Buildout to PyPI very slow

seanupton · August 10, 2023, 1:56pm

Buildout seems incredibly slow lately re: fetching from PyPI — moreso than I generally recall it being. I'm using Python 3.8.17 (via pyenv) on Ubuntu 22.04 with these in a venv:

setuptools==65.7.0
pip==20.3.4
wheel==0.38.4
zc.buildout==3.0.1

vs pip (quick) on same machine(s) (same venv), the wheel discovery process seems super-slow, enough that it takes over 40 minutes to populate a ~93MB download cache of a pretty generic Plone 5.2 build from fast internet.

I have not yet profiled this — but from watching log output, it seems the issue is not download of the package, but reading index and deciding what wheel to grab, or some other such thing surrounding the download but not the simple act of grabbing the wheel.

Any clues or direction on how I can diagnose/improve this situation (other than punt and preload a download cache)?

yurj · September 1, 2023, 7:47am

github.com/pypa/pip

pip rerun is very slow with _PIP_USE_IMPORTLIB_METADATA=1

opened 03:47PM - 08 Jun 23 UTC

closed 08:47AM - 19 Jun 23 UTC

mauritsvanrees

type: bug S: needs triage

### Description A pip install of a package with many dependencies, followed by …the same pip install command, is much slower on Python 3.11 than on 3.10. In my case instead of about 10 seconds, it takes almost 4 minutes. Investigation shows that this is a side effect of using `importlib_metadata` instead of `pkg_resources` on 3.11. Setting env variable `_PIP_USE_IMPORTLIB_METADATA=1` on 3.10 gives the same slowdown. Setting the env variable to zero makes 3.11 fast again. I have found where the problem is, and will prepare a PR. ### Expected behavior `pip install Plone -c constraints.txt` can take a while because it has lots of transitive dependencies. But when this command is finished and you run it again in the same venv, it should be much faster. Instead, it is almost four times slower than a fresh install. ### pip version 23.1.2 ### Python version 3.11.2 ### OS macOS Monterey 12.6.6, Intel ### How to Reproduce ``` python3.11 -mvenv py311 cd py311 bin/pip install -U pip setuptools wheel time bin/pip install Plone -c https://dist.plone.org/release/6.0.5/constraints.txt time bin/pip install Plone -c https://dist.plone.org/release/6.0.5/constraints.txt export _PIP_USE_IMPORTLIB_METADATA=0 time bin/pip install Plone -c https://dist.plone.org/release/6.0.5/constraints.txt ``` ### Output The first pip install of Plone takes about 1 minute. Longer if you do not have the wheels in your cache. The second takes about four minutes. You will see a lot of lines with "Requirement already satisfied". The first 100 or so go by fast, but then you start to see this slow down more and more. The third takes about 13 seconds. I did some measurements and have managed to narrow it down to [this line in SpecifierRequirement.is_satisfied_by](https://github.com/pypa/pip/blob/23.1.2/src/pip/_internal/resolution/resolvelib/requirements.py#L92). Total time spent on this line is about four seconds with `pkg_resources`. With `importlib_metadata` it takes 113 seconds! Narrowing it down further, the thing on this line that takes the most time, is simple attribute access: `candidate.version`. This is a [method of `AlreadyInstalledCandidate`](https://github.com/pypa/pip/blob/23.1.2/src/pip/_internal/resolution/resolvelib/candidates.py#L377-L379) and it gets `self.dist.version`. This should be calculated once and then cached, similar to what pip already does a bit higher in that file for [`_InstallRequirementBackedCandidate`](https://github.com/pypa/pip/blob/23.1.2/src/pip/_internal/resolution/resolvelib/candidates.py#L190-L194). When I make that change, the pip install command using importlib takes only 9 seconds. This is about 25 times faster than before! Note that this change also positively influences the case with `_PIP_USE_IMPORTLIB_METADATA=0`, so using `pkg_resources`: it takes about 5 seconds instead of 13. I will create a PR. ### Code of Conduct - [X] I agree to follow the [PSF Code of Conduct](https://www.python.org/psf/conduct/).

maybe this?

mauritsvanrees · September 6, 2023, 8:57pm

That is not a problem in the older pip 20 that Sean is using.

Still, did you try latest pip and wheel?
Maybe setuptools as well, but this could give problems if you need to install old packages with non-standard version numbers. Also, stay below 68.1 if you need editable installs.