Pypi Deprecation of support for non-SNI clients breaks buildout for older Plone versions

TL;DR: pypi has deprecated access for non-SNI clients. This is a problem when running buildout for old versions of Plone, because of two pieces that may be affected: setuptools and the python binary.

For the python binary, try this:

(Pdb) import ssl; print(ssl.HAS_SNI)
True

If it says False, you'll have to get a newer version of python. However, it seems even fairly old versions of 2.7 have SNI support.

I have tried a buildout for Plone 4.3.20 with both setuptools 36.8.0 and 44.1.1 and it worked fine. Methinks that it will now be impossible to run a buildout for versions of Plone that support Python 2.6 and not 2.7, though (at least without a complete local egg cache with all dependencies).

If you are affected by this deprecation you'll get the ambiguous

Couldn't find index page for 'xyz' (maybe misspelled?)
Getting distribution for 'xyz==vvv'.
Couldn't find index page for 'xyz' (maybe misspelled?)
While:
  Installing.
  Loading extensions.
  Getting distribution for 'xyz==vvv'.
Error: Couldn't find a distribution for 'xzy==vvv'.

(where xyz and vvv could be any package needed by your buildout).

If you are curious, you can set a breakpoint in lib/python2.7/site-packages/setuptools/package_index.py. This probably varies between versions of setuptools but in my case I had this code:

def open_url(self, url, warning=None):
    if url.startswith('file:'):
        return local_open(url)
    try:    
        return open_with_auth(url, self.opener)
    except (ValueError, http_client.InvalidURL) as v:
        msg = ' '.join([str(arg) for arg in v.args])
        if warning: 
            self.warn(warning, msg)
        else:   
            raise DistutilsError('%s %s' % (url, msg))
    except urllib.error.HTTPError as v:
        return v

and the v in the last line looks like this:

(Pdb) pp v.__dict__
{'__iter__': <bound method _fileobject.__iter__ of <socket._fileobject object at 0x7f51380a23d0>>,
 'code': 403,
 'fileno': <bound method _fileobject.fileno of <socket._fileobject object at 0x7f51380a23d0>>,
 'fp': <addinfourl at 139986809292912 whose fp = <socket._fileobject object at 0x7f51380a23d0>>,
 'hdrs': <httplib.HTTPMessage instance at 0x7f51380a1e60>,
 'headers': <httplib.HTTPMessage instance at 0x7f51380a1e60>,
 'msg': '[[[!!! BREAKING CHANGE !!!]]] Support for clients that do not support Server Name Indication is temporarily disabled and will be permanently deprecated soon. See https://status.python.org/incidents/hzmjhqsdjqgb and https://github.com/pypa/pypi-support/issues/978 [[[!!! END BREAKING CHANGE !!!]]]',
 'next': <bound method _fileobject.next of <socket._fileobject object at 0x7f51380a23d0>>,
 'read': <bound method _fileobject.read of <socket._fileobject object at 0x7f51380a23d0>>,
 'readline': <bound method _fileobject.readline of <socket._fileobject object at 0x7f51380a23d0>>,
 'readlines': <bound method _fileobject.readlines of <socket._fileobject object at 0x7f51380a23d0>>,
 'url': 'https://pypi.python.org/simple/whateverpackageyouweretryingtoinstall/'}

So then you can find out more at Python Infrastructure Status - Deprecation of support for non-SNI clients. and Deprecation of non-SNI compatible clients. · Issue #978 · pypa/pypi-support · GitHub.

4 Likes

Running an old version of setuptools causes multiple problems.

Additional, setuptools>=42.0.1 is necessary so buildout can download e.g. latest
versions of cryptography, see update setuptools requirements by jugmac00 · Pull Request #549 · buildout/buildout · GitHub

I created a PR to raise the minimum required setuptools version to 42.0.1 for buildout, but I did not get a positive review.

I still think this change would have saved many, many hours of debugging.

Ugh. It’s happening…

For users of Python 2.7.{0...8}

Upgrading to the last Python 2.7 release is an option.

However, note that Python 2.7 series itself is now End of Life and support in pip was dropped with version 21.0.

For users of Python 2.6.x and lower:

Neither the Python core developers, or pip maintainers support Python 2.6 and below.

If someone is aware of a work around for this issue (SNI support specifically) they are welcome to share it here for others.

There is no recommended solution from the PyPI team.

I suppose a workaround would be to create a temporary private pypi mirror and point the index line in buildout to it.

maybe devpi can help there: GitHub - devpi/devpi: Python PyPi staging server and packaging, testing, release tool

One of our customers has an internal PyPI server using devpi. To avoid having to update the setuptools version or switching away from bootstrap.py, I have now uploaded today's Plone security hotfix to this internal PyPI. The buildouts can find the package from there when they have http://pypi.customername/local/root/test/+findlinks in their buildout find-links. This needs the devpi-findlinks package on the server.
It should work, but don't really know yet.

Meanwhile, locally on my Mac bootstrap.py is completely broken, even when I let it use a locally downloaded ez_setup.py script which is fixed to use setuptools 42.0.2 instead of 33.1.1. Normally I create a virtualenv and use pip install setuptools zc.buildout, but I wanted to try the old way.

Another option is to use a local folder as a pypi repository:

[buildout]
find-links +=
    file://${buildout:directory}/pypi-local

of course you have to:

  • create the folder in the proper path
  • download the eggs there manually
4 Likes

I am having trouble upgrading an older Plone 4.3.18 installation running on SuSE SLES 12-SP4
I reinstalled Python 2.7.18 from scratch, I updated setuptools to 44.1.1 but I am getting this:

plone-test@oxygen:~/zchl_buildout> bin/buildout -c prod.cfg 
Getting distribution for 'mr.developer==1.38'.
While:
 Installing.
 Loading extensions.
 Getting distribution for 'mr.developer==1.38'.
Error: Wheels are not supported

However, installing any used package using pip of my virtualenv is working.

Any ideas?

Try pip install wheel==0.29.0.

I found that by running:

grep wheel `locate requirements.txt`

on my machine.
Maybe newer versions will work as well.