Make buildout deferent again (to Pillow from my virtualenv)

Hey folks, having a problem with buildout that I have not seen before, and one that I suspect is due to newer setuptools. Is there a simple trick to get zc.buildout (2.9.3) and setuptools (33.1.1) to defer to the version of Pillow I have installed in my virtualenv (used to bootstrap my build)? This used to the be default, assuming that the version numbers matched, but does not seem to be now.

This is what I see resulting:

qi@qits2:~/upiq5/app$ ../python/python-2.7/bin/python  # virtualenv Python, was used to bootstrap buildout
Python 2.7.13 (default, Mar 30 2017, 22:02:05)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import PIL
>>> PIL
<module 'PIL' from '/home/qi/upiq5/python/parts/opt/lib/python2.7/site-packages/Pillow-3.3.1-py2.7-linux-x86_64.egg/PIL/__init__.pyc'>
>>>
qi@qits2:~/upiq5/app$ ./bin/zopepy  # generated scripts inside my buildout use this:

>>> import PIL
>>> PIL
<module 'PIL' from '/home/qi/upiq5/app/eggs/Pillow-3.3.1-py2.7-linux-x86_64.egg/PIL/__init__.pyc'>
>>>

Thoughts?

Edit/update: this is using collective/buildout.python, and it matters because the Pillow built there seems to be working, this is on a system (for the moment) without libjpeg installed by system package.

Edit #2: it appears I may be incorrect about this being the previous behavior, but I am still interested in pointers on means to make buildout deferent to a distribution from virtualenv and/or system python?

So far, it appears that I may have fond nostalgia for control I am remembering from and only previously found in zc.buildout 1.x. Is there a way to do this in builout 2.x?

Really strange, this seems like you runned your buildout without a virtualenv (with Pillow already installed) somewhere in time and it downloaded as an egg. I know this is a bit obvious, but what's the python shebang inside "./bin/zopepy"?

Just a thought, why are you using Pillow installed in site-packages instead of relying only on buildout itself? Does buildout.python does this?

It points to the virtualenv built using collective/buidlout.python for Python 2.7. That virtualenv has Pillow installed (buildout invokes pip, not directly installing egg), correctly built against libjpeg.

Because on Ubuntu 16.04, there are numerous reasons why Pillow 3.3.0 built by buildout fails to find libjpeg, and instances therefore fail to start. Buildout is having problems with Pillow >= 3.3.0, and the punting position right now is to pin 3.2.0 since buildout is not paying attention anymore to what's in my virtualenv, site-packages.

buildout.python builds libjpeg and links Pillow against it. The buildout.python Pillow 3.3.0+ works because it is installed by pip, not by buildout.

I think others in the community have seen related problems (also here) with Pillow on Ubuntu 16.04, including @hvelarde.

1 Like

yes, I experienced issues; I have no idea why, neither how I fixed them :slight_smile:

If that has any utility/importance, the nodes in jenkins.plone.org are installed with this ansible playbook targeting Ubuntu 16.04 that installs all dependencies for Pillow, and so far tests are passing :smile:

We just gave up and had buildout install Pillow 3.2.0, which for some reason does not have issues getting built (via buildout) with the libjpeg installed by default in Ubuntu 16.04.

I have a workaround that I suspect will work (collective.recipe.egg plus rpath usage), but have not tested. It may also be that we need to install some backward-compatibility packages via apt, but I have libjpeg8-dev installed.

Maybe I wait for buildout wheel support to mature, and this lack of deference to site-packages go away, but I'm not sure I should count on that.