Missing files from ZODB3 egg

Plone Support Deployment & hosting
1

I'm deploying a new Plone 4.3.11 server under Ubuntu 16.04.2 using Python 2.7.12 and I'm facing a very weird issue: when the ZODB3 egg is uncompressed all non-Python files are missing (that is all TXT and XML files inside that egg) and that is causing the following error:

$ bin/zeoserver fg
Traceback (most recent call last):
  File "bin/zeoserver", line 37, in <module>
    + sys.argv[1:]))
  File "/opt/plone/buildout/eggs/plone.recipe.zeoserver-1.2.9-py2.7.egg/plone/recipe/zeoserver/ctl.py", line 35, in main
    zeoctl.main(args)
  File "/opt/plone/buildout/eggs/ZODB3-3.10.5-py2.7-linux-x86_64.egg/ZEO/zeoctl.py", line 28, in main
    zdaemon.zdctl.main(args, options)
  File "/opt/plone/buildout/eggs/zdaemon-2.0.7-py2.7.egg/zdaemon/zdctl.py", line 683, in main
    options.realize(args)
  File "/opt/plone/buildout/eggs/zdaemon-2.0.7-py2.7.egg/zdaemon/zdctl.py", line 106, in realize
    RunnerOptions.realize(self, *args, **kwds)
  File "/opt/plone/buildout/eggs/zdaemon-2.0.7-py2.7.egg/zdaemon/zdoptions.py", line 290, in realize
    self.load_schema()
  File "/opt/plone/buildout/eggs/zdaemon-2.0.7-py2.7.egg/zdaemon/zdoptions.py", line 338, in load_schema
    self.schema = ZConfig.loadSchema(self.schemafile)
  File "/opt/plone/buildout/eggs/ZConfig-2.9.3-py2.7.egg/ZConfig/loader.py", line 33, in loadSchema
    return SchemaLoader().loadURL(url)
  File "/opt/plone/buildout/eggs/ZConfig-2.9.3-py2.7.egg/ZConfig/loader.py", line 65, in loadURL
    r = self.openResource(url)
  File "/opt/plone/buildout/eggs/ZConfig-2.9.3-py2.7.egg/ZConfig/loader.py", line 104, in openResource
    self._raise_open_error(url, e.reason)
  File "/opt/plone/buildout/eggs/ZConfig-2.9.3-py2.7.egg/ZConfig/loader.py", line 120, in _raise_open_error
    url)
ZConfig.ConfigurationError: error opening file /opt/plone/buildout/eggs/ZODB3-3.10.5-py2.7-linux-x86_64.egg/ZEO/zeoctl.xml: [Errno 2] No such file or directory: '/opt/plone/buildout/eggs/ZODB3-3.10.5-py2.7-linux-x86_64.egg/ZEO/zeoctl.xml'

sounds like an issue with the egg packaging and a new version of setuptools, or maybe an issue with the virtualenv.

if I check for setuptools version in the buildout script I see nothing:

$grep setuptools bin/buildout

in other servers I see something like this:

$ grep setuptools bin/buildout 
  join(base, 'eggs/setuptools-18.0.1-py2.7.egg'),

any hints?

2

I can confirm that this happens as soon as one uses setuptools 35.0.0.

Minimal buildout config:

[buildout]
parts = test

[versions]
ZODB3=3.10.5

[test]
recipe=zc.recipe.egg
eggs = ZODB3

Changes in setuptools: https://github.com/pypa/setuptools/issues/436

My suspicion here is that we are seeing a bug in setuptools. My further suspicion here is that the settings in setup.py from ZODB3 trigger the creation of a manifest file and that this manifest file is now being ignored.

Looking through the issue mentioned above it seems that the one fixing the bug did not understand the old behavior and was hoping for a code review, which he didn't get.

No more plans from my side to dig deeper, since I am not affected by it and know it will probably take half a day that I don't have.

2 Likes
3

thanks, a new issue has been opened: https://github.com/pypa/setuptools/issues/1016

the question for me is how can I avoid using setuptools 35.0.0? should I run virtualenv again with a different set of options? what I did was a simple:

$ virtualenv env27

should I use the --no-setuptools directive?

1 Like
4

I was able to workaround this issue:

  • stop any service using Python from the virtualenv
  • remove all eggs from the eggs directory
  • remove the virtualenv
  • create a new virtualenv using:
    virtualenv --no-setuptools env27
  • bootstrap your buildout again, in my case:
    python bootstrap.py --version=2.4.0 --setuptools-version=18.0.1
  • run buildout again

now everything is working as expected.

2 Likes
5

Thanks @hvelarde and @do3cc,
I feel like I'm getting mixed signals about setuptools, I've seen people who encourage pinning but I've seen situations where I've been told to use the latest version always which has been my strategy for a year or two now. Then this happens....

6

This looks complicated.

Here is what I do:

  1. Create normal virtualenv
  2. pip install zc.buildout
  3. pip install specific older version of setuptools (Only for now, it has been a while that setuptools broke buildout)

For projects, I have a requirements.txt file.

I suggested in the past to make recommend pip installing zc.buildout the recommended way, but @tisto rightfully suggested to bring this recommendation upstream: https://github.com/buildout/buildout/issues/289
I started to work on this but stopped after I realized it will be days to make tests work again.

1 Like
7

The problem is that there is no official API to use the packaging tools in python. As there is no official API zc.buildout is forced to use internal APIs and setuptools has no obligation to make changes to their internal API backward compatible.

In this specific case they just made an error independent of zc.buildout. I'd watch out for the bug report @hvelarde created and then go back to use the latest version of setuptools.

8

So the mode of operation is:
Use the latest setuptools, if it breaks things then pin to the last stable one until a new release that works?

This is my best understanding of the current setuptools situation :frowning:

9

seems the bug was fixed, if anybody wants to give a try:

2 Likes
10

@hvelarde thanks for the update! I just ran the Unified Installer for 5.0.7 and can confirm that it works now.

1 Like