Buildout.coredev broken? Not good for new developers

This is especially important for onboarding new developers for summer of code. Background, I'm running a workshop this week to introduce a few newbies to working with coredev.

Running these commands to get started:

git clone -b 5.0 https://github.com/plone/buildout.coredev ./plone5devel
virtualenv --no-site-packages plone5devpy
cd plone5devel
../plone5devpy/bin/pip install -r requirements.txt
../plone5devpy/bin/buildout bootstrap
bin/buildout -v

Resulted in

  1. An infinite upgrade/restart loop with setuptools
  2. Unable to find lazr.restfulclient 0.13.3

The solution was to unpin setuptools in versions.cfg
and unpin lazr.restfulclient in tests.cfg. After that the build worked.
meaning set the setuptools line equal to nothing and same for the lazr.restfulclient line

in versions.cfg

setuptools =

in tests.cfg

lazr.restfulclient =

This would not be a good experience for a new developer,

Now when I run

bin/instance fg

I get the following traceback

Traceback (most recent call last):
  File "/home/ubuntu/workspace/plone5devel/parts/instance/bin/interpreter", line 278, in <module>
    exec(compile(__file__f.read(), __file__, "exec"))
  File "/home/ubuntu/workspace/plone5devel/eggs/Zope2-2.13.23-py2.7.egg/Zope2/Startup/run.py", line 76, in <module>
    run()
  File "/home/ubuntu/workspace/plone5devel/eggs/Zope2-2.13.23-py2.7.egg/Zope2/Startup/run.py", line 17, in run
    import Zope2.Startup
  File "/home/ubuntu/workspace/plone5devel/eggs/Zope2-2.13.23-py2.7.egg/Zope2/__init__.py", line 60, in <module>
    from Zope2.Startup.run import configure
  File "/home/ubuntu/workspace/plone5devel/eggs/Zope2-2.13.23-py2.7.egg/Zope2/Startup/__init__.py", line 28, in <module>
    from zope.processlifetime import ProcessStarting
ImportError: No module named processlifetime

I'm almost sure I shouldn't fix it this way, but since I saw that the zope.processlifetime package was missing I ran the following command:

../plone5devpy/bin/pip install zope.processlifetime

It installed the egg and then I was able to successfully launch Plone with

bin/instance fg

Yes. I fixed this 2 weeks ago but @jensens broke it again by pinning a different setuptools version in versions.cfg than is installed via requirements.txt, despite the documentation comment.

Please make sure the versions in these two files are in sync and that at least resolves the infinite loop you're getting.

Also I noted that Python gdbm is not listed as a dependency. I was not able to install Plone without first installing python-gdbm on Ubuntu.

 sudo apt-get install python-gdbm

:frowning: and I was the one who introduced requirements.txt ...

1 Like

That happens if you introduce a second way of doing things without removing the first way. Usually Jenkins would have catched an error like this but since Jenkins uses the old bootstrap.py approach this did not happen. We could of course test both ways. Though, having two options for this (and testing both) is just nuts in the first place IMHO...

indeed, the world is not perfect especially if it come to things like pip/ setuptools and buildout/ bootstrap.py.
Fixing buildout to not loop would be the real solution.

So I'm introducing some new devs to Plone on Friday.

Before they can get started I will need to walk them through the following:

  1. Fixing setuptools to make sure we don't get into an upgrade/restart loop
  2. Fixing lazr.restfulclient
  3. Pip installing missing packages

I suppose this is a good learning experience for them, but it may also make it look harder than it needs to be.

These devs will have the benefit of some hand holding, but it is sure to scare away developers who won't have such hand holding.

I just synced both - version.cfg and requirements.cfg - with an update to latest setuptools 20.2.2.

Here it works w/o an extra pip install step.

Edit:
zope.processlifetime is a dependecy of Zope2 itself and should get pulled in while buildout.

Edit 2:
lazr.restfulclient does not seems to be very well maintained at PyPI at all and lives primary on Launchpad. You may need to add an entry to find-links in buildout.

What is in the way to make requirements.txt to be the new default?
We are using it for several months and it has a few advantages together with our CI and internal best practices.

  • A few checkers can be installed without running buildout, these checks finish/fail much faster in CI
  • A manual check if a project uses an up to date buildout and setuptools is faster

Even I faced the same problem while I am trying to setup dev environment for an add-on.

we tend to use a requirements.txt at least for setuptools and buildout eggs. its more reliable.

I have fixed the lazr.restfulclient problem. The problem arose this weekend after I merged release.cfg into core.cfg. Well, not sure actually.

Anyway, explanation is in this commit:

1 Like

in fact, is a very bad idea to have the same information in two different places because it leads to data inconsistency (see, @svx?).

so, the idea is to replace buildout's [versions] part with this requirements.txt file because is more pythonic?

@do3cc if you guys want to push this forward I would like to see it documented somewhere.

This thread is not about replacing buildouts version section with a requirements.txt.
There is a discussion in a bug on zc.buildout about this, but not here.
Here we talk about replacing bootstrap.py with requirements.txt as a possible new default.

The requirements.txt is just for bootstrapping zc.buildout, so it would contain version pins for setuptools and for zc.buildout.

Then
python bootstap.py bin/buildout
becomes
pip install -r requirements bin/buildout

One more point for requirements.txt vs bootstrap:


So grad I don't suffer this any more.

Okay the instructions at http://docs.plone.org/develop/coredev/docs/intro.html#getting-started-with-development now work as advertised. Thanks @jensens and @mauritsvanrees and the others who worked on this.

I think I'm starting to like the idea of

pip install -r requirements
bin/buildout

What would be in requirements.txt then? Just zc.buildout and setuptools or all the dependencies and version pins? If we would do that wouldn't it make sense to make this the default in zc.buildout itself as well? I'm concerned that if people look into the zc.buildout docs they will find different information than in buildout.cordev docs. This will cause even more confusion...we are adding layers over layers and hope to make things simpler by this...

1 Like