Not using bootstrap.py as default?

With the new developments in setuptools, I suggest pinning to a known working set for setuptools and zc.buildout. The pip install suggested does not guarantee success.

I've introduced the following combination for us at GW20E: zc.buildout=2.2.5 and setuptools=7.0
Also this works most of the time:

python bootstrap.py -c myfile.cfg --version=2.2.5

Combined with pinning:

setuptools = 7.0
zc.buildout = 2.2.5

The setuptools could probably be a bit newer, but I personally have no clue why they are developing setuptools at the rate they are. Only know it causes A LOT of problems which personally pisses me off. Spend so much time on issues related to this.

2 Likes

The whole point is: bootstrap.py is old, busted, no longer needed.

If you want to pin initial versions use a requirements.txt containing:

setuptools==17.11.1
zc.buildout==2.3.1

and doing a

./bin/pip install -r requirements.txt

This is the same as done with bootstrap --version in a much more reliable way.

Plone 4.3.x + 5.0x works here for me (and our environment at least) perfect with both - setuptools and zc.buildout - on current latest release. at time of writing this: setuptools 17.11.1 an zc.buildout 2.3.1

I do not see any reason to use older versions.

2 Likes

I'm all for dropping bootstrap.py, but only if the solution we provide and document is stable.
If you want to document, document something that works always, not just with the latest.

How many version of Plone are out there? How many specific workingsets?
How many 'not so default' versions of packages people might use? (ie.elementtree 1.2.7-20070827-preview).
Assuming that the latest version you suggested work for all those is assuming several bridges too far :wink:

Or you have to make a clear distinction between Plone > 4.3.X and < 4.3.X in documentation.

i fully agree. I'd introduce this officially with Plone 5 only, even if it works with 4.3.x. But with 5 its save to not break things accidentally I'd say.

I have been in setuptools/buildout hell many times, I can feel the pain and I would love to see a way out. With this proposal we are going to remove one of the components involved (bootstrap.py), which is good. Keeping bootstrap.py up-to-date is painful.

Though, we are not going to solve our setuptools/buildout problem with this, newer versions will still break things and I think there are quite a few disadvantages:

  1. We are going to introduce just another way of bootstrapping a buildout file. This will confuse people, since we will not be able to erase the old way of doing things in every single bit of our documentation. Some people started to use makefiles or even nix, which is even more confusing. I think we should only replace a solution if it offers a significant improvement.

  2. We will have a different approach for older Plone versions and Plone 5, this will confuse people.

  3. We are going to introduce a requirements.txt file that contains version pinning in addition to buildout.cfg/versions.cfg. We will have versions pinned in two different places. Instead of a bootstrap.py we will have a requirements.txt.

  4. The "buildout" command will not live in "bin/buildout" like all other buildout-like commands, this is confusing.

  5. Creating a virtualenv requires more typing than the two usual "bootstrap/buildout" steps. If you do this many times a day, this makes a difference (after trying the proposed approach for a few days this is actually my main concern :).

1 Like

ad 1) plone people are confused, python people are at home
ad 2) well, in Plone 2.5 we had no buildout at all, so change happens and is good if it improve things.
ad 3) indeed a problem. I'd love to see buildout using a requirements.txt instead of versions sections. this would unify things in python world a lot an reduce bloat of buildout coming from a time where pip just was not invented.
ad 4) wrong, its still in bin/buildout
ad 5) you can put in a bootstrap.sh script or (much better!) as @jaroel wrote just in a Makefile - this would unify things again, everybody in IT world knows makefiles (except some poor folks doing nothing else than Plone).

My far-away vision is to:

  • download a plone-5.0-requirements.txt
  • virtualenv .
  • ./bin/pip install -r plone-5.0-requirements.txt -r my-customer-specific-requirements.txt
  • ./bin/plone-create -c customer.cfg
  • ./bin/plone-customer start

or like so.

Just to get an idea what wrong with buildout in my opinion. Don't get me wrong, buildout is great (and I love this swiss knife), but way too much sometimes.

Re #4: it may be put in a location bin/buildout -- but I am under the impression that the virtualenv 'bin' dir is not always/often the same location as the 'bin' used in an application buildout?

All you need to know as a Plone developer right now is buildout:

All possible options discussed, introduce one or more additional concepts that people have to learn. Being more Pythonic is a worthwhile goal, and maybe even dropping buildout (though, I don't see how we could really replace that kind of functionality).

As long as we are not there, adding more concepts will just confuse people and at the same time not really improve the situation. The Python packaging hell will not freeze over just because we add a few more complexity layers. :wink:

Regarding bash scripts and make files: You are aware that you are proposing a solution to a problem that does not exist, if we stick with just buildout, right? Introducing a second layer of complexity because the first layer introduces a problem that would not exist without it, is never a good idea in my opinion.

1 Like

+100. couldn't agree more with @tisto

Seems to be the solution for now is to fix buildout rather than wrap it (for all the reasons @tisto mentions)

First: dont get the "far-away vision" too serious. I doubt we can implement it with our current stack. Maybe in Plone 10 :snail:

The major problem right now with buildout are version conflicts in several flavors. All the other stuff works great. So lets fix this and we have much less headache.

bootstrap.py is one source of such version conflicts, so getting rid of it first is a good start.

Next would be in my opinion to define the versions pip-style instead of in a version section. There are a bunch of implications then, so this need to be discussed in-depth and maybe I'am wrong.

The whole virtualenv can be put in an extra directory. I saw some folks doing a virtualenv ./venv or virtualenv ./python and so on. In that case all pip-installed scripts are living in ./venv/bin/* or ./python/bin/*.

I never do this and instead use virtualenv . (dot) - even if splitting up buildout generated scripts and virtualenv generated scripts is slightly cleaner.

On the other hand I found buildouts (in inherited customer projects) defining a custom bin-dir into the virtualenv - bin-dir in buildout is configurable too.

I prefer to have one central bin-dir for virtualenv and buildout together, even if there is a small possibility to override scripts installed by virtuenv with a buildout part generated script.

Let me note that we already have nearly all the problems that have been mentioned as a negative for replacing bootstrap.py. Everyone that is checking out an old buildout or trying to update an old site is running into the fact that bootstrap.py is broken without extraordinary workarounds.

So, we already need to do some documentation to help these poor souls out of their bootstrap/setuptools hell. At the moment, it looks easiest to me to document the virtualenv/buildout approach.

2 Likes

I've just found time to come back & read this. I want to say thanks to Jens - for the virtualenv idea & everyone for the very helpful open discussion. I think the best thing that can be put in the docs is a clear admission that this area gets difficult at times... that should keep new adopters with us, rather than giving up!

Btw yesterday I found the following command which seems like it would help with version conflicts when I get them (is this obvious, or not?) In this example for plone.resource, it shows me what other products require it & what pins they apply:

$ grep plone.resource  eggs/*.egg/EGG-INFO/requires.txt 
eggs/plone.app.theming-1.1.6-py2.7.egg/EGG-INFO/requires.txt:plone.resource>=1.0b5
eggs/plone.app.theming-1.1.6-py2.7.egg/EGG-INFO/requires.txt:plone.resourceeditor
eggs/plone.app.themingplugins-1.0b1-py2.7.egg/EGG-INFO/requires.txt:plone.resource
eggs/plone.resourceeditor-1.0-py2.7.egg/EGG-INFO/requires.txt:plone.resource

Worked fine for a while but ...

see

I think this was solved meanwhile.

Indeed, imissed to pin plone.recipe.zope2instance to 4.4.0 (was 4.3)
Sorry ....

@jensens :+1:

I has been many nightmares trying to fixed the correct pinnies for setuptools or zc.buildout packages every son often, because my buildout.py file is outdated.

I'm not having bootstrap.py problems for a while. The last version is on https://bootstrap.pypa.io/bootstrap-buildout.py (it's hasn't been updated recently though).

For people coming just now about the bootstrap.py discussion, don't forget to check https://github.com/plone/Products.CMFPlone/issues/2001

You can even skip step one and do bin/pip install -r https://raw.githubusercontent.com/plone/buildout.coredev/5.1/requirements.txt