Installing Plone with just pip (without buildout)

Let's not let this block 5.2. To make this officially supported requires possibly a sprint worth of work:

  • pip branch of z3c.autoinclude should be released somehow
  • use includeDependencies-directive in Plone should probably be deprecated and it should be removed from the few remaining core packages using it; currently pip installed Plone only starts when that directive is disabled by special environment variable
  • full path for site.zcml should be configurable in zope.conf (now Zope2 loads it from instancehome/etc)
  • scripts generated by plone.recipe.zope2instance and plone.recipe.zeoserver need a new home (now they are in my branch of plonecli)
  • we need tool for generating site.zcml and zope.conf (e.g. templates bundled with plonecli)
1 Like

Early July good for you?

3 Likes

Just an update to this, I made a comment on this ticket https://github.com/plone/plone.docker/issues/89 with some work we did on this for creating a Docker image that does not bundle a packaging tool in a packaging tool (no more buildout).

I'd be interested in seeing if there is interest in getting the pip install Plone story moved further along as it will pave the way to nice Docker images.

1 Like

Join us at the Alpine City Sprint :smiley: it is a topic there! https://alpinecity.tirol/2020

1 Like

As @jensens noticed yesterday, it might be a time to get rid of z3c.autoinclude. It's better to be explicit in including configuration. And as a bonus, we are closer to pip compatibility.

As an update, the current status:

  • Plone 5.2 installs and starts fine with pip when Zope configuration and instance scripts are provided somehow and site.zcml has <meta:provides feature="disable-autoinclude" />

  • Some time ago, I forked plonectl https://github.com/datakurre/plonectl from plonecli and extracted instance script from plone.recipe.zope2instance; Unfortunately, that was before WSGI, so currently plonectl is only able to start ZServer based Plone 5.2 instances (installed with pip).

  • A floating idea has been that plonecli could include or use code from plone.recipe.zope2instance for generating instance configuration without buildout. Or should that be also part of plonectl?

(For myself, we are very late in Python 3 game, but we do have Nix-based Zope instance configuration generators waiting, also packaging Plone with Nix [pip style without buildout]. We'll go forward with my TxZServer and plonectl so I don't need to resolve WSGI related plone.recipe.zope2instance issues for pip.)

1 Like

I tweeted this, if someone wonders where I wrote about it. In a sense to make all IT I use greener I wrap a lot Py-Spy arround all long running Python processes. Maybe startup time is not that important, but it feels now much better.

The essence is, startup time of a complex (addon-rich) site came down from ~16-17sec to 6-7 seconds.

So after all, if we remove z3c.autoinclude and include ZCML dependencies explicit, we get a pip installable Plone. Plone 5.2 is already pip installable as long one does not use addons with z3c.autoinclude.
Reason: z3c.autoinclude does not work with wheels. It's complex discovery of matching pip/pypi package names to dotted paths uses some egg-specific heuristics and just does not work with wheels. This mechanism is also very slow. I do not see why we need to support it if some explicit line what to load just does the job.

So, in the plonectl it would be awesome to just have some configurations like where I have a list of addons with parameters and an zcml=true as one of those:

[addons]
collective.easyform zcml=true

or something like this.

3 Likes

As @jensens noticed yesterday, it might be a time to get rid of z3c.autoinclude. It's better to be explicit in including configuration. And as a bonus, we are closer to pip compatibility.

We should remove it from bobtemplates.plone as well (example: https://github.com/plone/bobtemplates.plone/blob/e7a8de972521f5abd3b2fa74b346df8c04d481f5/bobtemplates/plone/addon/setup.py.bob).

3 Likes

I am also interested in getting Plone installed just in pip instead of buildout. I'm not looking to get rid of buildout per se, I just want to have the python environment regulated by pip and let buildout do other things. This may end up being a bigger ask than just going completely without buildout.

The reason for my interest is the python community as a whole has increasingly relied on venv/pyenv/pipenv/whatever as a given in any project. This historically caused problems with tools like Pycharm Good news for pycharm users where buildout support was de-prioritized because it represents such a small part of the python community. But there are other tools I've been looking at like ossaudit and safety that just flat out only work with a venv.

It looks like a lot of work has been done already since this issue was created. I was able to create a test environment with everything pip installed just fine (I don't know what the issue with z3c.autoinclude was or if that's still a concern). As others mentioned, mr.developer is still really nice for managing several dev src eggs and I'm comfortable using it to create several parts/executables.

As I test I took at pip installed Plone and a barebones buildout with just a plone.recipe.zope2instance egg. Every egg this needs gets added to the eggs/ section though, even if it's already in the venv. I don't really know the details of this, but I would guess this would be a problem with any recipe that uses zc.recipe.egg. Do we need to abandon these eggs and use solutions in other tools like Asko's plonecli changes?

If your pip-installed Plone-environment started without issues, then this is no longer an issue.

AFAIK, with pip, buildout is only required because plone.recipe.zope2instance

  1. creates Zope application configuration
  2. creates instance scripts

Neither of these really require buildout as such, but it will be hard to extract these from the recipe without requiring maintaining the same knowledge in two different places.

Creating a simple Zope application configuration for Plone is not really that hard, and could be possible with mrbob or cookiecutter templates. Implementing template with all options from the recipe could be hard.

I consider the latter issue more important. From pip perspective it is strange that starting Plone requries buildout generated scripts. I'm haven't actually tried running Plone without the code in the recipe (that I copied into plonecli). Running instance might be possible without it, but debugging and executing scripts requires is not without replicating code from the recipe. I wonder, if it would be possible to extract scripts related code from the recipe into something like plonecli (but without dependency on click) and then use that in the recipe to avoid maintaining the same code twice...

Disclaimer: This is not official.
A naked Plone 5.2.2 installs and runs with pip install only, the same way Zope 4+ does.

python3 -m venv py
./py/bin/pip install Plone Paste -c https://dist.plone.org/release/5.2.2/constraints.txt
./py/bin/mkwsgiinstance -d .
./py/bin/runwsgi -v etc/zope.ini

Note, any addon using z3c.autoincludes <includeDependencies package="." /> wont work.
The other part, loading an addons zcml on startup, works (CMFPlone uses <includeDependencies package="." />). That said this may lead to problems. It is not well tested and z3c.autoinlcude is known to have not full working wheel support.

For example I did a pip install collective.easyform and started runwsgi again. The plugin appears in the add-ons section and works as expected.

For Plone 6 it is planned to fully support pip based installs. Any help is appreciated to make this true!

Further notes:

  • @mauritsvanrees is working on plone.autoinclude as an light weight replacement for z3c.autoinclude.
  • @datakurre wrote a script replacing plone.recipe.zope2instance which needs some attention and love.
  • it would be great to have egg-less buildout (or a pip based replacement) for all the recipes and extensions we love (this one probably needs a lot of discussion).
  • all this needs documentation
3 Likes

I updated all of my packages to use the below in setup.py awhile ago. This seems to work when just pip installed

entry_points="""
      # -*- entry_points -*-
      [z3c.autoinclude.plugin]
      target = plone
      """,

Where is @datakurre's script? I would like to check it out

Nothing special in them. I took scripts from plone.recipe.zope2instance, placed them into a plonecli branch, but after recommendations by others, renamed it to plonctl and remove other plonecli commands.

But since I did this a long time ago, this is probably missing wsgi support. I've only used it myself with Plone 5.2.1, Python 3 and my ZServer fork, but that polishing that path remains on my TODO. (I'm returning to a Plone project soon, so I'll try to complete my work on that.)

I'd recommend trying out Jens' unofficial Zope 4+ path at first. It's probably enough for just running Plone. I've been used to use debug and script running features of the recipe scripts and therefore had to extract those from the recipe package for my use.

I'm using buildout to generate the list of the requirements and then per installation I use it to generate Nix packages of the Plone/Zope scaffolding, configuration and instance script, See https://gitlab.com/metapensiero/plone-nix

1 Like

my 20 cents is that pipenv is an unmantained project, we should look for poetry instead :wink:

1 Like

Nice! I have not tried mach-nix yet by myself. Maybe I have to if upgrading to 5.2.2 gets too hard with my old tools :slight_smile:

But now that someone else is also using nix, did you look into my zconfig-generator https://github.com/datakurre/plone-nix/tree/master/zconfig before choosing to use simple string templates?

3 Likes

I've been testing the installation of Plone with pip and the biggest lesson so far is to pin pip to a version lower than 20.3. This version of pip introduced backtracking for dependency resolution and it makes it close to impossible to install Plone.

The following snippet works like a charm:

python -m venv . 
./bin/python -m pip install "pip==19.2.3"
./bin/pip install -U setuptools wheel 
./bin/pip install Plone Paste -c https://dist.plone.org/release/5.2.4/constraints.txt

While running the following snippet will prove the case (with pip install "hanging" after downloading all packages and creating the wheels)

python -m venv . 
./bin/pip install -U pip setuptools wheel 
./bin/pip install Plone Paste -c https://dist.plone.org/release/5.2.4/constraints.txt
2 Likes

Confirmed. For now, it works even with latest pip when using the legacy resolver:

./bin/pip install Plone Paste -c https://dist.plone.org/release/5.2.4/constraints.txt --use-deprecated legacy-resolver

Since the constraints have an older setuptools, I thought I would try it like this, and also install only Products.CMFPlone:

./bin/pip install -U pip setuptools wheel -c https://dist.plone.org/release/5.2.4/constraints.txt
./bin/pip install Products.CMFPlone -c https://dist.plone.org/release/5.2.4/constraints.txt

But that stalls after a while too. Making it more verbose (-vvv) does not help.

When I first install everything with the legacy resolver, and then in the same venv install it with the standard resolver, pip still stalls.

Every package that we need should be in constraints.txt, so special resolvers should not be needed. Theoretically there might be an inconsistency in the constraints, but then pip should fail, and not try to determine if other versions are better.
I installed with the legacy resolver, then ran pip freeze and used the results as constraints file, but that still gave the same result.

I suppose the most likely cause is that there is some cyclical dependency in our packages.

Could you raise this in the pip issue tracker?

2 Likes

Another way to install Plone would be to get all the wheels and then install them from the wheelhouse, but without installing dependencies.

./bin/pip wheel Plone Paste -c https://dist.plone.org/release/5.2.4/constraints.txt -w ~/.cache/wheelhouse
./bin/pip pip install --force-reinstall --no-index --no-deps ~/.cache/wheelhouse

But this seems to have undesirable side effects, in one of my tests, I noticed that zope-testrunner command was not created (for a package that required plone.app.testing).

I do plan to dig a bit more over the weekend to try to pin where the problem occurs, then open an issue on pypa.

1 Like

@mauritsvanrees, not sure this is the main culprit, but we sure need to tackle it:

An analysis using pipdeptree of a Plone 5.2.4 installation with pip returns the following report:

Warning!! Cyclic dependencies found
* Products.CMFPlone => plone.app.content => Products.CMFPlone
* Products.CMFPlone => plone.app.contentlisting => Products.CMFPlone
* Products.CMFPlone => plone.app.contentmenu => Products.CMFPlone
* Products.CMFPlone => plone.app.contentrules => Products.CMFPlone
* Products.CMFPlone => plone.app.contenttypes => Products.CMFPlone
* Products.CMFPlone => plone.app.dexterity => Products.CMFPlone
* Products.CMFPlone => plone.app.discussion => Products.CMFPlone
* Products.CMFPlone => plone.app.layout => Products.CMFPlone
* plone.app.portlets => plone.app.layout => plone.app.portlets
* Products.CMFPlone => plone.app.multilingual => Products.CMFPlone
* Products.CMFPlone => plone.app.portlets => Products.CMFPlone
* plone.app.layout => plone.app.portlets => plone.app.layout
* plone.app.vocabularies => plone.app.querystring => plone.app.vocabularies
* Products.CMFPlone => plone.app.registry => Products.CMFPlone
* Products.CMFPlone => plone.app.theming => Products.CMFPlone
* Products.CMFPlone => plone.app.users => Products.CMFPlone
* plone.app.querystring => plone.app.vocabularies => plone.app.querystring
* plone.app.z3cform => plone.app.widgets => plone.app.z3cform
* Products.CMFPlone => plone.app.workflow => Products.CMFPlone
* plone.app.widgets => plone.app.z3cform => plone.app.widgets
* plone.app.workflow => Products.CMFPlone => plone.app.workflow
* plone.app.users => Products.CMFPlone => plone.app.users
* plone.app.theming => Products.CMFPlone => plone.app.theming
* plone.app.registry => Products.CMFPlone => plone.app.registry
* plone.app.portlets => Products.CMFPlone => plone.app.portlets
* plone.app.multilingual => Products.CMFPlone => plone.app.multilingual
* plone.app.layout => Products.CMFPlone => plone.app.layout
* plone.app.discussion => Products.CMFPlone => plone.app.discussion
* plone.app.dexterity => Products.CMFPlone => plone.app.dexterity
* plone.app.contenttypes => Products.CMFPlone => plone.app.contenttypes
* plone.app.contentrules => Products.CMFPlone => plone.app.contentrules
* plone.app.contentmenu => Products.CMFPlone => plone.app.contentmenu
* plone.app.contentlisting => Products.CMFPlone => plone.app.contentlisting
* plone.app.content => Products.CMFPlone => plone.app.content

Should I open an issue on Products.CMFPlone?

1 Like

Plone Foundation Code of Conduct