Installing Plone with just pip (without buildout)

Did you take a look at pipenv? I use it already for some buildouts locally, just travis has currently some problems I couldn't solve.

That can be easily accomplished with pipenv/pipfile as well :wink:

1 Like

includePlugins part of z3c.autoinclude (for entrypoint=plone) works fine also with pip

Is it worth a PLIP for Plone 5.2? We would need an official requirements.cfg and constraints.cfg, documentation and whats else?

2 Likes

I'm interpreting the negation as a typo - you mean like pip install -e git+https://github.com/plone/Products.CMFPlone.git@5.1rc2#Products.CMFPlone?

That does a source checkout egg develop into your virtualenv and you can then point your editor at it and hack at it as usual. You can use any @refspec (omitting this parametre means HEAD of default branch). Also can go into requirements.txt. The #eggname syntax is a bit weird, but it flags up a warning if you omit it, so it's probably there to make sure the eggname and reponame are not different from each other.

1 Like

Let's not let this block 5.2. To make this officially supported requires possibly a sprint worth of work:

  • pip branch of z3c.autoinclude should be released somehow
  • use includeDependencies-directive in Plone should probably be deprecated and it should be removed from the few remaining core packages using it; currently pip installed Plone only starts when that directive is disabled by special environment variable
  • full path for site.zcml should be configurable in zope.conf (now Zope2 loads it from instancehome/etc)
  • scripts generated by plone.recipe.zope2instance and plone.recipe.zeoserver need a new home (now they are in my branch of plonecli)
  • we need tool for generating site.zcml and zope.conf (e.g. templates bundled with plonecli)
1 Like

Early July good for you?

3 Likes

Just an update to this, I made a comment on this ticket https://github.com/plone/plone.docker/issues/89 with some work we did on this for creating a Docker image that does not bundle a packaging tool in a packaging tool (no more buildout).

I'd be interested in seeing if there is interest in getting the pip install Plone story moved further along as it will pave the way to nice Docker images.

1 Like

Join us at the Alpine City Sprint :smiley: it is a topic there! https://alpinecity.tirol/2020

1 Like

As @jensens noticed yesterday, it might be a time to get rid of z3c.autoinclude. It's better to be explicit in including configuration. And as a bonus, we are closer to pip compatibility.

As an update, the current status:

  • Plone 5.2 installs and starts fine with pip when Zope configuration and instance scripts are provided somehow and site.zcml has <meta:provides feature="disable-autoinclude" />

  • Some time ago, I forked plonectl https://github.com/datakurre/plonectl from plonecli and extracted instance script from plone.recipe.zope2instance; Unfortunately, that was before WSGI, so currently plonectl is only able to start ZServer based Plone 5.2 instances (installed with pip).

  • A floating idea has been that plonecli could include or use code from plone.recipe.zope2instance for generating instance configuration without buildout. Or should that be also part of plonectl?

(For myself, we are very late in Python 3 game, but we do have Nix-based Zope instance configuration generators waiting, also packaging Plone with Nix [pip style without buildout]. We'll go forward with my TxZServer and plonectl so I don't need to resolve WSGI related plone.recipe.zope2instance issues for pip.)

1 Like

I tweeted this, if someone wonders where I wrote about it. In a sense to make all IT I use greener I wrap a lot Py-Spy arround all long running Python processes. Maybe startup time is not that important, but it feels now much better.

The essence is, startup time of a complex (addon-rich) site came down from ~16-17sec to 6-7 seconds.

So after all, if we remove z3c.autoinclude and include ZCML dependencies explicit, we get a pip installable Plone. Plone 5.2 is already pip installable as long one does not use addons with z3c.autoinclude.
Reason: z3c.autoinclude does not work with wheels. It's complex discovery of matching pip/pypi package names to dotted paths uses some egg-specific heuristics and just does not work with wheels. This mechanism is also very slow. I do not see why we need to support it if some explicit line what to load just does the job.

So, in the plonectl it would be awesome to just have some configurations like where I have a list of addons with parameters and an zcml=true as one of those:

[addons]
collective.easyform zcml=true

or something like this.

3 Likes

As @jensens noticed yesterday, it might be a time to get rid of z3c.autoinclude. It's better to be explicit in including configuration. And as a bonus, we are closer to pip compatibility.

We should remove it from bobtemplates.plone as well (example: bobtemplates.plone/bobtemplates/plone/addon/setup.py.bob at e7a8de972521f5abd3b2fa74b346df8c04d481f5 · plone/bobtemplates.plone · GitHub).

3 Likes

I am also interested in getting Plone installed just in pip instead of buildout. I'm not looking to get rid of buildout per se, I just want to have the python environment regulated by pip and let buildout do other things. This may end up being a bigger ask than just going completely without buildout.

The reason for my interest is the python community as a whole has increasingly relied on venv/pyenv/pipenv/whatever as a given in any project. This historically caused problems with tools like Pycharm Good news for pycharm users where buildout support was de-prioritized because it represents such a small part of the python community. But there are other tools I've been looking at like ossaudit and safety that just flat out only work with a venv.

It looks like a lot of work has been done already since this issue was created. I was able to create a test environment with everything pip installed just fine (I don't know what the issue with z3c.autoinclude was or if that's still a concern). As others mentioned, mr.developer is still really nice for managing several dev src eggs and I'm comfortable using it to create several parts/executables.

As I test I took at pip installed Plone and a barebones buildout with just a plone.recipe.zope2instance egg. Every egg this needs gets added to the eggs/ section though, even if it's already in the venv. I don't really know the details of this, but I would guess this would be a problem with any recipe that uses zc.recipe.egg. Do we need to abandon these eggs and use solutions in other tools like Asko's plonecli changes?

If your pip-installed Plone-environment started without issues, then this is no longer an issue.

AFAIK, with pip, buildout is only required because plone.recipe.zope2instance

  1. creates Zope application configuration
  2. creates instance scripts

Neither of these really require buildout as such, but it will be hard to extract these from the recipe without requiring maintaining the same knowledge in two different places.

Creating a simple Zope application configuration for Plone is not really that hard, and could be possible with mrbob or cookiecutter templates. Implementing template with all options from the recipe could be hard.

I consider the latter issue more important. From pip perspective it is strange that starting Plone requries buildout generated scripts. I'm haven't actually tried running Plone without the code in the recipe (that I copied into plonecli). Running instance might be possible without it, but debugging and executing scripts requires is not without replicating code from the recipe. I wonder, if it would be possible to extract scripts related code from the recipe into something like plonecli (but without dependency on click) and then use that in the recipe to avoid maintaining the same code twice...

Disclaimer: This is not official.
A naked Plone 5.2.2 installs and runs with pip install only, the same way Zope 4+ does.

python3 -m venv py
./py/bin/pip install Plone Paste -c https://dist.plone.org/release/5.2.2/constraints.txt
./py/bin/mkwsgiinstance -d .
./py/bin/runwsgi -v etc/zope.ini

Note, any addon using z3c.autoincludes <includeDependencies package="." /> wont work.
The other part, loading an addons zcml on startup, works (CMFPlone uses <includeDependencies package="." />). That said this may lead to problems. It is not well tested and z3c.autoinlcude is known to have not full working wheel support.

For example I did a pip install collective.easyform and started runwsgi again. The plugin appears in the add-ons section and works as expected.

For Plone 6 it is planned to fully support pip based installs. Any help is appreciated to make this true!

Further notes:

  • @mauritsvanrees is working on plone.autoinclude as an light weight replacement for z3c.autoinclude.
  • @datakurre wrote a script replacing plone.recipe.zope2instance which needs some attention and love.
  • it would be great to have egg-less buildout (or a pip based replacement) for all the recipes and extensions we love (this one probably needs a lot of discussion).
  • all this needs documentation
3 Likes

I updated all of my packages to use the below in setup.py awhile ago. This seems to work when just pip installed

entry_points="""
      # -*- entry_points -*-
      [z3c.autoinclude.plugin]
      target = plone
      """,

Where is @datakurre's script? I would like to check it out

Nothing special in them. I took scripts from plone.recipe.zope2instance, placed them into a plonecli branch, but after recommendations by others, renamed it to plonctl and remove other plonecli commands.

But since I did this a long time ago, this is probably missing wsgi support. I've only used it myself with Plone 5.2.1, Python 3 and my ZServer fork, but that polishing that path remains on my TODO. (I'm returning to a Plone project soon, so I'll try to complete my work on that.)

I'd recommend trying out Jens' unofficial Zope 4+ path at first. It's probably enough for just running Plone. I've been used to use debug and script running features of the recipe scripts and therefore had to extract those from the recipe package for my use.

I'm using buildout to generate the list of the requirements and then per installation I use it to generate Nix packages of the Plone/Zope scaffolding, configuration and instance script, See https://gitlab.com/metapensiero/plone-nix

1 Like

my 20 cents is that pipenv is an unmantained project, we should look for poetry instead :wink:

1 Like