Installing Plone with just pip (without buildout)

Development Roadmap and Future
1

For me it seems that, a story for installing Plone with just pip is quite close (thanks to many things prior to this).

I did an example repository with the latest Plone requirements and minimal Plone instance skeleton

git clone https://github.com/datakurre/plone-pip
cd plone-pip

All required packages should install nicely with just pip install on a fresh Python 2.7 virtual env

virtualenv env
env/bin/pip install -U pip setuptools
env/bin/pip install -r requirements-5.1rc2.txt -r requirements.txt

I still required buildout to extracted those requirements using buildout.requirements from Plone 5.1.rc2-pending versions. In addition, I needed to

  • Disable the default z3c.autoinclude and replace it with the branch with pip compatibility.
  • Add my branch of plonecli that includes compatible instance, zeoserver and zeopack scripts I extracted from plone.recipe.zope2instance and plone.recipe.zeoserver.

In addition, the repository contains

  • ./etc/site.zcml, because currently Zope2 is hardcoded to load site.zcml from {config.instancehome}/etc/site.zcml (it would be easy fix into Zope2 to make that fully configurable)
  • minimal ./zope.conf that configures $PWD as instancehome and var (maybe not best practice, but simple for demo; also I could imagine creating custom zope.conf from bobtemplate using plonecli); also <includeDependencies /> directive (the evil twin of good <includePlugins />) is disabled with an environment variable in zope.conf

After installation, Plone can be started in foreground with running a simple script for initializing the admin user and then starting the instance

env/bin/plonecli instance -C zope.conf run admin.py
env/bin/plonecli instance -C zope.conf fg

But that's just the start for the fun. Installing add-ons is simply pip install and restarting the instance

env/bin/pip install collective.easyform
env/bin/pip install plone.app.imagecropping

And creating a new add-on is just an another call to plonecli and python setup.py develop to install the add-on into the virtualenv

source env/bin/activate
plonecli create addon my.addon
cd my.addon
python setup.py develop

I admit that pip may not ever replace existing professional buildout based development and deployment toolchains, but what are the reasonse why we shouldn't polish and promote this approach for new users?

4 Likes
2

Can we see all the commands you ran? I think you must have skipped over the part that installs Plone :slight_smile:

My first question coming to this is why do you think buildout is bad that it needs to be avoided?

Re: new users, I think an encapsulating installer rather than a command line method is going to attract more novices. I guess it depends if by 'new users' you mean 'new developers'. I also think the Plone Docker is a super easy way to get novices going with a local instance of Plone. Decision makers evaluating Plone are an entirely different category; for those, we need to promote live demo instances as we do on Plone CMS: Open Source Content Management — Plone: Enterprise Level CMS - Free and OpenSource - Community Driven - Secure

3

OK that is pretty slick.

On the other hand, it reminds me of easy_install. I assume there is a reason why we stopped adding add-ons via easy_install. Does your new method avoid those gotchas?

4

That is very nice! Being compatible with default Python installation mechanisms is definitely a big win when it comes to explain Plone to non-Plone users. That might even replace buildout on simpler deployment projects.

Still, buildout has it's advantages. I wouldn't miss mr.developer to be able to check out any repository while developing.

Does installing via pip mean that no C-optimized but only the Python fallback code is used? (e.g. AccessControl, Acquisition, etc)

5

I didn't say it's bad (well, it is pretty slow nowadays). Personally, I'm investigating buildoutless build for building containers (either with Docker or with Nix), and it would help also me if there were official requirements.txts maintained :slight_smile:

I skipped that era :frowning: When I did Plone for the first time there was only Products. When I came back, there was already buildout. Yet, I'd guess it would be related to version pins.

Agreed. mr.developer is definitely more flexible way to manage multiple developed packages than just installing packages in develop mode into virtualenv. Also there is no yet alternatives for creating Plone configuration with buildout (not to mention all the other recipes). (Well, I have now a zope.conf-generator for Nix, but that one is not compatible with this thread :slight_smile:

That depends on the availability of wheels for those packages. If there is properly build wheel available, that should include optimizations.

6

So, having a pip based Docker image could be even easier, because using and extending it would follow common Docker practices? :slight_smile:

7

I'm not aware of any disadvantages of installing addons via pip. Buildout is doing the same. We need z3c.autoinclude for that to work, but that shouldn't be a problem either...

8

Did you take a look at pipenv? I use it already for some buildouts locally, just travis has currently some problems I couldn't solve.

9

That can be easily accomplished with pipenv/pipfile as well :wink:

1 Like
10

includePlugins part of z3c.autoinclude (for entrypoint=plone) works fine also with pip

11

Is it worth a PLIP for Plone 5.2? We would need an official requirements.cfg and constraints.cfg, documentation and whats else?

2 Likes
12

I'm interpreting the negation as a typo - you mean like pip install -e git+https://github.com/plone/Products.CMFPlone.git@5.1rc2#Products.CMFPlone?

That does a source checkout egg develop into your virtualenv and you can then point your editor at it and hack at it as usual. You can use any @refspec (omitting this parametre means HEAD of default branch). Also can go into requirements.txt. The #eggname syntax is a bit weird, but it flags up a warning if you omit it, so it's probably there to make sure the eggname and reponame are not different from each other.

1 Like
13

Let's not let this block 5.2. To make this officially supported requires possibly a sprint worth of work:

  • pip branch of z3c.autoinclude should be released somehow
  • use includeDependencies-directive in Plone should probably be deprecated and it should be removed from the few remaining core packages using it; currently pip installed Plone only starts when that directive is disabled by special environment variable
  • full path for site.zcml should be configurable in zope.conf (now Zope2 loads it from instancehome/etc)
  • scripts generated by plone.recipe.zope2instance and plone.recipe.zeoserver need a new home (now they are in my branch of plonecli)
  • we need tool for generating site.zcml and zope.conf (e.g. templates bundled with plonecli)
1 Like
14

Early July good for you?

3 Likes
15
16

Just an update to this, I made a comment on this ticket https://github.com/plone/plone.docker/issues/89 with some work we did on this for creating a Docker image that does not bundle a packaging tool in a packaging tool (no more buildout).

I'd be interested in seeing if there is interest in getting the pip install Plone story moved further along as it will pave the way to nice Docker images.

1 Like
17

Join us at the Alpine City Sprint :smiley: it is a topic there! https://alpinecity.tirol/2020

1 Like
18

As @jensens noticed yesterday, it might be a time to get rid of z3c.autoinclude. It's better to be explicit in including configuration. And as a bonus, we are closer to pip compatibility.

As an update, the current status:

  • Plone 5.2 installs and starts fine with pip when Zope configuration and instance scripts are provided somehow and site.zcml has <meta:provides feature="disable-autoinclude" />

  • Some time ago, I forked plonectl https://github.com/datakurre/plonectl from plonecli and extracted instance script from plone.recipe.zope2instance; Unfortunately, that was before WSGI, so currently plonectl is only able to start ZServer based Plone 5.2 instances (installed with pip).

  • A floating idea has been that plonecli could include or use code from plone.recipe.zope2instance for generating instance configuration without buildout. Or should that be also part of plonectl?

(For myself, we are very late in Python 3 game, but we do have Nix-based Zope instance configuration generators waiting, also packaging Plone with Nix [pip style without buildout]. We'll go forward with my TxZServer and plonectl so I don't need to resolve WSGI related plone.recipe.zope2instance issues for pip.)

1 Like
19

I tweeted this, if someone wonders where I wrote about it. In a sense to make all IT I use greener I wrap a lot Py-Spy arround all long running Python processes. Maybe startup time is not that important, but it feels now much better.

The essence is, startup time of a complex (addon-rich) site came down from ~16-17sec to 6-7 seconds.

So after all, if we remove z3c.autoinclude and include ZCML dependencies explicit, we get a pip installable Plone. Plone 5.2 is already pip installable as long one does not use addons with z3c.autoinclude.
Reason: z3c.autoinclude does not work with wheels. It's complex discovery of matching pip/pypi package names to dotted paths uses some egg-specific heuristics and just does not work with wheels. This mechanism is also very slow. I do not see why we need to support it if some explicit line what to load just does the job.

So, in the plonectl it would be awesome to just have some configurations like where I have a list of addons with parameters and an zcml=true as one of those:

[addons]
collective.easyform zcml=true

or something like this.

3 Likes
20

As @jensens noticed yesterday, it might be a time to get rid of z3c.autoinclude. It's better to be explicit in including configuration. And as a bonus, we are closer to pip compatibility.

We should remove it from bobtemplates.plone as well (example: bobtemplates.plone/bobtemplates/plone/addon/setup.py.bob at e7a8de972521f5abd3b2fa74b346df8c04d481f5 · plone/bobtemplates.plone · GitHub).

3 Likes