Our pip based development workflow for Plone

Together with @zworkb and with input from discussions in Sorrento (thanks @ericof et al) I developed a workflow with

  • pip
  • make (for automation)
  • stable upstream constraints from dist.plone.org
  • a main and several development packages
  • run test and instance
  • debugging in vscode

This is all still in the making, but now I am at a point to share the insights. Details may get improvements.

Preparation:

  • Create a Plone add-on with plonecli or manually.
  • ignore or better remove all buildout specific parts (latter to not confuse yourself).

Files to be created

Hint: Philipp built a cookiecutter to ceate all below called plone-kickstarter with a good README explaining its usage together with plonecli.

requirements.txt

-c constraints.txt
-e .[test]
zope.testrunner

Here tools like black, isort, pdbpp, ... (one per line) can be appended as needed for development

constraints.txt

-c https://dist.plone.org/release/6.0.0a1/constraints.txt

sources.ini

The problem with pip install on top of stable constraints like with above reference is, pip does not allow to just override a constraint with -e GITSSHURL. First it has to be removed from the constraints. The next problem then is, pip always resets the cloned package to the upstream state. Changes would get lost if not committed before subsequent pip runs.

Last Sunday I developed a replacement for buildout's mr.developer. It is called mxdev (pronounced mixdev) and can be found on Github or PyPI. It is fresh and still called alpha, because I am sure missing features and problems are popping up while it is in use. OTOH it is simple, lo-dependency and does one thing: fetch and inject sources in an existing pip configuration.

mxdev reads a sources.ini with some basic settings and one section per sources with URL, branch and so on. Details about it are in the README.
It then recursively gets all constraints and requirements. It replaces constraints with a comment if it is in the sources config. It adds editable installs to the requirements for each package in the sources.
Packages are fetched from given VCS (libvcs is used here) to the configured target directory.
pip is not run, mxdev just prepares everything.

Given we need for our package the master of collective.richdescription and a branch barceloneta-lts of plone.app.mosaic the sources.ini would look like so:

[settings]
# first we could define in- and outfiles, but since ours are named as the defaults we can skip this
# the "target = " directory could be set, but we stick with default "sources".

#variables
github_plone = git+ssh://git@github.com/plone/
github_collective = git+ssh://git@github.com/collective/

[collective.richdescription]
url = ${settings:github_collective}/collective.richdescription.git

[plone.app.mosaic]
url = ${settings:github_plone}/plone.app.mosaic.git
branch = barceloneta-lts

Makefile

make dates back to 1976. It might feel ancient while working with it, but it is mission-proven like nothing else and it is used and known everywhere. And it just works.

Out Makefile has to

  • install everything
  • make an instance configuration
  • run tests
  • run the instance
  • clean up

The Makefile below declares a PACKAGENAME variable. Change it to the name of your package.

### Defensive settings for make:
#     https://tech.davis-hansson.com/p/make/
SHELL:=bash
.ONESHELL:
.SHELLFLAGS:=-xeu -o pipefail -O inherit_errexit -c
.SILENT:
.DELETE_ON_ERROR:
MAKEFLAGS+=--warn-undefined-variables
MAKEFLAGS+=--no-builtin-rules

# We like colors
# From: https://coderwall.com/p/izxssa/colored-makefile-for-golang-projects
RED=`tput setaf 1`
GREEN=`tput setaf 2`
RESET=`tput sgr0`
YELLOW=`tput setaf 3`


# Specifcs of this Makefile
PACKAGENAME=collective.testpackage
PIP_VERSION=21.3.1
PIP_PARAMS=--use-deprecated legacy-resolver

# install and run
.PHONY: all
all: install instance run

# Add the following 'help' target to your Makefile
# And add help text after each target name starting with '\#\#'
.PHONY: help
help: ## This help message
	@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}'

.PHONY: install
install: .installed.txt ## pip install all dependencies and scripts

.installed.txt: Makefile requirements.txt sources.ini constraints.txt setup.cfg setup.py
	@pip install -U "pip==${PIP_VERSION}" wheel mxdev
	@mxdev -c sources.ini
	@pip install -r requirements-dev.txt  ${PIP_PARAMS}
	@pip freeze >.installed.txt


.PHONY: instance
instance: instance/etc/zope.ini  ## create configuration for an zope (plone) instance

instance/etc/zope.ini: install
	@mkwsgiinstance -d instance -u admin:admin
	@cp skel/zope.ini instance/etc/
	@mkdir -p instance/etc/package-includes
	@sed 's/PROJECT/${PACKAGENAME}/g' skel/project-configure.zcml >instance/etc/package-includes/${PACKAGENAME}-configure.zcml

test: install
	@zope-testrunner --test-path=src

run: instance  ## run Plone
	@runwsgi -v instance/etc/zope.ini

.PHONY: clean
clean: install  ## remove instance configuration (keep data)
	@rm -fr instance/etc instance/inituser
	@pip uninstall -y -r requirements.txt
	@rm .installed.txt

.PHONY: style
style: install  ## format code (black, isort, zpretty)
	@isort src
	@black src
	@find src -name "*.zcml"|xargs zpretty -iz
	@find src -name "*.xml"|grep -v locales|xargs zpretty -ix

Skeleton files

The Makefile needs some files to prepare the Plone instance confiuration. Create a folder skel and add there:

skel/project-configure.zcml

Makefile replaces PROJECT and renames it on copy.

<configure
    xmlns="http://namespaces.zope.org/zope"
    xmlns:meta="http://namespaces.zope.org/meta"
    xmlns:five="http://namespaces.zope.org/five">

  <include package="PROJECT" />

</configure>

skel/zope.ini

The file is kept 1:1

[app:zope]
use = egg:Zope#main
zope_conf = %(here)s/zope.conf

[server:main]
use = egg:waitress#main
host = 0.0.0.0
port = 8080
threads = 4
clear_untrusted_proxy_headers = false
max_request_body_size = 1073741824


[filter:translogger]
use = egg:Paste#translogger
setup_console_handler = False

[pipeline:main]
pipeline =
    translogger
    egg:Zope#httpexceptions
    zope

[loggers]
keys = root, plone, waitress.queue, waitress, wsgi

[handlers]
keys = console, accesslog, eventlog

[formatters]
keys = generic, message

[logger_root]
level = INFO
handlers = console, eventlog

[logger_plone]
level = INFO
handlers = eventlog
qualname = plone

[logger_waitress.queue]
level = INFO
handlers = eventlog
qualname = waitress.queue
propagate = 0

[logger_waitress]
level = INFO
handlers = eventlog
qualname = waitress

[logger_wsgi]
level = INFO
handlers = accesslog
qualname = wsgi
propagate = 0

[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = NOTSET
formatter = generic

[handler_accesslog]
class = FileHandler
args = (r'%(here)s/../var/log/instance-access.log', 'a')
kwargs = {}
level = INFO
formatter = message

[handler_eventlog]
class = FileHandler
args = (r'%(here)s/../var/log/instance.log', 'a')
kwargs = {}
level = NOTSET
formatter = generic

[formatter_generic]
format = %(asctime)s %(levelname)-7.7s [%(name)s:%(lineno)s][%(threadName)s] %(message)s

[formatter_message]
format = %(message)s

Usage of make

With all above in place and an activated Python virtualenv just do make run or make test. On the first run it will install all, create the instance and run it. On subsequent call it don't. make only executes dependent steps if dependent files like requirement.txt or setup.py have been touched.

Debugging Plone in Visual Studio Code

probably possible in PyCharm and other IDE's too

@zworkb figured it out: Having all packages installed with pip enables debugging of a runnin zope instance or debuging test runs directly in the IDE. Create a file .vscode/launch.json with

{
    "version": "0.2.0",
    "configurations": [

        {
            "name": "Python: Plone",
            "type": "python",
            "request": "launch",
            "module": "Zope2.Startup.serve",
            "args": [
                "-v",
                "${workspaceFolder}/instance/etc/zope.ini"
            ],
            "justMyCode": false
        },
        {
            "name": "Python: Test Plone",
            "type": "python",
            "request": "launch",
            "module": "zope.testrunner",
            "args": [
                "--test-path=src"
            ],
            "justMyCode": false
        }

    ]
}

Then in VSCode go to "Run and Debug" (Ctrl-Shift-D), select one of the configurations and click on the play button
Screenshot from 2021-11-23 11-43-47
You can set breakpoints directly in your code left of the line number and use post-mortem debugging after an exception occurred.

My Conclusio

This is a first shot forward in a pip based and buildout free future. We have to be careful to keep it simple while covering more advanced use cases with this approach, but I am lucky looking forward.

Being able to debug in VSCode with ease is the sugar on top.

For now it is a development workflow. I soon will extend it with a build workflow for Docker containers based on our Sorrento sprint work. Stay tuned.

16 Likes

The pip approach is neat on one side. On the other side, we give up the auto-generated configuration of zope.conf and wsgi.ini through a declarative approach with the zope2instance recipe. Is there a story for solving this issue? Providing hand-crafted configurations is unlikely what we want in deployments.

1 Like

:heart: :heart: :heart:

Not to take anything away from the new pip approach but debugging in vscode works fine with buildout. I use it all the time. just add GitHub - nazrulworld/collective.recipe.vscode: Buildout Recipe for Visual Studio Code which has been updated to use pylance and sets python paths in all right places with no extra effort.

Since there seems to be a misconception this isn't possible I posted howto here Vscode debugging with instance and tests using your existing buildout

2 Likes

I think the python based can use python -m <buildout replacement section> <do stuff>

We are not giving up the generated configuration, it is on my todo list. One thing after the other.

Nice! Thanks for sharing.

I am becoming a fan of tox to do all kinds of commands/installation instead of Buildout or Make. It is not only for tests, but can be used as a more general tool. Being a Python tool, it is geared more towards Python projects. As an example, this is the tox.ini of my own website (freshly on Plone 6.0.0a1):

[tox]
envlist =
    plone
    backup
    varnish
    supervisor

[testenv]
basepython = python3.9
usedevelop = false
skip_install = true
deps = zc.buildout==2.13.5

[testenv:plone]
# For the moment, this still uses buildout, 
# but this may change to pure pip.
changedir = {toxinidir}/plone
deps =
# The [] means you could pass '-c production.cfg' where needed.
commands = ./bootstrap.sh []

[testenv:plonebuildout]
# Not used by default
changedir = {toxinidir}/plone
deps =
# The [] means you could pass for example 'install instance' or 'annotate'
commands =
# Despite the changedir, we must use the full path to bin/buildout,
# otherwise the executable is not found...
    {toxinidir}/plone/bin/buildout []

[testenv:varnish]
changedir = {toxinidir}/varnish
allowlist_externals = mkdir
commands =
    mkdir -p {toxinidir}/var/run
    {envdir}/bin/buildout

[testenv:backup]
changedir = {toxinidir}/backup
commands = {envdir}/bin/buildout

[testenv:supervisor]
changedir = {toxinidir}/supervisor
commands = {envdir}/bin/buildout

Next to this are directories that tox changes into (changedir above) when running the various environments: plone, backup, varnish, supervisor. All four currently still have a buildout.cfg, most of them extending a central shared.cfg with at least the port numbers. But now I am able to replace these one by one with pip if I want:

  • The plone env can become similar to what Jens does above.
  • I may keep varnish in buildout, using plone.recipe.varnish, as it is very useful, but some configure/make/make install plus config files could work too. Note that in this env I let tox do mkdir -p var/run, which I would otherwise have done with z3c.recipe.mkdir: such simple recipes are easily replaced with a single command.
  • The supervisor env can probably become pip install supervisor plus one configuration file.
  • Same could eventually be done for the backup env, but this would require stripping Buildout away from collective.recipe.backup.

Regardless of using tox or Makefile, I am not sure what the best way is to handle configuration files where you need to replace some of the contents, like a port number or a file path:

  • The Makefile uses sed to replace one variable. This will likely be harder with more variables that need to be replaced in more files.
  • You could define a skeleton and let plonecli or cookiecuttr render it once. That does not cover differences like /Users/maurits on Mac and /home/maurits on Linux.
  • Maybe Jinja templates, and some command that knows how to replace the variables.
1 Like

I do really like this buildout feature:

bin/buildout instance:http-address=12080 home:directory=/Users/gotcha

which lets you override some value from the command-line.

2 Likes

Short update on mxdev: It's now enhanced and bug-fixed in a to some extend stable alpha 9.
See mxdev · PyPI

I think it can go beta with the next release. Any feedback is highly appreciated!

Plone Foundation Code of Conduct