Together with @zworkb and with input from discussions in Sorrento (thanks @ericof et al) I developed a workflow with
- pip
- make (for automation)
- stable upstream constraints from dist.plone.org
- a main and several development packages
- run test and instance
- debugging in vscode
This is all still in the making, but now I am at a point to share the insights. Details may get improvements.
Preparation:
- Create a Plone add-on with plonecli or manually.
- ignore or better remove all buildout specific parts (latter to not confuse yourself).
Files to be created
Hint: Philipp built a cookiecutter to ceate all below called plone-kickstarter with a good README explaining its usage together with plonecli.
requirements.txt
-c constraints.txt
-e .[test]
zope.testrunner
Here tools like black, isort, pdbpp, ... (one per line) can be appended as needed for development
constraints.txt
-c https://dist.plone.org/release/6.0.0a1/constraints.txt
sources.ini
The problem with pip install on top of stable constraints like with above reference is, pip does not allow to just override a constraint with -e GITSSHURL. First it has to be removed from the constraints. The next problem then is, pip always resets the cloned package to the upstream state. Changes would get lost if not committed before subsequent pip runs.
Last Sunday I developed a replacement for buildout's mr.developer. It is called mxdev (pronounced mixdev) and can be found on Github or PyPI. It is fresh and still called alpha, because I am sure missing features and problems are popping up while it is in use. OTOH it is simple, lo-dependency and does one thing: fetch and inject sources in an existing pip configuration.
mxdev reads a sources.ini with some basic settings and one section per sources with URL, branch and so on. Details about it are in the README.
It then recursively gets all constraints and requirements. It replaces constraints with a comment if it is in the sources config. It adds editable installs to the requirements for each package in the sources.
Packages are fetched from given VCS (libvcs is used here) to the configured target directory.
pip is not run, mxdev just prepares everything.
Given we need for our package the master of collective.richdescription and a branch barceloneta-lts of plone.app.mosaic the sources.ini would look like so:
[settings]
# first we could define in- and outfiles, but since ours are named as the defaults we can skip this
# the "target = " directory could be set, but we stick with default "sources".
#variables
github_plone = git+ssh://git@github.com/plone/
github_collective = git+ssh://git@github.com/collective/
[collective.richdescription]
url = ${settings:github_collective}/collective.richdescription.git
[plone.app.mosaic]
url = ${settings:github_plone}/plone.app.mosaic.git
branch = barceloneta-lts
Makefile
make dates back to 1976. It might feel ancient while working with it, but it is mission-proven like nothing else and it is used and known everywhere. And it just works.
Out Makefile has to
- install everything
- make an instance configuration
- run tests
- run the instance
- clean up
The Makefile below declares a PACKAGENAME variable. Change it to the name of your package.
### Defensive settings for make:
# https://tech.davis-hansson.com/p/make/
SHELL:=bash
.ONESHELL:
.SHELLFLAGS:=-xeu -o pipefail -O inherit_errexit -c
.SILENT:
.DELETE_ON_ERROR:
MAKEFLAGS+=--warn-undefined-variables
MAKEFLAGS+=--no-builtin-rules
# We like colors
# From: https://coderwall.com/p/izxssa/colored-makefile-for-golang-projects
RED=`tput setaf 1`
GREEN=`tput setaf 2`
RESET=`tput sgr0`
YELLOW=`tput setaf 3`
# Specifcs of this Makefile
PACKAGENAME=collective.testpackage
PIP_VERSION=21.3.1
PIP_PARAMS=--use-deprecated legacy-resolver
# install and run
.PHONY: all
all: install instance run
# Add the following 'help' target to your Makefile
# And add help text after each target name starting with '\#\#'
.PHONY: help
help: ## This help message
@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}'
.PHONY: install
install: .installed.txt ## pip install all dependencies and scripts
.installed.txt: Makefile requirements.txt sources.ini constraints.txt setup.cfg setup.py
@pip install -U "pip==${PIP_VERSION}" wheel mxdev
@mxdev -c sources.ini
@pip install -r requirements-dev.txt ${PIP_PARAMS}
@pip freeze >.installed.txt
.PHONY: instance
instance: instance/etc/zope.ini ## create configuration for an zope (plone) instance
instance/etc/zope.ini: install
@mkwsgiinstance -d instance -u admin:admin
@cp skel/zope.ini instance/etc/
@mkdir -p instance/etc/package-includes
@sed 's/PROJECT/${PACKAGENAME}/g' skel/project-configure.zcml >instance/etc/package-includes/${PACKAGENAME}-configure.zcml
test: install
@zope-testrunner --test-path=src
run: instance ## run Plone
@runwsgi -v instance/etc/zope.ini
.PHONY: clean
clean: install ## remove instance configuration (keep data)
@rm -fr instance/etc instance/inituser
@pip uninstall -y -r requirements.txt
@rm .installed.txt
.PHONY: style
style: install ## format code (black, isort, zpretty)
@isort src
@black src
@find src -name "*.zcml"|xargs zpretty -iz
@find src -name "*.xml"|grep -v locales|xargs zpretty -ix
Skeleton files
The Makefile needs some files to prepare the Plone instance confiuration. Create a folder skel and add there:
skel/project-configure.zcml
Makefile replaces PROJECT and renames it on copy.
<configure
xmlns="http://namespaces.zope.org/zope"
xmlns:meta="http://namespaces.zope.org/meta"
xmlns:five="http://namespaces.zope.org/five">
<include package="PROJECT" />
</configure>
skel/zope.ini
The file is kept 1:1
[app:zope]
use = egg:Zope#main
zope_conf = %(here)s/zope.conf
[server:main]
use = egg:waitress#main
host = 0.0.0.0
port = 8080
threads = 4
clear_untrusted_proxy_headers = false
max_request_body_size = 1073741824
[filter:translogger]
use = egg:Paste#translogger
setup_console_handler = False
[pipeline:main]
pipeline =
translogger
egg:Zope#httpexceptions
zope
[loggers]
keys = root, plone, waitress.queue, waitress, wsgi
[handlers]
keys = console, accesslog, eventlog
[formatters]
keys = generic, message
[logger_root]
level = INFO
handlers = console, eventlog
[logger_plone]
level = INFO
handlers = eventlog
qualname = plone
[logger_waitress.queue]
level = INFO
handlers = eventlog
qualname = waitress.queue
propagate = 0
[logger_waitress]
level = INFO
handlers = eventlog
qualname = waitress
[logger_wsgi]
level = INFO
handlers = accesslog
qualname = wsgi
propagate = 0
[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = NOTSET
formatter = generic
[handler_accesslog]
class = FileHandler
args = (r'%(here)s/../var/log/instance-access.log', 'a')
kwargs = {}
level = INFO
formatter = message
[handler_eventlog]
class = FileHandler
args = (r'%(here)s/../var/log/instance.log', 'a')
kwargs = {}
level = NOTSET
formatter = generic
[formatter_generic]
format = %(asctime)s %(levelname)-7.7s [%(name)s:%(lineno)s][%(threadName)s] %(message)s
[formatter_message]
format = %(message)s
Usage of make
With all above in place and an activated Python virtualenv just do make run or make test. On the first run it will install all, create the instance and run it. On subsequent call it don't. make only executes dependent steps if dependent files like requirement.txt or setup.py have been touched.
Debugging Plone in Visual Studio Code
probably possible in PyCharm and other IDE's too
@zworkb figured it out: Having all packages installed with pip enables debugging of a runnin zope instance or debuging test runs directly in the IDE. Create a file .vscode/launch.json with
{
"version": "0.2.0",
"configurations": [
{
"name": "Python: Plone",
"type": "python",
"request": "launch",
"module": "Zope2.Startup.serve",
"args": [
"-v",
"${workspaceFolder}/instance/etc/zope.ini"
],
"justMyCode": false
},
{
"name": "Python: Test Plone",
"type": "python",
"request": "launch",
"module": "zope.testrunner",
"args": [
"--test-path=src"
],
"justMyCode": false
}
]
}
Then in VSCode go to "Run and Debug" (Ctrl-Shift-D), select one of the configurations and click on the play button
You can set breakpoints directly in your code left of the line number and use post-mortem debugging after an exception occurred.
My Conclusio
This is a first shot forward in a pip based and buildout free future. We have to be careful to keep it simple while covering more advanced use cases with this approach, but I am lucky looking forward.
Being able to debug in VSCode with ease is the sugar on top.
For now it is a development workflow. I soon will extend it with a build workflow for Docker containers based on our Sorrento sprint work. Stay tuned.
