Technical Sprint Report
Alpine City Strategic Sprint 2019
This is a first and technical report of Alpine City Strategic Sprint 2019 results.
Our primary goal was to make Plone 5.2 release ready by fixing open issues and merging open Plone Improvement Proposals (PLIPs), as well as porting as many add-ons as possible to Plone 5.2 and Python 3.
With 19 sprinters (two of them remote), the fifth was the largest Alpine City Sprint in Innsbruck.
It was organized by Klein & Partner KG, member of BlueDynamics Alliance.
Thanks to the Plone Foundation for sponsoring the strategic sprint!
Thanks to Kombinat Media Gestalter for sponsoring the Tiroler Bier!
Thanks to Zest Software for the delicious Pizza!
And thanks to Markus for his individual donation, invested in wine!
TL;DR
In detail we worked on the following primary tasks:
- Fixing open issues in Plone 5.2 and Zope 4,
- Fixing the upgrade from existing Plone 5 sites to Plone 5.2 on Python 2.7,
- Migrating the ZODB from Python 2.7 to Python 3.6/3.7,
- Finishing and merging of the Navigation PLIP,
- Finishing and merging of the overhaul of the Static Resources PLIP,
- Reduced Javascript Footprint,
- Removal of old portal_css/portal_js in code and DB.
- Major improvements to the Related Items Widget,
- Improve the Jenkins CI setup for Python 2.7/3.6/3.7 testing,
- Porting of add-ons (about 14 touched and lots of them finished).
- Foundation work regarding assimilation of Zope Foundation and integration of Pylons project.
And yes, we sprinted one day at ICE-Q at a height of 3048m above sea level. Probably the highest Plone sprint ever! While some focused at code sprinting and enjoying the view, Peter Mathis and I took ski and snowboard and sprinted down the slopes!
Open issues in Plone 5.2 and Zope 4
There were, and still are, some open issues in Plone 5.2 and Zope 4.
Here alone for Plone 18 (!) issues were closed by Philip, Alessandro, Wolfgang and others (Use the search, to many to reference them all).
Michael worked on Zope4, after some fixes and cleanups a fresh Zope 4 beta 9 was released! After the sprint we found problem in this Zope release and so Plone 5.2 beta 1 will still use Zope 4.0 beta 8. Thomas improved the debug- and deprecation-message output in WSGI.
Upgrade from Plone 5.0/5.1 to Plone 5.2 on Python 2.7
After Plone 5.2 works very well we also looked deeper into upgrading existing Plone 5 sites to Plone 5.2.
One problem was the renaming of the plone.richtext and plone.leadimage behaviors. The dotted path changed and as a side effect the plone.app.iterate broke, because old revisions still stored the old dotted name. Wolfgang added a feature to plone.behavior to deal with it by introducing a new field in the zcml behavior directive.
Another problem was identified inside the relation catalog. It uses Interface classes as keys in its indexes. If the code of an Interface (in our case one of the Zope 2 webdav stack) was removed, the catalog broke. Plone itself does not use that feature of the relation-catalog provided by z3c.relationfield. Jan and Jens improved relationsfield’s setup code to make installed catalog configurable and modified plone.app.relationfield to only configure the necessary indexes, including an upgrade step removing existing interface indexes.
Wolfgang also fixed a problem with broken member data after upgrade.
Migrating ZODB: Python 2.7 to Python 3.6/3.7
In Python 3 the pickle format changed as well as string handling and thus Python 2.7 ZODB can not be read in Python 3. There was already amazing work done on this topic prior to the sprint, but nobody knew in detail how and if this will work with a Plone Site.
Jan and Jens worked on this topic. The zodbupdate branch by David Glick looked most promising. We created a Vanilla Plone 5.2 coredev on Python 2.7.15, copied it the Data.fs and blobstorage to a fresh Plone 5.2 coredev on Python 3.7.1 buildout. coredev5.2multipy is very handy here. Important is to not start the Plone before migration, this will destroy the database. After running bin/zodbupdate --convert-py3 --file var/filestorage/Data.fs --encoding=utf8 the instance was started without any issues for now.
Jens added a section and sources of zodbupdate to the buildout.coredev so it is available for further testing. Next we switched on more features, such as multilingual support, working copy support, all shipped features. Migration of a site with working copy support switched on failed, but it wasn’t zodbupdate to blame. After fixing the upgrade problems all seemed to worked fine.
The next test was to use a real world production database. Here it gets more difficult, since not all addons were already ported to Python 3. Paul gave us as DB we thought it is reasonable to try on. We did some quick fixes on collective.isotope, collective.mailchimp and Products.PloneKeywordManager to make them at least start up on Python 3. We gave up on collective.plonetruegallery and skipped porting it (only a few pages in the DB used it). After getting the code in shape we run the migration and started the instance. The state so far: It starts, login is possible and it shows us the upgrade form. Upgrading itself does not work due to a problem somewhere in the five.intid tool. We suspect it is related to a string/bytes swapping problem while migration. This needs now further investigation.
Also Jan documented our findings, but the whole documentation also needs some cleanup. Jan did not remove any existing information at this stage. This is for sure important work for the Plone-Tagung Sprint in Munich.
Finishing and Merging of the PLIP Navigation
Peter and Alessandro worked on the navigation PLIP. The new fast dropdown Navigation got merged! At the sprint it was polished and lots of time was invested in good accessibility.
Finishing and Merging PLIP Static Resources
Until now, Products.CMFPlone contained a big blob of JavaScript managed by outdated tools like Bower. Johannes worked on this for a while and finished the PLIP while sprinting. Documentation is still missing, but Johannes will add it later.
Finishing and merging PLIP Removal of portal_css/portal_js
Since Plone 5.0 the old Products.ResourceRegistries were still available as a deprecated backward compatible bridge-head for old code. Katja took the effort to remove the code and added an upgrade step removing the tools from the site-root for 5.2. We figured out, often removal of code if more difficult than adding new!
Finishing and merging PLIP Merge Products.RedirectionTool into core
Maurits worked to finish the implementation started by Stephan and Asko 2 years ago on Nathans PLIP. Now automatic created redirections can be managed and manual ones be created, edited and removed in the control panel.
Reduced Javascript Footprint
At least for momentjs Davi managed to reduce the amount JS loaded from 172kB to 55kB by loading only the locales configured in the site:
Major improvements to the Related Items Widget
Maik took the effort to improve the Related Items Widget. The current UX with the widget was not good and it also was not very accessible. Here a lot of time flew in detail-work and discussions as also in solving technical problems. The updated version now shows images in a pleasant size and has an overall better arrangement of all elements. It is now possible to browser thru folders and select items by keyboard. Already selected items, can also be deleted by keyboard. A short demo can be found here:
Improve the Jenkins CI setup for Python 2.7/3.6/3.7 testing
Gil did amazing work to improve our Jenkins CI and mr.roboto. If we are polite mr.roboto/ Jenkins will create all jobs needed to test a coredev pull request on all configured Python versions. Just tell him “@jenkins-plone-org please run jobs” as a comment in the pull request and the magic happens. The very same works for addons now!
Porting of add-ons
pas.plugins.ldap & node.ext.ldap (& smbpasswd)
Manuel did an tremendous effort to port the ldap-plugins to Python 3. And it works now! The Pull-Requests are pending review!
plone.app.mosaic
Peter Mathis - with help of Markus 
- ported plone.app.mosaic and all related packages to Python 3. All except the plone.app.mosaic package itself are merged and released:
- plone.app.blocks -> 4.3.0
- plone.app.drafts -> 1.1.3
- plone.app.tiles -> 3.1.1
- plone.jsonserializer -> 0.9.7
The versions are pinned in versions.cfg file which can be used in custom buildouts.
(Note from @petschki: Unfortunately the pull request https://github.com/plone/plone.app.mosaic/pull/431 got merged too early by my mistake … I reverted it already and will Open a new PR with the mosaic Python 3 changes by the end of the week.)
collective.lineage
The important multi-site plugin for Plone was ported in remote work by Cillian. There are some problems left to be addressed in future.
collective.tiles.responsiveslides
Markus took this one. It was a bit more work, since it needed first to be ported to Plone 5.1. A Version 1.0.0 for Plone 5.1 is now released. The next step, porting to Plone 5.2/ Python3 need still to be done.
wildcard.media
Phillip worked on this one as an interactive as a training on Monday to live-code how to upgrade code to Python 3. It is not finished yet.
collective.impersonate
Cillian worked on this impersonate too. Only small changes were needed. A pull request is pending:
plone.app.imagecropping
Cillian worked on this one too. A pull request is pending.
collective.workspace
Another one by Cillian. Also here a pull request is pending. https://github.com/collective/collective.workspace/pull/31
bda.plone.shop
Peter Holzer worked on the BlueDynamics shop stack. It works now on Python 3. All Archetypes related code is gone. Also improved: The JavaScript handling. YAFOWIL forms now load resources on request, collective.js.jqueryui is no longer a package dependency and the jquerytools script was also added to bdajax so p.a.jquerytools is no longer needed. The changes are pending review.
Products.membrane
Cillian took this one too. Since it is an old and very Archetypes centric addon it got more difficult to port to Python 3 than the other ones. The work is not finished. An important todo for another sprint!
collective.js.jqueryui
Peter Holzer tested it with 3.7, added an uninstall profile and merged the changes to master. A release is pending.
collective.isotope, collective.mailchimp, Products.PloneKeywordManager
Jens made some changes to make it start on Python 3, more work/testing need to be done.
Documentation
The past showed that Robotframework as a tool for documentation screenshots is way to complex and fragile. Paul invested time to figure out if Puppeteer is an alternative.
Plone Foundation work
Paul did some paper work regarding the assimilation of the Zope Foundation and integration of the Pylons project.
catched all?
Lot of things happended! I hope there is no ones work missing here, please tell me otherwise and I will add the missing pieces.