Porting Plone to Python 3

Development
28

In a comment on


I posted a status-update on this.
Here is the same as a cross-post because not everyone is following the updates from github:

Status report

Here is report of the current state of Plone on Python 3. It covers what was done in the last couple of weeks since the Buschenschank Sprint (This was the Buschenschanksprint 2018!) and lists some things that still need to be done.

&tldr;~Things are looking good but we still need to do a lot of work to migrate some remaining packages, fix all tests and provide a solid database migration-story. If you can, please come to the sprint in Halle in October (https://www.meetup.com/de-DE/Zope-Sprint/events/252468356/)!

On June 25-27 there was a Mini-Sprint in Munich where Jens Klein, David Glick and Philip Bauer worked mainly on fixing the tests with Python 3.

For that sprint we started out with 7459 Tests with 138 failures and 40 errors and ended with 8147 Tests with 113 failures and 56 errors.

We worked on a bunch of tasks:

Weird tests

There were a lot of tests failing where the issue was not obvious. We fixed all of the the following:

  • -s plone.app.z3cform -t test_widget_base_notimplemented
  • -s plone.dexterity -t test_container_manage_delObjects
  • testUnicodeSplitter.py
  • -s Products.CMFPlone -t safe_unicode
  • -s Products.ZopeVersionControl
  • -s Products.CMFEditions -t test06_retrieveWithNoMoreExistentObject
  • -s Products.CMFEditions -t test16_delete_history_on_content_deletion
  • -s Products.CMFPlone -t test_site_logo_is_stored_in_registry: "Object is of wrong type" in fieldErrorBox. Check def validate()...
  • -s plone.protect -t test_change_password_on_root_does_not_throw_other_csrf_protection
  • -s plone.formwidget.namedfile -t widget.rst

Fail on Jenkins only

Some tests still fail when running in Jenkins but pass when running locally. Weird.

  • -s z3c.form -t datamanager.txt
  • -s five.customerize

plone.rfc822

After the sprint Jens tackled the package plone.rfc822 that deals with serialization (also of schemata) and fixed all remaining issues (I think...). Here are a couple of tests that failed before:

  • -s plone.rfc822
  • -s plone.namedfile -t marshaler.rst
  • -s plone.app.contentrules -t TestMailAction
  • -s Products.CMFPlone -t test_request_reset_password

Port plone.app.robotframework

David ported plone.app.robotframework to Python 3. For this we had to re-add XMLRPC support to Zope (it was removed for Zope 4) and ported it to Python 3. We also had to work around a pretty tough bug in Python itself. See https://github.com/plone/plone.app.robotframework/issues/80 for details.

Locally the robot-test are now running and some are passing. We still need to setup Jenkins properly to run them there and then fix the failing robot-tests.

Migrate tests away from PloneTestCase and PloneTestCaseFixture

We migrated a ton of tests from PloneTestCase to plone.app.testing. I also changed PloneTestCase to use Dexterity instead of Archetypes to make porting some tests easier that rely heavily on self.publish (see https://github.com/plone/plone.app.testing/pull/50). A lot of additional tests are now running because of this change but I also found a couple of issues where dexterity is failing (e.g. testClearFindAndRebuildKeepsModificationDate).

Add PY23DoctestChecker

To run doctests in Python 3 and Python 2 we use a pattern PY23DoctestChecker that can modify the expected and received values in doctests for Python 2 (for Python 3 they should pass without any tricks). See https://github.com/plone/plone.app.relationfield/pull/24/files for a simple example.

Strategy to deal with random ordering

In Python 3 (as actually in Py2) keys in a dict do not have a deterministic order. There are many tests which implicitly test the order of dicts. They had to be fixed to work in Py2 and Py3).

We applied a couple of different patterns for different tests. The best is to simply sort the keys.

Where are some of the test that used to fail:

  • -s Products.DCWorkflow -t test_exportimport
  • -s plone.app.multilingual -t test_formvariables_sequences
  • -s plone.app.contentrules -t testExport

Cannot adapt lists

A issue with __slice__ prevented lists of brains from being adaptable to IContentListing. See https://github.com/plone/Products.CMFPlone/issues/2429

Fix issues in GenericSetup

David fixed some more issues in the import/export code of GenericSetup. The main issue there was (as usual) serialization of lists of bytes and text.

Issue with IObjectInitializedEvent

When creating Dexterity-content with invokeFactory there is a issue when a new item is moved before it's completely added. Using createContentInContainer instead fixes this. See https://github.com/plone/plone.app.contentrules/pull/37

Open tasks

Plone runs fine with the current configuration (https://github.com/plone/buildout.coredev/blob/5.2/py3.cfg). But there are a lot of open tasks that need to be done before we can make a release.

Packages that need to be migrated

plone.outputfilters is being woked on by @MrTango (see https://github.com/plone/plone.outputfilters/issues/27).

Products.PortalTransforms still has a lot of failing tests and needs someone to take care of those.

plone.rest and plone.restapi. Both packages were merged into Plone 5.2 but they are not yet compatible with Python 3. I started working on this in https://github.com/plone/plone.rest/pull/75 and https://github.com/plone/plone.restapi/pull/542. See https://github.com/plone/Products.CMFPlone/issues/2474. @lukasgraf is working on this.

Fix all tests

We are currently at 9408 tests with 170 failures and 331 errors. The added tests and failures are mostly because I changed PloneTestCase to use Dexterity and also included plone.rest and plone.restapi in the Python 3 build.

Merge all open pull-requests

We currently have 88 open pull-requests that need to be reviewed and merged. They also need change notes and updated trove classifiers to specify support for python3:

Migration-Story

We need to start testing the migration-script zodbupdate with Plone and need to add and write the required zodbupdate_decode_dict for the packages in Plone that need it. For details see https://blog.gocept.com/2018/06/07/migrate-a-zope-zodb-data-fs-to-python-3/. This is likely the biggest and most important task for the sprint in Halle.

Document Support for Windows

So far no one tested Plone with Python 3 on Windows.

zope2instance

We need to add the necessary scripts to the WSGI start script:

  • run (for running scripts)
  • debug
  • adduser

Improve Jenkins setup

The Jenkins setup for the PLIP job is basically OK but we soon need to start testing all pull-requests and changes in Python 2 and Python 3. For this we need some serious Tox know how.

Migrate mtest to python 3

@rotonen did a lot of work on test parallelization with serious speed-improvements (see https://community.plone.org/t/ci-run-speedups-for-buildout-coredev/6225/14). As soon as we test everything with Py2 and Py3 tests take even longer. We are looking forward for the tests speedup be in use.

Adapt Plone's ZMI pages

There is a lot of work being done to port the ZMI to Bootstrap (see https://github.com/zopefoundation/Zope/pull/249). We will have to make a list all ZMI templates that Plone adds and test/fix them the new ZMI theme.

Port development plugins

The most important developer tools are probably plone.reload, Products.PDBDebugMode and plone.app.debugtoolbar. These packages need to be ported.

Port other plugins

We would love to see that some frequently used plugins are already compatible with Python 3 when when the first alpha of 5.2 is released.
Two that come to mind are plone.app.easyform and Mosaic, where mosaic is probably much more work since it comes with a whole bunch of dependencies.

Test against Python 3.7

Python 3.7 is out. We need to update the jenkins-jobs to test against it. One of the changes that might be relevant is that dicts preserve the order of items by insertion.

Deal with Deprecation- and ResourceWarnings

Running Plone on Python 3 yields many deprecation warnings. We should fix these. Also there are a lot of ResourceWarnings for unclosed files, we want to get rid of these by using with open(x) as fd: ...

Final words

Please pitch in to bring Plone to Python 3:

7 Likes