Porting Plone to Python 3

Development
21

I probably broke creating a site in py3 this morning because of https://github.com/zopefoundation/Products.GenericSetup/pull/51#issuecomment-378857370. I don't really have time to look into this issue until next week.

22

I have a branch which I am testing locally against Python 2 and Python 3:

It contains the patches from the PRs:

plus some WIP code which I have to recheck.

The next tests I am trying to fix requires a Python 3 functional Products.GenericSetup.
I think I will focus on that ASAP, maybe contributing to https://github.com/zopefoundation/Products.GenericSetup/pull/51/files.

1 Like
23

getting back to this 3 weeks later... quite a few things changed now.
Since everyone has a specific setup I have one too.

on a Ubuntu 18.04 LTS VM:
sudo apt-get install python3-venv python3-dev gcc build-essential libpcre3-dev
(this has not a much to do with Plone 5.2 aka Plone python 3)

git clone https://github.com/plone/buildout.coredev coredev_py3
cd coredev_py3
git checkout 5.2
python3 -m venv .
source bin/activate
pip install -r requirements.txt
buildout -c py3.cfg

How easier can it be ? I cheated a bit, as for the first time I tried it's always necessary to tweak a bit py3.cfg to change a few repos that have switched from a python3 specific branch to master. The symptom is buildout messages like
mr.developer: b'fatal: Remote branch python3 not found in upstream origin\n'
Easy to fix, change python3 to master in py3.cfg

I stumbled on a few breakages but at this point the progress is so fast (and impressive) I am not sure it's worth the bandwidth to report them they are probably already fixed :slight_smile:

24

I updated py3.cfg after more branches were merged. The situation is pretty volative at the moment. So it makes little sense to report issues. What makes sense is to come to the sprint in Halle (https://www.meetup.com/de-DE/Zope-Sprint/events/248187180/).

Once the situation is a little more stable and testing makes sense we will communicate that loudly :smiley:

1 Like
25

first of all, thank you very much for pushing this and taking care of it; I just want to suggest a change in the current workflow:

we have been using Pylint on Python 3 porting mode to help us with our add-ons with very good results and I think we should include it as a check on Plone's core packages.

I just ran it in Products.CMFPlone and this is what I found:

$ pylint --py3k --disable=no-absolute-import Products/CMFPlone/
No config file found, using default configuration
************* Module Products.CMFPlone.PloneFolder
W:274,11: cmp built-in referenced (cmp-builtin)
************* Module Products.CMFPlone.PloneBatch
W: 30, 4: __nonzero__ method defined (nonzero-method)
W: 53,15: map built-in referenced when not iterating (map-builtin-not-iterating)
************* Module Products.CMFPlone.PloneTool
W:1101,44: dict.keys referenced when not iterating (dict-keys-not-iterating)
************* Module Products.CMFPlone.CatalogTool
W:247,15: division w/o __future__ statement (old-division)
W:249,34: division w/o __future__ statement (old-division)
************* Module Products.CMFPlone.MigrationTool
W:218,14: dict.items referenced when not iterating (dict-items-not-iterating)
************* Module Products.CMFPlone.Portal
W:152,36: cmp built-in referenced (cmp-builtin)
************* Module Products.CMFPlone.WorkflowTool
W:248,18: dict.values referenced when not iterating (dict-values-not-iterating)
************* Module Products.CMFPlone.DublinCore
W: 46,15: map built-in referenced when not iterating (map-builtin-not-iterating)
W:390,31: map built-in referenced when not iterating (map-builtin-not-iterating)
************* Module Products.CMFPlone.utils
W:411,26: dict.keys referenced when not iterating (dict-keys-not-iterating)
W:484,29: unicode built-in referenced (unicode-builtin)
W:486,31: basestring built-in referenced (basestring-builtin)
W:488,24: unicode built-in referenced (unicode-builtin)
************* Module Products.CMFPlone.browser.search
W: 57,12: map built-in referenced when not iterating (map-builtin-not-iterating)
W:283,24: dict.keys referenced when not iterating (dict-keys-not-iterating)
W:285,27: dict.keys referenced when not iterating (dict-keys-not-iterating)
************* Module Products.CMFPlone.browser.ploneview
W:122,19: division w/o __future__ statement (old-division)
************* Module Products.CMFPlone._scripts.compile_resources
E: 49, 8: print statement used (print-statement)
E: 55, 4: print statement used (print-statement)
E: 57, 4: print statement used (print-statement)
E:137, 8: print statement used (print-statement)
E:138, 8: print statement used (print-statement)
E:158, 8: print statement used (print-statement)
E:165, 8: print statement used (print-statement)
E:171, 8: print statement used (print-statement)
E:174, 4: print statement used (print-statement)
************* Module Products.CMFPlone._scripts._generate_gruntfile
E: 47, 0: print statement used (print-statement)
E: 52, 0: print statement used (print-statement)
E:200,24: print statement used (print-statement)
E:203,20: print statement used (print-statement)
E:216, 8: print statement used (print-statement)
E:263,12: print statement used (print-statement)
E:306,12: print statement used (print-statement)
E:359,12: print statement used (print-statement)
E:383, 8: print statement used (print-statement)
E:389,12: print statement used (print-statement)
E:390,12: print statement used (print-statement)
E:399,12: print statement used (print-statement)
E:400,12: print statement used (print-statement)
E:441,24: print statement used (print-statement)
************* Module Products.CMFPlone.tests.testResourceRegistries
W:578,12: filter built-in referenced when not iterating (filter-builtin-not-iterating)
W:622,12: filter built-in referenced when not iterating (filter-builtin-not-iterating)
************* Module Products.CMFPlone.tests.testNextPrevious
W: 68,15: Called a next() method on an object (next-method-called)
W: 82,31: Called a next() method on an object (next-method-called)
************* Module Products.CMFPlone.tests.testBatch
W: 10,22: range built-in referenced when not iterating (range-builtin-not-iterating)
W: 17,32: range built-in referenced when not iterating (range-builtin-not-iterating)
************* Module Products.CMFPlone.UnicodeSplitter.config
W: 38, 5: dict.values referenced when not iterating (dict-values-not-iterating)
************* Module Products.CMFPlone.controlpanel.browser.quickinstaller
W:614,25: dict.values referenced when not iterating (dict-values-not-iterating)
W:644,15: dict.values referenced when not iterating (dict-values-not-iterating)
W:647,15: dict.values referenced when not iterating (dict-values-not-iterating)
W:650,15: dict.values referenced when not iterating (dict-values-not-iterating)
W:653,15: dict.values referenced when not iterating (dict-values-not-iterating)
************* Module Products.CMFPlone.controlpanel.browser.usergroups_usermembership
W: 65,15: filter built-in referenced when not iterating (filter-builtin-not-iterating)

-----------------------------------
Your code has been rated at 9.93/10

I think this is a better indicator of what's our current status than just rely on what you are currently proposing because:

  • we have a list of things that need to be fixed
  • we can focus on specific issues on every commit (for instance, filter-builtin-not-iterating)
  • we can avoid people reintroducing incompatible code in further commits

here is an example of the results on one of our add-ons:

https://travis-ci.org/collective/collective.fingerpointing/jobs/379725533#L1690-L1694

26

Thanks for the tip. The more tools the merrier. I will give pylint certainly a try. python-modernize had a lot of false positives that could be ignored, I hope pylint is better.

27

I wrote a blogpost about the current status:


The short version: Progress is good, Plone is running nicely on Python 3, there are some bugs left and we have to fix over 500 failing tests. There also will be a demo-site for you to test soon.

1 Like
28

In a comment on


I posted a status-update on this.
Here is the same as a cross-post because not everyone is following the updates from github:

Status report

Here is report of the current state of Plone on Python 3. It covers what was done in the last couple of weeks since the Buschenschank Sprint (This was the Buschenschanksprint 2018!) and lists some things that still need to be done.

&tldr;~Things are looking good but we still need to do a lot of work to migrate some remaining packages, fix all tests and provide a solid database migration-story. If you can, please come to the sprint in Halle in October (https://www.meetup.com/de-DE/Zope-Sprint/events/252468356/)!

On June 25-27 there was a Mini-Sprint in Munich where Jens Klein, David Glick and Philip Bauer worked mainly on fixing the tests with Python 3.

For that sprint we started out with 7459 Tests with 138 failures and 40 errors and ended with 8147 Tests with 113 failures and 56 errors.

We worked on a bunch of tasks:

Weird tests

There were a lot of tests failing where the issue was not obvious. We fixed all of the the following:

  • -s plone.app.z3cform -t test_widget_base_notimplemented
  • -s plone.dexterity -t test_container_manage_delObjects
  • testUnicodeSplitter.py
  • -s Products.CMFPlone -t safe_unicode
  • -s Products.ZopeVersionControl
  • -s Products.CMFEditions -t test06_retrieveWithNoMoreExistentObject
  • -s Products.CMFEditions -t test16_delete_history_on_content_deletion
  • -s Products.CMFPlone -t test_site_logo_is_stored_in_registry: "Object is of wrong type" in fieldErrorBox. Check def validate()...
  • -s plone.protect -t test_change_password_on_root_does_not_throw_other_csrf_protection
  • -s plone.formwidget.namedfile -t widget.rst

Fail on Jenkins only

Some tests still fail when running in Jenkins but pass when running locally. Weird.

  • -s z3c.form -t datamanager.txt
  • -s five.customerize

plone.rfc822

After the sprint Jens tackled the package plone.rfc822 that deals with serialization (also of schemata) and fixed all remaining issues (I think...). Here are a couple of tests that failed before:

  • -s plone.rfc822
  • -s plone.namedfile -t marshaler.rst
  • -s plone.app.contentrules -t TestMailAction
  • -s Products.CMFPlone -t test_request_reset_password

Port plone.app.robotframework

David ported plone.app.robotframework to Python 3. For this we had to re-add XMLRPC support to Zope (it was removed for Zope 4) and ported it to Python 3. We also had to work around a pretty tough bug in Python itself. See https://github.com/plone/plone.app.robotframework/issues/80 for details.

Locally the robot-test are now running and some are passing. We still need to setup Jenkins properly to run them there and then fix the failing robot-tests.

Migrate tests away from PloneTestCase and PloneTestCaseFixture

We migrated a ton of tests from PloneTestCase to plone.app.testing. I also changed PloneTestCase to use Dexterity instead of Archetypes to make porting some tests easier that rely heavily on self.publish (see https://github.com/plone/plone.app.testing/pull/50). A lot of additional tests are now running because of this change but I also found a couple of issues where dexterity is failing (e.g. testClearFindAndRebuildKeepsModificationDate).

Add PY23DoctestChecker

To run doctests in Python 3 and Python 2 we use a pattern PY23DoctestChecker that can modify the expected and received values in doctests for Python 2 (for Python 3 they should pass without any tricks). See https://github.com/plone/plone.app.relationfield/pull/24/files for a simple example.

Strategy to deal with random ordering

In Python 3 (as actually in Py2) keys in a dict do not have a deterministic order. There are many tests which implicitly test the order of dicts. They had to be fixed to work in Py2 and Py3).

We applied a couple of different patterns for different tests. The best is to simply sort the keys.

Where are some of the test that used to fail:

  • -s Products.DCWorkflow -t test_exportimport
  • -s plone.app.multilingual -t test_formvariables_sequences
  • -s plone.app.contentrules -t testExport

Cannot adapt lists

A issue with __slice__ prevented lists of brains from being adaptable to IContentListing. See https://github.com/plone/Products.CMFPlone/issues/2429

Fix issues in GenericSetup

David fixed some more issues in the import/export code of GenericSetup. The main issue there was (as usual) serialization of lists of bytes and text.

Issue with IObjectInitializedEvent

When creating Dexterity-content with invokeFactory there is a issue when a new item is moved before it's completely added. Using createContentInContainer instead fixes this. See https://github.com/plone/plone.app.contentrules/pull/37

Open tasks

Plone runs fine with the current configuration (https://github.com/plone/buildout.coredev/blob/5.2/py3.cfg). But there are a lot of open tasks that need to be done before we can make a release.

Packages that need to be migrated

plone.outputfilters is being woked on by @MrTango (see https://github.com/plone/plone.outputfilters/issues/27).

Products.PortalTransforms still has a lot of failing tests and needs someone to take care of those.

plone.rest and plone.restapi. Both packages were merged into Plone 5.2 but they are not yet compatible with Python 3. I started working on this in https://github.com/plone/plone.rest/pull/75 and https://github.com/plone/plone.restapi/pull/542. See https://github.com/plone/Products.CMFPlone/issues/2474. @lukasgraf is working on this.

Fix all tests

We are currently at 9408 tests with 170 failures and 331 errors. The added tests and failures are mostly because I changed PloneTestCase to use Dexterity and also included plone.rest and plone.restapi in the Python 3 build.

Merge all open pull-requests

We currently have 88 open pull-requests that need to be reviewed and merged. They also need change notes and updated trove classifiers to specify support for python3:

Migration-Story

We need to start testing the migration-script zodbupdate with Plone and need to add and write the required zodbupdate_decode_dict for the packages in Plone that need it. For details see https://blog.gocept.com/2018/06/07/migrate-a-zope-zodb-data-fs-to-python-3/. This is likely the biggest and most important task for the sprint in Halle.

Document Support for Windows

So far no one tested Plone with Python 3 on Windows.

zope2instance

We need to add the necessary scripts to the WSGI start script:

  • run (for running scripts)
  • debug
  • adduser

Improve Jenkins setup

The Jenkins setup for the PLIP job is basically OK but we soon need to start testing all pull-requests and changes in Python 2 and Python 3. For this we need some serious Tox know how.

Migrate mtest to python 3

@rotonen did a lot of work on test parallelization with serious speed-improvements (see https://community.plone.org/t/ci-run-speedups-for-buildout-coredev/6225/14). As soon as we test everything with Py2 and Py3 tests take even longer. We are looking forward for the tests speedup be in use.

Adapt Plone's ZMI pages

There is a lot of work being done to port the ZMI to Bootstrap (see https://github.com/zopefoundation/Zope/pull/249). We will have to make a list all ZMI templates that Plone adds and test/fix them the new ZMI theme.

Port development plugins

The most important developer tools are probably plone.reload, Products.PDBDebugMode and plone.app.debugtoolbar. These packages need to be ported.

Port other plugins

We would love to see that some frequently used plugins are already compatible with Python 3 when when the first alpha of 5.2 is released.
Two that come to mind are plone.app.easyform and Mosaic, where mosaic is probably much more work since it comes with a whole bunch of dependencies.

Test against Python 3.7

Python 3.7 is out. We need to update the jenkins-jobs to test against it. One of the changes that might be relevant is that dicts preserve the order of items by insertion.

Deal with Deprecation- and ResourceWarnings

Running Plone on Python 3 yields many deprecation warnings. We should fix these. Also there are a lot of ResourceWarnings for unclosed files, we want to get rid of these by using with open(x) as fd: ...

Final words

Please pitch in to bring Plone to Python 3:

7 Likes
29

At the Plonator-sprint last week in Munich (https://pads.ccc.de/vco1AVDkgj) we created a demo-site for Plone on Python 3: http://demo-latest-py3.plone.de
Give it a try!

It is not yet error-free since we still have 279 failing test (of 9412 total). Please report issues with Plone itself in https://github.com/plone/Products.CMFPlone/issues and issues with the demo-site in https://github.com/collective/demo.plone.de.

We also created a demo-site for 5.2 with python 2: http://demo-latest-py2.plone.de. Both site are being rebuilt from scratch every night from the current state of development.

Thanks to all participants of the sprint!

6 Likes
30

For a more detailed report of the Plonator-Sprint see

Also: A list of tasks for the sprint in Halle is now published here:

Please sign up!

2 Likes
31

awesome! does collective.sidebar supports the horizontal toolbar also?

32

No. So far only a vertical sidebar is planned. There is some functionality missing and we have to smash some bugs before we are able to make a release. Additional improvements are welcome!

33

The Saltlabs-Sprint in Halle (https://www.meetup.com/de-DE/Zope-Sprint/events/252468356) is approaching. More than 30 people are coming to the sprint (!!!) and I hope we will get a lot done. In preparation for the sprint there is some work under way:

Merge branches: @jensens, @thet and I are currently trying to merge as many python3-branches as possible. We're currently down to 44 unmerged braches. Many of the remaining branches work fine in python 3 but depend on changes in other packages (mosty plone.app.testing, plone.testing, Products.CMFPlone or plone.rfc822), so they are not easily merged.

Fix failing tests: We currently have 216 failing tests in Python 3, most of those are in CMFPlone (because I changed PloneTestCase to use Dexterity) and in plone.restapi.

With https://jenkins.plone.org/view/PLIPs/job/py3_on_py2/ I created a jenkins-job that runs the branches of this PLIP in Python 2. That should make it easier to find issues that need to be fixed before merging.

I would have loved to have a alpha-release of 5.2 out before the sprint but fixing all remaining tests and merging all remaining branches seems more and more unlikely (even I have to sleep sometimes and need to earn some money...).

But: Things are looking good:

  • http://demo-latest-py3.plone.de is working most of the time (unless someone merges a branch without updating the py3-config)
  • There are promising first result from migrating Plone Databases to Python 3
  • @gforcada will add some additional jenkins nodes so that jenkins will not be the bottleneck

If you want to help:

  • review, test and merge some branches
  • fix some failing tests
  • come to the sprint

The tasks for the sprint are basically still the same as mentioned above. We'll publish a more detailed list next week.

2 Likes
34

Three wishes:

  1. The executor count of each node matches the hyperthread count of the backing container / VM / naked CPU
  2. We use the heavy job plugin to indicate the parallelity of each build Heavy Job
  3. We use the least load plugin to distribute the builds Least Load

The last one is the most important when increasing the node count, as per default Jenkins prefers to run the builds on the node and executor it previously ran on.

35

In reverse order:

The third one is already in (just installed it now). At least, the plugin details page says that just install and have fun, no configuration needed.

About the other plugin, I can not see its docs as jenkins' wiki is down :confused:

Lastly, as for the first one, I try to get those numbers right, I will ask for help on IRC if not.

36

Correct, no configuration needed.

It adds an extra build attribute, weight, which makes the build take up more executors from the queue, as the build really does. The field defaults to 1, so if the builds are not configured to use it, it does not change anything.

Can you document the nodes in regards to what the tests actually run on? Bare metal? VMs? Containers? Something exotic?

Run getconf _NPROCESSORS_ONLN where the tests would run and use that number as the executor count.

37

Our containers are LXD ones, and each of them (nodes 1 to 3) report 8 as well as the host system that hosts them.

Right now we have 4 executors per node (i.e. 12) while the getconf value reports 8, should we scale that down to 3 per node (i.e. 9 total executors?).

Node 4 is also reporting 8... @jensens is it really a 8 cores machine that one?

38

Configure your containers to only expose the correct amount of threads to the processes running in them and match the executor count to that.

https://github.com/lxc/lxd/blob/master/doc/containers.md

Set limits.cpu on the container resource limit configuration.

4 'CPUs' and executors per node and 2 nodes per server (assuming 4 cores and 8 hyperthreads) sounds sane to me. So does 3 'CPUs' and 3 nodes per server.

Also, it's most likely giving you the hyperthread count, not the CPU core count. Check out the output of lscpu on the host side to gain a better insight as to what you're running on.

39

No, well, it is a kvm virtual machine configured like so:

  <vcpu placement='static' cpuset='22-23,42-43'>8</vcpu>
  <numatune>
    <memory mode='strict' nodeset='3'/>
  </numatune>

I am not sure where I got this settings from. I can change this to 4 CPUs which now makes more sense to me.
The cpuset is shared with a second very low-cpu consuming virtual machine.

40

No, that is sane as it is. It takes 4 cores from NUMA node 3 and 4 hyperthreads from NUMA node 3 and ensures all memory it can allocate is also from within NUMA node 3. Please keep as is.

1 Like