Report after Alpine City Strategic Sprint

Events Sprints
1

Note: I have to split this report into parts, since community.plone.org has a limit of 10 mentions in one post.

Last week of January we sprinted at our (Klein & Partner KG, BlueDynamics Alliance) office in Innsbruck, Tyrol, Austria (official sprint page).

After five long days of talking, coding, poking into the depth of Zope2 and also after 5 days of fun, wine, beer and delicious food, great views on and from our mountains, some culture, lots of WTF and opening of pandoras boxes we have to report a lot on what was titled with "Plone - dig deep, tidy up".

But first a big thanks also to all the sprinters who were here and to our sponsor of the lunch-break, the Flying Circus. Also thanks a lot to my wife and business partner @chrimba who did all the organization of the sprint.

Before I start with the technical report: @thet took some pictures.

Zope 4
@thet and @pbauer worked (with partial help of the whole group) on the task of levering Plone to the so called Zope4. Zope4 is about updating Plone's framework dependencies (ZCA, ZTK and Zope) to the latest possible versions. Lots of this latest versions are already Python 3 compatible. In order to make Plone Python 3 compatible this step is essential.

This was hard work with some nuts to crack. It is not all finished, but the majority of problems are solved and the remaining are identified.

For a more details best read the PLIP 1351: Zope 4 / Update Dependencies

PluggableAuthenticationService (PAS)
Jens Vagelpohl (@dataflake) and I worked on the PAS. Jens Vagelpohl is one of the authors of Products.PluggableAuthService and so his input was (not only here) very valuable.

I must say that touching user-authentication-story is like opening pandoras box. The whole code is influenced by backward compatibility to GRUF (Group-User-Folder, was kicked out with Plone 3 release) and the 4 tools (portal_groupdata, portal_groups, portal_memberdata and portal_membership). This started all as an add-on: PlonePAS. Now this code does lots of patches to PAS itself. It also provides PAS-plugins which then use the tools, while the tools are using PAS/PlonePAS. The problem is, that PAS itself has no convenient API and so the BBB code for GRUF is used as convenience API everywhere. And much more.

First we picked the low hanging fruits and moved plugins, some patches and the events down to a forked Products.PluggableAuthenticationService. The tools were moved up to Products.CMFPlone. Interfaces were moved as well. So since this are often persistent elements in ZODB we provided deprecated BBB imports at the old places.

This is still work in progress. I think we also need to move forward in iterative steps.

  1. The first step is to cleanup everything that does not break using the GRUF API.
  2. The second step is to write a PAS API providing the convinience needed to work with users, groups, (local)roles.
  3. The third step is then to use the PAS API everywhere in Plone core and provided still deprecated calls at the old places. Provide migration steps for moved persistent information. At this step also solve the portrait-image problem.
  4. With a major Plone release remove the deprecated API and the 4 tools.

This is a lot of work, but if we can do this step by step its doable.

We also created a PLIP (that will be adjusted soon to only cover step one) PLIP 1334: Cleanup PlonePAS patches and tools

Read further in the next comment

6 Likes
2

Assimilate Collective Indexing

collective.indexing increases indexing performance and makes indexers pluggable (to support third party indexing serives like Solr). @gforcada and @MrTango worked on moving all functionality to existing packages (Products.ZCatalog, Products.CMFCore and Products.CMFPlone) and apply all monkey patches directly to the packages they belong to, like Products.CMFCore or Products.Archetypes.

Some tests in Plone core will need to be updated to work around the indexing optimizations (as tests expect that objects are indexed/reindexed/unindexed right away).

Follow the state of this task at the PLIP 1343: assimilate collective.indexing

Get rid of CMFQuickinstaller

There are two ways of installing products: GenericSetup and CMFQuickInstaller. Checking if a package is currently installed, means having to check portal_setup and portal_quickinstaller, or having code that makes sure those two are in sync. The syncing is mostly in place, with some recent changes, but supporting both remains tricky. Cleanup would make this code much easier to understand

@mauritsvanrees worked on this, read the PLIP 1340 for details

AccessControl/ RestrictedPython on Python 3

@loechel worked on making AccessControl and RestrictedPython work on Python 3. In order to make Plone in future run on Python 3 this is an important task. It is everything but trivial. Good news: it is possible. But: its a lot of work.
Alexander analysed and documented the steps and background.

We may need a dedicated sprint on this topic and it could help to have persons there with deep knowledge of Python AST, C-Python and compilers.

Testing
@do3cc dug deep into our test and tried to bump zope.testrunner to the latest release. In the minor release update of zope.testrunner the order of test execution was optimized which may speedup the tests and also run unit tests before all others, so failures on this level appear earlier.

But this was a disappointing job, since he figured out that this is not that easy and needs rewrites/updates of lots of tests. Mixing ZopeTestCase and plone.app.testing within the same distribution renders all its tests unstable. This also is a possible cause of our current test isolation problems. Also PloneMockTestCase and mocker usage in plone.dexterity is a source of problems. We need to rewrite all the plone.dexterity tests using the mock packages (addon in Python 2.7, stdlib in Python 3).

For details read the buildout.coredev ticket #155: Bump zope.testrunner / fix test isolation problems

Other testing related work:

  • created a jenkins job template to run code analysis on specific distributions that report back to github (@gforcada)
  • brought back jenkins node4 (@gforcada)
  • update jenkins jobs here and there, create a Zope tab for ZODB, ZTK, Zope and CMF tests. Highly untested (@gforcada)
  • created jenkins jobs for PLIP cfg files (@gforcada)

Read further in the next comment

4 Likes
3

Documentation

Various/ Core

Various/ Other

Add-Ons

I hope I catched all topics, thanks to all sprinters and supporters!

We indeed need more sprints like this to make progress. @gforcada would like to sprint in Berlin later this year. This would be a great opportunity to push the cleanup/zope4 work forward.

Jens Klein

End of Report

5 Likes
4

Fantastic work, everyone!

5

Great stuff, thanks for the writeup @jensens.

6

Wow, that's a lot of progress on scary parts of our code base! Thanks a lot everyone!

1 Like
7

We owe these sprinters a lot for having done all this. I feel very humbled to be part of such a great community!

1 Like
8

@pbauer and @thet made it: zope4-tests are finally green: http://jenkins.plone.org/view/PLIPs/job/plip-zope4/16/

4 Likes
9

just an annotation here: @agitator worked a lot on this, which I really appreciate, but the work is far from finished; I think now is more obvious that is not going to be easy to achieve such compatibility on add-ons in the short term.

it's also clear to me that we need to split the huge task in many smaller ones to be able to succeed.

10

resource registry and controlpanel stuff is quite easily doable for plone 4 and plone 5.

the pain is within tests ...

11

When it is not possible to write the very same test for both, you can write specific tests for Plone 4 or Plone 5 in the same file using @unittest.skipIf(HAS_PLONE_xx).
See https://github.com/plone/plone.api/blob/master/src/plone/api/tests/test_content.py for instance.

3 Likes
12

every add-on will have its own problems; in this case, yes, to have tests running without basic issues we need to fix Dexterity API because I don't want to have to deal with such problems in all our packages.

if someone on the Framework Team wants to guide us, we can reserve some time from @rodfersou for doing that.

2 Likes
13

Thank you all for all your hard work!

14

Thanks you all for sharing and for your best work ever......