Alpine City Strategic Sprint 2018 - Technical Report

This is a first and more technical report of Alpine City Strategic Sprint 2018 results. A press release style one will follow soon!

Our primary goal was to push the efforts of porting Plone to Python 3 in the right direction, remove blockers here and figure out ways - and document how to - port Plone to Python 3. Thanks to the Plone Foundation for sponsoring the strategic sprint!

Releases and Updates

At first Maurits and Michael worked on new releases of Zope 2 and Zope 4 including with the latest hotfixes and updated dependency versions. Releasing Zope 2 also removed the last blocker for a Plone 5.1 release.

After this releases Jens updated the pinned versions of Plone 5.2 to newest possible versions. Since we are using an old version of Unidecode package for a while he took the opportunity to have a look why test are failing there. Its all the normalizers in plone.i18n which are doing some too fancy stuff not using the API of Unidecode. Together with Gogo we managed to fix this in a sane way and are on the latest release. This includes more fancy normalizing in languages like Chinese.

Blocker for Plone: GenericSetup etc.

Godefroid joined Saturday and reviewed/fixed Davids Products.GenericSetup pull request, which contains a series of commits from Andreas, Tom, Godefroid, Michael and David. All those targeting Python 3 support and are essential for Plone on Python 3. It is still open. As usual some nasty bugs needed to be hunted down resulting in open pull requests in AccessControl and Zope.

Also Michael and Godefroid started porting plone.testing to Python3. It uses ZServer (which wont be ported) and the WSGI server. It is not finished yet, so functional testing on WSGI is not possible for now. Next step is then plone.app.testing.

Porting to Python 3

Philip, Davi, Katja, Max, Jens and Gogo worked on porting several packages to Python 3. First, we found using sixer is the way to go, but even sixer is not smart enough to detect all cases and also creates false replacements, so we had to use our brain to make it work. The sixer process is documented now. We identified 139 packages to port and 109 of them are now checked with sixer. It does not mean that all of this works in Python 3, but probability that it imports are high. After we had several packages importable Philip went the other way around and used a buildout with Python3, Zope 4 and Plone and started it up, followed the failures and fixed them. So more and more startup code started to work. Plenty of this changes are still on his hard disc, so stay tuned to have a look at plenty of pull requests in near future.

While porting, Katja stumbled over PlacelessTranslationsService (PTS) which is a relict from former times. We don't want to port it to Python 3. While Katja and Jens tried to get rid of the package we found two things: old i18n-Folder would not be supported any more (which is fine, we may provide a Python 2.7 of PTS version for BBB reasons, like if an addon needs it). The language negotiator from here is still used and so we be moved it to plone.i18n as is. Also we will loose some questionable micro optimizations implemented as ugly patches on zope.i18n. If we want to micro-optimize, this should go directly in zope.i18n.

PLIPs

Timo pointed us to another problem with Plone 5.2, Zope 4 and the plone.restapi PLIP. Johannes worked on it, but it is still work in progress.

Johannes worked on a different important area: Getting the Javascript story step by step better. Some time ago he wrote PLIP 1653: Restructure CMFPlone static resources. After he worked on it, all static js resources are now in an own package, which acts as a publisher for them. We are now Bower-free! This needs to be reviewed by Framework Team next.

Mosaic improvements

Mosaic got some love too. Peter and Davi worked to make fluid rows working. We updated the documentation for advanced features. A pull request is pending and waiting for review.

UI cleanup

Also Peter worked on some UI cleanup, like variables for mediaqueries in barceloneta to hopefully make it more understandable and fixing weird scrollbar bugs.

Markus fixed some UI problems as well, like fixed pagination for folders and next prev navigation (arrows are now css content).

Global Allow Discussion

Markus started a discussion about global-allow semantics: it can be confusing, because together with constrain types its meaning is a bit fuzzy. Documentation may improve here!

Event and Multilingual

Also, plone.app.event now has language independent fields for start, end, whole_day, open_end. Peters pull request was already merged.

YAFOWIL - autoform/ alternative form engine for Plone

Robert worked on porting YAFOWIL to Python 3. This is primary a boring work converting several 1000s lines of doctests to unittests.

While doing so, we discussed (not the first time) how much work it might be to create a YAFOWIL autoform as a drop-in replacement for plone.autoform and its z3c.form generator. In fact the idea is to have way more flexible declarative forms following the YAFOWIL philosophy of declaring forms (using data structures) and not writing code for forms. On Saturday morning Robert and Jens decided to give it a try. After 4 hours pair-programming we got a working prototype (see branch). Robert continued some more hours and now we have a foundation for further work which is already impressive. Next work to be done is writing tests, mapping widgets, create special Plone widgets (like RelatedItems using its pattern in YAFOWIL), vocabulary handling, ordering, ... Markus is interested working on it in the next months if we provide instructions how to work on it, which we will do! Gogo was really happy to see this happen!

Wrap Up

Plone still works after all our merges!

At the end of the Alpine City Strategic Sprint we were all happy to get so many things done. In the discussion after the final stand up meeting we identified necessity for discussion in the broader community:

  • webdav, still important, for example for other multi-upload/download and external editor,
  • clockserver is missing if we get rid of zserver - how to deal with it?

More Sprints

Finally Johannes announced Buschenschanksprint 2018 (South of Graz/ Austria) in June 11.-15.!!!
Also Michael told us, Gocept plans to organize a sprint dependend on the outcomes of this one, stay tuned!

Fun in the snow

The day after the Sprint, Peter, Michael, Godefroid and Jens did some tobogganing in the Wildschönau. It was a great day in the the snow with about 16km of speeding down the mountains on a sledge! At the end we had a great Austrian lunch at the Söllerwirt in Thierbach.

8 Likes

:raised_hands::clap:

1 Like

Not to forget, on Friday afternoon we had also fun at our Mars Mission Support Center tour, where we learned about Analog Mars Missions - scientific simulations and trainings of future real mars missions - and we had fun there with a Occulus Rift ISS simulator and controlling a Mars Rover.
The volunteer based AMADEE18 Mission starts tomorrow, if you want to follow their one month simulation in the desert of Oman, have a look at http://oewf.org/en/amadee18-live/

There are a lot of detail about the Python 3 goal, but nothing on the strategy.
It was my understanding that breaking changes should go to different major version, so that should really be Plone 6.0; since add-on Python code will usually require some changes to work with Python 3, switching to Python 3 is by itself a breaking change even if all Apis are unchanged.
Unless it's planned to have the 5.2 version working with python 3 and 2 ? seems unlikely, that would be a lot of work.

Plone 5.2 is planned to run on both, Python 2.7 and Python 3. Plone 6.0 then will be Python 3 only. ZServer ist still default in Plone 5.2 - including webdav and FTP and all it has. WSGI is also an option for those who want to configure it. In Plone 6.0 WSGI is the only publisher. Here are the major changes.

I don't think this is true.

Here is the roadmap as I imagine it. Please note that this is my personal opinion and not a official document.

Plone 5.2

  • Run on Zope4 with WSGI though ZServer will still be the default
  • Include plone.restapi
  • Ship with mosaic (still needs to be plip'ed)
  • Deprecate Archetypes, Skins and some more old technologies
  • Runs on Python 2.7 though much of the code will be compatible with Python 3.6+
  • I'd love this version to support Python 3 but I'm pessimistic about the timing

Plone 5.3

  • There will maybe not be a 5.3

Plone 6.0

  • Runs on Python 2.7 and Python 3.6
  • Most Addons will probably only run in Python 2.7 untill they are migrated
  • No more Archetypes Support (probably still works as a addon in Python 2.7 but not on Python 3)
  • New Pastanaga UI (rendered in the client), a server-side-rendered UI will still be there as a alternative. - I hope the new UI will be the default.
  • At some point we will need to drop support for Python 2.7 but that depends on how long the releases will take.

Keep in mind that 1.1.2020 support for Python 2.7 will end for good. There will be no more releases of fixes. That means the Plone Community will have to offer a stable solution to upgrade projects and databases to Python 3 some time before that. I feel a sense of urgency.

There are many fun things to work on:

2 Likes

In Sorrento last year this is the roadmap discussion document that we created:

From my perspective, some new UI features need to be included in releases, and of course I think that some of CastleCMS's features should be among those. Mosaic is a great start.

2 Likes

PyPy is not dropping 2.7, so if someone wants to tackle at least ExtensionClass and Acquisition to be ported over to use CFFI instead of the pile of arcane cpyext they are, that could still actually be a thing for quite some things Zope, if not even for some things Plone.

Also the Python 2 sunset is at the end of / during PyCon 2020, so probably in mid-to-late April.

This is a tough one and even somewhat worse than having to offer a migration path from UCS2 to UCS4 within Python 2 (which no one publicly did so far?). There needs to be a migration path from both and this will be a nightmare to communicate.

people will continue using Python 2.7 for a long time no matter what the deadlines are. there are lots of sites running under Python 2.4 many years after its EOL.

anyway, I share your concerns and looking at the time it takes to release (30 months between Plone 4.3 and Plone 5; around 18 months between Plone 5 and Plone 5.1) I think it would be better to focus on the urgent things for Plone 6.0: Python 3 support and fixing the resource registries story. IMO, all other enhancements must have a lower priority.

what scares me is that seems nobody wants to talk about the resource registries at all: everybody just ignores it when roadmaps are drawn and nobody answers the questions:

also, porting add-ons to Python 3 is a strategic issue that has to be taken seriously; makes no sense to have Plone 6 running on Python 3 with no add-ons for it; let's not repeat the story of Plone 5.

It’s not in that roadmap document from April 2017 but you’ve got a lot of agreement on this and Asko and others have been working with webpack, and we saw it in Barcelona. I don’t know if that represents consensus but I think that’s progress.

@pbauer what's the relationship between mosaic and Pastanaga? Will it live along-side or does it replace it? Would tiles be compatible? Sorry, if this is obvious to others but I haven't been able to go the conference or sprints and Pastanaga doesn't seem to have docs or planning docs I can see.
The reason I ask is that, like @tkimnguyen I believe we need have a carrot in every release. Some UI enhancement that makes it worth the pain of upgrading and dealing with incompatible plugins and themes that have to be reworked. Mosaic should be that carrot. But if its going to create churn by going away in 6.0 then the carrot turns into a stick and would create a lot of churn. Same question applies to potential castle features like its new add system, toolbar, preview etc, how does that related to Pastanaga?

Haha, did you know that "pastanaga" means "carrot" in Catalan? :slight_smile: :carrot:

I missed Albert's talk on Pastanaga in Barcelona (ironically) but I suspect he explains where he thought they were heading. Timo's talk on headless Plone was part of it. There was an open space on it in Barcelona (that I missed too, mea culpa). There was a sprint in Bonn recently on Pastanaga.

https://2017.ploneconf.org/presentation_videos

I was kind of hoping that we would get a lot of this discussion done in Sorrento this April, alas.

We plan on keeping CastleCMS working with whatever base technologies are part of Plone core.

Regarding python2: I know stragglers still run Python 2.4 stuff (not us) but no client will start new projects with Python 2.7 after its end-of-life.

Regarding resource registries in Plone 5: As far as I know @thet finished this PLIP in Innsbruck:

Pastanaga is a UI framework whereas Mosaic is mostly a layout editor/concept. So I guess the Mosaic editor can be implemented based on the Pastanaga UI. But: Pastanaga is currently only developed for front-end based rendering (plone-angular, plone-react). I've seen a raw version of the mosaic editor in plone-react a year ago. My guess is that will focus on implementing Pastanaga. A plan for implementing mosaic layout editor will exists when someone comes up with it.

Tiles are basically browserviews with added features (most importantly a schema). They usually render in the backend. If tiles can be reused in a frontend-driven UI is a good question for the team working on that.

I share that sentiment, Mosaic should be shipped with Plone. I don't think that we will remove server-side rendering from Plone for a very long time. We cannot expect people to rewrite their templates and tiles in react, angular or vue. Plone will probably be able to run traditionally for quite a while, though probably not using Pastanage.

Regarding Addons: I think getting addons to work on python3 and zope4 is much easier that migrating to dexterity and new-style resources was. There are some simple changes to be made to your code to run on py2 and py3. With 4 people we migrated over 100 packages in 4 days in Innsbruck. There are some non-trivial issues in Plone and Zope itself but migrating addons is really easy in most cases.

good to know; I already started work on that direction.

sadly, that solves a completely different problem.

6 posts were split to a new topic: Future of Mosaic

3 posts were split to a new topic: JS + Resource Registry Improvements (and Rants)

CFFI support is not the only thing needed for PyPy. There is a severe issue in PyPy which prevents RestrictedPython from doing its restrictions. Thus through the web code cannot be secured when running on PyPy. (Anyone with access to the ZMI is able to write TTW code!) See https://github.com/zopefoundation/RestrictedPython/blob/master/docs/CHANGES.rst#40b1-2017-09-15 and https://bitbucket.org/pypy/pypy/issues/2653 for details.

Thank you for the clarification. I've been wondering for the details after the sentiment passed out from the last conference.

CFFI issues are a layer I've hit myself.