Plone 2020 Strategy

Development
1

At Ploneconf 2016 the roadmap / strategy "Plone 2020" was discussed. The main focus (again) seems to be on the backend / development side. Besides of REST API and Headless CMS here is my personal wishlist for a better user / editor / integrator experience:

  • Workflow UI - If you ask people what it is special about Plone mostly the first answer is (besides the community :slight_smile:) workflow and permissions. It is unfortunate that this gem is hidden in the deep dungeons of the ZMI. I'd like to see a drag & drop? UI for creating workflows in Plone. We can do better than Nintex for SharePoint :wink:

  • Login overhaul - The login pages of Plone are the last pieces that use CMFFormController. Replacing these would be a good chance to continue the work on plone.login and allow a more modern an pluggable login UI with an easy possibility to access 3rd party login services like Facebook, Google, Twitter or Github

  • User customizable listing views (this could probably be an addon)

  • Trash can - This is a often requested feature and there are some implementations already but none of them is complete and userfriendly.

  • Mosaic everywhere - I guess this is common sense and only needs to be PLIPed.

  • Remove portlets and viewlet - as a consequence from the item above

  • WSGI and Python 3 - This is not an integrator topic but I guess it is important for a Python web framework in 2020.

  • Content import / export story via CMIS or REST API.

  • API for content / seo quality checkers - For easy managing custom checks like title length, h1,h2, .. order, etc.

4 Likes
2

We started collecting tasks to get rid of technical debt at Cleanup Tasks Projects Board:

I added the login overhaul task there.

3

+100 for content import and export story!!

1 Like
4

I am working on something now (slowly, but with confidence :smirk:) based on zc.beforestorage and event-based audit logging of change events (modification, deletion, addition, move/rename -- all stored in annotations on the site, in ZODB, with various pruning options). I might have something suitable for review and testing at "proof of concept" level within the next week or two.

My slides on the idea from the conference:

Very limited work so far is only dealing with audit logging backend:

If anyone who has permission could create a repository for this in Plone Foundation · GitHub I would appreciate it. :grin:

The actual restoration is all science-fiction at this point, excepting the fact that I've fixed customer human error a handful of times using beforestorage programmatically with both reasonable results and remarkably repetitive patterns. Making this user facing ought to be doable, with security caveats and a few fun technical gotchas [1].

My goal is to deliver proof of concept as an add-on that is easily installable, uninstallable without mess. If the approach and implementation seem reasonable, which I hope they will, I will PLIP for future release.

As far as UX: making this "user friendly" is a bit tricky, there are lots of things to consider -- right now, I'm starting on making this a power user tool for Site Administrators. Idea is FIFO log of changes, presented in view, filterable by user, date range, path base, and change type); restorable items (deletions, modifications, move/rename) would have check boxes allowing restoration of one or more items of content; confirmations for special cases would intervene between selection and complete restoration, if necessary.

But my storage is (loosely) placeful (I store paths and UIDs, and will support restoration to new/old locations for moved items; we can restore items even when parents are deleted, but with care and possibly into a holding area); we might later be able to enable users to restore their own deletions (e.g. in a folder they have add permission on, for an item they previously had permission to modify[2]) in certain circumstances.

That plus DAV, but I have a fair guess that plone.server (or any other future re-implementation that eliminates ZServer/ZPublisher) might kill WebDAV, which saddens me a bit if we lose that by starting over. Maybe I'm just sentimental. :weary:

At minimum, content ought to be importable/exportable from trees of folders and files, or zip files thereof. It ought to be possible to import/export file-like content without loss of metadata, and it ought to be possible to represent structured/fielded content as JSON, with filename reference to exported BLOBs.

Sean

Notes:

  1. Aligning ZODB timestamp/TID with datetime stored by event subscribers before tpc_begin() is a bit tricky, and involves seeking a one to a few transactions from the storage starting at a time boundary near-before the logged date, iterating over transaction records in order, and checking if OIDs involved are modified in that transaction. Once you have the right TID, you can construct the date you need to put in your zc.beforestorage (ZConfig) configuration (necessary to open storage and time travel). There is also the matter of how much history one keeps; in the ideal future implementation, time-travel beyond this into FileStorage backups ought to be considered.

  2. Indexing permission to modify deleted items might require some kind of shadow-tree approach or an alternate index of local roles and permission maps for deleted items. I just don't want to deal with that yet.

5

Forget CMIS. It is not widely used in the CMS world and does not solve the content portability problem between our competitor systems due to their lack of CMIS. If you deal with large enterprise DMS then CMIS plays some role but this is truly a very small nice. Focus on easy to use APIs and robust import/export options like through ZIP data, transmogrifier pipelines or a REST service.

-aj

1 Like
6

Tim Simkins (Penn State) presented a product called Content Checks as a lightning talk last week at the Plone conference. I believe it is not posted anywhere yet but it sounds like what you are talking about in your last bullet point. It even has a guard for the workflow if certain thresholds are not met.

7

In addition to the PLIP there is also a lot of work left - just to not give the impression that this could be merged right away. Integrating Mosaic means that we have to switch to plone.app.blocks content and site layouts and giving up the main_template to not have multiple site templates which have to be kept in sync. As far as I understand, we will have to rewrite much of the browser views to tiles to make a full switch to a plone.app.blocks layout engine based stack.
That warning aside, I'd love to see this happen.

1 Like
8

@tomgross Some of these have PLIPS with really good ideas in them, just need people to work on them

Content import export - https://github.com/plone/Products.CMFPlone/issues/1386

The main idea is an UI that allows zip import/export of partial data (metadata) via csv as well as blob data and more structured data via json (perhaps embedded in csv or as seperate files in the zip).

User customisable listing views we currently have - https://github.com/collective/collective.listingviews (not plone 5 yet) but there is an even nicer idea in here https://github.com/plone/plone.app.mosaic/issues/239.

The gist is that we allow "inline" tiles, ie smaller replacement of placeholders inside rich text tiles in mosaics. This will allow a form of visual templating an editor can do. Then it becomes possible to visually layout a repeating tile which becomes a listing. Combine this with tiles for search fields and we then have a more integrated version of c.facetednavigation built into mosaic.

Workflow UI I would love to see too. I don't know what happened with the plip to integrate the workflow manager into plone - http://lists.plone.org/pipermail/plone-plip-advisories/2012-November/002920.html. Maybe plone 5 happened? Maybe it wasn't as intuitive as it could be? but at least it was a start. Making a UI for workflow is not easy.

9

@seanupton. So if I have this right the idea is to keep all your history in zodb data fs files + a seperate meta audit log that gives you the context of what those changes were (who, action etc) which is linked to the zodb transactions, and then use beforestorage to just pull the data back out and dump it onto the current state?

I think that's a really clever idea. Not sure how you'd handle issues like where plugins in past don't exist now but thats probably an edge case. Would love to see this. It's always been a shame that we have all the history data there but we can't get to it. And to say we have versioning but we don't when it comes to actions over many objects or deletions is a bit misleading. Many gov requirements require the ability to restore/view complete records from previous snapshots for legal reasons.

10

Well, to start, we can use beforestorage to wrap history from the production database, to the extent that your pack keeps it. Beyond that, we likely need to reach into backups, which I would suggest be FileStorage. For now, I am just starting with support for the former, and expect to get to the latter in due time.

I use RelStorage, then zodbconvert can create FileStorage backups and snapshot-rotate them on a retention policy (using bash scripts). I suspect I would have no problem using these (given proper configuration information).

Disclaimer: There is no way to go outside of production database's kept history without configuration, as I do not think we can introspect (a) how many days last pack kept; (b) location of backup FileStorage files; (c) naming convention. But making such configuration and function possible ought to be a goal for cases where the original source of error pre-dates history kept by the last pack.

This is simply kept in simple persistent records stored in site annotations on the live database by event subscribers. The date stamps may be slightly off from the transaction timestamps, but being close enough, we can infer the correct TID, then generate a correct datestamp to use for beforestorage. I am close to positive that this (application-level logging) will be more reliable and sufficient than trying to use ZODB transaction records alone.

This would, of course, depend on (a) ability to go beyond kept production history into (a copy of) your backups, (b) your data retention/rotation strategy. FileStorage is a relatively compact format, it is likely possible to have zodbconvert use zc.zlibstorage wrapping FileStorage to make backups even more compact, if you need near-line access to many of them.

Sean