Let there be TTW add-ons

Plone Support Add-on Development
1

Disclaimer: I'm not PLIP'n anything nor have implemented much yet.

We have discussed about the possibility of "through-the-web" (TTW) installable add-ons for years. Talk, a lot. Results, not so much. But in practice, for the last years, at least I've deployed TTW supported features bundled with themes for all of our new Plone 5 based sites (with the help of collective.themesitesetup and collective.themefragments): content types, restricted templates, frontend resources, custom localizations, workflows, permissions, content rules and, obviously, a lot of configuration. For the most, I've just used the currently existing features in Plone, mostly GenericSetup and RestrictedPython.

How unique feature of Plone is that thanks to Zope 2 legacy, we can deploy more features securely through the browser? (Yes, Wordpress has supported TTW upgrades for ages, but only by allowing it to rewrite its own code.) If so, why are we so shy of that? Or is there known security limitation in RestrictedPython, which we all should be better aware of?

Sure our TTW approach does not support everything. There are very strict limitations, which are support by either GenericSetup or RestrictedSupport. And especially with RestrictedPython the limitations often surprising (resulting hard to understand and debug unauthorized errors). Adding new features for GenericSetup is doable, RestrictedPython is more tricky. This is probably also the reason, why we still don't have official TTW add-on story: we have been afraid of the confusing of TTW development, hard to understand differences between TTW and filesystem add-on, and have been unable to decide, which features should be supported by TTW add-ons.

Since we've been unable figure out "the right way", I suggest, that we should just accept the current status quo and two tiers of add-on development and embrace it:

  • eggs, file system add-ons are trusted and can do anything
  • TTW uploadable "apps" remain untrusted and can only build on top of the features provided by file system add-ons (e.g. use behaviors, content-rules, portlets, tiles, etc defined in those) within limits of RestrictedPython

The confusing differences between file system and TTW development would remain, but the benefits of good TTW story should be big enough that users would just learn and accept it.

Personally, I'm interested in developing framework and UI for TTW add-ons, possibly by hijacking collective.ttw initiative, but only within the scope of features supported by the current Plone backend (e.g. I would not like to support z3c.jbot, because as seen in collective.jbot, that requires breaking out from RestrictedPython sandbox):

  • manifest format for "apps" (deps, versions, update address, signature, etc... features similar to old Firefox add-ons)
  • configuration panel for installing, updating apps, disabling, enabling and uninstalling apps (but note that in practice, "clean uninstall" would really depend on add-on and would be really hard to enforce)
  • apps would be separate from theme and managed under custom plone.resource namespace
  • apps could also be developed TTW using filemanager-pattern based UI (it's getting pretty cool after GSOC by @b4oshany)

But now it's the time for "stop energy". Why this should not be done? Am I missing something obvious? Is the two-tier add-on story more threat than possibility? Are there issues in RestrictedPython security, which makes this not worth of effort (as in why not just allow uploading wheels, write to filesystem and restart instance)?

5 Likes
2

Hell yes. +100

3

I realized that Firefox add-on signing has changed quite a bit from what it was 10 years ago when used that last time.

Any suggestions for signing? My initial idea was simply

  • allow to include gpg pubkey in manifest
  • updates could be signed with detached gpg signature (.asc-file next to .zip-file)
  • detached gpg signature is verified against downloaded update

This would allow to optionally whitelist keys and prevent apps from unknown developers. This would also allow to use and update apps without verifying if gnupg is too much of a dependency.

Or would certificate + openssl -based approach be better? I feel certificates more complex than keys, so I would appreciate examples.

Update: Mozilla's current approach could be an alternative, but would probably require developers setting their own autograph-services to automate the process. I'm not sure what's the main difference from just signing the final zip with gpg.

Update 2: As I understand, Mozilla's approach is to sign every file in zip and include signatures within the zip. Unless good reasons, I'd prefer using detached signature for simpler implementation and use.

4

Seems in line with the the Plone 2020 document published in 2014:

Here's the vision from then regarding TTW....

TTW (through the web) Themes and Templates

  • Easy add-on installs - click and run
  • Unify theming solutions
  • Two ways/techniques for theming that 90% of us are satisfied with
  • Theme editor that can be used for real in production
  • Consistent TTW experience so someone newish to the web can create a completely custom site in a day
  • Empower users with robust tiling/mixing/embedding
  • TTW view creation/customization and export - and/or Mosaic
  • Finish TTW development story
  • Simplified configuration - single source/way of defining content types/registrations (vs. ZCML, GS, ...)
  • Improve caching support (response times)
1 Like
5

I recall, I participated that discussion :slight_smile:

1 Like
6

I believe, I have only a few open question related to the design.

  1. Naming. I believe, I would call these TTW add-ons "apps". It would be distinct enough from add-ons and not yet used in Plone context. Package could be collective.ttw, collective.appsupport, whatever else or also in plone namespace if wanted. I don't have too strong opinions. Suggestions?

  2. Caching. Using raw resources (like templates, permissions, and various settings) from plone.resource requires some cooking and therefore caching to be performant. Should I a) copy the ZODB connection based cache from plone.app.theming or b) just use RAM cache from plone.memoize? The cache implementation in plone.app.theming is easy to invalidate when app has been updated. On the other hand, plone.memoize.ram uses Zope RAM cache, whose size can be configured and also supports memcached backend.

  3. App structure. App is a zip file with manifest.cfg and any amount of supported feature-folders with files. Possibly the names of those folders should be configurable, but we need good defaults. Each folder would be handled by a named adapter by the folder name:

./manifest.cfg  # plone.resource INI-manifest with metadata, all options and configuration, new permissions, etc...
./preview.png   # preview, logo or other visual representation of the app
./install       # GS profile to import on install
./uninstall     # GS profile to import on uninstall
./upgrade/to_2  # GS upgrade profiles named after p.a.upgrade conventions
./public        # publicly traversable assets
./models        # Dexterity XML schemas (I already tried inline in GS profiles, not convenient)
./locales/LC_MESSAGES/fi/   # PO files both for existing and new namespaces
./views         # restricted view templates and python scripts
./resources     # other plone.resource directories (possibly whitelisted namespaces) copied during install
./tiles         # Mosaic tiles [probably not built-in, but later implemented by Mosaic]

Note that ./resources together with ./install should allow app to also provide and activate a theme.

This folder based approach should make it easy to support extensions later, e.g. there should obviously be support for ./rapido folder later (but implemented in plone.rapido).

I know that rapido uses yaml for configuration, but I'd prefer INI, because it forces to think, how to keep the configuration flat and simple.

Opinions?

7

@b4oshany I recall that you were planning scaffolding support and inline documentation in to theme editor. This app project is an example, why it would be nice, if those features would be generic enough to use configure and use with custom data also outside theme editor (possibly through file manager pattern configuration).

8

What you seem to be talking about is a fat theme right? Is there difference is that you can have multiple plugins installed at the same time? Do you have a couple of examples from existing plugins that can be replaced with this zip approach?
Also not sure why you want views when rapido already handles that. Rapido will also do first class tiles soon too.

9

I've used all the features above already in fat themes, but I want to decouple all this from themes to make them reusable (installable and ugradeable) separate from theme.

This approach allows installing and maintaining simple content types with custom views, permissions and workflows separate from theme TTW without buildout or restarts. An example of a site build with this technique is https://kielikeskus.jyu.fi/eoto/en/info. It's a complete app as a fat theme: custom content types, permissions, workflows, views, email notifications etc. Yet, it's hard to reuse because bundled with theme. Also all fatness in theme is exposed through ++theme++, which could be a security issue depending the contents.

About Rapido. AFAIK Rapido views do not support the default toolbar Display menu. Also Rapido expects trusted TTW admins (its Python scripts allow plone.api by default that allows escaping its sandbox; here I like boring tech – the current scripts supported by AccessControl). I don't want to depend on it myself. Yet, it should be easy to implement Rapido support for those, who want to use primarly it. Main issue is possible namespace conflicts (AFAIK there can be currently only one active rapido folder – the on in the default theme).

10

I like apps too. They also could be named "plugins" (a lot of systems referer to this kind of thing that way).

1 Like
11

I also think .ini is probably better.

1 Like
12

For wider audiences it seems to me both plugins and apps might be confusing / hard to differentiate from regular add-ons.

Thus I'd suggest something more explicit for newcomers, like one of:

  • TTW add-ons
  • Restricted add-ons
  • Light add-ons
  • Uploadable add-ons
  • ...?
1 Like
13

Yes, this is probably the most difficult decisions here. TTW, Restricted and Uploadable are very technical terms. Light underestimates the possibilities. Simple add-ons might be little better (indicating they are simple to create), but still underestimating.

  • Plug'n Play add-ons
  • Live add-ons
  • Instant add-ons
  • Scaffolds (they can be customized TTW)
  • Jet packs
14 
There are only two hard things in Computer Science: 
cache invalidation and naming things.
-- Phil Karlton

(and the variation with '... and off-by-one-errors')

To borrow from gaming (admittedly also nerdy but maybe less so) I'd say

  • mods

but let's see if we can get some more suggestions in. It is indeed hard, and once done you're stuck with it...

1 Like
15

Loving that.

16

Thats easily fixed.

Not true. it has the same security model as other restricted python. It's the same tech. If plone.api is a security risk then its a bug that needs fixing in plone.api. Not much point having an api if it doesn't work in restrictedpython.

I think perhaps you should learn and use it before reinventing it.

zip addons or upload addons.

17

From the suggestions, I like:

  • Live add-ons
  • Plug'n'play addons
  • Mods (I think this is better than plugins/apps which will IMO really create trouble)

I agree TTW is too hermetic and that Light is too shy but Restricted, Uploadable and Zip Addons do not strike me as so bad either.

Another idea: Hotplug/Hotswapping add-ons, or something similar, which would go inline with Webpack's similar concept.

1 Like
18

I asked some non technical people today what they think would be better: no one understood what mods are. But everybody seem to know what apps are (we now have app stores, play stores, ...). So this seems to be very common and understandable. Also, everybody understood when I explained that add-ons are coming with the Plone installation and can be activated, and apps could be installed like any browser extension.

Btw, looking at the Chrome store they now have:

  • Extensions
  • Themes
  • Apps
  • Games

Thomas

1 Like
19

live add-ons gets my vote. I also like "Hot" add-ons but that might sound scary to some persons.

1 Like
20

If it's hard to choose a word, which means something in our context, what about something with less meaning

  • chords
  • pods
  • pearls
  • rocks
  • spots
  • specks
  • tapes