Add-on deployment TTW

Wordpress allows to download and install add-ons directly from its admin interface for quite a long time (and we already talked about it).
I just want to mention that BackDrop does the same now:
https://backdropcms.org/news/backdrop-1-4-0-released
(BackDrop is the Drupal fork which occured between Drupal 7 and Drupal 8)

1 Like

if TTW addon install was a mode you could enable in the filesystem. So if you wanted to accept the security risk you can. Even if was linked to debug/fg mode. We can have a big banner in teh control panel saying Plone is running in an unsafe mode.
Then anyone trying out plone doesn't have to learn buildout. That is a big win for onboarding.

I was wondering if it would be better to have 2 concepts:

  1. add-ons, which can be installed TTW(and they live and hook in like themes currently do)
  2. extensions, which are only FS
1 Like

This more-or-less boils down to:

  1. downloading addons
  2. storing them in ZODB (no disk access because security)
  3. custom importer that loads modules/code from ZODB.
  4. loading ZCA configuration after install and on startup.

I do see a challenge in resolving dependencies and compiling c-extensions: how can buildout and the TWW addon installer both resolve the same dependencies. How would pinning and upgrades work?

Sidestepping: my biggest use case would be installing hotfixes. An addon could phone home to a trusted/internal server and fetch security patches. One could run a medusa thread to do just this :slight_smile:

1 Like

I would have some concerns on this: one is security, as @djay already stated, and the other is add-on removal.

we all know that not all add-on developers care enough to write code to remove their features and this could lead to broken sites; this problem has been reduced lately but I would say we're not free of it.

IMO, I think we are again dealing with a lot of new features; don't get me wrong, this is not bad per se: all developers like to play with new technologies and that's pretty important; the problem is leaving behind a collection of half baked functionalities that don't work as expected for everybody else because that will backfire at the end.

so, yes, this could be interesting but not now or not with high priority.

having said that, if someone wants to bring a proof-of-concept in the form of an add-on, I'm totally fine with that.

installing security updates or updating Plone TTW is more interesting for me as high priority and could be a good starting point to learn the implications of this.

It could make sense, that's true.

For instance, an "add-on" could be a Rapido app (so just a folder to add in the current Diazo theme, no need to hook anything, and it is perfectly harmless to do it TTW, it would just automatize what we already do manually using the theming editor).
While an extension would be an actual Python egg.

1 Like

+1 for separating TTW "add-ons" and filesystem "extensions".

Instead of allowing Plone instance to rewrite its code, I'd embrace everything that can already be done TTW, like everythinging supported by GenericSetup (DX types, workflows, registry...), everything supported by plone.app.theming. Then we have Rapido and, in the future, also Mosaic.

+1 @vangheem for ttw addons. I think some of our current addons could be done using rapido, fragments, DX, GS etc if we provide a bit more infrastructure/apis. For example we'd need ways to create tiles TTW, content rules TTW, menu items etc. Some way to have resources installed and uninstalled into the current theme (a better way to have one theme be included in our main theme?)

It would be useful to work out which of the current popular plugins could be done TTW in the near future

  • solgema.fullcalander or ftw.calender?
  • eea.facetednavigation?
  • webcouturier.dropdownmenu (personally I think this should be builtin. probably as an enhanced navigation tile).
  • Products.PloneKeywordManager?
  • shopping cart?
  • collective.disqus
  • collective.addthis
  • others?

It would be really nice to have an app store (perhaps within sitesetup itself), so you can select a plugin and install easily.

@jaroel: To install fs packages I'm not sure you need to load code from the zodb. You could store a zip in the zodb and each instance can run setuptools on it to install it to disk and then restart itself. Obviously very insecure and would not be recommended. Perhaps there is some automatic roll back if an instance doesn't restart. And perhaps security could be improved slightly if setuptools is run with a special user account?

I don't really see the benefit of redoing the buildout eggs += ... story via the ZODB.

From a UX perspective, all that this achieves is that there's a set of extra functionalities in a vanilla Plone site, which is not enabled by default, but which can be easily installed TTW by clicking a button. However, the simplest way to achieve that, is by shipping a set of selected extras as eggs += ... in the installer buildout, which you can then activate via the Add-ons control panel, without requiring any new technology.

A more fancy solution would be, to maintain a extends += dist.plone.org/5.1/extras.cfg cfg with all these add-ons, which means the set of verified add-ons can be extended easily (especially with a re-buildout control panel button). That would appear to me the most easy route towards a TTW app store, with the option to centrally provide QA that mitigates the security and stability risks.

IMO this is quite a different story from the Rapido / theme hacking TTW customization story where you're actively reconfiguring and changing site-specific logic. Not bloating the "real" TTW hacking story with a risky ZODB buildout scope and keeping the two scenarios separate makes each of those more attainable.

1 Like

Not entirely sure I understand @gyst. Are you suggesting something like distributions could be another way to solve this? ie someone selects, tests and curates all the possible plugins someone could need and the standard installer has them all included already? Isn't the problem there that someone has to choose, and test and curate? and its good to have competition in an ecosystem so there is innovation.

I think other two options you have somehow combined but they are different suggestions

  1. restricted python addons (from @vangheem)
  2. mode to allow fs package install via sitesetup
  3. distribution(s)

Whats interesting is none solve all the use cases.

  • The only one that makes security patching easier is sitesetup fs packages.
  • The only one that allows easily trying out new stuff securely is restricted python addons.
  • Distributions is the only really secure way to give the full use of tools you need to build many kinds of sites (esp given most restrictedpython regularly needs whitelisting of many python libraries which can only be done with a fs package)

So I'm not sure we can dismiss any of these options yet.

+1

I really don't get the point of this whole discussion.

Who is the target audience of this feature or which target audience do you want to win for Plone?

With the shrinking importance of Plone (and other CMS) you will not win a new target audience with such a feature.

Perhaps 10% of our Plone customers of the last 15 years was running Plone itself. In the majority of projects the hosting and Plone administration is done by Plone administators and integrators. I can not remember a case where a Plone customer had the need or the wish to install some add-on by itself. It would be nice having this functionality but technical details are too complex for moving a complex add-on infrastructure to a TTW installation. This is never going to work properly. In the end you are trying to create a two-class ecosystem of add-ons. The TTW installation may work for a small class of add-ons with a limited functionality...the overall set of add-ons providing a broader functionality and a deep integration into Plone will never ever be installable this work...so please focus on the things that matter instead of wasting your limited and shrinking resources into a feature giving us a major advantage...as said: very high risks and hard to solve if not planned carefully in advance (otherwise you need up with the same nightmare as we see it right now with the resource registries).
Happy to discuss this in person @PyCon...

-aj

1 Like

@zopyx while plone is losing market share yes CMS is not. Only if you see a CMS as a way to build an app in which case its losing to frameworks like django. However 50% is cms sites and growing.
One of the reasons plone is heading to less relevancy is because of exactly what you describe. Plone only works if you outsource it to a paid for integrator and their own hosting. In the world of CMS that makes plone expensive and inassesible despite it being a very good CMS which is easy to use and easy to theme.
Given how cloud is now, we can have one button plone sites very easily. Except for two things

  • how do they write custom code given all the training requires them to learn buildout and provides bo guidance on how to get their code from their machine to their host.
  • how do they install plugins.

Ttw plugin install is probably one of the most importabt things we can cocentrate on to help plone and save all our techincal and business investment in plone IMO. It is exactly this old way of thinking about CMS @zopyx that is causing plone to concentrate on the wrong things and lose relevance.

Like @zopyx says: what't the point?

If you want site admins to be able to one-click install anything, just ship a universe.cfg with a gazillion add-ons. What capabilities does a high-risk project of re-engineering existing add-ons as TTW RestrictedPython ZODB installables deliver, that a simple extends += universe.cfg cannot do?

1 Like

Maybe 'most used add-ons' should be in the buildout.cfg (they can be commented out for that matter)

#uncomment the line below to make it possible to add Disqus commenting platform API into Plone ( https://github.com/collective/collective.disqus )
#collective.disqus 

Or maybe even:
#uncomment the add on below to check out all all available add-ons
#collective.everythingthatisavalableforplone5

Re-running buildout after months or years scares anyone and can lead to a non working environment, plus need a restart. Buildout is ok to build a project, not to add add-ons on the fly, I think.

It is not easy to solve this problem with Python in general, is there a cms written in Python with ttw addons based on eggs?

This is an old Plone3 gripe. Plone itself has been properly version pinned since ages, and anybody who runs a custom buildout in production without proper version pinning is a fool.

If restart is really that big deal for you, you can run ZEO on RelStorage which gives you zero-downtime rolling restarts.

@yuri +1 Buildout scares me whenever I run it and I do this everyday for a living. It's not for the faint of heart. I just spent 2 hours today trying to build plone because something changed in homebrew that made pillow and lxml not compile that lead to cryptic buildout errors. Last few weeks we've had many requests on the forums that amount to issues when installing addons.
Having a buildout with all addons, or even a docker image, is just not realistic.

@gyst whats with the stop energy? The usecase is crystal clear. If we want to grow plone we need an easier addon story and an easier hotfix story. Yes it doesn't help my business or your business directly, or any core developer on this list. But CMS's that are relevant are the ones that are widely used. Widely used is because you are able to get up and going yourself and not require integration services or special hosting. No one expects a CMS to be this hard these days. If we don't help people be able to do plone themselves, no one will ask us to help them when they need more help.

Even if I'am more the Plone-as-a-framework user I would appreciate a story, where simple integrations and customization including TTW-templates and code are installable as zip-files by upload.

I know that there are more people out there using Vanilla Plone TTW than it seems. This are not the loud people speaking up or even reading in here. I think @tkimnguyen was it who told us a good story about such people at PLOG last year.

For sure this would be kind of limited to whats possible TTW and and I dont see Python eggs here as the way to go. I'd also not make a difference between theme package and functionality package, because thats only confusing.

Keep in mind to not run unrestricted code or even do filesystem modifications done TTW, then all is possible.

I think all in control-panel including types and theme, simple template customizations and possibly add ons like rapido are perfect candidates for such a feature.

Since Plone is a community driven project with different needs I understand that there are different points of view on this, but this means also: Go on, PLIP it, discuss it as a PLIP!

1 Like

A said there is a huge technical risk here. If you can not provide a technical solution that is working for all add-ons then you will end up in two-class society for add-ons - good add-ons vs. bad add-ons, old add-ons vs. new add-ons. Since I doubt that a TTW installer will work for all and everything you will introduce a new installation method while leaving the old buildout method in place for the "bad" and "old" add-ons....what do you win?

"Old thinking"....perhaps better "realistic thinking"...True, the world is moving to the cloud. Since we all (including to CMS) lost the "CMS war" against Wordpress we will not see a growing market share for Plone in the future...everything else is wishful thinking. There will be a niche for Plone to exist in the future but this is not the niche where "going into the cloud" is a huge argument. Perhaps the prospective in US is different from the one in Europe or Germany in particular. But the large and important consumers of Plone at this time are organizations where moving their data into the cloud in not an option - for the simple reason of privacy.

And my fear is again that such a feature with lots of risks ends up in some major release - with a half-baked or broken implementation as we have seen it in Plone 5.0 (speaking of the toolbar and resource registries).

Again: please focus on the things that matter and that are well-understood..otherwise you are continuing to dig Plone's grave.

Andreas

1 Like

Some brainstorming
To help the fools: could it be possible that when you (first) run buildout, whatever add-on that is not pinned is 'automagically' added to some version.cfg file.
Then, it could maybe be possible to rerun buildout without much risk of messing up too much ?
maybe even 'rerun buildout from Plone itself' with a 'dry run option' ?