Future of JavaScript in Plone

Development Roadmap and Future
1

Another report from PLOG, sorry for the delay :bow:

Attendees

  • Jens Klein
  • Víctor Fernandez
  • Gil Forcada
  • Johannes Raggam

We (i.e. the attendees listed above) discussed the current status of the JavaScript in Plone and a possible way forward.

DISCLAIMER

is the header big enough? That was just a proposal, there is still no PLIP for it, although the idea is to create one if the feedback is positive. So, please, we warmly welcome constructive feedback.

PROPOSAL

First we set some goals and we tried to identify the use cases:

Goals

  • Define a way to make JavaScript integration in Plone sane and great again (same with CSS)
  • Use modern JavaScript tooling and widely accepted way of bundling
  • Decouple Plone Python code and Plone JavaScript/CSS code
  • from Python POV be agnostic which tools are used for JS/CSS

Use cases

  • Plone core itself
  • add-ons that want to provide extra JS/CSS on top of Plone
  • themers that want to provide extra or completely replace Plone JS/CSS

Prerequisites

The default install experience, should not change at all: you create a Plone instance and JS and CSS will be provided by the default theme (barceloneta in Plone 5.x) without having to have any knowledge on JS tooling.

So in short: the usual workflow to develop Plone (not JS/CSS for Plone) should not need any change.

TL;DR

  • all JS/CSS released as npm modules
  • webpack will be used to assemble all JS/CSS and released as an NPM module as well to be used by for Plone core and extended by themers
  • integrators are encouraged to use the same approach
  • add-ons are encouraged to dually release on NPM and PYPI but a fallback (a simple resource registry) will be provided

The story

What will change is that those JS and CSS coming from the default theme will be created and maintained through a Webpack configuration.

For that all Plone related CSS and JS will be released as npm modules as well as the Webpack configuration itself. This way integrators will be able to re-create the default bundles on their own, as well as modifing them as much as they wish with only JS tooling, no Plone/Python needed at all.

In this scenario, the only Python egg package released on pypi that should contain JS and CSS will be the default theme, everything else will be in npm modules.

Add-on developers, just like core Plone, will be encouraged to dually release their add-ons: the Python part as an egg at PyPI, the JS/CSS at npm.
To install them one will have to add a line in buildout (for the python egg) and a line on the webpack configuration (for the npm module).
Run buildout/webpack and update their servers with them.

As a fallback for small/tiny JS/CSS fixes a simple resource registry will still exist but will not do any dependency management/concatenation/minification/etc it will deliver all JS/CSS registered there one after the other. So, it will be the responsibility of the add-on developer to ensure that these JS/CSS have all their dependencies resolved, etc.

With this fallback resource registry, add-ons that are still not ported might work if their JS provides all the dependencies or are lucky enough that the dependencies are already loaded by the time they run (they will be appended at the end of the scripts tags).
This resource registry will be prominently advertised as a fallback practice and documentation on how to use the best practice (i.e. release it as npm modules) will be created.
This way, hopefully, add-on developers are not left in the cold on how to port their add-ons while at the same time their are allowed to have a minimal integration in Plone without having to go through the JS/CSS route (i.e. avoiding webpack all together if they wish).

If an add-on is released as an npm module, one can integrate it in their own theme's webpack configuration. That will disable the bundle provided by the Python egg itself (if it is using the fallback resource registry at all). This ensures bundles are not loaded twice. Each add-on must provide a unique ID (namespace) for each bundle it provides (anonymous, logged-in, ...). The npm module should provide those IDs as metadata in order to let webpack read that data and add it to the theme MANIFEST.in. Plone will read the MANIFEST.in and disable those bundles.

The TTW theme editor will still allow to add plain CSS/LESS/SASS that will get compiled and added as the last extra CSS file being loaded.

TTW no JavaScript will be compiled nor we will encourage TTW developers to add JavaScript. The only JavaScript added is - as it is in 5.1 and earlier already - those for Analytics configured in Control Panel. However, if a developer knows what she/he is doing Diazo may be used to add a custom JavaScript file at the end of the chain. This will not be officially supported nor encouraged.

Why webpack

http://webpack.github.io/

  • maturity: a major refactoring of it already happened (v2), so it should be fairly stable and it has been around for quite a while (2012, which by JS standards that's A LOT)
  • it supports all kinds of JS: AMD, commonJS, ES6, babel (to transpile...)
  • there is integrations with Plone already: see Asko's and Eric Brehault's work
  • small overhead: together with ES6 (preferably), npm scripts and webpack itself is the minimal needed to start hacking on JS, so the amount of tools/technologies to learn is really small
  • it is backed by a huge open source community: more active developers than Plone itself

Migration

  • make the current (5.1) resource registry output a webpack configuration that can be used standalone

Roadmap/TODO

  • it will be nice to secure the @collective namespace in npmjs
  • zest.releaser could be improved to deal with npm packages as well to automate things (there is https://pypi.python.org/pypi/yarn.build that could be used as a base)
  • put all JS/CSS from Plone in npm packages
  • create a webpack configuration (based on Asko's work) and publish that as well as an npm module
  • write guidelines to port existing JS/CSS to the new approach
  • improve https://github.com/datakurre/plonetheme.webpacktemplate and host it on plone org itself (?)
  • (?) migration script to export current (Plone 5.1) JS/CSS to the filesystem so that webpack can process it

Final notes

On PLOG there were no major concerns (anyway we were only discussing rather than implementing) and the sprint in Finland later on this year can be a good opportunity to start working on it, specially given that the main idea is to make @datakurre's webpack plugin saner by avoiding lots of the black magic is currently doing to achieve more or less what we have outlined here.

5 Likes
2

I must learn ... but its ok. Standstill is the death. :wink:
The js world is so fast, the tools, the package managers and the frameworks. Is webpack really the solution for the future? i don't know. I have only superficial knowledge in the modern JS-World. I think we need also good Tutorials for Beginners if they would customize her the Plone Installation, e.g. add a JS Image Gallery Plugin.

3

Can you please be more specific about how this affects addons?

I want to use some JS addons with its JS and CSS files for my addon...are you saying that I need to package and release this stuff together with my own JS glue code as NPM package?

-aj

4

Please explain what this means and what the impacts are.

  • I must be able to integrate and maintain an arbitrary JS add-on easily (e.g. Datatables)

  • development changes to JS and CSS must be reflected reflected without invoking a build chain in between (local changes to resources files -> saving them -> browser reload -> changes should be live)

  • I don't want to be forced to build and release any packages for JS and CSS at all. In almost all projects since 2001 we release and deploy directly through the VCS. This workflow must continue to work - if necessary with some building pipeline that will run automatically as part a buildout run during deployment or update.

  • you mention PyPI/NPMJS: not suitable for projects where your project code is not disclosed to the public. It is also not suitable to require private egg repositories or JS repositories for private deployments. As said: releasing and deploying from VCS is common practice and it would not be acceptable to impose such requirements for deployments of Plone 6

  • it is still unclear how you deal with JS modules that are commonJS or AMD or whatever which require in Plone 5 special and different handling, Put yourself into the role of a half-talented web tinkerer or a researcher with basic knowledge in HTML/JS/CSS....you come across DataTables with this documentation (Installation) ... what would be the step to map this type of information with the Plone 6 stack. As always requested: the process must be easy and robust, it has to be documented and it has to be foolproof.

-aj

5

The idea is that you have two ways (at the end it's Plone, so no single path for all :slight_smile:)

  • fully embrace JS/CSS tooling, in this case use webpack just like you use buildout to assemble all your python eggs
  • provide the JS/CSS that the add-on developer on its own will have built

The greatest complains about Plone 5 has been that the resource registry is flaky, half backed, etc etc, so we cut on it and provide something that could lookalike a Plone 4 resource registry so add-on developers can, once again if they want, upload their JS and CSS and that will be put verbatim at the end of JS and CSS imports.

The verbatim part above means that the resource registry will be literally that, just a registry, no fancy magic will happen: no compression, no minification, no hashing of resources. We don't want anyone to use it, but as a fallback or for some small glue code that's probably fine, anything else, please use JS tooling that is as good (or as bad however you look at it) as python tooling to assemble it.

As for having to release on npmjs.com: no you are not forced if it's an internal add-on, if it's a public one, then we will encourage you to do so. If not, on packages.json you can put references to repositories directly, as long as you can reach them.

Truth be told, for this new resource registry there zero lines of code created, for the JS tooling there's tons of tools available and there's integrations (like @datakurre's approach) that can be used right now. You can ignore the Plone 5 registry altogether if you wish and have a perfectly compressed, minified and optimized JS and CSS right now. You don't have to wait for this to become a PLIP, get implemented and use it.

Nonetheless, if you wait for this to become a PLIP and all the bits and pieces are aligned as envisioned, you will not have to go through the hurdles that @datakurre and others are having now to have a decent JS/CSS integration with Plone.

1 Like
6

Sorry but I can't help to like your must on each bullet point: are you working with the same tools as 5 years ago? Or 10? Or 15?

Probably not, programmers moved from assembly to C because it was better and more productive. Up until some years ago, with only some jQuery it was enough as websites UI/UX had not so many requirements, but you can not try to keep that very same approach when the UI/UX requirements have changed so much.

As for integrating Datatables, there you go: https://www.npmjs.com/package/datatables it's already on npm, just add it to your packages.json, let webpack bundle it and problem solved. I'm unaware if you configure datatables somehow on Plone as well, or is completely JS only (i.e. your glue code just looks for specific tables and initializes datatables on them), but if it's JS only then you can save yourself a python egg and instead move everything on JS.

As for development mode: have you read about Hot Module Replacement? That's even better than live reload, but again, oh surprise, just like you have to install plone.reload to have the @@reload view available, you have to install and prepare your JS tooling to have Hot Module Replacement available.

As having to be forced to release JS and CSS packages: there are lots of JS/CSS packages that you can pull them through VCS, actually they call it a best practice (I hope that they at least point to a tag rather than to a branch to keep some sanity though).

As for the deployment pipeline: again @datakurre's approach has a zero downtime deployment. After fixing the minimal CSS/JS bug, webpack builds everything up and uploads to Plone as a theme.

As per your last point: that's exactly why we want to move to a complete JS/CSS split with Python. Webpack assembles and produces JS and CSS out of source files, kind of exactly the same as Plone 5 resource registry. The big difference is that the Plone 5 resource registry has been developed by just a few developers while Webpack (or probably quite a lot of other JS tooling) is being developed by a huge community (bigger than Plone community itself). So ask your newbie that reaches for the first time Plone: will you trust your JS/CSS to an unmaintained-halfbaked resource registry to bundle your JS/CSS or another tooling that is being actively developed by a huge community.

We are always talking about Webpack here but that is only what we recommend for Plone core, that does not mean that you are forced to use it. As usual if you use buildout or the unified installer to install plone it will be far easier to get it right and get help from the community.

Now that I look at the datatables installation you point to: https://datatables.net/manual/installation#NPM there you go, they already explain you how to install it with NPM. Then you go to your favorite/all-time-loved CMS (Plone of course) and you see that they are also using NPM and webpack to bundle all JS dependencies: perfect, no discrepancy between the two.

I myself still have to get deeper on JS tooling, just like any Plone newcomer needs to get deeper on buildout, ZCA, etc etc. It's not going to be perfect, there's going to be rough edges, just like the ones we have now. If not there will not be a new version of Plone nor more pull requests. We are just trying to make JS/CSS development saner and take advantage of the new and great tools that are available.

Final note: Webpack, AngularJS and React, all of them just had a major release recently, and given their huge community we probably will not see another one of them (i.e. massive breaking changes) for a while. Their communities are mature enough to not go "oops, major changes as we skrewed up*. That with Webpack being able to chew everything (RequireJS, CommonJS, ES6 Modules, TypeScript), is literally the buildout of JS tooling. So IMHO JS ecosystem will continue to change and evolve a lot, but these few technologies/projects I mentioned will most probably still be around by 2020 (don't quote me on that though :smiley:).

2 Likes
7

There is no reason for being thin-skinned just because one must remind you over and over again that the current JS/CSS situation in Plone 5 was broken and is still broken. The Plone 5 resource registry architecture and implementation never passed the prototype and alpha code quality stage. It's rotten, it's broken broken, it's unusable...and everyone knows it and but nobody dares to confess it...it's the mess we have to live with for at least two years or even longer until Plone 6 or whatever it will be called to become mature. Meanwhile integrators leave, customers leave.

So this is likely your last chance for fixing things and bringing the JS/CSS story back on track. And I did nothing more and nothing less than telling you what requirements from the integrators point of view and what non-integrators with average Plone skills expect and can handle. Many of you have a tunnel view from the developers and core developers prospektive and many of you do not care about technology adopters. So I will continue to poke you and raise my voice - like it or not.

-aj

2 Likes
8

It's pointless to say it again to you, but anyway, attempt 1000k: there is constructive, reasoned and polite feedback and there is finger pointing with blaming with discouraging, why do you always take the second? You want things to be fixed/improved the way you want to be, in my dictionary that usually means "asking someone else to do some work for you, and unpaid by the way". Where you see software of (quoting you) alpha code quality stage. It's rotten, it's broken broken, it's unusable, I see the people that worked on it and I just feel sorry for them to have to read that (most probably they will not even read that at this point anymore anyway).

You want JS/CSS back on track, then do what nowadays companies do: embrace JS tooling, integrate it in Plone via a diazo theme and be happy.

Tunneled vision or not, Plone is FOSS so you are always encouraged to make your contributions, and for an experienced developer (far more than me) like you it should not be that difficult, we don't ask every person that comes to community.plone.org to send pull requests rather than opening threads here, but someone with your knowledge could also take time from their paid projects to improve the tools he relies on.

At the end of the day, if this plan/idea that we set on PLOG is going to get implemented is because someone will do it, it will not fall from the sky.

2 Likes
9

I don't know what to say except that I want to support Gil's great report and his attempt to clarify what amazing possibilities this opens up for Plone in terms of new developer audiences and markets.

Andreas, please stop complaining about the resource registry. You've discouraged developers and core developers. Instead, you could have proposed a solution, and then we could have discussed that solution's pros and cons concretely and perhaps you would have discovered that even your solution had problems.

1 Like
10

Regardless of Andreas' criticism for the current RR, his use cases are real. And those are pretty much the same, why we invested a lot of time into making it possible to build and deploy all CSS and JS (and more) in a zip-deployable theme. No matter how vanilla Plone we are deploying, we always need to customize its theme anyways.

With webpack-buildable theme

  • we can integrate any JS or CSS library by following its normal installation instructions (most of current libraries have maintained NPM releases, but NPM can also checkout repositories – also our private ones)

  • we don't need to maintain a Plone specific JS or CSS add-ons, because we can use the upstream releases directly

  • we don't need to care about the different JS module types, because webpack can handle all of them; sometimes we need to shim modules with webpack loaders, but nowadays there's plenty documentation for that

  • we don't need to run buildout or restart Plone for only deploying of new CSS or JS

  • we can serve all frontend resources (CSS, JS, theme graphics, webfonts, ...) outside Plone

1 Like
11

Plone has traditionally embraced loose coupling, and this gets thorny with every attempt to solve the build/dependency problems in JS (which are distinct issues, I think).

I am a webpack user (for one of my add-ons, which has complex code built in ES6), but I am not sure about this claim that webpack in a theme is going to solve all the problems:

  • I want to distribute add-ons to various places (internally, externally, whatever).
  • JBOA: just a bunch of add-ons. The JavaScript in my forms add-on has nothing to do with the JavaScript for my extranet add-on, and neither has anything to do with my theme. I want to maintain this independence and loose coupling.

I still need that loose coupling; I need the ability to deploy from VCS without a compile step just like Andreas.

I like the idea of a bare/basic resource registry for this case.

Sean

12

See my comments above about loose coupling; there are going to be folks who are using it for darn-good reasons, with much technical consideration.

I dislike the normative baggage of frowny-face discouraging loose coupling (or minifying/compiling your own add-on resources for JBOA use-case) as a second-class citizen.

In the docs and in forums like this, if we tell folks we disapprove of their use case, we have problems. We DO want them to use a basic registry, if it fits their needs better than using a big-tooling ideology or throwing everything in a theme. If we are going to support a resource registry, even if basic, let's not demote its merit in the docs -- the people preferring loose coupling may be just as technically able as folks slinging webpack/npm pipelines.

Sean

2 Likes
13

It's correct that in similar use cases we have to both buildout & install the add-on and compile & upload a new version of theme. Sometimes it makes sense to wrap the add-on specific JS into a pattern and manage it separately from the add-on, but compile & upload is still needed. Also we need to compile & upload theme whenever add-on is updated. It's a nuisance, but we still remember several times when "harmless" add-on update managed to break JS on Plone 4, and feel more safe now.

That said, I don't oppose JBOD-approach. Even the current RR works for me :slight_smile:

14

I'm all for fixing the JS situation especially if it can still allow for TTW development, ie theme based JS and CSS. The idea to disapprove and disallow theme JS is not good as thats a very realistic scenario. The idea that all development should be "proper" and use "proper" complex build chains is short sighted. For core plone and for addons I think no TTW compile and bundling is ok.

This idea has been brought up over and over in the past. I don't really get it. Having easy to understand JS helps anyone building addons for plone, but it doesn't open up new markets. How? Rok said this, so many have said it before? Somehow JS developers are going to rush towards developing with plone? Why? It's still hard for them to understand and always will be. Same for other python developers. We concentrate on converting them. Why? They are happy with flask and django. They like to build custom apps. They will almost always pick a django based CMS even if it has less features.
So can I ask what possibilities and what markets @tkimnguyen?

15

Your replies shows [quote="gforcada, post:6, topic:4118"]
Sorry but I can't help to like your must on each bullet point: are you working with the same tools as 5 years ago? Or 10? Or 15?
[/quote]

People are used to work with tools they know, with tools that work. This is even more true in the field of web development for companies and institutions having implemented Plone by themselves without the expert knowledge of high-profile Plone developer. You must not forget this audience (which is often the case).
In particular you discourage the further adaptation of Plone and the future of Plone due to such a brokenness and bad experience by integrators, existing Plone users and potential Plone users. I can name at least three cases over the last 12 months. Such major brokenness harms the complete eco system and it must not happen again.

[quote="gforcada, post:6, topic:4
118"]
As for development mode: have you read about Hot Module Replacement? That's even better than live reload, but again, oh surprise, just like you have to install plone.reload to have the @@reload view available, you have to install and prepare your JS tooling to have Hot Module Replacement available.
[/quote]

Have I complained about hot module replacement? I did not. If you read my initial posting again: such an implementation must be robust, as easy as possible to use for integrators and adoptors (see my point above).

Also there is no chance for refinance any major Plone development at all. The Plone project market is almost dead. The only thing you can sell at the moment is support for legacy systems. At most systems will not see a future on an updated Plone version.

What a nonsense pointer to Diazo., Integration of CSS/JS modules has to work seamlessly for arbitrary backend code that will run on the default Plone theme. We are talking here about enhanced JS functionality that is needed for the Plone editor inside its working environment. Again, you are ignoring real world usecases.

Yes, Plone is FOSS. Your translation of FOSS seems to be: don't tell us about our issues, please go away and fix them yourself. The frontend development is moving too fast and changing too fast in order to stay in top . As stated above: this is even more true for Plone adoptors outside. If you forget about these: you will harm the complete ecosystem and in the end you will kill business opportunities and sink the Plone ship completely. Please change your glasses and try to understand the problems from a different prospective other than the Plone core dev point of view.

-aj

16

It was not me who brought this discussion up again. Do not always shoot the messenger. I made clear statements about expectations for the new implementation in order make it usable for integrators and potential developers in organizations and companies - nothing more, nothing less.

About discouragement: I have not seen much anticipation for the obvious problems and facts where the RR in Plone 5.X is broken. It is discouraging for people outside the Plone core dev community dealing with the brokenness. That's the problem.

Also: as an integrator I am not on top with all aspects of frontend development. You have to be a frontend expert nowadays in order to the frontend part right. It is pretty wild to demand to come up for proposals and solutions for every problem made by others. It is a valid position to expect things to work in a reasonable way after years.

-aj

17

First, I appreciate your work and your level and depth of understanding of real world problems at lot - not only in this case.

This all looks promising from what I have read about Webpack and from what I have seen in your former work related to Webpack. As stated earlier:; it is hard to stay on top with all aspects of frontend programming. How does the "new" approach handle JS/CSS in the context of backend functionality (when I need a JS module for a particular widget or functionality in the @@edit view) compared the usage of CSS/JS in a theme?

-aj

18

@gforcada I think it would help a lot to take a documentation led development approach. or at least a usecase approach.

ie for a bunch of scenarios, what are the likely steps someone would have to do to develop, and then deploy?

For example, lets say you want plone + PFG + plomino and a off the shelf theme. Is it something like

  1. Get initial buildout working
  2. add PFG and Plomino eggs to buildout
  3. rerun buildout
  4. npm install
  5. npm run build
  6. ??

etc.
It's really hard for us without an indepth knowledge of the JS tooling to imagine how this will affect us.

19

Probably not as flexibly as the old RR.

Webpack itself supports both building dedicated "bundles" for different use cases or "asynchronous loading" of rarely used assets within a single big "bundle". But all those "code splitting" points must be explicitly defined. So, it's possible to optimize JS/CSS for a particular site, but probably not in a reusable manner for a single add-on like with conditions in old RR.

Currently I don't use "asynchronous loading", but build static bundles similar to the current registry:

  • default bundle for all users
  • additional bundle for all logged-in users
  • additonal bundles for some JS/CSS heavy add-ons like faceted navigation

Then I include all generated bundles in theme HTML (injected by Webpack build ) and drop the unnecessary bundles with Diazo-rules (mostly according to the content HTML body tag, which includes e.g. the current roles, content-type and template in use).

20

For me this thread sounds like moving back to the Plone 4 RR would make almost everyone happy:

Plone could ship with webpack-built default bundles in the registry, but it would be easy for integrators to drop extra resources into it. And those who would build everything in webpack anyways would not care less.

But that would be a lot of work, and those who built Plone 5 RR may not be too motivated to help us in tearing it apart :confused: It's quite tightly knitted into CMFPlone and plone.app.theming (and some other packagesl).

2 Likes