I am a bit late to this party, and I see that there are a lot of good points by everyone here that I can respond to. In order to do so, first I am going to air a bit of "grievances" with regards to the current Plone JS development and tools, then lead into the solution I have worked on for the last half year or so that may be of use for everyone here.
One thing that I find after working with Plone mockup is that the existing way (Plone 5) of providing/using JavaScript is that the reuse story is quite specific to Plone. Reusing the JS code outside in a more generalized Python environment (i.e. with other Python web frameworks that is not Plone) requires a lot of hand holding and configuration file crafting to make sure the development environment can execute the relevant tests. (In my case, with my project was able to symlink to the mockup directory to pick up the library but this really is not portable/cross-platform (Windows users be damned) and not ideal and is an absolutely terrible solution, given that this is at mercy to some file structure, rather than depending on what Python speaks, modules). The alternative is to have every project write up custom, special snowflake configuration file that is minimally reusable/portable outside of that Python project (which is what everyone else does, and it drives me mad) - some use Bower, but this leads to dupes between the Python and Bower installation which is a different form of madness.
Secondly, the integration story. It is a great thing that Plone operates on a loosely coupled principle, however in practice (with how things are currently done) this also leads to a lack of cohesion between the projects such that integration testing suffers. There was a case where the Plone upstream library required an API change, but the mockup library did not update the test data for those tests and a new pull request that was merged simply commented them out because those errors looked irrelevant. Then this XSS issue came along, a fix was written, tests passed, but that feature was silently broken by it until those tests were uncommented and corrected before the actual XSS fix was proven to function correctly. Point is, if the mockup library had used actual test data generated by Products.CMFPlone, the API mismatch could have get picked up earlier. This is where I realized that a way to pin the versions against each other must be done somewhere, especially for testing.
Thirdly, no unified rendering source for both server and client side rendering. Specifically to Plone, the folder_contents view is JavaScript only. I like the idea where there is a dedicated module that deal with navigable tabular data with all the client-side bells and whistles, except for the lack of server-side rendering. I get it, people think server side rendering is legacy, however aside from Google no other search engines will run client-side JavaScript to trigger the rendering, thus damaging the SEO prospect (never mind the usability prospects for users that don't want to enable JavaScript at all for security reasons). With Plone mockup, the rendering was done using the Backbone library and as far as I know, no Python library can render that. I've heard solutions such as passing it through to PhantomJS on the server side to trigger the initial rendering but seriously that sounds like a headache to do. Why not have a templating language/system that can be used on both Python server side and JavaScript client side.
I could go on more (actually forgot a couple more points that I wanted to add to that while writing this; I could go on about inadequate testing with mockup but that's just reiterating points), but for the mean time, a solution will be nice.
To fix the last point, I abandoned mockup and decided to roll a solution that bring together Jinja2 for Python server side rendering and nunjucks for JavaScript client side rendering, and did it in a way that Plone can be out of the picture until I really need it by building on top of Python directly (i.e. use distutils entry points for registry like features). Do note, I pick Jinja2 mostly because the greater Python web development community is most familiar with it.
However, the templating alone does not solve the development and deployment story. I need a way to support the live reloading use case for rapid prototyping, being able to build and run tests against not only the code, but also the resulting artifacts that will also be generated as part of the system, and that the Python packages that I create must have a way to export the various npm packages required and the versions required. Lastly, not coupled to any specific JavaScript framework, but the thing need to be extensible to support future frameworks that may come along.
After about three months of work, I created and released calmjs as the core system to glue what I have together - it provides the basic integration with npm by introducing a package_json key for setuptools / distutils, so that these end up as package metadata that can be reused by other packages through the included calmjs npm tool which will generate a package.json for a given Python package in a way that also include npm dependencies declared by its Python dependencies. It also introduce another keyword that allows the declaration of JavaScript sources that should be exported under some specific key out of node_modules. Finally, allow the declaration of Python namespaces within that package that should also export JavaScript modules, creating a framework agnostic way to export JavaScript sources that is embedded within a Python module.
Using that as the foundation, I have also created calmjs.rjs which is the integration package with RequireJS (where I learned to use, love and hate with Plone 5), where the calmjs.rjs provides a tool (calmjs rjs <package>) that will generate an AMD artifact that will contain ALL required JavaScript (provided that all npm dependencies have been installed) for the given Python package(s). I did plan to also provide something similar to webpack, but due to the time constraints I have I was unable to start this (yet).
As for testing, I have created calmjs.dev which extends upon calmjs so that a test registry is declared, and introduces karma integration (along with the suggested minimum devDependencies which can be seen in the setup.py file) where to run JavaScript tests provided by a Python package (to tests the JavaScript exported by that module) through the RequireJS AMD framework is simply calmjs karma rjs <package> will build and execute the tests for the Python <package> and generate the artifact only on success. All of the above packages are released and available on pip for some time now, with more information available on the top level README file that goes a little bit deeper to the points I've made, along with a description on how the packages work and their goals.
With calmjs in place, I continued on the development of the Jinja2 / nunjucks integration package, nunja. This package is currently not released yet, but I've had success building on top of calmjs. Also, in order to facilitate the live reloading development methodology, nunja.serve was created for integration with various serving mechanism, most notably for the hot reloading without rebuilding (only apply to nunja templates and scripts, as I had neglected to do this for generic modules declared under the calmjs registry). First implemented serving mechanism was Sanic, but Plone support will be done if/when I figure out how my sponsors are going to address Plone going forward.
For demonstration, I have also created a separate project, nunja.stock, where a number of useful reusable templates have been created, one of which (nunja fsnavtree) is reminiscent of the structure pattern from mockup. Navigation on the client side will trigger pushState and client-side rendering, with server side rendering also available at any given path. To demonstrate the entire system, I have an example written; under the prerequisites section; a description on how the entire stack can be setup is available. The nunja bits are very alpha/beta quality (i.e. no release on pypi yet), but everything is fully tested (note the test coverage; yes this includes the JavaScript code) and fully cross-platform (tested under Linux/Windows/OSX, Firefox/Chrome/IE/Safari). Test data for the template rendering is also used by both Python and JavaScript tests and are exported, thus making it easier to pick out API changes that result in potential breakages, addressing the second "grievance" point.
Working at the Python package level ensures compatibility and reusability with other Python frameworks and packages and also other JavaScript frameworks, provided the compatibility packages are written at the correct layer. This is the approach I've done and so far it is working out.
It is still not perfect, there are still things I need to do. There needs to be a way to declare pre-built artifacts, so that when the Python wheel is generated, the relevant JavaScript artifact(s) will also be generated, included and be exported/declared in a way that makes reuse from other frameworks straightforward (i.e. the goal is to make npm/nodejs completely optional in production usage; only Python wheels are to be required for deployment). Also CSS declarations (or SASS or whatever CSS framework) are not done yet, so those who have tried the example and is attentive will note that the CSS is declared in a rather ad-hoc way. While I would like other developers to adopt the system I've created, having others just look and see the idea that having a more generic, lower level compatibility layer that bridges Python/pypi and JavaScript/Node.js/npm that is reusable for all Python package is basically one of my main goals here. Perhaps the Plone community might find the work done in calmjs and nunja be interesting and of use.
