I am pleased to announce the release of calmjs-3.0.0, which supports the generation of webpack artifacts as part of the Python package build through the calmjs.webpack package. Some time ago I had a couple inquiries about webpack support, and it is here now.
Also, about five weeks ago ago I gave a talk at Kiwi PyCon 2017 about using Node.js/npm with Python/setuptools. In the talk, the Calmjs framework was introduced as a method of ensuring dependencies on npm for a given Python package are communicated to the Python environment it may be installed in, so that its dependants may also know about them. If watching someone talking is not palatable, the following is basically a personal anecdote on working with JavaScript in Plone, or rather another way to cover the material that was in the talk.
I had started this project originally due to increasing amounts of frustration trying to get Plone mockup working in a well-tested manner that actually tests against other browsers that it was supposed to work for. Note that on the CI services, a number of platforms are listed as tested however if one were to look at the logs closely, Firefox/Chrome aren't actually used, but the standard headless browser PhamtomJS was used in every case (I probably should have made a pull request to correct that, but I've grown severely allergic to working with the configuration soup that mockup has after all the time I've spent swimming in the likes of it while building Calmjs).
While mockup from the surface looked like a well-tested, feature-rich library, with the structure pattern looking like a reusable pattern that can be adapted to display any navigation table. Turns out that was not exactly the case, but I was able to make various changes that moved it towards that direction, and the mockup development team had graciously accepted the patches I made towards making that story better, which I am thankful for.
However, the difficulty of the integration dance really bare its fangs when I started building my own extensions to the structure pattern in a separate library that reused mockup. The programming part was not hard, but getting the development environment working with tests running was, at best, mildly annoying. As usual, a number of manual copying/editing and locating needs to be done to configuration files before it works for the development tree, repeat ad nauseam for every package that need to reuse and test against all this work. This situation is untenable, especially if I have to pass this on to other, less experienced programmers that don't know or lack the time to learn all the intricacies of various Node.js build systems (plus this complicated soup with Makefiles, grunt/gulp and whatever else).
Also, given that I found that I will need server side rendering done using the same look and feel and that mockup is client-side only (unless somehow the same view can be rendered through Node.js, but I am not even going to attempt trying to get Plone proxy'd through Node.js), so I started something afresh.
Initially, I built a rendering framework that make use of the commonly known Jinja2 templates (putting this politely, TAL is alien to Python programmers that haven't seen Plone), so both server-side and client-side rendering (through Mozilla nunjucks) can be done using the same template code (it's called nunja, and is very much a work in progress). Then I realised I do really need a reusable toolchain to glue all this stuff together and to build the resulting artifacts, and three months later Calmjs with RequireJS support was done (I thought that was the end, how naive).
After I posted about the Calmjs project in this thread about future of JavaScript in Plone, a discussion in another thread got me to fully realize that Webpack support would definite be useful (and that the Plone JS development was definitely moving towards that). So the next thing I know I ended up spending more than a year cleaning and polishing everything up so that webpack is supported, while being side-tracked with bugs in various libraries, such as slimit (and also getting very fed up about its pile of bugs (I referenced a number them in the changelog for calmjs.parse), lack of maintenance (read: abandoned) by its owner). Ended up forking that (into calmjs.parse as mentioned) and spent three months forking/rebuilding that JavaScript parser library (do note, Plone does use slimit for JavaScript minification, and there are various edge cases with the parser and minify printing that result in broken "cooked" JavaScript file). After doing all of that, I was able to generate source maps as part of the unparsing (pretty-printing) process for the case where I need to manipulate the input source file.
Oh, I should mention that calmjs makes it possible to run JavaScript tests against a finalised built artifact without having to copy/edit yet another configuration file. This is necessary given the state of perpetual buggyness of Node.js/JavaScript build tools, and as mentioned if someone used slimit to minify some JavaScript code, potential bugs can be introduced and having test cases being run against that final artifact to figure out what the bugs are will be crucial for a quality assurance process.
In fact, as a demonstration, I have pushed a development branch for nunja.stock showing how building and testing might work. That package is exists independently from nunja but depends on both its Python AND JavaScript code for complete functionality. The travis-ci configuration serves as a listing of all the commands required to get complete testing and development environment for that package, and one such one on travis (shows the build itself, with karma running on with a development only config, and then the calmjs artifact build process which built the webpack, then RequireJS bundle, then karma running against both those bundles using the same tests). Those artifact build/test commands make use of the declarations by these entry points, so they are defined as part of the Python package itself for reuse by other packages (e.g. locating them). Do note that I only just got this checked into CI, so only PhantomJS actually works with both Firefox/Chrome tests appear broken (as I am in the process of ripping out RequireJS specific test setup things, so bunch of skipped tests when running with webpack, and that still doesn't work in real browsers).
After all that, I got back to finishing up the the webpack support, then finish up the artifact building/inclusion process that it is now possible to use Calmjs to declare JavaScript artifacts that should be built as part of a Python wheel, so that integrators/end-users will not need Node.js to build/make use of the JavaScript code that may be included or needed by the given Python package.
So yes, while my journey originally started at Plone mockup, I ended up building an integration framework for Node.js build/dev environment. When I started working on this, I had hoped I would have time to build a fork of mockup that would build using calmjs, however as I have moved on from using mockup this is no longer on my priority list (plus I have other objectives that my employer is moving me towards, given that I've spent a year not really working directly on them). However, if the Plone developer team finds the Calmjs framework worthwhile for the JavaScript development with Python and would like help to make the best use out of it, I will be more than happy to work towards that.
Lastly (this post is getting rather long), there are some more points that I want to make/emphasize but I am just going to leave them as a summary note... I guess this is a tl;dr (if it is still too long, oh well, go read it):
- Calmjs is NOT a replacement for npm or Node.js/JavaScript packaging; a standlone, pure JavaScript library is possibly better served by npm, instead of Python/setuptools.
- However, Calmjs makes it possible to integrate the above npm package (or any other ones) back into Python in a managed manner, and for management of JavaScript code specific to their host Python package, in a way that is reusable by the rest of the Python ecosystem through setuptools (which is ubiquious to Python these days).
- My problem isn't with Plone, but with the general Python community about their lack of good software management practices when it comes to JavaScript. Treating that as a naive asset management issue will quickly turn a whole project into various forms of liablities (how many .conf.js files (and *test*.js) are there in a typical Plone or Python project these days that cannot
easilybe reused outside of their git repo but is critical for the development/build/testing process?) - Developers should be able to easily run the complete test suites offered by their dependencies against the artifacts produced by their packages to ensure their extensions to the upstream framework did not break any existing functionalities, if they care enough to.
- Calmjs can now be used to produce both RequireJS and Webpack artifacts from JavaScript code exported by Python packages, and test against them.