Can I use Calmjs to handle the JavaScript dependencies of my package?

Development Roadmap and Future
,
1

Continuing the discussion from Future of JavaScript in Plone:

we have many Plone add-ons that integrate some JS feature and are tedious to maintain; for the sake of simplicity let's consider collective.lazysizes which includes lazysizes.

every time a new version of lazysizes is released I have to manually download the new files, and put them on the right directory before creating the new version; take, for instance, this PR:

can Calmjs help me on this, @metatoaster? could it be possible to just update setup.py:

package_json = {
    "dependencies": {
        "lazysizes": "~3.0.0",
    }
}

run bin/buildout again and then maybe some other command to copy the new files to the expected location or this is not a use case considered on Calmjs?

1 Like
2

Updating the setup.py to include that information is only the first step so that the linkage from that Python package to that JavaScript package on npm is established. The tool to acquire those dependencies onto your development environment (i.e. installing that into node_modules) is done, and there is already a way (a not so good one that I am looking to improve). So right now it won't do what you want, but I want to get there. The example I also linked in the previous thread shows what currently can be done, basically using the declarations (in the current adhoc format) along with the JavaScript included within the various Python modules to create a complete (or partial) AMD bundle for consumption by some user agent through the implemented integration layers (plain Python CGI, or Sanic - Plone to follow at some point, hopefully).

Originally I really aimed to solve this issue is at the packaging level - I want to be able to have the standard python setup.py bdist_wheel command to also extract the require bits from the node_modules directory and put everything into the right place so the final Python wheel will include it, along with option to build/bundle the artifact(s) into the wheel, specifically the package's metadata sections (i.e. package's egg-info or dist-info directory). Given that Plone package doesn't do this to aid zcml declarations, I will need to do this when I get around to do the Plone support/integration bit.

If that part is done, and say if we can have entry points defined like this as per your use case:

    'plone.bundle.node_modules': [  # or maybe collective.bundle.node_modules
        'lazysizes/lazysizes-umd.js = collective.lazysizes.static',
        'lazysizes/lazysizes-umd.min.js = collective.lazysizes.static',
        'lazysizes/plugins/twitter/ls.twitter.js = collective.lazysizes.static',
        'lazysizes/plugins/twitter/ls.twitter.min.js = collective.lazysizes.static',
    ],

And a command can be created to put them into the right places - this is something I had considered doing, but I need to put that new registry together and the functionality together. While it might not address all your issues this might help with building the wheels a bit easier (i.e. without having to manually copy those four JS files every time a new release is made), and users of the package will basically get the copy from the wheel.

For development and situations where a user want complete control over the artifact generation (i.e. using calmjs.rjs), the other declaration will be needed so that, so that their generated artifact for use with collective.lazysizes outside of the Plone managed JavaScript/resource registry can also be done more easily.

Thank you for your question, it is giving me ideas on how I can make this better for everyone, even though right now it can't quite be used for what you want to do.

1 Like
3

thanks for your answer; my use case currently includes Plone 4.3 and Plone 5 and I'm against bundling everything together inside Plone for reasons I have stated in the past: is complex and is overrated.

we're currently using webpack to process and bundle things together at an add-on level; you can see such integration in another packages, like collective.newsticker and collective.upload.

ask @rodfersou in case you have any doubts.

4

I actually do have another question on your ideal use case

If it isn't putting some sources into the static locations accessible via zcml, what exactly do you mean? From the usage I saw in those collective packages, I see that it ultimately just generate a self-contained bundle to dump into browser/static within the final Python package. This is the use case I want to target. However, the way those two collective packages have done so isn't exposed to the standard Python packaging tools. The source files are only accessible in the repo, with only those who use zcml and Resource Registry can know where those finalised webpack artifact files are.

My idea for webpack integration might be something like this - Python packages to include their JS sources at a location accessible via pkg_resources so that they can also be accessed/declared using the Calmjs registry system (which are just entry points). A tool will be able to use the registry to generate webpack.config.js to either only bundle that one package, or include all relevant JS code from the parent packages, through command line options (or configurations declared again to the registry system, to executed automatically on package generation or when setup.py is executed). This will also allow end users of packages to not worry about figuring out where exactly the generated files are because they will be able to query them using the entry point system (specifically using the Calmjs helpers, or using the pkg_resources module for lower level use cases). As mentioned, these are just ideas and are mostly not implemented yet, but I am looking to do them as the time available to do so comes. Would something like this be ideal or useful?

The reason why I want this to be a reality is simply so that Python programmers who work with JavaScript are no longer shackled to a single CMS way of doing things. If a generic component can be created, declared via the entry point system, other systems will have a bit easier time to include the relevant scripts/artifacts without having to resort to manually figuring out where things are by looking at the source/repo. Then Django/Plone/Flask users can come in, take those packages, subclass/extend them in ways that make them more compatible with their system, which hopefully promote more code reuse across the entire Python world. This is why I ultimately started working on nunja as I do have Flask-based webapps and also Plone apps while maintaining server side rendering, though the goal is also to create generic templates that can be reused across both those frameworks, with the client side scripts be also usable across them.

5

@metatoaster you know Fanstatic, right? It is framework agnostic - not shackled to a single CMS way of doing things, as you say. Thus the package maintenance burden would be easier. I am actually surprised nobody has created a NPM -> Fanstatic tool. Or maybe someone has?

What do you (and @hvelarde ?) think of it? I wonder, could Fanstatic be part of what you're trying to achieve with calmjs etc?

6

Hmm, that actually is new for me, thanks for showing me that (for whatever reason in all my searching for terms like Python JavaScript module management I never came across it). At a glance I might be able to take some ideas, but as some others noted in the previous thread, repackaging existing JavaScript libraries available via npm into PyPI isn't exactly something I see being a desirable trait. They do use the entry point system to achieve similar to what I've done, except they do require a bit of glue code declared within the Python package (which I find intrusive) whereas Calmjs only require entry points. What fantastic has that Calmjs hasn't implemented is the ability to generate the markup snippets to trigger the script inclusion in the output html, but I am moving towards getting this implemented.

What they have is an interesting approach. While I dislike the glue code approach, what I plan isn't incompatible with what they got either, once Calmjs can support the declaration of what and how JS artifact(s) are to be generated for the given package (and dumped where). The result is that they would be at the expected location, and then some other framework (like fantastic or zcml/Plone RR) can then access/consume those artifact files.

7

I don't know if that's a good (or even desirable) idea; at the end I'm only interested in Plone add-ons… but that's me.

I was checking the package.json file and seems we already have what I was asking for:

so I only have to change the version number there and run a bunch of webpack commands to have this package updated.

learning the JavaScript way of configuring things is like having to learn Buildout again; but let's face it: JSON sucks for this!

I think @rodfersou solution is the way to go for us; now we need a recipe instead of adding a bunch of ugly parts in the buildout.

8

Hmm, that is fair. As I said before I do find myself building components that can be generic enough to be shared across different systems that I use, so if you ever find yourself in that situation please do consider the approach I've taken.