Building and operating a site using the Volto Plone 6 frontend with addons and customizations deployed behind TLS

Development Plone 6 UI (Volto)
, , ,
1

I ventured through a long and elaborate path to get to the point where I can build and maintain a Volto-based site with addons and customizations, all accessible via https/TLS. I've made many missteps and found myself in many dead-ends along the way. I do not think I could have avoided some of them, but do not mean that as an indictment of the guidance I found. I started when Plone 6 was just still a beta release and Volto was (and is) still taking shape, so it makes sense that there is not yet a “garden path” to follow. There's such great potential in Plone 6 + Volto that I found it worth the effort to find my way. I’m hoping that by describing what I discovered I can make it easier for others to use the system, so more of us can enjoy and contribute to the system's growth.

  1. Using the containers method for a vanilla build running behind a reverse proxy for remote access

    The first introduction on the website to running Plone 6 is Containers – Install in the Plone 6 documentation. It provides a simple recipe to use Docker to start the Plone 6 backend and Volto frontend so they can work together and give you a chance to try the system for yourself. I was delighted with how easy it was to do that, and happy to start to get acquainted with using containers via Docker. Operation based on containers quickly became part of my requirements.

    Trying it I noticed that I could access my Plone trial site with a browser local to the servers, but not when accessing it remotely. In the process of shaking this problem I learned a lot about how the frontend and backend cooperate. This was also key to my forward progress.

    I posted about my efforts, and the thread spells out the details:

    @tiberiuichim's suggestion resolved an obstacle so I could start the Volto frontend such that it could be accessed from a remote host. I described how I set things up in this thread followup. (I describe a more useful configuration in the next steps.)

    I submitted a documentation issue describing the problem that Tiberiu’s suggestion solved:

  2. Implementing TLS in the reverse proxy so I can access my sites via https

    TLS is an inevitable requirement for most of the sites I might build. Looking for instructions led me to the nginx, Frontend, Backend container example – Install – Containers – Examples of Plone 6 using containers documentation.

    Arranging to get proper TLS certificates using LetsEncrypt. I then had to resolve involved issues with Cross-Origin Resource Sharing (CORS) restrictions.

    For reliable security browser CORS restrictions require that resources like scripts embedded in a page are from the same address as the page itself. “Same address” includes the same port: to respect CORS security the browser can’t allow https pages to run scripts from the same host but a different, non-https port, like requests to the Plone backend server typically on 8080, or the Volto frontend server on port 3000. Yet Volto is designed to embed such requests in pages.

    The solution is to encode references to frontend and backend URLs as regular https URLs to the server host and use rewrites in the proxy server (nginx) to detect them and direct the requests to the intended server. Having the redirection happen in the proxy server circumvents the browser’s CORS restrictions without betraying security concerns: the provision is for the specific ports in our services and server - an explicit agreement with ourselves.

    Here’s the relevant nginx configuration for requests to the backend:

    location ~ /\\+\\+api\\+\\+($|/.*) {
      rewrite ^/(\\+\\+api\\+\\+\\/?)+($|/.*) /VirtualHostBase/https/$server_name/Plone/++api++/VirtualHostRoot/$2 break;
      proxy_pass <http://backend>;
  }

    In this community posting I asked for help with the CORS problems:

    @sneridagh's suggestions in comment 2 led me to consider how to express the backend and frontend links so that they went to the same host and port as the page (https://host) but the actual frontend or backend port could be discerned by the proxy service, which would use what it detected to direct the request accordingly. This got it mostly working: Comment 3 in that thread

    There was one small glitch remaining in this TLS connection. Pages would momentarily contain a “Connection Refused” message when they initially showed, almost immediately replaced by the content. I describe the mostly working configuration and the glitch in comment 4. In comment 5 @tiberiuichim once more resolves things, again clarifying the frontend / backend cooperation sequence. That led me to focus on the RAZZLE_INTERNAL_API_PATH setting, and arrive at a fully working configuration, which I lay out fully in comment 6.

    So now I’m able to interact with my trial Plone 6 installation through Volto behind TLS from a remote browser. Along the way I got familiar with using docker compose to coordinate operation of the service containers (proxy, backend, frontend). I’m totally sold on Docker and docker compose - they make orchestrate operation of the system components as easy as I can image. The results of all the subsequent steps will be run using this operational framework.

  3. Using the packages method to build a Volto container image that includes addons of your choosing

    As far as I can tell, the GitHub - plone/plone-frontend: Plone frontend Docker images using Node 16. image doesn’t provide a way to specify optional addons from the command line. To include addons we need to build them into a Volto image from packages using cookiecutter: Install Plone from its packages. We can then use that image in the docker compose configuration described in step #2.

    The resulting packages build setup is elaborate. make help tells you the available commands. You can include addons in the frontend and backend builds, using their respective subdirectories.

    While working on this I encountered a Dockerfile syntax error in the 'make build-images step that has not been fixed as of this writing:

    The syntax evidently works with docker buildx commands but not docker build. The fix is to change the Makefile to use the right docker buildx commands (which I didn’t investigate) or to use the Dockerfile syntax which works with the general docker build command. I submitted a pull request for the latter, which is easy for you to use if you need it:

    When including addons in the frontend you need to be sure to do a yarn add ... in the frontend subdirectory for each one. That will establish the current addon version as the one that will be used in subsequent builds. When you want to increment to a more recent version of an addon you can do another yarn add ... for that addon. (I don’t know whether doing an unqualified yarn add will update the entries for all your included addons, and would be interested to hear whether or not that is the case.)

    You can plug the images you build into the docker-compose.yml configuration arrived at in step #2 above. After building an image use docker images to see the name of the new image, and don’t forget to use docker image prune to dispose of intermediate container images produced in the image building process. (If you abort the build process you may need to first remove the residual containers. Look for two-word container names in the docker container ls (or docker ps -a) listing.)

    You can now run the system per step 2 with backend and frontend images including addons that you choose.

    There is one step further that I needed to learn to be able to fully customize my Volto installation.

  4. Using an addons development build to develop a customizations package or other addon.

    I did not find a way to incorporate custom Volto code source directly in the packages method described in step #3. (I tried using GitHub repositories as addons but encountered several problems using yarn to add GitHub repositories as addons. Some were trivial, but some were clearly not. It’s evident that people are not including addons directly from GitHub repositories or these problems would be shaken out. In any case, the addons development approach I’m about to relate is logistically better overall.)

    Plone 2022 training documentation includes Volto add-ons development documentation. (This is distinct from the Plone 6 documentation Volto Addons section, which did not lead me anywhere useful.) It guides you through using the yeoman GitHub - plone/generator-volto: Volto App Yeoman Generator (@plone/generator-volto) Volto project generator and @plone/volto Volto addon generator to establish a Volto frontend project that you can run from source. You can then add your addon and other sources to the project and run them directly from it.

    To use project framework with the docker compose orchestration:

    • In the directory where you established the docker-compose.yml file from step #2
      • Use docker compose start to start the composite system
      • Then stop the frontend using docker compose stop frontend.
    • Back in the generated Volto project directory use yarn start to start the project.

    The yarn-started frontend will run on port 3000, which is where the docker compose started nginx proxy will pass frontend traffic. The yarn-started frontend will connect to the docker compose started backend on port 8080 (or whatever you configured).

    A very nice feature of this Volto project system is hot reloading of the sources. Any file changes detected in the src directory triggers reloads of the Volto code, updating the frontend with code changes you soon as you make as you make them.

    Once I’ve got an addon developed you can plug the result into the package build of step #3 by publishing the addon as an NPM package. (You can do so at will with a free NPM account. You have to pay for a subscription if you want to publish packages with access restrictions. I can then rebuild my images including the new modules.

    Here’s a elementary addon I created which just configures some Volto settings: @myriadicity/myr-frontend-config. Visit the addon at npmjs.org to view the code.

    I should note that I found it necessary to publish the addons in an organization scope in order for yarn add in step #3 to work. I may be mistaken about that, but mention it for you to try if you find it’s not working otherwise. You also need to update the version numbers and republish the code in order for subsequent yarn adds to get the new version.

Wrap-up

I included a lot of the details that I learned and that I feel are essential to understanding what is going on. While hopefully all the details are useful, it helps to get perspective on what are essential three distinct activities:

  • Operate the facilities as containers: The docker compose operation of the container images
    • The result of step #2 above.
  • Build the facilities as container images: The means to change addons included in the frontend and backend images that are run
    • Detailed in step #3.
  • Develop and publish Volto addons for incorporation in the frontend image: The means to develop new addons and publish them for inclusion in the container images
    • Detailed in step #4.

You can also build backend addons for incorporation in step #3, but that is covered extensively in classic Plone documentation.

I can’t guarantee that the way I’m doing all this is the intended way for the system to be used. Similarly, it may be that other ways of doing things will emerge and take over. But I know that these steps have enabled me to move forward with using Plone 6 + Volto, and I’m hoping that the info will help others do so, too. The more of us that can use and develop the system, the better it will move forward and grow.

Lastly:

  • Thanks to everyone who answered my questions and provided guidance along the way, especially @tiberiuichim. It makes a huge difference to know that questions will be addressed when trying to use an elaborate system!
  • I welcome suggestions and corrections! I would like to make it as useful as possible. It may be that there are better ways to do what I described, and I would like to learn what those are, as well.
9 Likes
2

@klm Thanks for the appreciation!

The Volto documentation (which you linked to and said they're not good) sure could use a revamp. The reason is "historical", have to do with the way Volto is developed and the purpose of the documentation. It's no wonder that you found the Trainings to be more useful.

I have very high regards for the Pyramid documentation, they're very very well thought out and organized. And they're comprehensive. In that documentation, there are two "paths": the tutorials and the reference documentation.

With Volto's documentation we only have the "reference documentation" part, or at least that's how it's written by the contributors: whenever we add something we add a bit of reference information for that something. If we touch a greater part of the system, or we fundamentally change it, it's harder to reflect those changes in the documentation, as they'd require a big overhaul, systematic changes and there's still people that use the docs (using older versions of Volto) that would be confused, as something relevant to them is incorectly addressed because of new changes.

This is not too big of a deal except with the deployment story. "In the beginning" Volto was deployed and relied on the RAZZLE_API_PATH runtime envs and this is covered by the simple (classic) deployment documentation page. Notice it's the first option in the navtree. Then the seamless mode was introduced, which is a deployment method that relies on: the absence of RAZZLE_API_PATH and having the latest plone.rest 2.0 egg available in your Plone backend. Because at the time Seamless mode was introduced that plone.rest (and ++api++ traverser that is needed) was only available in alpha version, seamless mode wasn't "the primary" entry point. This should be rectified, cleaned up, overhauled, etc.

So, we still miss the tutorials part of Volto. Perhaps we could clearly link to the tutorials. I would actually be in favour of moving the tutorials to be part of Volto documentation, as that would imply that we make at least an effort to update them, otherwise they're updated only once a year, at Plone Conf time. And the Volto Addons training, which is something that I value highly, hasn't been updated in two years.

2 Likes
3

I appreciate that complexity of the situation. There's only so much time and attention, and it needs to be divided between making things work and describing how to use the stuff that's made. Any changes to how things work affects the descriptions. And people's time is limited in the first place! There's no simple answer.

Big course changes, like seamless mode, complicated matters. That's not a reason to refuse changing course, though. Development needs to be flexible, or you wind up solving only the problems you see at the start, not the ones you discover along the way – the latter ones often better serving your purposes.

Given that things are going to change and documentation is going to get stale, it becomes crucial to identify priorities. I see a crucial strategic benefit in prioritizing identification and elevation of guidance for the kinds of things that people who are new to the system and interested in trying it out want to do. "Pathways." Ensuring that such people find traction means growing the pool of contributors. I could have used such broad-perspective guidance – well maintained pathways to do the things I identified in my wrap-up.

I suppose that the tutorials you mention are the pathways that I'm talking about! So I'm for what you're suggesting. And of course people writing tutorials should actively pursue solving problems they encounter along the way, making bug reports as I've done and helping prioritize the bugs by indicating the pathways the bugs block. (I did a lot of that.) This seems like a way to ensure that attention goes to maintaining what's crucial.

1 Like
4

For all attempting to install and build, fresh cookiecutter-plone-starter installs (step #3) seem to be blocked, per this bug report: Recently installed cookiecutter-plone-starter frontend make install fails with ENAMETOOLONG · Issue #56 · collective/cookiecutter-plone-starter (github.com). I am still able to install and build using the less recent version I established, but this problem (with a particular npm package) has me stumped. (I copied over the working version I have and am able to use it on the host where the fresh version fails. That suggests that the problem is not specific to the host environment.)

5

@klm

I'm glad someone raised this issue here. Somehow I have the feeling that the topic of the Plone 6 installation and its problems is being neglected here in the forum.

When I first started exploring Plone 6, the first thing I looked for was a simple installation guide. Of course, I quickly found the Plone Documentation and the Plone Training pages, but I wasn't really satisfied with either of them.

In the video Mastering Plone 6 from the Ploneconf 2021 you could get a good impression of the possibilities.

And so I actually wanted to reach a state where I had the possibility to add addons in the backend as well as in the frontend. Or to create your own reusable data types as described in the training video. Of course, I also wanted to make changes to the appearance with Volto and create my own theme that I could use later.

Why the existing instructions did not bring me to this goal or what problems I had would lead too far here, but I then found a way that differs a little from the known instructions.

I first installed the basic requirements as described in the Plone documentation and then not only used the backend but also the frontend from the Volto repository ( GitHub - plone/volto: React-based frontend for the Plone Content Management System).

So make build-backend && make start-backend and in a new console make build-frontend && make start-frontend

Now I had the backend under api and could add my addons or my own packages and use bin/builout and my Volto paths under src and theme, which I could edit for my own theme.

I think the Volto developers only integrated the backend here for testing because each buildout deletes the backend data, but I can handle that

With this I am able to develop a website according to my ideas, but for the next step, to put the website online, I still don't have the right idea.

Maybe this is the right place to get some suggestions.

1 Like
6

Plone 6 is actually made of at least a few building blocks – the frontend, backend, and typically a proxy server, and often more – and there are various ways to get and use the pieces. I came to focus on building containers with the frontend and backend elements customized as I chose. I can run them (and a proxy server) using docker compose. That affords using docker volumes to mount filestorage and blobstorage from my filesystem so I can persist the storage across container versions and back it up, etc. Similarly, I use volumes to mount the LetsEncrypt certs for the proxy server. See comment 6 that I mention above for the docker-compose.yml file with these and other features.

Section #4 in my writeup details how to develop and package addons to include in the container build of step #3. That is how you can customize your styles and create new Volto functionality to add to your system.

That path is elaborate, more so because there are still some obstacles that need to be navigated. My writeup and some details in these follow-up notes describe what I think is a full path, including workarounds for the current obstacles. It hinges on using containers to install and deploy it all, which presents some overhead if you're not already acquainted with using containers (I wasn't). Getting acquainted with them, it seems clear to me that containers are an incredibly useful tool. I believe they're essential if you want to install, deploy, and/or customize Plone 6.

7

As I mentioned in a follow-up in cookiecutter-plone-starter ENAMETOOLONG issue #56, it turns out that the yarn install failure is apparently incorrect, the ENAMETOOLONG error is spurious and the yarn install succeeds (unless something completely separate goes awry). So a workaround is to ignore the yarn install and proceed with building the images (incorporating the workaround I provided for this bug, cookiecutter-plone-starter #55). I change yarn install in the frontend/Makefile to yarn install || true.

This workaround is not ideal because it means the frontend make install action disregards all errors that may arise. Intil this is properly fixed you'll have to watch for other errors, and in general the workaround can't be released as part of the build process.

8

Unfortunately, I was too optimistic about the ENAMETOOLONG problem. It prevents the link step of yarn add operations from happening, so you can't add stuff to your frontend. The problem doesn't show up with less recent builds, but that doesn't help anyone freshly trying to build the frontend container.

9

@klm

I try to reproduce your problem by following the instructions without problems:

Maybe there are two points making a different, i use python 3.10.9 with pyenv on a kubuntu 22.04 and i do not upgrade npm to 9.2.0 suggested after the installation of yo ... instead i install the 9.1.3 (npm install -g npm@9.1.3) this was suggested in a former version ...

After that make install and make start-frontend or any yarn add, for example yarn add volto-subfooter works fine

1 Like
10

When this happens, please create an issue, or add to an existing one, in the Documentation or Training repo, explaining exactly what is unsatisfactory and how we can improve. We have many open issues to improve the Plone 6 docs, and we need help.

The Volto docs using buildout for the backend are out of date, and should be updated. Plone 6 now uses other tools as explained in Manage add-ons and packages – Install — Plone Documentation v6.0.

This so much. Plone 6 developers are pushing other developers to use containers going forward. I used to resist containers, but have come to rely upon them and their tooling to make builds and deploys easy, fast, and reproducible. I don't regret it.

11

This is very good news.

I've been hoping someone else would report whether or not they encountered the problems, to gauge whether they are widespread or specific to particular builds I've tried to do. From what you say and recent experience trying to build on another host, it's looking like it's more towards the latter. (I still think there are rough edges that need attention. That said, it's early in this iteration of the platform, and people are working on it in various ways. I continue to feel this new iteration is very promising, and worth the investment.)

12

I believe I learned what I needed to do to get the heredoc syntax to work in this problem I described:

Instead of changing the syntax I used the docker buildx install command so that buildx would be used wherever docker build is specified (or something to that effect).

Buildx uses buildkit, an improved implementation of the Docker build backend. Install buildx so it is used by default improves build speed and functionality, but it allows syntax that is not backwards compatible. That is what tripped me up. I am going to switch to it, to get the benefits of speeding up the build process (particularly valuable while I'm in a phase where I'm repeatedly rebuilding) and not just so I don't have to change the packages-method build files syntax. I recognize, though, that people managing Plone 6 policies will need to invest some attention to institute this change in general.

14

I found an easy way to start multiple copies of the Volto frontend, based in the step #4 addons-development procedure above:

yarn start (actually, razzle, which is invoked by yarn start) will use the value of a PORT environment variable for the frontend port, plus the +1 secondary port. So, eg:

  • PORT=4000 yarn start

Will start the frontend and the secondary port on 4000 and 4001. This can be very handy for running alternate builds while developing, while leaving the primary build going.