How do I do cool devops with cookiecutter?

I have a quick project to throw together, and used GitHub - collective/cookiecutter-plone-starter: Cookiecutter Plone Starter is a framework for jumpstarting Plone 6 projects quickly. to get a quick start on it. This worked fine. I was able to do all my 'nocode' setup without opening up an IDE.

I did this on my local machine, and now I'm ready to throw this up on my hosting provider (an aws instance) and, well, I'm just going to spin up an ubuntu instance, do an adduser plone and scp the whole bunch of files cookiecutter created, plus my data.fs. Then start the servers. Yay.

But this takes no forethought and will this be a repeatable, maintainable build? When am I going to paint myself into a corner here?

I want to ask you guys what your processes are for deploying - especially with cookiecutter.


I'm used to the buildout process. I would create my plonesite as an addon, check that bit of customization into github, and push my custom buildout up to a hosted machine. Running buildout would build plone, and my addon, and I would just manage everything in buildout.

So, what's the new game here? Do I create the entire file structure locally, package it up, and deploy black-boxes up to the cloud as images?

I certainly don't want to create a repo for my entire backend or frontend. Probably just repositories for my backend python package (what I named the thing) and my volto customizations.

I've read some stuff in cookiecutter's devops folder, and I also see there are some github workflow .yml stuff in .github/ but I'm not sure how to use them. I have spun up docker instances before, but never created them myself, never used vagrant, ansible, etc....

I've been procrastinating this journey forever now. I need to start learning and doing this, and that's the advice I'm hoping to get. What to practice on first.

Here's what I got out of devops

devops
├── ansible.cfg
├── group_vars
│   └── all
│       ├── groups.yml
│       ├── host.yml
│       ├── swap.yml
│       ├── ufw.yml
│       └── users.yml
├── host_vars
│   └── rfaqsl-local.yml
├── inventory
│   └── dev.yml
├── Makefile
├── playbook-setup.yml
├── README.md
├── requirements
│   ├── ansible.yml
│   └── python.txt
├── stacks
│   ├── default.conf
│   ├── docker-compose-latest.yml
│   ├── docker-compose-local.yml
│   └── plone.yml
└── Vagrantfile

@flipmcf Take a look at the materials from the deployment training at ploneconf: Plone Deployment — Plone Training 2022 documentation -- this covers the Docker-based approach to deployment that is set up by the cookiecutter template.

The cookiecutter gives you:

  • An ansible playbook to do the basic setup to install Docker on the target machine
  • Dockerfiles to build images for the frontend and backend
  • A Docker stack configuration that includes the backend, frontend, and traefik as a frontend webserver.
  • A Makefile with commands to deploy the stack to the server that was set up with the ansible playbook.

The training goes through how to use it. (The training materials have you run the ansible playbook against a local virtualbox VM rather than a remote one, but the latter is what you would do for a real deployment.) Please have a look and feel free to ask questions if there are things that are still unclear.

1 Like

Oh, and the data. Usually for a production deployment I would use Relstorage so that it's possible to use the relational db's replication features. But then you would need to convert your Data.fs. It's also possible to put the Data.fs in a folder on the host VM, and configure the backend image to mount it in the correct location. If you want to run multiple backend instances then you need either a ZEO server or Relstorage.

1 Like

This is exactly what I needed. Thanks!

Having only a few hours a week to play with this, I'm finally coming back with the feedback. I'm juggling so many things.

The local development and deployment works perfectly following the docs and the cookiecutter repo.

Question: Is it correct to say that the remote deployment steps are targeted for someone using linnode or a vanilla AWS EC2 instance?

I think this was my first obstacle in understanding the big picture.

I think it's a "What is Ansible" question that I never asked. I know, I'm kind of late to the game to ask that question, but hey... we all have gaps in our knowledge somewhere. A quick 1 or 2 sentences to describe what Ansible is and why we are using it here would be useful.

"Ansible is a way to create repeatable virtual machines. We are using Ansible to create at least two VM's: a local, development VM to deploy and run our docker images in development, and eventually an identical VM on remote 'production' environment. With this, we can create repeatable, stable environments everywhere. Note we are not running the docker engine on our local OS, but on a guest VM that Ansible will manage."

I mention this because on the other side of the interface - the deployment side - I've been handed an AWS Management Console with Elastic Container Registry (ECR) and Fargate - which is "the AWS way".

So, I'm taking a different tack when deploying docker images. I'm not sure how useful this is to give back to the community as official Plone docs or how much we want to even appear to lean towards the "AWS way" (seems a bit amazon-gnostic) but it's what my employer wants, and I totally understand why.

I'll at least get my instructions into the community forums.

As for Relstorage vs Data.fs - yes, Relstorage is the path forward. But I'm wondering if it's even worth it for this 'small website' stage. Seems like a good place to take on some technical debt at this point, maybe even launch without it, and plan the migration or upgrade depending on the catalog growth. In other words, deliver the product and make the CTO decide what the risk/reward is on that one.

Oh, are we deploying the webserver (traefik) into a docker image, or onto the Host VM? I think it's getting installed onto the VM and we're scaling the plone application horizontally under the webserver using docker images Proxying port 80/443 to the frontend docker image?

So, if I were to prefer nginx, apache, or something else, I would change that in the Ansible config, not a docker config?

only frontend (volto) and backend (plone classic) are really docker images, so I think I answered this myself through process of elimination.

Any Linux (Ubuntu/Debian) server, it could be even your Raspberry Pi :slight_smile:

I'm going with Relstorage even for a small setup, as this helps to grow the solution later on (but we could ask, during the project creation) if you want a zeo setup.

We are deploying with Traefik in the Docker stack. We prefer Traefik to nginx/apache in this setup because it is way easier to configure and maintain (and get https working)

Of course, you could expose ports 3000 and 8080 from the stack and have your webserver running on the host itself.

Usually, in larger projects, I deploy Traefik on its own Docker Stack, which will rarely change, and then have many Docker Stacks for each site/service.

This is confusing to me. Please help unconfuse.

There are two docker images: "frontend" and "backend" Is Traefik included in each image? Only frontend? I could probably answer this myself looking at config files - but that involves work and learning and taking personal responsibility. I'm just not in the mood at the moment.

Yeah. I don't like that. Not sure why. I just feel odd exposing the actual application directly without a webserver doing the proxying. Some things are just much easier to do on a webserver than in plone/zope/watiress. I know it's possible, but seems like a bad way to deliver even a prototype. No good reason other than my lizard brain telling me it's a bad idea.

I will use 2022.ploneconf.org Docker stack as the example.

In line 4 we declare a service named traefik that is going to be our webserver.

  traefik:
    image: traefik:v2.6

    ports:
      - 80:80
      - 443:443

showing lines 4 to 9 of 2022.ploneconf.org.yml

You can see in the code above the traefik service uses the Docker image traefik:v2.6 and exposes ports 80 and 443 to the host.

Lines 11 to 34 will configure this service's deployment using labels:

    deploy:
      labels:
        - traefik.enable=true
        - traefik.docker.network=traefik-public
        - traefik.constraint-label=traefik-public
        - traefik.http.middlewares.admin-auth.basicauth.users=admin:$$apr1$Jv7k2JvK$uNmYFBXyov5NYPm/SzB09/
        - traefik.http.routers.traefik-public-https.rule=Host(`traefik.ploneconf.org`)
        - traefik.http.routers.traefik-public-https.entrypoints=https
        - traefik.http.routers.traefik-public-https.tls=true
        - traefik.http.routers.traefik-public-https.service=api@internal
        - traefik.http.routers.traefik-public-https.middlewares=admin-auth
        - traefik.http.services.traefik-public.loadbalancer.server.port=8000

        # GENERIC MIDDLEWARES
        - traefik.http.middlewares.https-redirect.redirectscheme.scheme=https
        - traefik.http.middlewares.https-redirect.redirectscheme.permanent=true
        - traefik.http.middlewares.gzip.compress=true
        - traefik.http.middlewares.gzip.compress.excludedcontenttypes=image/png, image/jpeg, font/woff2

        # GENERIC ROUTERS
        - traefik.http.routers.generic-https-redirect.entrypoints=http
        - traefik.http.routers.generic-https-redirect.rule=HostRegexp(`{host:.*}`)
        - traefik.http.routers.generic-https-redirect.priority=1
        - traefik.http.routers.generic-https-redirect.middlewares=https-redirect

There are many things in this snipped, including HTTP to HTTPS redirection, GZIP compression and so on (There is also the configuration of a UI, but let's not dive into that).

One thing you do not see here is any configuration regarding which sites will be served. That happens because other services use the same concept of labels to register themselves and configure webserver access to them.

Let's use the frontend service (a.k.a Volto) as an example:

  frontend:
    image: plone/ploneconf-frontend:2022
    environment:
      RAZZLE_INTERNAL_API_PATH: http://backend:8080/Plone
    ports:
    - "3000:3000"
    depends_on:
      - backend
    networks:
    - traefik-public
    - backend
    deploy:
      replicas: 2
      labels:
        - traefik.enable=true
        - traefik.docker.network=traefik-public
        - traefik.constraint-label=traefik-public
        # SERVICE
        - traefik.http.services.plone-frontend.loadbalancer.server.port=3000
        # HOSTS: Main
        - traefik.http.routers.frontend.rule=Host(`2022.ploneconf.org`)
        - traefik.http.routers.frontend.entrypoints=https
        - traefik.http.routers.frontend.tls=true
        - traefik.http.routers.frontend.tls.certresolver=le
        - traefik.http.routers.frontend.service=plone-frontend
        - traefik.http.routers.frontend.middlewares=gzip

showing lines 59 to 84 of 2022.ploneconf.org.yml

Under labels we have the following configuration:

  • traefik.enable=true : Tell traefik this service is enabled for proxying
  • traefik.docker.network=traefik-public : Tell traefik to connect to this service using the Docker network named traefik-public
  • traefik.constraint-label=traefik-public : Tell traefik this service should be grouped under the label traefik-public (There are cases where you want distinct traefik services to handle just part of the services)
  • traefik.http.services.plone-frontend.loadbalancer.server.port=3000 : Register, on traefik, a service named plone-frontend and this service will be listening on port 3000 (Default Volto port)
  • traefik.http.routers.frontend.rule=Host(2022.ploneconf.org) : Register, on traefik, a router named frontend that will respond to requests with Host-Header 2022.ploneconf.org (This is equivalent to adding a new server on nginx)
  • traefik.http.routers.frontend.entrypoints=https : The router frontend will listen to https requests.
  • traefik.http.routers.frontend.tls=true : The router frontend will have tls enabled.
  • traefik.http.routers.frontend.tls.certresolver=le : Traefik will generate certificates for the router frontend using let's encrypt.
  • traefik.http.routers.frontend.service=plone-frontend : The router frontend will use the service plone-frontend (Think of proxy-pass on nginx)
  • traefik.http.routers.frontend.middlewares=gzip : The router frontend will use the gzip middleware (To compress some resources).

A more complex example of labels configuration can be seen on plone.org stack. In there we have, for the same service (ploneorg-backend), distinct routers for:

  • beta.plone.org/++api++/
  • plone.org/++api++/
  • plone.org/ClassicUI/

In summary:

  • Traefik exposes port 80 and 443 (And those are the only ports exposed on the host)
  • Traefik handles Let's Encrypt certificate generation / renew
  • Plone Backend: Uses labels to tell Traefik how to route requests to it
  • Volto: Uses labels to tell Traefik how to route requests to it
2 Likes

Oh my. This was amazing. Thank you so much for spending the time to spell it out like this.

I'll read this and walk through it all myself and really internalize it.

I predict that this thread is going to get pretty popular with google searches now.

1 Like

Feel free to ask more questions here, as I know some of those concepts are too obscure at the moment.