[RESOLVED] Plone 6 docker backend - Multiple backends with a load balancer - Waitress Task queue depth growing

1

I'm noticing a growing queue in waitress when I do operations like copy and paste items. I only have one instance of the Plone backend and I'm guessing that additional instances will resolve this. Here's what I see in my logs.

backend_1  | 2022-12-07 17:14:01 WARNING [waitress.queue:114][MainThread] Task queue depth is 17
backend_1  | 2022-12-07 17:14:10 WARNING [waitress.queue:114][MainThread] Task queue depth is 18

I'm using docker-compose with relstorage and a standalone postgres (not running in docker). Is it enough to simply run?:

docker-compose up -d --scale backend=5

I'll test it out and come back and report.

2 Likes
2

We had this behaviour in a Plone 6 setup with 8 clients. Only a caching / load balancer setup reduced the requests. Since this has been active, we have never seen these warnings in the log.

3

Documentation is more sparse than I had hoped on running multiple zeo backends with a load balancer. I'm looking for the best way to load balance the setup if I use
command below.

docker-compose up -d --scale backend=5

That command launches the multiple backends :point_up_2:t6:

I noticed that if I have an entry like this in my docker-compose.yml, I get port conflicts when I use the scale option, as they all attempt to launch on port 8080.

  backend:
    build: .
   ...
    ports:
     - "8080:8080"

Changing the line to a list of ports ensures the backends launch on different ports

    ports:
     - "8080-8086:8080"
1 Like
4

Okay... I've found an example with haproxy. Created by @avoinea himself.
Let's see where this leads.

5

I looked at the example and came up with this configuration (I use relstorage)

version: "3"

services:
  lb:
    image: plone/plone-haproxy
    depends_on:
    - backend
    ports:
    - "80:8080"
    - "1936:1936"
    environment:
      FRONTEND_PORT: "8080"
      BACKENDS: "backend"
      BACKENDS_PORT: "8080"
      DNS_ENABLED: "True"
      HTTPCHK: "GET /"
      INTER: "5s"
      LOG_LEVEL: "info"

  backend:
    build: .
    environment:
      RELSTORAGE_DSN: "dbname='plone' user='plone' host='${DB_HOST}' password='${DB_PASSWORD}'"
    volumes:
    - ./var/import:/app/import
    ports:
     - "8080:8080"

volumes:
  data: {}

Seems identical, apart from using relstorage instead of a zeo db.

Running this command

docker-compose up -d --scale backend=5

I get the following error:

Error response from daemon: driver failed programming external connectivity on endpoint backend-5 (5b3c05d1c012d82f8275ede783b5cadcda4823e98bd575bc5520a50e936f25ef): Bind for 0.0.0.0:8080 failed: port is already allocated
6

I've settled on a simple, nginx-based load balancer (not running in docker) in front of my zeo-client replicas running in docker-compose.

Here's the configuration (for my future self):
docker-compose.yml
(the db DSN config is stored in environment variables)


services:

  backend:
    build: .
    deploy:
      replicas: 4
    environment:
      RELSTORAGE_DSN: "dbname='plone' user='plone' port='${DB_PORT}' host='${DB_HOST}' password='${DB_PASSWORD}'"
    volumes:
    - ./var/import:/app/import
    ports:
     - "8080-8083:8080"

volumes:
  data: {}

This configuration launches instances running on ports 8080, 8081, 8082 etc....
The deploy, replicas lines replace using ---scale on the command line.

My site.conf file for nginx (placed in the sites-available/sites-enabled folder) looks like this:
upstream www {
server 127.0.0.1:8080;
server 127.0.0.1:8081;
server 127.0.0.1:8082;
server 127.0.0.1:8083;
}
server {

server_name www.example.org example.org;

proxy_set_header        Host $http_host;
    proxy_set_header        X-Real-IP $remote_addr;
    proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header        X-Forwarded-Proto $scheme;

    client_max_body_size    20m;
    client_body_buffer_size 128k;
    proxy_connect_timeout   60s;
    proxy_send_timeout      90s;
    proxy_read_timeout      200s;
    proxy_buffering         off;
    proxy_temp_file_write_size 64k;
    proxy_redirect          off;
location / {
    rewrite ^/(.*)$ /VirtualHostBase/https/$http_host:443/Plone/VirtualHostRoot/$1 break;
       proxy_pass http://www;
}
listen 443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/www.example.org/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/www.example.org/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

}

server {
if ($host = example.org) {
return 301 https://$host$request_uri;
} # managed by Certbot

if ($host = www.example.org) {
    return 301 https://$host$request_uri;
} # managed by Certbot




server_name www.example.org example.org;

listen 80;
return 404; # managed by Certbot

}

1 Like
7

For caching, I used Plone's built-in caching configuration.

Step 1 - Import the "without caching proxy" profile:

Go to Site Setup > Cache > Import Settings > Without Caching Proxy
Screenshot 2022-12-14 at 1.44.36 PM

Step 2 - Enable caching

Go to Site Setup > Cache and select Enable Caching.

1 Like
8

Where the "replica" and "deploy" came from?

9

It's part of docker compose
https://docs.docker.com/compose/compose-file/deploy/

(Not sure if that's what you were asking)

1 Like
10

One issue I've come upon is that my current setup does not support rolling restarts.
This is a simple setup for a single machine but the lack of rolling restarts may force me to look at docker swarm or similar.

11

Coming back to this:
I still believe docker swarm or kubernetes is the next step, but...
The incremental step would be to add rolling deploy capabilities on top of my docker compose setup.
For this I will be borrowing from this guide: