Apache/zeocluster proxy errors

Plone Support Deployment & hosting
1

I frequently receive proxy errors from Apache when it tries to connect with the zeocluster.

Here is an example of one of my Apache virtual host configuration (with the names changed to protect the guilty):

<VirtualHost *:80>
        ServerAdmin example-admin@example.com
        ServerAlias test.example.com

        ErrorLog /var/www/example/wiki/logs/example/error-log
        CustomLog /var/www/example/wiki/logs/example/access-log combined

        RewriteEngine on
        RewriteCond %{SERVER_NAME} =test.example.com
        RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,QSA,R=permanent]
</VirtualHost>

<VirtualHost *:443>
    ServerAdmin example-admin@example.com
    ServerAlias test.example.com

    ErrorLog /var/www/example/wiki/logs/example/ssl-error-log
    CustomLog /var/www/example/wiki/logs/example/ssl-access-log combined

    SSLEngine On
    SSLCertificateFile /etc/letsencrypt/live/test.example.com/fullchain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/test.example.com/privkey.pem
    Include /etc/letsencrypt/options-ssl-apache.conf

    Header set X-Frame-Options "SAMEORIGIN"
    Header set Strict-Transport-Security "max-age=15768000; includeSubDomains"
    Header set X-XSS-Protection "1; mode=block"
    Header set X-Content-Type-Options "nosniff"
    Header set Content-Security-Policy-Report-Only "default-src 'self'; img-src *; style-src 'unsafe-inline'; script-src 'unsafe-inline' 'unsafe-eval'"

    ProxyVia On

    <LocationMatch "^[^/]">
            Deny from all
    </LocationMatch>

    <Proxy balancer://example>
            BalancerMember http://localhost:18080/ route=1
            BalancerMember http://localhost:18081/ route=2
            BalancerMember http://localhost:18082/ route=3
            BalancerMember http://localhost:18083/ route=4
            ProxySet lbmethod=bybusyness timeout=30
    </Proxy>
    <Location /balancer-manager>
            SetHandler balancer-manager
            Order Deny,Allow
            Allow from 127.0.0.1
    </Location>

    Header add Set-Cookie "ROUTEID=.%{BALANCER_WORKER_ROUTE}e; path=/" env=BALANCER_ROUTE_CHANGED

    ProxyPass /balancer-manager !
    ProxyPass / balancer://example/http://localhost/VirtualHostBase/https/test.example.com:443/example/VirtualHostRoot/ stickysession=ROUTEID
    ProxyPassReverse / balancer://example/http://localhost/VirtualHostBase/https/test.example.com:443/example/VirtualHostRoot/ stickysession=ROUTEID

    CacheEnable disk /
    CacheRoot "/var/cache/plone/example"
    CacheLastModifiedFactor 0.1
    CacheIgnoreHeaders Set-Cookie

    Header append X-PloneSite yes
</VirtualHost>

This was put together from the various bits of documentation, so something may have been missed in translation. Is there anything obvious here that could be causing problems?

2

I've had an issue like that when one of my zeoclients wasn't working.

I'd recommend manually checking 18080, 18081, 18082 and 18083 to rule out the possibility that one of them is misbehaving.

3

Unfortunately, it's a non-deterministic failure. When I encounter it, the next load works, and it only seems to happen every few hundred page loads. There aren't any errors in the server, client, or Apache logs.

4

It might be useful to add Sentry to your stack so that you can get real time log events captured. You would be able to know what type of request is causing this.

see:

5

After a lot of chasing my tail, @tkimnguyen made me think to look in dmesg, where it was clear that python was crashing (mostly due to segfaults). Further research led me to find my problems were a symptom of a buggy system-installed version of Python 2.7.11 from Ubuntu 16.04:

I had installed my Plone instance last July and never looked back, dealing with this annoying bug while getting my sites up and running. It's only now that I'm getting close to launch that this became worthy of tracking down.

In hindsight, I remember finding that issue and thinking that my regular upgrading of Ubuntu would eventually fix it. Since the installer copies the system python into the plone tree, the system upgrade... it did nothing. Meanwhile, I forgot about the root cause, leading me back here today. I had the joy of realizing (and not for the first time) that the feeling I'm running around in circles isn't always just a feeling.... :slight_smile:

Thus, my solution was simply to use the installer to create a new installation, copy over my data, and run buildout in the new tree. There were even instructions to help me do that in the Plone documentation: http://docs.plone.org/manage/deploying/copy.html

I hope this helps someone save some time in the future.

2 Likes
6

Ooooh, that seems really useful for alpha/beta testing new code..... Thanks for the suggestion!