Sites not available without any visible reason

dbitouze · March 10, 2020, 10:26am

Context

Plone 5.2 with Python 3 and Plone 4.3 with Python 2.7
Nginx 1.16.1

Problem

My Plone sites sometimes are not available without any visible reason. One of them (Plone 5.2) is down:

$ ~/site-gte/zinstance//bin/plonectl status
instance: daemon manager not running

but not the other one:

$ ~/stage-latex/zinstance/bin/plonectl status
instance: program running; pid=2116

It happened this morning whereas these sites were working nicely yesterday, hence a possible reboot of the server isn't involved:

$ uptime 
 11:01:02 up 11 days, 18:16,  1 user,  load average: 0,01, 0,00, 0,00

(and I added a script at boot to start these sites).

Breaking news: what is very strange is that it was enough to restart nginx to make these Plone sites available and working again, even the one with "daemon manager not running" status!

And ps aux | grep lone returns only something about the latter:

$ ps aux | grep lone
root      1906  0.0  0.4 109216 18252 ?        Ssl  févr.27   0:18 /home/gest/site-gte/zinstance/bin/python3.7 /home/gest/site-gte/zinstance/parts/instance/bin/interpreter /home/gest/site-gte/buildout-cache/eggs/zdaemon-4.3-py3.7.egg/zdaemon/zdrun.py -S /home/gest/site-gte/buildout-cache/eggs/plone.recipe.zope2instance-6.3.0-py3.7.egg/plone/recipe/zope2instance/wsgischema.xml -b 10 -d -s /home/gest/site-gte/zinstance/var/instance/zopectlsock -m 0o22 -x 0,2 -z /home/gest/site-gte/zinstance/parts/instance /home/gest/site-gte/zinstance/bin/python3.7 /home/gest/site-gte/zinstance/parts/instance/bin/interpreter /home/gest/site-gte/buildout-cache/eggs/Zope-4.1.1-py3.7.egg/Zope2/Startup/serve.py /home/gest/site-gte/zinstance/parts/instance/etc/wsgi.ini
gest     12566  0.0  0.0  13284   752 pts/0    S+   11:22   0:00 grep --color lone

Well, I'm completely lost... Any help very appreciated!

zopyx · March 10, 2020, 10:39am

Check your Plone logfile first.

dbitouze · March 10, 2020, 10:49am

Check your Plone logfile first.

Before I restarted nginx, for the Plone 5.2 site, the last 10 lines of instance.log were:

$ tail -n10 var/log/instance.log 
2020-03-02 21:20:47,100 INFO    [plone.protect:38][waitress] auto rotating keyring _forms
2020-03-02 21:20:47,101 INFO    [plone.protect:38][waitress] auto rotating keyring _anon
2020-03-02 21:20:47,101 INFO    [plone.protect:38][waitress] auto rotating keyring _anon
2020-03-02 22:33:14,138 INFO    [waitress:359][waitress] Client disconnected while serving /VirtualHostBase/https/gte.univ-littoral.fr:443/iutgte-dk/VirtualHostRoot/Members/denis-bitouze/pub/latex/formations/installation-latex.pdf/@@download/file/installation-latex.pdf
2020-03-02 22:33:14,138 INFO    [waitress:359][waitress] Client disconnected while serving /VirtualHostBase/https/gte.univ-littoral.fr:443/iutgte-dk/VirtualHostRoot/Members/denis-bitouze/pub/latex/formations/installation-latex.pdf/@@download/file/installation-latex.pdf
2020-03-05 16:05:38,162 WARNING [waitress.queue:122][MainThread] Task queue depth is 1
2020-03-06 17:07:06,613 WARNING [waitress.queue:122][MainThread] Task queue depth is 1
2020-03-06 19:48:27,307 WARNING [waitress.queue:122][MainThread] Task queue depth is 1
2020-03-07 08:04:06,791 WARNING [waitress.queue:122][MainThread] Task queue depth is 1
2020-03-09 11:16:38,511 WARNING [waitress.queue:122][MainThread] Task queue depth is 1

and the last 10 lines of instance-access.log were:

$ tail -n10 var/log/instance-access.log 
127.0.0.1 - - [10/mars/2020:01:25:16 +0200] "GET /VirtualHostBase/https/gte.univ-littoral.fr%3A443/iutgte-dk/VirtualHostRoot/%2B%2Bplone%2B%2Bproduction/%2B%2Bunique%2B%2B2019-09-18T21%3A49%3A19.093006/default.js HTTP/1.0" 200 528387 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
127.0.0.1 - - [10/mars/2020:01:43:07 +0200] "GET /VirtualHostBase/https/gte.univ-littoral.fr%3A443/iutgte-dk/VirtualHostRoot/corrige-du-controle-de-mathematiques-du-18-01-2019-en-ligne HTTP/1.0" 200 20318 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
127.0.0.1 - - [10/mars/2020:02:26:24 +0200] "GET /VirtualHostBase/https/gte.univ-littoral.fr%3A443/iutgte-dk/VirtualHostRoot/plonejsi18n?domain=widgets&language=fr HTTP/1.0" 200 15487 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b"
127.0.0.1 - - [10/mars/2020:02:42:37 +0200] "GET /VirtualHostBase/https/gte.univ-littoral.fr%3A443/iutgte-dk/VirtualHostRoot/ HTTP/1.0" 200 21859 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36"
127.0.0.1 - - [10/mars/2020:02:55:07 +0200] "GET /VirtualHostBase/https/gte.univ-littoral.fr%3A443/iutgte-dk/VirtualHostRoot/futurs-etudiants/licence-professionnelle HTTP/1.0" 200 21893 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
127.0.0.1 - - [10/mars/2020:03:03:50 +0200] "GET /VirtualHostBase/https/gte.univ-littoral.fr%3A443/iutgte-dk/VirtualHostRoot/robots.txt HTTP/1.0" 200 884 "-" "Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; AspiegelBot)"
127.0.0.1 - - [10/mars/2020:03:06:09 +0200] "GET /VirtualHostBase/https/gte.univ-littoral.fr%3A443/iutgte-dk/VirtualHostRoot/Members/denis-bitouze/pub/latex/divers/traitements-de-texte-stupides-et-inefficaces HTTP/1.0" 200 25128 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.92 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
127.0.0.1 - - [10/mars/2020:03:08:09 +0200] "POST /VirtualHostBase/https/gte.univ-littoral.fr%3A443/iutgte-dk/VirtualHostRoot/vendor/phpunit/phpunit/src/Util/PHP/eval-stdin.php HTTP/1.0" 404 26 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36"
127.0.0.1 - - [10/mars/2020:03:28:09 +0200] "GET /VirtualHostBase/https/gte.univ-littoral.fr%3A443/iutgte-dk/VirtualHostRoot/ HTTP/1.0" 200 21859 "-" "Mozilla/5.0 (compatible; Nimbostratus-Bot/v1.3.2; http://cloudsystemnetworks.com)"
127.0.0.1 - - [10/mars/2020:03:43:08 +0200] "GET /VirtualHostBase/https/gte.univ-littoral.fr%3A443/iutgte-dk/VirtualHostRoot/Members/denis-bitouze/pub/latex/formations/college-doctoral-lille-nord-de-france/cours HTTP/1.0" 200 24381 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.92 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

Nothing suspect, IMHO.

espenmn · March 10, 2020, 11:51am

A guess. Kill the processes and restart it again (maybe in debug mode?).

dbitouze · March 10, 2020, 1:14pm

I already stopped and restarted the sites:

/bin/plonectl stop && /bin/plonectl start

but the problem reappeared a few days later.

Do you mean with:

/bin/plonectl fg

I could but the problem is the sites usually works normally for a few days and I read somewhere it is not recommended to run this for sites in production.

espenmn · March 10, 2020, 4:07pm

Yes.

But I meant that you should do this only to check if it started. If it did, then stop it and start it the normal way.

( Just to check that something has not 'frozen')

dbitouze · April 15, 2020, 12:09pm

Sorry for the delay: I was awaiting the next unavailability of my sites and here it is

The problem is, the unavailability generally happens several days after I restarted the sites. Nevertheless, I will this time restart them in foreground mode in hopes of tracking the trouble.

Here are the few last lines of instance.log:

2020-04-14 19:28:39,389 WARNING [waitress.queue:122][MainThread] Task queue depth is 1
2020-04-14 19:28:39,524 WARNING [waitress.queue:122][MainThread] Task queue depth is 2
2020-04-14 19:28:39,800 WARNING [waitress.queue:122][MainThread] Task queue depth is 2
2020-04-14 19:28:39,933 WARNING [waitress.queue:122][MainThread] Task queue depth is 3
2020-04-14 19:28:40,717 WARNING [waitress.queue:122][MainThread] Task queue depth is 3
2020-04-14 19:28:40,733 WARNING [waitress.queue:122][MainThread] Task queue depth is 4
2020-04-14 19:28:40,814 WARNING [waitress.queue:122][MainThread] Task queue depth is 5
2020-04-14 19:28:41,228 WARNING [waitress.queue:122][MainThread] Task queue depth is 6
2020-04-14 19:28:41,919 WARNING [waitress.queue:122][MainThread] Task queue depth is 5
2020-04-14 19:28:41,950 WARNING [waitress.queue:122][MainThread] Task queue depth is 6
2020-04-14 19:28:42,041 WARNING [waitress.queue:122][MainThread] Task queue depth is 7
2020-04-14 19:28:42,222 WARNING [waitress.queue:122][MainThread] Task queue depth is 8
2020-04-14 19:28:42,262 WARNING [waitress.queue:122][MainThread] Task queue depth is 9
2020-04-14 19:28:42,263 WARNING [waitress.queue:122][MainThread] Task queue depth is 10
2020-04-14 19:28:42,428 WARNING [waitress.queue:122][MainThread] Task queue depth is 11
2020-04-14 19:28:42,566 WARNING [waitress.queue:122][MainThread] Task queue depth is 11
2020-04-14 19:28:42,810 WARNING [waitress.queue:122][MainThread] Task queue depth is 12
2020-04-14 19:28:43,077 WARNING [waitress.queue:122][MainThread] Task queue depth is 12
2020-04-14 19:28:43,246 WARNING [waitress.queue:122][MainThread] Task queue depth is 13
2020-04-14 19:28:43,608 WARNING [waitress.queue:122][MainThread] Task queue depth is 14
2020-04-14 19:28:43,948 WARNING [waitress.queue:122][MainThread] Task queue depth is 15
2020-04-14 19:28:44,038 WARNING [waitress.queue:122][MainThread] Task queue depth is 16
2020-04-14 19:28:44,043 WARNING [waitress.queue:122][MainThread] Task queue depth is 17
2020-04-14 19:28:44,102 WARNING [waitress.queue:122][MainThread] Task queue depth is 18
2020-04-14 19:28:45,196 WARNING [waitress.queue:122][MainThread] Task queue depth is 16
2020-04-14 19:28:45,879 WARNING [waitress.queue:122][MainThread] Task queue depth is 16
2020-04-14 19:28:46,011 WARNING [waitress.queue:122][MainThread] Task queue depth is 17
2020-04-14 19:28:48,457 WARNING [waitress.queue:122][MainThread] Task queue depth is 13
2020-04-14 19:28:48,654 WARNING [waitress.queue:122][MainThread] Task queue depth is 14
2020-04-14 19:28:49,372 WARNING [waitress.queue:122][MainThread] Task queue depth is 12
2020-04-14 19:28:49,683 WARNING [waitress.queue:122][MainThread] Task queue depth is 13
2020-04-14 19:28:49,798 WARNING [waitress.queue:122][MainThread] Task queue depth is 14
2020-04-14 19:28:50,272 WARNING [waitress.queue:122][MainThread] Task queue depth is 14
2020-04-14 19:28:50,407 WARNING [waitress.queue:122][MainThread] Task queue depth is 15
2020-04-14 19:28:50,910 WARNING [waitress.queue:122][MainThread] Task queue depth is 15
2020-04-14 19:28:51,171 WARNING [waitress.queue:122][MainThread] Task queue depth is 16
2020-04-14 19:28:51,214 WARNING [waitress.queue:122][MainThread] Task queue depth is 16
2020-04-14 19:28:51,226 WARNING [waitress.queue:122][MainThread] Task queue depth is 17
2020-04-14 19:28:52,838 WARNING [waitress.queue:122][MainThread] Task queue depth is 15
2020-04-14 19:28:53,023 WARNING [waitress.queue:122][MainThread] Task queue depth is 16
2020-04-14 19:28:53,690 WARNING [waitress.queue:122][MainThread] Task queue depth is 17
2020-04-14 19:28:53,701 WARNING [waitress.queue:122][MainThread] Task queue depth is 18

and of instance-access.log:

127.0.0.1 - - [14/avril/2020:22:46:16 +0200] "GET /VirtualHostBase/https/gte.univ-littoral.fr%3A443/iutgte-dk/VirtualHostRoot/Members/olivier--perrot/pub/ray-iut-poly.pdf HTTP/1.0" 200 512396 "https://www.google.com/" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36 CCleaner/80.0.3765.152"
127.0.0.1 - - [14/avril/2020:22:46:18 +0200] "GET /VirtualHostBase/https/gte.univ-littoral.fr%3A443/iutgte-dk/VirtualHostRoot/favicon.ico HTTP/1.0" 200 5430 "https://gte.univ-littoral.fr/Members/olivier--perrot/pub/ray-iut-poly.pdf" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36 CCleaner/80.0.3765.152"
127.0.0.1 - - [14/avril/2020:22:50:08 +0200] "GET /VirtualHostBase/https/gte.univ-littoral.fr%3A443/iutgte-dk/VirtualHostRoot/espace-de-travail/documents-pedagogiques/mathematiques/corrige-1.pdf HTTP/1.0" 200 93999 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
127.0.0.1 - - [14/avril/2020:22:57:00 +0200] "GET /VirtualHostBase/https/gte.univ-littoral.fr%3A443/iutgte-dk/VirtualHostRoot/corrige-du-controle-de-mathematiques-en-2e-annee-pel-en-date-du-19-02-2020-en-ligne HTTP/1.0" 200 20462 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
127.0.0.1 - - [14/avril/2020:23:14:34 +0200] "GET /VirtualHostBase/https/gte.univ-littoral.fr%3A443/iutgte-dk/VirtualHostRoot/sitemap.xml.gz HTTP/1.0" 200 2041 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
127.0.0.1 - - [14/avril/2020:23:20:15 +0200] "GET /VirtualHostBase/https/gte.univ-littoral.fr%3A443/iutgte-dk/VirtualHostRoot/Members/olivier--perrot/pub/thermo_1sem_poly.pdf HTTP/1.0" 200 794621 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
127.0.0.1 - - [14/avril/2020:23:33:56 +0200] "GET /VirtualHostBase/https/gte.univ-littoral.fr%3A443/iutgte-dk/VirtualHostRoot/ HTTP/1.0" 200 21859 "-" "Mozilla/5.0 zgrab/0.x"
127.0.0.1 - - [14/avril/2020:23:34:27 +0200] "GET /VirtualHostBase/https/gte.univ-littoral.fr%3A443/iutgte-dk/VirtualHostRoot/ads.txt HTTP/1.0" 404 15601 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
127.0.0.1 - - [14/avril/2020:23:34:35 +0200] "GET /VirtualHostBase/https/gte.univ-littoral.fr%3A443/iutgte-dk/VirtualHostRoot/rss.xml HTTP/1.0" 200 5084 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
127.0.0.1 - - [14/avril/2020:23:57:53 +0200] "GET /VirtualHostBase/https/gte.univ-littoral.fr%3A443/iutgte-dk/VirtualHostRoot/Members/denis-bitouze/pub/latex/formations/installation-de-latex HTTP/1.0" 302 86 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
127.0.0.1 - - [14/avril/2020:23:57:54 +0200] "GET /VirtualHostBase/https/gte.univ-littoral.fr%3A443/iutgte-dk/VirtualHostRoot/iutgte-dk/resolveuid/d385600f932a448ca495222f08c62144 HTTP/1.0" 301 0 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
127.0.0.1 - - [14/avril/2020:23:57:54 +0200] "GET /VirtualHostBase/https/gte.univ-littoral.fr%3A443/iutgte-dk/VirtualHostRoot/Members/denis-bitouze/pub/latex/diapositives-cours-d/installation-latex.pdf HTTP/1.0" 200 556047 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
127.0.0.1 - - [15/avril/2020:00:01:16 +0200] "GET /VirtualHostBase/https/gte.univ-littoral.fr%3A443/iutgte-dk/VirtualHostRoot/Members/denis-bitouze/pub/latex/formations/installation-de-latex HTTP/1.0" 302 86 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
127.0.0.1 - - [15/avril/2020:00:01:17 +0200] "GET /VirtualHostBase/https/gte.univ-littoral.fr%3A443/iutgte-dk/VirtualHostRoot/iutgte-dk/resolveuid/d385600f932a448ca495222f08c62144 HTTP/1.0" 301 0 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
127.0.0.1 - - [15/avril/2020:00:01:17 +0200] "GET /VirtualHostBase/https/gte.univ-littoral.fr%3A443/iutgte-dk/VirtualHostRoot/Members/denis-bitouze/pub/latex/diapositives-cours-d/installation-latex.pdf HTTP/1.0" 200 556047 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
127.0.0.1 - - [15/avril/2020:00:12:36 +0200] "GET /VirtualHostBase/https/gte.univ-littoral.fr%3A443/iutgte-dk/VirtualHostRoot/vendor/phpunit/phpunit/src/Util/PHP/eval-stdin.php HTTP/1.0" 404 26 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36"
127.0.0.1 - - [15/avril/2020:00:12:36 +0200] "POST /VirtualHostBase/https/gte.univ-littoral.fr%3A443/iutgte-dk/VirtualHostRoot/vendor/phpunit/phpunit/src/Util/PHP/eval-stdin.php HTTP/1.0" 404 26 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36"
127.0.0.1 - - [15/avril/2020:01:14:52 +0200] "GET /VirtualHostBase/https/gte.univ-littoral.fr%3A443/iutgte-dk/VirtualHostRoot/ HTTP/1.0" 200 21859 "-" "Go-http-client/1.1"
127.0.0.1 - - [15/avril/2020:01:48:35 +0200] "GET /VirtualHostBase/https/gte.univ-littoral.fr%3A443/iutgte-dk/VirtualHostRoot/ HTTP/1.0" 200 21859 "-" "-"
127.0.0.1 - - [15/avril/2020:02:06:14 +0200] "GET /VirtualHostBase/https/gte.univ-littoral.fr%3A443/iutgte-dk/VirtualHostRoot/Members/denis-bitouze/pub/maths-pel/cours.pdf HTTP/1.0" 200 555822 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
127.0.0.1 - - [15/avril/2020:02:23:57 +0200] "GET /VirtualHostBase/https/gte.univ-littoral.fr%3A443/iutgte-dk/VirtualHostRoot/ HTTP/1.0" 200 21859 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0"
127.0.0.1 - - [15/avril/2020:02:23:59 +0200] "GET /VirtualHostBase/https/gte.univ-littoral.fr%3A443/iutgte-dk/VirtualHostRoot/asdasdasd HTTP/1.0" 404 26 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0"
127.0.0.1 - - [15/avril/2020:02:27:44 +0200] "GET /VirtualHostBase/https/gte.univ-littoral.fr%3A443/iutgte-dk/VirtualHostRoot/Members/d-bitouze HTTP/1.0" 404 6842 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
127.0.0.1 - - [15/avril/2020:02:27:57 +0200] "GET /VirtualHostBase/https/gte.univ-littoral.fr%3A443/iutgte-dk/VirtualHostRoot/Members/d-bitouze/pub/latex/formations HTTP/1.0" 404 8609 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
127.0.0.1 - - [15/avril/2020:02:28:18 +0200] "GET /VirtualHostBase/https/gte.univ-littoral.fr%3A443/iutgte-dk/VirtualHostRoot/Members/d-bitouze/pub HTTP/1.0" 404 10283 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
127.0.0.1 - - [15/avril/2020:02:28:37 +0200] "GET /VirtualHostBase/https/gte.univ-littoral.fr%3A443/iutgte-dk/VirtualHostRoot/Members/d-bitouze/pub/latex HTTP/1.0" 404 10467 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
127.0.0.1 - - [15/avril/2020:02:28:58 +0200] "GET /VirtualHostBase/https/gte.univ-littoral.fr%3A443/iutgte-dk/VirtualHostRoot/Members/d-bitouze/pub/latex/exercices HTTP/1.0" 404 10067 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
127.0.0.1 - - [15/avril/2020:02:39:16 +0200] "GET /VirtualHostBase/https/gte.univ-littoral.fr%3A443/iutgte-dk/VirtualHostRoot/futurs-etudiants/stages HTTP/1.0" 200 25782 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
127.0.0.1 - - [15/avril/2020:02:48:35 +0200] "GET /VirtualHostBase/https/gte.univ-littoral.fr%3A443/iutgte-dk/VirtualHostRoot/ HTTP/1.0" 200 21859 "-" "-"
127.0.0.1 - - [15/avril/2020:02:59:04 +0200] "GET /VirtualHostBase/https/gte.univ-littoral.fr%3A443/iutgte-dk/VirtualHostRoot/futurs-etudiants/qui-sommes-nous HTTP/1.0" 200 24860 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"

nielssteenkrogh · April 17, 2020, 12:50pm

Hi, Just some "wild" guessing.
If you have print statements in your python code this type of symptom could happen (this should normally also give a clear error-message, but maybe in your setup does not). The symptom you describe with the "2 days factor" of some kind could be symptom of some type of "buffer" in your OS being filled to the top.

dbitouze · April 17, 2020, 2:01pm

If you have print statements in your python code this type of symptom could happen (this should normally also give a clear error-message, but maybe in your setup does not).

I'm not sure to understand what you mean by "print statements in your python code". AFAICT, no such statements in my sites.

The symptom you describe with the "2 days factor" of some kind could be symptom of some type of "buffer" in your OS being filled to the top.

Currently, while my sites are up, that's not the case:

df
Sys. de fichiers Taille Utilisé Dispo Uti% Monté sur
devtmpfs           2,0G       0  2,0G   0% /dev
tmpfs              2,0G       0  2,0G   0% /dev/shm
tmpfs              2,0G    680K  2,0G   1% /run
/dev/sda1          7,7G    258M  7,0G   4% /
/dev/sda6           16G    2,7G   13G  19% /usr
tmpfs              2,0G       0  2,0G   0% /sys/fs/cgroup
/dev/sda7           29G    3,1G   26G  11% /home
/dev/sda9          3,8G     16M  3,6G   1% /tmp
/dev/sda8           63G    4,4G   55G   8% /var
tmpfs              394M       0  394M   0% /run/user/1000

But I'll check next time they will be down: thanks for the hint!

zopyx · April 17, 2020, 3:48pm

This indicates that your incoming requests are stacking up because the Plone backend is unable to answer the requests quickly enough. No idea about the nature and setup of your site but usually you would add more ZEO clients and balance requests been ZEO clients.

dbitouze · April 18, 2020, 12:55pm

@zopyx Good to know: thanks! Unfortunately, I didn't take the ZEO route. Though the Plone backend is unable to answer the requests quickly enough, I guess it answers them later: do you think the delay is a problem for the site visitors?

tkimnguyen · April 18, 2020, 12:57pm

It's not a good sign that there is a delay. You should set up some kind of monitoring to track the behaviour of the site (the entire stack from nginx down to Plone) to be able to observe what is normal and what is not for your particular setup and traffic patterns.

dbitouze · April 18, 2020, 1:08pm

@tkimnguyen What kind of monitoring do you advise?

tkimnguyen · April 18, 2020, 1:13pm

It depends on how you've deployed the site(s). If, say, on Amazon AWS there are tools there that help you monitor sites. Prometheus, Grafana, zabbix, xymon, but even just basic "is my site up" like uptime robot.