i'd like to use zope's icp-server for health checks on haproxy (to save ram compared to additional zserver threads only accessed by haproxy)
seems that icp-server does not work as expected (see below). would like to start discussion or get some pointers how others to health checks on zope instances in their setups
motivation:
-
zserver-treads = 1
(to save RAM, second thread could load objects in cache (zodb-cache-size)) -
haproxy:
server instance1 127.0.0.1:8080 check 5s fastinter 2s downinter 2s fall 1 rise 2 maxconn 1
if you've got a long running request (longer then checkintervall + downinter * fall) on the zope innstance that occupies the only available thread, haproxy marks it as down
instead of setting zserver-threads = 2
(so the second thread can handle the health checks) i thought using
zope's icp server would be an elegant solution. (as i can be sure not to load a lot of objects into the cache so the second thread uses RAM that is better spent elsewhere)
setup:
plone.recipe.zope2instance allows to activate it via
icp-address = 8090
configure haproxy to use the icp port
server instance1 127.0.0.1:8080 check port 8090 5s fastinter 2s downinter 2s fall 1 rise 2 maxconn 1
however, the zope instance is never recognized as up.
reason: zserver is not listening on the icp port:
on startup zope reports the server as running
2017-11-28T15:27:15 INFO ZServer HTTP server started at Tue Nov 28 15:27:15 2017
Hostname: 0.0.0.0
Port: 8080
------
2017-11-28T15:27:15 INFO ZServer ICP server started
Address: 0.0.0.0
Port: 8090
but telnet can't connect
$ telnet 127.0.0.1 8080
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
$ telnet 127.0.0.1 8090
Trying 127.0.0.1...
telnet: Unable to connect to remote host: Connection refused
if others can confirm this problem i'll write a ticket for zope2
icp doesn't work - what else to do?
i guess as long as i only check http://127.0.0.1:8080 with haproxy (zope root)
the number of objects loaded into the cache will be quite low (compared to check for a path within the plone site)
on the other hand a check on / does not really mean the plone site can be served by this zope server already.
therefore one would need to check /Plone/@@some_view
(option httpchk /Plone/@@some_view)
what do you use for health checks on your setups?
-
icp server (how/why does it work for you?)
-
zope with zserver-threads = haproxy.maxonn + 1
(/Plone, or root / for the check?)or custom view that does not - or does - involve portal_catalog?
-
custom scripts outside of zope - as HAProxy version 1.7.14 - Configuration Manual suggests:
Using the "port" parameter, it becomes possible to use a different port to send health-checks. On some servers, it may be desirable to dedicate a port to a specific component able to perform complex tests which are more suitable to health-checks than the application. It is common to run a simple script in inetd for instance. This parameter is ignored if the "check" parameter is not set. See also the "addr" parameter.