[Solved] Haproxy and Plone docker images 5.2.8 and 5.2.9: CPU 100%

Plone Support Deployment & hosting
1

Hi all,
I just faced a strange CPU 100% issue with docker images of plone 5.2.8 and 5.2.9 (5.2.7 is safe).

The problem raises when you start a zeoclient behind an already started haproxy daemon. If you start a zeoclient and then the haproxy, the cpu behaves normally.

The very weird thing is that starting zeoclient after haproxy yields CPU 100% BUT shutting down haproxy has no effect and the cpu still remains stuck on 100% of load (the process that loads the CPU is the /plone/instance/parts/instance/bin/interpreter one).

I have reproduced the problem using official vanilla plone docker images and eeacms/haproxy on two different linux distributions (different docker/docker-compose versions) so I'm pretty sure it's a deterministic issue.

immagine
immagine903×596 30.9 KB

Starting a Plone5.2.9/5.2.8 in a local buildout behind the same eeacms/haproxy docker daemon is free of this issue so far.

alessandro.

2

Reading here

found that setting http-fast-listen = off in the buildout [instance] section, seems to solve the issue.

3

If you read further it didn't solve our problem. Switching to a different WSGI Server was our solution.

4

So also asyncore-use-poll=on was not helpful?

1 Like
5

Thanks @petschki @yurj.

For the record...

Yesterday, we ran into the same issue again but in slightly different conditions:

  • Plone upgraded to 5.2.13
  • no matter if haproxy was up
  • http-fast-listen = off (it's defaulted now in the recipe)
  • zeoclient pushes cpu at 100% after a while (before it was at startup), and keeps the load until restart

The main big problem in this conditions is that the issue is no so deterministic to reproduce.

In a site with 60-70k visit/day the instances starts to load cpu at 100% after about 2-3 hours of normal work.

Fortunately, it seems that applying asyncore-use-poll=on, as @yurj suggested, solved even this scenario.

alessandro.

3 Likes