Advice on caching

If all requests were created equal, or if a ZEO client actually worked on all requests it received (rather than just the number for which it has threads), then a least-request balancing algorithm would work. Since neither is true, I still think the best balancing algorithm is for the load balancer to maintain the queue and to only dispense requests to clients that are handling no more requests than they have threads available.

One exception to this analysis is that a ZEO client creates streaming threads to handle blob returns. Those do not tie up the main threads. If a load balancer could distinguish between when a client was returning blobs and when not, we could increase efficiency. Does anyone know of a load balancer that can do that trick?

we have a site that had been using the hash algorithm without issues until we implemented a subscribers area; now I have a lot of requests on a specific instance and sometimes it gets sick.

I'm willing to give the least connections a try and compare the results as we're using 2 identical servers with nginx, Varnish and 4 Zope instances (1 thread) running ZEO.

@smcmahon I raised this is an potential feature request with the author of haproxy. If enough people asked perhaps he would consider it. The feature was to consider backend instance "free" on first byte received (or end of headers) rather than last byte received.

His answer from 2012 was

That's quite a strange software architecture, isn't it ? Do you know what
makes the request parsing limited to one connection if the machine is
capable of processing the remaining data asynchronously ?

I'm not opposed to finding a solution to make haproxy capable of improving
such a use case, but it would be nice to understand why such bottlenecks
exist in the first place, so that we know whether we should cover other
use cases at the same time.

Regards,
Willy```

You can use GitHub - collective/collective.xsendfile: Deliver blobs direct through your webserver using X-Sendfile/ X-HTTP-ACCEL to free haproxy for the streaming of a file. Apache or Nginx will read the blob directly from the blobstorage folder (a zope streaming thread is not even used).

That is a very interesting idea. I may have a testable case...

Does blobstorage still have a weird permission issue if you use different users for instances and front-end?

Sean

@seanupton, yes its still an issue.

Yes we use c.xsendfile as a workaround for now. Its not ideal with begginers as it complicates your setup with more moving parts. It also does monkey patching and I'm not sure it gets all the cases where blobs are used. We did improve it recently so its better than it used to be. It works with wildcard.media now for example.

Yes, indeed. I use the patch with collective.recipe.patch approach in my project.

as promised:

2 Likes

@hvelarde is your blog not showing up on http://planet.plone.org ?

Would one of you update our github.com/plone/documentation with a digest of the above?

it was before, seems is not now; how can I fix that?

just to be clear, even as I like to discuss a lot, I don't consider myself an expert on this topic (in fact, I don't consider myself an expert on anything but procrastination… but that's another story).

as @smcmahon mentioned in his comment, the site has some specifics and probably should not be used as a guideline; my main idea what to put some thoughts together and make some tests.

feel free to criticize my approach and give more advice.

BTW, @smcmahon @vincentfretin , I was checking the issue on large blob files and I would like to understand what problem are you trying to solve with the usage of collective.xsendfile.

in the Olympics site we served a lot of big ZIP files (around 50-100 MB) containing high resolution photos and we never had any issue on it; as mentioned by Martijn:

serving a blob file is very, very efficient in Zope. The original request thread is closed, and the file streaming is handled entirely by the underlying async polling implementation, Zope / Plone doesn't have to do anything anymore once the blob file has been located.

transferring the file between Zope and the front end web server will be fast (we had that on different machines on the same network, but it was still fast), and then you can cache it so it will never touch the backend on following requests.

at least that's the way it worked for us.

Check how it's listed at GitHub - plone/planet.plone.org: Planet Plone Site Configuration and buildout. Add your feed here.

thanks, @tkimnguyen, I was filtering the feed using good-old Yahoo! Pipes, but the service was shut down in September, 2015… and yes, I haven't posted anything since eons.

I think is fixed now.

1 Like

I'll try to answer the question about why one would want to use collective.xsendfile.

It's about efficiency of load balancing. If the load balancer is configured to send only one request to a backend at a time, then if that backend is serving a blob using a Zope stream iterator, the load balancer is not sending other requests to that backend for processing even though it is available. Or you could decide not to limit the requests per backend for this reason and allow the load balancer to send multiple requests to queue up at the async loop of each backend--but then if some requests take a long time, others will be waiting in that queue for a particular backend even though other backends are available.

So using collective.xsendfile makes it possible to control queueing in the load balancer (by enforcing a per-backend limit) while also making sure that the RAM-hungry Zope instances spend their time actually doing computation rather than just reading files off disk.

(Of course, if you're load balancing based on content or sessions, then you're going to be queueing up requests on particular backends anyway, so in that case there's no reason to limit requests per backend, and therefore not much to be gained by serving blobs outside of Zope.)

1 Like

I think this misses the essence of the tradeoff.

  • If you have plenty of RAM and your app is CPU bound then run one instance per CPU with one thread each.
  • If you have less RAM then you can sacrifice some average request time to gain more total throughput by running more threads per instance.
  • if you have an IO bound site. For example you call an external api, or your traffic patterns results in frequent ZODB loads. In this case, to maximise your throughput for a given set of cpus, use more threads per instance, or better yet, seperate out the IO bound traffic to instances with high thread counts. Why? because the CPU is idle most of the time waiting on IO so it might as well be using it for another request, and it costs less RAM to do this via threads than instances with little downside in this case.

For example I've seen a one of the largest plone sites in the world brought to its knees through lack of understand of this concept. It ran with 4 quad core servers using 16 instances (1 thread per CPU). We proposed switching it to 8 CPU bound instances (1 thread per CPU) and 16 IO bound instances (3 threads per instance or 6 threads per CPU, or 48 threads total) and using rules in the load balancer to seperate the traffic. This greatly increased the capability of the same hardware to handle this particular kind of traffic spikes. Note CDNs and caching don't help with this IO bound issue.

Here is a graph of the a single quad core server tested with replay traffic showing the difference between those configurations.


Note this was testing max response time, not average response time, and the external api could take a long time to respond in this case so this test results also included the response times of that backend api server.

1 Like

Interesting article about using shared memory with multiple processes to reduce RAM usage. https://engineering.instagram.com/dismissing-python-garbage-collection-at-instagram-4dca40b29172#.1qae8ggrv I wonder if this can be used with plone since it has a LOT of code that takes a fair chunk of the RAM when you want to run one process per cpu instead of more threads.

I did a quick test in production and this doesn't work on Plone, or maybe I implemented it wrongly:

I added the following on my instances:

[instance]
...
initialization =
    # disable GC to reduce memory footprint and improve CPU LLC cache hit ratio
    import gc; gc.set_threshold(0)
    import atexit; atexit.register(os._exit, 0)

restarted the instances and wait to see the results.

this is the server with no changes:

this is the server with the hack:

as you can see, the second one is eating the memory faster.

@hvelarde I think you skipped over the important first step. I agree they didn't highlight it much. They run their django using uwsgi is a special mode that forks the processes in such a way is to take advantage of shared memory. Everything they did after that, was to make that work properly. The essential problem they were solving was that shared memory doesn't work for python because both reference counting and GC meta information is stored with the objects and is updated on read. This means that as soon as you GC your shared memory for code becomes non shared.
I don't know how to run plone instances to they fork in that way and if you can do it without wsgi but I'm sure someone else does.

1 Like

@hvelarde also there is a good chance it won't work for zope/plone if the ZODB cache is changing a lot. The trade off they made for turning off GC was possible because their services did not change the RAM much and were quick to restart and restarted often anyway. That might not be the same for a typical plone site.