Multiple Zope instances == slow buildout

Plone Support Administering Plone sites
1

Hello,

I have a web site where I have one machine running 12 Zope instances. In my buildout.cfg.

What is bothering me is that almost every time I run bin/buildout the instance parts are updated, and this process is quite slow. It is is slow for the first instance and takes the same time for each of the remaining instances.

Is there anything I can do to minimize buildout time? Do you suffer of long buildout times as well ?

I'm curious to hear about your strategies.

In my buildout I have something like this:

[instance]
recipe = plone.recipe.zope2instance/
http-address = 8080
(...)

[instance-1]
<= instance
http-address = 8081

# Repeat until [instance-11]

Relevant versions:

  • setuptools= 36.2.0
  • zc.buildout= 2.9.4
  • plone.recipe.zope2instance= 4.2.18
  • Products.CMFPlone= 4.3.7

Thanks!

2

Can you please post more information from your buildout.cfg? And maybe the output from your buildout run?

If nothing changed in your buildout, why should your instances be updated?

3

The instances updates happen when I change something in the buildout. Sorry for not being clear about that.

What bothers me is only the time it takes to update the instances.

Here are the buildout files if you want to take a look: https://github.com/rafaelbco/zope_buildout

Feedback unrelated to the issue I'm having is also highly welcome.

Thanks!

4

what Python version are you running on?

5

buildout will remove and recreate all the parts in your buildout.cfg so it will do so to each instance*

Is there a reason why you need buildout to run more quickly?

If it's because you're running buildout on your production (live) server, you should know that it doesn't usually affect running processes; they'll keep running.

6

A colleague is using a single "buildout" "instance" for multiple Zope instances. It essentially works like this:

  • each of the Zope instances has its own "folder" which is a symbolic link to the common Zope folder. The name of this instance folder contains an instance identifying instance number.

  • A Zope instance is controlled (i.e. start|stop|restarted) by a specialized shell script. This determines the instance number, derives from it instance differencing parameters (such as port and logfile) and then activates the normal instance controlling program (e.g. bin/client1) with these parameters (overriding corresponding definitions in zope.conf).

  • The controlling shell script is assumed to be always started from the instance folder (giving it a means to determine the instance number from the folder is has been started from).

1 Like
7

Upgrade ZConfig to at least 3.1.0
Check the docs for ZConfig, it allows you to use environment variables, for example for the http-address.
Then you can build one instance and set the env variable in supervisor.
We are using this for quite some time already.

4 Likes
8

2.7.12

9

Cool. I guess this approach can be simplified using @do3cc idea in this thread.

10

Life's short! :smile:

I use the buildout in production and I'm aware the instances keep running. What want to speed up is this process:

  1. Pull changes from VCS, containing most of the times updates to versions of add-ons in versions.cfg.
  2. Run bin/buildout. This is the slow part.
  3. Restart the Zope instances.
11

I've seen weird behavior with old Python versions; in one site using Python 2.7.3, for instance, running buildout takes like an hour.

obviously, that's not your case.

12

Anyone here have ever profiled a buildout trying to find a cause for slowness?

13

I"ll try that. The provided workarounds presented here are very clever, but I suspect that once the first instance is built then some things could be cached for the next instances. Perhaps at the level of the zope2instance recipe.

The code is quite scary though.

14

yes. The issue is this

However the advice to use env variables instead is the best solution. For each instance you really are using buildout change the instancename, path, fs location and port. Thats overkill.

2 Likes
15

can you share a full configuration example? I went to the whole ZConfig documentation and was unable to decipher it.

16

@djay is right. I did some quick profiling and most of the time taken to build/update an instance is spent creating the "working set" i.e the set of Python distributions available to the Zope instance.

I think caching the working sets makes sense, regardless of the existence of the environment variables approach.

So I made a PR to zc.buildout and would be very glad if you help to review and/or test it. If you have buildouts with many instances you'll experience an N-fold speedup without changing anything in the config files :wink:. @hvelarde, @idgserpro, @tkimnguyen

Please leave your feedback on GitHub:

2 Likes
17

I saw all the comments and activity in your PR. Thank you for your contribution!

18

It's been fun! Another Brazilian, @leorochael is reviewing the PR. It seems the quality assurance in order to get code into zc.buildout is quite high. Which is reassuring because it's such an important piece of software.

19

After many iterations the performance improvement was finally merged into zc.recipe.egg, which is part of the buildout repo.

Now is your job to poke the guys in order to do a new release :stuck_out_tongue_winking_eye:. @tkimnguyen, @hvelarde, @idgserpro

If you're curious here's the timing comparison: https://github.com/rafaelbco/buildout_improvements/tree/master/timings_ws_caching

4 Likes
20

Nice work!

...but if you could check occasionally if they've made the release that would work better than to pass that on to someone else :slight_smile: