[SOLVED] Doing port calculations in zc.buildout based on some port base

It is a common requirement that you want to setup a Zope cluster with zeo server, zeo instances and haproxy based on some common port base that you want to define once in your buildout configuration and perform all other ports based on the port base.

For ZEO clients there is the port-base configuration option which works fine.

For ZEO servers you can only hard-code the zeo server's port - no option to calculate it or configure it in your buildout configuration

For HAproxy you need to generate the haproxy.conf file e.g. using collective.recipe.template which also does not seem to provide an option to perform calculations.

What I basically want is to provide a port base for all services and configure all other ports of a cluster as offset once. Any idea?

Unfortunately I don't have an nice solution lying around.

When I did hosting I had a custom configuration management tool that would generate a buildout with the correct services and ports mapping for our whole platform.

You could use buildout to generate the configs needed and then use sed to tweak the configs to your needs.
ie, use 123 to prefix the ports and then replace that with the base port. This won't help you when the ports need to wrap around, ie zeoclients ranging from 999-1010.

An alternative might be to use a container (docker, vps, or vm ), where you can use a known set of ports and map them on the host.

I'm using this in my buildout. Is this what you're after?

[v]
port-base = 82

[hosts]
zeoserver = 127.0.0.1
instance1 = 127.0.0.1
instance2 = 127.0.0.1
instance3 = 127.0.0.1
instance4 = 127.0.0.1
instance5 = 127.0.0.1
instance6 = 127.0.0.1
instance-debug = 127.0.0.1
instance-export = 127.0.0.1
redis = 127.0.0.1
varnish = 127.0.0.1
haproxy = 127.0.0.1
supervisor = 127.0.0.1

[ports]
zeoserver = ${v:port-base}00
instance1 = ${v:port-base}81
instance2 = ${v:port-base}82
instance3 = ${v:port-base}83
instance4 = ${v:port-base}84
instance5 = ${v:port-base}85
instance6 = ${v:port-base}86
instance-debug = ${v:port-base}80
instance-export = ${v:port-base}89
redis = 6379
varnish = ${v:port-base}50
haproxy = ${v:port-base}55
supervisor = ${v:port-base}91

[zeoserver]
recipe = plone.recipe.zeoserver
zeo-address = ${hosts:zeoserver}:${ports:zeoserver}

[instance1]
shared-blob = on
zeo-client = true
zeo-address = ${hosts:zeoserver}:${ports:zeoserver}
effective-user = ${users:zope}
http-address = ${hosts:instance1}:${ports:instance1}

[instance2]
<= instance1
http-address = ${hosts:instance2}:${ports:instance2}

[instance3]
<= instance1
http-address = ${hosts:instance3}:${ports:instance3}

[instance4]
<= instance1
http-address = ${hosts:instance4}:${ports:instance4}

[instance5]
<= instance1
http-address = ${hosts:instance5}:${ports:instance5}

[instance6]
<= instance1
http-address = ${hosts:instance6}:${ports:instance6}

2 Likes

That's cool. I did not know that trick...this will solve problems!

-aj

more or less what I wrote mr.scripty for.

[ports-base]
recipe=mr.scripty
START = 0
init=
  ... for key,value in self.buildout['ports-offset'].items():
  ...   self.options[key] = str(int(value)+int(self.START))
  ... for key,value in self.buildout['ports-fixed'].items():
  ...   if len(value) > 0:
  ...     self.options[key] = str(value)
  ... return ''

[ports-prod]
recipe=mr.scripty
init = ${ports-base:init}
START = 8000

[ports-uat]
recipe=mr.scripty
init = ${ports-base:init}
START = 7000

but now we use docker so don't have to worry about this anymore.

1 Like

Nice too

Some more praise for mr.scripty, we use it to extract the value of a version.txt in our buildout repository (managed by zest.releaser) and have it available as a variable.

recipe = mr.scripty
version =
    import os
    bodir = self.buildout['buildout']['directory']
    versionfile = os.path.join(bodir, 'version.txt')
    if os.path.exists(versionfile):
        with open(versionfile, 'r') as infile:
            version = infile.readline().strip()
    else:
        version = 'unknown'
    return version

Then we inject the version in zeoclients sentry config so each reported stack trace has the buildout release included. Mr.scripty is very useful for simple glueing.

event-log-custom =
    %import raven.contrib.zope
    <sentry>
      dsn ${conf:sentry-dsn}
      level ERROR
      string_max_length 100000
      list_max_length 500
      environment ${conf:release}
      release ${vars:version}
    </sentry>

I tried @vincentfretin 's solution and it works perfectly.