Plone docker container with volumes has permissions error

We're trying to set up Plone 5.1.5 with Docker and running into some problems. We're following the instructions here to set up Plone with a separate ZEO parent cluster on ECS (Amazon Linux 2017.9). When we run without volumes, it works fine, but if we run it mounted to an EFS volume, we see this:

[root@ip-whatever ~]# docker run -v /mnt/efs:/data plone
. Traceback (most recent call last):
  File "/plone/instance/parts/instance/bin/interpreter", line 285, in <module>
    exec(compile(__file__f.read(), __file__, "exec"))
  File "/plone/buildout-cache/eggs/zdaemon-4.2.0-py2.7.egg/zdaemon/zdrun.py", line 734, in <module>
    main()
  File "/plone/buildout-cache/eggs/zdaemon-4.2.0-py2.7.egg/zdaemon/zdrun.py", line 731, in main
    d.main(args)
  File "/plone/buildout-cache/eggs/zdaemon-4.2.0-py2.7.egg/zdaemon/zdrun.py", line 229, in main
    self.run()
  File "/plone/buildout-cache/eggs/zdaemon-4.2.0-py2.7.egg/zdaemon/zdrun.py", line 233, in run
    self.opensocket()
  File "/plone/buildout-cache/eggs/zdaemon-4.2.0-py2.7.egg/zdaemon/zdrun.py", line 255, in opensocket
    sock.bind(tempname)
  File "/usr/local/lib/python2.7/socket.py", line 228, in meth
    return getattr(self._sock,name)(*args)
socket.error: [Errno 13] Permission denied

This happens even though the EFS mount looks like it has the correct folder structure and permissions:

[root@ip-whatever ~]# ls -R /mnt/efs
/mnt/efs:
blobstorage  filestorage  instance  log  test.txt

/mnt/efs/blobstorage:

/mnt/efs/filestorage:

/mnt/efs/instance:

/mnt/efs/log:
instance.log

Any pointers from those who have done this before? @oshani, I saw you had this topic open.... were you ever successful?

Thanks!

This is a docker problem not Plone. We use plone with docker and EFS fine. As always with docker you have to ensure the numeric id of the users match when sharing volumes not the symbolic names which to me is the likely cause but as I said. This is unrelated to Plone

actually your error is about binding to sockets so it's not related to mounts or EFS.

The socket error I was able to resolve by changing the permissions to 777 on the mounted disk with all of its subdirectories. Unfortunately, the container still doesn't start in ECS, though I can run it manually.

When you were running with EFS, were you running Plone in a container or a whole EC2 instance?

in docker.

Your setup doesn't really make sense. You shouldn't be sharing your filestorage between your plone instances and the zeo server via EFS. Use EFS only for blobs and have the settings explicitly set to shared blobs and never share filestorage.

The intended design was for the zeo containers to share storage via EFS, but the individual plone clients not to mount EFS at all. So should I not be sharing filestorage between zeo containers either, if they aren't acting as plone clients?

ZEO is the database server. You can only have one ZEO instance active and it needs exclusive access to its database files (with the exception of blobs in shared mode). Same as MySQL or Postgres.

In addition every plone client can't share its filestorage files either.

Read https://docs.plone.org/manage/deploying/stack.html

If you are specifically trying to setup a HA master/slave database then use ZRS or Relstorage, We use ZRS.

This is starting to make much more sense now.... the documentation for the docker container uses the example of having one container as the database server and what look like sidecar containers as clients, but it doesn't call out the fact that they aren't sidecar containers, nor that you can only have a single ZEO server running at a time. Given that new information, you're right, what I had in mind makes no sense.

Two more questions... in a docker scenario, do the clients need their own separate filestorage, and do those need to persist after the container has exited (i.e, persistent volumes)? I'm guessing the answer is "yes," but I wanted to be sure.

Second, do you have any ideas on why I'd be having trouble only in cases where the zeo server is using mounted volumes? If I start a separate zeo container, it works fine. If I mount a volume with a plone client container, it works fine. But if I start the separate zeo container with the volume mounted, it won't start.

@djay Is this the architecture I should be aiming for, if I'm using a single database?

plone_arch (1)

(Edit: updated for future reference)

yes

No. But there is a persistent-cache file (.zec) which can be turned on by a setting. That helps speed up startup. Our production rancher system we don't mount filestorage but if we did it shouldn't be shared.

I'm assuming its own non-shared volume, and file owner has the same numeric id in both, in which case I don't know what the problem would be. We generally run it with mounting the host volume though as we've run into problems using data volumes for database files. Are you still getting the same BIND error because that seemed related to file sockets? Looking at teh code the script to start ZEO process is using file based sockets and there is a permission problem with them so seems like permissions for temp files being used. I'd keep the mount very specific to just filestorage, blobs and log dir on ZEO and just blobs and log dir on the instances

Thank you so much for the answers!

I set the permissions on the EFS mount to 777 hoping that would handle it, but it sounds like that won't cut it. With ECS, I'd expect the uid to change every time a new instance is spun up, but perhaps I can count on it being root starting the plone container. I find that with 777, the it does start, though it remains to be seen whether I can do that with just ZEO running.

Is there a reason why you went with EFS instead of EBS, or is hat because you were using ZRS for HA?

The only thing you should have on EFS is shared blobs and only if you set the shared blob mode on in your build out which from the commands you sent it doesnt seem like you have done.

Everything else should be local disks or EBS. Otherwise you are just throwing away money.

Yikes, don't want to do that! Will sharing just the blobs allow the container's data to be persistent, though? I'm booting this from a CloudFormation script, so the reason I went with EFS was that if the container went offline, I didn't want to lose the data. With an EBS volume, I wouldn't be able to re-mount it in ECS; I can save the data, but would manually have to mount the old volume, since ECS creates a new one every time.

You should draw the architecture again, like above, but with filestorage and blobstorage in your drawing.

It'd seem there is a some sort of a trip-up on the name 'file' of filestorage. I guess it is an easy confusion as the files are stored in the blobstorage.

Simplest possible scenario to get the ball rolling:

  • Blobstorage is not configured to be shared and is only mounted on the ZEO.
  • Filestorage cannot be shared and is only mounted on the ZEO.

From 'inside out':

  • One ZEO server
  • Many ZEO clients
  • Loadbalancer
  • Webfront

Does this illustrate the intended architecture better?

plone_arch (1) (1)

BTW, I'm using draw.io to edit this.... so here's a gist if you want to edit it to correct my inevitable mistakes...

draw.io file

Where are you setting in buildout to have blobs shared? Maybe you need to share your buildout.

Forgive me, but I do not have a buildout.... I'm just using the plain old Plone container by itself with a volume mounted to /data.... since I only have one DB server, I don't need blobs shared, correct?

Youe diagram shows you mounting the same blobs between all instances and the DB. You are sharing blobs. Therefore you need to work out how to turn on shared blobs support using docker.

I'm sorry, that was not my intented architecture.... the idea is that the docker container hosting the DB is the only thing that has blobs on it. The plone clients are separate, independent, and share absolutely nothing with the DB; they just have that environment variable that points the client to the DB.

Running under the assumption that the only thing using the blobs is the single database container, is this an architecture that will work?