On a brand new server, after a successful run of the ansible playbook, my zeoclient won't launch.
I was able to identify the issue by checking the zeoclient error log.
Here's the tail output of the /var/log/supervisor/zeoserver_zeoclient1-stderr.. file
for ss in strs:
File "/usr/local/plone-5.0/zeoserver/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2576, in _get_metadata
for line in self.get_metadata_lines(name):
File "/usr/local/plone-5.0/zeoserver/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1472, in get_metadata_lines
return yield_lines(self.get_metadata(name))
File "/usr/local/plone-5.0/zeoserver/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1468, in get_metadata
value = self._get(self._fn(self.egg_info, name))
File "/usr/local/plone-5.0/zeoserver/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1577, in _get
with open(path, 'rb') as stream:
IOError: [Errno 13] Permission denied: '/usr/local/plone-5.0/buildout-cache/eggs/plumber-1.3.1-py2.7.egg/EGG-INFO/entry_points.txt'
More info.
I repeated the same thing on a new Ubuntu 16.04 server and the playbook fails when it reaches Everything in buildout cache is group-readable
Below I've included two small parts of a huge error message, I highlight them because the first part is the error itself and the second part talks about operation not permitted chmod. (I have a hunch that it is important)
fatal: [new]: FAILED! => {"failed": true, "msg": "The conditional check 'ran_buildout.changed' failed. The error was: {u'cmd': [u'bin/buildout'],
... snip ... uleObjectMetaclass):\\n ^\\nSyntaxError: invalid syntax\\n\\nwarning: no previously-included files matching \\'*.pyc\\' found under directory \\'src\\'\\nwarning: no previously-included files matching \\'*.pyo\\' found under directory \\'src\\'\\nchmod: changing permissions of \\'/var/local/plone-5.0/zeoserver\\': Operation not permitted\\nchmod: changing permissions of \\'/var/local/plone-5.0/zeoserver/.python-eggs\\': Operation not permitted\\nchmod: cannot read directory \\'/var/local/plone-5.0/zeoserver/.python-eggs\\': Permission denied\\nchmod: changing permissions of \\'/var/local/plone-5.0/zeoserver\\': Operation not permitted\\nchmod: changing permissions of \\'/var/local/plone-5.0/zeoserver/.python-eggs\\': Operation not permitted\\nchmod: cannot read directory \\'/var/local/plone-5.0/zeoserver/.python-eggs\\': Permission denied', u'rc': 0, 'stdout_lines': [u\"Getting distribution for 'zc.buildout==2.4.1'.\", u'Got zc.buildout 2.4.1.', u\"mr.developer: Queued
When I run it after that failure it runs through with no failure but the zeoclients fail to launch due to the permission issue.
It is evident that the task named Everything in buildout cache is group-readable is responsible for setting the group permissions. The fact that it is is failing prevents plone_daemon from doing further stuff.
I ran it on a new playbook with less customization and paid more careful attention.
It seems like the buildout fails
when the buildout fails the next task (which listens for ran_buildout.changed) has an issue.
Running it a second time doesn't seem to help.
Here's the error that comes up on the first run (I've cleaned up the output a bit and removed the middle since it is basically reporting on the buildout.
TASK[plone.plone_server: Everything in buildout cache is group - readable] * * * * *
fatal: [new]: FAILED! => {
"failed": true,
"msg": "The conditional check 'ran_buildout.changed' failed. The error was: {u'cmd': [u'bin/buildout'], u'end': u'2017-01-29 14:56:42.119838', u'stdout': u\"Creating directory '/usr/local/plone-5.0/zeoserver/parts'.
---Creating directory '/usr/local/plone-5.0/zeoserver/develop-eggs'.
---Getting distribution for 'ZODB3==3.10.5'.
---Got ZODB3 3.10.5.---Getting distribution for 'zope.interface==3.6.7'.
---Got zope.interface 3.6.7.-
--Getting distribution for 'zope.security==3.7.4'.
---Go
py\", line 128
--- return results---SyntaxError: 'return' outside function
------ File \"/usr/local/plone-5.0/buildout-cache/eggs/tmpXPz1Dv/Products.CMFPlone-5.0.6-py2.7.egg/Products/CMFPlone/skins/plone_login/login.py\", line 17
--- return context.restrictedTraverse('external_login_return')()
---SyntaxError: 'return' outside function------ File \"/usr/local/plone-5.0/buildout-cache/eggs/tmpXPz1Dv/Products.CMFPlone-5.0.6-py2.7.egg/Products/CMFPlone/skins/plone_login/require_login.py\", line 19
--- return portal.restrictedTraverse(login)()---SyntaxError: 'return' outside function------ File \"/usr/local/plone-5.0/buildout-cache/eggs/tmpXPz1Dv/Products.CMFPlone-5.0.6-py2.7.egg/Products/CMFPlone/skins/plone_form_scripts/plone_change_password.py\", line 16
--- return context.REQUEST.RESPONSE.redirect(---SyntaxError: 'return' outside function
------ File \"/usr/local/plone-5.0/buildout-cache/eggs/tmpXPz1Dv/Products.CMFPlone-5.0.6-py2.7.egg/Products/CMFPlone/skins/plone_prefs/prefs_error_log_update.py\", line 15
--- return request.RESPONSE.redirect(context.absolute_url() + '/prefs_error_log_form')---SyntaxError: 'return' outside function
------ File \"/usr/local/plone-5.0/buildout-cache/eggs/tmpXPz1Dv/Products.CMFPlone-5.0.6-py2.7.egg/Products/CMFPlone/skins/plone_prefs/prefs_error_log_setProperties.py\", line 17
--- return request.RESPONSE.redirect(---SyntaxError: 'return' outside function
------ File \"/usr/local/plone-5.0/buildout-cache/eggs/tmpXPz1Dv/Products.CMFPlone-5.0.6-py2.7.egg/Products/CMFPlone/skins/plone_templates/standard_error_message.py\", line 32
--- return '{{\"error_type\": \"{0:s}\"}}'.format(error_type)
---SyntaxError: 'return' outside function
------chmod: changing permissions of '/var/local/plone-5.0/zeoserver': Operation not permitted---chmod: changing permissions of '/var/local/plone-5.0/zeoserver/.python-eggs': Operation not permitted---chmod: cannot read directory '/var/local/plone-5.0/zeoserver/.python-eggs': Permission denied
---chmod: changing permissions of '/var/local/plone-5.0/zeoserver': Operation not permitted
---chmod: changing permissions of '/var/local/plone-5.0/zeoserver/.python-eggs': Operation not permitted
---chmod: cannot read directory '/var/local/plone-5.0/zeoserver/.python-eggs': Permission denied
------The error appears to have been in '/home/dbain/Downloads/buildouts/ansible_work/flyj-playbook/roles/plone.plone_server/tasks/main.yml': line 236, column 3, but may
---be elsewhere in the file depending on the exact syntax problem.
------The offending line appears to be:
---------- name: Everything in buildout cache is group-readable
--- ^ here---"}
Hmm, IIUC seems it's the ownership of the file, not the permissions on it that affects your ability to do chmod
See: linux - What permissions one needs to run chmod, chown command on a folder/item? - Unix & Linux Stack Exchange
I'm guessing that task needs a become_user: "{{ instance_config.plone_daemon_user }}"
or maybe become_user: root if indeed its not possible to predict the ownership of those directories (file might be created by plone_buildout user)
My goal is to get to a repeatable deploy. I don't want to have to log in and manually fix stuff. I've had success in the past with the playbook but most recently I'm seeing these issues. It isn't valuable if a system admin or integrator needs to do this every time.
@smcmahon might be a good person to give input on this scenario. Maybe I'll figure it out and just send a pull request.
I started wondering if this was due to a low memory system (1024mb ram) so I tried it on a 2048mb system but had the same result. Then I thought it might be due to the newer Ubuntu 16.04, so I used a machine running Ubuntu 14.04 but also had the issues.
@smcmahon might be a good person to give input on this scenario. Maybe I'll figure it out and just send a pull request.
We do test on 16.04. What cloud vendor (or vbox) are you running? Cloud vendors do customize their images and something like default umask changes or r/o partition mounts can have big effects.
Indeed, the goal is repeatability without command line intervention, so we'll keep working on it until we fix it.
Essentially all of the error message is misleading, because what's really going on is that Ansible is choking on the shear volume of output from buildout when it's run the first time.
The various "operation not permitted" messages are from the buildout -- the section that's adjusting permissions after the run -- and are harmless.
It's my observation that you may run the playbook again and get a success because the buildout output is much smaller and doesn't foul Ansible. See if that's true for you.
I was able to get it to build successfully the first time by: using the "shell" module instead of the "command" module, because the "shell" module allowed me to send output to /dev/null (probably a bad idea, I could push the output to a log file or something). Redirecting to /dev/null does not work with commands in the "command module".
I made my changes to the "Run buildout" task under plone.plone_server > tasks > main.yml
- name: Run buildout
when: instance_config.plone_always_run_buildout or
instance_config.plone_autorun_buildout and
(instance_status.changed or extra_dir_copy_result.changed or not buildout_status.stat.exists)
shell: bin/buildout > /dev/null 2>&1
chdir={{ plone_instance_home }}
become_user: "{{ instance_config.plone_buildout_user }}"
register: ran_buildout