ServerException: ('exceptions.IOError', (2, 'No such file or directory'))

Recently moved a server and upgraded from Plone 5.1 to Plone 5.1.1
When I attempt to edit an item I get this traceback

Module ZPublisher.Publish, line 146, in publish
Module Zope2.App.startup, line 303, in commit
Module transaction._manager, line 131, in commit
Module transaction._transaction, line 310, in commit
Module transaction._transaction, line 301, in commit
Module transaction._transaction, line 446, in _commitResources
Module transaction._transaction, line 423, in _commitResources
Module ZODB.Connection, line 693, in tpc_vote
Module ZEO.ClientStorage, line 750, in tpc_vote
Module ZEO.asyncio.client, line 764, in call
Module ZEO.asyncio.client, line 743, in call
Module ZEO.asyncio.client, line 756, in wait_for_result
Module concurrent.futures._base, line 429, in result
Module concurrent.futures._base, line 381, in __get_result
ServerException: ('exceptions.IOError', (2, 'No such file or directory'))

Looking into it now but thought I should note it here.

Update... I restored the old Data.fs and blobs and rebuilt with Plone 5.1.0 and still am getting the same error.

I've been able to validate that everything works as expected on the old server, so clearly I've made some breaking changing between the two machines.

Used this trick to ensure that I had the exact same versions of packages pinned on the new server

Still nothing.
I think it's the version of Python (python 2.7.12)... upgrading to python 2.7.14 now.

Do you have the exact same OS version on both boxes ? (thinking about locking configuration, because of this 'tpc' in the trace) ?

As the root user In installed python 2.7.14 and pip

version=2.7.14
wget https://www.python.org/ftp/python/$version/Python-$version.tgz
tar -xvf Python-$version.tgz
cd Python-$version
./configure
make
sudo checkinstall
# installed pip
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
python get-pip.py 
pip install virtualenv

I was getting errors until I did installed a new version of setuptools

pip install setuptools==38.5.1

I have been using rsync to pull the data from the old server.
This time I stopped the services before rsyncing.
I don't know which one fixed it, the stopping before rsyncing or the installation with Python 2.7.14

I think it was wiser to stop the ZEO though and it works now.

The stopping before rsyncing.

@Rotonen,
After a few more tests, I was able to confirm that it was definitely the stopping before rsyncing.

Further testing and it seems there's a specific content type that is giving trouble on the new server. I only get the error when I attempt to add or edit that particular content type.

Back to troubleshooting :slight_smile:

@gp54321 To answer your question,
Both machines are running: Ubuntu 16.04.4 LTS (GNU/Linux 4.4.0-122-generic x86_64)
The new machine has more ram and more hard drive space. That's about it.

After following a few incorrect leads, here is the final update.

1. The real issue was "masked" because of shared-blobs. This setting:

shared-blob= on

I switched it to off

shared-blob= off

and then restarted the zeo and clients. I also removed the blobstorage/.layout file because it was configured as a zeocache.

2. After disabling shared blobs I was able to get a much more explicit error:

   - __traceback_info__: (<NamedImageWidget 'form.widgets.committee_chair_photo'>, 'thumb_tag', ())
  Module plone.formwidget.namedfile.widget, line 250, in thumb_tag
  Module plone.namedfile.scaling, line 433, in scale
  Module plone.scale.storage, line 210, in scale
  Module plone.namedfile.scaling, line 216, in __call__
  Module plone.namedfile.file, line 331, in open
  Module ZODB.Connection, line 808, in setstate
  Module ZEO.ClientStorage, line 694, in loadBlob
  Module ZEO.asyncio.client, line 764, in call
  Module ZEO.asyncio.client, line 743, in call
  Module ZEO.asyncio.client, line 756, in wait_for_result
  Module concurrent.futures._base, line 429, in result
  Module concurrent.futures._base, line 381, in __get_result
POSKeyError: u'No blob file at /home/plone/site/var/var/blobstorage/0x00/0x00/0x00/0x00/0x00/0x00/0xb5/0xa6/0x03bc4407629fffdd.blob'

The real problem

It turns out that there was variation between the buildout on the target server and the source server. The new buildout was setting the blobstorage location to be var/var/blobstorage. My rsync script was placing blobstorage in var/blobstorage. As a result the system was returning a No blob file.

After fixing the blob location in my buildout and rerunning buildout the issue went away.
:tada: