Restore ZEO Data.fs / blobstorage from backups: Newer transactions than server!

Plone Support Deployment & hosting
, , ,
1

I let this here in order for YOU not waste time on this issue.

I had to restore Data.fs and blobstorage from backups on a production setup (ZEO + 2 ZEO-Clients)

Starting ZEO went well, but when I started the first zeo-client the nightmare!!!

instance.log

2021-03-17T13:43:59 CRITICAL ZEO.ClientStorage zeostorage Client has seen newer transactions than server!
------
2021-03-17T13:43:59 ERROR ZEO.zrpc (9711) CW: error in notifyConnected
Traceback (most recent call last):
  File "/eggs/ZODB3-3.10.7-py2.7-linux-x86_64.egg/ZEO/zrpc/client.py", line 619, in notify_client
    self.client.notifyConnected(self.conn)
  File "/eggs/ZODB3-3.10.7-py2.7-linux-x86_64.egg/ZEO/ClientStorage.py", line 641, in notifyConnected
    self.verify_cache(stub)
  File "/eggs/ZODB3-3.10.7-py2.7-linux-x86_64.egg/ZEO/ClientStorage.py", line 1364, in verify_cache
    raise ClientStorageError(message)
ClientStorageError: zeostorage Client has seen newer transactions than server!

zeo.log

2021-03-17T13:05:40 ERROR ZODB.ConflictResolution Unexpected error
Traceback (most recent call last):
  File "/eggs/ZODB3-3.10.7-py2.7-linux-x86_64.egg/ZODB/ConflictResolution.py", line 243, in tryToResolveConflict
    oldData = self.loadSerial(oid, oldSerial)
  File "/eggs/ZODB3-3.10.7-py2.7-linux-x86_64.egg/ZODB/FileStorage/FileStorage.py", line 434, in loadSerial
    raise POSKeyError(oid)
POSKeyError: 0x7...

I've cleanup blobcache several times, removed uuid.txt and async.txt without any luck!

The website was up-and-running, but when trying to add/edit object it would fail in mysterious ways.

Solution

Inspecting zeo-clients folders I found within parts/instance/var 2 files called:

  • instance.zec
  • instance.zec.lock

Removing these files on all zeo-clients fixed the issue.

See more topics on this issue:

4 Likes
2

My PR to improve the docs concerning this issue was closed.

I then created an issue to improve the log message, see Improve log and exception message · Issue #142 · zopefoundation/ZEO · GitHub

Jason already created a very good wording ( Add paragraph about how to fix ClientStorageError. by jugmac00 · Pull Request #139 · zopefoundation/ZEO · GitHub ) but at least back then the ZEO test suite was super flaky, so I did not create a pull request.

Also, currently the test suite even does not pass on Python 3.8 ( Support for Python 3.8 (broken?) · Issue #165 · zopefoundation/ZEO · GitHub ).

1 Like