TypeError/ReadError with ZEO.asyncio.client in verify

Apologize for the lengthy post.

I am running the following Plone stack on Ubuntu 18 LTS:

Plone 5.1.5 (5115)
CMF 2.2.12
Zope 2.13.27
Python 2.7.15rc1 (default, Nov 12 2018, 14:31:15) [GCC 7.3.0]
PIL 5.4.1 (Pillow)

I have a custom DX content-type I created that is used to enroll hs-students into College Courses in a special program that our Uni-provides free of charge.

  • One Zeoserver on 8100
  • Two Zope Clients 8080 & 8081
  • NGINX using ip_hash to connect requests to backend
  • Impelmented cipher.encryptingstorage for secure DataFS on an alternate mount point (/enc/plone) --my client requires an encrypted data store.

I should mention that adding cipher.encryptingstorage to my 5.1.5 installer derived buildout requires the following packages to be pinned to the following versions:

  • PasteDeploy = 2.0
  • zope.configuration = 3.8.0
  • webob = 1.8.3

Must give a shout out to Kim et al who recommended Sentry for capturing errors... For a free service, it's provided me meaningful ways to make some sense of the error messages (let alone capture them!).

===

I have 4.4K events of the following:

AFAICT, invalidation failures + read/conflict errors leave my Zopes unable to connect with their DB. Only a restart of Zope solves the stalled client/site.

EXCEPTION(most recent call last)
TypeError: invalidate() takes exactly 3 arguments (4 given)
  File "cipher/encryptingstorage/__init__.py", line 153, in invalidate
    return self.db.invalidate(transaction_id, oids, version)
  File "ZEO/ClientStorage.py", line 979, in invalidateTransaction
    self._db.invalidate(tid, oids)
  File "ZEO/asyncio/client.py", line 503, in verify
    self.client.invalidateTransaction(server_tid, oids)

Preformatted textcipher/encryptingstorage/__init__.py in invalidate at line 153
        return self.db.invalidateCache()
    def invalidate(self, transaction_id, oids, version=''):
        """ For IStorageWrapper
        """
        return self.db.invalidate(transaction_id, oids, version)
    def references(self, record, oids=None):
        """ For IStorageWrapper
        """
        return self.db.references(self._untransform(record), oids)
oids	
[
'\x00\x00\x00\x00\x00\x00\x00\xd9', 
'\x00\x00\x00\x00\x00\x02c\xfd', 
'a'
]
self	
<cipher.encryptingstorage.EncryptingStorage object at 0x7f831db25550>
transaction_id	
'\x03\xd0\x05\xdbG`\xb3\x99'
version	
''
ZEO/ClientStorage.py in invalidateTransaction at line 979
            self._db.invalidateCache()
    def invalidateTransaction(self, tid, oids):
        """Server callback: Invalidate objects modified by tid."""
        if self._db is not None:
            self._db.invalidate(tid, oids)
    # IStorageIteration
    def iterator(self, start=None, stop=None):
        """Return an IStorageTransactionInformation iterator."""
oids	
[
'\x00\x00\x00\x00\x00\x00\x00\xd9', 
'\x00\x00\x00\x00\x00\x02c\xfd', 
'a'
]
self	
<ZEO.ClientStorage.ClientStorage object at 0x7f830b7b7250>
tid	
'\x03\xd0\x05\xdbG`\xb3\x99'
ZEO/asyncio/client.py in verify at line 503
                    if vdata:
                        self.verify_result = "quick verification"
                        server_tid, oids = vdata
                        for oid in oids:
                            cache.invalidate(oid, None)
                        self.client.invalidateTransaction(server_tid, oids)
                    else:
                        # cache is too old
                        self.verify_result = "cache too old, clearing"
                        try:
                            ZODB.event.notify(
cache	
<ZEO.cache.ClientCache object at 0x7f830b7b7210>
cache_tid	
"\x03\xd0\x048\x0f\x93'\xdd"
exc	
TypeError('invalidate() takes exactly 3 arguments (4 given)',)
oid	
'a'
oids	
[
'\x00\x00\x00\x00\x00\x00\x00\xd9', 
'\x00\x00\x00\x00\x00\x02c\xfd', 
'a'
]
protocol	
Protocol(('127.0.0.1', 8100), 'enc', False)
self	
<ZEO.asyncio.client.Client object at 0x7f830af95250>
server_tid	
'\x03\xd0\x05\xdbG`\xb3\x99'
vdata	
[
'\x03\xd0\x05\xdbG`\xb3\x99', 
[
'\x00\x00\x00\x00\x00\x00\x00\xd9', 
'\x00\x00\x00\x00\x00\x02c\xfd', 
'a'
]
]

Things I've done to keep my application running are:

I was using Supervisor to run my application but Zeo would "flap". Essentially it was connecting and disconnecting and not providing any uptime of more than a few seconds.

Running Zeo in foreground mode (which was ironic considering that Supervisord runs it too via console fg mode) improved uptime so I just decided to start Plone using plonectl.

I tried to research the invalidation-queue-size in regards to Zeo but I did not feel comfortable toying with it. invalidateTransaction is being called but the argument requirement is not being satisfied (I guess due to the latency?).

I am thinking of going to a single instance because I feel powerless (given my lack of knowledge) looking for possible reasons as to why I am losing DB connection and ZEO.asyncio.client issues. I don't like forming surgery while the patient is currently running a marathon etc.

Plone Foundation Code of Conduct