Apologize for the lengthy post.
I am running the following Plone stack on Ubuntu 18 LTS:
Plone 5.1.5 (5115)
CMF 2.2.12
Zope 2.13.27
Python 2.7.15rc1 (default, Nov 12 2018, 14:31:15) [GCC 7.3.0]
PIL 5.4.1 (Pillow)
I have a custom DX content-type I created that is used to enroll hs-students into College Courses in a special program that our Uni-provides free of charge.
- One Zeoserver on 8100
- Two Zope Clients 8080 & 8081
- NGINX using ip_hash to connect requests to backend
- Impelmented cipher.encryptingstorage for secure DataFS on an alternate mount point (/enc/plone) --my client requires an encrypted data store.
I should mention that adding cipher.encryptingstorage to my 5.1.5 installer derived buildout requires the following packages to be pinned to the following versions:
- PasteDeploy = 2.0
- zope.configuration = 3.8.0
- webob = 1.8.3
Must give a shout out to Kim et al who recommended Sentry for capturing errors... For a free service, it's provided me meaningful ways to make some sense of the error messages (let alone capture them!).
===
I have 4.4K events of the following:
AFAICT, invalidation failures + read/conflict errors leave my Zopes unable to connect with their DB. Only a restart of Zope solves the stalled client/site.
EXCEPTION(most recent call last)
TypeError: invalidate() takes exactly 3 arguments (4 given)
File "cipher/encryptingstorage/__init__.py", line 153, in invalidate
return self.db.invalidate(transaction_id, oids, version)
File "ZEO/ClientStorage.py", line 979, in invalidateTransaction
self._db.invalidate(tid, oids)
File "ZEO/asyncio/client.py", line 503, in verify
self.client.invalidateTransaction(server_tid, oids)
Preformatted textcipher/encryptingstorage/__init__.py in invalidate at line 153
return self.db.invalidateCache()
def invalidate(self, transaction_id, oids, version=''):
""" For IStorageWrapper
"""
return self.db.invalidate(transaction_id, oids, version)
def references(self, record, oids=None):
""" For IStorageWrapper
"""
return self.db.references(self._untransform(record), oids)
oids
[
'\x00\x00\x00\x00\x00\x00\x00\xd9',
'\x00\x00\x00\x00\x00\x02c\xfd',
'a'
]
self
<cipher.encryptingstorage.EncryptingStorage object at 0x7f831db25550>
transaction_id
'\x03\xd0\x05\xdbG`\xb3\x99'
version
''
ZEO/ClientStorage.py in invalidateTransaction at line 979
self._db.invalidateCache()
def invalidateTransaction(self, tid, oids):
"""Server callback: Invalidate objects modified by tid."""
if self._db is not None:
self._db.invalidate(tid, oids)
# IStorageIteration
def iterator(self, start=None, stop=None):
"""Return an IStorageTransactionInformation iterator."""
oids
[
'\x00\x00\x00\x00\x00\x00\x00\xd9',
'\x00\x00\x00\x00\x00\x02c\xfd',
'a'
]
self
<ZEO.ClientStorage.ClientStorage object at 0x7f830b7b7250>
tid
'\x03\xd0\x05\xdbG`\xb3\x99'
ZEO/asyncio/client.py in verify at line 503
if vdata:
self.verify_result = "quick verification"
server_tid, oids = vdata
for oid in oids:
cache.invalidate(oid, None)
self.client.invalidateTransaction(server_tid, oids)
else:
# cache is too old
self.verify_result = "cache too old, clearing"
try:
ZODB.event.notify(
cache
<ZEO.cache.ClientCache object at 0x7f830b7b7210>
cache_tid
"\x03\xd0\x048\x0f\x93'\xdd"
exc
TypeError('invalidate() takes exactly 3 arguments (4 given)',)
oid
'a'
oids
[
'\x00\x00\x00\x00\x00\x00\x00\xd9',
'\x00\x00\x00\x00\x00\x02c\xfd',
'a'
]
protocol
Protocol(('127.0.0.1', 8100), 'enc', False)
self
<ZEO.asyncio.client.Client object at 0x7f830af95250>
server_tid
'\x03\xd0\x05\xdbG`\xb3\x99'
vdata
[
'\x03\xd0\x05\xdbG`\xb3\x99',
[
'\x00\x00\x00\x00\x00\x00\x00\xd9',
'\x00\x00\x00\x00\x00\x02c\xfd',
'a'
]
]
Things I've done to keep my application running are:
I was using Supervisor to run my application but Zeo would "flap". Essentially it was connecting and disconnecting and not providing any uptime of more than a few seconds.
Running Zeo in foreground mode (which was ironic considering that Supervisord runs it too via console fg mode) improved uptime so I just decided to start Plone using plonectl.
I tried to research the invalidation-queue-size in regards to Zeo but I did not feel comfortable toying with it. invalidateTransaction is being called but the argument requirement is not being satisfied (I guess due to the latency?).
I am thinking of going to a single instance because I feel powerless (given my lack of knowledge) looking for possible reasons as to why I am losing DB connection and ZEO.asyncio.client issues. I don't like forming surgery while the patient is currently running a marathon etc.