Migration Error v5.1.6 to v5.2.4 | Cache values must be persistent objects

riker11451 · March 25, 2021, 8:06pm

I developed an add-on using ploneCLI some years back and it works well for our internal client. I've made some modifications over its life but for the most part it's simple, with two types, one for student and one for course. Counselors, HS liaisons, and managers are allowed to register students into courses that they have permission to, there's a cap on the # of students, etc. I have approx 6800 records in the ZODB.

I'm trying to get things to migrate and I'm getting problems with the upgrade steps. Before I started migrating the Data.fs, in the old site I did the following:

uninstalled unnecessary addons (I was using collective.importexport to extract records but switched to datatables and views instead).
cleared and rebuilt catalog
packed the database
GoogleFu seemed to indicate that the inability to load state may be due to the catalog? so I verified that the catalog was in decent shape with collective.catalogcleanup (didn't actually use it but the info it provided I thought was useful i.e no broken brains well except mine ).

Handling catalog portal_catalog.
Brains in portal_catalog: 6958
portal_catalog: removed 0 brains without UID.
portal_catalog: removed no brains in object check.
portal_catalog: 0 non unique uids found.
portal_catalog: 0 items need new unique uids.
portal_catalog: total problems: 0

Ignored nonexisting catalog uid_catalog.

Ignored nonexisting catalog reference_catalog.

Done with catalog cleanup.
Dry run selected: aborted any transaction changes.

When I run @@plone-upgrade things bork at:

[plone.app.upgrade:32][waitress-0] Indexing new indexes exclude_from_nav.
2021-03-25 15:34:37,709 ERROR [ZODB.Connection:809][waitress-0] Couldn't load state for yc.cnow.content.student.Student 0x0d497e
Traceback (most recent call last):
File "/Users/etyrer/Development/p-524-z2/buildout-cache/eggs/cp27m/ZODB-5.6.0-py2.7.egg/ZODB/Connection.py", line 795, in setstate
self._reader.setGhostState(obj, p)
File "/Users/etyrer/Development/p-524-z2/buildout-cache/eggs/cp27m/ZODB-5.6.0-py2.7.egg/ZODB/serialize.py", line 633, in setGhostState
state = self.getState(pickle)
File "/Users/etyrer/Development/p-524-z2/buildout-cache/eggs/cp27m/ZODB-5.6.0-py2.7.egg/ZODB/serialize.py", line 626, in getState
return unpickler.load()
File "/Users/etyrer/Development/p-524-z2/buildout-cache/eggs/cp27m/ZODB-5.6.0-py2.7.egg/ZODB/serialize.py", line 492, in _persistent_load
return self.load_persistent(*reference)
File "/Users/etyrer/Development/p-524-z2/buildout-cache/eggs/cp27m/ZODB-5.6.0-py2.7.egg/ZODB/serialize.py", line 534, in load_persistent
self._cache.new_ghost(oid, obj)
TypeError: Cache values must be persistent objects.

So I am in unfamiliar territory here. I'm not sure why I'm having an issue with persistence when I don't think I've explicitly toyed with that feature... or at least I don't think I did. The migration fails when new indexes are indexed, could I have an issue with my index? Search works really well btw, so I'm not sure where to troubleshoot.

Searching this error seems to suggest very little either. I would appreciate any thoughtful comments or suggestions. Thanks in advance. -est2

dieter · March 25, 2021, 9:04pm

riker11451:

2021-03-25 15:34:37,709 ERROR **[ZODB.Connection:809][waitress-0] Couldn't load state for yc.cnow.content.student.Student 0x0d497e**
Traceback (most recent call last):
File "/Users/etyrer/Development/p-524-z2/buildout-cache/eggs/cp27m/ZODB-5.6.0-py2.7.egg/ZODB/Connection.py", line 795, in setstate
...
5.6.0-py2.7.egg/ZODB/serialize.py", line 534, in load_persistent
self._cache.new_ghost(oid, obj)
**TypeError: Cache values must be persistent objects.**

Apparently, Student or a formerly direct persistent subobject is no longer persistent. I would use debugging to determine which class is to blame (and then let it derive again from Persistent).

riker11451 · March 26, 2021, 4:07am

Thanks, Dieter

I'm weak on debugging can you assist/pointers?

I did the following:

added Products.PDB to my buildout.
ran the upgrade step and waited till it borked
at the first breakpoint I used some of the canned snippets from the directions

(Pdb) print locals() {'self': <ZODB.serialize.ObjectReader object at 0x10ca9ffd0>, 'oid': '\x00\x00\x00\x00\x00\x02a\xd1', 'obj': <Products.PlonePAS.tools.memberdata.MemberData object at 0x110e63c10>, 'klass': <class 'Products.PlonePAS.tools.memberdata.MemberData'>}

(Pdb) print obj.__class__ <class 'Products.PlonePAS.tools.memberdata.MemberData'>

I'm not really sure what I'm looking at. I see the object ref, the OID, the path to the OID, and the associated memberData object ref. which is great, but I'm not sure what to do with it now?

How do I read what this object is so that I can go to the next step? I did initially have my content type bring in the users' location from MemberData but that was long ago...

Again, thanks in advance

-est2

dieter · March 26, 2021, 6:31am

The (formerly reported) error message suggests that MemberData is no longer persistent - which is surprising. To verify you can use:

from inspect import getmro
getmro(obj.__class__)

One of the listed classes should be persistent.Persistent.
You can issue (one line) Python commands inside the debugger by starting the line with !. In the debugger, the above code would be:

!from inspect import getmro
p getmro(obj.__class__)

dieter · March 26, 2021, 7:09am

I have looked at the Products.PlonePAS code: its MemberData is now an adapter which means a temporary (and no longer a persistent) object. I assume that this is a bug because the class structure differs significantly from that of Products.CMFCore.MemberDataTool: likely, Products.PlonePAS.memberdata should actually use MemberDataAdapter as name in its current MemberData definition and define MemberData as a tiny wrapper around the MemberData of CMFCore. In any case, it is a very bad idea to change the "persistency" of a class as this makes loading from the ZODB almost impossible. I filed "Wrong class name `memberdata.MemberData`? · Issue #60 · plone/Products.PlonePAS · GitHub"; let's see what the Products.PlonePAS authors think about this.

You could try this out:

rename MemberData in Products.PlonePAS.tool.memberdata and the associated configure.zcml to MemberDataAdapter

define MemberData as:

from Products.CMFCore.MemberDataTool import MemberData as BaseMemberData
class MemberData(BaseMemberData): pass```

retry your migration

riker11451 · March 27, 2021, 3:20am

Dieter

Thank you again for the advice.

I was going to clone Products.PlonePAS and modify as suggested but a little voice in my head (I've got lot's of 'em) said maybe look around first and see if anyone else has done anything.

Sure enough, vangheem had started a Pull Request 46 some time ago WIP: fix existing errors from existing ZODB objects in db.

The Persistent-fixes branch appeared to be in line with what you've suggested. I re-ran buildout using this branch and.....

Dry run selected.
Starting the migration from version: 5116
.........
End of upgrade path, main migration has finished.
Starting upgrade of core addons.
Types tool imported.
'Discussion Item' type info imported.
Workflow tool imported.
Done upgrading core addons.
> Your Plone instance is now up-to-date.
Dry run selected, transaction aborted

This branch borks the current Portal Members though. While the migration runs successfully, the

Rebuilding member data information. This step can take a while if your site has many users. Skip broken memberdata for EOlmos2: __init__() takes exactly 2 arguments (3 given) Skip broken memberdata for HDavidThoreau: __init__() takes exactly 2 arguments (3 given) Skip broken memberdata for JosephKing2: __init__() takes exactly 2 arguments (3 given)

After the migration, I cannot access the site as it fails with:

(Pdb) 2021-03-26 22:48:33,159 ERROR [CMFCore.MembershipTool:38][waitress-2] Error during wrapUser

Traceback (most recent call last):
File "/Users/etyrer/Development/p-524-z2/buildout-cache/eggs/cp27m/Products.CMFCore-2.5.0-py2.7.egg/Products/CMFCore/MembershipTool.py", line 159, in wrapUser
u = mdtool.wrapUser(u)
File "/Users/etyrer/Development/p-524-z2/zeocluster/src/Products.PlonePAS/src/Products/PlonePAS/tools/memberdata.py", line 194, in wrapUser
return MemberData(u, self)
TypeError: init() takes exactly 2 arguments (3 given)

/Users/etyrer/Development/p-524-z2/zeocluster/src/Products.PlonePAS/src/Products/PlonePAS/tools/memberdata.py(194)wrapUser()
-> return MemberData(u, self)

So I'll review it further on Sunday prob as the weather will be crummy here... and see what I can do with MemberData.py to get it working.

Any thoughtful suggestions or helpful pointers greatly appreciated.

Again thanks!

In health -est2

dieter · March 27, 2021, 5:53am

The modern (i.e. recent CMFCore) MemberData is a pure data container -- all logic has been moved out to the MemberDataAdapter. Correspondingly, it no longer needs a reference to the tool; the signature of its __init__ was changed accordingly: formerly def __init__(self, tool, id) now def __init__(self, id).

riker11451 · March 29, 2021, 3:11am

OK, I think I've got a working solution.

MrDeveloper configged to get WIP: fix existing errors from existing ZODB objects in db.
Ran migration and then shutdown instance.
Re-ran buildout using stock P5.2.4 installer (PlonePAS==6.0.7).
Rerun-upgrade step (just said everything was good in 4 lines)
I got a mysterious Conflict error (not sure what to make of that at the moment) buuuut .......
I am able to access the site and members are there!

Is this weird that I'm the only reporting with persistence on MemberData?

This particular add-on/site uses local roles/users because of people from disparate locations/entities. as they're not in any sort of LDAP store.

Thanks Dieter, really feel like TIL something