Corrupted Pickle object in Zope database

Plone Support Administering Plone sites
1

I have a Data.fs file that was partially corrupted due to some underlying hardware failures which I have been working on recovering. After using fsrecover to address the corrupted transactions and then patching the resulting PosKeyErrors I am now left with the situation described by https://github.com/mitodl/salt-ops/issues/652

I am currently investigating how to modify the fsrecover script to purge the invalid pickle object but if anyone has either done this before or knows of a better approach I would appreciate any advice.

2

So what I think you're saying is that so far you've got a working zeo/zope but can't load anything because your ZODB is still looking for missing objects?

Have you tried:

http://www.zodb.org/en/latest/articles/multi-zodb-gc.html

Are your missing objects from an add-on?

Have you tried taking a bare vanilla site and attempt a restore?

3

Thank you for the response. I have the site up and running and I am able to interact with it, the problem is that I can't run Zeopack due to the corrupted data.

I was able to locate the object IDs and transaction IDs that correspond to the truncated data records but now I'm not able to determine how to delete them from the database. I have also tried filtering for those records in the fsrestore.py script but it ends up crashing due to running out of memory when it tries to unpack the transaction ID for the corrupted record.

When I try to overwrite the corrupted records with an empty Persistent object it throws a ConflictError. If anyone has advice on how to delete or otherwise remove those objects that would be immensely helpful.

4

Do you have an instance with DB in question set up on a VM that you can experiment on?

What you're going to do next is not something you want to try on your only copy.

  1. Start your instance.
  2. Run /bin/zopepy (zopepy is just python loaded with the same modules as your Plone instance).

Look at the examples on the following page about running Zope scripts from the CLI. Its a bit old but most of the examples work.

Some examples here on how to delete objects that behave poorly (you know permissions).

How to remove an object by ID

And finally an old, old post I finally found via google-fu from the PloneChix, their definitive guide to the POSkey error.

I also think Andreas or Dieter had a guide somewhere on broken objects too, which probably doesn't relate anymore due to programming changes but maybe they can add to this if so inclined.

5

Thank you for those links, that's very helpful. Fortunately I have an entirely separate environment that I've been working on migrating to that is available for testing.

After using the steps in your third link I started zeopack and it's looking promising so far. I will report back with the results :slight_smile:

6

I figured you had a separate environment, but I'd be remiss if I didn't provide the standard disclaimer of making a copy! :smile:

We've had issues too with stubborn POSkey's from the old Plone4Artists days (:face_vomiting:).

I hope you're able to migrate and get on to more important stuff! Perhaps if you have a moment you could post your use case and we can add to the docs how to recover as instructions ar fairly thin.