A POSKeyError in my index

Plone Support
2

Update: it is certainly in my metadata index, which is really fun.

I would like to document / update the procedure:

zodbpack --prepack with relstorage

Plone 5.2 Python3

First, add stuff to buildout...

parts += relstorage-zodbpack-conf
         relstorage-zodbpack

Make sure to match your Relstorage Version on Production
I put in "RelStorage==2.1.1" in my eggs and buildout didn't give me an error :slight_smile:

[relstorage-zodbpack]
recipe = zc.recipe.egg:scripts
eggs =
    RelStorage==2.1.1
scripts = zodbpack
initialization =
    config = '${buildout:directory}/etc/zodbpack.conf'
arguments = config

you need a config file for zodbpack. It's basically the xml straight from your zcml.conf

This recipe creates a config file for you - if you want. Just make sure that:

  • pack-gc true
  • create-schema false
  • point to your broken database.
[relstorage-zodbpack-conf]
recipe = collective.recipe.template
input = inline:
    <relstorage>
        pack-gc true
        create-schema false
      <mysql>
        host locahost
        port 3306
        db plone_borked
        user plone
        passwd plone
      </mysql>
    </relstorage>
output = ${buildout:directory}/etc/zodbpack.conf

Now, the zodbpack script is broken, you have to fix it. Create a ticket in github while you're at it.

zodbpack: error: argument config_file: can't open 'h': [Errno 2] No such file or directory: 'h'

that 'h' is because the full path to my config file is '/home/flipmcf/path/to/instance'. Yes folks, Strings are iterables in python.

edit bin/zodbpack. remove 'config' argument

if __name__ == '__main__':
    sys.exit(relstorage.zodbpack.main())

then, run bin/zodbpack --prepack etc/zodbpack.conf

The 'bug' is that args aren't passed correctly into relstorage.zodbpack.main(), so we will just put everything in argv.

no module named mysql and/or MySQLdb

Unable to use the driver 'None' for the database 'mysql'. Available drivers are: [].

Throw this into
egg += mysqlclient
PyMySQL

One of those will fix it.
But, to be honest, I really hacked at the ./bin/zodbpack script and just added egg paths I found in ./bin/instance until it 'worked'. :man_shrugging:

Back to my unique POSKeyError, As shown by this little guy on ./bin/instance debug - it's the IOBTree iterator that's blowing up while reading metadata:

generator = catalog.getAllBrains()
while True:
    try:
        brain = generator.__next__()
        ## brain.getObject()
    except:
        import pdb; pdb.set_trace()

I DO NOT get the object, the comment is there because that's typically what you do to generate a POSKeyError, but no 'getObject()' is needed to make it blow up.

the POSKeyError is generated during the __next__() call on the generator. At this point, I'm inside the IOBTree "walk" implementation, and I haven't opened the C code there much. I'd like to... I mean, that's cool stuff, but I'm kind of busy right now.

References:
https://zodb.org/en/latest/articles/multi-zodb-gc.html