Utility with persistent data

I have a dict that I need to be persistent. It's not tied to content per se nor is it even specific to one Plone site. There is some functionality associated with it so I was originally going to just write a global utility. But it does need to actually save some data, and thus be persistent. I could make the utility a subclass of Products.CMFCore.utils.UniqueObject but that seems like overkill here. I don't need it to be accessible in the ZMI or traversable, I just want a dict stored in a db.

What I tried to do was something like this

class IMyUtility(Interface):
    _history = schema.Dict(
        title='History',
        key_type=schema.TextLine(title='code'),
        value_type=schema.TextLine(title='title'),
    )
    

@implementer(IMyUtility)
class MyUtility(Persistent):
    def __init__(self):
        self._history = OOBTree()

and in zcml

<utility provides=".mycode.IMyUtility" factory=".mycode.MyUtility"/>

This is incomplete - self._history will always be reinitialized to a new OOBTree on start up. What am I missing to make this something actually persistent, or is this just a completely wrongheaded approach?

Store your data as annotation on the Zope or Plone root. Not sure why I would use/need a utility here? Just for the sake of having one?

Zope utilities live in two worlds: global utilities, maintained in the so called GlobalComponentRegistry and managed via ZCML, and so called local (persistent) utilities, maintained in a local component registry and managed via the registry's API.

You can find examples how to register a local (persistent) utility in Products.CMFCore (and dm.zope.saml2.authority.move_handler).

I agree with @zopyx - if it is just a dict. I would get it like so:

def get_data(self):
    ann = IAnnotation(api.portal.get())
    if 'MYTOOL' not in ann:
        ann['MYTOOL'] = PersistentDict()
    return ann['MYTOOL']

Cool, I actually already have a dev site where it's working with annotations, I was just concerned that was the wrong pattern for this case. Sounds like that was unfounded.

If you want to handle large amounts of objects is better to use objects from BTrees package than objects from persistent package. See http://www.zodb.org/en/latest/guide/writing-persistent-objects.html#using-persistent-data-structures

To handle write conflicts properly, you should subclass and define a _p_resolveConflict function. See http://www.zodb.org/en/latest/articles/ZODB2.html#resolving-conflicts

I did notice that BTrees were suggested as acceptable in the docs here https://docs.plone.org/develop/plone/misc/annotations.html#make-your-code-persistence-free and saw that plone.contentrules uses a OOBTree for performance reasons. I don't think I need to define _p_resolveConflict anywhere anymore, since I abandoned using a Persistent class in favor of using annotations?

In an annotation you store a BTree object, so I think conflict might happen. Let's say you have an OOBTree in an annotation and then one thread assigns a key value in there, while another thread does the same just before the change done by the first is stored. In such case I would expect a conflict.

Take it with a grain of salt, maybe I am missing something.

Indeed, but then also consider using souper instead, which comes with a search facility.

1 Like

I look into souper before, but I was not sure how easy -- compared to creating an utility with BTree objects -- it was.

Do I need to create a new catalog to have the search facility working?

Souper is great, very easy to use.

You need to create a catalog, but it's specific to Souper, not a regular Zope catalog. I got it working by following the docs in less than an hour.

You only need to define _p_resolveConflict if you're not satisfied with the default conflict resolution provided by Zope i.e retry transactions. I would say it is not very common to do that.

I would think of defining it only if the application had very high traffic. You're probably ok without it.

Fact! I was able to implement it also in less than an hour. Thanks @jensens (and everyone else involved with it)

1 Like