Continous number generation

Plone Support Add-on Development
1

Will this work as expected?

I need a way to

  • generate a continuous number,
  • increased by one each time,
  • starting at a specific offset (like 1000).
  • it will be stored in an dexterity field of a my new content item.

Implemenation idea:

  • add a behavior to the site root with one additional int field
  • edit site root field and store initial offset
  • An ObjectCreated subscriber then:
    • reads value from site root,
    • increments it
    • store increment on site root
    • store increment on my content

This should create a transaction conflict on the site root if two items increment in the same time range and so ensures only one wins. I do not have that many writes that it will create a problem, usually the retry should then work.

Or did I miss something?

2

Jens W. Klein via Plone Community wrote at 2022-8-26 09:37 +0000:

Will this work as expected?
...

This should work.

1 Like
3

I‌ have a INameChooser adapter which does something similar. It should set the id as a number which is increased by 1 each time a content is created in this specific folder:

class NumericIdChooser(object):
    """A name chooser for class contributions.
    """
    implements(INameChooser)

    def __init__(self, context):
        self.context = context

    def checkName(self, name, ob):
        container = aq_inner(self.context)
        return name not in container.contentIds()

    def chooseName(self, name, ob):
        container = aq_inner(self.context)

        def sort_key(key):
            try:
                ret = int(key)
            except ValueError:
                ret = 0  # dummy
            return ret
        ids = map(int, container.contentIds())
        max_id = max(ids, key=sort_key) if ids else 0
        name = max_id + 1
        return str(name)

This works fine probably as long as you do not have a huge amount of items in that folder.

1 Like
4

Couldn't you use the registry for that?

3 Likes
5

Yes, why not. I already have a custom control panel for project specific values, so this is easy too. Good idea.

6

Not sure if I understand completely. Too me it sounds 'quite straightforward'.
I would create an entry in portal_registry, then read it (from ObjectCreated), store it , then save it on content (update content).

Is it technically possible that two ObjectCreated runs at the exactly same time?
Since you store it in registry before updating your content, it seems (to me) very unlikely that there could be a conflict

UPDATE: I 'forgot to save', so I actually wrote this before 'others answered
'.

7

Write requests may occur concurrently. However, they are serialized on the underlying layer. First one wins, the second request will receive a ZODB conflict error. The ZPublisher will repeat the second request up to three times. The application code may define a custom conflict resolution handler that would determine the new state based on the old state, the changed state and the state of the request in conflict.

https://zodb.org/en/latest/ConflictResolution.html

8

Does the registry still provide a transaction lock like your dexterity object would?

And also (because rabbit holes are fun and a great waste of time) can you create a special kind of transaction that is an 'increment' transaction so if many hit at the same time, they can just wait on each other instead of trying to set a value?

(nevermind that last part, I have an over-engineering curse)

9

flipmcf via Plone Community wrote at 2022-8-26 17:11 +0000:

...
And also (because rabbit holes are fun and a great waste of time) can you create a special kind of transaction that is an 'increment' transaction so if many hit at the same time, they can just wait on each other instead of trying to set a value?

In principle, you can create an "increment transaction".
But then it would not be guaranteed that the incremented value
would be used (it would not if the enclosing ("normal") transaction would get
aborted). This may or may not be a problem.

10

CMFUid's UID generator uses an approach very similar to what you described: Products.CMFUid/UniqueIdGeneratorTool.py at master · zopefoundation/Products.CMFUid · GitHub

1 Like
11

If you want a single step increasing number, you should store the max_value_added on the btree itself. This will make sure that you'll have to recompute the new max_value_added as the transaction fails if max_value_added was modified.

ie.
moment in time: now; now+1; now+2; now+3; now+4; now+5
transaction a: start; compute max_value = 2; wait; commit; failed; retry request
transaction b: wait; start; compute max_value = 2; commit; committed; done

At now+5 transaction will have to recompute it's max_value. It will do so, because of transaction retry.
The retry will read the updates max_value_added from the container object (2) and computes max_value_added = 3.

Which is why I think you we should store a counter (max_value_added) on container._tree, which needs to updated when adding an object to the container. Updating can be done using a objectAdded event or custom __setitem__ in a class for your container.

An example could be:

class SomeStorage(dict):
    pass


class SomeContainer:

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self._tree = SomeStorage()
        self._tree.max_numeric_id_seen = -1

    def __setitem__(self, key: str, value):
        try:
            self._tree.max_numeric_id_seen = max(int(key), self._tree.max_numeric_id_seen)
        except ValueError:
            pass
        self._tree[key] = value


class NumericIdChooser:
    def __init__(self, context):
        self.context = context

        # Grab from registry, or from a content field, or provide via an adapter
        self.min_numeric_id = 0

    def checkName(self, name:str, ob=None):
        try:
            name_number = int(name)
            current_max: int = self.context._tree.max_numeric_id_seen
            return name_number > current_max and name_number >= self.min_numeric_id
        except ValueError:
            return name not in self.context._tree

    def chooseName(self, name, ob=None)-> str:
        try:
            chosen = max(self.min_numeric_id, self.context._tree.max_numeric_id_seen + 1, int(name))
        except ValueError:
            chosen = name
        chosen = str(chosen)
        if self.checkName(chosen, ob):
            return chosen
        raise RuntimeError(f"Sound like a you-problem to me.")

Some test cases

container = SomeContainer()
chooser = NumericIdChooser(context=container)
def av(output: str):
    print(output)
    assert output.endswith(('=True', '= True')), f"Failed: {output}"


av(f"{chooser.checkName('0') is True=}")
av(f"{chooser.checkName('1') is True=}")
av(f"{chooser.checkName('3') is True=}")
av(f"{chooser.checkName('henk') is True=}")
av(f"{chooser.checkName(1)=}")
av(f"{chooser.checkName(3)=}")
av(f"{chooser.chooseName('harf') == 'harf'=}")
av(f"{chooser.chooseName('999') == '999'=}")
av(f"{chooser.chooseName('-1') == '0'=}")
av(f"{chooser.context._tree.max_numeric_id_seen == -1=}")
av(f"{container._tree == {}=}")


chooser.min_numeric_id = 44
av(f"{chooser.checkName('0') is False=}")
av(f"{chooser.checkName('1') is False=}")
av(f"{chooser.checkName('3') is False=}")
av(f"{chooser.checkName('henk') is True=}")
av(f"{chooser.checkName(1) is False=}")
av(f"{chooser.checkName(3) is False=}")
av(f"{chooser.chooseName('harf') == 'harf'=}")
av(f"{chooser.chooseName('999') == '999'=}")
av(f"{chooser.chooseName('-1') == '44'=}")
av(f"{chooser.context._tree.max_numeric_id_seen == -1=}")
av(f"{container._tree == {}=}")

container['henk'] = 'I exist'
container['0'] = 'I exist 0'
container['3'] = 'I exist 3'
container['1'] = 'I exist 1'
av(f"{chooser.context._tree.max_numeric_id_seen == 3=}")
av(f"{list(chooser.context._tree.keys()) == ['henk', '0', '3', '1']=}")

# This is allowed according to the container, but the namechooser enforces the min value for the keys.
container[-1] = 'I exist -1'
av(f"{chooser.context._tree.max_numeric_id_seen == 3=}")
av(f"{list(chooser.context._tree.keys()) == ['henk', '0', '3', '1', -1]=}")

The starting number is a bit finicky.

Having typed all this, it might be easier to maintain as a behaviour like id from title, where you apply a marker interface to either all content types that need this or the instances via local behaviors.

1 Like
12

Maybe too late but this product could interest you (collective.behavior.internalnumber · PyPI).
It's used on Plone 4 but could work with little adaptations on current releases.

1 Like