In Plone, how can I remove all missing broken relation values?

I am trying to make a control panel for my Plone 4 site so that when a button is clicked, broken relations are removed from the site. Unfortunately, I'm having trouble removing broken relations.

I am testing for two dexterity content types currently:

Agreement (my.product.agreement)
Project Sample(my.project.projectsample)

The Agreement has a workflow with several states. When the Agreement reaches the Activated state, a Project Sample is created (through a Products.DCWorkflow.interfaces.IAfterTransitionEvent subscriber) . The Agreement's relatedItems is set to include the Project Sample and the Project Sample's relatedItems is set to include the Agreement.

from plone import api
from z3c.relationfield import RelationValue
from zope.component import getUtility
from zope.intid.interfaces import IIntIds

def getProjectSamplesFolder():
    ....
    ....

def agreementTransitioned(obj,event):

    workflow_tool = api.portal.get_tool(name='portal_workflow')
    current_user = api.user.get_current()
    projects_folder = getProjectSamplesFolder()
    action = event.status['action']
    if action <> 'activate':
        return
    
    if len(obj.relatedItems) == 0:
        int_ids = getUtility(IIntIds)
        #use api to create ProjectSample and place it in projects_folder
        new_project_sample = api.create(....)
        ....
        ps_id = int_ids.getId(new_project_sample)
        obj.relatedItems = [RelationValue(ps_id)]
        
        obj_id = int_ids.getId(obj)
        new_project_sample.relatedItems = [RelationValue(obj_id)]
        
        new_project.reindexObject()
        obj.reindexObject()

When viewing an Agreement or Project Sample, the related items portlet shows the related item as it should. However, if I delete one or the other, the relatedItems portlet on the one that wasn't deleted won't render correctly, because the relations are broken.
It shows up as a message where the portlet should be shown:
error while rendering plone.belowcontentbody.relateditems

Originally I was going to do a delete event, but I was having trouble with the delete event firing multiple times. So I figured a control panel that I could run once in a while would clean up broken relations, outside of just the two content types.

from plone.app.controlpanel.form import ControlPanelForm
from zope.interface import Interface
from zope.formlib import form as formlib
from plone import api

from my.product import MessageFactory as _

class IBrokenRelationsCleanupUtility(Interface):
    """Interface for utility that removes broken relations
    """
         pass

class BrokenRelationsCleanupUtility(ControlPanelForm):

    form_fields = IBrokenRelationsCleanupUtility

    label = u"Broken Relations Cleanup Utility"
    description = u"Utility for cleaning up broken relations"
    form_name = u"Broken Relations Cleanup Utility"

    @formlib.action(_(u'label cleanup broken relations', default=u'Run Cleanup'),
                name=u'clean-broken-relations')
    def handle_cleanup_broken_relations(self, action, data):
        #run code for cleaning up broken relations

The handler runs. I tried two different approaches:

The first approach is based on an answer I found on a StackOverflow question:

...
from z3c.relationfield.event import updateRelations
from z3c.relationfield.interfaces import IHasRelations
from zc.relation.interfaces import ICatalog
from zope.component import getUtility
....

class BrokenRelationsCleanupUtility(ControlPanelForm):
    ....                                         
    def handle_cleanup_broken_relations(self, action, data):

        rcatalog = getUtility(ICatalog)
        rcatalog.clear()

        pc = api.portal.get_tool(name='portal_catalog')
        brains = pc.searchResults(object_provides=IHasRelations.__identifier__)
        for brain in brains:
            obj = brain.getObject()
            updateRelations(obj, None)
        print "Catalog rebuilt for %s objects" % len(brains)

The code runs and it prints:
Catalog rebuilt for 885 objects
However, there are still broken relations as the relatedItems portlet is broken still.

The second approach uses collective.cleanup, which just redirects the browser to the page where collective.catalogcleanup (https://github.com/collective/collective.catalogcleanup) runs its code:

class BrokenRelationsCleanupUtility(ControlPanelForm)
    ....
    def handle_cleanup_broken_relations(self, action, data):

        catalogcleanup = api.portal.get().absolute_url() + "/@@collective-catalogcleanup?dry_run=false"
        self.request.response.redirect(catalogcleanup)

It redirects to the page and it says:

Starting catalog cleanup.

NO dry_run SELECTED. CHANGES ARE PERMANENT.


Handling catalog portal_catalog.
Brains in portal_catalog: 885
portal_catalog: removed 0 brains without UID.
portal_catalog: removed no brains in object check.
portal_catalog: 0 non unique uids found.
portal_catalog: 0 items given new unique uids.
portal_catalog: total problems: 0

Handling catalog uid_catalog.
Brains in uid_catalog: 0
uid_catalog: removed 0 brains without UID.
uid_catalog: removed no brains in object check.
uid_catalog: 0 non unique uids found.
uid_catalog: 0 items given new unique uids.
uid_catalog: total problems: 0

Handling catalog reference_catalog.
Brains in reference_catalog: 0
reference_catalog: removed 0 brains without UID.
reference_catalog: removed no brains in object check.
reference_catalog: removed no brains in reference check.
reference_catalog: total problems: 0

Done with catalog cleanup.

Unfortunately, its the same thing with relatedItems rendering improperly.

I'm afraid I'm stuck on what to do. Am I approaching relations incorrectly?

Maybe post the code that you tried?

I apologize, I will try to piece together the code I was trying and post it soon.

No need to apologize! It just helps us answer your questions by looking at specifics. If you can include the errors you got that would be helpful too.

The answer may depend on what Plone version you are using and whether you are using Archetypes or Dexterity content types.

Formerly (Plone 4; Archetypes), references where handled by the reference_catalog (implemented byProducts.Archtetypes.ReferenceEngine.ReferenceCatalog). Its _queryFor method allows you to search for references. It does not return the references themselves but catalog proxies (so called brains) for them. Likely, you can get the reference object from a proxy via its _getObject method even if one of the reference's objects no longer exists. You can then check whether the objects associated with the reference still exist (the reference is unbroken) and otherwise delete the reference. As references are stored as annotations of the source object, you likely need only check the target object.

I do not know how references are implemented with Dexterity (or newer Plone versions).

2 Likes

you can find a description of how references are implemented in Dexterity on its documentation.

I updated my question and provided how I'm attempting to remove broken relations.

yes, I think so; IMO, you're trying to patch something that is broken: you have to avoid broken relations, not to create a configlet to fix them.

what event you were using? are you deleting the items from the user interface? are you using plone.app.linkintegrity?

please don't further edit the question as you're making it more difficult to understand the problem and your rationale.

I don't remember if collective.catalogcleanup can fix broken relations in Dexterity (I think it doesn't), but we can ask @mauritsvanrees.

I'm deleting items from the Actions menu. I'm not using plone.app.linkintegrity at the moment, but I'll check that out.
Originally, I was using OFS.interfaces.IObjectWillBeRemovedEvent to call a function that would remove the relations when an Agreement is removed.

I did try zope.lifecycleevent.interfaces.IObjectRemovedEvent, but I couldn't get the relation value for the Agreement, which makes sense since that occurs after the Agreement was moved.

So in the event, I got the relation value of the Agreement and then removed that value from the list of the Project's relatedItems. I set the Agreement's relatedItems to .

So currently, I'm trying to remove all broken relations and in this particular case, I was dealing with relatedItems. I figured maybe a utility configlet would be appropriate. I was thinking maybe this approach would be useful for other broken relations that may arise.

that's fair; I don't see any references to IObjectWillBeRemovedEvent in Dexterity code so probably you can't use that event (it could be a nice feature to add, BTW; please open a new issue on GitHub, so we don't forget about it). you have to take a look on how plone.app.linkintegrity handles this.

on the other side, it will be better to add the clean up feature to collective.catalogcleanup instead of to a configlet on a customer package; this is a feature that can benefit more people and you will find others willing to contribute if you do so.

good night and good luck!

I figured out why it wasn't working. The references catalog is being called in their code (or atleast in cleanup), which is for archetypes, which dieter mentioned.

However, there's a catalog utility (zc.relation.interfaces.ICatalog) for Relation Values that I've used to get back_references. I was getting the two confused.

Unfortunately, there appears to be an issue with the relation catalog. It won't pick up RelationValues from RelationLists unless the RelationList contains only one item.

My Agreements upon being created are also linked to a second dexterity content type.
Regarding printing the relation value queries I make.

#From the context of an Agreement

catalog = getUtility(ICatalog)
intids = getUtility(IIntIds)
this_agreement = intids.getId(self)

print [i for i incatalog.findRelations(dict(from_attribute='relatedItems',to_id=this_agreement))
#Outputs [<z3c.relationfield.relation.RelationValue object at 0xb3837e2c], meaning it finds the project samples, who happen to have only one related item (the Agreement)

A ProjectSample only has one relatedItem:

#From the context of a ProjectSample

catalog = getUtility(ICatalog)
intids = getUtility(IIntIds)
this_project = intids.getId(self)

print [i for i in catalog.findRelations(dict(from_attribute='relatedItems',to_id=this_project))]
#Outputs [], meaning it can't find the agreements linked to project sample, because the agreements happen to have more than one related item

I did a couple of print statements in Project Sample:

print self.relatedItems
for i in self.relatedItems:
    print i.to_object
#This output None

I'll try to see if I can get a list of all the relations, but I just can't seem to get relation values from relation lists.
So in otherwords this query (even if both the Agreement and Project Sample still exist) isn't working appropriately, only getting the value from Project Sample:

catalog = getUtility(ICatalog)
relations = catalog.findRelations()
print relations
#Outputs only the relation values that are from project to customer agreement

Just to reiterate, its important to note that the relations for "from project sample to agreement" still exist even if agreement gets deleted.

If I find out how to get all relation values from a catalog query, I'll share as maybe it could still be useful as the next step would be removing relations found that are broken.

The question is also how the broken references get there in the first place. I filed this issue around three years ago but couldn't debug what exactly is going wrong. plone.app.relationfield/z3c.relationfield should clean references up as soon as objects are removed.

I don't remember if collective.catalogcleanup can fix broken relations in Dexterity (I think it doesn't), but we can ask @mauritsvanrees.

No, it doesn't fix broken relations.
If you use the referenceablebehavior and have simple references that use UUID, then those references will end up in the reference_catalog, and collective.catalogcleanup may help here. But the package does not touch 'real' relations, with zc.relationfield and intids.