AttributeError on catalog rebuild

Plone Support Administering Plone sites
1

I'm facing the following error that aborts the process while trying to rebuild the catalog of one of our sites:

2017-02-16T01:57:21 ERROR Zope.SiteErrorLog 1487217441.430.766221285057 http://localhost:8281/example/portal_catalog/manage_catalogRebuild
Traceback (innermost last):
  Module ZPublisher.Publish, line 138, in publish
  Module ZPublisher.mapply, line 77, in mapply
  Module ZPublisher.Publish, line 48, in call_object
  Module Products.CMFPlone.CatalogTool, line 442, in manage_catalogRebuild
  Module plone.app.discussion.patches, line 47, in patchedClearFindAndRebuild
  Module OFS.FindSupport, line 239, in ZopeFindAndApply
  Module OFS.FindSupport, line 239, in ZopeFindAndApply
  Module OFS.FindSupport, line 239, in ZopeFindAndApply
  Module OFS.FindSupport, line 239, in ZopeFindAndApply
  Module OFS.FindSupport, line 239, in ZopeFindAndApply
  Module OFS.FindSupport, line 198, in ZopeFindAndApply
  Module Products.ZCatalog.Lazy, line 190, in __getitem__
  Module plone.folder.ordered, line 93, in <lambda>
  Module plone.folder.ordered, line 58, in _getOb
AttributeError: 'trabalhadores-poderao-opinar-se-concordam-ou-nao-com-a-cobranca-do-imposto-sindical'

any hints on how can I solve this?

2

Looks like that folder thinks it contains an item that it doesn't. This might fix it: folder.getOrdering().notifyRemoved('trabalhadores-poderao-opinar-se-concordam-ou-nao-com-a-cobranca-do-imposto-sindical')

2 Likes
3

thanks, David; it's working! the hardest part was find out what was the folder containing the offending objects (there were lots!).

I used this modified version of Martijn's script to walk over the site:

from collections import deque
from plone.dexterity.interfaces import IDexterityContent
from Products.ATContentTypes.interfaces.folder import IATFolder

import transaction


def tree_walker(root):
    # stack holds (parent, id, obj) tuples
    stack = deque([(None, None, root)])
    while stack:
        parent, id, next = stack.popleft()
        if callable(getattr(next, 'objectItems', None)):
            try:
                stack.extend((next, id, child) for id, child in next.objectItems())
            except AttributeError:
                print (parent, id, next)
        yield parent, id, next


def is_folder(obj):
    return IATFolder.providedBy(obj)


count = items = 0
for parent, id_, obj in tree_walker(site):
    if is_folder(obj):
        print 'Found folder {0} at {1}'.format(id_, '/'.join(obj.getPhysicalPath()))
        obj._p_changed = True
        count += 1
        if count % 100 == 0:
            transaction.savepoint(True)
    else:
        items += 1
        if items % 1000 == 0:
            print '{0} items processed, {1} matches so far'.format(items, count)

transaction.commit()

then, when finding a folder with issues I ran the following script to clean it up:

def clean_folder(folder):
    while True:
        try:
            folder.objectItems()
        except AttributeError as e:
            id_ = str(e.message)[1:-1]
            print id_
            folder.getOrdering().notifyRemoved(id_)
        else:
            break

I'm trying to rebuild the catalog now, but seems to be working so far.

1 Like
4

Are you aware of collective.catalogcleanup ? It was if I remember correctly created by Maurits for a project where a migration from Plone 3 to Plone 4 gave a lot of issues with stale catalog objects (and Plone 4 migration doing a number of content sweeps (reindexing, blob migration), but it's useful in general for keeping catalogs into shape for Plone 4 and probably Plone 5 as well.

2 Likes
5

no, but now I am… it could have saved me 2 days of work.

what's the current status? I see this on the README:

Compatibility

I have tried this on Plone 3.3 and 4.1. It will likely work on other versions as well.

I have not tried it with Dexterity items. It might not be needed for those. Likely, it will either succeed normally or you get an AttributeError somewhere and the cleanup run is aborted.

6

@mauritsvanrees have you tested it recently on Plone 4.3 sites using Dexterity?

7

@hvelarde I see one site with dexterity where I added it in 2013. It had version 1.4. When I ran it, it fixed a few problems. But latest version 1.6 should be better. But that one gives an error:

Traceback (innermost last):
  Module ZPublisher.Publish, line 138, in publish
  Module ZPublisher.mapply, line 77, in mapply
  Module ZPublisher.Publish, line 48, in call_object
  Module collective.catalogcleanup.browser, line 56, in __call__
  Module collective.catalogcleanup.browser, line 166, in check_references
  Module collective.catalogcleanup.browser, line 287, in get_object_or_status
AttributeError: 'NoneType' object has no attribute 'getId'

From a quick look, this seems a problem in the code that has nothing to do with Archetypes versus dexterity. Something with a brain in the uid catalog for a reference... I can probably fix that.

But it should be okay to use with dexterity too.

1 Like
8

just test it in the same site I was fixing the other day and run into an issue:

2017-02-22T02:23:31 ERROR Zope.SiteErrorLog 1487741011.420.737143190852 http://localhost:8281/example/@@collective-catalogcleanup
Traceback (innermost last):
  Module ZPublisher.Publish, line 138, in publish
  Module ZPublisher.mapply, line 77, in mapply
  Module ZPublisher.Publish, line 48, in call_object
  Module collective.catalogcleanup.browser, line 57, in __call__
  Module collective.catalogcleanup.browser, line 220, in non_unique_uids
  Module collective.catalogcleanup.browser, line 18, in path_len
  Module Products.ZCatalog.CatalogBrains, line 51, in getPath
  Module Products.ZCatalog.ZCatalog, line 518, in getpath
KeyError: 1304083406