Broken persistent object in formcontroller after removing PloneHotfix20160830

[frisi (Harald Frießnegger)]

updated a 4.3.11 plone site to 4.3.14 and removed PloneHotfix20160830 (among other hotfixes not needed any longer)
(yes - i ran the upgrade to 4.3.14

after that - creating new content items did not work:

2017-06-25 21:39:18 ERROR Zope.SiteErrorLog 1498419558.880.704178625194 http://cms.localhost:8081/plone//termine/portal_factory/KKVEvent/kkvevent.2017-06-25.5318547167/atct_edit
Traceback (innermost last):
  Module ZPublisher.Publish, line 138, in publish
  Module ZPublisher.mapply, line 77, in mapply
  Module Products.PDBDebugMode.runcall, line 70, in pdb_runcall
  Module ZPublisher.Publish, line 48, in call_object
  Module Products.CMFPlone.FactoryTool, line 478, in __call__
  Module ZPublisher.mapply, line 77, in mapply
  Module ZPublisher.Publish, line 48, in call_object
  Module Products.CMFFormController.FSControllerPageTemplate, line 91, in __call__
  Module Products.CMFFormController.BaseControllerPageTemplate, line 29, in _call
  Module Products.CMFFormController.ControllerBase, line 232, in getNext
  Module Products.CMFFormController.Actions.TraverseTo, line 38, in __call__
  Module ZPublisher.mapply, line 77, in mapply
  Module ZPublisher.Publish, line 48, in call_object
  Module Products.CMFFormController.FSControllerPythonScript, line 107, in __call__
  Module Products.CMFFormController.ControllerBase, line 232, in getNext
  Module Products.CMFFormController.Actions.TraverseTo, line 38, in __call__
  Module ZPublisher.mapply, line 77, in mapply
  Module ZPublisher.Publish, line 48, in call_object
  Module Products.CMFFormController.FSControllerPythonScript, line 107, in __call__
  Module Products.CMFFormController.ControllerBase, line 232, in getNext
  Module Products.CMFFormController.Actions.TraverseTo, line 38, in __call__
  Module ZPublisher.mapply, line 77, in mapply
  Module ZPublisher.Publish, line 48, in call_object
  Module Products.CMFFormController.FSControllerPythonScript, line 107, in __call__
  Module Products.CMFFormController.ControllerBase, line 232, in getNext
  Module Products.CMFFormController.Actions.TraverseTo, line 38, in __call__
  Module ZPublisher.mapply, line 77, in mapply
  Module ZPublisher.Publish, line 48, in call_object
  Module Products.CMFFormController.FSControllerPythonScript, line 107, in __call__
  Module Products.CMFFormController.ControllerBase, line 232, in getNext
SystemError: error return without exception set

to see what's wrong i wanted to visit /portal_form_controller/manage_formActionsForm
there i get this error

2017-06-25 21:44:57 ERROR Zope.SiteErrorLog 1498419897.350.459190811937 http://cms.localhost:8081/plone/portal_form_controller/manage_formActionsForm
Traceback (innermost last):
  Module ZPublisher.Publish, line 138, in publish
  Module ZPublisher.mapply, line 77, in mapply
  Module Products.PDBDebugMode.runcall, line 70, in pdb_runcall
  Module ZPublisher.Publish, line 48, in call_object
  Module Shared.DC.Scripts.Bindings, line 322, in __call__
  Module Shared.DC.Scripts.Bindings, line 359, in _bindAndExec
  Module Products.PageTemplates.PageTemplateFile, line 130, in _exec
  Module Products.PageTemplates.PageTemplate, line 87, in pt_render
  Module zope.pagetemplate.pagetemplate, line 132, in pt_render
  Module zope.pagetemplate.pagetemplate, line 240, in __call__
  Module zope.tal.talinterpreter, line 271, in __call__
  Module zope.tal.talinterpreter, line 343, in interpret
  Module zope.tal.talinterpreter, line 858, in do_defineMacro
  Module zope.tal.talinterpreter, line 343, in interpret
  Module zope.tal.talinterpreter, line 852, in do_condition
  Module zope.tal.talinterpreter, line 343, in interpret
  Module zope.tal.talinterpreter, line 821, in do_loop_tal
  Module zope.tal.talinterpreter, line 343, in interpret
  Module zope.tal.talinterpreter, line 376, in do_startEndTag
  Module zope.tal.talinterpreter, line 405, in do_startTag
  Module zope.tal.talinterpreter, line 482, in attrAction_tal
  Module Products.PageTemplates.Expressions, line 225, in evaluateText
  Module zope.tales.tales, line 696, in evaluate
   - URL: manage_formActionsForm
   - Line 39, Column 10
   - Expression: <PathExpr standard:'action/getObjectId'>
   - Names:
      {'container': <FormController at /plone/portal_form_controller>,
       'context': <FormController at /plone/portal_form_controller>,
       'default': <object object at 0x7f1c21946520>,
       'here': <FormController at /plone/portal_form_controller>,
       'loop': {'action': <Products.PageTemplates.Expressions.PathIterator object at 0x7f1bf95debd0>},
       'nothing': None,
       'options': {'args': ()},
       'repeat': <Products.PageTemplates.Expressions.SafeMapping object at 0x7f1bec773940>,
       'request': <HTTPRequest, URL=http://cms.localhost:8081/plone/portal_form_controller/manage_formActionsForm>,
       'root': <Application at >,
       'template': <PageTemplateFile at /plone/portal_form_controller/manage_formActionsForm>,
       'user': <PropertiedUser 'meister'>}
  Module zope.tales.expressions, line 217, in __call__
  Module Products.PageTemplates.Expressions, line 147, in _eval
  Module zope.tales.expressions, line 124, in _eval
  Module Products.PageTemplates.Expressions, line 74, in boboAwareZopeTraverse
  Module ZODB.Connection, line 860, in setstate
  Module ZODB.Connection, line 914, in _setstate
  Module ZODB.serialize, line 612, in setGhostState
  Module ZODB.serialize, line 605, in getState
  Module copy_reg, line 93, in __newobj__
  Module ZODB.broken, line 106, in __new__
TypeError: ('object.__new__(RedirectTo) is not safe, use Persistence.Persistent.__new__()', <function __newobj__ at 0x7f1c2194ab90>, (<class 'Products.PloneHotfix20160830.redirectto.RedirectTo'>,))

so it looks like i registered or updated formcontroller actions with the hotfix beeing installed.
this also stored an object with class Products.PloneHotfix20160830.redirectto.RedirectTo somewhere in the zodb.
after removing the hotfix this objects can't be loaded and my formcontroller actions are broken.

i temporarily worked around the problem by adding Products.PloneHotfix20160830 to my instance-eggs again.

any pointers how to solve this very appreciated.

[dieter (Dieter Maurer)]

@frisi

after removing the hotfix this objects can't be loaded and my formcontroller actions are broken.

i temporarily worked around the problem by adding Products.PloneHotfix20160830 to my instance-eggs again.

any pointers how to solve this very appreciated.

Excellent analysis!

A reliable (even though somewhat tedious) approach is to locate and fix the (to be) broken objects. It may be necessary to do this before the objects get broken or install something more intelligent than ZODB.broken to handle broken objects (wildcard.fixpersistentutilities may show how to do that).

In your specific case, the location can likely be facilitated by looking where manage_formActionsForm looks for the objects.

[frisi (Harald Frießnegger)]

i managed to fix the broken formcontroller actions by creating the missing module (Products.PloneHotfix20160830.redirectto) dynamically and use zope.deprecation.moved to point to the correct location (Products.CMFFormController.Acttions.RedirectTo):

hope this helps others with similar problems.

start the instance in debug mode: bin/instance debug:

accessing formcontroller actions raises error due to broken imports

>>> plone = app.plone
>>> from zope.component.hooks import setSite
>>> setSite(plone)


>>> fc = plone.portal_form_controller
>>> fc.actions.actions
...
TypeError: ('object.__new__(RedirectTo) is not safe, use Persistence.Persistent.__new__()', <function __newobj__ at 0x7f8286072b90>, (<class 'Products.PloneHotfix20160830.redirectto.RedirectTo'>,))

fix broken imports

>>> import imp
>>> hotfix = imp.new_module('Products.PloneHotfix20160830')
>>> hotfix.redirectto = imp.new_module('Products.PloneHotfix20160830.redirectto')

>>> sys.modules['Products.PloneHotfix20160830'] = hotfix
>>> sys.modules['Products.PloneHotfix20160830.redirectto'] = hotfix.redirectto

# test import
>>> from Products.PloneHotfix20160830 import redirectto

# add zope.deprecation.moved declaration to our dynamic module 
# (see https://stackoverflow.com/questions/3799545/dynamically-importing-python-module)
>>> redirectto_code="""
>>> import zope.deprecation
>>> zope.deprecation.moved('Products.CMFFormController.Actions.RedirectTo')
>>> """
>>> exec redirectto_code in hotfix.redirectto.__dict__

# test if import loads new location:
>>> from Products.PloneHotfix20160830.redirectto import RedirectTo
>>> RedirectTo.__module__
'Products.CMFFormController.Actions.RedirectTo'

# formcontroller actions can be loaded now
>>> fc = plone.portal_form_controller
>>> fc.actions.actions
{<FormActionKey at >: <FormAction at >, <FormActionKey at >: <FormAction at >, ...}

remove all actions with redirect_to action

>>> keys_to_delete = []
>>> for key,action in fc.actions.actions.iteritems():
...     if action.getActionType() == 'redirect_to':
...         keys_to_delete.append(key)
>>> for key in keys_to_delete:
...     fc.actions.delete(key)

re-run setup code that creates the formcontroller actions we just deleted

>>> from my.package.setuphandlers import registerFormControllerActions
>>> registerFormControllerActions(plone)

>>> import transaction; transaction.commit()