We need to stop using an add-on that registered a browserlayer. Which is the best way to do it and avoid pickling errors between upgradeSteps?

Plone Support Add-on Development
1

We have a project called brasil.gov.portal that is mainly a policy adding a bunch of dependencies and a specific theme. Right now we're into creating a new release, 1.1.4, and some installations are using an old version, 1.0.5. All releases are using upgradeSteps between them.

For this 1.1.4 release, we're planning to remove plone.app.collection 2.0b5 pinning (since it's not needed after plone.app.contenttypes 1.0) but we now have a problem: this plone.app.collection version registered plone.app.collection.interfaces.IPloneAppCollectionLayer as a browserlayer, so if we remove the pinning, run buildout and try to run our upgradeSteps, at the end of the first one from 1.0.5 to 1.1.4, we get:

2016-06-14 15:51:25 ERROR Zope.SiteErrorLog 1465930285.170.776153375467 http://localhost:8080/Plone/portal_setup/manage_doUpgrades
Traceback (innermost last):
  Module ZPublisher.Publish, line 146, in publish
  Module Zope2.App.startup, line 301, in commit
  Module transaction._manager, line 89, in commit
  Module transaction._transaction, line 329, in commit
  Module transaction._transaction, line 443, in _commitResources
  Module ZODB.Connection, line 567, in commit
  Module ZODB.Connection, line 623, in _commit
  Module ZODB.Connection, line 658, in _store_objects
  Module ZODB.serialize, line 422, in serialize
  Module ZODB.serialize, line 431, in _dump
PicklingError: Can't pickle <class 'plone.app.collection.interfaces.IPloneAppCollectionLayer'>: import of module plone.app.collection.interfaces failed

We know this can be solved by:

  • Adding 2.0b5 branch again to the instance buildout;
  • Uninstall plone.app.collection (in our policy we would need to create an "uninstall" script and use "bin/instance run script.py" since a lot of products and profiles are hidden);
  • Run the upgradeSteps;
  • Remove 2.0b5 branch from buildout.

...but we have dozens of sites using brasil.gov.portal and doing this manual process is really cumbersome, specially for users that aren't into ZODB and Plone internals and are just trying to upgrade a policy product from http://localhost:8080/Plone/prefs_install_products_form. An upgradeStep needs to be run ASAP after buildout because there's a new record in registry from collective.cover that, if not run, breaks the frontpage.

Is there a way to solve this programatically (adding a mock, a fake class, something, we don't mind having this fake class if it means less pain for integrators and users) or are we stuck into adding a note to our README between upgrades about this manual solution? We know the correct way is the manual one to avoid crap in ZODB, but sometimes we need to be pragmatic. Should we just copy some snippets from https://github.com/collective/wildcard.fixmissing and adapt to our use case? What do you think?

1 Like
2

It is difficult to answer questions like this without a detailed analysis.

In my personal view, you should get the ZODB fixed. I.e. find the objects referencing no longer available resources and clean them up. With a bit of trickery, you should be able to do that in your upgrade step.

The trickery might involve a concept called "module alias". It is implemented by sys.modules[<alias>] = <module>. In this way, you can let a module of your own (with replacement definitions) act as replacement for a maybe missing other module.

1 Like
3

I recently had to uninstall a whole slew of old addons as part of an upgrade to Plone5 project.

After trying to fix the mess with wildcard.fixpersistentutilities and not really getting rid of all the errors, I settled for the solution of forking the upstream package, add a proper uninstaller, uninstall the package (and submit a PR). Just install the fork on all your sites, run the uninstaller and remove the package completely from buildout afterward.

Here's an example of solving the same problem you're having: getting rid of a browser layer.

2 Likes
4

Completely unrelated, but remember that it is possible for
metadate.xml to depend on your own profile, too, meaning you dont have to write to many scripts_

default profile
<dependency>profile-my.product:install

install profile
<dependency>profile-my.product:uninstall

uninstall profile
etc.....

1 Like
5

I would write an upgrade step, like I did for collective.geolocationbehavior:


(don't forget to raise the metadata.xml version)

Basically:

from plone.browserlayer.utils import unregister_layer
def remove_browserlayer(context):
    try:
        unregister_layer(name=u"collective.geolocationbehavior")
    except KeyError:
        # No browser layer with name collective.geolocationbehavior registered
        pass
1 Like
6

@thet Let me try to give a better explanation where the problem lies about using upgradeSteps in this situation.

We have brasil.gov.portal version 1.0.5, that uses plone.app.collection.interfaces.IPloneAppCollectionLayer from 2.0b5 branch.

From 1.1.4 version onwards, we remove plone.app.collection from our buildout.cfg.

From 1.0.5 to 1.1.4, we have 4 upgradeSteps: so I can't create an upgradeStep (the 5th one) that removes/fixes this browserlayer, because we have 4 upgradeSteps to be executed before being able to call the last one. Well, we can call the last one first, but them I need to call again all old upgradeSteps and add a note to the release explaining the situation (this is really backwards and confusing).

Suppose 2 months from now we have release 1.1.5. If the user didn't upgrade to 1.1.4 before from 1.0.5, s/he needs to know that the upgradeStep, specific from 1.1.4 needs to be called before all of the others (or a PicklingError will be thrown), and than call all upgrades from 1.0.5 onwards, and I need to redocument this behavior presented in 1.1.4, but now in 1.1.5 changelog notes.

Suppose, again, 3 months from now we have release 1.1.6: if the user didn't upgrade... you get the idea, the same problem as above, the same need to document it for every new release onwards. This approach is error prone and cumbersome. The packages are already on pypi.

We can change the first upgradeStep in 1.0.5 version to remove the browserlayer, but it's an incorrect approach, because the layer was only removed in 1.1.4.

It's kind of a "upgradeStep chicken and egg problem".

I'm starting to think that, in this specific case (because the only thing that's giving problems in ZODB is the layer), the best thing to do is:

  • Have a bbb.py or a good documented init.py that tries to import the problematic interfaces: if it can't import them, we set a WARNING using logging module, and @dieter approach of using module alias as a fallback;
  • Document in the release changelog that these modules should be removed, telling the integrator to add them again in buildout and them uninstalling them to fix;
  • Suppose that, in a 2.x release, there's an interest to remove this bbb for cleaning up code base: we can use a zope deprecation warning to warn integrators and ease the pain of migration.

This way, you don't have a broken website that can't run the upgradeSteps but you still show the user that something needs to be fixed. It's a responsible and pragmatic approach for this situation: it may not be the best one for other packages that have more complex uninstall profiles.

7

Surely you'll need an upgrade step to uninstall that browserlayer in the version before the version that removes plone.app.collection. So you should put that uninstall script in 1.1.4 and remove p.a.collection only in 1.1.5. If you already released 1.1.4 you should re-add p.a.collection in 1.1.5 and include the uninstall step there, then remove it only in 1.1.6 from the buildout.

With this approach you need to document that the version that has both the p.a.collection egg and the uninstall step needs to be installed first, before you can move to the next version. The alternative is to not remove p.a.collection from the buildout, only uninstall.

8

I think for your case best is to first add a "module alias" as @dieter suggested and remove plone.app.collection. Now you can start Zope as usal. Then with this provide an upgrade step in brasil.gov.portal that removes the layer from the portal, so the bbb is not needed anymore. Keep the bbb until the next major release.

9

@gyst this approach works, but now the user is stuck into upgrading in specific sequential versions, and this needs to be thoroughly documented in the changelog, if possible, expliciting the exception that it's thrown when it's not being upgraded in the correct order.

This is not optimal from a user perspective: suppose you upgrade from 1.0.5 to 1.1.8, 6 months from now: a completely strange error blows in your face, and them s/he needs to check all versions from the package in changelog, to see that, in 1.1.4, there's a documentation saying "please, follow this sequential order to upgrade".

We could create a major release, 2.x, because with major releases at least some breakage is expected, but I don't agree that a major release should be created just by removing a simple package like plone.app.collection, and I don't want this "setuptools syndrome" in our package :slight_smile:

But thanks for all the suggestions, this will be indeed useful in another scenarios. @jensens one is similar to ours, but instead of the manual step of editing the buildout, do an upgradeStep and maintain the bbb module.

10

Thinking loud: I think a Plip - and its implementation - with goal to change the persistent registration in plone browserlayer is a good idea :wink: storing dotted path instead of interface class.

2 Likes