Plone 5.2.5 released

Announcements
1

Release Manager Maurits van Rees has released Plone 5.2.5 on Friday August 6, 2021.
Some links:

Note: this is a fresh release. Installers are not ready yet, but will be made available, and the above pages updated.

Some highlights of this release are:

  • Security fixes in AccessControl and Products.isurlinportal.
  • Security fixes from Products.PloneHotfix20210518 taken over in core.
  • Zope: 4.5.5 to 4.6.3
  • Products.CMFPlone: Add PLONE52MARKER Python marker.
  • plone.app.iterate: Add proper support for Dexterity folderish content.
  • plone.folder: restore webdav support.
  • plone.registry: Allow plone.schema.JSONField to be stored in registry (dictionary-like).
  • plone.namedfile: Cache stable image scales strongly.
  • plone.recipe.zope2instance: customize WSGI, profiling, python-env.
  • plone.restapi: JSONField, sub blocks, use_site_search_settings.
  • Lots of bugfixes, especially improving Python 3 compatibility.
9 Likes
2

Thanks! https://demo.plone.org is now running Plone 5.2.5

2 Likes
3

For those working with M1 (arm) macs and trying to install Plone 5.2.5 locally with python 3.8.X, you

could run into installation issues with packages Btrees, persistent and cffi.

The error output contains something like this with:

File "/private/var/folders/bc/qzs1h_3n2dggkdkm5t07_2m80000gn/T/pip-wheel-c1ha9g2_/persistent/.eggs/cffi-1.14.6-py3.8-macosx-11.0-arm64.egg/cffi/api.py", line 48, in __init__
        import _cffi_backend as backend
    ImportError: dlopen(/private/var/folders/bc/qzs1h_3n2dggkdkm5t07_2m80000gn/T/pip-wheel-c1ha9g2_/persistent/.eggs/cffi-1.14.6-py3.8-macosx-11.0-arm64.egg/_cffi_backend.cpython-38-darwin.so, 2): Symbol not found: _ffi_prep_closure

I've been trying to figure out what exactly is going on, there are similar issue reports from other Python projects online where cryptography installs show the same errors, but I haven't got to a 80+% theory yet of what is going on. There are caches involved, and the error you see is not actually from building a binding, but a call to the python3 executable on the path/venv used in building the binding.

If I pin back BTrees and Persistent to the versions used in Plone 5.2.4 all is well. If I first install cffi, persistent and btrees in my virtualenv with pip using parameter "--no-binary :ALL:" and use Python 3.8.6 or lower I can install these packages and bin/buildout.

Python 3.9 works fine but shouldn't be used with Plone 5.2/Zope 4.X. . Python 3.8.9-3.8.11 and a local venv where I try to install the separate packages with pip from source fail.

If more people experience this problem, let's create a separate thread.

2 Likes
4

My issues with installing Plone 5.2.5 seem to be have been caused by wrong builds of the cffi module that were stored in the pip wheel cache. And that cffi was pulled from that cache when buidout/setuptools tries to build new bindings for Btrees / Persistent. Which is very strange as buildout 2.x is setuptools/dist-utils based and AFAIK shouldn't use any local pip infrastructure or cached eggs. Unless they were installed first in the virtualenv one is also running buildout from.

It probably lay dormant for months on my development machine until the Btrees/Persistent pinnings were updated. As soon as I replaced the cffi in the pip cache with a freslly compiled version with the right bnding, bin/buildout worked flawless. Until now I was only concerned with buildouts shared egg directory you can configure with an eggs-directory statement in your local buildout, or shared default.cfg. :open_mouth:

For future reference: the issue on Macs for this particular error is easy to check once you know where to look: create a virtualenv, bin/pip install cffi. cffi is either downloaded and compiled for the first time, but if it already got into the local pip cache it is collected from there. If you want to check beforehand, run pip cache list cffi*

Now you can check with otool -L ./lib/python3.8/site-packages/_cffi_backend.cpython-38-darwin.so in your virtualenv root to which low level libraries the python module is dynamically linked. You'll see output like this:

otool -L lib/python3.8/site-packages/_cffi_backend.cpython-38-darwin.so
lib/python3.8/site-packages/_cffi_backend.cpython-38-darwin.so:
	/opt/homebrew/opt/libffi/lib/libffi.7.dylib (compatibility version 9.0.0, current version 9.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.100.5)

In this case my cffi module is linked against the version I installed with Homebrew. If your cffi on arm Mac is linked to /usr/lib/libffi.dylib (compatibility version 1.0.0, current version 27.0.0) and this cffi egg gets stored in the pip or buildout egg cache and retieved later, you might run into similar issues.

You can use otool -L (the equivalent of ldd on linux) also for other python modules like Pillow (libjpeg, libpng) or libevent (libmemcached bindings) to check afterwards to which libraries the binding was linked at compile time.

5

Hi @mauritsvanrees, thanks for the release!

I just gave it a try and found out that accessing /manage_main gives me a 404. Trying it over /manage seems to work partially. It gives me 404 for e.g. acl_users, portal_setup, portal_catalog etc.

Does anyone else encounter that?

6

@fredvd I ran into this issue on my iMac Pro 2017 (no M1!) as well when running the plone.restapi tests:

make test
bin/test
Traceback (most recent call last):
  File "bin/test", line 273, in <module>
    sys.exit(zope.testrunner.run((['-s', 'plone.restapi', '--auto-color', '--auto-progress']) + [
  File "/Users/timo/.buildout/eggs/zope.testrunner-5.3.0-py3.8.egg/zope/testrunner/__init__.py", line 31, in run
    failed = run_internal(defaults, args, script_parts=script_parts, cwd=cwd,
  File "/Users/timo/.buildout/eggs/zope.testrunner-5.3.0-py3.8.egg/zope/testrunner/__init__.py", line 55, in run_internal
    runner.run()
  File "/Users/timo/.buildout/eggs/zope.testrunner-5.3.0-py3.8.egg/zope/testrunner/runner.py", line 181, in run
    feature.global_setup()
  File "/Users/timo/.buildout/eggs/zope.testrunner-5.3.0-py3.8.egg/zope/testrunner/find.py", line 525, in global_setup
    tests = find_tests(self.runner.options, self.runner.found_suites)
  File "/Users/timo/.buildout/eggs/zope.testrunner-5.3.0-py3.8.egg/zope/testrunner/find.py", line 174, in find_tests
    for suite in found_suites:
  File "/Users/timo/.buildout/eggs/zope.testrunner-5.3.0-py3.8.egg/zope/testrunner/find.py", line 195, in find_suites
    for fpath, package in find_test_files(options):
  File "/Users/timo/.buildout/eggs/zope.testrunner-5.3.0-py3.8.egg/zope/testrunner/find.py", line 263, in find_test_files
    for f, package in find_test_files_(options):
  File "/Users/timo/.buildout/eggs/zope.testrunner-5.3.0-py3.8.egg/zope/testrunner/find.py", line 291, in find_test_files_
    for (p, package) in test_dirs(options, {}):
  File "/Users/timo/.buildout/eggs/zope.testrunner-5.3.0-py3.8.egg/zope/testrunner/find.py", line 346, in test_dirs
    p = import_name(p)
  File "/Users/timo/.buildout/eggs/zope.testrunner-5.3.0-py3.8.egg/zope/testrunner/find.py", line 423, in import_name
    __import__(name)
  File "/Users/timo/workspace/plone/plone.restapi/src/plone/restapi/__init__.py", line 1, in <module>
    from AccessControl import allow_module
  File "/Users/timo/.buildout/eggs/AccessControl-4.3-py3.8-macosx-11-x86_64.egg/AccessControl/__init__.py", line 18, in <module>
    from AccessControl.Implementation import setImplementation
  File "/Users/timo/.buildout/eggs/AccessControl-4.3-py3.8-macosx-11-x86_64.egg/AccessControl/Implementation.py", line 99, in <module>
    setImplementation(_default_implementation)
  File "/Users/timo/.buildout/eggs/AccessControl-4.3-py3.8-macosx-11-x86_64.egg/AccessControl/Implementation.py", line 53, in setImplementation
    from AccessControl import ImplC as impl  # NOQA
  File "/Users/timo/.buildout/eggs/AccessControl-4.3-py3.8-macosx-11-x86_64.egg/AccessControl/ImplC.py", line 16, in <module>
    from AccessControl.cAccessControl import PermissionRole
  File "/Users/timo/.buildout/eggs/AccessControl-4.3-py3.8-macosx-11-x86_64.egg/AccessControl/SimpleObjectPolicies.py", line 48, in <module>
    from BTrees.IIBTree import IIBTree
  File "/Users/timo/.buildout/eggs/BTrees-4.9.2-py3.8-macosx-11-x86_64.egg/BTrees/__init__.py", line 59, in <module>
    mod = create_module(family)
  File "/Users/timo/.buildout/eggs/BTrees-4.9.2-py3.8-macosx-11-x86_64.egg/BTrees/_module_builder.py", line 219, in create_module
    populate_module(vars(mod), key_type, val_type, iface, mod)
  File "/Users/timo/.buildout/eggs/BTrees-4.9.2-py3.8-macosx-11-x86_64.egg/BTrees/_module_builder.py", line 134, in populate_module
    from ._base import _fix_pickle
  File "/Users/timo/.buildout/eggs/BTrees-4.9.2-py3.8-macosx-11-x86_64.egg/BTrees/_base.py", line 19, in <module>
    from persistent import Persistent
  File "/Users/timo/.buildout/eggs/persistent-4.7.0-py3.8-macosx-11-x86_64.egg/persistent/__init__.py", line 36, in <module>
    from persistent import picklecache as _picklecache
  File "/Users/timo/.buildout/eggs/persistent-4.7.0-py3.8-macosx-11-x86_64.egg/persistent/picklecache.py", line 30, in <module>
    from persistent.ring import Ring
  File "/Users/timo/.buildout/eggs/persistent-4.7.0-py3.8-macosx-11-x86_64.egg/persistent/ring.py", line 22, in <module>
    from persistent import _ring
ModuleNotFoundError: No module named '_cffi_backend'
make: *** [test] Error 1
➜  plone.restapi git:(master) bin/python3 --version
Python 3.8.10
➜  plone.restapi git:(master) bin/pip cache list cffi*
Cache contents:

 - cffi-1.14.4-cp37-cp37m-macosx_11_0_x86_64.whl (176 kB)
 - cffi-1.14.4-cp39-cp39-macosx_11_0_x86_64.whl (176 kB)

I fixed the error by just upgrading cffi from 1.14.4 to:

cffi = 1.14.6

Will give it a shot later today on my M1 mac...

7

I haven't seen that. Did you upgrade from 5.2.4? Or from an earlier version?

Do you have STRICT_TRAVERSE_CHECK in your OS environment (or in parts/zeoclient/etc/zope.conf? That could lead to these problems, although I don't know why you would have this.

Are you overriding any base Zope templates maybe?

Can you show a traceback of a 404?

Could this be something in a web server like Apache or nginx which partially forbids access to the ZMI? I have that for several Plone 4 sites where I don't want to add plone4.csrffixes. Should not be needed in Plone 5, but it can still be a valid idea.

8

@tisto Did you get it working again?

Just to clarify to maybe others, The ModuleNotFoundError doesn't mean it isn't installed/on file system, but it could also mean that Python tries to load the cffi library and it exists on the python path, but one of the dynamic links in _cffi_backend.so is broken/missing, so the loading failes and Python then says 'not found'. :open_mouth:

You could check with first starting python in your buildout that has the extended path (i.e. zopepy in coredev), requesting sys.path and figuring out which cffi module location is used. And then quering with otool -L as in my previous example which libffi library was linked to while building that cffi. It's either /usr/lib/libffi.dylib (OSX) or some path to a homebrew/custom libffi build.

Upping the cffi version pin in the requirement/buildout can help to disables any cached versions with wrong bindings as a new cffi python module needs to be build again.

9

@fredvd yeah. I just upgrade to cffi to 1.14.6 (buildout/plone-5.2.x.cfg at 5.2 · kitconcept/buildout · GitHub) and that made it work for me.

10

Hi Maurits,
thanks for your reply.
I have neither STRICT_TRAVERSE_CHECK in my environment nor in my buildout.cfg.

This is the traceback that occurs:

2021-08-18 10:54:03,256 ERROR   [Zope.SiteErrorLog:252][waitress-0] 1629276843.260.980793085333 http://localhost:6060/senaite/manage_main
Traceback (innermost last):
  Module ZPublisher.WSGIPublisher, line 162, in transaction_pubevents
  Module ZPublisher.WSGIPublisher, line 371, in publish_module
  Module ZPublisher.WSGIPublisher, line 274, in publish
  Module ZPublisher.mapply, line 85, in mapply
  Module ZPublisher.WSGIPublisher, line 63, in call_object
  Module Shared.DC.Scripts.Bindings, line 335, in __call__
  Module Shared.DC.Scripts.Bindings, line 372, in _bindAndExec
  Module Products.PageTemplates.PageTemplateFile, line 145, in _exec
  Module Products.PageTemplates.PageTemplate, line 85, in pt_render
  Module zope.pagetemplate.pagetemplate, line 135, in pt_render
  Module Products.PageTemplates.engine, line 378, in __call__
  Module z3c.pt.pagetemplate, line 176, in render
  Module chameleon.zpt.template, line 302, in render
  Module chameleon.template, line 215, in render
  Module chameleon.template, line 192, in render
  Module 152632436f412eb62a9d6f3bdd780600, line 238, in render
  Module zope.tales.expressions, line 250, in __call__
  Module Products.PageTemplates.Expressions, line 217, in _eval
  Module zope.tales.expressions, line 153, in _eval
  Module Products.CMFPlone.earlypatches.expressions, line 173, in boboAwareZopeTraverse
  Module Products.CMFPlone.earlypatches.expressions, line 112, in shared_traverse
  Module Products.CMFPlone.earlypatches.expressions, line 79, in guarded_import_module
NotFound: getSecurityManager

 - Expression: " modules/AccessControl/SecurityManagement/getSecurityManage"
 - Filename:   main
 - Location:   (line 9: col 22)
 - Arguments:  repeat: <Products.PageTemplates.engine.RepeatDictWrapper object at 0x11789c640>
               has_order_support: 0
               template: <PageTemplateFile at /senaite/main>
               modules: <Products.PageTemplates.ZRPythonExpr._SecureModuleImporter object at 0x10577db90>
               here: <PloneSite at /senaite>
               user: <PropertiedUser 'admin'>
               nothing: None
               target_language: None
               container: <PloneSite at /senaite>
               default: <DEFAULT>
               request: <WSGIRequest, URL=http://localhost:6060/senaite/manage_main>
               loop: {}
               context: <PloneSite at /senaite>
               translate: <function translate at 0x118337550>
               root: <Application at >
               options: {'args': ()}
               attrs: {u'action': u'string:${request/URL1}/', u'name': u'objectItems', u'method': u'post'}

No idea yet why it occurs...

11

The NotFound is raised on this line. But your case should have been caught by the check directly above it. You could try setting a breakpoint there to investigate.

The problem is explained in the lines above. The template OFS/zpt/main.zpt used to have an expressing modules/AccessControl/SecurityManagement/getSecurityManager which no longer works due to the stricter checks. It works if you replace it with modules/AccessControl/getSecurityManager. But as said, the code in CMFPlone 5.2.5 (and the hotfix) has a workaround. When I edit the page template locally to use the older code, it still works.

Maybe you have an override for this template? Or you have some variant of this code?
Search for templates that have AccessControl/SecurityManagement or SecurityManagement/getSecurityManager.

12

Hey Maurits, thanks for your reply and help!

However, it does not seem like I have this overwritten in the codebase and I can't find out any side effect causing it.

This is what I have at this codepoint:

base = <module 'AccessControl.SecurityManagement' from 'AccessControl-4.3-py2.7-macosx-11.0-x86_64.egg/AccessControl/SecurityManagement.pyc'>
path_items = ['SecurityManagement', 'getSecurityManager']
(Pdb++) name
'getSecurityManager'

still scratching my head about this...

13

Can you figure out which template this is? What I would do in the pdb is go up a few frames with u and then use a to see the args and kwargs of the method, and go further up when needed.

The main suggests it is indeed the main.zpt from OFS, which lives in the Zope package. But the Plone 5.2.5 release uses Zope 4.6.3, which has fixed this problem in the template. So that is suspicious.

Perhaps look in portal_view_customizations in the ZMI.

14

Hi @mauritsvanrees,

we found the reason for the behavior in our codebase.
There was an very old snippet that basically looked like this:

from AccessControl import allow_module
allow_module("AccessControl")

we removed all of this stale code and now it works again.

Thanks again for your support and the 5.2.5 release!