A Plone zip/unzip package to share

Hustle and Bustle
1

I've been working with Plone for over 10 years and recently started moving some of our general tools from an internal Gitlab into public Github (and pypi). Here's the first of these: ims.zip · PyPI source: GitHub - imsweb/ims.zip at master. This package may or may not be useful to others but I thought I'd share :slight_smile:

This package contains the ability to zip and unzip files using the python zip library. Unzipping is probably the most common - unpack a zip file into a Plone site and maintain the folder structure of the zip folder. All content is uploaded as Files or Images using the Plone content mimetype registry.

Zipping is also supported though arguably less common. Our use case is for sites that have a set shelf life - generally the site is a collaborative extranet for a government grant research study. Several Dexterity content types are supported and it is possible to create adapters for others. Zipping can be an expensive process if zipping large content so take advantage of the included custom permissions and do not assign them casually. For our use, only the Manager is allowed to zip. The process is not completely lossless and is not meant as a way to port site functionality to another platform. It's only intended to be a means of extracting most content and does not include metadata like permissions or relations.

Zipping VERY large content is possible with the zip64 format. The default zip format has a cap of 4GB, but the 64 bit version allows it to be bigger. ims.zip offers different behavior for each. We use the uncompressed total size as an estimate and if this is under 4GB we create the zip and return it as the response in one request. If greater than 4GB, the request is sent as an AJAX request. The resulting zip file is uploaded to the site and the requester is emailed.

4 Likes
2

Which version of Plone did you test with?
(I tried on an old 5.1.5 site without luck)

3

We're using it on 5.2 but I don't think there's anything particular unique about any of the 5.x releases that it wouldn't work with. Got a traceback?

4

I have not done any debugging, yet

2022-02-23 11:48:15 INFO Products.PloneFormGen gpg_subprocess initialized, using /usr/bin/gpg
Traceback (most recent call last):
  File "/home/medialog/vol2/medialog.dutchtheme/parts/instance/bin/interpreter", line 355, in <module>
    exec(compile(__file__f.read(), __file__, "exec"))
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/Zope2-2.13.27-py2.7.egg/Zope2/Startup/run.py", line 76, in <module>
    run()
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/Zope2-2.13.27-py2.7.egg/Zope2/Startup/run.py", line 22, in run
    starter.prepare()
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/Zope2-2.13.27-py2.7.egg/Zope2/Startup/__init__.py", line 92, in prepare
    self.startZope()
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/Zope2-2.13.27-py2.7.egg/Zope2/Startup/__init__.py", line 268, in startZope
    Zope2.startup()
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/Zope2-2.13.27-py2.7.egg/Zope2/__init__.py", line 47, in startup
    _startup()
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/Zope2-2.13.27-py2.7.egg/Zope2/App/startup.py", line 120, in startup
    load_zcml()
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/Zope2-2.13.27-py2.7.egg/Zope2/App/startup.py", line 52, in load_zcml
    load_site()
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/Zope2-2.13.27-py2.7.egg/Zope2/App/zcml.py", line 46, in load_site
    _context = xmlconfig.file(site_zcml)
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/xmlconfig.py", line 649, in file
    include(context, name, package)
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/xmlconfig.py", line 548, in include
    processxmlfile(f, context)
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/xmlconfig.py", line 380, in processxmlfile
    parser.parse(src)
  File "/usr/lib/python2.7/xml/sax/expatreader.py", line 110, in parse
    xmlreader.IncrementalParser.parse(self, source)
  File "/usr/lib/python2.7/xml/sax/xmlreader.py", line 123, in parse
    self.feed(buffer)
  File "/usr/lib/python2.7/xml/sax/expatreader.py", line 213, in feed
    self._parser.Parse(data, isFinal)
  File "/usr/lib/python2.7/xml/sax/expatreader.py", line 365, in end_element_ns
    self._cont_handler.endElementNS(pair, None)
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/xmlconfig.py", line 359, in endElementNS
    self.context.end()
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/config.py", line 558, in end
    self.stack.pop().finish()
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/config.py", line 706, in finish
    actions = self.handler(context, **args)
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/Zope2-2.13.27-py2.7.egg/OFS/metaconfigure.py", line 46, in loadProducts
    xmlconfig.include(_context, zcml, package=product)
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/xmlconfig.py", line 548, in include
    processxmlfile(f, context)
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/xmlconfig.py", line 380, in processxmlfile
    parser.parse(src)
  File "/usr/lib/python2.7/xml/sax/expatreader.py", line 110, in parse
    xmlreader.IncrementalParser.parse(self, source)
  File "/usr/lib/python2.7/xml/sax/xmlreader.py", line 123, in parse
    self.feed(buffer)
  File "/usr/lib/python2.7/xml/sax/expatreader.py", line 213, in feed
    self._parser.Parse(data, isFinal)
  File "/usr/lib/python2.7/xml/sax/expatreader.py", line 365, in end_element_ns
    self._cont_handler.endElementNS(pair, None)
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/xmlconfig.py", line 359, in endElementNS
    self.context.end()
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/config.py", line 558, in end
    self.stack.pop().finish()
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/config.py", line 706, in finish
    actions = self.handler(context, **args)
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/xmlconfig.py", line 548, in include
    processxmlfile(f, context)
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/xmlconfig.py", line 380, in processxmlfile
    parser.parse(src)
  File "/usr/lib/python2.7/xml/sax/expatreader.py", line 110, in parse
    xmlreader.IncrementalParser.parse(self, source)
  File "/usr/lib/python2.7/xml/sax/xmlreader.py", line 123, in parse
    self.feed(buffer)
  File "/usr/lib/python2.7/xml/sax/expatreader.py", line 213, in feed
    self._parser.Parse(data, isFinal)
  File "/usr/lib/python2.7/xml/sax/expatreader.py", line 365, in end_element_ns
    self._cont_handler.endElementNS(pair, None)
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/xmlconfig.py", line 359, in endElementNS
    self.context.end()
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/config.py", line 558, in end
    self.stack.pop().finish()
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/config.py", line 706, in finish
    actions = self.handler(context, **args)
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/xmlconfig.py", line 548, in include
    processxmlfile(f, context)
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/xmlconfig.py", line 380, in processxmlfile
    parser.parse(src)
  File "/usr/lib/python2.7/xml/sax/expatreader.py", line 110, in parse
    xmlreader.IncrementalParser.parse(self, source)
  File "/usr/lib/python2.7/xml/sax/xmlreader.py", line 123, in parse
    self.feed(buffer)
  File "/usr/lib/python2.7/xml/sax/expatreader.py", line 213, in feed
    self._parser.Parse(data, isFinal)
  File "/usr/lib/python2.7/xml/sax/expatreader.py", line 365, in end_element_ns
    self._cont_handler.endElementNS(pair, None)
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/xmlconfig.py", line 359, in endElementNS
    self.context.end()
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/config.py", line 558, in end
    self.stack.pop().finish()
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/config.py", line 706, in finish
    actions = self.handler(context, **args)
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/xmlconfig.py", line 548, in include
    processxmlfile(f, context)
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/xmlconfig.py", line 380, in processxmlfile
    parser.parse(src)
  File "/usr/lib/python2.7/xml/sax/expatreader.py", line 110, in parse
    xmlreader.IncrementalParser.parse(self, source)
  File "/usr/lib/python2.7/xml/sax/xmlreader.py", line 123, in parse
    self.feed(buffer)
  File "/usr/lib/python2.7/xml/sax/expatreader.py", line 213, in feed
    self._parser.Parse(data, isFinal)
  File "/usr/lib/python2.7/xml/sax/expatreader.py", line 365, in end_element_ns
    self._cont_handler.endElementNS(pair, None)
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/xmlconfig.py", line 359, in endElementNS
    self.context.end()
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/config.py", line 558, in end
    self.stack.pop().finish()
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/config.py", line 706, in finish
    actions = self.handler(context, **args)
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/xmlconfig.py", line 548, in include
    processxmlfile(f, context)
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/xmlconfig.py", line 380, in processxmlfile
    parser.parse(src)
  File "/usr/lib/python2.7/xml/sax/expatreader.py", line 110, in parse
    xmlreader.IncrementalParser.parse(self, source)
  File "/usr/lib/python2.7/xml/sax/xmlreader.py", line 123, in parse
    self.feed(buffer)
  File "/usr/lib/python2.7/xml/sax/expatreader.py", line 213, in feed
    self._parser.Parse(data, isFinal)
  File "/usr/lib/python2.7/xml/sax/expatreader.py", line 365, in end_element_ns
    self._cont_handler.endElementNS(pair, None)
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/xmlconfig.py", line 359, in endElementNS
    self.context.end()
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/config.py", line 558, in end
    self.stack.pop().finish()
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/config.py", line 706, in finish
    actions = self.handler(context, **args)
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/xmlconfig.py", line 548, in include
    processxmlfile(f, context)
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/xmlconfig.py", line 380, in processxmlfile
    parser.parse(src)
  File "/usr/lib/python2.7/xml/sax/expatreader.py", line 110, in parse
    xmlreader.IncrementalParser.parse(self, source)
  File "/usr/lib/python2.7/xml/sax/xmlreader.py", line 123, in parse
    self.feed(buffer)
  File "/usr/lib/python2.7/xml/sax/expatreader.py", line 213, in feed
    self._parser.Parse(data, isFinal)
  File "/usr/lib/python2.7/xml/sax/expatreader.py", line 365, in end_element_ns
    self._cont_handler.endElementNS(pair, None)
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/xmlconfig.py", line 359, in endElementNS
    self.context.end()
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/config.py", line 558, in end
    self.stack.pop().finish()
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/config.py", line 706, in finish
    actions = self.handler(context, **args)
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/xmlconfig.py", line 548, in include
    processxmlfile(f, context)
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/xmlconfig.py", line 380, in processxmlfile
    parser.parse(src)
  File "/usr/lib/python2.7/xml/sax/expatreader.py", line 110, in parse
    xmlreader.IncrementalParser.parse(self, source)
  File "/usr/lib/python2.7/xml/sax/xmlreader.py", line 123, in parse
    self.feed(buffer)
  File "/usr/lib/python2.7/xml/sax/expatreader.py", line 213, in feed
    self._parser.Parse(data, isFinal)
  File "/usr/lib/python2.7/xml/sax/expatreader.py", line 365, in end_element_ns
    self._cont_handler.endElementNS(pair, None)
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/xmlconfig.py", line 359, in endElementNS
    self.context.end()
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/config.py", line 558, in end
    self.stack.pop().finish()
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/config.py", line 706, in finish
    actions = self.handler(context, **args)
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/xmlconfig.py", line 548, in include
    processxmlfile(f, context)
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/xmlconfig.py", line 380, in processxmlfile
    parser.parse(src)
  File "/usr/lib/python2.7/xml/sax/expatreader.py", line 110, in parse
    xmlreader.IncrementalParser.parse(self, source)
  File "/usr/lib/python2.7/xml/sax/xmlreader.py", line 123, in parse
    self.feed(buffer)
  File "/usr/lib/python2.7/xml/sax/expatreader.py", line 213, in feed
    self._parser.Parse(data, isFinal)
  File "/usr/lib/python2.7/xml/sax/expatreader.py", line 365, in end_element_ns
    self._cont_handler.endElementNS(pair, None)
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/xmlconfig.py", line 359, in endElementNS
    self.context.end()
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/config.py", line 558, in end
    self.stack.pop().finish()
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/config.py", line 706, in finish
    actions = self.handler(context, **args)
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/z3c.autoinclude-0.3.8-py2.7.egg/z3c/autoinclude/zcml.py", line 104, in includePluginsDirective
    includeZCMLGroup(_context, info, filename)
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/z3c.autoinclude-0.3.8-py2.7.egg/z3c/autoinclude/zcml.py", line 30, in includeZCMLGroup
    include(_context, filename, includable_package)
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/xmlconfig.py", line 548, in include
    processxmlfile(f, context)
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/xmlconfig.py", line 380, in processxmlfile
    parser.parse(src)
  File "/usr/lib/python2.7/xml/sax/expatreader.py", line 110, in parse
    xmlreader.IncrementalParser.parse(self, source)
  File "/usr/lib/python2.7/xml/sax/xmlreader.py", line 123, in parse
    self.feed(buffer)
  File "/usr/lib/python2.7/xml/sax/expatreader.py", line 213, in feed
    self._parser.Parse(data, isFinal)
  File "/usr/lib/python2.7/xml/sax/expatreader.py", line 365, in end_element_ns
    self._cont_handler.endElementNS(pair, None)
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/xmlconfig.py", line 359, in endElementNS
    self.context.end()
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/config.py", line 558, in end
    self.stack.pop().finish()
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/config.py", line 706, in finish
    actions = self.handler(context, **args)
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/xmlconfig.py", line 548, in include
    processxmlfile(f, context)
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/xmlconfig.py", line 380, in processxmlfile
    parser.parse(src)
  File "/usr/lib/python2.7/xml/sax/expatreader.py", line 110, in parse
    xmlreader.IncrementalParser.parse(self, source)
  File "/usr/lib/python2.7/xml/sax/xmlreader.py", line 123, in parse
    self.feed(buffer)
  File "/usr/lib/python2.7/xml/sax/expatreader.py", line 213, in feed
    self._parser.Parse(data, isFinal)
  File "/usr/lib/python2.7/xml/sax/expatreader.py", line 365, in end_element_ns
    self._cont_handler.endElementNS(pair, None)
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/xmlconfig.py", line 359, in endElementNS
    self.context.end()
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/config.py", line 558, in end
    self.stack.pop().finish()
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/config.py", line 705, in finish
    args = toargs(context, *self.argdata)
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/config.py", line 1397, in toargs
    args[str(name)] = field.fromUnicode(s)
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/fields.py", line 137, in fromUnicode
    value = self.context.resolve(name)
  File "/home/medialog/vol2/medialog.dutchtheme/eggs/zope.configuration-3.7.4-py2.7.egg/zope/configuration/config.py", line 179, in resolve
    mod = __import__(mname, *_import_chickens)
zope.configuration.xmlconfig.ZopeXMLConfigurationError: File "/home/medialog/vol2/medialog.dutchtheme/parts/instance/etc/site.zcml", line 16.2-16.23
    ZopeXMLConfigurationError: File "/home/medialog/vol2/medialog.dutchtheme/eggs/Products.ATContentTypes-2.3.9-py2.7.egg/Products/ATContentTypes/configure.zcml", line 18.2-18.44
    ZopeXMLConfigurationError: File "/home/medialog/vol2/medialog.dutchtheme/eggs/plone.app.collection-1.2.7-py2.7.egg/plone/app/collection/configure.zcml", line 9.2-9.45
    ZopeXMLConfigurationError: File "/home/medialog/vol2/medialog.dutchtheme/eggs/plone.app.querystring-1.4.11-py2.7.egg/plone/app/querystring/configure.zcml", line 11.2-11.42
    ZopeXMLConfigurationError: File "/home/medialog/vol2/medialog.dutchtheme/eggs/plone.app.registry-1.7.2-py2.7.egg/plone/app/registry/configure.zcml", line 12.4-12.34
    ZopeXMLConfigurationError: File "/home/medialog/vol2/medialog.dutchtheme/eggs/plone.app.registry-1.7.2-py2.7.egg/plone/app/registry/browser/configure.zcml", line 6.4-6.43
    ZopeXMLConfigurationError: File "/home/medialog/vol2/medialog.dutchtheme/eggs/plone.app.z3cform-3.0.8-py2.7.egg/plone/app/z3cform/configure.zcml", line 10.2-10.41
    ZopeXMLConfigurationError: File "/home/medialog/vol2/medialog.dutchtheme/eggs/plone.app.widgets-2.4.1-py2.7.egg/plone/app/widgets/configure.zcml", line 12.2-12.41
    ZopeXMLConfigurationError: File "/home/medialog/vol2/medialog.dutchtheme/eggs/Products.CMFPlone-5.1.5-py2.7.egg/Products/CMFPlone/configure.zcml", line 105.2-109.8
    ZopeXMLConfigurationError: File "/home/medialog/vol2/medialog.dutchtheme/eggs/ims.zip-4.2.1-py2.7.egg/ims/zip/configure.zcml", line 16.4-16.33
    ZopeXMLConfigurationError: File "/home/medialog/vol2/medialog.dutchtheme/eggs/ims.zip-4.2.1-py2.7.egg/ims/zip/browser/configure.zcml", line 13.4-18.39
5

I can't reproduce this but it looks like that error traceback may be incomplete. I think that error type may have an accompanying secondary error?