Out of memory when editing the title of large files

Plone Support
1

Hi, I am working with Plone 5 installed using a minimal buildout in a small machine, with less than 500 MB available for the Plone instance. I upload some videos. Some of them are small (around 40MB) and others are bigger (around 200MB). Regardless of the size, I have no problems when reproducing the videos. The problem is when trying to edit the big videos (i.e., changing the title or the description). I got the following error message:

Traceback (innermost last):
  Module ZPublisher.Publish, line 146, in publish
  Module Zope2.App.startup, line 303, in commit
  Module transaction._manager, line 131, in commit
  Module transaction._transaction, line 310, in commit
  Module transaction._transaction, line 301, in commit
  Module transaction._transaction, line 446, in _commitResources
  Module transaction._transaction, line 420, in _commitResources
  Module ZODB.Connection, line 498, in commit
  Module ZODB.Connection, line 547, in _commit
  Module ZODB.Connection, line 579, in _store_objects
  Module ZODB.serialize, line 419, in serialize
  Module ZODB.serialize, line 428, in _dump
MemoryError: out of memory

I guess that the issue is because the system is loading the video as part of the object when editing it. I found a post that supports my guess (https://sourceforge.net/p/plone/mailman/message/28161385/). Do you think that this is the source of the error message? If that were the case, I think that it is inefficient loading the blob in memory when only editing its metadata.

2

The "sourceforge" message reports a problem with CMFEditions. You did not tell us that you are using CMFEditions; in addition, the MemoryError from that message is related to a "%s" % obj while your MemoryError comes from the commit phase (i.e. significantly later). Thus, the "sourceforge" message (almost surely) is not related to your problem.

I fully agree with you here: modifying the metadata should do nothing with the blob unless you have CMFEditions (or similar versioning) active. For simplicity reasons, CMFEdition likely copies the entire object when it must create a new version -- potentially including a blob. If you have no versioning active, then you almost surely see a bug.

You could try to use Products.PDBDebugMode to analyse the problem. It activates the Python debugger in case of unhandled exceptions. In the debugger, you could check which object is about to be dumped. It is not sure that this approach will succeed: as Python has run out of memory, there might not be enough space left to run the debugger.

A quick look at the traceback and the associated code seems to indicate that the problem is not directly related to blobs (and their modification): instead, some "normal" persistent object seems to contain some unexpectedly huge (non persistent) data; maybe something like a preview or a cached transformation result.

Should you come back, it would be good to tell us precisely which Plone version you are using. I see from the traceback that you are not using the ZODB version coming with Plone 5.0.8.

1 Like
3

Hi Dieter,
Thanks for your answer.

The Plone version I am using is:

Plone 5.1.4 (5114)
CMF 2.2.12
Zope 2.13.27
Python 2.7.13 (default, Sep 26 2018, 18:42:22) [GCC 6.3.0 20170516]
PIL 5.2.0 (Pillow)

I have installed no extra package. The buildout I used is the following:

[buildout]
extends =
    http://dist.plone.org/release/5-latest/versions.cfg

parts =
    instance
[instance]
recipe = plone.recipe.zope2instance
user = admin:admin
http-address = 8080
eggs =
    Plone

I have not set versioning for files manually. In the site configuration, section Content Settings for type file, the attribute Versioning Policy is set to "No versioning".

I will try with Products.PDBDebugMode tomorrow.

4

I installed the egg Products.PDBDebugMode, and I run the site with bin/instance fg. As a test, I tried to edit three files of 71 MB, 167 MB and 222 MB, respectively. Editing the first two succeeded and the last one failed. I got the following error message in the terminal:

(Pdb) 2018-12-22 13:51:05 ERROR Zope.SiteErrorLog 1545486665.470.72723422401 http://.../mvi_9545.mov/@@edit
Traceback (innermost last):
  Module ZPublisher.Publish, line 138, in publish
  Module ZPublisher.mapply, line 77, in mapply
  Module Products.PDBDebugMode.runcall, line 70, in pdb_runcall
  Module ZPublisher.Publish, line 48, in call_object
  Module plone.z3cform.layout, line 63, in __call__
  Module plone.z3cform.layout, line 57, in update
  Module z3c.form.form, line 162, in render
  Module zope.browserpage.viewpagetemplatefile, line 49, in __call__
  Module zope.pagetemplate.pagetemplate, line 137, in pt_render
  Module five.pt.engine, line 98, in __call__
  Module z3c.pt.pagetemplate, line 163, in render
  Module chameleon.zpt.template, line 261, in render
  Module chameleon.template, line 171, in render
  Module f1a0fe8767bf74ad50c025a5fa1ebb5d.py, line 91, in render
  Module fba930545fa874cdb48971ba9dd03567.py, line 1826, in render_titlelessform
  Module fba930545fa874cdb48971ba9dd03567.py, line 451, in render_fields
  Module fba930545fa874cdb48971ba9dd03567.py, line 126, in render_widget_rendering
  Module fba930545fa874cdb48971ba9dd03567.py, line 1069, in render_field
  Module five.pt.expressions, line 161, in __call__
  Module Products.Five.browser.metaconfigure, line 485, in __call__
  Module zope.browserpage.viewpagetemplatefile, line 81, in __call__
  Module zope.browserpage.viewpagetemplatefile, line 49, in __call__
  Module zope.pagetemplate.pagetemplate, line 137, in pt_render
  Module five.pt.engine, line 98, in __call__
  Module z3c.pt.pagetemplate, line 163, in render
  Module chameleon.zpt.template, line 261, in render
  Module chameleon.template, line 171, in render
  Module 922b59e08dc6da377e8ec57f51e8c785.py, line 610, in render
  Module 922b59e08dc6da377e8ec57f51e8c785.py, line 481, in render_widget_wrapper
  Module five.pt.expressions, line 161, in __call__
  Module z3c.form.widget, line 154, in render
  Module zope.browserpage.viewpagetemplatefile, line 49, in __call__
  Module zope.pagetemplate.pagetemplate, line 137, in pt_render
  Module five.pt.engine, line 98, in __call__
  Module z3c.pt.pagetemplate, line 163, in render
  Module chameleon.zpt.template, line 261, in render
  Module chameleon.template, line 191, in render
  Module chameleon.template, line 171, in render
  Module 0ad07260cc1a19be8530babe5e3e48f1.py, line 442, in render
  Module five.pt.expressions, line 154, in __call__
  Module five.pt.expressions, line 126, in traverse
  Module zope.traversing.adapters, line 142, in traversePathElement
   - __traceback_info__: (<NamedFileWidget 'form.widgets.file'>, 'file_upload_id')
  Module zope.traversing.adapters, line 42, in traverse
   - __traceback_info__: (<NamedFileWidget 'form.widgets.file'>, 'file_upload_id', ())
  Module plone.formwidget.namedfile.widget, line 110, in file_upload_id
  Module plone.namedfile.file, line 348, in _getData
MemoryError: MemoryError

 - Expression: "widget/@@ploneform-render-widget"
 - Filename:   ... rm-3.0.6-py2.7.egg/plone/app/z3cform/templates/macros.pt
 - Location:   (line 100: col 81)
 - Source:     ... place="structure widget/@@ploneform-render-widget"/>
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 - Expression: "view/file_upload_id"
 - Filename:   ... 2.0.6-py2.7.egg/plone/formwidget/namedfile/file_input.pt
 - Location:   (line 31: col 30)
 - Source:     ... if tal:define="up_id view/file_upload_id" tal:condition="up_ ...
                                        ^^^^^^^^^^^^^^^^^^^
 - Arguments:  repeat: {...} (0)
               context: <ImplicitAcquisitionWrapper mvi_9545.mov at 0x7efc8410bb90>
               exists: True
               views: <ViewMapper - at 0x7efc81348590>
               modules: <TraversableModuleImporter - at 0x7efc89e27f90>
               args: <tuple - at 0x7efc91fa6050>
               nothing: <NoneType - at 0x55b0f9fd74d0>
               icon: http://.../video.png
               doc_type: QuickTime video
               allow_nochange: True
               target_language: <NoneType - at 0x55b0f9fd74d0>
               default: <object - at 0x7efc91ec3540>
               request: <instance - at 0x7efc810995a8>
               wrapped_repeat: {...} (0)
               download_url: http://...
               filename: MVI_9545.MOV
               loop: {...} (0)
               template: <ViewPageTemplateFile - at 0x7efc842e5990>
               action: nochange
               translate: <function translate at 0x7efc801a7938>
               options: {...} (0)
               view: <NamedFileWidget file at 0x7efc814aa950>
> /.../plone5/eggs/plone.namedfile-4.2.5-py2.7.egg/plone/namedfile/file.py(348)_getData()
-> data = fp.read()
5

If you monitor your system memory or swap with top can you tell if it’s happening because you ran out of system memory or swap? Otherwise I would look at the source where the error occurred to see if there is a way to prevent it if, say, the file size is larger than a certain amount.

6

I will do a report with the memory usage of the instance when loading the edit view. I think that there is a problem with the way the object is loaded into memory when displaying the edit interface. It results extrange that Plone can stream the vĂ­deo, but cannot edit the title, even when the edition interface does not require showing the vĂ­deo as the view of the object do.

7

Clearly there is a bug involved.

Can you try renaming the video via the ZMI?

8

I have no problems to edit the file properties if using the ZMI interface.

9

This tells you that the problem comes from the frontend: while you only want to edit the metadata, the frontend prepares for a complete edit -- including the main (file) content.

You might looks at the code of file_upload_id and determine why it wants to get the file data. Obviously, it is not a good idea to read a huge amount of data to prepare for a manual edit. Maybe, you see ways to avoid the data getting for cases like yours.