Hi,
We are using Plone 3.X version and created multiple courses in our application. Couple of courses actually having large number of media resources. Previously course was taking 30-40 minutes to save these courses but not it stop saving them. There are around 4500 objects (total) in this course now.
Can somebody suggest any possible solution to resolve this issue? Any solution to skip Media Resource while saving course metadata only?
Thanks
Media resources can be large. Thus, it can take some time to save them (though 30 to 40 minutes seems to be really much).
On the other hand, your description is not very clear to me. Why do you want to save all of your courses? I would expect that you save a course at a time (or even save the individual course components).
Hearing about the huge time (in the order of 1/2 hour), I get the feeling that you are doing something wrong. One possible problem could be that your transaction becomes too huge (should you commit all your thousands of media objects in a single transaction): without special interventions (--> transaction.savepoint), Zope keeps all modified objects of a transaction in memory which can lead to heavy swapping (and huge a runtime).
In order to help you effectively, we must better understand your application and the problem it involves. You could use a profiler (e.g. Products.ZopeProfiler) to find out where the time is spent for (you will need to try with a course which still allows saving); you could use one of the ZODB related tools to find out about the size of your transactions (on approach could be looking at the ZMI: --> Control_Panel --> "Datebase Management" --> "main" --> "Activity").
Could it be that the folders in your version of Plone are not b-tree folders? The more things you put in non-b-tree folders, the slower they'll get (exponentially).
Also you need to check your system's disk and other activity (e.g. memory, file system space left).
How large are your media files? Is there any indexing involved (unlikely)?
Pointer to BTree folders is a good pointer. Do you see some ZODB conflict errors in your event.log?
Are you using blob storage? In general it is a pretty bad idea storing large files in Plone directly although it is working to some degree. The general approach is to storage large files like videos somewhere else....what large means depends on various factors...in generall it is a bad idea to use Plone as multimedia server.
Hi Dieter,
Thanks for quick reply. Here is my current structure on content:
Course
|--->Sections
|----------------->Section
|------------------------------>Media Resource (100s of media files in each section)
When we try to updated Course (Single course one time) metadata it takes 30-40 minutes but when we update Section metadata or media resource metadata it just take 2-3 minutes. But when we copy this course and deleted big number of media resources then course metadata is getting save in 3-4 minutes.
Zope client logs just showing that validators (custom) are getting called multiple time when we have large number of media resources but in other case validators runs only once.
Not able to understand the root cause for that. Control_Panel is available in Zope 2.x? We are not able to find this.
Thanks
Hi,
Sorry but how can we check that if this is B-Tree structure? We have sharing structure in previous comment please check and let us know it there is any problem with structure and possible solution on live site structure.
Thanks
Hi,
Yes everything (memory, space) is fine as this is production environment.
Thanks.
Hi,
Yes, we are using blob storage. Not saving anything in Plone site itself.
Thanks.
In Plone 3 there were Folders and Large Folders. The Large Folders were implemented using b-trees.
Your sharing settings probably don't have anything to do with the performance problem.
Control_Panel is available in Zope 2. In may be hidden by some special Plone configuration. In those case, you usually can access it (using Acquisition) via http://your host/Control_Panel/manage_main (of course, you must replace your host by the real host, potentially together with the Zope port; in my environment, the url is http://localhost:8080/Control_Panel/manage_main).
Times in the order of 1/2 hour indicate that something is severely wrong (several minutes, as well, seem too high for a metadata update).
We still know very little about your application: are your "course"s, "section"s and "resource"s standard Plone objects (or do they use custom content types)? With standard Plone objects, the time of a metadata update on a course object should not be significantly affected by the number of objects below the object (apart from the number of "sections", in case, the "course" object derives from a "normal" rather than a "large" folder).
Hi,
Yes, These are using custom content types in our application. But only those few courses are having issue which have more number of media resources or other objects otherwise everything is working fine.
Is there anything related to Acquisition behavior of Plone? But why validators are getting called multiple times in these courses only? No clue till now...
Thanks
It's a Folder vs Large Folder issue. If you share with us more code about your application we can try to help more.
I am not convinced that this is the real cause. The difference between a Folder and a Large Folder is that a Folder maintains its content items in a non persistent attribute while a Large Folder uses a persistent tree with persistent subcomponents. As a consequence, loading a Folder from the ZODB creates ghost (!) objects for all content items (but does not load the content items themselves); writing a Folder to the ZODB writes persistent references for each content items (but does not write the content items themselves). This means, that a Large Folder is more efficient than a Folder if large numbers of content items are involved -- but nevertheless, the Folder handling should not use minutes.
Still waiting for solution !!
Thanks
What analysis steps have you performed?
Have you looked with a profiler (e.g. Products.ZopeProfiler) what the time is spent for?
Have you looked how many persistent objects are written (--> Control_Panel --> "Database Management" --> "main" --> "Activity") (and maybe also read)?
Have you checked the size of your transactions (--> fsdump)?
Keep in mind that it is your installation that has a problem and that we cannot look into your installation. Therefore, it is you who must do the analysis; we can only give some hints.
You haven't let us see any of the code or buildout so we cannot help you further.
You should also know that the way you phrased your response is far from ideal: we are not paid to help you. This is a volunteer organization. Here's how to ask for help: How to Ask for Help