I'm planning a project which manages access control to information based on folder structure and local permissions. For performance reasons I'm considering souper.plone. Just putting my main "research" considerations out there, in case someone already knows the answers off their head.
If I use souper.plone to create lightweight records instead of using dexterity:
Are souper.plone records aware of local roles and permissions? (my guess is that soups are aware NOT individual soup records)
Can I organise souper records hierarchically? (my guess is that I'll have to organise the hierarchy at the soup level NOT the individual soup record)
Use case - replace dexterity content with souper.plone records
Let's say this is a school and the information to be stored in souper.plone records is student information.
There are three Groups types:
School Level (I can see all information) e.g Main School Group
Grade Level ( I can see all information for a grade) - eg. Grade 2 Group
Class Level (I can see all information for a class) - eg. Class 2-1 of Grade 2
Using a pure dexterity approach, I would create "student" content types and then place them in their various classes. Permissions would be managed based on a folder hierarchy and local sharing permissions. The folder structure would end up looking like this (Grade 2, Class 2-1 expanded for illustration):
no souper records aren't heirachical or have per record security. They also aren't searchable in the main plone catalog. The whole idea is they are independently indexed table of data.
Sounds like you want dexterity rather than souper.
A souper model you likely have one soup for students and one for class and then store ids into them linking them.
A lot also depends on how you want to query this data.
I like folder hierarchy and local sharing permissions and already have my head wrapped around that. For my use case performance issues may force my hand.
It's thousands of records not dozens of students
Importing dexterity content is super slow compared to rows in a relational db (or souper.plone I assume)
If I can't improve the speed of importing dexterity (my major source of pain at the moment) I'll have to sacrifice dexterity. No dexterity also seems to mean reworking the security approach and architecture. At some point I wonder if I should just look at a traditional relational db (thinking aloud)....
Good question, since your data does not exist in Plone (you generate a view based on the SQL query) I am not sure that you would be able to use local permissions. Maybe store your permission attributes in your source data and use a filter before returning it to Plone? Depends on how complicated, extensible etc. your permissions model is.
We use it to generate a view of product information which is stored in an external legacy system. Have dabbled a bit with edit forms and looks promising (planning to work on that feature for the next major update of our system).
Helps if you state your problem not solution up front.
If your problem is import speed then I'd be looking at ways to make indexing more efficient. I think there has been much written about that and how much affect that has on import speed. Obviously not using restapi and having a more direct method to import is going to help too.
Souper import I'm not sure is going to be super fast but I don't know. @ebrehault is the person to ask. He imported wikipedia into souper.
As far as I know souper still writes its indexes at teh same time you import a record. Look at solutions like elastic or solr to see if you can delay indexing and do it external to plone?
I assume you are breaking it down into seperate transaction (not just subtransactions). That will reduce your ram. Using subtransactions you have ot be very careful you aren't keep references to items between subtransactions as then GC won't do anything.
@pigeonflight another idea (I've no idea if someone tried or not before) is to use zopes low level copy functionality if your objects are largely the same then just update any values that have changed. That could reduce the amount of code run to create each dexterity object a lot?
If you also disable zope events for copy then helps since there is a bunch of code hooked into that which does nothing in the end (like do you really need content rules and redirectiontool evaluation stuff during inport). We had a plugin somewhere that disabled zope events during zxp imports and it sped things up a lot.