BTreeFolder2 questions

Plone Support
1

Hello,

From what I have read, I have seemed to understand that the folders (root and all others) of Plone 5.1 and later are BTreeFolder2. Correct?

If I have tens of thousands of items in a BTreeFolder2 folder and I want to change the order of a single item (for example, so that a new item appears second instead of last), can I easily do it without dragging it?

If I want to make a directory website with unlimited people:
mysite.es/person1
mysite.es/person2
mysite.es/person3
...
what would be better? Host all people in Plone root folder or host them in a subfolder and use rewrite rules?

Thanks,

Manuel

2

In my newly installed Plone 5.2, none of the normal folders are BTreeFolder2. I dont remember about 5.1, but I think it should be the same.

I am not sure what is the right approach here, but basically I would say this.

It is not a good idea to have a lot of content in the root folder. It is better to keep them in folders.

If you dont want to drag items, I would suggest using a smart folder ( what used to be called 'Collection') to show the items. Then you could add some logic for: 'in which order to view it'. For example sort on 'Modified' or Published. If you modify an item it will 'show first'. Thern you modify another and the new will show first and this second.

If you have a lot of items, I would suggest using collective.collectionfilter or something similar to make it easy for the visitor to sort. It can work like this: - YouTube

Personally, I dont like to overpopulate the root folder (but it is a 'personal' , not a technical answer), so I would put person1 etc in 'persons' folder. That said: would it matter if the urls were mysite.es/persons/person3 ?

3

Likely, almost folders in Plone implement IOrderedContainer.

So you can use

ordered_folder = IOrderedContainer(obj,aq_parent)
ordered_folder.moveObjectToPosition(obj.getId(), position)
obj.reindexObject(idxs=['getObjPositionInParent'])

for moving content programmatically.

But usually you have some criteria like sort by title, sort by some date etc. for generating listings and overview pages...that's where you would use the portal_catalog for searching and sorting content.

4

Thanks, Espen.

My newly installed Plone 5.1.6 site includes the following egg by default: Products.BTreeFolder2-2.14.0. Do I have to do something to activate it?

According to Products.BTreeFolder2 · PyPI "BTreeFolder2 solves this issue. It stores not only the subobjects but also the IDs of the subobjects in a BTree. It also batches the list of items in the UI, showing only 1000 items at a time. So if you write your application carefully, you can use a BTreeFolder2 to store as many items as will fit in physical storage.".

I will try to import or programmatically create 100,000 items in the root folder and in a subfolder, to see what happens (I will stay away from the laptop in case it explodes ...).

5 
ordered_folder = IOrderedContainer(obj,aq_parent)
ordered_folder.moveObjectToPosition(obj.getId(), position)
obj.reindexObject(idxs=['getObjPositionInParent'])

Noted (I'll test it).

Thanks!

6

BTreeFolder2 is used internally by Plone to provide support for large folders. It is not used directly (and you should not do it) because it lacks Plone content features. In particular, it has no ordering support.

7

Someone that is more used to very large sites should answer, since I am not sure if I know enough about this.

That said, unless there is a very good reason to stick with Plone 5.1, I would upgrade to 5.2 and then put all the 'persons folders' in the Members folder and maybe duplicate the Members folder if you need another very large folder (since the members folder is 'supposed to have many items'. You should change the settings in the control panel (/@@usergroup-controlpanel ) and enable 'many users'. And maybe activate 'members folder' in /@@security-controlpanel if you need that:

PS: I got the info about 'persons folder' in another message, it was not in this 'question'.

8

I have created a dexterity content type (Person) and just imported 100,000 items of this type into the root of a Plone 5.2 (5.5 hours with transmogrify.dexterity).

No problem, everything is perfect (although I will have to create a private folder with links to the different ZMI folders, because I imagine that with so much element in the root of the site it will be difficult to use /manage_main ...).

Another day I will try with a million people, to see how it responds ...

Thanks and hugs,

Manuel

9

Yes, as I imagined, /manage_main with more than 100,000 items is not manageable (but it is solved by creating links to /portal_catalog, /portal_setup, /portal_skin, etc. in the new private /zmi folder).

I've already started Importing another 900,000 items with transmogrify.dexterity.

Until now, everything is fine ...

\ :wink:

Hugs

10

The 900,000 items import failed (I have not investigated the cause).

Importing 100,000 more items in foreground mode did work, but:

1.- Immediately afterwards, I received the following warning (there are no images, files or media in the current 200,000 items, only title, description and 9 url fields):

/home/plone/52/buildout-cache/eggs/ZODB-5.6.0-py2.7.egg/ZODB/Connection.py:580: UserWarning: The <class 'Products.CMFPlone.Portal.PloneSite'> object you're saving is large. (19859973 bytes.)
Perhaps you're storing media which should be stored in blobs.
Perhaps you're using a non-scalable data structure, such as a PersistentMapping or PersistentList.
Perhaps you're storing data in objects that aren't persistent at all. In cases like that, the data is stored in the record of the containing persistent object.
In any case, storing records this big is probably a bad idea.
If you insist and want to get rid of this warning, use the large_record_size option of the ZODB.DB constructor (or the large-record-size option in a configuration file) to specify a larger size.
warnings.warn(large_object_message % (obj.class, len(p)))

Do I need to change something?

2.- Now, deleting a user takes 14 minutes (very uncomfortable and inefficient ...).

Does anyone know why this happens and if it could be solved?

Thanks.

I'll keep testing ...

Hugs,

Manuel

11

A guess, but could it be that the 'whole folder / its content gets reindexed every time you delete something.

Slightly related: Membrane and/or similar products? used to have an option for using a custom content type for 'Member' ( I have not used it for many, many years, but think it is still OK) . If you also set 'many users' in the control panel, maybe that is an option ?

https://docs.plone.org/develop/plone/members/membrane.html

1 Like
12

The portal object has no special features to handle a large number of content objects. The warning is a direct result.

13

Thanks to both of you: I'll try importing the 200,000 items into the Members folder.

Anyway I have seen this in /portal_membership/manage_mapRoles: "The members folder has to be in the same container as the membership tool" ( and on this site each Person element will be the user folder of each member ... ).

Hugs

14

Check the link I sent: It states that 'members can be in different folders'. That sounds like a good idea with so many (?)

15

Thanks, Espen. A few months ago I tried Membrane and related. If necessary, I will use them. Although I value simplicity (the fewer add-ons, the better - less complexity -). Hugs, Manuel

16

I just created 100,000 users in less than an hour with atreal.usersinout 1.2.3 on Plone 5.2 (I was impressed with the speed ...).

Now, I will create their personal folders in /Members with transmogrify.dexterity (it should take 5.5 hours ...).

Hugs

17

With 100,000 users:

1.- By default, it takes 3.5 minutes to load the users page. But after activating the "Too many users" option, it does it like lightning.

2.- The modal window of manual creation of a new user is frozen (it does not close), but the new user is created quickly and can log in at normal speed.

3.- The manual edition of users also works quickly.

4.- The search and the removal of users takes 4-5 seconds.

So far pretty good (taking into account that there are 100,000 users -we'll see what happens when I've created a million ...-).

I'm going to create the 100,000 custom dexterity personal folders ...

Hugs

18

Done: now, there are 100,000 users and 100,000 dexterity persons.

Everything is fine except the removal of users: it takes 7,5 minutes and slows the first search and first collection load during the 7.5 minute user deletion.

Is there a way to fix this?

I begin to prepare the rise to the million users and dexterity persons ...

Hugs

19

You could try profiling to find out for what this time is used.

One profiling solution is dm.zope.profile; there is also a WSGI based solution.

20

I would use py-spy · PyPI to attach to the running instance via its PID and watch its top like output while user is removed to get an idea whats going on and where the time is spent.

Classic profiling slows down Python a lot, not so Py-Spy.
For classic profiling plone.recipe.zope2instance can be configured to switch on profiling with repoze.profile, see plone.recipe.zope2instance · PyPI