Help with pas.plugins.ldap 1.4?

Plone Support
1

Guys, I need some help
I installed Plone 5.0.4 on Ubuntu and now I´m trying to use pas.plugins.ldap 1.4.0 with buildout, but when I run buildout appears the error below:

Error: There is a version conflict.
We already have: yafowil 2.1.4
but yafowil.widget.array 1.4 requires 'yafowil>2.1.99'.
but yafowil.widget.dict 1.6 requires 'yafowil>2.1.99'.

I tried to update yafowil but the problem remains as if I never had installed the latest version.

2

@crisnreis oh, that needs to be resolved - yafowil 2.2 doesn't have a final release yet, this is why buildout doesn't pick it when installing pas.plugins.ldap.

/cc @jensens

until yafowil gets a new release, you can fix it yourself. just declare the version dependencies of yafowil and pas.plugins.ldap (because I recommend to use the newer version with Plone 5) in your buildout configuration like so:

[versions]
yafowil = 2.2b2
pas.plugins.ldap = 1.5b1
1 Like
3

Robert just released yafowil 2.2 and pas.plugins.ldap 1.5
so it should pick latest, but to be sure you can pin it like so:

[versions]
yafowil = 2.2
pas.plugins.ldap = 1.5
1 Like
4

I can confim, that the latest versions of node.ext.ldap and pas.plugins.ldap install fine on Plone 5.

But what kind of performance should we expect with pas.plugins.ldap? What kind and size of LDAP setup it should work well enough? I ask, because I've been very puzzled about pas.plugins.ldap performance with our LDAP (AD based LDAP with a lot of users and groups, but most of the groups still being smaller than 500 members).

I don't mean to whine. We are really grateful that pas.plugins.ldap finally works as a replacement for LDAPUserFolder! I'm just curious, how others have managed to adapt pas.plugins.ldap for their installations.

Some releases ago pas.plugins.ldap didn't work at all for large directories, because every now and then it loaded whole director (took a long time for 70 000 users) and also it was missing paging. These have now been fixed, and logging-in and loading groups for the current user works quite ok. Unfortunately, browsing even small groups was pretty slow.

Big part of the slowness is the design of PAS, which is unaware of lazy evaluation or iterators, and wants to load attributes and groups for each new user immediately.

Yet, while spending a this week with pas.plugins.ldap, I noticed the following and made some optimizations (most probably incompatible with the upstream):

  • At first, node.ext.ldap expects LDAP queries to be cached for it to be usable. Correct?

  • The default supported cache is memcached. After Zope profiler showed a lot of time being spent in pickling searches in and out memcached (bda.cache used both pure-python memcached-lib and pure-python pickle), I patched bda.cache to use pylibcm: libmemcached based C-optimized memcached library, which also happens to use cPickle.

  • Next I noticed multiple LDAP queries being made to retrieve the same objects from LDAP (and miss the cache). Just with different baseDN, queryFilter and attrlist. And after all the optimizations for queried attrlist, eventually the attribute behavior in node.ext.ldap fetched all attributes for the previously fetched objects. I spent a couple of days in normalizing the queries so that only mapped and mandatory attributes were queried and in the best case each object were queried only once (and then retrieved from cache afterwards).

  • The final profiling found was that (after the optimizations above) the stack spent third of it time in decoding the data retrieved from LDAP into unicode. Because the cached data was encoded, all this decoding did miss the caching. I ended up removing all decoding/encoding code from node.ext.ldap, assumed all data in node.ext.ldap to be always utf-8 encoded and only added decoding/encoding into PAS interfaces in pas.plugins.ldap.

After all this, browsing "large" groups (with members over a few hundred) still DOS the site from tens of seconds to minutes. Yet, this can be attributed to design problems in PAS, and I resolved these simply by adding limits on how many users could be "enumerated" by search, or how big groups could be browsed.

I'd be happy to hear others experiences on pas.plugins.ldap with Plone and know if I have simply missed something obvious.

1 Like
5

@datakurre @jensens as any of this merged? I can't see any commits or PR from you.

6

I did finish some merged pulls into bda.cache and node.ext.ldap, but run out of resources on this.

I recall that @fredvd has talk at @ploneconf on their fork on top of my fork


1 Like
7

@datakurre That's correct. I thought, what can we show that we dit this year that might be interesting for other Plonista's. My plan is to show a small set up of all the moving parts involved in setting up pas.plugins.ldap and I'll show what issues we ran into in are project and the changes we made to make user searching work.(The problems were most in understanding what happens and what was wrong with the directory/ldap :fearful:)

The patches are rather small by the way. We didn't have any performance issues. I guess unless you are a univerrsity or multinational most middle sized companies have 500-2000 members in their directory. But it all depends on setup and settings, if you have nested groups etc.

But most us only set up ldap/AD integration occasionally, so you're happy when things work and when the next install is needed the client has changed, the situation has changed and your memory is gone. :stuck_out_tongue:

1 Like