State of profiling under Zope 4/ Python 3?

jensens · September 11, 2019, 1:29pm

It seems my beloved Products.ZopeProfiler did not make it into Python 3 land.

What is the state of profiling under Zope 4? What to use, best practices?

jugmac00 · September 11, 2019, 1:53pm

@thet posted a screenshot and a link to

a couple of weeks ago.

I haven't used it much, but it looks promising and you can attach the profiler to a running process!

jensens · September 11, 2019, 2:34pm

I was looking more for some kind of integration of the stdlibs profile (cProfile) module. Something where I can profile a single request through Zope, write the result to a file and start analyzing using the stdlibs pstats module in order to see callers, callees, cumulative time, number of calls.

If there is some nice UI on top of it, ok. Downloading a the requests profile would be enough. There are tools like snakeviz and others helping with analysis.

icemac · September 12, 2019, 5:27am

You could use repoze.profile, a profiling WSGI middleware. It uses cProfile under the hood.
See https://repozeprofile.readthedocs.io/en/latest/

pbauer · September 12, 2019, 7:24am

I usually use profilehooks for profiling together with snakeviz for visualization. It is a pity that CacheManager and DebugManager were removed from Zope. It's a but manual but it makes it easy to profile specific methods/views/tests.

@dieter: Are you planning to port Products.ZopeProfiler to Zope4/Python3?

@jensens: Please report about your experiences with repoze.profile and/or py-spy.

dieter · September 12, 2019, 8:06am

Philip Bauer via Plone Community wrote at 2019-9-12 07:35 +0000:

...
@dieter: Are you planning to port Products.ZopeProfiler to Zope4/Python3?

The main feature Products.ZopeProfiler provided over other
profiling solutions has been its "high level" (i.e. Zope rather
than Python level) profile support.
This was important when most logic was in ZODB objects
("Script Python", "DocumentTemplate", "PageTemplate");
nowadays, it resides mostly in views and therefore is profilable at
the Python level.
Therefore, I do not know whether it is important to have
Products.ZopeProfiler in a modern environment.

jensens · September 12, 2019, 9:13am

py-spy is not what I am looking for, so I did not test it deeper. I think @thet collected already some experience, maybe he can report here?

I tested repoze.profile and yeah, it just works!

Lazy as I am, I wrote a quick integration for plone.recipe.zope2instance here as a draft PR with a TODO-list (help welcome, I have no idea when I have time to finish it). See its README for available options. Lazy, because this took me including trying out, fiddling with KCacheGrind and this post) only 2 hours of work!

TL;DR: use branch, in [instance] set profile = on.

Experience: It gives access to the profile of the last request executed on an instance. There is a web-interface, the raw-data is written to a file and - a big plus - the data is written in callgrind/cachegrind format and ready for further analysis with a tool supporting this general format (if the additional package is installed). I used KCachegrind (also available as QCacheGrind for Windows/OSX) and its complete easy to get good results here.

flipmcf · July 8, 2020, 5:54pm

@jensens Thanks for your work on this. I don't have much time to contribute to the branch, but I do need to use a profiler for plone 5.2.2 and this seems to be the right path IMHO

I just added plone.recipe.zope2instance to my buildout 'sources' with your branch - like any normal development egg - and all worked.

I got a localhost:8080/__profile__ endpoint - which I think is success.

Additionally, I was able to take the default output ( /instance/var/log/profile-instance.raw) and launch kCacheGrind:

pyprof2calltree -k -i profile-instance.raw

fredvd · July 9, 2020, 9:43am

@jensens If I remember correctly you showed me py-spy to profile low level ZCA performance at this years Alpine City Sprint.

Is py-spy suitable for PloneZope profiling after all, or do you still prefer a more integrated profiler like repoze.profile?

pbauer · July 9, 2020, 10:04am

I use py-spy regularly to get the current stacktrace of a running instance with
sudo py-spy dump --pid 22747 and sudo py-spy top --pid 22747 to inspect what functions are currently taking the most time. It is great to find out what is going wrong when something takes unexpectedly long. You don't need to install anything in your Plone or use buildout because you can use it from a separate virtualenv. I think you can even inspect py2 threads from a py3 virtualenv.

I have not played with py-spy record, instead i usually profile individual methods with profilehooks.

jensens · July 9, 2020, 2:49pm

py-spy is valuable, yes, but I think it needs both: classic profiling gives more possibilities to figure out whats going on in detail while py-spy helps to get an good overview of an running app, even on prod instances.

Today I use both and like them.

dieter · July 9, 2020, 3:30pm

I am working on a successor for Products.ZopeProfiler; its name will be dm.zope.profile.

dieter · July 20, 2020, 6:36am

Published on PyPI a few minutes ago.

jugmac00 · July 20, 2020, 6:53am

And here is the link for @dieter's profiler:

thet · March 15, 2021, 2:34pm

@dieter tnx for dm.zope.profile!

It currently fails for me with this error:

zope.configuration.config.ConfigurationConflictError: Conflicting configuration actions
  For: ('resource', 'orderedselect_input.js', <InterfaceClass zope.publisher.interfaces.browser.IBrowserRequest>, <InterfaceClass zope.publisher.interfaces.browser.IDefaultBrowserLayer>)
    File "/home/ikath/.buildout/eggs/z3c.form-3.7.1-py3.8.egg/z3c/form/browser/orderedselect.zcml", line 46.2-49.8
        <browser:resource
            name="orderedselect_input.js"
            file="orderedselect_input.js"
            />
    File "/home/ikath/.buildout/eggs/dm.zope.profile-1.0-py3.8.egg/dm/zope/profile/configure.zcml", line 35.4-38.8
          <browser:resource
            name="orderedselect_input.js"
            file="orderedselect_input.js"
            />

Should that maybe go into a includeOverrides instead?

dieter · March 15, 2021, 4:05pm

dm.zope.profile has introduced orderedselect_input.js in order to work around a bug in z3c.form (inline JS containing < which poses problems in XML documents -- the workaround moved the JS out of the template into its own JS file). Apparently, you are using a newer version of z3c.form which fixed the bug similar to the workaround of dm.zope.profile causing the conflict.

Your options:

comment out the registration of ordered_select_input.js in dm.zope.profile:configure.zcml
use z3c.unconfigure to unconfigure one of the conflicting registrations
use an older version of z3c.form.

thet · March 15, 2021, 8:24pm

Did manually edit the configure.zcml of dm.zope.profile.

What about accepting a PR for dm.zope.profile as another option?

dieter · March 15, 2021, 9:53pm

At the moment, I do not know how to support both older (with bug) and newer (with conflict) versions of z3c.form. Do you have an idea? Once it is clear how to handle the case, actual doing it should be trivial.

jensens · March 16, 2021, 8:28am

AFAIK you can register new components as well in includeOverrides. So having all in the overrides should work well for buggy and fixed z3c.form. Code here at least looks like it works that way zope.configuration/src/zope/configuration/xmlconfig.py at master · zopefoundation/zope.configuration · GitHub

dieter · March 16, 2021, 9:03am

Maybe it will work. If it does, however, it will use my resources over those provided by z3c.form. I would prefer a way where z3c.form resources are used if available and my own only when this is not the case: i.e. if z3c.form has no orderedselect_input.js use my JS and my template; otherwise use those of z3c.form.