Warming ZODB connection caches at Zope startup?

Plone Support Add-on Development
1

In my site policy, I intend to preload and warm some content at startup via IDatabaseOpenedWithRoot subscriber. For now, I am probably contented with (process-global) side-effects of memoizing the result of some views on content, and I have some scaffolding set up to my satisfaction to traverse to content and call/render it this from my at-startup handler. I’m borrowing the idea of opening a new connection from this post/thread.

However, it occurs to me that one thing I may want is simply to guarantee warming connection cache for the things I am traversing to, for every connection in event.database.pool — I am trying to determine if this (warm ZODB connection cache, not just ram cache) at startup is a reasonable and feasible objective?

I do not know if the pool actually has connections kept in it at this point in the process lifecycle, and my read of ZODB.DB.ConnectionPool is that I would need to pop() all connections until exhausted (pop() returns None), then repush() them all back in sequnce. Are there risks or difficulties to doing this that I should be looking out for?

Thoughts?

Sean

2

@seanupton Here's what I am doing these days. I cheated and hardcoded the number of connections to warm up, to match the number of threads that are configured for the application server. As far as I know, the connection pool can provide more connections than there are ZServer threads, but I only need enough warm connections for the configured threads.

from Testing.makerequest import makerequest
from ZPublisher.WSGIPublisher import publish

import logging
import time


PATHS = (
    "/Plone/++api++/de/?expand=breadcrumbs,navigation,translations",
    "/Plone/++api++/en/?expand=breadcrumbs,navigation,translations",
)
THREADS = 2

logger = logging.getLogger(__name__)


def warm_zodb_cache(event):
    """
    Warm the ZODB by fetching a specified list of paths during startup.

    This is a handler for the IDatabaseOpenedWithRoot event.

    This also serves as a smoke test that will halt deployment
    if these paths can't be served without an error.
    """
    import Zope2

    t0 = time.time()
    conns = []
    # Open one connection for each publisher thread
    for x in range(THREADS):
        # Get the Zope root with a new connection from the ZODB pool
        app = Zope2.bobo_application()
        # Don't try to do warmup if the Plone site hasn't been created yet
        if "Plone" in app:
            # Publish the configured paths
            for path in PATHS:
                request = makerequest(
                    app,
                    environ={
                        "PATH_INFO": path.split("?")[0],
                        "QUERY_STRING": "" if "?" not in path else path.split("?")[-1],
                    },
                ).REQUEST
                try:
                    response = publish(request, (app, "Zope", False))
                    logger.info(f"Warmed {path}: {response.status}")
                finally:
                    request.close()
        # Keep the connection open until we've processed one per thread,
        # so we don't get the same one from the pool again
        conns.append(app._p_jar)
    for conn in conns:
        conn.close()
    logger.info(f"Warmed {len(conns)} connections in {time.time() - t0}s")
    logger.info(f"{event.database.cacheSize()} objects in ZODB caches")

Improvements I've considered but not implemented yet:

  • Get the number of threads from configuration
  • Do the warming in parallel threads
1 Like