Defanging development databases when using Relstorage and Plone 6

Plone Support
1

When working with relstorage and Plone 6 is there a simple way to defang a development database?

For the sake of comparing notes, I've summarised my setup and approach below:

TL;DR... we create a snapshot of our development database in a container and develop against that.

My setup:

  • custom built docker containers based on plone/plone-backend.
  • docker-compose on the server to manage launching the container (will likely move to something like ansible and kubernetes in the future for more automation).
  • Relstorage that connects to an external managed postgres

Development approach:
We pg_dump the production database to a local pg container to create a local "snapshot". Development is done against the local snapshot container.

I'd love to defang the local data somehow.
This used to be "a thing" when using a Data.fs, there was very useful package: isotoma.plone.defang · PyPI

2

A simple way is to export the whole Plone site as a zexp and import it into the local instance. This does not work well for large databases as it takes a lot of time.

3

Ah... instead of populating the development database using postgres-related tools I would rely on the zexp to populate the development database.

How does this relate to the "defanging" requirement?

4

I would go for a similar strategy as the one used by isostoma.plone.defang. It could be described as folows:

Write a set of defang steps and conversely also the refang steps

Do the following:

  1. run the defang steps in your dev db
  2. pg_dump the "defanged" dev db (and pg_restore it to the target db)
  3. run the refang steps in your dev db

Alternatively, to make sure you don't get a modified current/original db:

  1. pg_dump the current/original dev db
  2. run the defang steps in your dev db
  3. pg_dump the "defanged" dev db (and pg_restore it to the target db)
  4. run the refang steps in your dev db
  5. pg_restore the current/original db
5

@mekell,
Makes sense. I like the fact that isotoma handed me a recipe :slight_smile: . Will just have to make the time to roll my own recipe now.

Thanks @jensens and @mekell ... let's see where this goes.

6

Well, what about a run-script to "defang" any Plone database (after whatever import) before it is started up?

Scenarios I can think of:

  • reset password of all users in /Plone/acl_users
  • remove sub-trees of data
  • replace fields of a Type or Behavior with a random value

And do not forget to pack the DB and remove the .old file in case of FileStorage.

7

and evtl.

  • remove users
  • delete content types
  • plone-uninstall (as oposed to pip-uninstall) add-ons like plone.reload or plone.restapi that for whatever reason might have security issues or might not be pip-installed in the target (production) environment
  • delete languages (in multilingual sites)
  • ...
8

This is one of the ideas of isostoma.plone.defang: to defang the database before it is started up.

They open a dbfile via ZODB.FileStorage and ZODB.DB. This could be adapted to work on app after opening it via Zope2.Startup.run import make_wsgi_app with a zope.conf.

Another interesting idea of isotoma's approach is the use of - a rather rudimentary - kind of interface Fang with two methods fang and defang which get app as argument (see isotoma.plone.defang.fangs) .

9

:heart_eyes: exactly... useful stuff.

1 Like