Developing sites - using a subset of data in testing

Plone Support
1

This is a continuous development problem. Just wondering if anyone has already solved the problem of developers having to work on full copies of a Plone site.

When working on a site that relies on a large dataset, I'd like developers to be able to pull a representative subset of the data. Is there a tool that does this now?

I'm not talking big data but when the site needs to be tested, there's no fun in having to pull 10GB of a full database, especially if there's a simple way to pull a representative subset.

2

If the dataset is caused by a large blobstorage and your testing can live without images or other blob-stored fields, you could skip copying the blobstorage and install experimental.gracefulblobmissing .

This is a real developer tool though. You can use it to fix templates or use an older blobstorage and suppress errors for the blobs that are missing, but know what it is doing:

  • some risks are that you might miss detecting errors in blob-handling in your code while developing because you expect the blobs to fail because they're not there.
  • And don't run this in preview/production or on a filestorage/blobstorage you want to re-use (see the readme): gracefulblobmissing will actively create new empty blob files in your blobstorage for the files it cannot open. So it can potentially corrupt an existing blobstorage if that one isn't complete.
3

i have used collective.loremipsum to create Testdata a long time ago.

4

We use

for creating specific site fixtures according to customer requirements...requires some more code
than a generic approach but we usually want to have the content only that is actually needed for styling a site.

The module is also used for plone-demo.info.

-aj

5

It should be possible to use a ZODB DemoStorage to load the real data from production but write any changes to a local file (sort of like how plone.app.testing uses stacked storages to be able to tear down test layers). I've never gotten around to actually trying this, however, so don't have step-by-step instructions.

6

Thanks @fredvd, @1letter, @zopyx and @davisagli... all of these are useful directions to look at.
I think, assuming we have the energy and it, building a site fixture with zopyx.ipsumplone is a very enticing option, since it runs plone-demo.info I think I can assume that it works with Plone 5.