Zodbconvert: filestorage --> RelStorage much slower than other direction

Plone Support Deployment & hosting
1

I have a Plone 4.3 instance using RelStorage. I noticed that the restore of a FileStorage to Relstorage is much slower than saving the RelStorage as FileStorage; in my case (Data.fs size about 3.6 GiB):

RelStorage --> FileStorage: below 20 minutes,
FileStorage --> RelStorage: about 4050 minutes (more than 200 times slower!)

I tried to switch off autovacuum in PostgreSQL during that process, reloading the PostgreSQL demon configuration, but this didn't seem to change much.

  • Is this considered normal?
  • Could this improve by using newer packages?
  • Are there some important configuration tweeks to check?

My current versions:

  • PostgreSQL 9.3 (ancient, I know; because of CentOS 7)
  • RelStorage 1.6.3
  • psycopg2 2.8.6
  • ZODB3 3.10.5
2

Wow, that is slow. I remember it was slow with 1.6, but not that slow.
Did you pack the filestorage before convert?
How are blobs handled?

And yes, RelStorage 3 is much faster, also with convert. But its all Python 3 only, so no option for an ancient site/setup.

3

Well, the RelStorage is packed once a week. That specific Data.fs was probably not created immediately after packing, but it shouldn't be too badly bloated.
BlobStorage is in the filesystem (var/blobstorage); in the zodb.conf:

<zodb_db main>
    # Main database
    cache-size 300000
    %import relstorage
    <relstorage>
        blob-dir /opt/zope/instances/plone1/var/blobstorage
        <postgresql>
            dsn dbname='zodb_plone1' user='zodb_plone1' host='localhost' password='S0mes3curepassw0rd'
        </postgresql>
    </relstorage>
    mount-point /
</zodb_db>

So I don't expect the blob storage to have any effect, right?

4

No, the blobstorage is not in the way. In the zodb convert configuration you even do not need to configure it for both sides, because it wont touch it (or load blobs) anyway.

5

Right; it isn't.

6

Ok; I try a conclusion.

  • At least for old versions of RelStorage, the "restore" (FileStorage --> RelStorage) is slow. Not necessarily as slow as in my case, but much slower than "saving" the RelStorage to FileStorage.
  • This fact makes zodbconvert (for such cases) nearly useless as a backup method -- you'd rather not wait so long for the restore to complete.
  • Instead, the RelStorage database should be backed up and restored using the native database utilities.

That's a pity, because a zodbconvert run could be integrated easily in the backup configuration (as a pre_command) -- to feed the resulting export in the further backup -- but I couldn't find a way to integrate the backup of sql dumps there.
Nor to restore relational databases;
I still need my own home-brewed backup and restore to integrate this.
(Yes, I can integrate a script which creates the backups and copies them somewhere, but it won't be connected in any way to a certain dated recipe-created backup).

This makes me seriously consider to replace my current RelStorage setup by ZEO.
The reason I didn't do so in the first place:

  • I inherited a non-ZEO setup
  • which included a supporting PostgreSQL database already anyway,
  • and the UnifiedInstaller creates such a very different structure for zeo than for standalone that I never really looked into converting.
7

I really do not use any recipe related backup. PostgreSQL plus some files in a filesystem are easy to backup. In my opinion its a big plus to be completely agnostic to any Zope/Plone specifics. If it comes to deployment any administrator knows what to do (opposite to snapshot/backup a filestorage).
With RelStorage 3.x things are becoming easier, since blobs can be stored without problems in the DB, so there is only one storage to backup.

1 Like
8

Ah, ok; I thought it would be only me, and I felt a little bit bad about it ...

9

I wonder if it is the same reason as I had with zodbconvert: Zodbconvert incredibly slow [solved] - #3 by ngoeddel

I fixed it by adding an other unique index to the database.