Where do blobstorage savepoints come from?

[mauritsvanrees]

I have seen several Plone sites where the blobstorage has a tmp directory with several savepoints that are months old. Just this morning a sysadmin gave me a copy from a Plone 4.3 site with these savepoint directories from October last year:

.../blobstorage/tmp# ls -l
total 32
drwx------ 2 buijs buijs 4096 Oct 17 22:46 savepoints6ut1eZ
drwx------ 2 buijs buijs 4096 Oct 17 22:46 savepointsMfhIES
drwx------ 2 buijs buijs 4096 Oct 17 22:46 savepointsrhuP2J
drwx------ 2 buijs buijs 4096 Oct 17 22:46 savepointsThW_xK
drwx------ 2 buijs buijs 4096 Oct 17 22:46 savepointsUs51UT
drwx------ 2 buijs buijs 4096 Oct 17 22:46 savepointsvgUQA5
drwx------ 2 buijs buijs 4096 Oct 17 22:46 savepointsVjXfPx
drwx------ 2 buijs buijs 4096 Oct 17 22:46 savepointsx3rXnX

Only one of the directories has actual contents:

blobstorage/tmp# ls -l savepoints6ut1eZ
total 8448
-rw------- 1 buijs buijs   64531 Oct 17 22:46 0x0a7b72-0x00.spb
-rw------- 1 buijs buijs  141698 Oct 17 22:46 0x0a7b7b-0x00.spb
-rw------- 1 buijs buijs   24133 Oct 17 22:46 0x0a7b7c-0x00.spb
-rw------- 1 buijs buijs    2984 Oct 17 22:46 0x0a7ef9-0x00.spb
-rw------- 1 buijs buijs    3560 Oct 17 22:46 0x0a7efd-0x00.spb
-rw------- 1 buijs buijs    2031 Oct 17 22:46 0x0a8463-0x00.spb
-rw------- 1 buijs buijs    2031 Oct 17 22:46 0x0a8464-0x00.spb
-rw------- 1 buijs buijs    5821 Oct 17 22:46 0x0a8468-0x00.spb
-rw------- 1 buijs buijs    5821 Oct 17 22:46 0x0a8469-0x00.spb
-rw------- 1 buijs buijs   14001 Oct 17 22:46 0x0a846e-0x00.spb
-rw------- 1 buijs buijs   11362 Oct 17 22:46 0x0a8472-0x00.spb
-rw------- 1 buijs buijs    5397 Oct 17 22:46 0x0a8473-0x00.spb
-rw------- 1 buijs buijs    2720 Oct 17 22:46 0x0a8474-0x00.spb
-rw------- 1 buijs buijs    2046 Oct 17 22:46 0x0a8475-0x00.spb
-rw------- 1 buijs buijs   30518 Oct 17 22:46 0x0a8476-0x00.spb
-rw------- 1 buijs buijs    1009 Oct 17 22:46 0x0a8477-0x00.spb
-rw------- 1 buijs buijs    5024 Oct 17 22:46 0x0a847c-0x00.spb
-rw------- 1 buijs buijs   22098 Oct 17 22:46 0x0a84b2-0x00.spb
-rw------- 1 buijs buijs    3395 Oct 17 22:46 0x0a84b3-0x00.spb
-rw------- 1 buijs buijs   16510 Oct 17 22:46 0x0a84b4-0x00.spb
-rw------- 1 buijs buijs 7977124 Oct 17 22:46 0x0a84b5-0x00.spb
-rw------- 1 buijs buijs   36304 Oct 17 22:46 0x0a863d-0x00.spb
-rw------- 1 buijs buijs   37543 Oct 17 22:46 0x0a8643-0x00.spb
-rw------- 1 buijs buijs   49869 Oct 17 22:46 0x0a8826-0x00.spb
-rw------- 1 buijs buijs   64131 Oct 17 22:46 0x0a8827-0x00.spb
-rw------- 1 buijs buijs    1865 Oct 17 22:46 0x0a889a-0x00.spb
-rw------- 1 buijs buijs    6578 Oct 17 22:46 0x0a889b-0x00.spb
-rw------- 1 buijs buijs   25027 Oct 17 22:46 0x0a889c-0x00.spb
-rw------- 1 buijs buijs   22876 Oct 17 22:46 0x0a889d-0x00.spb
-rw------- 1 buijs buijs    2980 Oct 17 22:46 0x0a889e-0x00.spb

Looking for the biggest one, I do not see a matching file in the real blobstorage:

.../blobstorage# find . -name '0x0a84b5*'
./tmp/savepoints6ut1eZ/0x0a84b5-0x00.spb

Obviously something went wrong in October. But I haven't heard anything about it, the site seems fine, and zeopack has been run.

So questions:

1. What causes these savepoints to be made? I guess code calling transaction.savepoint may do this.

2. What causes these savepoints to stay there, even after a zeopack and several months? Hah, I now see one that still has savepoints from 2014!

3. Is it safe to remove them? I guess so. I have already removed them earlier in at least two sites, and have not had a problem. The ones that I remember were all Plone 4.3 sites.

[mauritsvanrees]

@mauritsvanrees

What causes these savepoints to be made? I guess code calling transaction.savepoint may do this.

Out of simple curiosity I looked for large files, and got this:

$ ls -lh $(find . -size +10M)
-r--------  1 maurits  staff    33M 17 Oct 22:46 ./0x00/0x00/0x00/0x00/0x00/0x06/0x28/0xf4/0x038e0d0cc394bbdd.blob
-r--------  1 maurits  staff    18M 17 Oct 22:45 ./0x00/0x00/0x00/0x00/0x00/0x2e/0x96/0xd3/0x03b3f579f6ac1d77.blob
-r--------  1 maurits  staff    16M 17 Oct 22:46 ./0x00/0x00/0x00/0x00/0x00/0x47/0x39/0x6f/0x03bf1825e69ebe11.blob
-r--------  1 maurits  staff   128M 17 Oct 22:44 ./0x00/0x00/0x00/0x00/0x00/0x58/0xb8/0x8f/0x03cd726dd147d300.blob
-r--------  1 maurits  staff    84M 17 Oct 22:45 ./0x00/0x00/0x00/0x00/0x00/0x5a/0xc7/0x77/0x03cfbfae1e6e4366.blob
-r--------  1 maurits  staff    94M 17 Oct 22:45 ./0x00/0x00/0x00/0x00/0x00/0x5a/0xc7/0xcd/0x03cfbfb04fbcc755.blob
-r--------  1 maurits  staff    78M 17 Oct 22:45 ./0x00/0x00/0x00/0x00/0x00/0x5a/0xc8/0x25/0x03cfbfb0da9aee33.blob
-r--------  1 maurits  staff    88M 17 Oct 22:45 ./0x00/0x00/0x00/0x00/0x00/0x5a/0xc8/0x78/0x03cfbfb1a4824222.blob
-r--------  1 maurits  staff   106M 17 Oct 22:45 ./0x00/0x00/0x00/0x00/0x00/0x5a/0xc8/0xcc/0x03cfbfb2795c6055.blob
-r--------  1 maurits  staff    29M 18 Nov 09:59 ./0x00/0x00/0x00/0x00/0x00/0x5d/0xfa/0x7d/0x03d3e0fba31b50bb.blob
-r--------  1 maurits  staff    14M 12 Dec 12:50 ./0x00/0x00/0x00/0x00/0x00/0x5e/0x5f/0x4e/0x03d46e46a98470cc.blob
-r--------  1 maurits  staff    14M  5 Mar 15:30 ./0x00/0x00/0x00/0x00/0x00/0x5f/0xc4/0x80/0x03d652a64fb2af55.blob
-r--------  1 maurits  staff    15M  9 Mar 16:28 ./0x00/0x00/0x00/0x00/0x00/0x5f/0xd3/0xf6/0x03d669601bb538ee.blob

Most of them are from 17 October, at the same time as the savepoints, or a minute before.
And I can see in an issue tracker of this customer that around this time a new release was put in production. There was nothing special in this release. I don't have access to this machine. If I understand correctly, their process is to run buildout in a fresh directory, stop the old site, copy over the data, and start the new site. If a change was in progress at the point that they stopped the old site, then maybe this happens. But I see a comment from the sysadmin that he specifically checked that there were hardly any visitors on the site at this point.

So either someone was in the middle of doing changes anyway, or the copy or startup did not go completely right.
Or the savepoint would have been cleaned up automatically slightly later when the old site would have kept running.

Ah: almost all blobs (162.000 out of 168.000) have a last modification date between 17 Oct 22:44 and 22:46, so around that same time. So that would be the copy, and not real changes in the site.