We use ZEXP import/export a lot during our daily work to move content between different instances (with the exact same code). This works quite well. Though, it is cumbersome to do upload a file to the server, go to the ZMI and import stuff manually (and then run into a timeout because you are too lazy to do ssh forwarding and go to the instance itself to bypass the firewall).
Anyways. I was wondering if anybody ever looked into option to make this easier. I am thinking about either a command line option to import/export zexp or a REST API endpoint. Is is even possible to import a ZEXP file that is not located on the server (or is this a security feature)?
I am about to start investigating the options outlined above but I was wondering if anybody else might have worked on this already and I can spare my time. I found this as a good starting point but that doesn't solve the file upload problem:
Here is a commandline script I put together not so long ago to try and do this.
It didn't work out so well. The idea was to not have to have huge files for large sites and so stream the export from one zope into import process of the other.
I think the problem I ran into is that it ends up consuming a lot of RAM on two zopes at the same time which hit the limits of our machines.
So it probably needed to be reworked as a two stage process with a file inbetween. Possibly zexp to disk is also a lower memory process anyway.
Another enhancement that could be made is to turn off events during the import. We have a different patch that does this that can be incorporated. This speeds things up a lot.
from __future__ import print_function
from OFS.ObjectManager import customImporters
"""
Script to run on a UAT instance to get a zexp from prod and replace the uat portal with this import.
Requires entering a username and pw
"""
# plone5
import argparse
import getpass
import os
import sys
import tempfile
import requests
import transaction
IMPORT_FOLDER = 'var/instance/import'
def main(userid, path, server):
global app # app should be defined if running from zopepy or zopectl run
path = [p for p in path.split('/') if p]
portalid = path[-1]
mnt = '/'.join(path[:-1])
server = server.rstrip('/')
# Step 1: download the zexp
#userid = raw_input('username: ')
pw = getpass.getpass('password: ')
url = "{server}/{mnt}".format(server=server,mnt=mnt)
print("Exporting {}/{}".format(url,portalid))
r = requests.post(url + '/manage_exportObject', {'id':portalid, 'download:int':1}, auth=(userid, pw), stream=True)
if r.status_code != 200:
print(r.text)
return
size = int(r.headers['Content-length'])/1024
#fd, zexp_filename = tempfile.mkstemp(dir=IMPORT_FOLDER, suffix='.zexp')
zexp_filename = os.path.join(IMPORT_FOLDER, portalid+'.zexp')
if not os.path.exists(zexp_filename):
with open(zexp_filename, 'w') as tmp:
i = 0
for chunk in r.iter_content(chunk_size=1024):
tmp.write(chunk)
i+=1
print('Downloading {}KB/{}KB'.format(i,size), end='\r')
sys.stdout.flush()
print("Downloaded {} {}Kb".format(zexp_filename, os.path.getsize(zexp_filename)/1000))
transaction.get()
#Step 2: remove the old site
#print("Removing old site")
#app.unrestrictedTraverse(mnt).manage_delObjects([portalid])
# Step3: do zexp import
print("Importing into local instance")
#fileid = zexp_filename.split('/')[-1]
#app.unrestrictedTraverse(mnt).manage_importObject(file=fileid, set_owner=0)
obj = app.unrestrictedTraverse(mnt)
connection=obj._p_jar
while connection is None:
obj=obj.aq_parent
connection=obj._p_jar
#TODO: in theory we could do the import direct from the stream and skip the local file
# a file object can be passed in directly.
# however import process does seeks back instead of peeking so buffer would need to support that
# also this would keep the transaction open longer which could cause problems. It would save on diskspace
# but not on RAM on either side.
ob=connection.importFile(zexp_filename, customImporters=customImporters)
#if verify: self._verifyObjectPaste(ob, validate_src=0)
id=ob.id
if hasattr(id, 'im_func'): id=id()
obj._setObject(id, ob, set_owner=0, suppress_events=True)
# try to make ownership implicit if possible in the context
# that the object was imported into.
ob=obj._getOb(id)
ob.manage_changeOwnershipType(explicit=0)
transaction.commit()
# Step 4: clean up
print("Removing zexp file")
os.remove(zexp_filename)
if __name__ == "__main__":
parser = argparse.ArgumentParser(prog="replace_site", description='Import a site back to uat')
parser.add_argument('path', help='path to portal')
parser.add_argument('-u', '--user', help='username', default="admin")
parser.add_argument('-s', '--server', help='server to download from', default='http://localhost:8080/')
args = parser.parse_args(sys.argv[3:])
main(userid=args.user, server=args.server, path=args.path)
