ZEXP Import/Export via command line?

We use ZEXP import/export a lot during our daily work to move content between different instances (with the exact same code). This works quite well. Though, it is cumbersome to do upload a file to the server, go to the ZMI and import stuff manually (and then run into a timeout because you are too lazy to do ssh forwarding and go to the instance itself to bypass the firewall).

Anyways. I was wondering if anybody ever looked into option to make this easier. I am thinking about either a command line option to import/export zexp or a REST API endpoint. Is is even possible to import a ZEXP file that is not located on the server (or is this a security feature)?

I am about to start investigating the options outlined above but I was wondering if anybody else might have worked on this already and I can spare my time. I found this as a good starting point but that doesn't solve the file upload problem:

Importing in code is not hard. The method is called _importObjectFromFile. See https://github.com/collective/demo.plone.de/blob/master/src/plonedemo.site/src/plonedemo/site/setuphandlers.py#L208 for a example. Exporting is probably equally straightforward.

1 Like

Here is a commandline script I put together not so long ago to try and do this.
It didn't work out so well. The idea was to not have to have huge files for large sites and so stream the export from one zope into import process of the other.
I think the problem I ran into is that it ends up consuming a lot of RAM on two zopes at the same time which hit the limits of our machines.
So it probably needed to be reworked as a two stage process with a file inbetween. Possibly zexp to disk is also a lower memory process anyway.
Another enhancement that could be made is to turn off events during the import. We have a different patch that does this that can be incorporated. This speeds things up a lot.

from __future__ import print_function

from OFS.ObjectManager import customImporters

"""
Script to run on a UAT instance to get a zexp from prod and replace the uat portal with this import.
Requires entering a username and pw

"""

# plone5
import argparse
import getpass
import os
import sys
import tempfile

import requests
import transaction


IMPORT_FOLDER = 'var/instance/import'

def main(userid, path, server):
    global app  # app should be defined if running from zopepy or zopectl run

    path = [p for p in path.split('/') if p]
    portalid = path[-1]
    mnt = '/'.join(path[:-1])
    server = server.rstrip('/')

    # Step 1: download the zexp

    #userid = raw_input('username: ')
    pw = getpass.getpass('password: ')

    url = "{server}/{mnt}".format(server=server,mnt=mnt)
    print("Exporting {}/{}".format(url,portalid))
    r = requests.post(url + '/manage_exportObject', {'id':portalid, 'download:int':1}, auth=(userid, pw), stream=True)
    if r.status_code != 200:
        print(r.text)
        return
    size = int(r.headers['Content-length'])/1024
    #fd, zexp_filename = tempfile.mkstemp(dir=IMPORT_FOLDER, suffix='.zexp')
    zexp_filename = os.path.join(IMPORT_FOLDER, portalid+'.zexp')
    if not os.path.exists(zexp_filename):
        with open(zexp_filename, 'w') as tmp:
            i = 0
            for chunk in r.iter_content(chunk_size=1024):
                tmp.write(chunk)
                i+=1
                print('Downloading {}KB/{}KB'.format(i,size), end='\r')
                sys.stdout.flush()

    print("Downloaded {} {}Kb".format(zexp_filename, os.path.getsize(zexp_filename)/1000))

    transaction.get()

    #Step 2: remove the old site
    #print("Removing old site")
    #app.unrestrictedTraverse(mnt).manage_delObjects([portalid])

    # Step3: do zexp import
    print("Importing into local instance")
    #fileid = zexp_filename.split('/')[-1]
    #app.unrestrictedTraverse(mnt).manage_importObject(file=fileid, set_owner=0)
    obj = app.unrestrictedTraverse(mnt)
    connection=obj._p_jar

    while connection is None:
        obj=obj.aq_parent
        connection=obj._p_jar
    #TODO: in theory we could do the import direct from the stream and skip the local file
    # a file object can be passed in directly.
    # however import process does seeks back instead of peeking so buffer would need to support that
    # also this would keep the transaction open longer which could cause problems. It would save on diskspace
    # but not on RAM on either side.
    ob=connection.importFile(zexp_filename, customImporters=customImporters)
    #if verify: self._verifyObjectPaste(ob, validate_src=0)
    id=ob.id
    if hasattr(id, 'im_func'): id=id()
    obj._setObject(id, ob, set_owner=0, suppress_events=True)

    # try to make ownership implicit if possible in the context
    # that the object was imported into.
    ob=obj._getOb(id)
    ob.manage_changeOwnershipType(explicit=0)

    transaction.commit()
    # Step 4: clean up
    print("Removing zexp file")
    os.remove(zexp_filename)

if __name__ == "__main__":
    parser = argparse.ArgumentParser(prog="replace_site", description='Import a site back to uat')
    parser.add_argument('path', help='path to portal')
    parser.add_argument('-u', '--user', help='username', default="admin")
    parser.add_argument('-s', '--server', help='server to download from', default='http://localhost:8080/')
    args = parser.parse_args(sys.argv[3:])
    main(userid=args.user, server=args.server, path=args.path)
1 Like

@pbauer @djay thanks folks! Will look into it and see if I can come up with something useful to share with the community. :slight_smile:

Not my field of expertise, just a thought:

Would it make any difference if one split it up in smaller zexp files?

So basically just a loop thought the folders, exporting them as separate zexp-files

maybe if you only want content instead of the full site.

Plone Foundation Code of Conduct