Solved: Automate migration via commandline - How to fake a Post Request?

Plone Support Administering Plone sites
, ,
1

I disable CSRF in buidlout:

[instance-archetypes]
http-address = 10580
environment-vars +=
    PLONE_CSRF_DISABLED true

My upgrade script to run via commandline

# clearupscript5.py
import transaction
import pdb

from Testing.makerequest import makerequest
from zope.component.hooks import setSite
from AccessControl.SecurityManagement import newSecurityManager
from AccessControl.SecurityManager import setSecurityPolicy
from Testing.makerequest import makerequest
from Products.CMFCore.tests.base.security import PermissiveSecurityPolicy
from Products.CMFCore.tests.base.security import OmnipotentUser


def spoofRequest(context):
    """
    Make REQUEST variable to be available on the Zope application server.
    This allows acquisition to work properly
    """
    _policy=PermissiveSecurityPolicy()
    _oldpolicy=setSecurityPolicy(_policy)
    newSecurityManager(None, OmnipotentUser().__of__(context.acl_users))
    context =  makerequest(context)
    import pdb
    pdb.set_trace()
    return context

app = spoofRequest(app)
myplone = app.myplone
setSite(myplone)

migration_tool = myplone['portal_migration']
if migration_tool is not None:
    migration_tool.upgrade(myplone.REQUEST, dry_run=True)

The Traceback:

Traceback (most recent call last):
  File "/opt/mig-to-plone5/py2/parts/instance-archetypes/bin/interpreter", line 297, in <module>
    exec(_val)
  File "<string>", line 1, in <module>
  File "clearupscript5.py", line 32, in <module>
    migration_tool.upgrade(myplone.REQUEST,dry_run=True)
  File "<string>", line 3, in upgrade
  File "/opt/mig-to-plone5/Plone5/py2/eggs/AccessControl-4.2-py2.7-linux-x86_64.egg/AccessControl/requestmethod.py", line 77, in _curried
    raise Forbidden('Request must be %s' % methodsstr)
zExceptions.Forbidden: Request must be POST

What i do:
I call in my Browser the /@@plone-upgrade view and click the Button Upgrade, then the migration starts and run successfully. But if i call the my script via bin/instance-archetypes run clearupscript5.py, it fails.

How can i fake the Post Request to start the Upgrade via Commandline or is there another option availabel to automate the process? Any ideas or tips?

1 Like
2

Why do you need to make a fake request? Can't you just import the related browser view class, initialize the browser view and call the related method on this view?.

3

You mean this?

# clearupscript5.py
from zope.component import getMultiAdapter
from zope.component.hooks import setSite

setSite(app.myplone)
app.myplone.REQUEST.form.update({'form.submitted':True})
app.myplone.REQUEST.form.update({'dry_run':True})
upgrade_view = getMultiAdapter(
    (app.myplone, app.myplone.REQUEST),
    name=u'plone-upgrade'
)
print upgrade_view()

Also a Traceback:

Traceback (most recent call last):
  File "/opt/mig-to-plone5/py2/parts/instance-archetypes/bin/interpreter", line 297, in <module>
    exec(_val)
  File "<string>", line 1, in <module>
  File "clearupscript5.py", line 38, in <module>
    upgrade_view = getMultiAdapter(
  File "/opt/mig-to-plone5/src/Products.CMFPlone/Products/CMFPlone/browser/admin.py", line 310, in __call__
    dry_run=form.get('dry_run', False),
  File "<string>", line 3, in upgrade
  File "/opt/mig-to-plone5/py2/eggs/AccessControl-4.2-py2.7-linux-x86_64.egg/AccessControl/requestmethod.py", line 77, in _curried
    raise Forbidden('Request must be %s' % methodsstr)
zExceptions.Forbidden: Request must be POST
4

Usually I do something like

setSite(site)
makerequest(site)

from xx.yy.brower.my_view import MyView

view = MyView(site, site.REQUEST)
view.some_method()

Perhaps this won't work under all circumstances. But I can not recall having used getMultiAdapter in such a situation.

5

i will test it, thanks.

I tested your advice, but i get the same Traceback.

from Products.CMFPlone.browser.admin import Upgrade

myplone=app.myplone
setSite(myplone)
myplone = makerequest(myplone)
myplone.REQUEST.form.update({'form.submitted':True})
myplone.REQUEST.form.update({'dry_run':True})
view = Upgrade(myplone, myplone.REQUEST)
print view()

@pbauer How you automate your migrations? do you use an addon with a setuphandler?

6

I use upgrade-steps but I do not automate everything.

Usually the migration involves the follwing steps that I trigger manually:

  • Upgrade to latest 4.3.x
  • Run some custom upgrade-steps (remove addons and theme, cleanup the DB)
  • Run 5.2.x buildout in py2
  • Upgrade Plone to 5.2.x
  • Run some custom upgrade-steps (migrate default and custom AT to DX, uninstall AT etc.pp.)
  • Run 5.2.x buildout in py3
  • Run zodb migration
  • Run some more upgrade-steps in py3

Automation of all steps can be useful if the DB is very large. All these steps could be invoked from a instance-script and thus automated.

7

Thanks, for advice. I migrate from 2.5 to 5.2 with a bashscript, install the Plone Versions, Copy Data.fs, Run migration via instance script. But to start the migration_tool in the instance-script for Plone 5 ends with the Traceback above :frowning:

8

I don't see you setting the method to POST for the request. makerequest is creating an HTTPRequest, which sets the method based on an environment variable (https://github.com/zopefoundation/Zope/blob/master/src/ZPublisher/BaseRequest.py#L387).
But makrequest itself supports setting this: environ.setdefault('REQUEST_METHOD', 'GET'). So you could pass this in the call (currently you only provide the context).

9 
environ = {}
environ.setdefault('REQUEST_METHOD', 'POST')
myplone=app.myplone
setSite(myplone)
myplone = makerequest(myplone, environ=environ)
myplone.REQUEST.form.update({'form.submitted':True})
myplone.REQUEST.form.update({'dry_run':True})
view = Upgrade(myplone, myplone.REQUEST)
view.index = ViewPageTemplateFile("../../../../myview.pt")
view()

Traceback (most recent call last):
  File "/opt/mig-to-plone5/py2/parts/instance-archetypes/bin/interpreter", line 297, in <module>
    exec(_val)
  File "<string>", line 1, in <module>
  File "clearupscript5.py", line 41, in <module>
    view()
  File "/opt/mig-to-plone5/src/Products.CMFPlone/Products/CMFPlone/browser/admin.py", line 310, in __call__
    dry_run=form.get('dry_run', False),
  File "<string>", line 3, in upgrade
  File "/opt/mig-to-plone5/py2/eggs/AccessControl-4.2-py2.7-linux-x86_64.egg/AccessControl/requestmethod.py", line 77, in _curried
    raise Forbidden('Request must be %s' % methodsstr)
zExceptions.Forbidden: Request must be POST

I have the feeling that it is going in the wrong direction. i don't know if it's a good idea. I will create an addon with a global browserview and run curl, e.g curl --user "admin:admin" --request POST https://127.0.0.1:10580/myplone/@@automate-upgrade

10

Hmm, I wonder what method is actually used within the BaseRequest. Maybe makerequest is not compatible with the current BaseRequest implementation.

The failing check is this:

if request.method not in methods:
    raise Forbidden('Request must be %s' % methodsstr)

Can you check what method is on your actual request? Maybe that is not set correct.

11 
(Pdb++) pp request.method
'GET'
12

There are two options: it’s either not set correctly within makerequest or overwritten at some point. I would try to set it like this (and hope it goes through or make a curl request as you mentioned):

environ = {}
environ.setdefault('REQUEST_METHOD', 'POST')
myplone=app.myplone
setSite(myplone)
myplone = makerequest(myplone, environ=environ)
myplone.REQUEST.form.update({'form.submitted':True})
myplone.REQUEST.form.update({'dry_run':True})
myplone.REQUEST.method = 'POST'  # <-- add this
view = Upgrade(myplone, myplone.REQUEST)
view.index = ViewPageTemplateFile("../../../../myview.pt")
view()
1 Like
13

:+1: Thank for the tip, the script run now.

1 Like