Calling (slow) rest from Plone

I need suggestions on how to deal with timeouts when making a rest call

As stated in another post, I played around with 'making something that can take screenshots of a Plone site.

Using sanic and asyncio I have made a asyncio (rest) app so

http://someip/sitemap/http://myplonesite.xml.gz takes screenshots of 'everything' in the sitemap

and
http://someip/screenshot/http://somesite.com/page

Returns a screenshot of any page.

Now: I want to add a browser view in Plone, so it is possible to

http://myplonesite/somefolder/@@makescreenshots would make screenshots of all pages 'inside some folder'.

Since this might take several minutes: how do I deal with timeouts.

Is it possible

  • from Plone, call http://sanic_app/makescreenshots
  • from sanic_app: return a code (202 ? ) and a message 'work in progress' and a link or something
  • in sanic app: make the screenshots and send message

PS: Help with what I can google is also helpful

Update: I have put very basic working code here: https://github.com/espenmn/medialog.takescreenshhot/blob/8ecc7e25adf19317aa4d48ea3136de7cd0cd616e/make_screenshot_app.py

I think what you are looking for is to handle the request asynchronously by a worker process and not in the request to the plone site itself. The grand old lady to do this in plone is plone.app.async which builds on zc.async, but I'd stay away from that as it is very complex (zc.async) and there's not enough public resources and/or installed base anymore. I have used in a deployment of eea.pdf, and there is a an example in collective.documentviewer.

Modern more lightweight solutions: @datakurre built collective.taskqueue which might help you, I haven't deployed it myself.

Thanks.

Does this mean that I should 'deal with all the problems' in Plone, and add for example the requests for a screenshot "one by one"?

My plan was to pass a list of urls to the syncio/rest-app

It depends on how you want to get the screenshots back to the user.

Once I had plone.app.async in place for eea.pdf I also used it for a custom add'on where we implemented highcharts graphs as a custom content type. We also needed the charts as an SVG and highcharts offered a separate highcharts export server which you can feed a chart json definition and then receive svg/png/pdf.

I created a separate plone app.async task: when an editor modifies the highcharts graph an async job gets created. A separate zope instance which is the worker instance get the job with the context of the graph content item and starts to process it a few seconds later: sends the json to the highcharts export server, retrieves the svg and stores the svg again on the plone content item. Then later on we can use the svg file (for including in a generated PDF for example, no fun running a 700 page document with 400 interactive highcharts javascript graphs through wkhtml2pdf)

This way your timeouts and waiting is done in the async task on the zope worker process, not in the zope instance where you fired the original request and want to deliver a response to the user.

eea.pdf generates a pdf of content-item(s) in the background, stores the pdf in a separate filesystem location (using eea.download) and then sends the user who requested the PDF an e-mail with a download link to the generated PDF file. (retrievable in Plone through eea.download views).