GSoC 2018 Ideas: IFTTT plugin

[cewing]

This is one post in a series to begin focused discussion about ideas that came out of our 2018 GSoC Brainstorm.

Please use this post as a place to begin to discuss this idea more in depth.

This idea was suggested by @tkimnguyen, who has suggested that @fulv might be a good mentor.

The description: “create one or more IFTTT.com plugins for Plone”

Okay, community members. What sort of plugins would you most want to see? What would be most useful? Dive in and make this topic shine!

[MrTango]

@tkimnguyen can you please add more infos, what you want to archive with IFTTT?
If, we should propably thing about other IoT things too.

[tkimnguyen]

IFTTT.com lets you specify a trigger and an action to be taken when that trigger fires. A trigger can be many different types of checks or conditions, e.g. you received an email from a particular sender, and the action can be to send another email (or post to Slack etc. - there are many services integrated).

This is very similar to Plone content rules.

Off the top of my head, we could expand the scope of a content rule to include specifying an IFTTT.com “app” (I think that’s what they call them). It would require authenticating with IFTTT.com and giving the IFTTT app a local (to the Plone site) identifier.

Or someone could create a Plone integration for IFTTT so that an IFTTT trigger could be something happening on a Plone site, and an action could be something done on a (another?) Plone site. Would require authentication to the Plone site(s) involved. This way Plone sites could be part of the IFTTT ecosystem and could participate in orchestrated (I haven’t gotten tired of that word…yet) action taking place on the greater Internet.

This would have security implications too.

Here’s a possible example: someone publishes a piece of content on a Plone site, which results in it being tweeted automatically. I know this is doable with Plone scripts or in a Plone add-on, or by manually clicking on a social media sharing button/portal action, but the point of an IFTTT app and an integration between Plone and IFTTT is that a non-programmer could do it.

[cewing]

oh, interesting. I’ve been considering some approaches to emitting “events” from a Plone site using something like Kafka or Kinesis, so that a Plone site could be able to play along as a single component in a larger distributed system. This sounds similar.

I wonder if there’s room for the idea of implementing a generalized event bus that could have plugins added to emit to whatever external target there might be. Is there enough commonality to define a generic approach to hooking in to Zope events? I suspect that there is.

[djay]

That is pretty much content rules is already. Its just a matter of extra
action plugins. Or even a generic restapi/oauth plugin.

[tkimnguyen]

A generalized mechanism that lets people add more plug ins with as little programming as possible…

Should also have a way of ingesting external events

Webhooks - everyone seems to have implemented them; why not us?

[sally]

I think this is a great idea. I’m subscribed to the NTEN (non-profit tech) discussion digest, and one of the threads was “what are your favorite automations?”:
https://community.nten.org/communities/community-home/digestviewer/viewthread?GroupId=247&MessageKey=5234f55d-6e16-43c0-b1ca-9d501bc38473&CommunityKey=a8e1e11e-7bc9-471c-8d12-7b1386bff668&tab=digestviewer
There are a lot of mentions of using IFTTT recipes or Zapier zaps to move info from/to Eventbrite, Salesforce, Google Sheets, Slack, etc. etc. It would be really nice to have Plone added to these ecosystems (for marketing as well as usefulness, similar to the Siteimprove plugin GSoC idea). There are 2 parts, Plone acting as a trigger, and as an action.

Trigger: Plone content rules seem perfect, I don’t imagine we need another mechanism. For the non-technical people who are using IFTTT and Zapier it should be easy-peasy for them to trigger something from Plone. So we need an open source package with content rule conditions and actions that would be useful in IFTTT and Zapier. For the technically challenged it would help to have ready made conditions for obviously IFTTT/Zapier useful things (like “an image was added”, “a file was downloaded”) rather than have to build them out of multiple conditions. (Maybe the same package should also include actions that go directly directly to Slack, Twitter, Instagram, Facebook, SMS and other obviously useful targets.) There would also need to be a control panel or something to configure the connection from Plone to each service.

Action: Enabling Plone to handle IFTTT/Zapier events would I imagine require Plone to provide something like webhooks, which sounds like an interesting problem. The trigger part of this may be too easy for most GSoC students, but the action part should be more challenging.

[pigeonflight]

+1 on this.

[djay]

The trigger part of this may be too easy for most GSoC students, but the action part should be more challenging.

To do this right it needs to be asynchronous so its a little more complex. The reason is that anything in Plone to has to wait on an external network resource can easily become a bottleneck since each thread is so heavy and a typical stack doesn’t have hundreds of them.
But its not too hard to support c.async and c.taskqueue if they are installed.

Built in async or at least a plugin architecture for different async plugins is long long overdue. Is that a good GSOC FWT?

[pigeonflight]

Built in async or at least a plugin architecture for different async plugins is long long overdue. Is that a good GSOC FWT?

Yes! And probably a prerequisite for this project.

[tkimnguyen]

If any of those outside systems have web hooks (?) then Plone can just trigger them. Whether those systems respond quickly or not seems outside the scope… I would assume they’re built to perform. Even if they aren’t, adding async to this project means it isn’t going to happen …

[sally]

I’m probably being ignorant and naive, but isn’t a basic webhook just an HTTP POST request to a URL, with arbitrary data enclosed in the body of the request? Like data to submit a form, or an image or file to upload? I’m not seeing where the “wait on an external network resource” comes in. I was thinking that the authentication/permission stuff and figuring out how to make this work so it is super easy to use and in line with the way the rest of the world works would be the hard part, but that existing core functionality would support it. Doesn’t adding built in async or a new plugin architecture move this from “doable project” to “can of worms”?

[datakurre]

Trigger: Plone content rules seem perfect, I don’t imagine we need another mechanism.

Action: Enabling Plone to handle IFTTT/Zapier events would I imagine require Plone to provide something like webhooks.

If I understood properly, these would be reverse on Plone:

• Local Plone content rule action would fire remote IFTTT trigger.

• Remote IFTTT action would fire local Plone content rule trigger.

So, MVP would be a proof-of-concept generic Plone content rule action (for IFTTT trigger) or Plone content rule trigger (to be triggered from IFTTT action). I’d guess, Plone content rule action would be eaiser, because it only needs to trigger some IFTTT webhook (is there a such thing)?

And no async (for calling IFTTT trigger without blocking Plone) or queues (for queueing IFTTT calls without blocking Plone) to keep this GSOC-doable

[kakshay21]

I think it’s doable with async or queues also. It would be easier if there is already some plugin available for that. I’ve worked with Celery and RabbitMQ in Django to accomplish some tasks. If we have something similar to Celery, which is asynchronous task queue based on distributed message parsing, it would be a lot easier. Though Celery would be overkill here. But I think if we use distributed message parsing then it would be more scalable than any other alternative.
But overall, it’s a good GSOC project. I would definitely want to work on this.
Do we have any similar plugin for async task queues in Plone?
@tkimnguyen @datakurre

[tkimnguyen]

There are probably add-ons to do async queues. CastleCMS has that integrated (redis, celery). I think async for IFTTT-type functionality is overkill. Not everyone will want to run and manage separate processes for a small simple site. As a separate project, async queues for Plone could be good.

[kakshay21]

think async for IFTTT-type functionality is overkill

Yeah, That’s true. But the triggering task requires network calls and we don’t want to lose those so we do need a good queue mechanism that continuously calls the service, whenever the network is not busy. If the existing add-ons have these then it’s fine, we can use that.
Can you point me to that add-ons, please?

[datakurre]

Do we have any similar plugin for async task queues in Plone?

I have authored a couple:

• collective.zamqp provides robust auto-reconnecting and transaction aware RabbitMQ support for Plone with direct control of RabbitMQ various message routing features (we are using this everywhere we need guaranteed message delivery and communication between Plone instances and other systems)

• collective.taskqueue provides simple asynchronous in-memory queue for Plone instances without any external services; it can also be configured to use shared Redis based queue with guaranteed delivery (similar to collective.zamqp, but because existence of zamqp we are only using this for casual in-site async jobs [like RSS-updates] where we don’t need guaranteed delivery and can always requeue missed tasks

Then there’s also Celery plugin, which was developed under name collective.celery and then probably continued within Castle CMS. Plone Intranet has also its own async implementation (possibly using Celery). I did try both Celery and bare Kombu myself before implementing collective.zamqp in 2010, but was burned so badly by their various issues with AMQP back then that after losing a month of my life I have never looked them again…

[zopyx]

Sorry for destroying this thread but such a functionality is really artificial and not of common need. Focus on the things that matter.

-aj

[kakshay21]

@datakurre Amazing!

Webhooks - everyone seems to have implemented them; why not us?

@zopyx It would be a good experiment and as @tkimnguyen mentioned.

[djay]

I think async for IFTTT-type functionality is overkill. Not everyone will want to run and manage separate processes for a small simple site. As a separate project, async queues for Plone could be good.

Thats why collective.taskqueue is a good implementation as it provides built in default which requires no additional server. It can just create a seperate thread in the same instance to make the call outside of the the request thread.
Plone.app.async is also good as it requires nothing but ZEO.

What we really need is

1. An api in plone core that lets us run a task asynchronously regardless of how plone is installed. ie doesn’t require an extra service or ZEO. (obviously this means tasks are not serialised to a single queue).
2. Plugins that allow adding redis, celery or plone.app.async to this so you can serialise these async requests and get HA on the queue.

I’d love to see this come out of this GSOC, not just IFTTT integration.