Creating a Collection page of videos from another (non-Plone) site like Youtube, Vimeo, etc.?

So, we have a server with many different videos. Scattered through our website. They are tagged. The videos are on a non-Plone server.
On the Plone 5 server, we are able to manually embed those videos as desired within a Plone Page content item. One or many, it is no problem, manually.
But this is a very time consuming process for hundreds of videos, with new content every week.

Is it possible for a Plone Collection to grab from our webserver, based on tags/keywords, and create a nicely formatted video collection, similar to this:
http://rpgresearch.com/documents/primary/images/screenshot-20170105-144452-cropped.png

Thanks for any suggestions!

Plone collections are based on catalog queries and therefore stuff handled by a collection must exist as primary Plone content.

In your case you need to provide a custom video content-type (written in Dexterity) representing each video and content object in Plone (which does not mean that your video content lives as copy in Plone). You need to find a way to represent your external videos either by URL, file path or ID in Plone.

How to sync videos with their proxy objects in Plone? This is up to you. You need to write some code checking the existing videos on your external system against what you have in Plone and create new content proxies manually or automatically. plone.api will help you to interact with the Plone site. If you have an API on your video storage solution then use this in order to get the video names/ids/paths or whatever. http://corp.kaltura.com/Products/Kaltura-API might be of interest here.

For e-teaching.org we were in a similar situation with videos stored as MP4 files on a remote filesystem (NFS accessible). We created a "Video Source" content type where all videos showed up within a SELECT list. We added some candy for making this (long) video list searchable. theoretically you create a searchable source yourself using the Ajax widgets for Dexterity but we found them to be horrible rotten and broken.

OR: you interact directly with the Kultura API through browser views and Kultura API calls...depending on your requirements. I am not sure why you are looking into the collections here since the Kultura API appearch to be rich and gives you all and everything for searching, sorting and grouping videos. Rendering the result of a API query to Kultura inside a Plone browser view should not be a big deal. Again: programming work required.

-aj

3 Likes

we had a similar requirement for the 2016 Olympics site:

http://www.brasil2016.gov.br/en/videos

this is how we solved it:

  • videos are stored in YouTube
  • Plone site has sc.embedder package installed; this package is a content type that allows to include content from a external site importing only its metadata
  • we used the clock server feature of Zope to poll YouTube API every hour in search of new videos and then created (and published) instances of sc.embedder for it
  • we used a collection to display the results

Unfortunately, sc.embedder doesn't work on Plone 5 yet.

3 Likes

I'm collecting project ideas for an upper year computer science class I'm helping with this semester. Sounds like a promising project!

2 Likes

https://github.com/collective/collective.opensearch may be of some help. I used it in the past for similar tasks (without creating plone proxy content) https://github.com/collective/Products.feedfeeder may be worth investigating as well

2 Likes

Since so many of the replies to the requests for this site have been "not available in Plopne 5", we're setting up an instance of Plone 4.3.11 to see if we can get all these things to work as desired in Plone 4. Unfortunately they are getting a bit frustrated, so I need to get these things working by end of next week, or they're bailing back to Worpress (blech!). Hopefully going back to the better supported Plone 4 will solve these multiple issues for trying to build this platform.
I will try out sc.embedder, and the other suggestions on the other threads.
Many thanks everyone for your feedback! Here's hoping we can make it all work.

I tried using FeedFeeder, but most of the content sources (Kaltura CE, and such) didn't really have good RSS support, so that didn't work out as desired. But may still factor in for other parts.
I'll check out the collective.opensearch to see if that is helpful.
Thanks @cleder!

I wish I was a better ambassador for Plone for this project. I have until this coming Thursday afternoon, January 19th, to get at least a mockup working of what is possible, else they will give up completely on using Plone. :frowning: I believe Plone is the best centerpiece for their entire platform, but my lack of knowledge (and time to delve into coding), is unfortunately interfering with showing them just how great it could be.

They/we have pretty much given up on Plone 5. We really need what older versions of Plone do well, providing a "good enough" variety of features (rather than fighting with trying to integrate a dozen "best of breed" products), without having to write any (or very little) code, with an exceptionally secure platform, great granular content and user management. That has been the great power of it for so many of these community sites since switching over to plone in 2004.

So enough preamble :slight_smile: and back to the technical challenge, using Plone 4.3.11, I'm trying to combine opensearch with a collection. I'm not sure where I can put the opensearch syntax to grab from an external site. Trying to use youtube as an example.
Backing up to a simpler task, just trying to use the example from the opensearch site - https://pypi.python.org/pypi/collective.opensearch/1.5

"If you add a link content type with a open search description
compatible url (i.e. it has a parameter {searchTerms} in the querystring)
the view of the link will be set when the link is saved. The view consists of a simple searchform and the results of the query"

Here is the example opensearch string (that works in web browser): https://www.youtube.com/results?search_query={the_gamers}&page={startPage?}&utm_source=opensearch
That I am trying to get working in Plone.

But adding as a Link content item, and changing view to Opensearch view, nothing is found: "No results found". Though it works in regular link_redirect_view mode.

And in a collection, not sure where that can be shoehorned in.
Sc.embed working fine, and once I figure out the opensearch + collection challenge, would like to see about combining (if possible) collection + opensearch + sc.embed.

Suggestions on how (doesn't have to be the best way), just a way that can be done just for the sake of proof of concept for a prototype as quickly as possible?

They're about to throw in the towel and abandon Plone, and just use Wordpress (Even though they know how bad that is in so many other ways). I have until Thursday to try to at least get something working to see at least the potential of Plone. I will still use it for my own personal community sites, and I think this huge video/movie community would greatly benefit from being on Plone, but alas I'm failing to convince them due to my lack of time/knowledge to show them that Plone could do what they want eventually, and all the other benefits (security, user management, etc.) are well worth the extra "hassles" compared to php-based and other nasty "competing" products.

(Reminder note, I am doing this as a volunteer trying to help them out).

Appreciate any examples/guides.
Thanks!

Enough lamentation.

Kaltura has a well-defined and extensive API.

As indicated in my very first reply: it should be possible (easily) to integrate the API natively using a browser - means some Python code, some template code, some interaction with the Kaltura REST API. Instead you are tinkering with other second class solutions or approaches..

-aj

Sorry @zopyx , I forgot to mention that Kaltura is probably no longer part of the equation.
I was hoping, by going back to Plone 4, to follow up on what was suggested by @hvelarde.
The folks for this project evaluated the CE version of Kaltura, it is incredibly buggy, and is lacking too many of the functions needed. The other features are only available in the commercial hosting, which would cost many tens of thousands USD per month/year, based on the quotes. Kaltura is much more affordable than the several other competing video products, but causes more problematic vendor lock-in concerns, and would greatly reduce their budget to be able to make new content, just covering the Kaltura expenses.
They are working on a front-end, and possible peer-to-peer options to greatly reduce bandwidth expenses, but still need a central user community driven content platform, Plone ( ? )
So, Kaltura is unlikely to be any longer in the picture. It is possible a Plone setup may be the video repository down the road maybe . They were looking at Plumi, but he said it had far to few features for content providers. I whipped up a Plumi instance this weekend for them to try out with more flexibility than the previous online demo they tried, to give them a second chance to be sure.
But meanwhile, if I can just whip something up, grabbing from youtube, vimeo, and other sources, arranged in various ways within Plone (like the original screenshot in this post), such a collection.
Then it may be possible to encourage the extra time and effort needed for such integration.

@hvelarde On plone 4.3, do you have any additional information on how you made that work? Is it documented anywhere?
It might be just what is needed for now.
Thanks!

Have you looked at Plumi.org?

1 Like

Yes, 2 posts up, buried in text wall I guess. :slight_smile:

"They were looking at Plumi, but he said it had far to few features for content providers. I whipped up a Plumi instance this weekend for them to try out with more flexibility than the previous online demo they tried, to give them a second chance to be sure.But meanwhile, if I can just whip something up, grabbing from youtube, vimeo, and other sources, arranged in various ways within Plone (like the original screenshot in this post), such a collection.
Then it may be possible to encourage the extra time and effort needed for such integration."

There is a computer science class (software engineering) I will be helping with this semester to try to beef up Plumi.

But yeah, your client seems to have complex requirements. We don't have an out of the box solution for that exact set of requirements :wink: and trying to rush it while learning the ins and outs of customizing Plone was a valiant effort, so thank you.

1 Like

here is the script we used:

# -*- coding:utf-8 -*-
" " " 
Import YouTube feed into Plone using sc.embedder as content type.

Copyright 2015-2016, Simples Consultoria.
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 59 Temple Place, Suite 330, Boston, 
MA 02111-1307 USA.
" " "
from my.proyect.controlpanel import IYouTubeSettings
from my.proyect.logger import logger
from collections import namedtuple
from DateTime import DateTime
from gdata.youtube.service import YouTubeService
from plone import api
from plone.i18n.normalizer.interfaces import IIDNormalizer
from plone.namedfile.file import NamedBlobImage
from plone.protect.interfaces import IDisableCSRFProtection
from Products.CMFPlone.utils import safe_unicode
from Products.Five.browser import BrowserView
from zExceptions import BadRequest
from zope.component import queryUtility
from zope.interface import alsoProvides

import json
import urllib

BASE_URI = 'https://www.youtube.com/feeds/videos.xml?channel_id='
EMBED_TEMPLATE = '<iframe width="{width}" height="{height}" src="https://www.youtube.com/embed/{id}"></iframe>'

fields = [
    'id',  # w_sqsoiHkbc
    'title',
    'description',
    'url',  # https://www.youtube.com/watch?v=w_sqsoiHkbc
    'thumbnail',
    'width',
    'height',
    'creation_date',
    'modification_date',
]
Video = namedtuple('Video', fields)


class ProcessYouTubeFeedView(BrowserView):

    def __init__(self, context, request):
        self.context = context
        self.request = request

    def setup(self):
        alsoProvides(self.request, IDisableCSRFProtection)
        self.channel_id = api.portal.get_registry_record(
            IYouTubeSettings.__identifier__ + '.youtube_channel_id')
        if not self.channel_id:
            raise BadRequest(u'No channel ID specified on YouTube configlet.')
        logger.info(u'Using channel_id = ' + self.channel_id)
        self.feed_uri = BASE_URI + self.channel_id
        self.folder = self._get_folder()

    def _get_folder(self):
        """Return default folder to fill with YouTube Feed entry.
        Create folder if needed.
        """
        try:
            portal = api.portal.get()
            return portal['videos']
        except KeyError:
            logger.info(u'Default folder not found; creating it')
            with api.env.adopt_roles(['Manager']):
                folder = api.content.create(
                    container=portal,
                    type='Folder',
                    title='Videos',
                    exclude_from_nav=True,
                )
                api.content.transition(obj=folder, transition='publish')
            return folder

    def _get_feed(self):
        """Get feed with YouTube Service."""
        logger.info(u'Getting video feed from YouTube')
        yt_service = YouTubeService()
        feed = yt_service.GetYouTubeVideoFeed(self.feed_uri)
        if api.env.test_mode():
            feed.entry = feed.entry[:1]  # return only one entry when running tests
        logger.info(u'Feed contains {0} entries'.format(len(feed.entry)))
        return feed

    def _get_thumbnail(self, url):
        logger.info(u'Getting thumbnail from: ' + url)
        try:
            data = urllib.urlopen(url).read()
            return NamedBlobImage(data=data)
        except IOError:
            logger.warning(u'Connection error')
            pass

    def _get_entry_data(self, entry):
        return Video(
            id=entry.id.text.split(':')[2],
            title=safe_unicode(entry.media.title.text),
            description=safe_unicode(entry.media.description.text),
            url=entry.link[0].href,
            thumbnail=self._get_thumbnail(entry.media.thumbnail[0].url),
            width=int(entry.media.thumbnail[0].width),
            height=int(entry.media.thumbnail[0].height),
            creation_date=entry.published.text,
            modification_date=DateTime(entry.updated.text),
        )

    def _create_video(self, entry):
        """Create new video based on feed entry."""
        logger.info(u'Creating new video from feed entry')
        util = queryUtility(IIDNormalizer)
        video = self._get_entry_data(entry)

        last_created = self._get_creation_date_from_latest_video()
        if video.creation_date <= last_created:
            return 0

        video_id = util.normalize(video.title)
        if video_id in self.folder:
            return 0

        embed_html = EMBED_TEMPLATE.format(
            width=video.width, height=video.height, id=video.id)

        with api.env.adopt_roles(['Manager']):
            obj = api.content.create(
                container=self.folder,
                type='sc.embedder',
                id=video_id,
                title=video.title,
                description=video.description,
                url=video.url,
                width=video.width,
                height=video.height,
                embed_html=embed_html,
                image=video.thumbnail,
                creation_date=DateTime(video.creation_date),
                modification_date=video.modification_date,
            )
            api.content.transition(obj=obj, transition='publish')

        logger.info(u'Video "{0}" created and published'.format(video.title))
        return 1

    def _get_creation_date_from_latest_video(self):
        """Return the creation date of the most recent video in the catalog."""
        catalog = api.portal.get_tool('portal_catalog')
        folder_path = '/'.join(self.folder.getPhysicalPath())
        results = catalog(
            portal_type='sc.embedder', sort_on='created', sort_order='reverse', sort_limit=1,
            path={'query': folder_path, 'depth': 1})[:1]

        if results:
            # convert to UTC as that's the time zone used by YouTube entries
            title, date = results[0].Title, results[0].created.toZone('UTC').ISO()
            msg = u'Latest video available ("{0}") was created on {1}; importing from newer entries only'
            logger.info(msg.format(safe_unicode(title), date))
        else:
            # no videos in catalog; we use earliest date possible
            date = DateTime(0).toZone('UTC').ISO()

        return date

    def _update_feed_entries(self, feed):
        """Update feed entries if needed."""

        totals = {'total': len(feed.entry), 'created': 0}

        # add only videos created since last time
        for entry in feed.entry:
            totals['created'] += self._create_video(entry)

        if totals['created'] > 0:
            logger.info(u'{0} videos added.'.format(totals['created']))
        else:
            logger.info(u'No videos added.')

        return totals

    def __call__(self):
        """Process the YouTube feed."""
        self.setup()
        feed = self._get_feed()
        totals = self._update_feed_entries(feed)
        return totals

this is the ZCML registration:

  <browser:page
      name="process-youtube-feed"
      for="Products.CMFPlone.interfaces.IPloneSiteRoot"
      class=".youtube.ProcessYouTubeFeedView"
      permission="cmf.AddPortalContent"
      />

and this is the buildout configuration for a special instance:

[instance-youtube]
<=instance
event-log-level = INFO
port-base = 9
z2-log-level = INFO
zodb-cache-size = 5000
zope-conf-additional =
    <clock-server>
       method /Plone/@@process-youtube-feed
       period 3600
       user <user>
       password <password>
       host localhost:8089
    </clock-server>

replace user and password there and create an user with enough privileges to add content.

1 Like

@tkimnguyen, please keep me posted on how that goes, that will be helpful in conversations with them about the platform options. Thanks!

@hverlade, Thanks so much! I'll give that a shot.

Just realize that unless you stay involved we are unlikely to create a package that meets your exact (complex) requirements. The best way to get the most out of open source is to get involved and at least help identify features that are generally useful :slight_smile:

1 Like

@tkimnguyen Totally understand that. And I am so very grateful for the help you have provided.
Just in case you didn't know, I've been involved with "open source" (before that became recognized as that term) since 1979 through University of Utah, German programmers, and others, and am a huge advocate for it!
I release my documents under creative commons, and try to make all my research projects as open as I legally can, rather than restrictive closed journals.
Alas, I can only spread myself so much across so many projects. I worked with vangheem previously, after pointing out for years, the problems with registration spam of Plone, that lead to his creating the emailconfirmationregistration add-on initially, and then lead to the Plone security folks finally acknowledging the security issue that had been there in all version, and lead to the 20150910 hotfix. But though I had been pointing out the problems for 3-5+ years without much response, it took vangheem kindly stepping up, and asking for an account on my server (which was gladly provided), for someone with the skillset he had to track down what the problems were, and realize it was Plone, and just something wrong with my setup. :slight_smile: Just pointing out that although I don't have any bandwidth left for coding, I have been an advocate and supporter of Plone since 2004, and hope to continue being so for many years to come as time allows. :slight_smile:
Thanks again for all your help!

@hvelarde, okay, so I have the buildout entries created, and ran buildout (seemed to run without errors).
Restarted server (no errors).
I added the script through ZMI Script (Python) item. Assuming that was correct?
I'm not sure where/how to place the ZCML registration.
Was I suppposed to create a whole add-on to make this work?
Sorry for the newb question. Appreciate a smidgeon more guidance to get this in place.
Again, thanks very much!