May 02 2017
May 02

Custom Video Export/Import Process With Views and Feeds

In the Media Research Center's set of three main Drupal sites, MRCTV serves as our video platform where all videos are created and stored as nodes, and then using Feeds, specific videos are imported into the other two sites (Newsbusters and CNS News) as video nodes. Then, on NB and CNS, we use the Video Embed Field module with a custom VEF provider for MRCTV to play the embedded videos.

There are only specific videos that need to be imported into the destination sites, so a way to map channels between the two sites is needed. All three sites have a Channels vocabulary, a mapping is created between the appropriate channels. This mapping has two parts:

  1. A feed of channels terms on NB and CNS.
  2. A custom admin form that links source channels on MRCTV with target channels on the destination site.

On the receiving site side, in addition to the standard feed content, the following custom elements are needed for the Feeds import:

  1. The nid of the video node on MRCTV. This is used to create the URL that is put into the VEF field.
  2. The taxonomy terms for the Channels vocabulary terms in the destination sites (NB and CNS).

Since these are outside of the standard feed components, they will need to be added custom to the feed items.

I documented my custom Feeds importer on drupal.stackexchange, so you can see the code there.

MRCTV is finally in the process of being updated to D8 from D6 (insert derision here), so both the mapping form and the feed needed to be re-created. The first part of the structure is the channel mapping form. The following file VideoExportForm.php is placed in /modules/custom/video_export/src/Form:

/**
 * @file
 * Contains \Drupal\video_export\Form\VideoExportForm.
 */

namespace Drupal\video_export\Form;

use Drupal\Core\Config\ConfigFactoryInterface;
use Drupal\Core\Form\ConfigFormBase;
use Drupal\Core\Form\FormStateInterface;
use GuzzleHttp\Exception\RequestException;

class VideoExportForm extends ConfigFormBase {
  
  /**
   * {@inheritdoc}.
   */
  public function getFormId() {
    return 'video_export_settings';
  }
    
  /**
   * {@inheritdoc}
   */
  public function buildform(array $form, FormStateInterface $form_state) {
    $form = array();
    $channels = array();
    
    // Get list of channels.
    $terms =\Drupal::entityTypeManager()->getStorage('taxonomy_term')->loadTree('channels');
    foreach ($terms as $term) {
      $channels[$term->tid] = $term->name;
    }
    
    // Get config data from video_export.settings.yml.
    $config = \Drupal::config('video_export.settings');
    $mapping_config = \Drupal::config('video_export.mappings');
    $sites = $config->get('sites');
    
    foreach($sites as $site => $site_data) {
      // Get channels list.
      try {
        $response = \Drupal::httpClient()->get($site_data['channel_url'], array('headers' => array('Accept' => 'text/plain')));
        $data = $response->getBody();
        if (empty($data)) {
          return FALSE;
        }
      }
      catch (RequestException $e) {
        return FALSE;
      }
  
      $channel_data = new \SimpleXMLElement($data);
      foreach ($channel_data->channel as $channel) {
        $channel_name = $channel->name->__toString();
        $channel_tid = $channel->tid->__toString();
        $target_channels[$channel_tid] = $channel_name;
      }
      // Sort array alphabetically by element.
      asort($target_channels, SORT_STRING);
  
      $target_channel_options = array();
      $target_channel_options[0] = "No Channel";
      foreach ($target_channels as $target_tid => $target_name) {
        $target_channel_options[$target_tid] = $target_name;
      }
  
      //Get mappings from mappings conifg.
      $mappings = $mapping_config->get('sites');
      foreach ($mappings[$site]['mappings'] as $mrctv_channel => $target_channel) {
        $mapping_defaults[$mrctv_channel] = $target_channel;
      }
  
      $form[$site] = array(
        '#type' => 'details',
        '#title' => t($site . ' Channel Mappings'),
        '#description' => t('Map MRCTV channels to ' . $site . ' channels'),
        '#collapsible' => TRUE,
        '#collapsed' => TRUE,
        '#tree' => TRUE,
      );
  
      // Loop through all of the categories and create a fieldset for each one.
      foreach ($channels as $id => $title) {
        $form[$site]['channels'][$id] = array(
          '#type' => 'select',
          '#title' => $title,
          '#options' => $target_channel_options,
          '#tree' => TRUE,
        );
        if (in_array($id, array_keys($mapping_defaults))) {
          $form[$site]['channels'][$id]['#default_value'] = intval($mapping_defaults[$id]);
        }
      }
    }
    
    // Get mapping configs.
    $xml = array();
    $mapping_config = \Drupal::config('video_export.mappings');
    $sites = $mapping_config->get('sites');
    $channel_mappings = $sites[$site]['mappings'];
    // Get video nodes that belong to one of the selected channels.
    $query = \Drupal::entityQuery('node')
      ->condition('status', 1)
      ->condition('type', 'video')
      ->condition('changed', REQUEST_TIME - 59200, '>=')
      ->condition('field_channels.entity.tid', array_keys($channel_mappings), 'IN');
    $nids = $query->execute();
    // Load the entities using the nid values. The array keys are the associated vids.
    $video_nodes = \Drupal::entityTypeManager()->getStorage('node')->loadMultiple($nids);

    foreach ($video_nodes as $nid => $node) {
      $host = \Drupal::request()->getSchemeAndHttpHost();
      $url_alias = \Drupal::service('path.alias_manager')->getAliasByPath('/node/' . $nid);
      // Get channels values.
      $channel_tids = array_column($node->field_channels->getValue(), 'target_id');
      $create_date = \Drupal::service('date.formatter')->format($node->getCreatedTime(), 'custom', 'j M Y h:i:s O');
      $item = array(
        'title' => $node->getTitle(),
        'link' => $host . $url_alias,
        'description' => $node->get('body')->value,
        'mrctv-nid' => $nid,
        'guid' => $nid . ' at ' . $host,
        'pubDate' => $create_date
      );
      // Check for short title and add it if it's there.
      if ($node->get('field_short_title')->value) {
        $item['short-title'] = $node->get('field_short_title')->value;
      }
      foreach ($channel_tids as $ctid) {
        $item[$site . '-channel-map'][] = $ctid;
      }
      $xml[] = $item;
    }

    return parent::buildForm($form, $form_state);
  }
  
  /**
   * {@inheritdoc}.
   */
  public function validateForm(array &$form, FormStateInterface $form_state) {
  
  }
  
  protected function getEditableConfigNames() {
    return ['video_export.mappings'];
  }
  
  /**
   * {@inheritdoc}
   */
  public function submitForm(array &$form, FormStateInterface $form_state) {
    $values = $form_state->getValues();
    $config = $this->config('video_export.mappings');
    $sites = array();

    foreach($values as $site => $mappings) {
      if (is_array($mappings)) {
        foreach ($mappings['channels'] as $mrctv_channel => $target_channel) {
          if ($target_channel != 0) {
            $sites[$site]['mappings'][$mrctv_channel] = $target_channel;
          }

        }
        $config->set('sites', $sites);
      }
    }
    $config->save();
  
    parent::submitForm($form, $form_state);
  }
}

The setting for the channels feeds on NB and CNS are stored in /modules/custom/video_export/config/install/video_export.settings.yml:

sites:
  newsbusters:
    channel_url: 'http://www.newsbusters.org/path/to/channels'
  cnsnews:
    channel_url: 'http://www.cnsnews.com/path/to/channels'
list_time: 24
        

Since this is an admin settings form, I extend the ConfigFormBase class. This adds some additional functionality over the standard FormBase class, similar to the way the system_settings_form() function does in D7 and older (see the change record for details).

As mentioned above the form does the following things:

  1. Reads the channels feed from the destination sites
  2. Creates a fieldset for each site with a select list for each MRCTV channel where the user can select the destination channel.
  3. Saves the mappings in config.

The next thing that is needed is the feed of video nodes that are available to be imported. After trying unsuccessfully to create a custom REST API endpoint, I ended up going with a Feeds display in Views. Out of the box I can create my feed, but I still need to add my custom elements. In D6, I used hook_nodeapi($op = 'rss item') to add my custom elements. In other feeds on D7 sites I've been able to use the Views RSS module with its provided hooks to add custom RSS elements, but as of now it is currently unusable for D8 due to one major issue.

Finally, since everything in D8 is based on OOP, I knew there had to be a way to override a Views class at some level, so after some searching, I decided to override the display plugin. I poked around in the Views code and found the RssFields class that is used for the field level display for a Feeds display, so I overrode that.

namespace Drupal\video_export\Plugin\views\row;

use Drupal\views\Plugin\views\row\RssFields;
use Drupal\Core\Form\FormStateInterface;
use Drupal\Core\Url;

/**
 * Renders an RSS item based on fields.
 *
 * @ViewsRow(
 *   id = "mrctv_rss_fields",
 *   title = @Translation("MRCTV Fields"),
 *   help = @Translation("Display fields as RSS items."),
 *   theme = "views_view_row_rss",
 *   display_types = {"feed"}
 * )
 */
class MRCTVRssFields extends RssFields {
  
  /**
   * Override of RssFields::render() with additional fields.
   *
   * @param object $row
   *
   * @return array
   */
  public function render($row) {
    $build = parent:: render();
    $item = $build['#row'];
    
    // Add MRCTV nid
    $item->elements[] = array(
      'key' => 'mrctv-nid',
      'value' => $row->nid,
    );
  
    // Add channels and their target nids. We can get them from $row->_entity.
    $site = $this->view->args[0];
    // Get MRCTV nids from view.
    $channel_tids = array_column($row->_entity->field_channels->getValue(), 'target_id');
    // Now, get destination tids from config.
    $mapping_config = \Drupal::config('video_export.mappings');
    $all_mappings = $mapping_config->get('sites');
  
    foreach($channel_tids as $mrctv_channel) {
      if(in_array($mrctv_channel, array_keys($all_mappings[$site]['mappings']))) {
        $item->elements[] = array(
          'key' => $site . '-channel-map',
          'value' => $all_mappings[$site]['mappings'][$mrctv_channel],
        );
      }
    }
    
    // Re-populate the $build array with the updated row.
    $build['#row'] = $item;
    
    return $build;
  }
}

As you can see, the override is fairly simple; all I needed to do was override the render() method. This method returns a render array, so all I do is get the built array from the parent class, add my custom elements to the #row element in the array, and return it.

One thing that I couldn't do simply in the views UI was select the nodes that should be in the feed based on the associate Channels vocabulary terms. These are dynamic, based on the mappings selected in the admin form, so I can't pre-select them in the view settings. This is where hook_views_query_alter() comes to the rescue.

/**
 * Implements hook_views_query_alter().
 */
function video_export_views_query_alter(Drupal\views\ViewExecutable $view, Drupal\views\Plugin\views\query\Sql $query) {
  if ($view->id() == 'video_export' && $view->getDisplay()->display['id'] == 'feed_1') {
    // First, we need to get the site parameter from the view.
    $site = $view->args[0];
    
    // Next, we need to get the saved config for the channel mapping.
    $mapping_config = \Drupal::config('video_export.mappings');
    $all_mappings = $mapping_config->get('sites');
    $tids = array_keys($all_mappings[$site]['mappings']);
   
    // Modify query to get nodes that have the selected nids, which are the array keys.
    $query->addWhere(NULL, 'node__field_channels.field_channels_target_id', $tids, 'IN');
  }
}

All I do here is get the saved mappings from config and add them to the views query as a WHERE condition to limit the feed items to the appropriate nodes.

One issue I ran into with the results was duplicate records. Since field_channels (the entity reference field for the Channels vocabulary) is multiselect, the query returns multiple records for each node if there are multiple Channels terms selected. There are display settings to show multiple items in one row, but they don't take effect here. I didn't dig far enough into the code to know for sure, but my guess is that the grouping happens at a higher layer in the views rendering process, so they don't take effect in this situation.

To get around this, I implemented hook_views_pre_render(). At this point in the process, the results have been built, so I just loop through them and remove duplicates.

/**
 * Implements hook_views_query_pre_render().
 */
function video_export_views_pre_render(Drupal\views\ViewExecutable $view) {
  $unique_nids = $new_results = array();
  
  // Loop through results and filter out duplicate results.
  foreach($view->result as $index => $result) {
    if(!in_array($result->nid, $unique_nids)) {
      $unique_nids[] = $result->nid;
    }
    else {
      $new_results[] = $result;
    }
  }
  // Replace $view->result with new array. Apparently views requires sequentially keyed
  // array of results instead of skipping keys (e.g. 0, 2, 4, etc), so we can't just
  // unset the duplicates.
  $view->result = $new_results;
}

As noted in the code comment, views seems to require a sequentially numbered array, so you can't just unset the duplicate keys and leave it as is, so I chose to just add each item to a new array. In retrospect, I could have just used PHP functions like array_splice() and array_filter(), but this method works just as well.

It should also be noted that the views hooks need to go in a *.views_execution.inc file, so this one is in /modules/custom/video_export/video_export.views_execution.inc.

All I do at this point is use the Job Scheduler module with Feeds in the destination sites to schedule the import at the desired interval, and the process runs by itself. 

Sep 14 2016
Sep 14

Handling clients with more than one site involves lots of decisions. And yet, it can sometimes seem like ultimately all that doesn’t matter a hill of beans to the end-user, the site visitor. They won’t care whether you use Domain module, multi-site, separate sites with common codebase, and so on. Because most people don’t notice what’s in their URL bar. They want ease of login, and ease of navigation. That translates into things such as the single sign-on that drupal.org uses, and common menus and headers, and also site search: they don’t care that it’s actually sites search, plural, they just want to find stuff.

For the University of North Carolina, who have a network of sites running on a range of different platforms, a unified search system was a key way of giving visitors the experience of a cohesive whole. The hub site, an existing Drupal 7 installation, needed to provide search results from across the whole family of sites.

This presented a few challenges. Naturally, we turned to Apache Solr. Hitherto, I’ve always considered Solr to be some sort of black magic, from the way in which it requires its own separate server (http not good enough for you?) to the mysteries of its configuration (both Drupal modules that integrate with it require you to dump a bunch of configuration files into your Solr installation). But Solr excels at what it sets out to do, and the Drupal modules around it are now mature enough that things just work out of the box. Even better, Search API module allows you to plug in a different search back-end, so you can develop locally using Drupal’s own database as your search provider, with the intention of plugging it all into Solr when you deploy to servers.

One possible setup would have been to have the various sites each send their data into Solr directly. However, with the Pantheon platform this didn’t look to be possible: in order to achieve close integration between Drupal and Solr, Pantheon locks down your Solr instance.  That left talking to Solr via Drupal.

SearchAPI lets you define different datasources for your search data, and comes with one for each entity type on your site. In a datasource handler class, you can define how the datasource gets a list of IDs of things to index, and how it gets the content. So writing a custom datasource was one possibility.

Enter the next problem: the external sites we needed to index only exposed their content to us in one format: RSS. In theory, you could have a Search API datasource which pulls in data from an RSS feed. But then you need to write a SearchAPI datasource class which knows how to parse RSS and extract the fields from it.  That sounded like we’d be reinventing Feeds, so we turned to that to see what we could do with it. Feeds normally saves data into Drupal entities, but maybe (we thought) there was a way to have the data be passed into SearchAPI for indexing, by writing a custom Feeds plugin?  However, we found we had a funny problem of the sort that you don’t consider the existence of until you stumble on it: Feeds works on cron runs, pulling in data from a remote source and saving it into Drupal somehow. But SearchAPI also works on cron runs, pulling data in, usually entities. How do you get two processes to communicate when they both want to be the active participant?

With time pressing, we took the simple option: we defined a custom entity type for Feeds to put its data into, and SearchAPI to read its data from. (We could have just used a node type, but then there would have been an ongoing burden of needing to ensure that type was excluded from any kind of interaction with nodes.)

Essentially, this custom entity type acted like a bucket: Feeds dumps data in, SearchAPI picks data out. As solutions go, not the most massively elegant, at first glance. But if you think about it, if we had gone down the route of SearchAPI fetching from RSS directly, then re-indexing would have been a really lengthy process, and could have had consequences for the performance of the sites whose content was being slurped up. A sensible approach would then have been to implement some sort of caching on our server, either of the RSS feeds as files, or the processed RSS data. And suddenly our custom entity bucket system doesn’t look so inelegant after all: it’s basically a cache that both Feeds and SearchAPI can talk to easily.

There were a few pitfalls. With Search API, our search index needed to work on two entity types (nodes and the custom bucket entities), and while Search API on Drupal 7 allows this, its multiple entity type datasource handler had a few issues we needed to iron out or learn to live with.

The good news though is that the Drupal 8 version of Search API has the concept of multi-entity type search indexes at its core, rather than as a side feature: every index can handle multiple entity types, and there’s no such thing as a datasource for a single entity type.  With Feeds, we found that not all the configuration is exportable to Features for easy deployment. Everything about parsing the RSS feed into entities can be exported, except the actual URL, which is a separate piece of setup and not exportable. So we had to add a hook_updateN() to take care of setting that up.

The end result though was a site search that seamlessly returns results from multiple sites, allowing users to work with a network of disparate sites built on different technologies as if they were all the same thing. Which is what they were probably thinking they were all along anyway.

Author: Joachim Noreiko

Jan 02 2014
Jan 02

I took over a Drupal 7 project building a web application for college students to upload original videos about their school, and for schools to manage, group, and share the videos.

It's a startup privately funded by the principal, and we are literally working on a shoestring. My previous experience with media in Drupal led the principal to contact me via LinkedIn.

When it came time to build a video playlist in Drupal Views for JW Player version="6.7.4071" (formerly known as Longtail Video), I found very little useful documentation. In fact, someone suggested that those who know how are not interested in sharing their knowlege. -- but not me smiley

There are a couple of videos on YouTube by Bryan Ollendyke for Drupal 6. But a lot has changed in Drupal since then.

The Goal:

Back to the playlist: Site admins can mark a video featured by ticking a checkbox on the custom video content type. Now I want to display those featured videos as a playlist.

JW Player provides clear documentation about how to embed the player, and how to construct and load the playlist via a RSS feed.

The structure of the RSS file:

<rss version="2.0" xmlns:jwplayer="http://rss.jwpcdn.com/">
<channel>

  <item>
    <title>Sintel Trailer</title>
    <description>Sintel is a fantasy CGI from the Blender Open Movie Project.</description>
    <jwplayer:image>/assets/sintel.jpg</jwplayer:image>
    <jwplayer:source file="/assets/sintel.mp4" />
  </item>

</channel>
</rss>

There are various threads on drupal.org discussing video playlists for JW Player. None of them were particularly useful for me.

I constructed my Drupal View to to create a page display with a URL path and display fields:

jwplayer aml playlist drupal view

I tried using the Views Data Export module (a Drupal 7 successor of the Views Bonus Pack export functions) but it was difficult to create the proper syntax, particularly the xml namespace tag 

xmlns:jwplayer="http://rss.jwpcdn.com/"

I found the Views Datasource sub-module views_xml.module workable - mostly. I could add the namespace tag. The old videos by Bryan Ollendyke suggested the strategy to use the Views field labels for the xml tags like <title> and <description>.

I added the xml namespace in Format: XML data document: Settings:

jwplayer drupal view xml namespace

and set up the root and child syntax there too:

jw player xml root and child settings

But there were still two problems:

  1. The Video module does not expose the derivative transcoded video filenames to Views
  2. The JW Player syntax for the file source <jwplayer:source file="/assets/sintel.mp4" /> does not follow the convention of an opening and closing tag.

I used the Views PHP module to locate the filename from the node nid that was available in the the View. Yes, I could have written custom Views plugins and handlers, but remember: shoestring budget. It only took 3 queries:

$video_fid = db_query('SELECT field_video_fid FROM `field_revision_field_video` WHERE entity_id = ' . $data->nid)->fetchColumn();

$video_output_fid = db_query('SELECT output_fid FROM `video_output` WHERE original_fid = ' . $video_fid . ' ORDER BY output_fid DESC')->fetchColumn();

$video_uri = db_query('SELECT uri FROM `file_managed` WHERE fid = ' . $video_output_fid)->fetchColumn();

$video_source = str_ireplace("public://",$base_url,$video_uri);

return $video_source;

  1. Locate the original video_fid in the field_revision_field_video table
  2. Locate the video_output_fid (a different number generated by the Video module) in the video_output table
  3. Get the video_uri in the file_managed table

(yes we could use SQL JOIN - but we're only listing a few videos; TODO: rewrite the queries with placeholders for better security)

I considered creating php template files for these Views fields. But since I planned to embed the player in a custom block, I decided the parsing and processing could take place there.

Now we have the video file uri wrapped in tags <php></php> (from the label of the Views php field - it could be any arbitrary label, and we will be replacing it later).

The result of this view is something like:

<?xml version="1.0" encoding="utf-8"?> <rss version="2.0" xmlns:jwplayer="http://rss.jwpcdn.com/"> <channel> <item> <title>Who doesn&#039;t like free dessert?! </title> <description>Manhattan College</description> <jwplayer:image>http://[domain]/sites/default/files/[path]/thumbnail-690_0004.jpg</jwplayer:image> <php>path/690/MVI_4592_mp4_1384800425.mp4</php> </item> </channel>

Notice that it is not a valid XML or RSS document because there are no closing tags for the xml or the rss - but we will fix that later. Also I have used the value of the school field to rewrite the description field; but I might restore the description text in the playlist.

At first I thought I might use the view to write a file to the filesystem so I set the path to sites/default/files/playlists/featured_videos_playlist.rss.xml

Next, I had to get the playlist into the player.

I created a block with PHP code (or you can do it with a custom module and hook_block_info - but you already know I was working fast and cheap):

<?php

global $base_url;

$url = $base_url . "/sites/default/files/playlists/featured_videos_playlist.rss.xml";

$xml = fopen($url, "r");  

$playlist = stream_get_contents($xml);

$playlist = str_ireplace('<php>','<jwplayer:source file="/sites/default/files/',$playlist);

$playlist = str_ireplace('</php>','"/>',$playlist);

$playlist .= '</rss>';

file_put_contents('public://playlists/featured_videos_playlist.rss',$playlist);

?>

<div id="myElement">Loading the player ...</div>

<script type="text/javascript">

jwplayer.key="[put your key here if you have one]";

jwplayer("myElement").setup({

    playlist: "<?php print $base_path . '/sites/default/files/playlists/featured_videos_playlist.rss' ?>",

    listbar: {

        position: 'bottom',

        size: 80

    },

    height: 380,

    width: 420

    });

</script>

The block goes through 7 steps:

  1. Load the JW Player Javascript library
  2. Read the View output with PHP's fopen function (yes it works even though there is no actual file at the uri). I tried file_get_contents, but that failed because there was no actual file.
  3. Create the specialized jwplayer:source file tag from the file uri we placed within the <php></php> labels
  4. Close the rss tag (no need to close the xml tag)
  5. Write the playlist file to the filesystem (tip - use a different filename than the path in the View!!!)
  6. Embed the player in a div in the block
  7. Load the playlist we created

The generated playlist file (for the first item):

<?xml version="1.0" encoding="utf-8"?>

<rss version="2.0" xmlns:jwplayer="http://rss.jwpcdn.com/">

<channel>

  <item>

    <title>Who doesn&#039;t like free dessert?! </title>

    <description>Manhattan College</description>

    <jwplayer:image>http://[domain]/sites/default/files/[path]/thumbnail-690_0004.jpg</jwplayer:image>

    <jwplayer:source file="/sites/default/files/[path]/MVI_4592_mp4_1384800425.mp4"/>

  </item>

</channel>

</rss>

Finally, I used Context to place the block on the desired (home) page. Some additional CSS was required. JW Player is also fully skinnable.

The result may be viewed on the Proud Campus home pageOn that page, and the user pages and video upload page, we are using a simple responsive theme Ember selected via the Themekey module, to be more mobile-friendly. The web site is under active development and has some "rough edges."

Debriefing:

The project I inherited uses the "premium" GoVideo theme by ThemeSnap and the Video module.

If I were to start fresh, I would not have chosen to use them. GoVideo is a good example of how not to build a Drupal application all in the theme templates. I have started to move many of the functions into custom modules. One benefit of purchasing the GoVideo theme is that it includes a key for the licensed version of JW Player. Also, the Video module makes some assumptions and decisions that become tricky to customize. I will replace them with a responsive theme and component media modules, to better configure the file management, transcoding and display of the videos.

However, I think that much of my solution can be useful in general, without depending on the Video Module or the GoVideo theme. Also, we are not using the Drupal JW Player module - GoVideo includes the player's JavaScript library within its theme, and we are loading it in the block from the preferred location in Libraries.

I'm not going to into detail on how the content type is created, it uses the video field provided by the Video module, but the video could have been attached to the nodes in other ways.

I'm also not going to elaborate on the transcoding process; we are using avconv (formerly ffmpeg) on our server. I have used Encoding.com in the past at a client's request, and the Brightcove and Ooyala video management services as well. Eventually we will use Amazon AWS as a CDN for the videos - when we get more funding wink

 

<?php

global $base_url;

$url = $base_url . "/sites/default/files/playlists/featured_videos_playlist.rss.xml";

$xml = fopen($url, "r");  

$playlist = stream_get_contents($xml);

$playlist = str_ireplace('<php>','<jwplayer:source file="/sites/default/files/',$playlist);

$playlist = str_ireplace('</php>','"/>',$playlist);

$playlist .= '</rss>';

file_put_contents('public://playlists/featured_videos_playlist.rss',$playlist);

?>

<div id="myElement">Loading the player ...</div>

<script type="text/javascript">

jwplayer.key="[put your key here if you have one]";

jwplayer("myElement").setup({

    playlist: "<?php print $base_path . '/sites/default/files/playlists/featured_videos_playlist.rss' ?>",

    listbar: {

        position: 'bottom',

        size: 80

    },

    height: 380,

    width: 420

    });

</script>

The block goes through 7 steps:

  1. Load the JW Player Javascript library
  2. Read the View output with PHP's fopen function (yes it works even though there is no actual file at the uri). I tried file_get_contents, but that failed because there was no actual file.
  3. Create the specialized jwplayer:source file tag from the file uri we placed within the <php></php> labels
  4. Close the rss tag (no need to close the xml tag)
  5. Write the playlist file to the filesystem (tip - use a different filename than the path in the View!!!)
  6. Embed the player in a div in the block
  7. Load the playlist we created

The generated playlist file (for the first item):

<?xml version="1.0" encoding="utf-8"?>

<rss version="2.0" xmlns:jwplayer="http://rss.jwpcdn.com/">

<channel>

  <item>

    <title>Who doesn&#039;t like free dessert?! </title>

    <description>Manhattan College</description>

    <jwplayer:image>http://[domain]/sites/default/files/[path]/thumbnail-690_0004.jpg</jwplayer:image>

    <jwplayer:source file="/sites/default/files/[path]/690/MVI_4592_mp4_1384800425.mp4"/>

  </item>

</channel>

</rss>

(the school tag is ignored

Finally, I used Context to place the block on the desired (home) page. Some additional CSS was required. JW Player is also fully skinnable.

The result may be viewed on the Proud Campus home page. On that page, and the user pages and video upload page, we are using a simple responsive theme Ember selected via the Themekey module, to be more mobile-friendly. The web site is under active development and has some "rough edges."

Debriefing:

My project uses the "premium" GoVideo theme by ThemeSnap and the Video module. If I were to start fresh, I would not have chosen to use them. Govideo is a good example of how not to build a Drupal application all in the theme templates. I have started to move many of the functions into custom modules. One benefit of purchasing the Govideo theme is that it includes a key for the licensed version of JW Player. Also, the Video module makes some assumptions and decisions that become tricky to customize. I would have used a nice responsive theme and component media modules, to better configure the file management, transcoding and display of the videos. However, I think that much of my solution can be useful in general, without depending on the Video Module or the Govideo theme. Also, we are not using the Drupal JW Player module - GoVideo includes the player's JavaScript library within its theme, and we are loading it in the block from the preferred location in Libraries.

I'm not going to into detail on how the content type is created, it uses the video field provided by the Video module, but the video could have been attached to the nodes in other ways.

I'm also not going to elaborate on the transcoding process; we are using avconv (formerly ffmpeg) on our server. I have used Encoding.com in the past at a client's request, and the Brightcove and Ooyala video management services as well. Eventually we will use Amazon AWS as a CDN for the videos - when we get more funding wink

Jul 27 2012
Jul 27

If you receive our newsletter, you may have noticed that you recently got a HUGE list of posts we've written recently. Well, except that they weren't all really that recent — some of those we two months old, and every week in between. Our regular newsletter is sent out automatically based on our RSS feed, and it turns out that our RSS feed was broken. Once we tracked it all down and got it fixed, all of the posts that had never gotten queued up for the newsletter shot out in one big go. Sorry about that. Aside from the crazy long newsletter though, I thought I'd share how I got this sorted out, because this is the kind of problem that can happen to anyone, and it is really annoying to track down.

When we finally realized there was a problem, we didn't know what was causing the problem, or therefor how to fix it. The error that Drupal.org was showing when trying to update the feed was "The feed from Drupalize.Me seems to be broken, because of error "Reserved XML Name" on line 1." That wasn't really helpful, but a little bit of Googling pointed us in the direction of having a space in the feed. Lo and behold when we went back and looked at the feed itself, you could see that there was a blank space right at the beginning of the first line. Hm, where did that come from?

We hadn't made any changes to our view for the feed, and we didn't have any custom theming going on for it, but we checked those things out anyway. Nothing proved helpful. Trying to review all of the code on our site for an extra space was massively daunting. Then it struck me that I could narrow my search very easily because we use version control. We knew the feeds worked fine originally, so we just need to check the changes between it working correctly and not. I tracked down the rough date of when the feeds started to fail, and then I found the point in time in our code where it worked correctly. We use tags for our production code, so on my local I checked out a tag that worked right (3.3.0). I rolled forward on the tags until I saw when the feed broke (3.3.2). Now since this was a hot fix release, the changes in it were pretty small and in only a few files. A quick look at the diff on the files and I saw the mistake right away: a space was accidentally added to the top of our template.php file, right before the opening PHP tag. That added a space to the beginning of ALL pages on the site.

Now, if this hadn't been so quick to find with my basic method here, the next thing to do would have been to use a really handy Git command called bisect. If you've never used bisect before, it lets you take two endpoints in your code, and then goes halfway between those two points. You can then test if the bug is still there. You can keep bisecting the commits until you zero in on the commit that introduced the bug.

So, luckily, once I remembered that version control is my friend in so many ways, I quickly found the bug, fixed it and rolled out the new hot fix. Our feeds are now working properly and we can share with the world again.

Apr 25 2011
Apr 25

When a client asks for a way to pull content onto a site through RSS, the obvious choice is to use Drupal's Feeds module. I've never been really in love with this module but it does the job well. We recently had an interesting case that required extending the normal functionality of feeds to interact with custom content types.

The scenario was as follows:

  • The site lists many organizations and the events of those organizations.
  • Site users connect with and follow events and news from the various organizations.
  • Each organization may run several websites each having its own set of RSS feeds.
  • The organizations desired the ability to have their content added to their pages dynamically and without effort.
  • The user would visit the organization's page and see a list of news and articles from one or more of that organization's feeds.

The idea is fairly straight forward. But feeds does not support this kind of association by default.

If you're not familiar with feeds here's a brief rundown of how it works:

  1. Feeds allows you to designate a content type to be the source of a feed, or it will create a feed content type for you.
  2. You then create new nodes of this content type, adding the URL of the feed to be imported to each node.
  3. A designated times, the feed importer will be used to fetch information from the designated RSS URL. The data will be added to Drupal as nodes. You can choose what kind of node you would like the imported data to be.

This was a Drupal 7 site and our idea was to use references to reference each feed importer to the organization in question. For example the feed importers for developer.apple.com would reference the organization node for Apple as would the feed importer for news.apple.com while the feed importers for developer.microsoft.com and news.microsoft.com would reference the Microsoft organization node.

Make sense so far?

We then created a new content type for partner's news called Partner News. To this we added normal title, body, date, and ID information and also another reference field for organization. What we really wanted was for the partner news nodes to automatically inherit the reference field from the importer that created them. So playing off the example above, we wanted each news item that was added to Drupal from the feed at news.apple.com to inherit the reference to the Apple organization node so that we could later create a view on the Apple organization node that displayed all the imported feeds associated with Apple.

The trick is that this function didn't exist. Lucky for us Feeds module provides some useful hooks to extend its base functionality.

The custom module we created is fewer than 50 lines if you ignore all the comments.

The following screen shots show a typical Feeds importer setup.

In this case I am attaching my feed importer to content type called importer. This means that when I create a new importer node, I will see that feeds has added a new field to the content type giving me a place to add the URL of the feed to be imported.

Under settings I designate that imported nodes should be Partner News nodes. That means that each item in an RSS feed that is imported will become its own Partner News node. I also set Feeds to update nodes rather than replace them or creating new ones if it finds duplicate data.

Finally we designate the mapping. The mapping defines to feeds what elements from an imported RSS element should be added to what part of the new node. Some of these are pretty obvious. We map title to title, date to date, description to body, and GUID to GUID. This last one (GUID) provides a unique identifier for updating feed data and is required if you want the nodes to update rather than duplicate.

But what we want doesn't exist. We want to see a source element that says something like “Feed Importer's Organization Reference” and a target that says something like “Organization Reference”, so that we can map from one to the other.

To do this start a custom module in the standard fashion (http://drupal.org/node/1074360). I'll call my module feedmapper. In feedmapper.module add the following function:

<?php
/**
* Implements hook_feeds_parser_sources_alter().
*/
function feedmapper_feeds_parser_sources_alter(&$sources, $content_type) {
  $sources['field_importer_reference'] = array(
    'name' => t('Organizations\'s NID'),
    'description' => t('The node ID of the partner.'),
    'callback' => 'feedmapper_get_organization_nid',
  );
}

This adds a new source to the dropdown on the feed importer configuration.

The callback describes a function that will actually handle the data processing. You should prefix it with the name of you module but it can be named anything that makes sense. I haven't written this function yet, but we'll get to that shortly.

Next I'll add a function that specifies a new target.

/**
* Implements hook_feeds_processor_targets_alter().
*/
function feedmapper_feeds_processor_targets_alter(&$targets, $entity_type, $bundle_name) {
  $targets['field_importer_reference'] = array(
    'name' => 'Organization Reference',
    'description' => 'the node reference for the partner',
    'real_target' => 'field_importer_reference', // Specify real target field on node. This is on the content type.
    'callback' => 'feedmapper_feeds_set_target',
  );
}

Note that the source and the target both reference the same field field_importer_reference. This is due to the fact that I am reusing the field across content types. If you had different field names for each content type, you would need to make the target and source point to the specific field names you created.

Now you can assign this mapping. Of course it doesn't do anything yet because we haven't written the appropriate callbacks.

The set method is actually pretty easy because feeds handles that for us providing we pass the correct data at first. We need to focus on retrieving the correct node Id from the feed importer. To do this we access a property of the feed object. This property is feed_nid which as you can guess returns the value of the feed's node id. But now that we have the nid, retrieving another field's data is fairly trivial, we just need to make sure we're using the correct type of node so we run a check on the node type and then get the field in question:

/**
* Find the node id of the feed source and use it to find the associated organization.
*/
function feedmapper_get_organization_nid(FeedsSource $source) {
  $nid = $source->feed_nid;
  $feed = node_load($nid);
  if ($feed->type == 'importer') { //this needs to be the name of the importer content type.
    $partner_nid = $feed->field_importer_reference;
  }
  else {
    $partner_nid = NULL;
  }
  return $partner_nid;
}/**
* Implements hook_feeds_set_target().
*/
function feedmapper_feeds_set_target($source, &$entity, $target, $value) {
  $entity->$target = $value;
}

And that's it. Now the creating of a feed importer is tied to an organization and every time news is imported via feeds the incoming news item is automatically linked to its parent organization.

I've attached a working version of the module outlined here along with a Feature that should get you started.

Good luck.

AttachmentSize 3.52 KB 953 bytes
Oct 08 2010
Oct 08

I was recently asked to add a simple twitter feed to a Drupal 5 site I built a few years ago. You know the sort of thing - a block in the sidebar with the 5 most recent tweets from the organisation's twitter account. Unfortunately, this turned out to be one of those requests that sounds like it's going to be really easy, but is not (especially on Drupal 5). I came up with a solution which does the job quite nicely, but it's a bit of a hack; I wouldn't do it like this on a new site - but then of course I wouldn't be using Drupal 5.

There are several twitter modules for Drupal, as you'd expect. The twitter module does have a Drupal 5 version, but it's a minimal dev release and doesn't seem to offer much of the functionality of the other releases. The other module which sounded promising was twitter pull but this has no D5 release at all. I had a think about doing a backport, but this was supposed to be a quick favour and with the imminent release of D7 (ahem!), working on Drupal 5 code hardly seems like an investment in the future.

Then I checked, and found that twitter provides RSS feeds for users' timelines, and obviously D5 has a perfectly functional RSS aggregator module built-in. The RSS feed looks something like this:

http://twitter.com/statuses/user_timeline/30355784.rss

...so I set this up as a feed (admin/content/aggregator), and this automatically provided a block. This works okay, but the problem is that each item ends up like this:

<a href="http://twitter.com/mcdruid_co_uk/statuses/26748027524">mcdruid_co_uk: Drupal Quiz module: "action to be preformed after a user has completed Quiz: Ban IP address of current user" - seems a bit extreme, shirley?</a>

...so we end up with the entire tweet being a link, pointing at a page displaying just that one tweet. This is not exactly what I was hoping for.

So here's the slightly nasty hack; like all well-written Drupal modules, aggregator runs its output through a theme function, which can be overridden:

<?php
function theme_aggregator_block_item($item, $feed = 0) {
  global $user;
 
  if ($user->uid && module_exists('blog') && user_access('edit own blog')) {
    if ($image = theme('image', 'misc/blog.png', t('blog it'), t('blog it'))) {
      $output .= '<div class="icon">'. l($image, 'node/add/blog', array('title' => t('Comment on this news item in your personal blog.'), 'class' => 'blog-it'), "iid=$item->iid", NULL, FALSE, TRUE) .'</div>';
    }
  }
 
  // Display the external link to the item.
  $output .= '<a href="'. check_url($item->link) .'">'. check_plain($item->title) ."</a>\n";
 
  return $output;
}
?>

We wouldn't necessarily want to mess with the output of feeds other than this twitter one, so we need a way of identifying items from this feed. I noticed that twitter's RSS feed prepends the twitter username to each post, so we can use this to identify them.

I borrowed a function from the twitter module which deals with twitter @usernames and #hashtags. After that, running the tweet through the default input filter strips out potential nasties, and also converts URLs to links (assuming you have this switched on in your default input filter). The item includes a timestamp, so I used that to append a date/time. I didn't want the blog it stuff, so my theme code (in template.php) ended up looking like this:

define('ORGNAME_TWITTER_USERNAME',		'orgname');
 
/**
 * theme aggregator items
 */
function mytheme_aggregator_block_item($item, $feed = 0) {
  // horrible hack, but catch items which are part of the twitter feed
  $twitter_username = variable_get('orgname_twitter_username', ORGNAME_TWITTER_USERNAME);
  $tweet_prefix = "$twitter_username: ";
  if (strpos($item->title, $tweet_prefix) === 0) {
    // chop the username part from the start
    $tweet = substr($item->title, strlen($tweet_prefix));
    // @usernames
    $tweet = _mytheme_twitter_link_filter($tweet);
    // #hashtags
    $tweet = _mytheme_twitter_link_filter($tweet, '#', 'http://search.twitter.com/search?q=%23');
    // add date
    $tweet .= ' <span class="date">(' . format_date($item->timestamp) . ')</span>'; 
    // filter
    $tweet = check_markup($tweet);
    $output = trim($tweet);
  }
  else {
    // default implementation
    $output = '<a href="'. check_url($item->link) .'">'. check_plain($item->title) ."</a>\n";
  }
  return $output;
}
 
/**
 * stolen from twitter module (for D6)
 *
 * This helper function converts Twitter-style @usernames and #hashtags into 
 * actual links.
 */
function _mytheme_twitter_link_filter($text, $prefix = '@', $destination = 'http://twitter.com/') {
  $matches = array(
    '/\>' . $prefix . '([a-z0-9_]{0,15})/i',
    '/^' . $prefix . '([a-z0-9_]{0,15})/i',
    '/(\s+)' . $prefix . '([a-z0-9_]{0,15})/i',
  );
  $replacements = array(
    '><a href="' . $destination . '${1}">' . $prefix . '${1}</a>',
    '<a href="' . $destination . '${1}">' . $prefix . '${1}</a>',
    '${1}<a href="' . $destination . '${2}">' . $prefix . '${2}</a>',
  );
  return preg_replace($matches, $replacements, $text);
}

The result is nicely filtered and formatted tweets, with @usernames, #hashtags and URLs all converted to links.

If you don't like the way the aggregator module provides the block for the feed, you can always take what aggregator outputs and manipulate it with your own module code, e.g.:

  // get the block content from aggregator module
  $block = module_invoke('aggregator', 'block', 'view', 'feed-1');
  $block_content .= $block['content'];

It'll do the job until the site's upgraded to a newer version of Drupal.

Aug 15 2010
Aug 15

This lesson shows how to automatically import content from RSS or Atom feeds with the help of the Feeds module. The process relies on cron to run periodically so that the feed processor can check for new content. The feed processor creates regular Drupal nodes from the items contained in the feed. You can choose to have the imported items published by default or not. Once the items have been imported as nodes you can then create custom paths to those using Pathauto and/or customize the display using the Views module.

Note: Click the 'full screen' icon (to the right of the volume control) in order to watch online at full 1280x720 resolution.

Video Links

Flash Version

Jan 14 2010
Jan 14

The Views module in Drupal provides a Feed display that will allow for site owners to publish the content on their sites as RSS feeds. However, unlike a page or block display, the feed display does not allow for addition of custom tags inside the feed using the Views administration UI. There is however a way to add custom tags, attributes and values into the generated feed using hook_nodeapi.

Hook_nodeapi supports an 'rss item' operation which gets called whenever an item is inserted into the feed during the generation of a feed. The corresponding nodeapi call can be used by modules to insert custom tags to the feed.

For example, by inserting the following 'rss item' case to the nodeapi hook in a module

    case 'rss item':
      $custom_tags = some_function_call();
      $rss_items = array();
      $rss_items[] = array(
        'key' => 'key_name',
        'attributes' => array(
          'attribute1_name' => $attribute1_value,
          'attribute2_name' => $attribute2_value,
        ),
        'value' => $key_value,
      ); 
    return $rss_items;

you will be able to insert a tag

key_value

For the 'rss item' nodeapi hook the function should return an array of items that are to be inserted corresponding to each node in the feed. Each item in the array should be an array of one key, zero or more attributes and optionally a value.

For more complex capabilities you will have to use the Feeds module instead of the default core feed functionality.

Post your comments / questions

Aug 24 2008
Aug 24

I will be writing a few posts on connecting silverlight and drupal in which I will replicate my main site. I will be doing this piece by piece as I investigate each drupal module that I am using and how I can request the appropriate data to populate my silverlight controls. I have outlined the basics needed to accomplish this in my previous post and will be focusing on the new information that is needed to complete each new piece.

Drupal's aggregator module is a core module that is used for fetching syndicated content from other sites. However, it has the ability to provide an xml syndication button for your drupal site if you decide to show the syndication block. There is much more functionality that this module can provide, but all I needed was a simple link to my sites syndication. After doing some initial research I did not find a way to call some method that returned my sites syndication. So I decided to 'cheat' a little bit and produce my sites syndication link manually (at least I did not hard code this!). Here is the php below:

$mainSiteRSS = variable_get('site_name', 'Drupal');
$mainSiteRSS .= url('rss.xml');
return array('rssLoaded', $mainSiteRSS);

I simply grab the sites name, append 'rss.xml' and return a correctly formatted url string to my silverlight application. I then tie that url to my rss button:

private void rssLoaded(object feedSite)
{
    string rssLocation = ((MethodToCall)feedSite).MethodName;
    Uri rssLink = new Uri("http://" + rssLocation, UriKind.RelativeOrAbsolute);
    HtmlPage.Window.Navigate(rssLink);            
}

Remember, I have a generic method that gets called when drupal returns its data that casts the data to a MethodToCall object as outlined in my previous post.

And that's it! This was one of the simpler controls and that is one reason that I decided to start with this one. The silverlight code was very simple as well; just a button that looks like the standard rss icon that was made using Expression Blend that has a click event that sends a request to drupal to return my sites syndication link, as outlined above in the php code. Here is the XAML for the rss icon:

<Button Click="rss_Click">
    <Button.Template>
        <ControlTemplate TargetType="Button">
            <Canvas>
                <Rectangle Width="50" Height="50" Fill="Orange" RadiusX="5" RadiusY="5" />
                <Canvas>
                    <Ellipse Width="10" Height="10" Fill="White" Canvas.Left="10" Canvas.Top="30" />
                    <Path Stroke="White" StrokeThickness="5" StrokeStartLineCap="Round" StrokeEndLineCap="Round" Data="M 15,20 a 15,15 90 0 1 15,15" />
                    <Path Stroke="White" StrokeThickness="5" StrokeStartLineCap="Round" StrokeEndLineCap="Round" Data="M 15,10 a 25,25 90 0 1 25,25" />
                </Canvas>
            </Canvas>
        </ControlTemplate>
    </Button.Template>
</Button>

I will post my full source code when I complete this entire replication of my site to silverlight. Until then, I will post the source for each control in a separate solution. If there is a specific drupal module that you would like converted to a silverlight control let me know and I will try to help you out!

AttachmentSize 141.97 KB 939 bytes 652 bytes
Oct 22 2005
Oct 22

Here's a bunch of smaller updates which probably don't warrant extra blog posts, but might be interesting nevertheless:

  • On the right-hand side of my homepage you now have three new blocks which should simplify site navigation. Each contains a weighted list of tags (tag clouds) for my blog, my podcast, and my photoblog, powered by Drupal's tagadelic module. Of course, the "traditional" navigation menu is still there.
  • There's also an RSS feed for comments on this site now, courtesy to Drupal's Comment RSS module.
  • On the lower left-hand side of my homepage, there's another new block. It shows where the visitors of my site come from in (almost) realtime. This is a service provided by MapStats (I already talked about it earlier).
  • I have uploaded updated versions of my bashrc, vimrc, and muttrc config files.
  • My podcast is now listed in iTunes (thanks Daniel Reutter for adding it). You can subscribe by clicking on this button: Subscribe to my podcast in iTunes!. The amount of people who subscribe to my podcast have almost doubled, figures went up from 30 subscribers to more than 50 today! If you're reading this and subscribed through iTunes: thanks ;)
  • My video iPod will (probably) be shipped on October 31, and it'll take a few more days until it arrives. Arghh, /me wants it now!
Oct 01 2005
Oct 01

Green Leaves
Two Flowers
Pipes

Bear in mind, I'm no professional photographer or something — I just happen to own a crappy 1.3 MegaPixel camera and take photos with it from time to time.

The photoblog is (again) implemented with/on my existing Drupal site by abusing the taxonomy system.

Comments are welcome.

About Drupal Sun

Drupal Sun is an Evolving Web project. It allows you to:

  • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
  • Facet based on tags, author, or feed
  • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
  • View the entire article text inline, or in the context of the site where it was created

See the blog post at Evolving Web

Evolving Web