May 02 2017
May 02

Custom Video Export/Import Process With Views and Feeds

In the Media Research Center's set of three main Drupal sites, MRCTV serves as our video platform where all videos are created and stored as nodes, and then using Feeds, specific videos are imported into the other two sites (Newsbusters and CNS News) as video nodes. Then, on NB and CNS, we use the Video Embed Field module with a custom VEF provider for MRCTV to play the embedded videos.

There are only specific videos that need to be imported into the destination sites, so a way to map channels between the two sites is needed. All three sites have a Channels vocabulary, a mapping is created between the appropriate channels. This mapping has two parts:

  1. A feed of channels terms on NB and CNS.
  2. A custom admin form that links source channels on MRCTV with target channels on the destination site.

On the receiving site side, in addition to the standard feed content, the following custom elements are needed for the Feeds import:

  1. The nid of the video node on MRCTV. This is used to create the URL that is put into the VEF field.
  2. The taxonomy terms for the Channels vocabulary terms in the destination sites (NB and CNS).

Since these are outside of the standard feed components, they will need to be added custom to the feed items.

I documented my custom Feeds importer on drupal.stackexchange, so you can see the code there.

MRCTV is finally in the process of being updated to D8 from D6 (insert derision here), so both the mapping form and the feed needed to be re-created. The first part of the structure is the channel mapping form. The following file VideoExportForm.php is placed in /modules/custom/video_export/src/Form:

/**
 * @file
 * Contains \Drupal\video_export\Form\VideoExportForm.
 */

namespace Drupal\video_export\Form;

use Drupal\Core\Config\ConfigFactoryInterface;
use Drupal\Core\Form\ConfigFormBase;
use Drupal\Core\Form\FormStateInterface;
use GuzzleHttp\Exception\RequestException;

class VideoExportForm extends ConfigFormBase {
  
  /**
   * {@inheritdoc}.
   */
  public function getFormId() {
    return 'video_export_settings';
  }
    
  /**
   * {@inheritdoc}
   */
  public function buildform(array $form, FormStateInterface $form_state) {
    $form = array();
    $channels = array();
    
    // Get list of channels.
    $terms =\Drupal::entityTypeManager()->getStorage('taxonomy_term')->loadTree('channels');
    foreach ($terms as $term) {
      $channels[$term->tid] = $term->name;
    }
    
    // Get config data from video_export.settings.yml.
    $config = \Drupal::config('video_export.settings');
    $mapping_config = \Drupal::config('video_export.mappings');
    $sites = $config->get('sites');
    
    foreach($sites as $site => $site_data) {
      // Get channels list.
      try {
        $response = \Drupal::httpClient()->get($site_data['channel_url'], array('headers' => array('Accept' => 'text/plain')));
        $data = $response->getBody();
        if (empty($data)) {
          return FALSE;
        }
      }
      catch (RequestException $e) {
        return FALSE;
      }
  
      $channel_data = new \SimpleXMLElement($data);
      foreach ($channel_data->channel as $channel) {
        $channel_name = $channel->name->__toString();
        $channel_tid = $channel->tid->__toString();
        $target_channels[$channel_tid] = $channel_name;
      }
      // Sort array alphabetically by element.
      asort($target_channels, SORT_STRING);
  
      $target_channel_options = array();
      $target_channel_options[0] = "No Channel";
      foreach ($target_channels as $target_tid => $target_name) {
        $target_channel_options[$target_tid] = $target_name;
      }
  
      //Get mappings from mappings conifg.
      $mappings = $mapping_config->get('sites');
      foreach ($mappings[$site]['mappings'] as $mrctv_channel => $target_channel) {
        $mapping_defaults[$mrctv_channel] = $target_channel;
      }
  
      $form[$site] = array(
        '#type' => 'details',
        '#title' => t($site . ' Channel Mappings'),
        '#description' => t('Map MRCTV channels to ' . $site . ' channels'),
        '#collapsible' => TRUE,
        '#collapsed' => TRUE,
        '#tree' => TRUE,
      );
  
      // Loop through all of the categories and create a fieldset for each one.
      foreach ($channels as $id => $title) {
        $form[$site]['channels'][$id] = array(
          '#type' => 'select',
          '#title' => $title,
          '#options' => $target_channel_options,
          '#tree' => TRUE,
        );
        if (in_array($id, array_keys($mapping_defaults))) {
          $form[$site]['channels'][$id]['#default_value'] = intval($mapping_defaults[$id]);
        }
      }
    }
    
    // Get mapping configs.
    $xml = array();
    $mapping_config = \Drupal::config('video_export.mappings');
    $sites = $mapping_config->get('sites');
    $channel_mappings = $sites[$site]['mappings'];
    // Get video nodes that belong to one of the selected channels.
    $query = \Drupal::entityQuery('node')
      ->condition('status', 1)
      ->condition('type', 'video')
      ->condition('changed', REQUEST_TIME - 59200, '>=')
      ->condition('field_channels.entity.tid', array_keys($channel_mappings), 'IN');
    $nids = $query->execute();
    // Load the entities using the nid values. The array keys are the associated vids.
    $video_nodes = \Drupal::entityTypeManager()->getStorage('node')->loadMultiple($nids);

    foreach ($video_nodes as $nid => $node) {
      $host = \Drupal::request()->getSchemeAndHttpHost();
      $url_alias = \Drupal::service('path.alias_manager')->getAliasByPath('/node/' . $nid);
      // Get channels values.
      $channel_tids = array_column($node->field_channels->getValue(), 'target_id');
      $create_date = \Drupal::service('date.formatter')->format($node->getCreatedTime(), 'custom', 'j M Y h:i:s O');
      $item = array(
        'title' => $node->getTitle(),
        'link' => $host . $url_alias,
        'description' => $node->get('body')->value,
        'mrctv-nid' => $nid,
        'guid' => $nid . ' at ' . $host,
        'pubDate' => $create_date
      );
      // Check for short title and add it if it's there.
      if ($node->get('field_short_title')->value) {
        $item['short-title'] = $node->get('field_short_title')->value;
      }
      foreach ($channel_tids as $ctid) {
        $item[$site . '-channel-map'][] = $ctid;
      }
      $xml[] = $item;
    }

    return parent::buildForm($form, $form_state);
  }
  
  /**
   * {@inheritdoc}.
   */
  public function validateForm(array &$form, FormStateInterface $form_state) {
  
  }
  
  protected function getEditableConfigNames() {
    return ['video_export.mappings'];
  }
  
  /**
   * {@inheritdoc}
   */
  public function submitForm(array &$form, FormStateInterface $form_state) {
    $values = $form_state->getValues();
    $config = $this->config('video_export.mappings');
    $sites = array();

    foreach($values as $site => $mappings) {
      if (is_array($mappings)) {
        foreach ($mappings['channels'] as $mrctv_channel => $target_channel) {
          if ($target_channel != 0) {
            $sites[$site]['mappings'][$mrctv_channel] = $target_channel;
          }

        }
        $config->set('sites', $sites);
      }
    }
    $config->save();
  
    parent::submitForm($form, $form_state);
  }
}

The setting for the channels feeds on NB and CNS are stored in /modules/custom/video_export/config/install/video_export.settings.yml:

sites:
  newsbusters:
    channel_url: 'http://www.newsbusters.org/path/to/channels'
  cnsnews:
    channel_url: 'http://www.cnsnews.com/path/to/channels'
list_time: 24
        

Since this is an admin settings form, I extend the ConfigFormBase class. This adds some additional functionality over the standard FormBase class, similar to the way the system_settings_form() function does in D7 and older (see the change record for details).

As mentioned above the form does the following things:

  1. Reads the channels feed from the destination sites
  2. Creates a fieldset for each site with a select list for each MRCTV channel where the user can select the destination channel.
  3. Saves the mappings in config.

The next thing that is needed is the feed of video nodes that are available to be imported. After trying unsuccessfully to create a custom REST API endpoint, I ended up going with a Feeds display in Views. Out of the box I can create my feed, but I still need to add my custom elements. In D6, I used hook_nodeapi($op = 'rss item') to add my custom elements. In other feeds on D7 sites I've been able to use the Views RSS module with its provided hooks to add custom RSS elements, but as of now it is currently unusable for D8 due to one major issue.

Finally, since everything in D8 is based on OOP, I knew there had to be a way to override a Views class at some level, so after some searching, I decided to override the display plugin. I poked around in the Views code and found the RssFields class that is used for the field level display for a Feeds display, so I overrode that.

namespace Drupal\video_export\Plugin\views\row;

use Drupal\views\Plugin\views\row\RssFields;
use Drupal\Core\Form\FormStateInterface;
use Drupal\Core\Url;

/**
 * Renders an RSS item based on fields.
 *
 * @ViewsRow(
 *   id = "mrctv_rss_fields",
 *   title = @Translation("MRCTV Fields"),
 *   help = @Translation("Display fields as RSS items."),
 *   theme = "views_view_row_rss",
 *   display_types = {"feed"}
 * )
 */
class MRCTVRssFields extends RssFields {
  
  /**
   * Override of RssFields::render() with additional fields.
   *
   * @param object $row
   *
   * @return array
   */
  public function render($row) {
    $build = parent:: render();
    $item = $build['#row'];
    
    // Add MRCTV nid
    $item->elements[] = array(
      'key' => 'mrctv-nid',
      'value' => $row->nid,
    );
  
    // Add channels and their target nids. We can get them from $row->_entity.
    $site = $this->view->args[0];
    // Get MRCTV nids from view.
    $channel_tids = array_column($row->_entity->field_channels->getValue(), 'target_id');
    // Now, get destination tids from config.
    $mapping_config = \Drupal::config('video_export.mappings');
    $all_mappings = $mapping_config->get('sites');
  
    foreach($channel_tids as $mrctv_channel) {
      if(in_array($mrctv_channel, array_keys($all_mappings[$site]['mappings']))) {
        $item->elements[] = array(
          'key' => $site . '-channel-map',
          'value' => $all_mappings[$site]['mappings'][$mrctv_channel],
        );
      }
    }
    
    // Re-populate the $build array with the updated row.
    $build['#row'] = $item;
    
    return $build;
  }
}

As you can see, the override is fairly simple; all I needed to do was override the render() method. This method returns a render array, so all I do is get the built array from the parent class, add my custom elements to the #row element in the array, and return it.

One thing that I couldn't do simply in the views UI was select the nodes that should be in the feed based on the associate Channels vocabulary terms. These are dynamic, based on the mappings selected in the admin form, so I can't pre-select them in the view settings. This is where hook_views_query_alter() comes to the rescue.

/**
 * Implements hook_views_query_alter().
 */
function video_export_views_query_alter(Drupal\views\ViewExecutable $view, Drupal\views\Plugin\views\query\Sql $query) {
  if ($view->id() == 'video_export' && $view->getDisplay()->display['id'] == 'feed_1') {
    // First, we need to get the site parameter from the view.
    $site = $view->args[0];
    
    // Next, we need to get the saved config for the channel mapping.
    $mapping_config = \Drupal::config('video_export.mappings');
    $all_mappings = $mapping_config->get('sites');
    $tids = array_keys($all_mappings[$site]['mappings']);
   
    // Modify query to get nodes that have the selected nids, which are the array keys.
    $query->addWhere(NULL, 'node__field_channels.field_channels_target_id', $tids, 'IN');
  }
}

All I do here is get the saved mappings from config and add them to the views query as a WHERE condition to limit the feed items to the appropriate nodes.

One issue I ran into with the results was duplicate records. Since field_channels (the entity reference field for the Channels vocabulary) is multiselect, the query returns multiple records for each node if there are multiple Channels terms selected. There are display settings to show multiple items in one row, but they don't take effect here. I didn't dig far enough into the code to know for sure, but my guess is that the grouping happens at a higher layer in the views rendering process, so they don't take effect in this situation.

To get around this, I implemented hook_views_pre_render(). At this point in the process, the results have been built, so I just loop through them and remove duplicates.

/**
 * Implements hook_views_query_pre_render().
 */
function video_export_views_pre_render(Drupal\views\ViewExecutable $view) {
  $unique_nids = $new_results = array();
  
  // Loop through results and filter out duplicate results.
  foreach($view->result as $index => $result) {
    if(!in_array($result->nid, $unique_nids)) {
      $unique_nids[] = $result->nid;
    }
    else {
      $new_results[] = $result;
    }
  }
  // Replace $view->result with new array. Apparently views requires sequentially keyed
  // array of results instead of skipping keys (e.g. 0, 2, 4, etc), so we can't just
  // unset the duplicates.
  $view->result = $new_results;
}

As noted in the code comment, views seems to require a sequentially numbered array, so you can't just unset the duplicate keys and leave it as is, so I chose to just add each item to a new array. In retrospect, I could have just used PHP functions like array_splice() and array_filter(), but this method works just as well.

It should also be noted that the views hooks need to go in a *.views_execution.inc file, so this one is in /modules/custom/video_export/video_export.views_execution.inc.

All I do at this point is use the Job Scheduler module with Feeds in the destination sites to schedule the import at the desired interval, and the process runs by itself. 

Jan 02 2014
Jan 02

I took over a Drupal 7 project building a web application for college students to upload original videos about their school, and for schools to manage, group, and share the videos.

It's a startup privately funded by the principal, and we are literally working on a shoestring. My previous experience with media in Drupal led the principal to contact me via LinkedIn.

When it came time to build a video playlist in Drupal Views for JW Player version="6.7.4071" (formerly known as Longtail Video), I found very little useful documentation. In fact, someone suggested that those who know how are not interested in sharing their knowlege. -- but not me smiley

There are a couple of videos on YouTube by Bryan Ollendyke for Drupal 6. But a lot has changed in Drupal since then.

The Goal:

Back to the playlist: Site admins can mark a video featured by ticking a checkbox on the custom video content type. Now I want to display those featured videos as a playlist.

JW Player provides clear documentation about how to embed the player, and how to construct and load the playlist via a RSS feed.

The structure of the RSS file:

<rss version="2.0" xmlns:jwplayer="http://rss.jwpcdn.com/">
<channel>

  <item>
    <title>Sintel Trailer</title>
    <description>Sintel is a fantasy CGI from the Blender Open Movie Project.</description>
    <jwplayer:image>/assets/sintel.jpg</jwplayer:image>
    <jwplayer:source file="/assets/sintel.mp4" />
  </item>

</channel>
</rss>

There are various threads on drupal.org discussing video playlists for JW Player. None of them were particularly useful for me.

I constructed my Drupal View to to create a page display with a URL path and display fields:

jwplayer aml playlist drupal view

I tried using the Views Data Export module (a Drupal 7 successor of the Views Bonus Pack export functions) but it was difficult to create the proper syntax, particularly the xml namespace tag 

xmlns:jwplayer="http://rss.jwpcdn.com/"

I found the Views Datasource sub-module views_xml.module workable - mostly. I could add the namespace tag. The old videos by Bryan Ollendyke suggested the strategy to use the Views field labels for the xml tags like <title> and <description>.

I added the xml namespace in Format: XML data document: Settings:

jwplayer drupal view xml namespace

and set up the root and child syntax there too:

jw player xml root and child settings

But there were still two problems:

  1. The Video module does not expose the derivative transcoded video filenames to Views
  2. The JW Player syntax for the file source <jwplayer:source file="/assets/sintel.mp4" /> does not follow the convention of an opening and closing tag.

I used the Views PHP module to locate the filename from the node nid that was available in the the View. Yes, I could have written custom Views plugins and handlers, but remember: shoestring budget. It only took 3 queries:

$video_fid = db_query('SELECT field_video_fid FROM `field_revision_field_video` WHERE entity_id = ' . $data->nid)->fetchColumn();

$video_output_fid = db_query('SELECT output_fid FROM `video_output` WHERE original_fid = ' . $video_fid . ' ORDER BY output_fid DESC')->fetchColumn();

$video_uri = db_query('SELECT uri FROM `file_managed` WHERE fid = ' . $video_output_fid)->fetchColumn();

$video_source = str_ireplace("public://",$base_url,$video_uri);

return $video_source;

  1. Locate the original video_fid in the field_revision_field_video table
  2. Locate the video_output_fid (a different number generated by the Video module) in the video_output table
  3. Get the video_uri in the file_managed table

(yes we could use SQL JOIN - but we're only listing a few videos; TODO: rewrite the queries with placeholders for better security)

I considered creating php template files for these Views fields. But since I planned to embed the player in a custom block, I decided the parsing and processing could take place there.

Now we have the video file uri wrapped in tags <php></php> (from the label of the Views php field - it could be any arbitrary label, and we will be replacing it later).

The result of this view is something like:

<?xml version="1.0" encoding="utf-8"?> <rss version="2.0" xmlns:jwplayer="http://rss.jwpcdn.com/"> <channel> <item> <title>Who doesn&#039;t like free dessert?! </title> <description>Manhattan College</description> <jwplayer:image>http://[domain]/sites/default/files/[path]/thumbnail-690_0004.jpg</jwplayer:image> <php>path/690/MVI_4592_mp4_1384800425.mp4</php> </item> </channel>

Notice that it is not a valid XML or RSS document because there are no closing tags for the xml or the rss - but we will fix that later. Also I have used the value of the school field to rewrite the description field; but I might restore the description text in the playlist.

At first I thought I might use the view to write a file to the filesystem so I set the path to sites/default/files/playlists/featured_videos_playlist.rss.xml

Next, I had to get the playlist into the player.

I created a block with PHP code (or you can do it with a custom module and hook_block_info - but you already know I was working fast and cheap):

<?php

global $base_url;

$url = $base_url . "/sites/default/files/playlists/featured_videos_playlist.rss.xml";

$xml = fopen($url, "r");  

$playlist = stream_get_contents($xml);

$playlist = str_ireplace('<php>','<jwplayer:source file="/sites/default/files/',$playlist);

$playlist = str_ireplace('</php>','"/>',$playlist);

$playlist .= '</rss>';

file_put_contents('public://playlists/featured_videos_playlist.rss',$playlist);

?>

<div id="myElement">Loading the player ...</div>

<script type="text/javascript">

jwplayer.key="[put your key here if you have one]";

jwplayer("myElement").setup({

    playlist: "<?php print $base_path . '/sites/default/files/playlists/featured_videos_playlist.rss' ?>",

    listbar: {

        position: 'bottom',

        size: 80

    },

    height: 380,

    width: 420

    });

</script>

The block goes through 7 steps:

  1. Load the JW Player Javascript library
  2. Read the View output with PHP's fopen function (yes it works even though there is no actual file at the uri). I tried file_get_contents, but that failed because there was no actual file.
  3. Create the specialized jwplayer:source file tag from the file uri we placed within the <php></php> labels
  4. Close the rss tag (no need to close the xml tag)
  5. Write the playlist file to the filesystem (tip - use a different filename than the path in the View!!!)
  6. Embed the player in a div in the block
  7. Load the playlist we created

The generated playlist file (for the first item):

<?xml version="1.0" encoding="utf-8"?>

<rss version="2.0" xmlns:jwplayer="http://rss.jwpcdn.com/">

<channel>

  <item>

    <title>Who doesn&#039;t like free dessert?! </title>

    <description>Manhattan College</description>

    <jwplayer:image>http://[domain]/sites/default/files/[path]/thumbnail-690_0004.jpg</jwplayer:image>

    <jwplayer:source file="/sites/default/files/[path]/MVI_4592_mp4_1384800425.mp4"/>

  </item>

</channel>

</rss>

Finally, I used Context to place the block on the desired (home) page. Some additional CSS was required. JW Player is also fully skinnable.

The result may be viewed on the Proud Campus home pageOn that page, and the user pages and video upload page, we are using a simple responsive theme Ember selected via the Themekey module, to be more mobile-friendly. The web site is under active development and has some "rough edges."

Debriefing:

The project I inherited uses the "premium" GoVideo theme by ThemeSnap and the Video module.

If I were to start fresh, I would not have chosen to use them. GoVideo is a good example of how not to build a Drupal application all in the theme templates. I have started to move many of the functions into custom modules. One benefit of purchasing the GoVideo theme is that it includes a key for the licensed version of JW Player. Also, the Video module makes some assumptions and decisions that become tricky to customize. I will replace them with a responsive theme and component media modules, to better configure the file management, transcoding and display of the videos.

However, I think that much of my solution can be useful in general, without depending on the Video Module or the GoVideo theme. Also, we are not using the Drupal JW Player module - GoVideo includes the player's JavaScript library within its theme, and we are loading it in the block from the preferred location in Libraries.

I'm not going to into detail on how the content type is created, it uses the video field provided by the Video module, but the video could have been attached to the nodes in other ways.

I'm also not going to elaborate on the transcoding process; we are using avconv (formerly ffmpeg) on our server. I have used Encoding.com in the past at a client's request, and the Brightcove and Ooyala video management services as well. Eventually we will use Amazon AWS as a CDN for the videos - when we get more funding wink

 

<?php

global $base_url;

$url = $base_url . "/sites/default/files/playlists/featured_videos_playlist.rss.xml";

$xml = fopen($url, "r");  

$playlist = stream_get_contents($xml);

$playlist = str_ireplace('<php>','<jwplayer:source file="/sites/default/files/',$playlist);

$playlist = str_ireplace('</php>','"/>',$playlist);

$playlist .= '</rss>';

file_put_contents('public://playlists/featured_videos_playlist.rss',$playlist);

?>

<div id="myElement">Loading the player ...</div>

<script type="text/javascript">

jwplayer.key="[put your key here if you have one]";

jwplayer("myElement").setup({

    playlist: "<?php print $base_path . '/sites/default/files/playlists/featured_videos_playlist.rss' ?>",

    listbar: {

        position: 'bottom',

        size: 80

    },

    height: 380,

    width: 420

    });

</script>

The block goes through 7 steps:

  1. Load the JW Player Javascript library
  2. Read the View output with PHP's fopen function (yes it works even though there is no actual file at the uri). I tried file_get_contents, but that failed because there was no actual file.
  3. Create the specialized jwplayer:source file tag from the file uri we placed within the <php></php> labels
  4. Close the rss tag (no need to close the xml tag)
  5. Write the playlist file to the filesystem (tip - use a different filename than the path in the View!!!)
  6. Embed the player in a div in the block
  7. Load the playlist we created

The generated playlist file (for the first item):

<?xml version="1.0" encoding="utf-8"?>

<rss version="2.0" xmlns:jwplayer="http://rss.jwpcdn.com/">

<channel>

  <item>

    <title>Who doesn&#039;t like free dessert?! </title>

    <description>Manhattan College</description>

    <jwplayer:image>http://[domain]/sites/default/files/[path]/thumbnail-690_0004.jpg</jwplayer:image>

    <jwplayer:source file="/sites/default/files/[path]/690/MVI_4592_mp4_1384800425.mp4"/>

  </item>

</channel>

</rss>

(the school tag is ignored

Finally, I used Context to place the block on the desired (home) page. Some additional CSS was required. JW Player is also fully skinnable.

The result may be viewed on the Proud Campus home page. On that page, and the user pages and video upload page, we are using a simple responsive theme Ember selected via the Themekey module, to be more mobile-friendly. The web site is under active development and has some "rough edges."

Debriefing:

My project uses the "premium" GoVideo theme by ThemeSnap and the Video module. If I were to start fresh, I would not have chosen to use them. Govideo is a good example of how not to build a Drupal application all in the theme templates. I have started to move many of the functions into custom modules. One benefit of purchasing the Govideo theme is that it includes a key for the licensed version of JW Player. Also, the Video module makes some assumptions and decisions that become tricky to customize. I would have used a nice responsive theme and component media modules, to better configure the file management, transcoding and display of the videos. However, I think that much of my solution can be useful in general, without depending on the Video Module or the Govideo theme. Also, we are not using the Drupal JW Player module - GoVideo includes the player's JavaScript library within its theme, and we are loading it in the block from the preferred location in Libraries.

I'm not going to into detail on how the content type is created, it uses the video field provided by the Video module, but the video could have been attached to the nodes in other ways.

I'm also not going to elaborate on the transcoding process; we are using avconv (formerly ffmpeg) on our server. I have used Encoding.com in the past at a client's request, and the Brightcove and Ooyala video management services as well. Eventually we will use Amazon AWS as a CDN for the videos - when we get more funding wink

About Drupal Sun

Drupal Sun is an Evolving Web project. It allows you to:

  • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
  • Facet based on tags, author, or feed
  • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
  • View the entire article text inline, or in the context of the site where it was created

See the blog post at Evolving Web

Evolving Web