Upgrade Your Drupal Skills

We trained 1,000+ Drupal Developers over the last decade.

See Advanced Courses NAH, I know Enough
Jul 30 2016
Jul 30

On a recent project, we had to create multiple sitemaps for each of the domains that we have setup on the site. We came across some problems that we had to resolve because of the nature of our pURL setup.

Goals

  • We want all of the front pages from each subdomain to be added to the sitemap and we are able to set the rules for them on the XMLSitemap settings page.
  • We want to make sure that the URLs that we are adding to the other pages no longer show up in the main domain's sitemap.

Problems

1) Only On The Primary Domain

The XML sitemap module only creates one sitemap based on the primary domain.

2) Prefixes not Distinguished

Our URLs for nodes are setup so that nodes can be prefixed with our subdomain (pURL modifier) and XMLSitemap doesn't see our prefixes as being different sites. At this point, all nodes are added to every single domain's sitemap.

3) URL Formats

Our URLs are not in the correct format when being added to the sitemap. Our URLs should look like http://subdomain.domain.org/*, however, because we are prefixing them, they show up as http://domain.org/subdomain/*. We want our URLs to look like they are from the right sub-domain and not all coming from the base domain.

Solution

We were able to add the ability to create sitemaps for each of the 15 domains by adding the XMLSitemap domain module. The XLMSitemap domain module allows us to define a domain for each sitemap, generate a sitemap and serve it on the correct domain.

We added xmlsitemap-dont-write-empty-element-in-xml-sitemap-file-2545050-3.patch to prevent empty elements from being added to the sitemap.

Then we used a xmlsitemap_element_alter inside of our own custom module that looks something like this:




function hook_xmlsitemap_element_alter(array &$element, array $link, $sitemap) {
  $domain = $sitemap->uri['options']['base_url'];
  $url_parts = explode('//', $domain);
  $parts = explode('.', $url_parts[1]);
  $subdomain = array_shift($parts);

  $current_parts = explode('/', $link['loc']);
  $current_prefix = array_shift($current_parts);

  $modifiers = _get_core_modifiers();

  
  if (in_array($subdomain, array_keys($modifiers))) {
    
    
    if ($current_prefix != $subdomain && $current_prefix != '') {
      
      $element = array();
        return $element;
      }
    else {
      
      $pattern = $current_prefix . '/';
      $element['loc'] = $domain . str_replace($pattern, '', $link['loc']);
    }
  }
  else {
    
    
    if (in_array($current_prefix, array_keys($modifiers))) {
      $element = array();
      return $element;
    }
  }
}


function _get_core_modifiers() {
  if (!$cache = cache_get('subdomains')) {
    $result = db_query("SELECT id, value FROM {purl} WHERE provider = 'og_purl_provider'")->fetchAllAssoc('value');
    cache_set('subdomains', $result, 'cache', time() + 86400);
    return $result;
  }
  else {
    return $cache->data;
  }
?>

If you have any questions, suggestions, feel free to drop a comment below!

Jul 27 2016
Jul 27
TL;DR Last week, I had worked on and developed tests to ensure that the similar images are grouped in accordance to the Image Properties feature of the Vision API. The code is under review by the mentors, and I would continue on it once the review is done. Meanwhile, they also reviewed the “Fill Alt Text” feature issue, and approved it is good to go. This week, I have worked on developing tests for this issue.

An important feature that I have implemented in the Google Vision API module is the filling of Alt Text field of an image file entity by any of the four choices- Label Detection, Landmark Detection, Logo Detection and Optical Character Detection. My mentor suggested me to check the availability of the response and then fill the field, as we can not fully rely on the third party responses. With this minor suggestion being implemented, now its time to develop tests to ensure the functionality of this feature.

I started developing simple web tests for this feature, to ensure that the Alt Text field is properly filled in accordance to the choice of the user. It requires the selection of the four choices one by one and verify that the field is filled correctly. Thus we require four tests to test the entire functionality. I have added an extra test to ensure that if none of the options are selected then the field remains empty.

I created the image files using the images available in the simpletests. The images can be accessed through drupalGetTestFiles(). The filling, however, requires call to the Google Cloud Vision API, thus inducing dependency on the API key. To remove the dependency, I mocked the function in the test module, returning the custom data to implement the feature.

The first test ensures that the Label Detection feature returns correct response and the Alt Text field is filled correctly. The simpletest provides a list of assertions to verify it, however, I found assertFieldByName() to be most suitable for the purpose. It asserts the value of a field based on the field name. The second test ensures that the Landmark Detection feature works correctly. Similarly, the third and fourth test ensures the correct functionality of the Logo and the Optical Character Detection feature.

The fifth test which I have included perform tests when none of the options are selected. It ensures that under this case, the Alt Text field remains empty, and does not contain any unwanted values.
I have posted the patch covering the suggestions and tests on the issue queue Fill the Alt Text of the Image File using Google Vision API to be reviewed by my mentors. Once they review it, I would work on it further, if required.
Jul 27 2016
Jul 27

It's been 3 months now and we're finally approaching the last steps of our project - "Solving Merge conflict in Drupal 8" with Multiversion module. So far, we have launched two Open source Libraries, Implemented Simple LCA resolver and a simple merge resolver in the Conflict  module to find lowest common ancestor and merge the updates respectively. Now all development phase is almost done and the integration part starts where we have to integrate the Libraries we created in the first phase of Google Summer of Code16. The libraries we created were relaxed/lca and relaxed/merge

Let's take a look at the project's target and the solution we are implemeting:

Project Details: With the introduction of Multiversion module in Drupal 8, we have a very powerful content revision API that can handle branching and conflict DETECTION. But there is not yet a way to SOLVE revision conflicts.

Proposed Solution: We store a graph of all the updated revisions from an entity and find a base revision from this graph with which two entities can be compared. This process is based on 3-way merge algorithm. According to this algorithm, we shall compare two revisions of an entity with another revision or actual node to detect what has been modified in those revisions. If the new revisions has modified content over different lines, the updated content shall be merged in the new revision along with the unchanged data from previous revisions. This is why we compare two entities with a base entity, to find out which part has been modified in the new revisions. If both the new revisions has modified the same data, a merge conflict would occur and user would be allowed to solve the conflict in a 3-pane window. You can find more about the project and the approach in the 3rd week blog post at my blog.

Work we did last week: The multiversion module stores the revision status and other data in form of trees. To detect and solve conflicts, we need to convert this data in graphs. Last week, we worked on writing code which could create a graph from a given tree.

We wrote a recursive function to store all revision Ids from the tree in an array. Then we created nodes from those revision IDs. After the recursive function extracted the revision ID correctly, we wrote a function which creates edges between these nodes in the exact same manner in which they were connected in the tree. The code is pushed here.

Work we did this week: We tested the code we developed last week for the Conflict Module to convert the tree returned by getTree method of Multiversion module. After we made sure it’s returning the expected output, we merged the code into Multiversion module as a new method named getGraph. Other than that, we have written tests for it in the multiversion module as well to ensure it’s functionality with the module. We also implemented Travis CI integration for the conflict module. We’re also making sure that our code is working on other versions of drupal than 8.2. All the code is pushed here.

Target for the next week: We will start with creating a complex LCA resolver which will use the relaxedws/lca library for finding LCA. This resolver will be available to only those sites which uses multiversion module and so we will implement this resolver in multiversion module only. We’d also be writing tests for the complex LCA resolver this week only.

Jul 27 2016
Jul 27

This blog post summarizes week #10 of the Google Summer of Code 2016 project - Mailhandler.

In the last blog post, I was writing about the comment and demo module improvements. This blog post will provide an overview of the work done in the past 7 days. Even though the plan was to work mostly on UI/UX issues we ended up in code refactoring.

During the last meeting with my mentors we identified 3 key Inmail issues to work on: Lack of standard result in collaboration of analyzers, Support switching the user context and Provide a hook_requirements fed by plugin instances. We agreed those issues will provide better overall value in the integration of Mailhandler and Inmail modules. It will allow Inmail to implement ideas from Mailhandler, make them generic in a way both modules can benefit from.

Lack of standard result in collaboration of analyzers was identified as the main blocker to achieve analyzer collaboration. After a long discussion and 7 patches, it was finally committed. As a result, Inmail will have a default analyzer result which can be extended sequentially by all enabled analyzers. As this was a big change in core Inmail module, there are several follow-ups created.

Another Inmail issue Support switching the user context was dependent on the default analyzer issue. It uses AccountSwitcher service which completely switches the user context to the given user/account. That is done after the handlers run. In case the account switching mechanism was activated, we make sure it is switched back after the handlers-processing. On a handler level, we can use \Drupal::currentUser() and be sure it represents the identified user sender or an anonymous user otherwise.

Last but not least, I have been working on Provide a hook_requirements fed by plugin instances. The goal of this issue is to create a way for each of the Inmail plugins (analyzers, deliverers, handlers) to provide information about its runtime requirements. They can be PHP extension requirements, valid credentials etc. This information is displayed on “Status report” page (admin/reports/status) or processed by contrib modules like Monitoring.

Image removed.PGP Analyzer requirements

Inmail plugins can produce several issues with unmet requirements. As displayed in the picture above, PGP Analyzer needs gnupg PHP extension in order to work with signed messages. On the other side, an IMAP deliverer needs valid credentials to be functional.

Since the most of the Inmail issues mentioned above were committed, Mailhandler module will need adaption which is going to be my focus for this week. Also, I will analyze the code of the module and try to simplify it. Besides the mentioned, I will work on Inmail UI/UX issues which will be described in the next blog post.

Jul 26 2016
Jul 26
TL;DR In the past two weeks I had worked on using the Image Properties feature offered by the Google Cloud Vision API to group the image files together on the basis of the dominant color components filling them. In addition, I had also worked on detecting the image files and filling the Alternate Text field based on the results of Label/Landmark/Logo/Optical Character Detection, based on the demand of the end user. This week, I have worked on and developed tests to ensure that the similar images are grouped in accordance to the Image Properties feature of the Vision API.

At present, the Google Vision API module supports the Label Detection feature to be used as taxonomy terms, the Safe Search Detection feature to avoid displaying any explicit contents or violence and the User Emotion detection to detect the emotions of the users in their profile pictures and notify them about it.

I had worked on grouping the images on the basis of the dominant color component(Red, Green or Blue) which they are comprised of. I got the code reviewed by my mentors, and they approved it with minor suggestions on injecting the constructors wherever possible. Following their suggestions, I injected the Connection object instead of accessing the database via \Drupal::database().

After making changes as per the suggestions, I started developing simple web tests for this feature, to ensure that the similar images gets displayed under the SImilarContents tab. It requires the creation of new taxonomy vocabulary and adding an entity reference field to the image file entity. After the creation of the new Vocabulary and addition of the new field to the image file, I created the image files using the images available in the simpletests. The images can be accessed through drupalGetTestFiles(). The first test ensures that if the Vocabulary named ‘Dominant Color’ is selected, the similar images gets displayed under the file/{file_id}/similarcontent link.

The grouping, however, requires call to the Google Cloud Vision API, thus inducing dependency on the API key. To remove the dependency, I mocked the function in the test module, returning the custom data to implement the grouping.

To cover the negative aspect, i.e. the case when the Dominant Color option is not selected, I have developed another test which creates a demo vocabulary to simply store the labels, instead of the dominant color component. In this case, the file/{file_id}/similarcontent link displays the message “No items found”.
I have posted the patch covering the suggestions and tests on the issue queue to be reviewed by my mentors. Once they review it, I would work on it further, if required.
Jul 25 2016
Jul 25

Project Details: With the introduction of Multiversion module in Drupal 8, we have a very powerful content revision API that can handle branching and conflict DETECTION. But there is not yet a way to SOLVE revision conflicts.

Proposed Solution: We store a graph of all the updated revisions from an entity and find a base revision from this graph with which two entities can be compared. This process is based on 3-way merge algorithm. According to this algorithm, we shall compare two revisions of an entity with another revision or actual node to detect what has been modified in those revisions. If the new revisions has modified content over different lines, the updated content shall be merged in the new revision along with the unchanged data from previous revisions. This is why we compare two entities with a base entity, to find out which part has been modified in the new revisions. If both the new revisions has modified the same data, a merge conflict would occur and user would be allowed to solve the conflict in a 3-pane window. You can find more about the project and the approach in the 3rd week blog post at my blog.

Work we did last week: We are done with the simple merge resolver as well now. The task was to resolve merge conflicts in linear entities which drupal implements by default. The approach we took for this was to return the entity last created in the revision. This way, the entity which was created last would be returned as it is the latest entity. We have also written tests for it. The code for the library can be found on the github repository.

Work we did this week: The multiversion module stores the revision status and other data in form of trees. To detect and solve conflicts, we need to convert this data in graphs. This week, we have been working on writing code which could create a graph from a given array.

We wrote a recursive function to create an array to store all revision Ids from the array and then we created nodes from those revision IDs. Next we are working on a function which creates edges between these nodes. The code is pushed here in the getGraphTest.php file.

We are done with the code and testing it to make sure it’s returning the expected output. Once we are sure it’s returning the correct data, we can implement the code into the multiversion module.

Task for the next week: We will complete the tests, get the code implemented in the multiversion module to return graph from the tree and will also try to complete the complex LCA resolver code as much as possible. We need to complete the graph creating code first as it’d be used in all next steps i.e, to find LCA using relaxedws/lca Library and to resolve merge conflicts using relaxedws/merge library.

Jul 25 2016
Jul 25

Yesterday all the accepted sessions for DrupalCon Dublin were announced, and we are delighted to report that 5 of our 8 session proposals were accepted! With Acquia being the only company receiving more acceptances, we are extremely proud of our achievement.

Testament to our high standing in the Drupal community, we are the only Irish company speaking at DrupalCon Dublin. Our accepted sessions this year span a number of different tracks, namely Business, Horizons, Site Building, Being Human and Core Conversations, and cover topics from accessibility to remote working to building mobile apps with the Ionic framework. Congratulations to all our speakers!

Here's a quick run down of each session.

Building a co-lingual website - lessons learned from ireland.ie

Speaker: Alan Burke
Track: Site Building

2016 marks the centenary of the 1916 rising in Dublin, a pivotal year in Irish history, and is marked with a series of high-profile events commemorating the rising. ireland.ie is the official state website for the 1916 commemoration and runs on Drupal 7.

While English is the main language in Ireland, Irish is the first official language. A decision was taken to present both languages side by side wherever possible for the 1916 commemorations - including on the website. This session will focus on the unusual co-lingual [2 languages side-by-side] approach, and how Drupal made it possible. 

Choosing Drupal - insider advice from an Irish multinational

Speaker: Alan Burke & Aisling Furlong from Glanbia
Track: Business

Struggling to sell Drupal to clients? Ever wondered what goes into the decision making process when choosing a CMS?
In 2014, Glanbia selected Drupal as the CMS of choice for marketing sites. This session will outline the decision-making process used, and what Drupal agencies can learn when pitching Drupal. This is a joint session proposal between Annertech and Glanbia.

Bridging the PhoneGap: Getting Started Creating Hybrid Mobile Apps with Drupal and Ionic Framework

Speaker: Mark Conroy
Track: Horizons

With the advent of hybrid mobile apps, you can continue being a Drupal frontend developer and also build apps without needing to learn new technologies. The mobile web is quickly catching up with native apps. The mobile web is free, and open, and available to all of us right now and doesn't bind us to proprietary systems. With the many advances being made in this area, we can create great mobile experiences for users.

Future Directions for Drupal Accessibility

Speaker: Andrew Macpherson
Track: Core Conversations

Drupal has made great advances in accessibility over several major releases, and nowadays ranks as a leading implementation of web accessibility standards.  This session will encourage contributors to look ahead at future challenges and opportunities for accessibility during the faster 8.x (and 9.x) release cycle. 

Happiness is... remote working

Speaker: Anthony Lindsay
Track: Being Human

Many Drupal agencies have remote workers. Some are entirely distributed. Whilst remote working is beneficial to all concerned in so many ways, it does come with its own challenges. This talk will cover the journey I took when I moved from a typical 9-5 office job and joined Annertech, which is an entirely distributed Drupal agency. It will highlight the challenges I found: the good, the bad, the funny and the downright surprising, and offer as examples, my experiences for staying happy and healthy in what has the potential to be an isolating environment. 

Congratulations to Alan, Anthony, Andrew and Mark on their great achievement. We look forward to seeing these and all the other great sessions at DrupalCon Dublin in September. Hope to see you there!

Jul 23 2016
Jul 23

Ever since Andrew joined Annertech, he's been a champion of accessible web design and has ensured that accessibility has remained a key focus area in everything we do. That combined with his dedication to open source and contributing back to the community, meant that we were not surprised when he was asked if he'd be interested in becoming a Drupal core accessibility maintainer.

Andrew is truly passionate about accessibility and has increased the knowledge and awareness of issues encountered by people with disabilities for all members of our team. We can not think of a better candidate for a new Drupal core accessibility maintainer.

His response when asked to be a Drupal Core maintainer?

I was really stoked when Mike asked if I'd consider becoming a core maintainer. I have barely stopped bouncing around my home.

Congratulations from everyone in Annertech Andrew!

Jul 20 2016
Jul 20

During the 8th week of Google Summer of Code 2016, we have started with adding support for posting comments via email. After getting the feedback from my mentors, there was some space to improve proposed solution. The suggestions were accepted and briefly explained below.

Comments in a submodule

As comments are supported through new handler plugin (MailhandlerComment), we decided to move it into a submodule - mailhandler_d8_comment. Following this approach, we can avoid too many dependencies on the main module. The submodule features: Inmail config entity that represents the handler, the plugin class and the related test coverage.

Configurable referenced entity type

With configurable referenced entity type of processed comments, we can support not only node entities but all the other “commentable” entities. In my opinion, this is a nice feature for site administrators in case they want to allow posting comments (for their articles or custom entities) as easy enough as sending an email.

Strict validation

Validation is the important topic in software development in general. In Drupal 8, there is an API made specifically for entity validation - Entity Validation API. It is recommended to use it before saving an entity. However, entity validation does not work as expected in a case of creating comments via API. The referenced entity ID and its bundle were not validated which required us to find a different approach. To accomplish this, we had to implement custom validation checks. They consist of:

  • Entity type validation - entity type is “commentable”

  • Referenced entity ID validation - referenced entity with provided ID actually exists

  • Bundle validation - identified entity bundle supports comments


Last week, besides the improvements made on the comment feature, I was working on the demo module improvements. The pull request has been submitted on Github and available for review. This module serves as an example module to show features of Mailhandler. It creates a sample “Mailhandler” content type (with a body field) and a new user with permissions to add content for “Mailhandler” content type and post comments. The GPG key field of this user was populated  with a sample key (both public and private GPG keys were added to the module) as well. Since last week, the example messages - sample comment and PGP clear-signed, MIME and MIME HTML emails are part of the demo module too. By enabling mailhandler_d8_demo, exploring Mailhandler features becomes very straightforward.

[embedded content]

Drupal 8 module demo: Mailhandler from Milos Bovan on Vimeo.

Also, the demo module adds a sample article attached to the front page. The idea is to have a descriptive article about Mailhandler features. Its content will be extended during the “documentation week”. Next week, I will be working on improving overall user experience and user interface of Mailhandler.

 

 

Jul 19 2016
Jul 19

Project Details: With the introduction of Multiversion module in Drupal 8, we have a very powerful content revision API that can handle branching and conflict DETECTION. But there is not yet a way to SOLVE revision conflicts.

Proposed Solution: We store a graph of all the updated revisions from an entity and find a base revision from this graph with which two entities can be compared. This process is based on 3-way merge algorithm. According to this algorithm, we shall compare two revisions of an entity with another revision or actual node to detect what has been modified in those revisions. If the new revisions has modified content over different lines, the updated content shall be merged in the new revision along with the unchanged data from previous revisions. This is why we compare two entities with a base entity, to find out which part has been modified in the new revisions. If both the new revisions has modified the same data, a merge conflict would occur and user would be allowed to solve the conflict in a 3-pane window. You can find more about the project and the approach in the 3rd week blog post at my blog.

Work we did last week: We finished writing code for simple LCA resolver along with tests and after that we started our work to implement a simple merge resolver for linear entities in drupal. We started by writing basic services, creating containers and I spent lot more time in reading about Symfony framework and Drupal8 module development.

Work we did this week: We are done with the simple merge resolver as well now. The task was to resolve merge conflicts in linear entities which drupal implements by default. The approach we took for this was to return the entity last created in the revision. This way, the entity which was created last would be returned as it is the latest entity. We have also written tests for it. The code for the library can be found on the github repository.

Next week’s task: After we’re done with the extensive testing of this simple merge resolver, we will start our major task with the complicated lca resolver where we’d integrate the library we created in coding phase 1 to find LCA with Drupal8 Conflict module. Our approach will be the same - write tests first and then code.

My mentors Dick Olsson, Andrei Jechiu and Tim Millwood have been really very supportive. Not only they have taught me how to code in a better way but they also help me understanding why it is important to test a software extensively. It’s only because of them I have came this far. All the credits to them only. I know my project is very interesting but the have made the whole journey interesting.

Jul 18 2016
Jul 18
Drupal Component Composer Console default avatar

Thought byRay Saltini

July 18, 2016

When we said we'd introduce you to the ABC's of Drupal we didn't just mean the easy stuff. Here are three C's that will help you understand the power behind Drupal 8.

Component: In Drupal 8 the word component is often used to identify one of the libraries managed by the Symfony project used by Drupal. Visit this link for a full list. https://ffwagency.com/blog/what-symfony-components-are-going-drupal-8 While Twig is often cited as a Symfony component, strictly speaking it is a PHP theming engine created by the same individuals who founded and maintain the Symfony project.

Composer: Composer is a dependency manager tool for PHP, Drupal’s core scripting language. If you are familiar with a package manager it is similar but not the same. Practically speaking Composer will help you download, install and manage specific code libraries and any additional code libraries they in turn may be dependent on for a given project. It’s important to understand that Composer is project centric - it defaults to managing code and dependencies within projects (like websites) rather than globally across projects. Composer’s use in the Drupal space is growing with the adoption of Drupal 8 and has begun to be used as an alternative to .make files which have been used to help manage Drupal installs and distributions.

Console: Console is a component of the Symfony project. Most often when a Drupal developer refers to using Console they are talking about using Drupal Console, a suite of tools run from a command line interface (CLI) to generate boilerplate code and interact with a Drupal 8 installation. In 2014 FFW hired Drupal Console’s lead project developer Jesus Olivas to develop the project and contribute the tool back to the Drupal community to strengthen Drupal 8 adoption. Drupal Console is used as a learning tool, module and theming scaffolding, debugger and configuration tool for Drupal 8. For more information on Drupal console visit https://ffwagency.com/blog/drupal-console-overview-new-drupal-cli

Jul 17 2016
Jul 17
  • Advisory ID: DRUPAL-PSA-2016-002
  • Project: Drupal
  • Version: 8.x
  • Date: 2016-July-17
  • Security risk: TBD
  • Vulnerability: TBD

Description

We will be doing a Drupal 8 core patch release on Monday, July 18th. This will occur between 14:15 UTC and 19:00 UTC.

There will not be a Drupal 7 release during this window.

Update: details and mitigating instructions

A release of Drupal core that fixes this issue is available via SA-CORE-2016-003. Details of the issue (dubbed httpoxy) are avialable at httpoxy.org. In addition to the background information available the site also includes mitigation instructions.

Why is this release being issued?

The Drupal security team has learned that a third-party Drupal 8 dependency will be making a security release on Monday, July 18th and in accordance we will be making a Drupal 8 release soon after. We will not disclose details of the third-party update in advance of that release and cannot respond to requests for further information. This security release is for the dependency only and does not affect Drupal 7 sites. Other mitigating factors will be included with our published SA.

What about the regularly scheduled release window on Wednesday, July 20?

We are moving the regularly scheduled window two days earlier to provide the third-party dependency update, so this replaces that window.

There will not be another core release on Wednesday, July 20th.

Contact and More Information

The Drupal security team can be reached at security at drupal.org or via the contact form at https://www.drupal.org/contact.

Learn more about the Drupal Security team and their policies, writing secure code for Drupal, and securing your site.

Follow the Drupal Security Team on Twitter at https://twitter.com/drupalsecurity

Edited to add:

  • July 18th 14:00 UTC: Link to details and mitigation instructions
Jul 15 2016
Jul 15

The overall test coverage of Mailhandler module has been improved in the week 7 of Google Summer of Code. The plan for the week 8 was to implement feature for posting comments by sending an email.

Similarly to MailhandlerNode (handler for nodes), we had to create a new config entity: inmail.handler.mailhandler_comment and a handler plugin class. Since comments will have limited support, during the last weekly meeting with my mentors (Miro and Primoz), we decided not to add more analyzers as proposed first, but rather to move comment specific business logic to MailhandlerComment Inmail handler plugin.

In order to simplify the logic in the comment handler, EntityTypeAnalyzer was updated to support partial entity type matching. The entity type was extracted from the subject independently of the second part, which can be bundle or entity ID in case of comments.

The current steps in the comment handler are:

  • Assert we are dealing with comments (the identified entity type is comment)

  • Parse the referenced entity ID from the mail subject: [comment][#entity_id]

  • Validate (authenticate and authorize) a user

  • Create a comment entity if all previous conditions are met

The pull request on Github was already created and it will request additional updates after it received some nice suggestions from my mentor.

The Inmail issue Lack of standard result in collaboration of analyzers progressed well during the last week. After several feedbacks and broad discussion, it is currently in “Needs review” state. In my opinion, it is quite close to be fixed and we will be able to implement the standard analyzer result object into Mailhandler module very soon.

Also, last week I made a few UX improvements in the module.
Inmail demo now supports sample mail messages from mailhandler_d8_demo module. As a related issue, PGP-signed sample mails were added to the demo.

The Mailhandler Demo is our focus for the following week. It will be extended with a sample Mailhandler user with already preconfigured Inmail settings, PGP keys and relevant form and display updates. The goal is to provide an easy start for new Mailhandler users. The progress made on the module so far, will be presented as a short (video) demo. Stay tuned!

Jul 14 2016
Jul 14
TL;DR Previous week I had worked on detecting the emotion in the profile pictures of the users, and notifying them to change the image if they do not look happy. The work is under review by the mentors. Once it gets reviewed, I would resume it if it needs any changes. This week I have worked on filling the ‘Alt Text’ field of an image file based on any one of the method selected by the end user- Label Detection, Landmark Detection, Logo Detection and Optical Character Detection.

Last week, I had worked on implementing the Face Detection feature in the Google Vision API module. The code is currently under the review by the mentors. Once, they review it, I would develop further on it if it requires any changes.

The Google Cloud Vision API provides the features to detect popular landmarks in an image(Landmark Detection), logos of popular brands(Logo Detection), texts within an image(Optical Character Detection), in addition to Label Detection. These features, though of less significance, are helpful in identifying an image. Hence, I have started working on implementing a new helpful case for the users- Filling of the Alternate Text field of an image file using these features.

The Alt Text field of the image file entity is modified to incorporate the options to fill the field using the features. The user may select any one of the four options to fill the Alt Text field of the image.

Coming to the technical aspect, I have made use of hook_form_BASE_FORM_ID_alter() to alter the Alternate Text field of the image file entity. I have modified the edit form of the Alt Text field to add four radio options, namely- Label Detection, Landmark Detection, Logo Detection and Optical Character Detection. The user may select any of the options and save the configuration. The Alternate Text field would be filled up accordingly.
Presently, the code is under the review by the mentors. Once it gets reviewed, I would make suggested changes, if required.
Jul 14 2016
Jul 14

Back in December, Tom Friedhof shared how we set up our Drupal 8 development and build process utilizing Docker. It has been working well in the several months we have used it and worked within its framework. Within the time-span however, we experienced a few issues here and there which led me to come up with an alternative process which keeps the good things we like and getting rid of/resolving the issues we encountered.

First, I'll list some improvements that we'd like to see:

  1. Solve file-syncing issues

    One issue that I keep running into when working with our development process is that the file-syncing stops working when the host machine powers off in the interim. Even though Vagrant's rsync-auto can still detect changes on the host file-system and initiates an rsync to propel files up into the containers via a mounted volume, the changes do not really appear within the containers themselves. I had a tough time debugging this issue, and the only resolution in sight was to do a vagrant reload -- it's a time-consuming process as it rebuilds every image and running them again. Having to do this every morning when I turn on my laptop at work was no fun.

  2. Performant access to Drupal's root

    Previously, we had to mount Drupal's document root to our host machine using sshfs to explore in it, but it's not exactly performant. For example, performing a grep or ag to search within files contents under Drupal 8's core takes ~10 seconds or more. Colleagues using PhpStorm report that mounting the Drupal root unto the host system brings the IDE to a crawl while it indexes the files.

  3. Levarage Docker Compose

    Docker Compose is a great tool for managing the life-cycle of Docker containers, especially if you are running multiple applications. I felt that it comes with useful features that we were missing out because we were just using Vagrant's built-in Docker provider. Also with the expectation that Docker for Mac Beta will become stable in the not-so-distant future, I'd like the switch to a native Docker development environment as smooth as possible. For me, introducing Docker Compose into the equation is the logical first-step.

    dlite just got into my attention quite recently which could fulfill the role of Docker for Mac before its stable release, but haven't gotten the chance to try it yet.

  4. Use Composer as the first-class package manager

    Our previous build primarily uses Drush to build the Drupal 8 site and download dependencies and relegating the resolution of some Composer dependencies to Composer Manager. Drush worked really well for us in the past and there is no pressing reason why we should abandon it, but considering that Composer Manager is deprecated for Drupal 8.x and that there is already a Composer project for Drupal sites, I thought it would be a good idea to be more proactive and rethink the way we have been doing Drupal builds and adopt the de-facto way of putting together a PHP application. At the moment, Composer is where it's at.

  5. Faster and more efficient builds

    Our previous build utilizes a Jenkins server (also ran as a container) to perform the necessary steps to deploy changes to Pantheon. Since we were mostly deploying from our local machines anyway, I always thought that perhaps running the build steps via docker run ... would probably suffice (and it doesn't incur the overhead of a running Jenkins instance). Ultimately, we decided to explore Platform.sh as our deployment target, so basing our build in Composer became almost imperative as Drupal 8 support (via Drush) on Platform.sh is still in beta.

With these in mind, I'd like to share our new development environment & build process.

1. File & directory structure

Here is a high-level tree-view of the file structure of the project:

/<project_root>
├── Vagrantfile
├── Makefile
├── .platform/ 
│   └── routes.yaml
├── bin/ 
│   ├── drupal*
│   ├── drush*
│   └── sync-host*
├── docker-compose.yml 
├── environment 
├── src/ 
│   ├── .gitignore
│   ├── .platform.app.yaml 
│   ├── Dockerfile
│   ├── LICENSE
│   ├── bin/ 
│   │   ├── drupal-portal*
│   │   └── drush-portal*
│   ├── composer.json
│   ├── composer.lock
│   ├── custom/
│   ├── phpunit.xml.dist
│   ├── scripts/
│   ├── vendor/
│   └── web/ 
└── zsh/ 
    ├── zshrc
    ├── async.zsh
    └── pure.zsh

2. The Vagrantfile

Vagrant.configure("2") do |config|

  config.vm.box = "debian/jessie64"
  config.vm.network "private_network", ip: "192.168.100.47"

  config.vm.hostname = 'activelamp.dev'

  config.vm.provider :virtualbox do |vb|
    vb.name = "activelamp.com"
    vb.memory = 2048
  end

  config.ssh.forward_agent = true

  config.vm.provision "shell",
    inline: "apt-get install -y zsh && sudo chsh -s /usr/bin/zsh vagrant",
    run: "once"

  config.vm.provision "shell",
    inline: "[ -e /home/vagrant/.zshrc ] && echo '' || ln -s /vagrant/zsh/zshrc /home/vagrant/.zshrc",
    run: "once"

  config.vm.provision "shell",
    inline: "[ -e /usr/local/share/zsh/site-functions/prompt_pure_setup ] && echo '' || ln -s /vagrant/zsh/pure.zsh /usr/local/share/zsh/site-functions/prompt_pure_setup",
    run: "once"

  config.vm.provision "shell",
    inline: "[ -e /usr/local/share/zsh/site-functions/async ] && echo '' || ln -s /vagrant/zsh/async.zsh /usr/local/share/zsh/site-functions/async",
    run: "once"

  if ENV['GITHUB_OAUTH_TOKEN']
    config.vm.provision "shell",
      inline: "sudo sed -i '/^GITHUB_OAUTH_TOKEN=/d' /etc/environment  && sudo bash -c 'echo GITHUB_OAUTH_TOKEN=#{ENV['GITHUB_OAUTH_TOKEN']} >> /etc/environment'"
  end

  
  config.vm.provision :docker

  config.vm.provision :docker_compose, yml: "/vagrant/docker-compose.yml", run: "always", compose_version: "1.7.1"

  config.vm.synced_folder ".", "/vagrant", type: "nfs"
  config.vm.synced_folder "./src", "/mnt/code", type: "rsync", rsync__exclude: [".git/", "src/vendor"]
end

Compare this new manifest to the old one and you will notice that we reduce Vagrant's involvement in defining and managing Docker containers. We are simply using this virtual machine as the Docker host, using the vagrant-docker-compose plugin to provision it with the Docker Compose executable and having it (re)build the images during provisiong stage and (re)start the containers on vagrant up.

We are also setting up Vagrant to sync file changes on src/ to /mnt/code/ in the VM via rsync. This directory in the VM will be mounted into the container as you'll see later.

We are also setting up zsh as the login shell for the vagrant user for an improved experience when operating within the virtual machine.

3. The Drupal 8 Build

For now let's zoom in to where the main action happens: the Drupal 8 installation. Let's remove Docker from our thoughts for now and focus on how the Drupal 8 build works.

The src/ directory cotains all files that constitute a Drupal 8 Composer project:

/src/
├── composer.json
├── composer.lock
├── phpunit.xml.dist
├── scripts/
│   └── composer/
├── vendor/ # Composer dependencies
│   └── ...
└── web/ # Web root
    ├── .htaccess
    ├── autoload.php
    ├── core/ # Drupal 8 Core
    ├── drush/
    ├── index.php
    ├── modules/
    ├── profiles/
    ├── robots.txt
    ├── sites/
    │   ├── default/
    │   │   ├── .env
    │   │   ├── config/ # Configuration export files
    │   │   │   ├── system.site.yml
    │   │   │   └── ...
    │   │   ├── default.services.yml
    │   │   ├── default.settings.php
    │   │   ├── files/
    │   │   │   └── ...
    │   │   ├── services.yml
    │   │   ├── settings.local.php.dist
    │   │   ├── settings.php
    │   │   └── settings.platform.php
    │   └── development.services.yml
    ├── themes/
    ├── update.php
    └── web.config

The first step of the build is simply executing composer install within src/. Doing so will download all dependencies defined in composer.lock and scaffold files and folders necessary for the Drupal installation to work. You can head over to the Drupal 8 Composer project repository and look through the code to see in depth how the scaffolding works.

3.1 Defining Composer dependencies from custom installation profiles & modules

Since we cannot use the Composer Manager module anymore, we need a different way of letting Composer know that we may have other dependencies defined in other areas in the project. For this let's look at composer.json:

{
    ...
    "require": {
        ...
        "wikimedia/composer-merge-plugin": "^1.3",
        "activelamp/sync_uuids": "dev-8.x-1.x"
    },
    "extra": {
        ...
        "merge-plugin": {
          "include": [
            "web/profiles/activelamp_com/composer.json",
            "web/profiles/activelamp_com/modules/custom/*/composer.json"
          ]
        }
    }
}

We are requiring the wikimedia/composer-merge-plugin and configuring it in the extra section to also read the installation profile's composer.json and one's that are in custom modules within it.

We can define the contrib modules that we need for our site from within the installation profile.

src/web/profiles/activelamp_com/composer.json:

{
  "name": "activelamp/activelamp-com-profile",
  "require": {
    "drupal/admin_toolbar": "^8.1",
    "drupal/ds": "^8.2",
    "drupal/page_manager": "^[email protected]",
    "drupal/panels": "~8.0",
    "drupal/pathauto": "~8.0",
    "drupal/redirect": "~8.0",
    "drupal/coffee": "~8.0"
  }
}

As we create custom modules for the site, any Composer dependencies in them will be picked up everytime we run composer update. This replicates what Composer Manager allowed us to do in Drupal 7. Note however that unlike Composer Manager, Composer does not care if a module is enabled or not -- it will always read its Composer dependencies and resolve them.

3.2 Drupal configuration

3.2.1 Settings file

Let's peek at what's inside src/web/settings.php:




$settings['container_yamls'][] = __DIR__ . '/services.yml';

$config_directories[CONFIG_SYNC_DIRECTORY] = __DIR__ . '/config';


include __DIR__ . "/settings.platform.php";

$update_free_access = FALSE;
$drupal_hash_salt = '';

$local_settings = __DIR__ . '/settings.local.php';

if (file_exists($local_settings)) {
  require_once($local_settings);
}

$settings['install_profile'] = 'activelamp_com';
$settings['hash_salt'] = $drupal_hash_salt;

Next, let's look at settings.platform.php:



if (!getenv('PLATFORM_ENVIRONMENT')) {
    return;
}

$relationships = json_decode(base64_decode(getenv('PLATFORM_RELATIONSHIPS')), true);

$database_creds = $relationships['database'][0];

$databases['default']['default'] = [
    'database' => $database_creds['path'],
    'username' => $database_creds['username'],
    'password' => $database_creds['password'],
    'host' => $database_creds['host'],
    'port' => $database_creds['port'],
    'driver' => 'mysql',
    'prefix' => '',
    'collation' => 'utf8mb4_general_ci',
];

We return early from this file if PLATFORM_ENVIRONMENT is not set. Otherwise, we'll parse the PLATFORM_RELATIONSHIPS data and extract the database credentials from it.

For our development environment however, we'll do something different in settings.local.php.dist:



$databases['default']['default'] = array(
    'database' => getenv('MYSQL_DATABASE'),
    'username' => getenv('MYSQL_USER'),
    'password' => getenv('MYSQL_PASSWORD'),
    'host' => getenv('DRUPAL_MYSQL_HOST'),
    'driver' => 'mysql',
    'port' => 3306,
    'prefix' => '',
);

We are pulling the database values from the environment, as this is how we'll pass data in a Docker run-time. We also append .dist to the file-name because we don't actually want settings.local.php in version control (otherwise, it will mess up the configuration in non-development environments). We will simply rename this file as part of the development workflow. More on this later.

3.2.2 Staged configuration

src/web/sites/default/config/ contains YAML files that constitute the desired Drupal 8 configuration. These files will be used to seed a fresh Drupal 8 installation with configuration specific for the site. As we develop features, we will continually export the configuration entities and place them into this folder so that they are also versioned via Git.

Configuration entities in Drupal 8 are assigned a universally unique ID (a.k.a UUID). Because of this, configuration files are typically only meant to be imported into the same (or a clone of the) Drupal site they were imported from. The proper approach is usually getting hold of a database dump of the Drupal site and use that to seed a Drupal 8 installation which you plan to import the configuration files into. To streamline the process during development, we wrote the drush command sync-uuids that updates the UUIDs of the active configuration entities of a non-clone site (i.e. a freshly installed Drupal instance) to match those found in the staged configuration. We packaged it as Composer package named activelamp/sync_uuids.

The complete steps for the Drupal 8 build is the following:

$ cd src
$ composer install
$ [ -f web/sites/default/settings.local.php ] && : || cp web/sites/default/settings.local.php.dist web/sites/default/settings.local.php
$ drush site-install activelamp_com --account-pass=default-pass -y
$ drush pm-enable config sync_uuids -y
$ drush sync-uuids -y
$ drush config-import -y

These build steps will result a fresh Drupal 8 installation based on the activelamp_com installation profile and will have the proper configuration entities from web/sites/default/config. This will be similar to any site that is built from the same code-base minus any of the actual content. Sometimes that is all that you need.

Now let's look at the development workflow utilizing Docker. Let's start with the src/Dockerfile:

FROM php:7.0-apache

RUN apt-get update && apt-get install -y \
  vim \
  git \
  unzip \
  wget \
  curl \
  libmcrypt-dev \
  libgd2-dev \
  libgd2-xpm-dev \
  libcurl4-openssl-dev \
  mysql-client

ENV PHP_TIMEZONE America/Los_Angeles


RUN docker-php-ext-install -j$(nproc) iconv mcrypt \
&& docker-php-ext-configure gd --with-freetype-dir=/usr/include/ --with-jpeg-dir=/usr/include/ \
 && docker-php-ext-install -j$(nproc) gd pdo_mysql curl mbstring opcache


RUN curl -sS https://getcomposer.org/installer | php
RUN mv composer.phar /usr/local/bin/composer
RUN echo 'export PATH="$PATH:/root/.composer/vendor/bin"' >> $HOME/.bashrc


RUN composer global require drush/drush:8.1.2 drupal/console:0.11.3
RUN $HOME/.composer/vendor/bin/drupal init
RUN echo source '$HOME/.console/console.rc' >> $HOME/.bashrc


RUN echo "date.timezone = \"$PHP_TIMEZONE\"" > /usr/local/etc/php/conf.d/timezone.ini
ARG github_oauth_token

RUN [ -n $github_oauth_token ] && composer config -g github-oauth.github.com $github_oauth_token || echo ''

RUN [ -e /etc/apache2/sites-enabled/000-default.conf ] && sed -i -e "s/\/var\/www\/html/\/var\/www\/web/" /etc/apache2/sites-enabled/000-default.conf || sed -i -e "s/\/var\/www\/html/\/var\/www\/web/" /etc/apache2/apache2.conf


COPY bin/drush-portal /usr/bin/drush-portal
COPY bin/drupal-portal /usr/bin/drupal-portal

COPY . /var/www/
WORKDIR /var/www/

RUN composer --working-dir=/var/www install

The majority of the Dockerfile should be self-explanatory. The important bits are the provisioning of a GitHub OAuth token & adding of the {drupal,drush}-portal executables which are essential for the bin/{drush,drupal} pass-through scripts.

Provisioning a GitHub OAuth token

Sometimes it is necessary to configure Composer to use an OAuth token to authenticate on GitHub's API when resolving dependencies. These tokens must remain private and should not be committed into version control. We declare that our Docker build will take github_oauth_token as a build argument. If present, it will configure Composer to authenticate using it to get around API rate limits. More on this later.

DrupalConsole and Drush pass-through scripts

Our previous build involved opening up an SSH port on the container running Drupal so that we can execute Drush commands remotely. However, we should already be able to run Drush commands inside the container without having SSH access by utilizing docker run. However the commands can get too lengthy. In fact, they will be extra lengthy because we also need to execute this from within the Vagrant machine using vagrant ssh.

Here are a bunch of scripts that makes it easier to execute drush and drupal commands from the host machine:

Here are the contents of bin/drush and bin/drupal:

#!/usr/bin/env bash
cmd="docker-compose -f /vagrant/docker-compose.yml  run --no-deps --rm server drupal-portal $@"
vagrant ssh -c "$cmd"
#!/usr/bin/env bash
cmd="docker-compose -f /vagrant/docker-compose.yml  run --no-deps --rm server drush-portal $@"
vagrant ssh -c "$cmd"

This allow us to do bin/drush to run Drush commands and bin/drupal ... to run DrupalConsole commands, and the arguments will be pass over to the executables in the container.

Here are the contents of src/bin/drupal-portal and src/bin/drush-portal:

#!/usr/bin/env bash
/root/.composer/vendor/bin/drupal --root=/var/www/web $@
#!/usr/bin/env bash
/root/.composer/vendor/bin/drush --root=/var/www/web $@

The above scripts are added to the container and is essential to making sure drush and drupal commands are applied to the correct directory.

In order for this to work, we actually have to remove Drush and DrupalConsole from the project's composer.json file. This is easily done via the composer remove command.

The docker-compose.yml file

To tie everything together, we have this Compose file:

version: '2'
services:
  server:
    build:
      context: ./src
      args:
        github_oauth_token: ${GITHUB_OAUTH_TOKEN}
    volumes:
      - /mnt/code:/var/www
      - composer-cache:/root/.composer/cache
    env_file: environment
    links:
      - mysql:mysql
    ports:
      - 80:80
  mysql:
    image: 'mysql:5.7.9'
    env_file: environment
    volumes:
      - database:/var/lib/mysql

volumes:
  database: {}
  composer-cache: {}

There are four things of note:

  1. github_oauth_token: ${GITHUB_OAUTH_TOKEN}

    This tells Docker Compose to use the environment variable GITHUB_OAUTH_TOKEN as the github_oauth_token build argument. This, if not empty, will effectively provision the Composer with an OAuth token. If you go back to the Vagrantfile, you will see that this environment variable is set in the virtual machine (because docker-compose is run under it) by appending it to the /etc/environment file. All it needs is that the environment variable is present in the host environment (OS X) during the provisioning step.

    For example, it can be provisioned via: GITHUB_OAUTH_TOKEN= vagrant provision

  2. composer-cache:/root/.composer/cache

    This tells Docker to mount a volume on /root/.composer/cache so that we can persist the contents of this directory between restarts. This will ensure that composer install and composer update is fast and would not require re-downloading packages from the web every time we run. This will drastically imrpove the build speeds.

  3. database:/var/lib/mysql

    This will tell Docker to persist the MySQL data between builds as well. This is so that we don't end up with an empty database whenever we restart the containers.

  4. env_file: environment

    This let us define all environment variables in a single file, for example:

    MYSQL_USER=activelamp
    MYSQL_ROOT_PASSWORD=root
    MYSQL_PASSWORD=some-secret-passphrase
    MYSQL_DATABASE=activelamp
    DRUPAL_MYSQL_HOST=mysql

    We just configure each service to read environment variables from the same file as they both need these values.

We employ rsync to sync files from the host machine to the VM since it offers by far the fastest file I/O compared to the built-in alternatives in Vagrant + VirtualBox. In the Vagrantfile we specified that we sync src/ to /mnt/code/ in the VM. Following this we configured Docker Compose to mount this directory into the server container. This means that any file changes we make on OS X will get synced up to /mnt/code, and ultimately into /var/www/web in the container. However, this only covers changes that originate from the host machine.

To sync changes that originates from the container -- files that were scaffolded by drupal generate:*, Composer dependencies, and Drupal 8 core itself -- we'll use the fact that our project root is also available at /vagrant as a mount in the VM. We can use rsync to sync files the other way -- rsyncing from /mnt/code to /vagrant/src will bring file changes back up to the host machine.

Here is a script I wrote that does an rsync but will ask for confirmation before doing so to avoid overwriting potentially uncommitted work:

#!/usr/bin/env bash

echo "Dry-run..."

args=$@

diffs="$(vagrant ssh -- rsync --dry-run --itemize-changes $args | grep '^[>)"

if [ -z "$diffs" ]; then
  echo "Nothing to sync."
  exit 0
fi

echo "These are the differences detected during dry-run. You might lose work.  Please review before proceeding:"
echo "$diffs"
echo ""
read -p "Confirm? (y/N): " choice

case "$choice" in
  y|Y ) vagrant ssh -- rsync $args;;
  * ) echo "Cancelled.][dfLDS]\|^\*deleted'";;
esac

We are keeping this generic and not bake in the paths because we might want to sync arbitrary files to arbitrary destinations.

We can use this script like so:

$ bin/sync-host --recursive --progress --verbose --exclude=".git/" --delete-after /mnt/code/ /vagrant/src/

If the rsync will result in file changes on the host machine, it will bring up a summary of the changes and will ask if you want to proceed or not.

Makefile

We are using make as our task-runner just like in the previous build. This is really useful for encapsulating operations that are common in our workflow:


sync-host:
	bin/sync-host --recursive --progress --verbose --delete-after --exclude='.git/' /mnt/code/ /vagrant/src/

sync:
	vagrant rsync-auto

sync-once:
	vagrant rsync

docker-rebuild:
	vagrant ssh -- docker-compose -f /vagrant/docker-compose.yml build

docker-restart:
	vagrant ssh -- docker-compose -f /vagrant/docker-compose.yml up -d

composer-install:
	vagrant ssh -- docker-compose -f /vagrant/docker-compose.yml run --no-deps --rm server composer --working-dir=/var/www install

composer-update:
	vagrant ssh -- docker-compose -f /vagrant/docker-compose.yml run --no-deps --rm server composer --working-dir=/var/www update --no-interaction



lock-file:
	@vagrant ssh -- cat /mnt/code/composer.lock

install-drupal: composer-install
	vagrant ssh -- '[ -f /mnt/code/web/sites/default/settings.local.php ] && echo '' || cp /mnt/code/web/sites/default/settings.local.php.dist /mnt/code/web/sites/default/settings.local.php'
	-bin/drush si activelamp_com --account-pass=secret -y
	-bin/drush en config sync_uuids -y
	bin/drush sync-uuids -y
	[ $(ls -l src/web/sites/default/config/*.yml | wc -l) -gt 0  ] && bin/drush cim -y || echo "Config is empty. Skipping import..."

init: install-drupal
	yes | bin/sync-host --recursive --progress --verbose --delete-after --exclude='.git/' /mnt/code/ /vagrant/src/

platform-ssh:
	ssh ></span>@ssh.us.platform.sh

The Drupal 8 build steps are simply translated to use bin/drush and the actual paths within the virtual machine in the install-drupal task. After cloning the repository for the first time, a developer should just be able to execute make init, sit back with a cup of coffee and wait until the task is complete.

Try it out yourself!

I wrote the docker-drupal-8 Yeoman generator so that you can easily give this a spin. Feel free to use it to look around and see it in action, or even to start off your Drupal 8 sites in the future:

$ npm install -g yo generator-docker-drupal-8
$ mkdir myd8
$ cd myd8
$ yo docker-drupal-8

Just follow through the instructions, and once complete, run vagrant up && make docker-restart && make init to get it up & running.

If you have any questions, suggestions, anything, feel free to drop a comment below!

Jul 12 2016
Jul 12

Update: Release Annoucements

The following modules have security releases that are now available, listed in order of severity. There are no more releases planned for today.

Description

There will be multiple releases of Drupal contributed modules on Wednesday July 13th 2016 16:00 UTC that will fix highly critical remote code execution vulnerabilities (risk scores up to 22/25). These contributed modules are used on between 1,000 and 10,000 sites. The Drupal Security Team urges you to reserve time for module updates at that time because exploits are expected to be developed within hours/days. Release announcements will appear at the standard announcement locations.

Drupal core is not affected. Not all sites will be affected. You should review the published advisories on July 13th 2016 to see if any modules you use are affected.

Contact and More Information

The Drupal security team can be reached at security at drupal.org or via the contact form at https://www.drupal.org/contact.

Learn more about the Drupal Security team and their policies, writing secure code for Drupal, and securing your site.

Follow the Drupal Security Team on Twitter at https://twitter.com/drupalsecurity

Edited to add: approximate usage of the modules, links to the final releases, that there are no more releases for today..

Jul 11 2016
Jul 11

Many people have now heard of the EFF-backed free certificate authority Let's Encrypt. Not only is it free of charge, it has also introduced a fully automated mechanism for certificate renewals, eliminating a tedious chore that has imposed upon busy sysadmins everywhere for many years.

These two benefits - elimination of cost and elimination of annual maintenance effort - imply that server operators can now deploy certificates for far more services than they would have previously.

The TLS chapter of the RTC Quick Start Guide has been updated with details about Let's Encrypt so anybody installing SIP or XMPP can use Let's Encrypt from the outset.

For example, somebody hosting basic Drupal or Wordpress sites for family, friends and small community organizations can now offer them all full HTTPS encryption, WebRTC, SIP and XMPP without having to explain annual renewal fees or worry about losing time in their evenings and weekends renewing certificates manually.

Even people who were willing to pay for a single certificate for their main web site may have snubbed their nose at the expense and ongoing effort of having certificates for their SMTP mail server, IMAP server, VPN gateway, SIP proxy, XMPP server, WebSocket and TURN servers too. Now they can all have certificates.

Early efforts at SIP were doomed without encryption

In the early days, SIP messages would be transported across the public Internet in UDP datagrams without any encryption. SIP itself wasn't originally designed for NAT and a variety of home routers were created with "NAT helper" algorithms that would detect and modify SIP packets to try and work through NAT. Sadly, in many cases these attempts to help actually clash with each other and lead to further instability. Conversely, many rogue ISPs could easily detect and punish VoIP users by blocking their calls or even cutting their DSL line. Operating SIP over TLS, usually on the HTTPS port (TCP port 443) has been an effective way to quash all of these different issues.

While the example of SIP is one of the most extreme, it helps demonstrate the benefits of making encryption universal to ensure stability and cut out the "man-in-the-middle", regardless of whether he is trying to help or hinder the end user.

Is one certificate enough?

Modern SIP, XMPP and WebRTC require additional services, TURN servers and WebSocket servers. If they are all operated on port 443 then it is necessary to use different hostnames for each of them (e.g. turn.example.org and ws.example.org. Each different hostname requires a certificate. Let's Encrypt can provide those additional certificates too, without additional cost or effort.

The future with Let's Encrypt

The initial version of the Let's Encrypt client, certbot, fully automates the workflow for people using popular web servers such as Apache and nginx. The manual or certonly modes can be used for other services but hopefully certbot will evolve to integrate with many other popular applications too.

Currently, Let's Encrypt's certbot tool issues certificates to servers running on TCP port 443 or 80. These are considered to be a privileged ports whereas any port over 1023, including the default ports used by applications such as SIP (5061), XMPP (5222, 5269) and TURN (5349), are not privileged ports. As long as certbot maintains this policy, it is generally necessary to either run a web server for the domain associated with each certificate or run the services themselves on port 443. There are other mechanisms for domain validation and various other clients supporting different subsets of them. Running the services themselves on port 443 turns out to be a good idea anyway as it ensures that RTC services can be reached through HTTP proxy servers who fail to let the HTTP CONNECT method access any other ports.

Many configuration tasks are already scripted during the installation of packages on a GNU/Linux distribution (such as Debian or Fedora) or when setting up services using cloud images (for example, in Docker or OpenStack). Due to the heavily standardized nature of Let's Encrypt and the widespread availability of the tools, many of these package installation scripts can be easily adapted to find or create Let's Encrypt certificates on the target system, ensuring every service is running with TLS protection from the minute it goes live.

If you have questions about Let's Encrypt for RTC or want to share your experiences, please come and discuss it on the Free-RTC mailing list.

Jul 07 2016
Jul 07

Recap: We need to detect and resolve merge conflicts in Drupal8 which might arise when two users try to push the same entities but modified at respective end to the server without pulling the latest changes. We are implementing a “conflict” module to help detecting conflicts otherwise merge. For a complete description of what the project is about, please review to the 3rd week’s blog where I have given a complete walk through of the project. We finished two libraries, one to find the Lowest common ancestor from a directed acyclic graph and the other one to use that LCA as a base to perform recursive 3-way Merge algorithm by comparing local, remote and base entity before mid term and now we’re creating a module which will use those libraries.

For the progress till now, please refer to the article from last week. All the work till last week has been written in that article.

Target for this week was to create a simple Lowest common ancestor in Drupal 8 as a service in “Conflict” module.

Working: Entities in Drupal store revisions in a linear graph, for example if a node “A” has been edited 4 times, a linear graph would be formed looking something like:

                                                                      A → B → C → D → E

This simple LCA resolver would find the parent of last edited revisions. In this case, it’d find and return “C” for revisions “D” and “E” as they both (D & E) have been modified from “C” and hence, “C” is their common parent. 

Although it looks to be a pretty simple feature but it was ironically very tough for me to implement this code. Not the actual implementation but the testing part. After creating two libraries, I was starting to think that the “Testing” part is not so difficult after all but it proved me wrong *pun*. I made some silly mistakes and was trying on my own to solve them because I didn’t want to bug my mentors with such small mistakes. Just opposite to what I was thinking, I had to ask my mentors after nearly spending 2 days on a single error. My mentor “Andrei Jechiu” hardly took 2 minutes to point me in the right direction and I was back on track.

One thing I noticed while creating the module is that you don’t really just create the module, you learn a lot more than what you need and this is what I believe is the best part of working with Drupal and Google summer of code. The graph of learning increases exponentially.

So after solving the error, when I thought it shouldn’t take much time, I was right where I started, on another error which wouldn’t let me through but I didn’t make the same mistake of being struck with that error, after trying for like half day, I again had to ask “Tim Millwood” (My mentor) and just like Andrei, not only he pointed me to where I was making mistake (Services.yml was not properly defined), he also taught me how actually service container works in a much simpler way than many of the articles I read.

All the code we created can be found in drupal-conflict repository over github. We are still working on the Tests and once we are sure that the code is passing the tests, we shall move to the next step of creating a simple merge resolver which would work for linear entities and merge the updates in them. I was expecting to get over with this task this week only but it took much longer than I was expecting but I will take it as long as I’m learning new things.

Other than this part of project, we had an awesome Open session this week to encourage students to start contributing to open source. Students from various organizations like Fossasia, Drupal, Mifos Initiative and KD were there. Not only we met really talented people, we also had a talk about their projects and their approach.

Jul 07 2016
Jul 07
TL;DR Previous week I had worked on grouping the contents based on the dominant color component in their images, if present. The work is under review of the mentors. And once, it gets reviewed, I would work further on that issue. Meanwhile, I have started developing and implementing the Emotion Detection feature of the Google Cloud Vision API. It would detect the emotion of the person in the profile picture uploaded, and if the person looks angry or unhappy, he would be notified thereof. This feature is especially important when building sites for professional purposes, as the facial looks matters a lot in such cases.

Last week, I had worked on implementing the Dominant Color Detection feature in the Google Vision API module. The code is currently under the review by the mentors. Once, they review it, I would develop further on it if it requires any changes.

Hence, meanwhile, I have started working on implementing a new feature Face Detection in an image. This feature gives us the location of the face in an image, and in addition, the emotions and expressions on the face.

I have used this feature to detect the emotion of the person in the profile picture uploaded by him/her. If the person does not seem happy in the image, he/she is notified thereof of their expressions. This is especially useful when the end users are developing a site for professional purposes, as in professional matters, expressions matters a lot.

Coming to the technical aspect, I have made use of hook_entity_bundle_field_info_alter() to alter the image fields, and check the emotions in the uploaded images. This function has been used, as we only want to implement this feature on the image fields. If the image is not a happy one, then appropriate message is displayed using drupal_set_message(). This feature also makes use of Constraints and Validators just like the Safe Search detection feature. Presently, the code is under the review by the mentors.

In addition to the implementation of Face Detection, I also worked on expanding the tests of the Safe Search Detection feature of the Google Vision API module to test other entities as well, in addition to the nodes. I have expanded the tests to test the safe search constraint on the comment entity as well. This requires the creation of a dummy comment type, adding an image field to the comment type, and attaching the comment to the content type. The image field contains the safe search as the constraint on it. This test is basically similar to the tests present in the module for the node entity. The code is under review by the mentors and would soon be committed to the module. For reference on how to create dummy comment types and attaching it to the content types, the CommentTestBase class is very helpful.
Jul 06 2016
Jul 06

The coding period of Google Summer of Code 2016 has been continued after midterm evaluation. After splitting the Mailhandler analyzer into more specific and independent analyzers for different parts of an email message (sender, entity type, PGP, body, footer), last week I was working on providing test coverage for those.

The current tests are mostly written as kernel tests because of their speed of execution. MailhandlerNodeTest represents an integration test for the node handler and tests the whole process of Inmail analyzers and handlers.

Newly written tests:

  • SenderAnalyzerKernelTest - covers the use case when a user is identified and when a user is not identified
  • PGPAnalyzerKernelTest - tests assertions of PGP-signed messages which inlude body, sender etc
  • FooterAnalyzerKernelTest tests plain mail messages with and without the message footer
  • EntityTypeAnalyzerKernelTestcovers entity type and bundle pair of the mail subject.
  • SenderAnalyzerKernelTest covers the mail sender detection using From mail header field.

As they were all similar in the implementation, AnalyzerTestBase was added. It contains the list of required enabled modules and set up configuration for easier analyzer-specific work. 

Regarding the web tests, I extended MailhandlerWebTest with "Manage display" assertions of GPG key field which can be displayed via the public key, GPG key's fingerprint or as both properties.

As previously mentioned I created a few inmail issues which will allow for a better DX. Here is a short update on status of those issues:

The plan for the next week seems quite interesting. Besides the mentioned Inmail issues, I will be working on adding support for creating comments by sending an email. All analyzers implemented in the previous week will be used as a part of new comment handler. I am looking forward to enriching Mailhandler with one more feature.

Jul 05 2016
Jul 05

Last week the Couchbase team released the new 4.5 version of Couchbase Server.

You can see all the new introduced features here:

http://www.couchbase.com/binaries/content/assets/us/product/couchbase-se...

A part from introducing a huge number of new features, many of them aimed at improved performance and scalability, the new version solves an outstanding issue that prevented the Drupal Couchbase module from operating on SASL protected buckets:

https://forums.couchbase.com/t/how-to-execute-n1ql-queries-against-a-sas...

We are also happy to see the Couchbase team actively working on the Couchbase PHP extension, and a PHP7 release is around the corner.

The Drupal Couchbase module has already been tested against an internal build and works perfectly on PHP7.

With the latest 8.x-1.9 Drupal Couchbase module release you can scale your Drupal based application to new heights with a much lower TCO.

Because you know that....

Caches should not be mixed with data on the same storage engine ®

Jul 01 2016
Jul 01

With three months left to go before DrupalCon Dublin, event planning is in full swing. Like all Irish Drupal events, Annertech are actively involved in preparations for DrupalCon. Our managing director, Stella Power, has taken on the role of local team lead as well as Business Track chair, and I myself am the Project Management track chair. There's less than a week now to get your sessions submitted for DrupalCon, so it's time to get writing!

Business Track

This year the business track is focusing on how agencies can grow their business and how to do so sustainably. We're looking for speakers who are willing to share their experiences and insights into growing their businesses, what worked, what didn't, and why.

Suggested topics include:

Marketing Drupal

Competition in the marketplace is growing, both from a growing number of agencies offering similar services, but also from alternative solutions being offered. We are looking for sessions on how you generate new leads for your business, and how you market and postiion Drupal and your agency as being the best solution available.

Going for Growth

Growing your business is all very well, but how do you manage this growth and ensure you have the necessary structures in place to ensure this growth is sustainable. We are seeking strategies which offer sustainable approaches to growth for the small and large Drupal agency alike.

Business Innovation and Diversification

Perhaps the solution to sustainable business growth is to innovate and diversify your service offering. Maybe you need to consider extending into additional or alternative markets or verticals. But when is the time right to do this? And what are your strategies to approaching this?

Drupal 8

With the release of Drupal 8, there are a number challenges, but also opportunities for agencies working with Drupal. What has been the impact of Drupal 8 on your business, on demand for your services, on your sales or marketing processes and on your team? How has it affected the estimation and delivery of your projects? How do you convince clients to upgrade to Drupal 8?

These are just some of the ideas we have for the Business Track this year. We want to hear from you, about your experiences and insights into the above topic areas. Got an idea for another session? Great, then let us know and submit it!

Project Management Track

Whatever the size of the project, no matter how many people are working on it, it will need to be managed. Some have dedicated project managers, others share the task within the team, whatever way you spin it it's got to be done, so why not make sure we can learn how to do it in the most appropriate way.

The focus is about sharing your experiences, good or bad, as long as you learned something which is relevant to others, you're welcome to submit a session, here are some topics which we're actively looking for:

  • Tips and Tricks: How did you improve your processes or learn how not to do something?
  • Becoming agile: How did you become more agile?
  • Getting the numbers right: Estimations, budgeting and reporting. How do you keep on top of things?
  • Freelance Project Management: Are you, or have you ever been, a freelance Project Manager? Tell us your story.
  • Managing an Open Source Project: Ever been involved in managing an open source project? How did you get by? Would you do it again?

As always, if you're unsure whether your session proposal is appropriate, reach out to myself or Stella and we will help you where we can.

Countdown to session submission closing starts now, are you ready?

Jul 01 2016
Jul 01

TL;DR

We were already prepared for a scenario like the Aberdeen Cloud breakdown, owing to our disaster recovery plan. Fortunately we didn't have to set it in motion. Each night we have a simple script which takes off-site backups of all of our hosted sites. We've made the source code available on github, so hopefully this will help others prepare for the likes of the Aberdeen Cloud implosion, and perhaps we can share ideas on how to improve each other's disaster recovery plans.

Our experience with the Aberdeen Cloud incident

We have a large number of sites hosted by various cloud services. Since autumn 2013, we'd mainly used Aberdeen Cloud, and in autumn 2015 we started to explore other options to see what else the market had to offer. Platform.sh was the one that we decided to give a serious test for new clients.

Soon after that, Aberdeen Cloud began to seem a bit flaky. Longer response times on support tickets. Solr services started to fail, along with random outages of various other services. We accepted that for a while, but after having lost and regained SSH access to all of our sites (including git and rsync), we eventually decided that enough was enough and we couldn't put our trust in them anymore.

We had to migrate everything to alternative hosting platforms. Given the similar price points and the fact that they seem well funded and offer excellent support, we decided on Platform.sh. And so began Project Exodus. 

Over the next three weeks we migrated over 20 sites to Platform.sh in a staged approach. I wouldn't say that it was a straight-forward process. Lots of clients had specific quirks to their setup. For example, some needed a PHPBB forum, others had FTP access for uploading files, some integrated with external systems that required firewall changes, etc, while others had custom .htaccess redirection rules that needed to be rewritten for Nginx. However, we were very lucky and had completed Project Exodus nearly a month before Aberdeen Cloud finally came tumbling down.

So what if you were not as fortunate as us, and still have sites whose assets are no longer accessible? Stuck in cyberspace, or maybe just plain deleted?

Well, I'm not sure there is a lot you can do. Maybe read Code Engima's blog post about the different things they've tried. However, to put it bluntly, it sucks! Enough said about the matter.

Now it's time to ensure you're never caught out again.

But what can we do to prevent this from happening again?

All companies probably have their own way to deal with this kind of scenario, but I'll tell you about how Annertech deals with backups and recoveries.

We have two sets of backups. A backup from each day, for each production/live/master environment, which is hosted by the cloud service. Then there is an off-site backup (again, daily) used for disaster recovery. The latter one is the important one in this scenario.

The idea is that even if Amazon (which hosts Aberdeen Cloud services) pulled the plug, we would still have access to our clients' data.

We have a server, hosted by a different company, that: 

  1. Pulls down a copy of the database, from every cloud-hosted site, every night, and saves it for four days, before it deletes it again.
  2. Runs `rsync` on every cloud-hosted site, to get an up-to-date version of the files folder, every night.

Sounds simple? It is. All it requires is that your hosting partner supports running Drush on your remote sites and you're good. If you run Drupal sites, and your hosting partner doesn't support running Drush on the remote sites, find somebody else who does. It's that important!

Regarding the code for the sites; we keep our source code repositories on dedicated git services. And, more than likely, we'll also have a copy or two on developers machines.

I'd like to show you the two backup scripts that I made, one for Aberdeen Cloud and one for Platform.sh.

The code is meant to work in our setup only, and is not (yet) generic enough to just work out of the box elsewhere. The release of the scripts is meant to give you a leg up and some inspiration. This is by no means the final end point for these scripts - we are continually evaluating and improving our system, and I look forward to hearing what ideas you have on where we could take it from here too.

The entire repository of code can be "stolen" from github.

When you have a disaster recovery plan you also need to make sure that it actually works. You can do this by downloading the latest backup from each of your sites once a month, installing and then testing them. I've tested a site, where the DB file was corrupt, but only for that site, so make sure that you test all of them. The setup of test sites can also be automated by a script so you don't have to setup 10, 50, or 300 sites and test each manually. Scripts are your friends. Make good use of them and have them do all the hard work.

Now, if you really want to push this further, you should implement a "Smoke test" in all of your installation profiles, so that you can trigger that to see if the site is alive; or perhaps tie it in with a Jenkins server.

If something is unclear, feel free to put a comment below. If you feel like this could be improved, feel free to contribute with a pull request. We are all ears.

Jun 29 2016
Jun 29
TL;DR The safe search constraint feature is now committed to the module along with proper web tests. So, this week I started off with a new feature offered by the Google Cloud Vision API- “Image Properties Detection”. It detects various properties and attributes of an image, namely, the RGB components, pixel fraction and score. I have worked on to detect the dominant component in the image present in any content, and display all the contents sharing similar dominant color. It is pretty much like what we see on the e-commerce sites.

Previous week I had worked on writing web tests for the safe search constraint validation on the image fields. This feature is now committed in the module Google Vision API.

This week I have worked on implementing another feature provided by the Google Cloud Vision API, namely, Image Properties Detection. This feature detects the color components of red, green and blue colors in the images along with their pixel fractions and scores. I have used this feature to determine the dominant color component (i.e. red, blue or green) in the image, and to display all those contents which have the same dominant color in their images.

I have developed the code which creates a new route- /node/{nid}/relatedcontent to display the related contents in the form of a list. This concept makes use of Controllers and Routing System of Drupal 8. The Controller class is extended to render the output of our page in the format we require. The contents are displayed in the form of list with the links to their respective nodes, and are named by their titles.

In addition to the grouping of similar contents, the colors are also stored in the form of taxonomy terms under a taxonomy vocabulary programmatically generated under the name Dominant Colors.

This issue is still under progress, and requires little modification. I need to add the link to the new route in each of the nodes, so as to  get a better interface to access those contents. Henceforth, I will put this patch for review.
A very simple example of creating routes and controllers in your module can be found here.
Jun 29 2016
Jun 29
TL;DR It has been over a month since I started working on my Drupal project “Integrate Google Cloud Vision API to Drupal 8”, and gradually I have crossed the second stage towards the completion of the project, first being selection in the Google Summer of Code 2016 programme. Here, I would like to share my experiences and accomplishments during this one month journey, and also I would like to summarize my further plans with the project and the features which I would be implementing in the coming two months.

Let me first describe the significance of this post and what actually does “midterm submission” means? The GSOC coding phase has been divided into two halves, viz. Midterm submission and Final submission. In the first half, the students try to accomplish around 50% of the project, and submit their work to the mentors for evaluation. Those who passed the midterm evaluations are allowed to proceed further and complete the remaining portion of their project.

Now coming back to my experiences, after successfully passing through the Community Bonding period of the GSOC 2016 programme, now it was the time for start coding our project proposal to reality. As I had shared earlier that during the Community Bonding period, I came to know that the project has already been initiated by Eugene Ilyin,(who is now a part of my GSOC team). So, we discussed upon the project and set a roadmap of the project and the goals we need to achieve in the GSOC period. I had started coding the very first day of the coding phase, moving the new as well as existing functions to services. My mentors Naveen Valecha, Christian López Espínola and Eugene Ilyin really helped me a lot and guided me whenever and wherever I needed their guidance and support. They helped me to get through new concepts and guided me to implement them in the most effective way to make the module the best that we could.

During this period, I also came to learn about a lot of new techniques and concepts which I had not implemented earlier. Right from the very first day of the coding period, I have been coming across new things everyday, and it is really interesting and fun to learn all those techniques. I also learnt about the use of validators and constraints and how can they be implemented both on general basis or specifically on fields. I also learnt about how to create test modules and alter various classes and their functions in our tests so as to remove the dependency on web access or on valid informations for our testing purposes. I learnt new things every day and enjoyed implementing them to code our module plan into reality. At present, the module supports the Label Detection feature of the Vision API, along with the tests to verify whether the API key has been set by the end user or not. Currently, the feature of Safe Search Detection is available as a patch which can be found here, which would soon be committed to the module.
[embedded content] I have shared all the details of my work on the Drupal Planet. Please watch this video for detailed information on how to use the Google Vision API module on your Drupal site.
Jun 29 2016
Jun 29

After finishing our work with the PHP libraries to find Lowest Common Ancestor from a graph and then to merge the changes from updated entites, we finally moved to the next step of the project i.e, Making these libraries work with Drupal system.

For those who are not familiar with the project, I would strongly recommend to go through my 3rd week blogpost of GSoC16 where I have given a complete and detailed description of the project.

Coming back to where we started, to make these libraries work with Drupal, we have started work on creating a D8 module "Conflict" which would take care of merging and conflicts.

I spent this week studying about dependency injection, developing pluggable services and other Drupal development stuff rather than starting with the implementation right now. The goal is to keep the module pluggable and to not to depend on Multiversion or Replication module. The initial stab at Conflict module for Drupal 8 will only do a few simple things:

  1. Implement two different ancestor resolvers: 

Since the LCA resolution is pluggable inside, we shall provide two different implementations for this:

  • A simple resolver which find the LCA for revisions that use the default linear revision system provided by core. This resolver will work with any Drupal 8 site.
  • A more complicated resolver that uses the relaxedws/lca library and RevisionTreeIndex from Multiversion module. This step depends on #2748403: Add getGraph() to RevisionTreeIndex . This resolver will only work with sites that uses Multiversion module.
  1. Implement two conflict resolvers: 
  • A simple resolver that just returns the revision with the largest revision ID. This resolver will work with any Drupal 8 site.
  • A more complex resolver that uses the Serialization module to normalize the given revisions into arrays and then use the relaxedws/merge library to perform the merge. This resolver will only work with sites that uses Multiversion module.

We set up an issue to track the progress of this project and the code will be pushed to the drupal-conflict public repository over github. It has the d7 code for it already and now we are creating the code for d8 over "dev-8.x" branch.

Note that the main focus is to make the services pluggable so that those users who don't use Multiversion module can also use this module. Also, there might be developers who might want to use some other approach to find LCA or solve conflicts, they can also add their own services easily if we make the module services pluggable.

To understand the use of services, I went through Services and dependency injection in Drupal 8 and the article about Service Tags over drupal.org. After going through these articles, I could easily see the clear picture of the approach my mentors told me about. 

We will be starting by implemeting a method to find Lowest common ancestor which would work with linear entities and would not require Multiversion module. It'd work with some predefined approached like the last entity updated. We'd be deciding about which approach would be best for this as there are many to use. Once we decide, we would start with the implementation. The best way would be to write the unit test and build the service until it passes the test. This is what we have decided this week. By the next week, we would start the implementation and probably would have done the LCA implementation with the module.

Jun 28 2016
Jun 28

As usual, Tuesday is the day to update you on the progress of Google Summer of Code 2016 project - Mailhandler.

Last week both mentors and students had to fill Google Summer of Code midterm evaluation. The evaluation happened after 5 weeks of work and consisted of questions about the chosen organization, program, mentors (for students) and students (for mentors).

I am happy to announce that I have passed the midterm evaluation. Yay! I would like to give thanks to my mentors Primož and Miro. They were supporting me with reviews, ideas and suggestions in the past weeks. I hope we will continue the great cooperation in the second phase of the project as well. Here is the review I received from my mentors:

Miloš is very diligent and capable of self organising. There were no instances where we needed to remind him of his obligations or upcoming milestones. This goes equally for the technical as for the non-technical side of the project. He is always prepared to investigate the subject very carefully and find the best solutions to his knowledge. As a result his code never feels sloppy or produced just for the sake to make progress. He genuinely cares about the project. Being very goal oriented he sometimes neglects the discussion part slightly. This could be improved by requesting more feedback before jumping to implementation.

This week, GSoC students will continue the coding until the final evaluation which is scheduled for the second part of August 2016.

Back to the project updates. The last meeting with my mentors was very productive. We were talking about the weekly goal and had the broader discussion about the second phase of the project.

More specifically, we discussed the possibility to introduce the user context as a core feature of Inmail. I was writing about Inmail’s concept of plugins (analyzers, deliverers, handlers). Each analyzer has an option to analyze the mail message that is being processed and update the properties of a shared result object. This would allow collaboration between Inmail analyzers. To discuss different approaches, I created an issue on this topic. For now, the properties are updated on MailhandlerAnalyzerResult object.

Based on the discussion with mentors, we decided to split huge MailhandlerAnalyzer into several smaller analyzers. A pull request with the implementation can be followed on Github. The following analyzers were created (sorted by defaults execution order):

  • PGP (Pretty Good Privacy) analyzer analyzes the PGP-signed email messages, verifies the signature, parses the mail body and sets the sender. Although there is specific BodyAnalyzer, for signed messages we have to parse the mail body to extract the signed text and PGP signature.

  • Entity type analyzer - we have a concept of detecting an entity type and bundle information for the mail subject. For now, we only support: [node][{node_type}]. Later on, we will extend it to support comments entities too. The purpose of this analyzer is to recognize [{entity_type}][{bundle}] pattern, extracts the metadata information, do the validation and update the subject - without metadata.

  • Sender analyzer uses a well-known feature of Mailhandler for Drupal 7. It extracts the mail address from From mail header field and finds the corresponding user. It is worth to mention that user is only set in case the user context is not already populated (by some other analyzer). This prevents us from changing the user context when it is set by PGPAnalyzer, for instance. Also, since this method is not entirely safe - From mail address can be faked by a malicious user, this analyzer is disabled by default.

  • Footer analyzer detects the mail footer/signature in a mail body and updates footer and body properties. Two most used footer separators are supported. This analyzer was described in the previous blog post.

  • Body analyzer works with the actual mail body. It has pretty limited functionality. It removes the white spaces before and after the body string using PHP’s standard method trim(). Also, in case processed body is not received as HTML, it replaces new lines \r\n with
    HTML tag. As the analyzer was implemented as a plugin, it can be easily extended.

MailhandlerNode is becoming much “cleaner”. Our algorithm has 3 steps:

  1. Get MailhandlerAnalyzerResult which contains the result of all Mailhandler analyzers

  2. Authenticate and authorize a user

  3. Create a node.

The original complexity from one analyzer is now shared between 5 independent Inmail analyzers. This architectural simplification was made thanks to the great Drupal 8 plugin API. If you are more interested in exploring this topic, Drupalize.me published a great article about Drupal 8 plugin system.

Next week, I am going to work on extending the test coverage for the module. The plan is to create one kernel test per each created analyzer. The existing MailhandlerNodeTest will serve as a general test of all Mailhandler analyzers and MailhandlerNode handler. Also, I will provide additional test coverage of the Mailhandler’s user interface.

Jun 23 2016
Jun 23
TL;DR In my last post Avoid Explicit Contents in the images using Google Vision module, I had discussed about the services which “Safe Search” feature of the Vision API provides us, and how we have put this into use in the Google Vision module as a constraint to all the image fields which would be validated when the option is enabled. This week I have worked on developing simple web tests for testing this feature whether it gives us the results as expected.

Last week I had worked on developing the code to use the Safe Search detection feature as a constraint to the image fields which would validate the images for the presence of explicit contents, provided that the user enables the configuration for the concerned image field.

Besides the code, testing the functionality using simple web tests are equally essential to ensure that the feature executes perfectly when necessary steps are implemented. Hence, this week I have worked on developing simple web tests, which ensures that we have a fully functional feature.

I have tested both the conditions with safe search enabled and disabled to verify the results which should be obtained. When the safe search is enabled, any image containing any sort of  explicit content, is detected, and asked for moderation. If the image is not moderated, then the image is not saved. When the same image was passed through the second test, with safe search disabled, it was stored successfully, thus providing us the expected results.

To conduct the test successfully, I had to create a demo content type in my tests using drupalCreateContentType(), which would have an image field with the ‘Enable Safe Search’ option. This was something new to me to how to add an extra field to the default content type settings. The Drupal documentation on FieldConfig and FieldStorageConfig were of great help to understand the underlying concepts and functions which the field offers, and thus helping me to create custom fields programmatically. However, in order to perform the test, I had to call the API directly, which required a valid API key and an image which actually contains explicit content. Hence, my mentors asked me to override the functions of the class (mocking the services) in such a way that it removes the dependency from both the key and the image. Thus, I created a test module inside the Google Vision module, and override the function.

Summarizing the above, I can say that in addition to learning how to test the constraints and validators, I also came to learn about some really cool techniques, including the creation of custom fields in tests and mocking the services.
Jun 22 2016
Jun 22

Google Summer of Code is halfway through and it’s now time for mid term evaluations. What that means is, we will be judged on the basis of our efforts and targets achieved and our mentors would decide if we pass of fail. If someone fails the mid term evaluation, he is removed from the program immediately.

Let us sum up all work we’ve done in the GSoC-coding phase 1.

Project: Solving content conflicts with merge algorithms in Drupal 8.

Proposed Solution: Merge the updated content in the new array or throw an Exception if there is a conflict. The real challenge was to detect the merge conflicts and to merge otherwise. We decided to use 3-way merge algorithm to compare the updated nodes with a parent node. Out of all parent nodes, the one closest to them(LCA) was our best option to compare them with as it’d have all the recent updations. We needed Libraries to find the closes common parent or Lowest common ancestor and then compare it to the updated nodes to find if there was a conflict or not.

We needed one more library to merge the updated content if there is not conflict.

Deliverables before mid term evaluations:

  1. Library to find LCA from a Directed acyclic graph.

  2. Library to perform a recursive 3-way merge algorithm.

Week 1: Library to find Lowest common ancestor(LCA) in a Directed acyclic graph.

LCA: The nearest common parent to 2 nodes. For ex:

                                                                                      1

                                                                                  /        \

                                                                                2            3

                                                                              /    \         /     

                                                                            4       5     6

LCA of (2,3) is 1 and (4,5) is 2 in the graph created above. LCA(4,5) = 2 because it is a common parent of both as well as closest to these nodes than other common parents.

Similarly LCA(5,6) = 1

Approach: The approach we used to find LCA is Breadth First Traversal in opposite Direction. We traversed from node1 to root storing all the elements encountered in an array. We, then traversed from node2 to root doing same thing. Thanks to the clue/graph and graphp libraries. We used these libraries to create graphs and traverse the graph using BFS in the reverse direction. The first element we get from the intersection of these array elements will be the LCA.

Week 2: Writing tests to assert the correct functionality.

Once we had the working prototype ready to find the LCA, we needed tests to make sure that the code works well in all the cases. We created multiple graphs based on their complexity such as Simple graph, a bit complex graph (Few Multiple parents), very complex graph (Many multiple parents). The visual representation of the graphs is available in the source code.

Then we implemented these graphs in our code and ran multiple tests on them to make sure that they were returning the correct LCA. It was a very crucial part of our project and we had to make sure that no wrong LCA were returned as it’d have caused data loss and probably would have resulted in unnecessary merge conflicts. We wrote tests for:

  • Single Parent in simple graph.

  • Multiple Parents in simple graph.

  • Single Parent in Complex graph.

  • Multiple Parents in Complex graph.

Week 3: Writing Library to perform recursive 3-Way Merge algorithm

After the completion of the LCA library, we were all set to start with the library to perform a recursive 3-way merge algorithm.

3-way merge algorithm: It turns a manual conflict into an automatic resolution. Part of the magic here relies on the VCS locating the original version of the entity (Base entity). This original version is better known as the "Lowest Common Ancestor (LCA)". The VCS then passes the common ancestor and the two contributors to the three-way merge tool that will use all three to calculate the result. The tool compares the local and remote entities with the LCA (base entity) and updates the base entity according to the modified entity. For a clear understanding, please read the article I wrote for 3rd week of GSoC.

Approach: It was a very complicated task and we had to make sure it was working fine for all edge cases including exceptions(Yes, there were many). To implement this library, I got a very clear understanding of how it works.

There were multiple cases I had to take care of while implementing the library. Some of them are:

  • Lines are added in Remote.

  • Lines are added in Local.

  • Lines are added in both Remote and Local.

  • Lines are removed in Remote.

  • Lines are removed in Local.

  • Lines are removed in both Remote and Local.

  • Lines are modified in Remote.

  • Lines are modified in Local.

  • Lines are modified in both Remote and Local.

  • Pre-existing Lines are modified and new lines are added in Remote.

  • Pre-existing Lines are modified and new lines are added in Local.

  • Pre-existing Lines are modified and new lines are added in both Remote and Local.

  • Pre-existing Lines are modified and some lines are removed from Remote.

  • Pre-existing Lines are modified and some lines are removed from Local.

  • Pre-existing Lines are modified and lines are removed from either Remote or Local.

  • Lines are added and removed at the same time.

The basic approach used for most of the cases is to count the number of lines and then we used:

  1. A For loop to run as many times as the minimum lines were there. For example: If ancestor array has a value at some key which has 3 lines, remote has a value with 4 lines an local array has a value with 2 lines, then the first loop will run for 2 lines as the minimum number of lines is 2. This loop will compare all 3 arrays and store the changes on the same key=>value in new array which would be returned after a successful merge.

  2. Second FOR loop  which ran for as many times as the the remaining lines in the second array with largest number of lines. In our case, ancestor array has 3 lines and 2 of which were audited in first FOR Loop and the 3rd line would be compared in this second FOR Loop. So In this scenario, second For loop would run for 1 line only.  It would compare the 3rd line of ancestor from the remote array to make sure they both are same. If they are same, it means only Local node has the 3rd line updated(as it was removed in local), so it’d be removed from final revision as well otherwise, it’d return a conflict Exception.

  3. Third For loop would run to just add new lines into the new revision. In our case, the line number 4 was only in remote and hence, it’d be added in the new array.

In most of the cases, only 1 or 2 “For loops” would run. All the 3 “For Loops” would run in the case where lines have been added as well as removed in remote, local or ancestor.

Week 4: Writing Tests, Custom Exception and Updated documentation.

At the end of the 3rd week, I thought I was done with almost everything until I started testing. With every new test case, I discovered a new edge case where the code was not working as expected and this is the part where I actually learnt about the real importance of testing.

The challenge was the ensure the code readability. With every edge case, the length and complexity of the code kept increasing. After +1,179 // −32 LOC, We were finally done with our as many test cases as possible and the code with it’s proper documentation.

We Covered the following cases in our tests:

  • Simple Merge in same number of lines in all 3 arrays.

  • Merge with same number of lines in 2 arrays (Addition or removal in 3rd).

  • Merge with different number of lines in all 3 arrays (Addition or removal in all 3 arrays).

  • Catching Merge Conflicts.

  • Key removal from a array(Remote or Local).

Other than writings tests this week, we created our own exception class (ConflictException) which extends the core Exception class. We also updated the documentation, made code readable and updated Readme file with the features and the examples.

All the code I have written has been merged into the core libraries from my Github repositories. The code is ready to be used, modifications and feedback. Please feel free to create a pull request with any issues, suggestions or patch for a bug.

A huge thanks to my mentors Dick Olsson, Tim Millwood and Andrei Jechiu for their guidance. They are always available to answer to my silly queries in an appropriate manner. They keep telling me about all those little mistakes I repeat and make sure that I don't repeat them multiple times. I must say they are equally inclined towards me learning more as well as the completion of this project. Never thought this summer would be this much fun. Learning curve is now going up exponentially.

Jun 21 2016
Jun 21

This is the 5th blog post of the Google Summer of Code 2016 project - Mailhandler.

Implementing authentication and authorization for a mail sender provided an additional layer of security for Mailhandler project. The module was extended to support both PGP signed and unsigned messages.

The goal for the last week was to create a mail Footer analyzer and to add support for node (content) type detection via mail subject. The pull request has been created and it is in the review status. This analyzer has a purpose of stripping the message footer/signature from the message body. As of now, 2 types of signature/footer separators are supported:

  • -- \n as the separator line between the body and the signature of a message recommended by RFC 3676
  • On {day}, {month} {date}, {year} at {hour}:{minute} {AM|PM} pattern which is trickier and currently used by Gmail to separate replied message from the response.

First of all, we had to create inmail.analyzer.footer config entity and the corresponding analyzer plugin - FooterAnalyzer. Since footer, subject and content type properties are relevant for all types of mail messages supported by Mailhandler, these properties were put in MailhandlerAnalyzerResultBase class.

FooterAnalyzer currently depends on the analyzed result provided by MailhandlerAnalyzer. The reason why one plugin depends on another is to support PGP signed messages. MailhandlerAnalyzer will try to analyze the message body of signed (and unsigned) messages and extract the actual mail body. Next, FooterAnalyzer will parse the processed body stored in MailhandlerAnalyzerResult. As mentioned above, the footer analyzer currently supports footers separated by -- \n and On {day}, {month} {date}, {year} at {hour}:{minute} {AM|PM} lines. The content after these lines is put into the footer property of the analyzer result. In case the body message has one of the supported separators, detected footer is stripped out from the actual message body.

Furthermore, the content type detection via message subject has been implemented. As we are going to support creating comments via email in the following weeks, we had to create a “protocol” that will allow us to differentiate between nodes and comments. We agreed to add [{entity_type}][{bundle}] before the actual message subject. For now, only node entity type and its bundle (content/node type) are parsed and extracted. All the assertions of the analyzed message are happening in the handler plugin (MailhandlerNode). The handler plugin will check if the configured content type is set to “Detect” mode and if so, it will get the parsed content type and create an entity of the parsed node type.

This week, students and their mentors are requested to submit mid-term evaluations. The evaluation represents a sum of the project after 5 weeks of the work. By finishing FooterAnalyzer, Mailhandler is now capable of processing signed (and unsigned) emails, extracting the actual body and creating a node of the detected node type for an authorized user.

The plan for the next week is to extend the project with validation support. We will use entity (node) validation and extend content type to bundle validation too. Also, I will work on splitting the Mailhandler analyzer to the smaller analyzers and adapting the handler to the changes.

Jun 20 2016
Jun 20

This year a significant number of students are working on RTC-related projects as part of Google Summer of Code, under the umbrella of the Debian Project. You may have already encountered some of them blogging on Planet or participating in mailing lists and IRC.

WebRTC plugins for popular CMS and web frameworks

There are already a range of pseudo-WebRTC plugins available for CMS and blogging platforms like WordPress, unfortunately, many of them are either not releasing all their source code, locking users into their own servers or requiring the users to download potentially untrustworthy browser plugins (also without any source code) to use them.

Mesut is making plugins for genuinely free WebRTC with open standards like SIP. He has recently created the WPCall plugin for WordPress, based on the highly successful DruCall plugin for WebRTC in Drupal.

Keerthana has started creating a similar plugin for MediaWiki.

What is great about these plugins is that they don't require any browser plugins and they work with any server-side SIP infrastructure that you choose. Whether you are routing calls into a call center or simply using them on a personal blog, they are quick and convenient to install. Hopefully they will be made available as packages, like the DruCall packages for Debian and Ubuntu, enabling even faster installation with all dependencies.

Would you like to try running these plugins yourself and provide feedback to the students? Would you like to help deploy them for online communities using Drupal, WordPress or MediaWiki to power their web sites? Please come and discuss them with us in the Free-RTC mailing list.

You can read more about how to run your own SIP proxy for WebRTC in the RTC Quick Start Guide.

Finding all the phone numbers and ham radio callsigns in old emails

Do you have phone numbers and other contact details such as ham radio callsigns in old emails? Would you like a quick way to data-mine your inbox to find them and help migrate them to your address book?

Jaminy is working on Python scripts to do just that. Her project takes some inspiration from the Telify plugin for Firefox, which detects phone numbers in web pages and converts them to hyperlinks for click-to-dial. The popular libphonenumber from Google, used to format numbers on Android phones, is being used to help normalize any numbers found. If you would like to test the code against your own mailbox and address book, please make contact in the #debian-data channel on IRC.

A truly peer-to-peer alternative to SIP, XMPP and WebRTC

The team at Savoir Faire Linux has been busy building the Ring softphone, a truly peer-to-peer solution based on the OpenDHT distribution hash table technology.

Several students (Simon, Olivier, Nicolas and Alok) are actively collaborating on this project, some of them have been fortunate enough to participate at SFL's offices in Montreal, Canada. These GSoC projects have also provided a great opportunity to raise Debian's profile in Montreal ahead of DebConf17 next year.

Linux Desktop Telepathy framework and reSIProcate

Another group of students, Mateus, Udit and Balram have been busy working on C++ projects involving the Telepathy framework and the reSIProcate SIP stack. Telepathy is the framework behind popular softphones such as GNOME Empathy that are installed by default on the GNU/Linux desktop.

I previously wrote about starting a new SIP-based connection manager for Telepathy based on reSIProcate. Using reSIProcate means more comprehensive support for all the features of SIP, better NAT traversal, IPv6 support, NAPTR support and TLS support. The combined impact of all these features is much greater connectivity and much greater convenience.

The students are extending that work, completing the buddy list functionality, improving error handling and looking at interaction with XMPP.

Streamlining provisioning of SIP accounts

Currently there is some manual effort for each user to take the SIP account settings from their Internet Telephony Service Provider (ITSP) and transpose these into the account settings required by their softphone.

Pranav has been working to close that gap, creating a JAR that can be embedded in Java softphones such as Jitsi, Lumicall and CSipSimple to automate as much of the provisioning process as possible. ITSPs are encouraged to test this client against their services and will be able to add details specific to their service through Github pull requests.

The project also hopes to provide streamlined provisioning mechanisms for privately operated SIP PBXes, such as the Asterisk and FreeSWITCH servers used in small businesses.

Improving SIP support in Apache Camel and the Jitsi softphone

Apache Camel's SIP component and the widely known Jitsi softphone both use the JAIN SIP library for Java.

Nik has been looking at issues faced by SIP users in both projects, adding support for the MESSAGE method in camel-sip and looking at why users sometimes see multiple password prompts for SIP accounts in Jitsi.

If you are trying either of these projects, you are very welcome to come and discuss them on the mailing lists, Camel users and Jitsi users.

GSoC students at DebConf16 and DebConf17 and other events

Many of us have been lucky to meet GSoC students attending DebConf, FOSDEM and other events in the past. From this year, Google now expects the students to complete GSoC before they become eligible for any travel assistance. Some of the students will still be at DebConf16 next month, assisted by the regular travel budget and the diversity funding initiative. Nik and Mesut were already able to travel to Vienna for the recent MiniDebConf / LinuxWochen.at

As mentioned earlier, several of the students and the mentors at Savoir Faire Linux are based in Montreal, Canada, the destination for DebConf17 next year and it is great to see the momentum already building for an event that promises to be very big.

Explore the world of Free Real-Time Communications (RTC)

If you are interesting in knowing more about the Free RTC topic, you may find the following resources helpful:

RTC mentoring team 2016

We have been very fortunate to build a large team of mentors around the RTC-themed projects for 2016. Many of them are first time GSoC mentors and/or new to the Debian community. Some have successfully completed GSoC as students in the past. Each of them brings unique experience and leadership in their domain.

Helping GSoC projects in 2016 and beyond

Not everybody wants to commit to being a dedicated mentor for a GSoC student. In fact, there are many ways to help without being a mentor and many benefits of doing so.

Simply looking out for potential applicants for future rounds of GSoC and referring them to the debian-outreach mailing list or an existing mentor helps ensure we can identify talented students early and design projects around their capabilities and interests.

Testing the projects on an ad-hoc basis, greeting the students at DebConf and reading over the student wikis to find out where they are and introduce them to other developers in their area are all possible ways to help the projects succeed and foster long term engagement.

Google gives Debian a USD $500 grant for each student who completes a project successfully this year. If all 2016 students pass, that is over $10,000 to support Debian's mission.

Jun 15 2016
Jun 15

The web development community can have a long list of requirements, languages, frameworks, constructs and tools that most companies or bosses want you to know.

This list may not include everything you need to know including PHP, HTML, CSS, responsive web development principles, and Drupalisms. Here is the list of some of the important skills, concepts, and tools that we think you should know as a beginner Drupal developer.

1. Version Control

Every developer should have some experience with version control and versioning. Version control is an essential part of the Drupal community. Versioning allows for Drupal projects to be easily managed, maintained and contributed in a uniform manner. Version control will also most likely be used in-house to manage each client project as well.

2. Command Line Interface (CLI)

It isn't necessary to be a CLI Ninja, however being able to work comfortably using a CLI is very important. One of the advantages to using a CLI is the ability to be more productive. You can quickly automate repetitive tasks, perform tasks without jumping from application to application, and the ability to use tools like Drush to perform tasks that would normally require you to navigate 3 or more mouse clicks to accomplish.

3. Package Managers

Using package managers is important to the installation of Drupal. Whether it is installing Sass or Bootstrap from node or Drush from composer, it is important to know how package managers work and exactly what you are running before running commands on your computer.

4. Contributing Back

An important part of the Drupal community is contributing back to projects and core. When you find an issue, such as something that just doesn't seem to work correctly, or you would like to implement a functionality to Drupal, you should think about giving back to the community. If you find an issue on an existing project or core, check to see if there is an existing ticket on that project. If there isn't, you can create one, and if you can debug it and resolve the issue you can contribute a patch to that issue. If you don't know exactly how to debug the issue you can have an open conversation with other developers and maintainers to help resolve the issue. Contributing and interacting in the community moves Drupal forward.

5. CSS Preprocessors

Within the last couple of years, there has been a movement to CSS preprocessors to add a programmatic feel to CSS2 and CSS3. There are some that are against preprocessors because it adds a little more overhead to a project. Whether you use them or not, you may have a client or framework that uses one that you might need to be familiar with how to use a preprocessor.

6. A Framework

Within the Drupal community, there is often talk of headless Drupal. We have seen some interesting ideas come from the adopters of headless Drupal. Headless Drupal setups usually use a framework for the front-end. It may be Angular, Angular 2, Backbone, Ember or something different, however, most of the frameworks have two things in common, they are often written in Javascript and almost always make use of templating.

7. Templating

It is important to know the principles of templating so that you can easily pick up and learn new frameworks. Whether it is Mustache, Twig, Jade, or the templating syntax from within Angular, there are similarities between the syntax and the principles can be applied to each of the languages that will allow you to quickly step from one to the next with a smaller learning curve.

8. Basic Debugging

Debugging a problem correctly can save you valuable time by getting you directly to the cause of an issue instead of looking over each line of code one by one. It is essential to know how to do basic debugging when working with Drupal. Sometimes the error messages can give you enough information, other times it is necessary to step into Devel or XDebug and step through the project to find the exact location where the code is not working correctly so that you can start to solve the problem.

9. Unit Testing / Code Testing

Testing your own code is important. When it comes to code testing you have many options, from TTD and BDD you can write unit tests to cover your classes, linting to make sure you are writing "good", standardized code. Linting can be helpful for writing code that others can easily navigate and sets up some best practices for you to follow.

10. A CMS

When starting with Drupal, it might be good to have familiarity with a CMS platform before jumping in. There are some advantages to knowing the constructs of other CMS platforms and being familiar with how to work within a platform. However, when working with Drupal it is important to think about the way Drupal works and not be stuck in the way other CMS platforms accomplish goals.

Conclusion

As a web developer, it is important to know many concepts and technologies. Many companies will not require you to know everything, do everything and be a jack-of-all-trades. In technology, there are so many new tools, frameworks, and languages coming out daily that it is impossible to stay on top of them all. It is far better to get a good base understanding of core web concepts that can be applied to multiple languages, tools, and technologies and then specialize.

Did I miss something you feel is important? Is there something you would like to have seen on the list? Leave a comment below.

Jun 14 2016
Jun 14

This blog post summarizes the third week of the Google Summer of Code 2016 project - Mailhandler.

As mentioned in the previous blog post, the Mailhandler prototype for Drupal 8 has been finished. The prototype has already one of the main features implemented - processing mail messages and creating nodes. However, processed messages could be created by anyone which is not really nice for a module aiming to be used in production. To solve this problem, I've been working on user authentication implementation for almost a week.

The goal was to authenticate a user (sender of an email) before doing any mail processing. The meta issue with its child issues formalizes the requirements.

An obvious way to identify a mail sender is by checking the From field from the mail header. As this field is required to be present in mail headers (RFC 2822) we can assume we will always be able to identify the sender's mail address.
While writing a blog post about building a prototype, I acquainted you with Inmail. This is the module we are going to rely on to process mail messages. Inmail has a nice concept of plugins separated into deliverers, analyzers, and handlers. Deliverers are used to specify where a mail is coming from, analyzers to identify different types of messages and handlers to do some actions with mail messages and analyzer results. Last week, I already added a handler plugin and here I am going to talk about analyzers.

I started implementing an Inmail analyzer plugin and named it MailhandlerAnalyzer. It inherits AnalyzerBase from Inmail and currently has only one method - analyze() which accepts $message (represents a mail message entity) and processor_result (collects outcomes of a single mail processing pass) parameters. To preserve analyzer results, MailhandlerAnalyzerResult is created with $sender, $user and $body properties. The $sender will represent a sender's mail address (obtained from From mail header field), $user is a user entity in the current system and the $body is analyzed message body. After populating those fields in the analyzer, the job of MailhandlerNode is to authenticate and authorize the sender. If the conditions are met (user is validated) the handler will process the message and do the actions (create a node). This gives us a basic level of security.

However, From mail handler field can be faked by a malicious user. To bridge this obstacle, we introduced support for digitally signed emails. By its definition: digital signature is a mathematical scheme for demonstrating the authenticity of digital documents and it provides:

  • Authentication - since the secret (private) key is bound to a specific user, successful signature verification will show the identity of the sender
  • Integrity - it guarantees the message was not changed during transmission. Any change of the message content after signing it invalidates the signature
  • Non-repudiation - a user can not deny sending and signing the message

Handling digital signature is not supported in PHP by default and we had to find the tools to do it. We decided to use GnuPG implementation of the OpenPGP (Pretty Good Privacy) standard defined by RFC 4880 and gnupg PHP extension. If you are not familiar with the concept of digital signature I would recommend you looking into this ~2 minutes read.

As a starting point, we have to add mailhandler_gpg_key field to store GPG Key for each user. I wanted to preserve public_key and fingerprint properties for each key and decided to go with custom field type. Drupal.org documentation provides a nice explanation how to create a custom field type, corresponding widgets and formatters in Drupal 8.

Mailhandler GPG Key fieldMailhandler GPG Key field

The next thing in our PGP journey is to extend analyzer and handlers to deal with signed messages. Usually, when handling emails, you will have to deal with regular expressions a lot. This case is not an exception. A new method called isSigned() is added to MailhandlerAnalyzer which analyzes the message and returns a boolean value. While we are dealing with messages at the low level, isSigned() populates the context data in case of signed messages. Context consists of:

  • PGP Type - the signed message could be inline-signed (clear-text signature) or as nowadays standard PGP/MIME.
  • Signed text - extracted the actual body without PGP signature
  • Signature - PGP signature

Populating the general result properties of the analyzer result instance (In case of signed messages MailhandlerAnalyzerResultSigned), the analyzer finishes its job. At this point, we can analyze the signed message and be sure if we are dealing with signed or unsigned mail messages.

Next, we have to adapt our handler to support signed messages too. That looks easy, as all the hard work is completed by the analyzer. The only thing we have to do is to verify the signature. We do that by getting the signed text and signature from the analyzer result and pass them to gnupg_verify() function. If the signature is valid and the corresponding user's GPG key is not disabled, revoked or expired we will continue the handling process. One last step to check before creating a node is to assert the user is authorized for that action.

To summarize, implementing support for digitally signed emails certainly provides an acceptable level of security for Mailhandler module. Keep in mind, that you will need to have GnuPG PHP extension enabled on your server to get GPG support. In either way, you will benefit from Mailhandler's default authentication system. The interesting thing about last week is that authentication support produced more than 1,2k new lines of code in the Github pull request

Next week, I am going to work on footer detection in a mail message. This will allow us to remove any unnecessary and unintended content from a node body. To support all those features, most of the work will be done in the analyzer. The mid-term evaluation period starts in less than a week which means the next week will be used to do preparation for it as well.

Over and out.

Jun 14 2016
Jun 14
TL;DR Safe Search detection of the Google Cloud Vision API allows the end users to avoid any explicit content or violence in images to be uploaded on the site. I worked on integrating this feature to the module as a constraint to those image fields which have the “Safe Search” feature enabled.

Let me first give a brief summary about the current status of the module Google Vision. In the earlier weeks, I had implemented the services and wrote tests for the module, which are now committed to the module.

Now, coming to the progress of this week.

I had started with integrating the Safe Search detection feature in the module. Safe Search detection allows its users to detect any explicit contents within the image, and hence can be very useful for site administrators who do not want to display any explicit images on their sites.

This feature was initially integrated using a function call in the module. However, my mentors suggested that this feature should rather be a Constraint on the image fields, which would be validated if the feature is enabled for the field. It depends on the user whether to keep safe search on their site or not, and they can implement it any time they want by just enabling/disabling the checkbox on the image field. Hence, now it is a user choice, rather than the developer’s choice whether to implement this feature or not.

Presently, the code is under review by my mentors whether it needs changes or is ready for commit.

Constraints and Validators are wonderful features of Drupal 8. Constraints, as the name goes, are certain restrictions which we pose on the various fields. Validators are used to implement the logic to create the situation under which the constraints are to be applied. Some helpful examples of applying custom constraints and validators can also be found Sitepoint.
This week had a lot of new things stored for me. I had no idea about the constraints and validators when I was asked to implement them at the first place. I spent hours on them, learning about them and seeking guidance from my mentors on the issues I faced. I developed gradually on it, and by the end of the week, I was able to implement them for the safe search detection feature.
Jun 08 2016
Jun 08

GSoC students have officially been coding since 23 May (about 2.5 weeks) and are almost half-way to the mid-summer evaluation (20 - 27 June). Students who haven't completed some meaningful work before that deadline don't receive payment and in such a large program, there is no possibility to give students extensions or let them try and catch up later.

Every project and every student are different, some are still getting to know their environment while others have already done enough to pass the mid-summer evaluation.

I'd like to share a few tips to help students ensure they don't inadvertently fail the mid-summer evaluation

Kill electronic distractions

As a developer of real-time communications projects, many people will find it ironic or hypocritical that this is at the top of my list.

Switch off the mobile phone or put it in silent mode so it doesn't even vibrate. Research has suggested that physically turning it off and putting it out of sight has significant benefits. Disabling the voicemail service can be an effective way of making sure no time is lost listening to a bunch of messages later. Some people may grumble at first but if they respect you, they'll get into the habit of emailing you and waiting for you to respond when you are not working.

Get out a piece of paper and make a list of all the desktop notifications on your computer, whether they are from incoming emails, social media, automatic updates, security alerts or whatever else. Then figure out how to disable them all one-by-one.

Use email to schedule fixed times for meetings with mentors. Some teams/projects also have fixed daily or weekly times for IRC chat. For a development project like GSoC, it is not necessary or productive to be constantly on call for 3 straight months.

Commit every day

Habits are a powerful thing. Successful students have a habit of making at least one commit every day. The "C" in GSoC is for Code and commits are a good way to prove that coding is taking place.

GSoC is not a job, it is like a freelance project. There is no safety-net for students who get sick or have an accident and mentors are not bosses, each student is expected to be their own boss. Although Google has started recommending students work full time, 40 hours per week, it is unlikely any mentors have any way to validate these hours. Mentors can look for a commit log, however, and simply won't be able to pass a student if there isn't code.

There may be one day per week where a student writes a blog or investigates a particularly difficult bug and puts a detailed report in the bug tracker but by the time we reach the second or third week of GSoC, most students are making at least one commit in 3 days out of every 5.

Consider working away from home/family/friends

Can you work without anybody interrupting you for at least five or six hours every day?

Do you feel pressure to help with housework, cooking, siblings or other relatives? Even if there is no pressure to do these things, do you find yourself wandering away from the computer to deal with them anyway?

Do family, friends or housemates engage in social activities, games or other things in close proximity to where you work?

All these things can make a difference between passing and failing.

Maybe these things were tolerable during high school or university. GSoC, however, is a stepping stone into professional life and that means making a conscious decision to shut those things out and focus. Some students have the ability to manage these distractions well, but it is not for everybody. Think about how leading sports stars or musicians find a time and space to be "in the zone" when training or rehearsing, this is where great developers need to be too.

Some students find the right space in a public library or campus computer lab. Some students have been working in hacker spaces or at empty desks in local IT companies. These environments can also provide great networking opportunities.

Managing another summer job concurrently with GSoC

It is no secret that some GSoC students have another job as well. Sometimes the mentor is aware of it, sometimes it has not been disclosed.

The fact is, some students have passed GSoC while doing a summer job or internship concurrently but some have also failed badly in both GSoC and their summer job. Choosing one or the other is the best way to succeed, get the best results and maximize the quality of learning and community interaction. For students in this situation, now it is not too late to make the decision to withdraw from GSoC or the other job.

If doing a summer job concurrently with GSoC is unavoidable, the chance of success can be greatly increased by doing the GSoC work in the mornings, before starting the other job. Some students have found that they actually finish more quickly and produce better work when GSoC is constrained to a period of 4 or 5 hours each morning and their other job is only in the afternoon. On the other hand, if a student doesn't have the motivation or energy to get up and work on GSoC before the other job then this is a strong sign that it is better to withdraw from GSoC now.

Jun 08 2016
Jun 08

Last month, I gave the third in a series of front-end web development talks here in Taipei. The 10 attendees and I discussed style-guide-centric development and front-end development automation.

The seminar was designed to be hands-on, so we started by installing Node.js and downloading Zen 8.x-7.0-alpha8 so we could play with code while discussing the evening’s topics.

Putting the style guide at the center of your development cycle is something that I’ve talked about many times before. As you build out each component of your design, it immediately goes into a style guide. Visualizing the component in an isolated manner makes it easier to construct the component, its elements, and its variations.

While I’ve given talks and written blog posts on this subject before, I’ve never built a style guide for a Drupal 8 site. This third talk in the Taipei seminars gave me the chance to build incrementally on the previous two talks, component design and Drupal 8’s Twig. Ideally your style guide should be using the exact same CSS, HTML and JS that your application is using. With Drupal 7, integrating the PHPTemplate system into a style guide was a non-starter for me; technically possible, but too difficult to invest the number of hours it would require. In the week leading up to this seminar, I started porting the Zen theme to Drupal 8.

Using the Twig template language we played with during the second seminar, I showed the attendees for the first time how I imagined you could build a Twig-based component library and integrate it with Drupal’s theme layer.

With vanilla Twig, you can construct a very simple component with the exact HTML you wanted and the exact variable names you wanted. Then you can convert Drupal’s very Drupal-ly variables and render arrays into the variables that your component expects. Just use the *.html.twig template to convert the variables from Drupal into the variables needed for the component and then extend/embed/include the *.twig component.

I’ll write a more detailed blog post on this soon. But it was really cool to share this method for the first time and get some early feedback.

From there I showed attendees how to “start living the dream” by taking the component written with Twig, CSS and JS and pull it directly into a style guide with KSS comments and the first of our automation tasks. Since we’ve pulled out the Drupal-specific variables out of our Twig files, that allows kss-node to use the component Twig file with Twig.js (the Node.js port of Twig). We have once set of HTML (the .twig file), of CSS, and of JS and both the style guide builder and Drupal uses it. Hooray!

We then launched into using a task runner to control all of the front-end development tasks we need to do. Automating our best practices means we can spare time for improving our craft.

I showed them Gulp.js and its very simple API that makes writing new tasks a breeze.

The most fun part of the evening when was when someone asked me to demo the Browser Sync task that comes with Zen. I ran “gulp watch” from my command line and that started up a BrowserSync proxy server that served up my Drupal site and style guide on a special URL. Then all of the attendees went to that URL and I could control the scrolling of the page on their machines with the mouse on my computer. It was pretty cool.

This seminar was amazing for me. I love doing hands-on deep dives of concepts and technology. If you missed the class or don’t live in Taipei, I’m available to teach your organization remotely or in person. Just contact me!

Jun 08 2016
Jun 08

Drupal-8-manina is at its highest. Modules are being ported, blog posts are being written, and new sites are being coded, so we in Gizra decided to join the party.

We started with a simple site that will replace an existing static site. But we needed to migrate node attachments, and we just couldn’t find an existing solution. Well, it was time to reach out to the community

Any example of #Drupal 8 migration of files/ images out there? (including copy from source into public:// )

— Amitai Burstein (@amitaibu) April 8, 2016

A few minutes after the tweet was published, we received a great hint from the fine folks at Evoloving Web. They were already migrating files into Drupal 8 from Drupal 7, and were kind enough to blog post about it.

However, we were still missing another piece of the puzzle, as we also wanted to migrate files from an outside directory directly into Drupal. I gave my good friend @jsacksick a poke (it’s easy, as he sits right in front of me), and he gave me the answer on a silver platter.

Post has a happy end - we were able to migrate files and attach to a node!

An example for super heroes

For this blog post I created a dummy Drupal 8 installation profile with way too much information about super heroes. The migration module can migrate some images along with some CSV data about them.

If you’ll look closely you can see that I’ve attached an SQL dump with raw tables. This raw table will be the source that eventually will migrated into nodes, and you can read here how it was created with csv2sql.

Basic structure of migration

The description of the mapping between the source table and the destination node type is in a yaml file.

Let’s go over the interesting parts of the process:

process:
  type:
    plugin: default_value
    default_value: super_heroes
  uid:
    plugin: default_value
    default_value: 1
  title: _title
  field_image:
    source: _image
    plugin: file_import
  field_alter_ego: _alter_ego
  'body/value': _description

In Drupal 7, because we wanted to prepare the value before populating the entity fields, we did it in a prepare method. In Drupal 8 we have process plugins.

For example, the default_value plugin will populate the (configurable) field of the entity with a raw value like the name of a content type or a user ID, in case we are migrating all the nodes with the same author (e.g. user ID 1).

But we can, of course, have our own logic. In the transform method of the process plugin we can massage our data and return any value which will eventually populate the field.

In our case, the transform method is responsible for adding the new file into Drupal using file_unmanaged_copy and friends.

Conclusion

Some of the know hows and best practices are still missing from Drupal 8. But they can and should be re-learned and re-published. So remember kids, if you ever feel frustrated about not finding a solution, always reach out to smart community members and then write a post about it, so everybody can profit.

Jun 07 2016
Jun 07
TL;DR I have already created services for the functions which bridges the module to the API, implementing the features offered by the Google Cloud Vision API, thus completing my first step towards Integrating Google Cloud Vision API to Drupal 8. This week I worked on generating error reports if the API key is not set by the user, and developing tests to test the API key configuration and whether the key is stored successfully or not.

The first step towards the integration of Google Cloud Vision API to Drupal 8 was completed with the functions moved to services. I had posted the patch for the review by my mentors. They provided their suggestions on the patch, which I worked on, and every step resulting in a better and more beautiful code.

I would also like to share that this week, our team expanded from a team of three, to a team of four members. Yes! Eugene Ilyin, the original maintainer of the module Google Vision API has joined us to mentor me through the project.

Now coming to the progress of the project, the schedule says I need to configure the Google Cloud Vision API at taxonomy field level, so that the end users may use the taxonomy terms to get the desired response from the API. However, the module already used the configuration for Label Detection, and in a discussion with my mentors, it was figured out that the current configuration does not need any changes, as at present the behaviour is pretty clear and obvious to let the developers use it easily; rather we should work on implementing the runtime verification and storage of API key supplied by the end users. I was required to write and implement the code which would give error report if the API key was not saved prior to the use of the module, and also to write tests for verifying the configuration and ensuring the storage of the key.

I created patches and posted it in the issue to get it reviewed by my mentors. I brought the suggested changes in the code and finally have submitted the patch implementing the required functionalities. I also worked on this issue, solving and implementing minor suggestions before it gets ready to be accepted and committed! And finally, my first patch in this project has been accepted and the changes has been reflected in the module.
At the end of these two weeks, I learnt about services and dependency injection which prove to be very useful concepts implemented in Drupal 8. I also had experiences of writing tests to check the runtime functionality of the module.
Jun 07 2016
Jun 07

Continuing from Evan's blog post on building pages with Paragraphs and writing custom blocks of content as fields, I will walk you through how to create a custom field-formatter in Drupal 8 by example.

A field-formatter is the last piece of code to go with the field-type and the field-widget that Evan wrote about in the previous blog post. While the field-type tells Drupal about what data comprises a field, the field-formatter is responsible for telling Drupal how to display the data stored in the field.

To recap, we defined a hashtag_search field type in the previous blog post whose instances will be composed of two items: the hashtag to search for, and the number of items to display. We want to convert this data into a list of the most recent n tweets with the specified hashtag.

A field-formatter is a Drupal plugin, just like its respective field-type and field-widget. They live in /src/Plugin/Field/FieldFormatter/ and are namespaced appropriately: Drupal\\Plugin\Field\FieldFormatter.



namespace Drupal\my_module\Plugin\Field\FieldFormatter;


use Drupal\Core\Field\FieldItemListInterface;
use Drupal\Core\Field\FormatterBase;
use Drupal\Core\Form\FormStateInterface;


class HashtagFormatter extends FormatterBase
{

    public function viewElements(FieldItemListInterface $items, $langcode)
    {
        return array();
    }
}

We tell Drupal important details about our new field-formatter using a @FieldFormatter class annotation. We declare its unique id; a human-readable, translatable label; and a list of field_types that it supports.

The most important method in a field-formatter is the viewElements method. It's responsibility is returning a render array based on field data being passed as $items\Core\Field\FieldItemListInterface>.

Let's look at the code:



use Drupal\my_module\Twitter\TwitterClient;
use Drupal\my_module\Twitter\TweetFormatter;

...

    
    protected $twitter;

    
    protected $formatter;

    ...

    public function viewElements(FieldItemListInterface $items, $langcode)
    {
        $element = [];

        
        foreach ($items as $delta => $item) {

            try {

                
                $results = $this->twitter->search($item->hashtag_search, $item->count);

                
                
                
                $statuses = array_map(function ($s) {
                    $s['formatted_text'] = $this->formatter->toHtml($s['text'], $s['entities']);
                    return $s;
                }, $results['statuses']);

                
                if (!empty($statuses)) {
                    $element[$delta]['header'] = [
                        '#markup' => '

#'</span> . $item->hashtag_search . '

'
]; } foreach ($statuses as $status) { $element[$delta]['status'][] = [ '#theme' => 'my_module_status', '#status' => $status ]; } } catch (\Exception $e) { $this->logger->error('[:exception]: %message', [ ':exception' => get_class($e), '%message' => $e->getMessage(), ]); continue; } } $element['#attached']['library'][] = 'my_module/twitter_intents'; return $element; } ...

See https://github.com/bezhermoso/tweet-to-html-php for how TweetFormatter works. Also, you can find the source-code for the basic Twitter HTTP client here: https://gist.github.com/bezhermoso/5a04e03cedbc77f6662c03d774f784c5

Custom theme renderer

As shown above, each individual tweets are using the my_module_status render theme. We'll define it in the my_module.module file:




function my_module_theme($existing, $type, $theme, $path) {
  $theme = [];
  $theme['my_module_status'] = array(
    'variables' => array(
      'status' => NULL
    ),
    'template' => 'twitter-status',
    'render element' => 'element',
    'path' => $path . '/templates'
  );

  return $theme;
}

With this, we are telling Drupal to use the template file modules/my_module/templates/twitter-status.twig.html for any render array using my_module_status as its theme.

Render caching

Drupal 8 does a good job caching content: typically any field formatter is only called once and the resulting collective render arrays are cached for subsequent page loads until the Drupal cache is cleared. We don't really want our Twitter block to be cached for that long. Since it is always great practice to keep caching enabled, we can define how caching is to be applied to our Twitter blocks. This is done by adding cache definitions in the render array before we return it:



      public function viewElements(...)
      {

        ...

        $element['#attached']['library'][] = 'my_module/twitter_intents';
        
        $element['#cache']['max-age'] = 60 * 5;

        return $element;
      }

Here we are telling Drupal to keep the render array in cache for 5 minutes. Drupal will still cache the rest of the page's elements how they want to be cached, but will call our field formatter again -- which pulls fresh data from Twitter -- if 5 minutes has passed since the last time it was called.

Jun 07 2016
Jun 07

Today we held our inaugural #TheGizraWay webinar. The web series is intended to showcase some of “The Gizra Way” principles - a set of best practices and methodologies, borrowed from the Open Source development world and applied to operations, workflow, and overall company culture.

For the first in the series, we chose the topic of price estimations because it provides a real - and perhaps radical - example of how transparent communication from the beginning about a project’s needs alongside its budget can turn the process on it’s head. In the video below Amitai Burstein discusses how a budget-and time-driven discovery process gets a project off on the right foot.

[embedded content]

The next webinar - to take place in July 2016 - will be announced shortly. If you have any suggestions for topics or an idea that you would like to present in a future session let us know.

Pages

About Drupal Sun

Drupal Sun is an Evolving Web project. It allows you to:

  • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
  • Facet based on tags, author, or feed
  • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
  • View the entire article text inline, or in the context of the site where it was created

See the blog post at Evolving Web

Evolving Web