Apr 07 2019
Apr 07

April 07, 2019

Accessibility tests can be automated to a degree, but not completely; to succeed at accessibility, it needs to be a mindset shared by developers, UX and front-end folks, business people and other stakeholders. In this article, we will attempt to run tests and produce meaningful metrics which can help teams who are already committed to produce more accessible websites.

Premise

Say your team is developing a Drupal 8 site and you have decided that you want to reduce its accessibility issues by 50% over the course of six months.

In this article, we will look at a subset of accessibility issues which can be automatically checked – color contrast, placement of tags and HTML attributes, for example. Furthermore, we will only test the code itself with some dummy data, not actual live data or environment. Therefore, if you use the approach outlined in this article, it is best to do so within a global approach which includes stakeholder training; and automated and manual monitoring of live environments, all of which are outside the scope of this article.

Approach

Your team is probably perpetually “too busy” to fix accessibility issues; and therefore too busy to read and process reports with dozens, perhaps hundreds, of accessibility problems on thousands of pages.

Instead of expecting teams to process accessibility reports, we will use a threshold approach:

First, determine a standard towards which you’d like to work, for example WCAG 2.0 AA is more stringent than WCAG 2.0 A (but if you’re working on a U.S. Government website, WCAG 2 AA is mandated by the Americans with Disabilities Act). Be realistic as to the level of effort your team is ready to deploy.

Next (we’ll see how to do this later), figure out which pages you’d like to test against: perhaps one article, one event page, the home page, perhaps an internal page for logged in users.

In this article, to keep things simple, we’ll test for:

  • the home page;
  • an public-facing internal page, /node/1;
  • the /user page for users who are logged in;
  • the node editing form at /node/1/edit (for users who are logged in, obviously).

Running accessibility checks on each of the above pages, we will end up with our baseline threshold, the current number of errors, for example this might be:

  • 6 for the home page
  • 6 for /node/1
  • 10 for /user
  • 10 for /node/1/edit

We will then make our tests fail if there are more errors on a given page than we allow for. The test should pass at first, and this approach meets several objectives:

  • First, have an idea of the state of your site: are there 10 accessibility errors on the home page, or 1000?
  • Fail immediately if a developer opens a pull request where the number of accessibility errors increases past the threshold for any given page. For example, if a widget is added to the /user page which makes the number of accessibility errors jump to 12 (in this example), we should see a failure in our continuous integration infrastructure because 12 >= 10.
  • Provide your team with the tools to reduce the threshold over time. Concretely, a discussion with all stakeholders can be had once the initial metrics are in place; a decision might be made that we want to reduce thresholds for each page by 50% within 6 months. This allows your technical team to justify the prioritization of time spent on accessibility fixes vs. other tasks seen by able-bodied stakeholders as having a more direct business value.

Principles

Principle #1: Docker for everything

Because we want to run tests on a continuous integration server, we want to avoid dependencies. Specifically, we want a system which does not require us to install specific versions of MySQL, PHP, headless browsers, accessibility checkers, etc. All our dependencies will be embedded into our project using Docker and Docker Compose. That way, all you need to install in order to run your project and test for accessibility (and indeed other tests) is Docker, which in most cases includes Docker Compose.

Principle #2: A starter database

In our continous integration setup, will will be testing our code on every commit. Although it can be useful to test, or monitor, a remote environment such as the live or staging site, this is not what this article is about. This means we need some way to include dummy data into our codebase. We will do this by adding dummy data into a “starter database” committed to version control. (Be careful not to rely on this starter database to move configuration to the production site – use configuration management for that – we only want to store dummy data in our starter database; all configuration should be in code.) In our example, our starter database will contain node/1 with some realistic dummy data. This is required because as part of our test we want to run accessibility checks agains /node/1 and /node/1/edit.

A good practice during development would be that for new data types, say a new content type “sandwich”, a new version of the starter database be created with, say, node/2 of type “sandwich”, with realistic data in all its fields. This will allow us to add an accessibility test for /node/2, and /node/2/edit if we wish.

Don’t forget, as per principle #1, above, you will never need to install anything other than Docker on your computer or CI server, so don’t attempt to install these tools locally, they will run on Docker containers which will be built automatically for you.

  • Pa11y: There are dozens of tools to check for accessibility; in this article we’ve settled on Pa11y because it provides clear error reports; and allows the concept of a threshold above which the script fails.
  • Chromium: In order to check a page for accessibility issues without actually having a browser open, a so-called headless browser is needed. Chromium is a fully functional browser which works on the command line and can be scripted. This works under the hood and you will have no need to install it or interact with it directly, it’s just good to know it’s there.
  • Puppeteer: most accessibility tools, including Pa11y, are good at testing one page. Say, if you point Pa11y to /node/1 or the home page, it will generate nice reports with thresholds. However if you point Pa11y to /user or /node/1/edit it will see those pages anonymously, which is not what we want to test. This is where Puppeteer, a browser scripting tool, comes into play. We will use Puppeteer later on to log into our site and save the markup of /user and /node/1/edit as /dom-captures/user.html and /dom-captures/node-1-edit.html, respectively, which will then allow Pa11y to access and test those paths anonymously.
  • And of course, Drupal 8, although you could apply the technique in this article to any web technology, because our accessibility checks are run against the web pages just like an end user would see them; there is no interaction with Drupal.

Setup

To follow along, you can install and start Docker Desktop and download the Dcycle Drupal 8 starterkit.

git clone https://github.com/dcycle/starterkit-drupal8site.git
cd starterkit-drupal8site
./scripts/deploy.sh

You are also welcome to fork the project and link it to a free CircleCI account, in which case continuous integration tests should start running immediately on every commit.

A few minutes after running ./scripts/deploy.sh, you should see a login link to a full Drupal installation on a random local port (for example http://0.0.0.0:32769) with some dummy data (/node/1). Deploying this site locally or on a CI server such as Circle CI is a one-step, one-dependency process.

In the rest of this article we will refer to this local environment as http://0.0.0.0:YOUR_PORT; always substitute your own port number (in our example 32769) for YOUR_PORT.

Introducing Pa11y

We will use a Dockerized version of Pa11y, dcycle/pa11y, here is how it works against, say, amazon.com:

docker run --rm dcycle/pa11y:1 https://amazon.com

No site that I know of has zero accessibility issues; so you’ll see a bunch of issues in this format:

• Error: This element's role is "presentation" but contains child elements with semantic meaning.
  ├── WCAG2AA.Principle1.Guideline1_3.1_3_1.F92,ARIA4
  ├── #navFooter > div:nth-child(2)
  └── <div class="navFooterVerticalColumn navAccessibility" role="presentation"><div class="navFooterVerticalRo...</div>

Running Pa11y against a local site

Developers and continuous integration servers will need to run Pa11y against a local site. We would be tempted to run Pa11y on 0.0.0.0:YOUR_PORT, but that won’t work because Pa11y is being run inside its own container and will not have access to the host machine. You could give it access, but that raises another issue: the port is not guaranteed to be the same at every run, which requires ugly logic to figure out the port. Ugh! Instead, we will attach Pa11y to the Docker network used by our Starter site, in this case called starterkit_drupal8site_default (you can use docker network ls to list networks). Because our docker-compose.yml file defines the Drupal container as having the name drupal and port 80 (the default port), we can now run:

docker run --network starterkit_drupal8site_default \
  --rm dcycle/pa11y:1 http://drupal

This has some errors, just as we expected. Before doing anything else, type echo $?; this will give a non-zero code, meaning running this will make your continuous integration script fail. However, because we decided earlier that we will tolerate, for now, 6 errors on the home page, let’s set a threshold of 6 (or however many errors you get – there are 6 at the time of this writing) instead of the default zero:

docker run --network starterkit_drupal8site_default \
  --rm dcycle/pa11y:1 http://drupal --threshold 6

If you run echo $? right after, you should get the “passing” exit code of zero. There, we’ve met our threshold, so we will not have a failure!

How about pages where you need to be logged in?

The above solution breaks down, though, when you want to test http://drupal/node/1/edit. Although it will produce results, what we are actually checking against here is the “Access denied” page, not /node/1/edit when we are logged in. We will approach this in the following way:

  • Set a random password for user 1;
  • Use Puppeteer (see “Tools”, above) to click around your local site with its dummy data, do whatever you want to, and, every step of the way, save the DOM (the document object model, or the current markup after it has been processed by Javascript) as a temporary flat file, named, say, http://drupal/dom-captures/user.html;
  • Use Pa11y to test the temporary file we just created.

Putting it all together

In our Drupal 8 Starterkit, we can test the entire process. Start by running the Puppeteer script:

./scripts/end-to-end-tests.sh

What does this look like?

Astute readers have realized that using Puppeteer to click through the site to create our dom captures has the added benefit of confirming that our site functionality works as expected, which is why I called the script end-to-end-tests.sh.

To confirm this actually worked, you can visit, in an incognito window:

Yes it looks like you’re logged in, but you are not: these are anonymous webpages which Pa11y can check.

So if this worked correctly (and it should, because we hav it under continuous integration), we can run our Pa11y tests agains all these pages:

./scripts/a11y-tests.sh
echo $?

You will see the errors, but because the number of errors is below our threshold, the exit code will be zero, allowing our Continuous integration tests to pass.

Conclusion

Making a site accessible is, in my opinion, akin to making a site secure: it is not something to add to a to-do list, but rather an approach including all site stakeholders. Neither is accessibility something which can be automated; it really is a team culture. However, approaches like the one outlined in this article, or whatever works in your organization, will give teams metrics to facilitate the integration of accessibility into their day-to-day operations.

Please enable JavaScript to view the comments powered by Disqus.

Mar 14 2019
Mar 14

March 14, 2019

Often, during local Drupal development (or if we’re really unlucky, in production), we get the dreaded message, “Unable to send e-mail. Contact the site administrator if the problem persists.”

This can make it hard to debug anything email-related during local development.

Enter Mailhog

Mailhog is a dummy SMTP server with a browser GUI, which means you view all outgoing messages with a Gmail-type interface.

It is a major pain to install, but we can automate the entire process with the magic of Docker.

Let’s see how it works, and discuss after. Follow along by installing Docker Desktop – no other dependencies are required – and installing a Drupal 8 starterkit:

git clone https://github.com/dcycle/starterkit-drupal8site.git
cd starterkit-drupal8site
./scripts/deploy.sh

This will install the following Docker containers: a MySQL server with a starter database, a configured Drupal site, and Mailhog. You wil see something like this at the end of the output:

If all went well you can now access your site at:

=> Drupal: http://0.0.0.0:32791/user/reset/...
=> Dummy email client: http://0.0.0.0:32790

You might be seeing different port numbers instead of 32791 and 32790, so use your own instead of the example ports.

Now, the magic

(In my example, DRUPAL_PORT is 32791 and MAILHOG_PORT is 32790. In your case it will probably be different.)

As you can see, all emails produced by Drupal are now visible on a cool GUI!

So how does it work?

A dedicated “Mailhog” docker container, using on the Mailhog Docker image is defined in our docker-compose.yml file. It exposes port 8025 for public GUI access, which is mapped to a random unused port on the host computer (in the above example, 32790). Port 1025 is the SMTP mailhog port as you can see in the Mailhog Dockerfile. We are not mapping port 1025 to a random port on the host computer because it’s only needed in the Drupal container, not the host machine.

In the same docker-compose.yml, the “drupal” container (service) defines a link to the “mail” service; this means that when you are inside the Drupal container, you can access Mailhog SMPT server “mail” at port 1025.

In the Starterkit’s Dockerfile, we download the SMTP modules, and in our configuration, we install SMTP (0, in this case, is the module’s weight, it doesn’t mean “disabled”!).

Next, configuration: because this is for local development, we are leaving SMTP off in the exported configuration; in production we don’t want SMTP to link to Mailhog. Then, in our overridden settings, we enable SMTP and set the server to “mail” and the port to 1025.

Now, you can debug sent emails in a very realistic way!

You can remove the starterkit environment by running:

docker-compose down -v

Please enable JavaScript to view the comments powered by Disqus.

Oct 27 2018
Oct 27

October 27, 2018

This article discusses how to use HTTPS for local development if you use Docker and Docker Compose to develop Drupal 7 or Drupal 8 (indeed any other platform as well) projects. We’re assuming you already have a technique to deploy your code to production (either a build step, rsync, etc.).

In this article we will use the Drupal 8 site starterkit, a Docker Compose-based Drupal application that comes with everything you need to build a Drupal site with a few commands (including local HTTPS); we’ll then discuss how HTTPS works.

If you want to follow along, install and launch the latest version of Docker, make sure ports 80 and 443 are not used locally, and run these commands:

cd ~/Desktop
git clone https://github.com/dcycle/starterkit-drupal8site.git
cd starterkit-drupal8site
./scripts/https-deploy.sh

The script will prompt you for a domain (for example my-website.local) to access your local development environment. You might also be asked for your password if you want the script to add “127.0.0.1 my-website.local” to your /etc/hosts file. (If you do not want to supply your password, you can add that line to /etc/hosts before running ./scripts/https-deploy.sh).

After a few minutes you will be able to access a Drupal environment on http://my-website.local and https://my-website.local. For https, you will need to explicitly accept the certificate in the browser, because it’s self-signed.

Troubleshooting: if you get a connection error, try using an incongnito (private) window in your browser, or a different browser.

Being a security-conscious developer, you probably read through ./scripts/https-deploy.sh before running it on your computer. If you haven’t, you are encouraged to do so now, as we will be explaining how it works in this article.

You cannot use Let’s Encrypt locally

I often see questions related to setting up Let’s Encrypt for local development. This is not possible because the idea behind Let’s Encrypt is to certify that you own the domain on which you’re working; because no one uniquely owns localhost, or my-project.local, no one can get a certificate for it.

For local development, the Let’s Encrypt folks suggest using trusted, self-signed certificates instead, which is what we are doing in our script.

(If you are interested in setting up Let’s Encrypt for a publicly-available domain, this article is not for you. You might be interested, instead, in Letsencrypt HTTPS for Drupal on Docker and Deploying Letsencrypt with Docker-Compose.)

Make sure your project works without https first

So let’s look at how the ./scripts/https-deploy.sh script we used above works.

Let’s start by making sure our project works without https, then add a https access in a separate container.

In our starterkit project, you can run:

./scripts/deploy.sh

At the end of that scripts, you will see something like:

If all went well you can now access your site at:

 => http://0.0.0.0:32780/user/reset/...

Docker is serving our application using a random non-secure port, in this case 32780, and mapping it to port 80 on our container.

If you use Docker Compose for local development, you might have several applications running at the same time on different host ports, all mapped to port 80 on their respective container. At the end of this article you should be able to see each of them on port 443, something like:

The secret to all your local projects sharing port 443 is a reverse proxy container which receives requests to port 443, and indeed port 80 also, and acts as a sort of traffic cop to direct traffic the appropriate container.

That is why your individual projects should not directly use ports 80 and/or 443.

Adding an Nginx proxy container in front of your project’s container

An oft-seen approach to making your project available locally via HTTPS is to fiddle with your Dockerfile, installing openssl, setting up the certificate there; and rebuilding your container. This can work, but I would argue that it has significant drawbacks:

  • If you have several projects running on https port 443 locally, you could only develop one at a time because you only have one 443 port on your host machine.
  • You would need to maintain the SSL portion of your code for each of your projects.
  • It would go against the principle of separation of concerns which makes containers so robust.
  • You would be reinventing the wheel: there’s already a well-maintained Nginx proxy image which does exactly what you want.
  • Your job as a software developer is not to set up SSL.
  • If you decide to deploy your project to production Kubernetes cluster, it would longer makes sense for each of your Apache containers to support SSL.

For all those reasons, we will loosely couple our project with the act of serving it via HTTPS; we’ll leave our project alone and place an Nginx proxy in front of it to deal with the SSL/HTTPS portion of our local deployment.

Local https for one or more running projects

In this example we set up only one starterkit application, but real-world developers often need HTTPS with more than one application. Because you only have one local 443 port for HTTPS, We need a way to differentiate between our running applications.

Our approach will be for each of our projects to have an assigned local domain. This is why the https script we used in our example asked you to choose a domain like starterkit-drupal8.local.

Our script stored this information in the .env file at the root or your project, and also made sure it resolves to localhost in your /etc/hosts file.

Launching the Nginx reverse proxy

To me the terms “proxy” and “reverse proxy” are not intuitive. I’ll try to demystify them here.

The term “proxy” means something which represents something else; that term is already widely used to denote a web client being hidden from the user. So, a server might deliver content to a proxy which then delivers it to the end user, thereby hiding the end user from the server.

In our case we want to do the reverse: the client (you) is not placing a proxy in front of it; rather the application is placing a proxy in front of it, thereby hiding the project server from the browser: the browser communicates with Nginx, and Nginx communicates with your project.

Hence, “reverse proxy”.

Our reverse proxy uses a widely used and well-maintained GitHub project. The script you used earlier in this article launched a container based on that image.

Linking the reverse proxy to our application

With our starterkit application running on a random port (something like 32780) and our nginx proxy application running on ports 80 and 443, how are the two linked?

We now need to tell our Nginx proxy that when it receives a request for domain starterkit-drupal8.local, it should display our starterkit application.

There are a few steps to this, most handled by our script:

  • Your project’s docker-compose.yml file should look something like this: it needs to contain the environment variable VIRTUAL_HOST=${VIRTUAL_HOST}. This takes the VIRTUAL_HOST environment variable that our script added to the ./.env file, and makes it available inside the container.
  • Our script assumes that your project contains a ./scripts/deploy.sh file, which deploys our project to a random, non-secure port.
  • Our script assumes that only the Nginx Proxy container is published on ports 80 and 443, so if these ports are already used by something else, you’ll get an error.
  • Our script appends VIRTUAL_HOST=starterkit-drupal8.local to the ./.env file.
  • Our script attempts to add 127.0.0.1 starterkit-drupal8.local to our /etc/hosts file, which might require a password.
  • Our script finds the network your project is running on locally (all Docker-compose projects run on their own local named network), and gives the reverse proxy accesss to it.

That’s it!

You should now be able to access your project locally with https://starterkit-drupal8.local (port 443) and http://starterkit-drupal8.local (port 80), and apply this technique to any number of Docker Compose projects.

Troubleshooting: if you get a connection error, try using an incongnito (private) window in your browser, or a different browser; also note that you need to explicitly trust the certificate.

You can copy paste the script to your Docker Compose project at ./scripts/https-deploy.sh if:

  • Your ./docker-compose.yml contains the environment variable VIRTUAL_HOST=${VIRTUAL_HOST};
  • You have a script, ./scripts/deploy.sh, which launches a non-secure version of your application on a random port.

Happy coding!

Please enable JavaScript to view the comments powered by Disqus.

Oct 05 2018
Oct 05

October 05, 2018

I recently ran into a series of weird issues on my Acquia production environment which I traced back to some code I deployed which depended on my site being served securely using HTTPS.

Acquia Staging environments don’t use HTTPS by default and require you to install SSL certificates using a tedious manual process, which in my opinion is outdated, because competitors such as Platform.sh and Pantheon, Aegir, even Github pages support lots of automation around HTTPS using Let’s Encrypt.

Anyhow, because staging did not have HTTPS, I could not test some code I deployed, which ended up costing me an evening debugging an outage on a production environment. (Any difference between environments will eventually result in an outage.)

I found a great blog post which explains how to set up Let’s Encrypt on Acquia environments, Installing (FREE) Let’s Encrypt SSL Certificates on Acquia, by Chris at Redfin solutions, May 2, 2017. Although the process is very well documented, I made some tweaks:

  • First, I prefer using Docker-based solutions rather than install softward on my computer. So, instead of install certbot on my Mac, I opted to use the Certbot Docker Image, this has two advantages for me: first, I don’t need to install certbot on every machine I use this script on; and second, I don’t need to worry about updating certbot, as the Docker image is updated automatically. Of course, this does require that you install Docker on your machine.
  • Second, I automated everything I could. This result in this gist (a “gist” a basically a single file hosted on Github), a script which you can install locally.

Running the script

When you put the script locally on your computer (I added it to my project code), at, say ./scripts/set-up-letsencrypt-acquia-stage.sh, and run it:

  • the first time you run it, it will tell you where to put your environment information (in ./acquia-stage-letsencrypt-environments/environment-my-acquia-project-one.source, ./acquia-stage-letsencrypt-environments/environment-my-acquia-project-two.source, etc.), and what to put in those files.
  • the next time you run it, it will automate what it can and tell you exactly what you need to do manually.

I tried this and it works for creating new certs, and should work for renewals as well!

Please enable JavaScript to view the comments powered by Disqus.

Apr 07 2018
Apr 07

April 07, 2018

The process documented process for setting up a local environment and running tests locally is, in my opinion, so complex that it can be a barrier to even determined developers.

For those wishing to locally test and develop core patches, I think it is possible to automate the process down to a few steps and few minutes; here is an example with a core issue, #2273889 Don’t use one language’s plural index formula with another language’s string in the case of untranslated strings using format_plural(), which, at the time of this writing, results in the number 0 being displayed as 1 in certain cases.

Is it possible to start useful local development on this within 10 minutes on a computer with nothing installed except Docker? Let’s try…

Step 1: install Docker

Install and launch Docker. Everything we need, Apache web server, MySql server, Drush, Drupal, will reside on Docker containers, so we won’t need to install anything locally except Docker.

Step 2: launch a dev environment

I have create a project hosted on GitHub which will help you set up everything you need in Docker contains without local dependencies other than Docker, or any manual steps. Set it up by running:

git clone https://github.com/dcycle/drupal8_core_dev_helper.git && \
  cd drupal8_core_dev_helper && \
  ./scripts/deploy.sh`

This will create everything you need: a webserver container and database container, and your Drupal core code which will be placed in ./drupal8_core_dev_helper/drupal; near the end of the output of ./scripts/deploy.sh, you will see a login link to your development environment. Confirm you can access that local development environment at an address like http://0.0.0.0:SOME-PORT. (The port is random.)

The first time you run this, it will have to download Docker images with Drupal, MySQL, and install everything you need for local development. Future runs will be a lot faster.

See the project’s README for more details.

In your dev environment, you can confirm that the problem exists (provided the issue has not yet been fixed) by following the instructions in the “To reproduce this problem:” section of the issue description on your local development environment.

Any calls to drush can be run on the Docker container like so:

docker-compose exec drupal /bin/bash -c 'drush ...'

For example:

docker-compose exec drupal /bin/bash -c 'drush en locale language -y'

If you want to run drush directly, you can connect to your container like so:

docker-compose exec drupal /bin/bash

This will result in the following prompt on the container:

[email protected]:/var/www/html#

Now you can run drush commands directly on the container:

drush eval "print_r(\Drupal::translation()->formatPlural(0, '1 whatever', '@count whatevers', array(), array('langcode' => 'fr')) . PHP_EOL);"

Because the drupal8_core_dev_helper project also pre-installs devel on your environment, you can also confirm the problem exists by visiting /devel/php and executing:

dpm((string) (\Drupal::translation()->formatPlural(0, '1 whatever', '@count whatevers', array(), array('langcode' => 'fr'))));

Whether you do this by Drush or /devel/php, the result should be the same if the issue has not been resolved: 1 whatever instead of 0 whatevers.

Step 3: get a local version of the patch and apply it

In this example, we’ll look at the patch in comment #32 of our formatPlural issue, referenced above. If the issue has been resolved since this blog post has been written, follow along with another patch.

cd drupal8_core_dev_helper
curl https://www.drupal.org/files/issues/2018-04-07/2273889-31-core-8.5.x-plural-index-no-test.patch -O
cd ./drupal && patch -p1 < ../2273889-31-core-8.5.x-plural-index-no-test.patch

You have now patched your local version of Drupal. You can try the “0 whatevers” test again and the bug should be fixed.

Running tests

Now the real fun begins… and the “fast-track” ends.

For any patch to be considered for inclusion in Drupal core, it will need to (a) not break existing tests; and (b) provide a test which, without the patch, confirms that the problem exists.

Let’s head back to comment #32 of issue #2273889 and see if our patch is breaking anything. Clicking on “PHP 7 & MySQL 5.5 23,209 pass, 17 fail” will bring us to the test results page, which at first glance seems indecipherable. You’ll notice that our seemingly simple change to the PluralTranslatableMarkup.php file is causing a number of tests to fail: HelpEmptyPageTest, EntityTypeTest…

Let’s start by finding the test which is most likely to be directly related to our change by searching on the test results page for the string “PluralTranslatableMarkupTest” (this is name of the class we changed, with the word Test appended), which shows that it is failing:

Testing Drupal\Tests\Core\StringTranslation\PluralTranslatableMarkupTest
.E

We need to figure out where that file resides, by typing:

cd /path/to/drupal8_core_dev_helper/drupal/core
find . -name 'PluralTranslatableMarkupTest.php'

This tells us it is at ./tests/Drupal/Tests/Core/StringTranslation/PluralTranslatableMarkupTest.php.

Because we have a predictable Docker container, we can relatively easily run this test locally:

cd /path/to/drupal8_core_dev_helper
docker-compose exec drupal /bin/bash -c 'cd core && \
  ../vendor/bin/phpunit \
  ./tests/Drupal/Tests/Core/StringTranslation/PluralTranslatableMarkupTest.php'

You should now see the test results for only PluralTranslatableMarkupTest:

PHPUnit 6.5.7 by Sebastian Bergmann and contributors.

Testing Drupal\Tests\Core\StringTranslation\PluralTranslatableMarkupTest
.E                                                                  2 / 2 (100%)

Time: 16.48 seconds, Memory: 6.00MB

There was 1 error:

1) Drupal\Tests\Core\StringTranslation\PluralTranslatableMarkupTest::testPluralTranslatableMarkupSerialization with data set #1 (2, 'plural 2')
Error: Call to undefined method Mock_TranslationInterface_4be32af3::getStringTranslation()

/var/www/html/core/lib/Drupal/Core/StringTranslation/PluralTranslatableMarkup.php:150
/var/www/html/core/lib/Drupal/Core/StringTranslation/PluralTranslatableMarkup.php:121
/var/www/html/core/tests/Drupal/Tests/Core/StringTranslation/PluralTranslatableMarkupTest.php:31

ERRORS!
Tests: 2, Assertions: 1, Errors: 1.

How to fix this, indeed whether this will be fixed, is a whole nother story, a story fraught with dependency injection, mock objects, method stubs… More an adventure, really, than a story. An adventure which deserves to be told, just not right now.

Please enable JavaScript to view the comments powered by Disqus.

Jan 24 2018
Jan 24

January 24, 2018

Here are a few things I learned about caching for REST resources.

There are probably better ways to accomplish this, but here is what works for me.

Let’s say we have a REST resource that looks something like this in .../my_module/src/Plugin/rest/resource/MyRestResource.php and we have enabled it using the Rest UI module and given anonymous users permission to view it:

<?php

namespace Drupal\my_module\Plugin\rest\resource;

use Drupal\rest\ResourceResponse;

/**
 * This is just an example.
 *
 * @RestResource(
 *   id = "this_is_just_an_example",
 *   label = @Translation("Display the title of node 1"),
 *   uri_paths = {
 *     "canonical" = "/api/v1/get"
 *   }
 * )
 */
class MyRestResource extends ResourceBase {

  /**
   * {@inheritdoc}
   */
  public function get() {
    $node = node_load(1);
    $response = new ResourceResponse(
      [
        'title' => $node->getTitle(),
        'time' => time(),
      ]
    );
    return $response;
  }

}

Now, we can visit http://example.localhost/api/v1/get?_format=json and we will see something like:

{"title":"Some Title","time":1516803204}

Reloading the page, ‘time’ stays the same. That means caching is working; we are not re-computing our Json output each time someone requests it.

How to invalidate the cache when the title changes.

If we edit node 1 and change its title to, say, “Another title”, and reload http://example.localhost/api/v1/get?_format=json, we’ll see the old title. To make sure the cache is invalidated when this happens, we need to provide cacheability metadata to our response telling it when it needs to be recomputed.

Our node, when it’s loaded, contains within it all the caching metadata needed to describe when it should be recomputed: when the title changes, when new filters are added to the text format that’s being used, etc. We can add this information to our ResourceResponse like this:

...
$response->addCacheableDependency($node);
return $response;
...

When we clear our cache with drush cr and reload our page, we’ll see something like:

{"title":"Another title","time":1516804411}

Even more fun is changing the title of node 1 and reloading our Json page, and seeing the title and time change without clearing the cache:

{"title":"Yet another title","time":1516804481}

How to set custom cache invalidation events

Let’s say you want to trigger a cache rebuild for some reason other than those defined by the node itself (title change, etc.).

A real-world example might be events: an “upcoming events” page should only display events which start later than now. If we invalidate the cache every day, then we’ll never show yesterday’s events in our events feed. Here, we need to add our custom cache invalidation event, in this case “rebuild events feed”.

For the purpose of this demo, we won’t actually build an events feed, but we’ll see how cron might be able to trigger cache invalidation.

Let’s add the following code to our response:

...
use Drupal\Core\Cache\CacheableMetadata;
...
$response->addCacheableDependency($node);
$response->addCacheableDependency(CacheableMetadata::createFromRenderArray([
  '#cache' => [
    'tags' => [
      'rebuild-events-feed',
    ],
  ],
]));
return $response;
...

This uses Drupal’s cache tags concept and tells Drupal that when the cache tag ‘rebuild-events-feed’ is invalidated, all cacheable responses which have that cache tag should be invalidated as well. I prefer this to the ‘max-age’ cache tag because it allows us more fine-grained control over when to invalidate our caches.

On cron, we could only invalidate ‘rebuild-events-feed’ if events have passed since our last invalidation of that tag, for example.

For this example, we’ll just invalidate it manually. Clear your cache to begin using the new code (drush cr), then load the page, you will see something like:

{"hello":"Yet another title","time":1516805677}

As always, the time remains the same no matter how many times you reload the page.

Let’s say you are in the midst of a cron run and you have determined that you need to invalidate your cache for response which have the cache tag ‘rebuild-events-feed’, you can run:

\Drupal::service('cache_tags.invalidator')->invalidateTags(['rebuild-events-feed'])

Let’s do it in Drush to see it in action:

drush ev "\Drupal::service('cache_tags.invalidator')->\
  invalidateTags(['rebuild-events-feed'])"

We’ve just invalidated our ‘rebuild-events-feed’ tag and, hence, Responses that use it.

This one is beyond my competence level, but I wanted to mention it anyway.

Let’s say you want to output your node’s URL to Json, you might consider computing it using $node->toUrl()->toString(). This will give us “/node/1”.

Let’s add it to our code:

...
'title' => $node->getTitle(),
'url' => $node->toUrl()->toString(),
'time' => time(),
...

This results in a very ugly error which completely breaks your site (at least at the time of this writing): “The controller result claims to be providing relevant cache metadata, but leaked metadata was detected. Please ensure you are not rendering content too early.”.

The problem, it seems, is that Drupal detects that the URL object, like the node we saw earlier, contains its own internal information which tells it when its cache should be invalidated. Converting it to a string prevents the Response from being informed about that information somehow (again, if someone can explain this better than me, please leave a comment), so an exception is thrown.

The ‘toString()’ function has an optional parameter, “$collect_bubbleable_metadata”, which can be used to get not just a string, but also information about when its cache should be invalidated. In Drush, this will look like something like:

drush ev 'print_r(node_load(1)->toUrl()->toString(TRUE))'
Drupal\Core\GeneratedUrl Object
(
    [generatedUrl:protected] => /node/1
    [cacheContexts:protected] => Array
        (
        )

    [cacheTags:protected] => Array
        (
        )

    [cacheMaxAge:protected] => -1
    [attachments:protected] => Array
        (
        )

)

This changes the return type of toString(), though: toString() no longer returns a string but a GeneratedUrl, so this won’t work:

...
'title' => $node->getTitle(),
'url' => $node->toUrl()->toString(TRUE),
'time' => time(),
...

It gives us the error “Could not normalize object of type Drupal\Core\GeneratedUrl, no supporting normalizer found”.

ohthehugemanatee commented on Drupal.org on how to fix this. Integrating his suggestion, our code now looks like:

...
$url = $node->toUrl()->toString(TRUE);
$response = new ResourceResponse(
  [
    'title' => $node->getTitle(),
    'url' => $url->getGeneratedUrl(),
    'time' => time(),
  ]
);
$response->addCacheableDependency($node);
$response->addCacheableDependency($url);
...

This will now work as expected.

With all the fun we’re having, though, let’s take this a step further, let’s say we want to export the feed of frontpage items in our Response:

$url = $node->toUrl()->toString(TRUE);
$view = \Drupal\views\Views::getView("frontpage"); 
$view->setDisplay("feed_1");
$view_render_array = $view->render();
$rendered_view = render($view_render_array);

$response = new ResourceResponse(
  [
    'title' => $node->getTitle(),
    'url' => $url->getGeneratedUrl(),
    'view' => $rendered_view,
    'time' => time(),
  ]
);
$response->addCacheableDependency($node);
$response->addCacheableDependency($url);
$response->addCacheableDependency(CacheableMetadata::createFromRenderArray($view_render_array));

You will not be surpised to see the “leaked metadata was detected” error again… In fact you have come to love and expect this error at this point.

Here is where I’m completely out of my league; according to Crell, “[i]f you [use render() yourself], you’re wrong and you should fix your code “, but I’m not sure how to get a rendered view without using render() myself… I’ve implemented a variation on a comment on Drupal.org by mikejw suggesting using different render context to prevent Drupal from complaining.

$view_render_array = NULL;
$rendered_view = NULL;
\Drupal::service('renderer')->executeInRenderContext(new RenderContext(), function () use ($view, &$view_render_array, &$rendered_view) {
  $view_render_array = $view->render();
  $rendered_view = render($view_render_array);
});

If we check to make sure we have this line in our code:

$response->addCacheableDependency(CacheableMetadata::createFromRenderArray($view_render_array));

we’re telling our Response’s cache to invalidate whenever our view’s cache invaliates. So, for example, if we have several nodes promoted to the front page in our view, we can modify any one of them and our entire Response’s cache will be invalidated and rebuilt.

Resources and further reading

Please enable JavaScript to view the comments powered by Disqus.

Dec 18 2017
Dec 18

December 18, 2017

I recently needed to port hundreds of Drupal 7 webforms with thousands of submissions from Drupal 7 to Drupal 8.

My requirements were:

  • Node ids need to remain the same
  • Webforms need to be treated as data: they should be ignored by config export and import, just like nodes and taxonomy terms are. The reasonining is that in my setup, forms are managed by site editors, not developers. (This is not related to migration per se, but was a success criteria for my migration so I’ll document my solution here)

Migration from Drupal 7

I could not find a reliable upgrade or migration path from Drupal 7 to Drupal 8. I found webform_migrate lacks documentation (I don’t know where to start) and migrate_webform is meant for Drupal 6, not Drupal 7 as a source.

I settled on a my own combination of tools and workflows to perform the migration, all of them available on my Github account.

Using version 8.x-5.x of webform, I started by enabling webform, webform_node and webform_ui on my Drupal 8 site, this gives me an empty webform node type.

I then followed the instructions for a basic migration, which is outside the scope of this article. I have a project on Github which I use as starting point from my Drpual 6 and 7 to 8 migrations. The blog post Custom Drupal-to-Drupal Migrations with Migrate Tools, Drupalize.me, April 26, 2016 by William Hetherington provides more information on performing a basic migration of data.

Once you have set up your migration configurations as per those instructions, you should be able to run:

drush migrate-import upgrade_d7_node_webform --execute-dependencies

And you should see something like:

Processed 25 items (25 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_node_type'
Processed 11 items (11 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_user_role'
Processed 0 items (0 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_user_role'
Processed 95 items (95 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_user'
Processed 109 items (109 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_node_webform'

At this point I had all my webforms as nodes with the same node ids on Drupal 7 and Drupal 8, however this does nothing to import the actual forms or submissions.

Importing the data itself

I found that the most efficient way of importing the data was to create my own Drupal 8 module, which I have published on Dcycle’s Github account, called webform_d7_to_d8. (I have decided against publishing this on Drupal.org because I don’t plan on maintaining it long-term, and I don’t have the resources to combine efforts with existing webform migration modules.)

I did my best to make that module self-explanatory, so you should be able to follow the steps the README file, which I will summarize here:

Start by giving your Drupal 8 site access to your Drupal 7 database in ./sites/default/settings.php:

$databases['upgrade']['default'] = array (
  'database' => 'drupal7database',
  'username' => 'drupal7user',
  'password' => 'drupal7password',
  'prefix' => '',
  'host' => 'drupal7host',
  'port' => '3306',
  'namespace' => 'Drupal\\Core\\Database\\Driver\\mysql',
  'driver' => 'mysql',
);

Run the migration with our without options:

drush ev 'webform_d7_to_d8()'

or

drush ev 'webform_d7_to_d8(["nid" => 123])'

or

drush ev 'webform_d7_to_d8(["simulate" => TRUE])'

More detailed information can be found in the module’s README file.

Treating webforms as data

Once you have imported your webforms to Drupal 8, they are treated as configuration, that is, the Webform module assumes that developers, not site builders, will be creating the forms. This may be fine in many cases, however my usecase is that site editors want to create and edit forms directly on the production site, and we don’t want them to be tracked by the configuration management system.

Jacob Rockowitz pointed me in the right direction for making sure webforms are not treated as configuration. For that purpose I am using Drush CMI tools by Previous Next and documented on their blog post, Introducing Drush CMI tools, 24 Aug. 2016.

Once you install Drush CMI tools in your ~/.drush folder and run drush cc drush, you can use druch cexy and druch cimy instead of drush cim and drush cex in your conguration management process. Here is how and why:

Normally, if you develop your site locally and, say, add a content type or field, or remove a content type of field, you can run drush cex to export your newly created configuration. Then, your colleagues can pull your code and run drush cim to pull your configuration. drush cim can also be used in continuous integration, preproduction, dev, and production environments.

The problem is that drush cex exports all configuration, and drush cim deletes everything in the database which is not in configuration. In our case, we don’t want to consider webforms as configuration but as data, just as nodes as taxonomy terms: we don’t want them to be exported along with other configuration; and if they exist on a target environment we want to leave them as they are.

Using Drush CMI tools, you can add a file such as the following to ~/.drush/config-ignore.yml:

# See http://blog.dcycle.com/blog/2017-12-18
ignore:
  - webform.webform.*

This has to be done on all developers’ machines or, if you use Docker, on a shared Docker container (which is outside the scope of this article).

Now, for exporting configuration, run:

drush cexy --destination='/path/to/config/folder'

Now, webforms will not be exported along with other configuration.

We also need to avoid erasing webforms on target environments: if you create a webform on a target environment, then run drush cim, you will see something like:

webform.webform.webform_9521   delete
webform.webform.webform_8996   delete
webform.webform.webform_8991   delete
webform.webform.webform_8986   delete

So, we need to avoid deleting webforms on the target environment when we import configuration. We could just do drush cim --partial but this avoids deleting everything, not just webforms.

Drush CMI tools provides an alternative:

drush cimy --source=/path/to/config/folder

This works much like drush cim --partial, but it allows you to specify another parameter, –delete-list=/path/to/config-delete.yml

Then, in config-delete.yml, you can specify items that you actually want to delete on the target environment, for example content types, fields, and views which do not exist in code. This is dependent on your workflow and they way to set it up isdocumented on the Drush CMI tools project homepage.

With this in place, we’ll have our Drupal 7 webforms on our Drupal 8 site.

Please enable JavaScript to view the comments powered by Disqus.

Oct 03 2017
Oct 03

October 03, 2017

This article is about serving your Drupal Docker container, and/or any other container, via https with a valid Let’s encrypt SSL certificate.

Edit: if you’re having trouble with Docker-Compose, read this follow-up post.

Step one: make sure you have a public VM

To follow along, create a new virtual machine (VM) with Docker, for example using the “Docker” distribution in the “One-click apps” section of Digital Ocean.

This will not work on localhost, because in order to use Let’s Encrypt, you need to demonstrate ownership over your domain(s) to the outside world.

In this tutorial we will serve two different sites, one simple HTML site and one Drupal site, each using standard ports, on the same Docker host, using a reverse proxy, a container which sits in front of your other containers and directs traffic.

Step two: Set up two domains or subdomains you own and point them to your server

Start by making sure you have two domains which point to your server, in this example we’ll use:

  • test-one.example.com will be a simple HTML site.
  • test-two.example.com will be a Drupal site.

Step three: create your sites

We do not want to map our containers’ ports directly to our host ports using -p 80:80 -p 443:443 because we will have more than one app using the same port (the secure 443). Port mapping will be the responsibility of the reverse proxy (more on that later). Replace example.com with your own domain:

DOMAIN=example.com
docker run -d \
  -e "VIRTUAL_HOST=test-one.$DOMAIN" \
  -e "LETSENCRYPT_HOST=test-one.$DOMAIN" \
  -e "[email protected]$DOMAIN" \
  --expose 80 --name test-one \
  httpd
docker run -d \
  -e "VIRTUAL_HOST=test-two.$DOMAIN" \
  -e "LETSENCRYPT_HOST=test-two.$DOMAIN" \
  -e "[email protected]$DOMAIN" \
  --expose 80 --name test-two \
  drupal

Now you have two running sites, but they’re not yet accessible to the outside world.

Step three: a reverse proxy and Let’s encrypt

The term “proxy” means something which represents something else. In our case we want to have a webserver container which represents our Drupal and html containers. The Drupal and html containers are effectively hidden in front of a proxy. Why “reverse”? The term “proxy” is already used and means that the web user is hidden from the server. If it is the web servers that are hidden (in this case Drupal or the html containers), we use the term “reverse proxy”.

Let’s encrypt is a free certificate authority which certifies that you are the owner of your domain.

We will use nginx-proxy as our reverse proxy. Because that does not take care of certificates, we will use LetsEncrypt companion container for nginx-proxy to set up and maintain Let’s Encrypt certificates.

Let’s start by creating an empty directory which will contain our certificates:

mkdir "$HOME"/certs

Now, following the instructions of the LetsEncrypt companion project, we can set up our reverse proxy:

docker run -d -p 80:80 -p 443:443 \
  --name nginx-proxy \
  -v "$HOME"/certs:/etc/nginx/certs:ro \
  -v /etc/nginx/vhost.d \
  -v /usr/share/nginx/html \
  -v /var/run/docker.sock:/tmp/docker.sock:ro \
  --label com.github.jrcs.letsencrypt_nginx_proxy_companion.nginx_proxy \
  --restart=always \
  jwilder/nginx-proxy

And, finally, start the LetEncrypt companion:

docker run -d \
  --name nginx-letsencrypt \
  -v "$HOME"/certs:/etc/nginx/certs:rw \
  -v /var/run/docker.sock:/var/run/docker.sock:ro \
  --volumes-from nginx-proxy \
  --restart=always \
  jrcs/letsencrypt-nginx-proxy-companion

Wait a few minutes for "$HOME"/certs to be populated with your certificate files, and you should now be able to access your sites:

A note about renewals

Let’s Encrypt certificates last 3 months, so we generally want to renew every two months. LetsEncrypt companion container for nginx-proxy states that it automatically renews certificates which are set to expire in less than a month, and it checks this hourly, although there are some renewal-related issues in the issue queue.

It seems to also be possible to force renewals by running:

docker exec nginx-letsencrypt /app/force_renew

So it might be worth considering to be on the lookout for failed renewals and force them if necessary.

Edit: domain-specific configurations

I used this technique to create a Docker registry, and make it accessible securely:

docker run \
  --entrypoint htpasswd \
  registry:2 -Bbn username password > auth/htpasswd

docker run -d --expose 5000 \
  -e "VIRTUAL_HOST=mydomain.example.com" \
  -e "LETSENCRYPT_HOST=mydomain.example.com" \
  -e "[email protected]" \
  -e "REGISTRY_AUTH=htpasswd" \
  -e "REGISTRY_AUTH_HTPASSWD_REALM=Registry Realm" \
  -e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd \ 
  --restart=always -v "$PWD"/auth:/auth \
  --name registry registry:2

But when trying to push an image, I was getting “413 Request Entity Too Large”. This is an error with the nginx-proxy, not the Docker registry. To fix this, you can set domain-specific configurations, in this example we are allowing a maximum of 600M to be passed but only to the Docker registry at mydomain.example.com:

docker exec nginx-proxy /bin/bash -c 'cp /etc/nginx/vhost.d/default /etc/nginx/vhost.d/mydomain.example.com'
docker exec nginx-proxy /bin/bash -c 'echo "client_max_body_size 600M;" >> /etc/nginx/vhost.d/mydomain.example.com'
docker restart nginx-proxy

Enjoy!

You can now bask in the knowledge that your cooking blog will not be man-in-the-middled.

Please enable JavaScript to view the comments powered by Disqus.

Feb 28 2017
Feb 28

February 28, 2017

As the maintainer of Realistic Dummy Content, having procrastinated long and hard before releasing a Drupal 8 version, I decided to leave my (admittedly inelegant) logic intact and abstract away the Drupal 7 code, with the goal of plugging in Drupal 7 or 8 code at runtime.

Example original Drupal 7 code

// Some logic.
$updated_file = file_save($drupal_file);
// More logic.

Example updated code

Here is a simplified example of how the updated code might look:

// Some logic.
$updated_file = Framework::instance()->fileSave($drupal_file);
// More logic.

abstract class Framework {

  static function instance() {
    if (!$this->instance) {
      if (defined('VERSION')) {
        $this->instance = new Drupal7();
      }
      else {
        $this->instance = new Drupal8();
      }
    }
    return $this->instance;
  }

  abstract function fileSave($drupal_file);

}

class Drupal8 extends Framework {
  public function fileSave($drupal_file) {
    $drupal_file->save();
    return $drupal_file;
  }
}

class Drupal7 extends Framework {
  public function fileSave($drupal_file) {
    return file_save($drupal_file);
  }
}

Once I have defined fileSave(), I can simply replace every instance of file_save() in my legacy code with Framework::instance()->fileSave().

In theory, I can then identify all Drupal 7 code my module and abstract it away.

Automated testing

As long as I surgically replace Drupal 7 code such as file_save() with “universal” code such Framework::instance()->fileSave(), without doing anything else, without giving in the impulse of “improving” the code, I can theoretically only test Framework::instance()->fileSave() itself on Drupal 7 and Drupal 8, and as long as both versions are the same, my underlying code should work. My approach to automated tests is: if it works and you’re not changing it, there is no need to test it.

Still, I want to make sure my framework-specific code works as expected. To set up my testing environment, I have used Docker-compose to set up three containers: Drupal 7, Drupal 8; and MySQL. I then have a script which builds the sites, installs my module on each, then run a selftest() function which can test the abstracted function such as fileSave() and make sure they work.

This can then be run on a continuous integration platform such as Circle CI which generates a cool badge:

CircleCI

Extending to Backdrop

Once your module is structured in this way, it is relatively easy to add new related frameworks, and I’m much more comfortable releasing a Drupal 9 update in 2021 (or whenever it’s ready).

I have included experimental Backdrop code in Realistic Dummy Content to prove the point. Backdrop is a fork of Drupal 7.

abstract class Framework {
  static function instance() {
    if (!$this->instance) {
      if (defined('BACKDROP_BOOTSTRAP_SESSION')) {
        $this->instance = new Backdrop();
      }
      elseif (defined('VERSION')) {
        $this->instance = new Drupal7();
      }
      else {
        $this->instance = new Drupal8();
      }
    }
    return $this->instance;
  }
}

// Most of Backdrop's API is identical to D7, so we can only override
// what differs, such as fileSave().
class Backdrop extends Drupal7 {
  public function fileSave($drupal_file) {
    file_save($drupal_file);
    // Unlike Drupal 7, Backdrop returns a result code, not the file itself,
    // in file_save(). We are expecting the file object.
    return $drupal_file;
  }
}

Disadvantages of this approach

Having just released Realisic Dummy Content 7.x-2.0-beta1 and 8.x-2.0-beta1 (which are identical), I can safely say that this approach was a lot more time-consuming than I initially thought.

Drupal 7 class autoloading is incompatible with Drupal 8 autoloading. In Drupal 7, classes cannot (to my knowledge) use namespaces, and must be added to the .info file, like this:

files[] = includes/MyClass.php

Once that is done, you can define MyClass in includes/MyClass.php, then use MyClass anywhere you want in your code.

Drupal 8 uses PSR-4 autoloading with namespaces, so I decided to create my own autoloader to use the same system in Drupal 7, something like:

spl_autoload_register(function ($class_name) {
  if (defined('VERSION')) {
    // We are in Drupal 7.
    $parts = explode('\\', $class_name);
    // Remove "Drupal" from the beginning of the class name.
    array_shift($parts);
    $module = array_shift($parts);
    $path = 'src/' . implode('/', $parts);
    if ($module == 'MY_MODULE_NAME') {
      module_load_include('php', $module, $path);
    }
  }
});

Hooks have different signatures in Drupal 7 and 8; in my case I was lucky and the only hook I need for Drupal 7 and 8 is hook_entity_presave() which has a similar signature and can be abstracted.

Deeply-nested associative arrays are a staple of Drupal 7, so a lot of legacy code expects this type of data. Shoehorning Drupal 8 to output something like Drupal 7’s field_info_fields(), for example, was a painful experience:

public function fieldInfoFields() {
  $return = array();
  $field_map = \Drupal::entityManager()->getFieldMap();
  foreach ($field_map as $entity_type => $fields) {
    foreach ($fields as $field => $field_info) {
      $return[$field]['entity_types'][$entity_type] = $entity_type;
      $return[$field]['field_name'] = $field;
      $return[$field]['type'] = $field_info['type'];
      $return[$field]['bundles'][$entity_type] = $field_info['bundles'];
    }
  }
  return $return;
}

Finally, making Drupal 8 work like Drupal 7 makes it hard to use Drupal 8’s advanced features such as Plugins. However, once your module is “universal”, adding Drupal 8-specific functionality might be an option.

Using this approach for website upgrades

This approach might remove a lot of the risk associated with complex site upgrades. Let’s say I have a Drupal 7 site with a few custom modules: each module can be made “universal” in this way. If automated tests are added for all subsequent development, migrating the functionality to Drupal 8 might be less painful.

A fun proof of concept, or real value?

I’ve been toying with this approach for some time, and had a good time (yes, that’s my definition of a good time!) implementing it, but it’s not for everyone or every project. If your usecase includes preserving legacy functionality without leveraging Drupal 8’s modern features, while reducing risk, it can have value though. The jury is still out on whether maintaining a single universal branch will really be more efficient than maintaining two separate branches for Realistic Dummy Content, and whether the approach can reduce risk during site upgrades of legacy custom code, which I plan to try on my next upgrade project.

Please enable JavaScript to view the comments powered by Disqus.

Oct 02 2016
Oct 02

October 02, 2016

Unless you work exclusively with Drupal developers, you might be hearing some criticism of the Drupal community, among them:

  • We are almost cult-like in our devotion to Drupal;
  • maintenance and hosting are expensive;
  • Drupal is really complicated;
  • we tend to be biased toward Drupal as a solution to any problem (the law of the instrument).

It is true that Drupal is a great solution in many cases; and I love Drupal and the Drupal community.

But we can only grow by getting off the Drupal island, and being open to objectively assess whether or not Drupal is right solution for a given use case and a given client.

“if you love something, set it free” —Unknown origin.

Case study: the Dcycle blog

I have built my entire career on Drupal, and I have been accused (with reason) several times of being biased toward Drupal; in 2016 I am making a conscious effort to be open to other technologies and assess my commitment to Drupal more objectively.

The result has been that I now tend to use Drupal for what it’s good at, data-heavy web applications with user-supplied content. However, I have integrated other technologies to my toolbox: among them node.js for real-time websocket communication, and Jekyll for sites that don’t need to be dynamic on the server-side. In fact, these technologies can be used alongside Drupal to create a great ecosystem.

My blog has looked like this for quite some time:

Very ugly design.

It seemed to be time to refresh it. My goals were:

  • Keeping the same paths and path aliases to all posts, for example blog/96/catching-watchdog-errors-your-simpletests and blog/96 and node/96 should all redirect to the same page;
  • Keep comment functionality;
  • Apply an open-source theme with minimal changes;
  • It should be easy for myself to add articles using the markdown syntax;
  • There should be a contact form.

My knee-jerk reaction would have been to build a Drupal 8 site, but looking at my requirements objectively, I realized that:

  • Comments can easily be exported to Disqus using the Disqus Migrate module;
  • For my contact form I can use formspree.io;
  • Other than the above, there is no user-generated content;
  • Upgrading my blog between major versions every few years is a problem with Drupal;
  • Security updates and hosting require a lot of resources;
  • Backups of the database and files need to be tested every so often, which also requires resources.

I eventually settled on moving this blog away from Drupal toward Jekyll, a static website generator which has the following advantages over Drupal for my use case:

  • What is actually publicly available is static HTML, ergo no security updates;
  • Because of its simplicity, testing backups is super easy;
  • My site can be hosted on GitHub using GitHub pages for free (although HTTPS is not supported yet for custom domain names Github pages now supports secure HTTPS via Let’s encrypt);
  • All content and structure is stored in my git repo, so adding a blog post is as simple as adding a file to my git repo;
  • No PHP, no MySQL, just plain HTML and CSS: my blog now feels lightning fast;
  • Existing free and open-source templates are more plentiful for Jekyll than for Drupal, and if I can’t find what I want, it is easier to convert an HTML template to Jekyll than it is to convert it to Drupal (for me anyway).
  • Jekyll offers plugins for all of my project’s needs, including the Jekyll Redirect Form gem to define several paths for a single piece of content, including a canonical URL (permalink).

In a nutshell, Jekyll works by regenerating an entirely new static website every time a change is made to underlying structured data, and putting the result in a subdirectory called _site. All content and layout is structured in the directory hierarchy, and no database is used.

Exporting content from Drupal to Jekyll

Depending on the complexity of your content, this will likely be the longest part of your migration, and will necessitate some trial and error. For the technical details of my own migration, see my blog post Migrating content from Drupal to Jekyll.

What I learned

I set out with the goal of performing the entire migration in less than a few days, and I managed to do so, all the while learning more about Jekyll. I decided to spend as little time as possible on the design, instead reusing brianmaierjr’s open-source Long Haul Jekyll theme. I estimate that I have managed to perform the migration to Jekyll in about 1/5th the time it would have taken me to migrate to Drupal 8, and I’m saving on hosting and maintenance as well. Some of my clients are interested in this approach as well, and are willing to trade an administrative backend for a large reduction in risk and cost.

So how do users enter content?

Being the only person who updates this blog, I am confortable adding my content (text and images) as files in Github, but most non-technical users will prefer a backend. A few notes on this:

  • First, I have noticed that even though it is possible for clients to modify their Drupal site, many actually do not;
  • Many editors consider the Drupal backend to be very user-unfriendly to begin with, and may be willing instead of it to accept the technical Github interface and a little training if it saves them development time.
  • I see a big future for Jekyll frontends such as Prose.io which provide a neat editing interface (including image insertion) for editors of Jekyll sites hosted on GitHub.

Conclusion

I am not advocating replacing your Drupal sites with Jekyll, but in some cases we may benefit as a community by adding tools other than the proverbial hammer to our toolbox.

Static site generators such as Jekyll are one example of this, and with the interconnected web, making use of Drupal for what it’s good at will be, in the long term, good for Drupal, our community, our clients, and ourselves as developers

Please enable JavaScript to view the comments powered by Disqus.

Sep 19 2016
Sep 19

September 19, 2016

Docker is now available natively on Mac OS in addition to Linux. Docker is also included with CoreOS which you can run on remote Virtual Machines, or locally through Vagrant.

Once you have installed Docker and Git, locally or remotely, you don’t need to install anything else.

In these examples we will leverage the official Drupal and mySQL Docker images. We will use the mySQL image as is, and we will add Drush to our Drupal image.

Docker is efficient with caching: these scripts will be slow the first time you run them, but very fast thereafter.

Here are a few scripts I often use to set up quick Drupal 7 or 8 environments for module evaluation and development.

Keep in mind that using Docker for deployment to production is another topic entirely and is not covered here; also, these scripts are meant to be quick and dirty; docker-compose might be useful for more advanced usage.

Port mapping

In all cases, using -p 80, I map port 80 of Drupal to any port that happens to be available on my host, and in these examples I am using Docker for Mac OS, so my sites are available on localhost.

I use DRUPALPORT=$(docker ps|grep drupal7-container|sed 's/.*0.0.0.0://g'|sed 's/->.*//g') to figure out the current port of my running containers. When your containers are running, you can also just docker ps to see port mapping:

$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                   NAMES
f1bf6e7e51c9        drupal8-image       "apache2-foreground"     15 seconds ago      Up 11 seconds       0.0.0.0:32771->80/tcp   drupal8-container
...

In the above example (scroll right to see more outpu), port http://localhost:32771 will show your Drupal 8 site.

Using Docker to evaluate, patch or develop Drupal 7 modules

I can set up a quick environment to evaluate one or more Drupal 7 modules. In this example I’ll evaluate Views.

mkdir ~/drupal7-modules-to-evaluate
cd ~/drupal7-modules-to-evaluate
git clone --branch 7.x-3.x https://git.drupal.org/project/views.git
# add any other modules for evaluation here.

echo 'FROM drupal:7' > Dockerfile
echo 'RUN curl -sS https://getcomposer.org/installer | php' >> Dockerfile
echo 'RUN mv composer.phar /usr/local/bin/composer' >> Dockerfile
echo 'RUN composer global require drush/drush:8' >> Dockerfile
echo 'RUN ln -s /root/.composer/vendor/drush/drush/drush /bin/drush' >> Dockerfile
echo 'RUN apt-get update && apt-get upgrade -y' >> Dockerfile
echo 'RUN apt-get install -y mysql-client' >> Dockerfile
echo 'EXPOSE 80' >> Dockerfile

docker build -t drupal7-image .
docker run --name d7-mysql-container -e MYSQL_ROOT_PASSWORD=root -d mysql
docker run -v $(pwd):/var/www/html/sites/all/modules --name drupal7-container -p 80 --link d7-mysql-container:mysql -d drupal-image

DRUPALPORT=$(docker ps|grep drupal7-container|sed 's/.*0.0.0.0://g'|sed 's/->.*//g')

# wait for mysql to fire up. There's probably a better way of doing this...
# See stackoverflow.com/questions/21183088
# See https://github.com/docker/compose/issues/374
sleep 15

docker exec drupal7-container /bin/bash -c "echo 'create database drupal'|mysql -uroot -proot -hmysql"
docker exec drupal7-container /bin/bash -c "cd /var/www/html && drush si -y --db-url=mysql://root:[email protected]/drupal"
docker exec drupal7-container /bin/bash -c "cd /var/www/html && drush en views_ui -y"
# enable any other modules here. Dependencies will be downloaded
# automatically

echo -e "Your site is ready, you can log in with the link below"

docker exec drupal7-container /bin/bash -c "cd /var/www/html && drush uli -l http://localhost:$DRUPALPORT"

Note that we are linking (rather than adding) sites/all/modules as a volume, so any change we make to our local copy of views will quasi-immediately be reflected on the container, making this a good technique to develop modules or write patches to existing modules.

When you are finished you can destroy your containers, noting that all data will be lost:

docker kill drupal7-container d7-mysql-container
docker rm drupal7-container d7-mysql-container

Using Docker to evaluate, patch or develop Drupal 8 modules

Our script for Drupal 8 modules is slightly different:

  • ./modules is used on the container instead of ./sites/all/modules;
  • Our Dockerfile is based on drupal:8, not drupal:7;
  • Unlike with Drupal 7, your database is not required to exist prior to installing Drupal with Drush;
  • In my tests I need to chown /var/www/html/sites/default/files to www-data:www-data to enable Drupal to write files.

Here is an example where we are evaluating the Token module for Drupal 8:

mkdir ~/drupal8-modules-to-evaluate
cd ~/drupal8-modules-to-evaluate
git clone --branch 8.x-1.x https://git.drupal.org/project/token.git
# add any other modules for evaluation here.

echo 'FROM drupal:8' > Dockerfile
echo 'RUN curl -sS https://getcomposer.org/installer | php' >> Dockerfile
echo 'RUN mv composer.phar /usr/local/bin/composer' >> Dockerfile
echo 'RUN composer global require drush/drush:8' >> Dockerfile
echo 'RUN ln -s /root/.composer/vendor/drush/drush/drush /bin/drush' >> Dockerfile
echo 'RUN apt-get update && apt-get upgrade -y' >> Dockerfile
echo 'RUN apt-get install -y mysql-client' >> Dockerfile
echo 'EXPOSE 80' >> Dockerfile

docker build -t drupal8-image .
docker run --name d8-mysql-container -e MYSQL_ROOT_PASSWORD=root -d mysql
docker run -v $(pwd):/var/www/html/modules --name drupal8-container -p 80 --link d8-mysql-container:mysql -d drupal8-image

DRUPALPORT=$(docker ps|grep drupal8-container|sed 's/.*0.0.0.0://g'|sed 's/->.*//g')

# wait for mysql to fire up. There's probably a better way of doing this...
# See stackoverflow.com/questions/21183088
# See https://github.com/docker/compose/issues/374
sleep 15

docker exec drupal8-container /bin/bash -c "cd /var/www/html && drush si -y --db-url=mysql://root:[email protected]/drupal"
docker exec drupal8-container /bin/bash -c "chown -R www-data:www-data /var/www/html/sites/default/files"
docker exec drupal8-container /bin/bash -c "cd /var/www/html && drush en token -y"
# enable any other modules here.

echo -e "Your site is ready, you can log in with the link below"

docker exec drupal8-container /bin/bash -c "cd /var/www/html && drush uli -l http://localhost:$DRUPALPORT"

Again, when you are finished you can destroy your containers, noting that all data will be lost:

docker kill drupal8-container d8-mysql-container
docker rm drupal8-container d8-mysql-container

Please enable JavaScript to view the comments powered by Disqus.

Aug 21 2016
Aug 21
TL;DR The Google Summer of Code period ends, and am glad that I am able to meet all the goals and develop something productive for the Drupal community. In this blog post, I will be sharing the details of the project, the functionality of the module and its current status.

I am glad that I was one of the lucky students who were selected to be a part of the Google Summer of Code 2016 program for the project “Integrate Google Cloud Vision API to Drupal 8”. The project was under the mentorship of Naveen Valecha, Christian López Espínola and Eugene Ilyin. Under their mentoring and guidance, I am able meet all the goals and develop something productive for the Drupal community.

Let me first share why the Google Vision API module may be required.

Google Cloud Vision API bring to picture the automated content analysis of the images. The API can not only detect objects ranging from animals to famous monuments, but also detects faces on emotions. In addition, the API can also help censor images, extract text from images, detect logos and landmarks, and even the attributes of the image itself, for instance the dominant color in the image. Thus, it can serve as a powerful content analysis tool for images.

Now let us see how can we put the module to use, i.e. what are its use cases. To start with, the Google Vision API module allows Taxonomy tagging of image files using Label Detection. Label Detection classifies the images into a number of general purpose categories. For example, classifying a war scenario to war, troop, soldiers, transport, etc. based on the surroundings in the images. This feature of the module is especially important to filter the images based on some tags.

Second feature listing our use case is the Safe Search Detection. It quickly identifies and detects the presence of any explicit or violent contents in an image which are not fit for display. When this feature is enabled in the module, the Safe Search technique validates any image for explicit/violent contents. If found, these images are asked for moderation, and are not allowed to be uploaded on the site, thus keeping the site clean.

Please click here for video demonstration of the two above-mentioned use cases.

Continuing with the other use cases, the third one is Filling the Alternate Text field of an image file. Label, Logo, Landmark and Optical Character Detection feature of the Google Cloud Vision API have been used to implement this use case. Based on the choice entered by the end user, he/she can have the Alternate Text for any image auto filled by one of the four above-mentioned options. The choice “Label Detection” would fill the field with the first value returned in the API response. “Logo Detection” identifies the logos of famous brands, and can be used to fill the field accordingly. Likewise, “Landmark Detection” identifies the monuments and structures, ranging from natural to man-made; and “Optical Character Detection” detects and identifies the texts within an image, and fills the Alternate Text field accordingly.

Next comes the User Emotion Detection feature. This feature is especially important in cases of new account creation. On enabling this feature, it would detect the emotion of the user in the profile picture and notify the new user if he/she seems to be unhappy in the image, prompting them to upload a happy one.

Lastly, the module also allows Displaying the similar image files. Based on the dominant color component (Red, Green or Blue), the module quickly groups all the images which share the same color component, and display them under the “Similar Content” tab in the form of a list. Each item links itself to the image file itself, and is named as per the filename saved by the user. Users should note here that by “similar contents”, we do not mean that the images would resemble each other always. Instead we mean here the same dominant color components.

All the details of my work, the interesting facts and features have been shared on the Drupal Planet.

Please watch this video to know more on how to use the above-mentioned use cases in proper way.

[embedded content]


This is the complete picture of the Google Vision API module developed during the Google Summer of Code phase (May 23, 2016- August 23, 2016).

With this, the three wonderful months of Google Summer of Code phase comes to an end, enriching me with lots of experiences, meeting great persons and working with them. In addition of giving me an asset, it also boosted and enhanced my skills. I learnt a lot of new techniques, which probably, I would not have learnt otherwise. The use of services and dependency injection, constraints and validators, controllers, automated tests and the introduction to concepts of entities and entity types to name a few.
I would put to use these concepts in best possible way, and try to contribute to the Drupal community with my best efforts.
Aug 16 2016
Aug 16
TL;DR Last week I had worked moving the helper functions for filling Alt Text of image file to a new service; and moving the reused/supporting functions of the tests to an abstract parent class, GoogleVisionTestBase. This week I have worked on improving the documentation of the module and making the label detection results configurable.

With all major issues and features committed to the module, this week I worked on few minor issues, including the documentation and cleanup in the project..

It is an immense pleasure for me that I am getting the feedbacks from the community on the Google Vision API module. An issue Improve documentation for helper functions was created to develop more on documentation and provide the minute details on the code. I have worked on it, and added more documentation to the helper functions so that they can be understood better.

In addition, a need was felt to let the number of results obtained from the Vision API for each of the feature as configurable, and allow the end user to take the control on that. The corresponding issue is Make max results for Label Detection configurable. In my humble opinion, most of the feature implementations and requests to the Google Cloud Vision API have nothing to do with allowing the end user to configure the number of results. For instance, the Safe Search Detection feature detects and avoids the explicit contents to be uploaded, and does not need the number of results to be configurable. However, the taxonomy tagging using Label Detection should be user dependent, and hence, I worked on the issue to make the value configurable only for Label Detection purpose. This value can be configured from the Google Vision settings page, where we set the API key. I have also developed simple web tests to verify that the value is configurable. Presently, the issue is under review.

I have also worked on standard coding fixes and pa-reviews and assisted my mentor, Naveen Valecha to develop interfaces for the services. I assisted him on access rights of the functions, and fixing the documentation issues which clashed with the present one.

Lastly, I worked on improving the README and the module page to include all the new information and instructions implemented during the Google Summer of Code phase.

With all these works done, and all the minor issues resolved, I believe that the module is ready for usage with all the features and end user cases implemented.
Next Week, I’ll work on creating a video demonstration on how to use Google Vision API to fill the Alt Text attribute of an image file, detect the emotion in the user profile pictures and to group the similar images which share the same dominant color.
Aug 09 2016
Aug 09
TL;DR Last week I had worked on modifying the tests for “Fill Alt Text”, “Emotion Detection” and “Image Properties” features of the Google Vision API module. The only tasks left are moving the supporting functions to a separate service, in addition to, creating an abstract parent class for tests and moving the functions there.

The issues Alt Text field gets properly filled using various detection features, Emotion Detection(Face Detection) feature and Implementation of Image Properties feature of the Google Vision API module are still under review by my mentors. Meanwhile, my mentors asked me to move the supporting functions of the “Fill Alt Text” issue to a separate service and use it from there. In addition, they also suggested me to create an abstract parent class for the Google Vision simple tests, and move the supporting functions to the parent class. Thus, this week, I contributed to follow these suggestions and implement them out.

There are few supporting functions, namely, google_vision_set_alt_text() and google_vision_edit_alt_text() to fill the Alt Text in accordance to the feature requested from the Vision API, and also to manipulate the value, if needed. I moved these functions to a separate service, namely, FillAltText, and have altered the code to use the functions from there instead of directly accessing them.

In addition, there are a number of supporting functions used in the simple web tests of the module, to create users, contents and fields, which were placed in the test file itself, which in one way, is a kind of redundancy. Hence, I moved all these supporting functions to abstract parent class named GoogleVisionTestBase, and altered the test classes to extend the parent class instead and in place of WebTestBase. This removed the redundant code, as well as, gave a proper structure and orientation to the web tests.
These minor changes would be committed to the module directly, once the major issues are reviewed by my mentors and committed to the module.
Aug 03 2016
Aug 03
TL;DR Last week, I had worked on and developed tests to ensure that the Alt Text field of an image file gets filled in accordance to the various detection features of the Vision API, namely Label Detection, Landmark Detection, Logo Detection and Optical Character Detection. This week I have worked to modify and add tests to various features of the Google Vision module, namely filling of Alt Text field, emotion detection of user pictures and grouping the image files on the basis of their dominant color component.

My mentors reviewed the code and the tests which I had put for review to get them committed to the Google Vision API module. However, the code needs some amendment pointed out by my mentors, which was to be corrected before commit. Hence, I spent this week working on the issues and resolving the flaws, rather than starting with a new feature. Let me start discussing my work in detail.

I had submitted the code and the tests which ensure that the Alt Text field gets properly filled using various detection features according to the end user choice. However, as was pointed out by my mentor, it had one drawback- the user would not be able to manipulate or change the value of the field if he wishes to. Amidst the different options available to the end user to fill the alt text field of the image file, there was a small bug- once an option is selected, it was possible to switch between the options, however, disabling it was not working. After, been pointed out, I worked on modifying the feature and introducing the end user ability to manipulate the value of the field as and when required. Also, I worked on the second bug, and resolved the issues of disabling the feature.

Regarding the Emotion Detection(Face Detection) feature of the Vision API, I was guided to use injections instead of using the static methods directly, and to modify variables. For example, the use of get(‘entity_type.manager’) over the static call \Drupal::entityTypeManager(). Apart from these minor changes, a major issue was the feature was being called whenever an image file is associated with. However, I need to direct it to focus only when the user uploads an image, and not on its removal (as both the actions involves an image file, hence the bug).

In the issue, Implementation of Image Properties feature in the Vision API, I had queried multiple times to the database in the cycle to fetch results and build the routed page using the controllers. However, my mentor instructed me that its really a bad way of implementing the database queries to fetch the results. Hence, I modified the code and changed them to single queries to fetch the result and use them to build the page. In addition, I was asked to build the list using ‘item_list’ instead of using the conventional ‘#prefix’ and ‘#suffix’ to generate the list. Another important change in my approach towards my code was the use of db_query(), the use of which is deprecated. Hence, I switched to use addExpressions() instead of db_query().
Presently, the code is under review by the mentors. I will work further on them, once they get reviewed and I get further instructions on it.
Jul 27 2016
Jul 27
TL;DR Last week, I had worked on and developed tests to ensure that the similar images are grouped in accordance to the Image Properties feature of the Vision API. The code is under review by the mentors, and I would continue on it once the review is done. Meanwhile, they also reviewed the “Fill Alt Text” feature issue, and approved it is good to go. This week, I have worked on developing tests for this issue.

An important feature that I have implemented in the Google Vision API module is the filling of Alt Text field of an image file entity by any of the four choices- Label Detection, Landmark Detection, Logo Detection and Optical Character Detection. My mentor suggested me to check the availability of the response and then fill the field, as we can not fully rely on the third party responses. With this minor suggestion being implemented, now its time to develop tests to ensure the functionality of this feature.

I started developing simple web tests for this feature, to ensure that the Alt Text field is properly filled in accordance to the choice of the user. It requires the selection of the four choices one by one and verify that the field is filled correctly. Thus we require four tests to test the entire functionality. I have added an extra test to ensure that if none of the options are selected then the field remains empty.

I created the image files using the images available in the simpletests. The images can be accessed through drupalGetTestFiles(). The filling, however, requires call to the Google Cloud Vision API, thus inducing dependency on the API key. To remove the dependency, I mocked the function in the test module, returning the custom data to implement the feature.

The first test ensures that the Label Detection feature returns correct response and the Alt Text field is filled correctly. The simpletest provides a list of assertions to verify it, however, I found assertFieldByName() to be most suitable for the purpose. It asserts the value of a field based on the field name. The second test ensures that the Landmark Detection feature works correctly. Similarly, the third and fourth test ensures the correct functionality of the Logo and the Optical Character Detection feature.

The fifth test which I have included perform tests when none of the options are selected. It ensures that under this case, the Alt Text field remains empty, and does not contain any unwanted values.
I have posted the patch covering the suggestions and tests on the issue queue Fill the Alt Text of the Image File using Google Vision API to be reviewed by my mentors. Once they review it, I would work on it further, if required.
Jul 26 2016
Jul 26
TL;DR In the past two weeks I had worked on using the Image Properties feature offered by the Google Cloud Vision API to group the image files together on the basis of the dominant color components filling them. In addition, I had also worked on detecting the image files and filling the Alternate Text field based on the results of Label/Landmark/Logo/Optical Character Detection, based on the demand of the end user. This week, I have worked on and developed tests to ensure that the similar images are grouped in accordance to the Image Properties feature of the Vision API.

At present, the Google Vision API module supports the Label Detection feature to be used as taxonomy terms, the Safe Search Detection feature to avoid displaying any explicit contents or violence and the User Emotion detection to detect the emotions of the users in their profile pictures and notify them about it.

I had worked on grouping the images on the basis of the dominant color component(Red, Green or Blue) which they are comprised of. I got the code reviewed by my mentors, and they approved it with minor suggestions on injecting the constructors wherever possible. Following their suggestions, I injected the Connection object instead of accessing the database via \Drupal::database().

After making changes as per the suggestions, I started developing simple web tests for this feature, to ensure that the similar images gets displayed under the SImilarContents tab. It requires the creation of new taxonomy vocabulary and adding an entity reference field to the image file entity. After the creation of the new Vocabulary and addition of the new field to the image file, I created the image files using the images available in the simpletests. The images can be accessed through drupalGetTestFiles(). The first test ensures that if the Vocabulary named ‘Dominant Color’ is selected, the similar images gets displayed under the file/{file_id}/similarcontent link.

The grouping, however, requires call to the Google Cloud Vision API, thus inducing dependency on the API key. To remove the dependency, I mocked the function in the test module, returning the custom data to implement the grouping.

To cover the negative aspect, i.e. the case when the Dominant Color option is not selected, I have developed another test which creates a demo vocabulary to simply store the labels, instead of the dominant color component. In this case, the file/{file_id}/similarcontent link displays the message “No items found”.
I have posted the patch covering the suggestions and tests on the issue queue to be reviewed by my mentors. Once they review it, I would work on it further, if required.
Jul 14 2016
Jul 14
TL;DR Previous week I had worked on detecting the emotion in the profile pictures of the users, and notifying them to change the image if they do not look happy. The work is under review by the mentors. Once it gets reviewed, I would resume it if it needs any changes. This week I have worked on filling the ‘Alt Text’ field of an image file based on any one of the method selected by the end user- Label Detection, Landmark Detection, Logo Detection and Optical Character Detection.

Last week, I had worked on implementing the Face Detection feature in the Google Vision API module. The code is currently under the review by the mentors. Once, they review it, I would develop further on it if it requires any changes.

The Google Cloud Vision API provides the features to detect popular landmarks in an image(Landmark Detection), logos of popular brands(Logo Detection), texts within an image(Optical Character Detection), in addition to Label Detection. These features, though of less significance, are helpful in identifying an image. Hence, I have started working on implementing a new helpful case for the users- Filling of the Alternate Text field of an image file using these features.

The Alt Text field of the image file entity is modified to incorporate the options to fill the field using the features. The user may select any one of the four options to fill the Alt Text field of the image.

Coming to the technical aspect, I have made use of hook_form_BASE_FORM_ID_alter() to alter the Alternate Text field of the image file entity. I have modified the edit form of the Alt Text field to add four radio options, namely- Label Detection, Landmark Detection, Logo Detection and Optical Character Detection. The user may select any of the options and save the configuration. The Alternate Text field would be filled up accordingly.
Presently, the code is under the review by the mentors. Once it gets reviewed, I would make suggested changes, if required.
Jul 07 2016
Jul 07
TL;DR Previous week I had worked on grouping the contents based on the dominant color component in their images, if present. The work is under review of the mentors. And once, it gets reviewed, I would work further on that issue. Meanwhile, I have started developing and implementing the Emotion Detection feature of the Google Cloud Vision API. It would detect the emotion of the person in the profile picture uploaded, and if the person looks angry or unhappy, he would be notified thereof. This feature is especially important when building sites for professional purposes, as the facial looks matters a lot in such cases.

Last week, I had worked on implementing the Dominant Color Detection feature in the Google Vision API module. The code is currently under the review by the mentors. Once, they review it, I would develop further on it if it requires any changes.

Hence, meanwhile, I have started working on implementing a new feature Face Detection in an image. This feature gives us the location of the face in an image, and in addition, the emotions and expressions on the face.

I have used this feature to detect the emotion of the person in the profile picture uploaded by him/her. If the person does not seem happy in the image, he/she is notified thereof of their expressions. This is especially useful when the end users are developing a site for professional purposes, as in professional matters, expressions matters a lot.

Coming to the technical aspect, I have made use of hook_entity_bundle_field_info_alter() to alter the image fields, and check the emotions in the uploaded images. This function has been used, as we only want to implement this feature on the image fields. If the image is not a happy one, then appropriate message is displayed using drupal_set_message(). This feature also makes use of Constraints and Validators just like the Safe Search detection feature. Presently, the code is under the review by the mentors.

In addition to the implementation of Face Detection, I also worked on expanding the tests of the Safe Search Detection feature of the Google Vision API module to test other entities as well, in addition to the nodes. I have expanded the tests to test the safe search constraint on the comment entity as well. This requires the creation of a dummy comment type, adding an image field to the comment type, and attaching the comment to the content type. The image field contains the safe search as the constraint on it. This test is basically similar to the tests present in the module for the node entity. The code is under review by the mentors and would soon be committed to the module. For reference on how to create dummy comment types and attaching it to the content types, the CommentTestBase class is very helpful.
Jun 29 2016
Jun 29
TL;DR The safe search constraint feature is now committed to the module along with proper web tests. So, this week I started off with a new feature offered by the Google Cloud Vision API- “Image Properties Detection”. It detects various properties and attributes of an image, namely, the RGB components, pixel fraction and score. I have worked on to detect the dominant component in the image present in any content, and display all the contents sharing similar dominant color. It is pretty much like what we see on the e-commerce sites.

Previous week I had worked on writing web tests for the safe search constraint validation on the image fields. This feature is now committed in the module Google Vision API.

This week I have worked on implementing another feature provided by the Google Cloud Vision API, namely, Image Properties Detection. This feature detects the color components of red, green and blue colors in the images along with their pixel fractions and scores. I have used this feature to determine the dominant color component (i.e. red, blue or green) in the image, and to display all those contents which have the same dominant color in their images.

I have developed the code which creates a new route- /node/{nid}/relatedcontent to display the related contents in the form of a list. This concept makes use of Controllers and Routing System of Drupal 8. The Controller class is extended to render the output of our page in the format we require. The contents are displayed in the form of list with the links to their respective nodes, and are named by their titles.

In addition to the grouping of similar contents, the colors are also stored in the form of taxonomy terms under a taxonomy vocabulary programmatically generated under the name Dominant Colors.

This issue is still under progress, and requires little modification. I need to add the link to the new route in each of the nodes, so as to  get a better interface to access those contents. Henceforth, I will put this patch for review.
A very simple example of creating routes and controllers in your module can be found here.
Jun 29 2016
Jun 29
TL;DR It has been over a month since I started working on my Drupal project “Integrate Google Cloud Vision API to Drupal 8”, and gradually I have crossed the second stage towards the completion of the project, first being selection in the Google Summer of Code 2016 programme. Here, I would like to share my experiences and accomplishments during this one month journey, and also I would like to summarize my further plans with the project and the features which I would be implementing in the coming two months.

Let me first describe the significance of this post and what actually does “midterm submission” means? The GSOC coding phase has been divided into two halves, viz. Midterm submission and Final submission. In the first half, the students try to accomplish around 50% of the project, and submit their work to the mentors for evaluation. Those who passed the midterm evaluations are allowed to proceed further and complete the remaining portion of their project.

Now coming back to my experiences, after successfully passing through the Community Bonding period of the GSOC 2016 programme, now it was the time for start coding our project proposal to reality. As I had shared earlier that during the Community Bonding period, I came to know that the project has already been initiated by Eugene Ilyin,(who is now a part of my GSOC team). So, we discussed upon the project and set a roadmap of the project and the goals we need to achieve in the GSOC period. I had started coding the very first day of the coding phase, moving the new as well as existing functions to services. My mentors Naveen Valecha, Christian López Espínola and Eugene Ilyin really helped me a lot and guided me whenever and wherever I needed their guidance and support. They helped me to get through new concepts and guided me to implement them in the most effective way to make the module the best that we could.

During this period, I also came to learn about a lot of new techniques and concepts which I had not implemented earlier. Right from the very first day of the coding period, I have been coming across new things everyday, and it is really interesting and fun to learn all those techniques. In this one month period, I learnt about services and containers and how to implement them. The post on Services and dependency injection in Drupal 8 and the videos of Drupalize.me were of great help to understand the concept of services and implement dependency injection. I also learnt about the use of validators and constraints and how can they be implemented both on general basis or specifically on fields. I also learnt about how to create test modules and alter various classes and their functions in our tests so as to remove the dependency on web access or on valid informations for our testing purposes. I learnt new things every day and enjoyed implementing them to code our module plan into reality. At present, the module supports the Label Detection feature of the Vision API, along with the tests to verify whether the API key has been set by the end user or not. Currently, the feature of Safe Search Detection is available as a patch which can be found here, which would soon be committed to the module.
[embedded content] I have shared all the details of my work on the Drupal Planet. Please watch this video for detailed information on how to use the Google Vision API module on your Drupal site.
Jun 23 2016
Jun 23
TL;DR In my last post Avoid Explicit Contents in the images using Google Vision module, I had discussed about the services which “Safe Search” feature of the Vision API provides us, and how we have put this into use in the Google Vision module as a constraint to all the image fields which would be validated when the option is enabled. This week I have worked on developing simple web tests for testing this feature whether it gives us the results as expected.

Last week I had worked on developing the code to use the Safe Search detection feature as a constraint to the image fields which would validate the images for the presence of explicit contents, provided that the user enables the configuration for the concerned image field.

Besides the code, testing the functionality using simple web tests are equally essential to ensure that the feature executes perfectly when necessary steps are implemented. Hence, this week I have worked on developing simple web tests, which ensures that we have a fully functional feature.

I have tested both the conditions with safe search enabled and disabled to verify the results which should be obtained. When the safe search is enabled, any image containing any sort of  explicit content, is detected, and asked for moderation. If the image is not moderated, then the image is not saved. When the same image was passed through the second test, with safe search disabled, it was stored successfully, thus providing us the expected results.

To conduct the test successfully, I had to create a demo content type in my tests using drupalCreateContentType(), which would have an image field with the ‘Enable Safe Search’ option. This was something new to me to how to add an extra field to the default content type settings. The Drupal documentation on FieldConfig and FieldStorageConfig were of great help to understand the underlying concepts and functions which the field offers, and thus helping me to create custom fields programmatically. However, in order to perform the test, I had to call the API directly, which required a valid API key and an image which actually contains explicit content. Hence, my mentors asked me to override the functions of the class (mocking the services) in such a way that it removes the dependency from both the key and the image. Thus, I created a test module inside the Google Vision module, and override the function.

Summarizing the above, I can say that in addition to learning how to test the constraints and validators, I also came to learn about some really cool techniques, including the creation of custom fields in tests and mocking the services.
The lingotek_test of the Lingotek Translation module is a good reference to learn about how to override the services in web tests. Other references which are useful for mocking are ServiceProviderBase and Mocking for Unit Tests.
Jun 14 2016
Jun 14
TL;DR Safe Search detection of the Google Cloud Vision API allows the end users to avoid any explicit content or violence in images to be uploaded on the site. I worked on integrating this feature to the module as a constraint to those image fields which have the “Safe Search” feature enabled.

Let me first give a brief summary about the current status of the module Google Vision. In the earlier weeks, I had implemented the services and wrote tests for the module, which are now committed to the module.

Now, coming to the progress of this week.

I had started with integrating the Safe Search detection feature in the module. Safe Search detection allows its users to detect any explicit contents within the image, and hence can be very useful for site administrators who do not want to display any explicit images on their sites.

This feature was initially integrated using a function call in the module. However, my mentors suggested that this feature should rather be a Constraint on the image fields, which would be validated if the feature is enabled for the field. It depends on the user whether to keep safe search on their site or not, and they can implement it any time they want by just enabling/disabling the checkbox on the image field. Hence, now it is a user choice, rather than the developer’s choice whether to implement this feature or not.

Presently, the code is under review by my mentors whether it needs changes or is ready for commit.

Constraints and Validators are wonderful features of Drupal 8. Constraints, as the name goes, are certain restrictions which we pose on the various fields. Validators are used to implement the logic to create the situation under which the constraints are to be applied. Some helpful examples of applying custom constraints and validators can also be found Sitepoint.
This week had a lot of new things stored for me. I had no idea about the constraints and validators when I was asked to implement them at the first place. I spent hours on them, learning about them and seeking guidance from my mentors on the issues I faced. I developed gradually on it, and by the end of the week, I was able to implement them for the safe search detection feature.
Jun 07 2016
Jun 07
TL;DR I have already created services for the functions which bridges the module to the API, implementing the features offered by the Google Cloud Vision API, thus completing my first step towards Integrating Google Cloud Vision API to Drupal 8. This week I worked on generating error reports if the API key is not set by the user, and developing tests to test the API key configuration and whether the key is stored successfully or not.

The first step towards the integration of Google Cloud Vision API to Drupal 8 was completed with the functions moved to services. I had posted the patch for the review by my mentors. They provided their suggestions on the patch, which I worked on, and every step resulting in a better and more beautiful code.

I would also like to share that this week, our team expanded from a team of three, to a team of four members. Yes! Eugene Ilyin, the original maintainer of the module Google Vision API has joined us to mentor me through the project.

Now coming to the progress of the project, the schedule says I need to configure the Google Cloud Vision API at taxonomy field level, so that the end users may use the taxonomy terms to get the desired response from the API. However, the module already used the configuration for Label Detection, and in a discussion with my mentors, it was figured out that the current configuration does not need any changes, as at present the behaviour is pretty clear and obvious to let the developers use it easily; rather we should work on implementing the runtime verification and storage of API key supplied by the end users. I was required to write and implement the code which would give error report if the API key was not saved prior to the use of the module, and also to write tests for verifying the configuration and ensuring the storage of the key.

I created issue for the new task, Implement a runtime requirement checking if API key is not set in the issue queues of the module, and started coding the requirement. I created patches and posted it in the issue to get it reviewed by my mentors. I brought the suggested changes in the code and finally have submitted the patch implementing the required functionalities. On the other hand, the previous issue Moving the common functions to services was also under the review process. I also worked on this issue, solving and implementing minor suggestions before it gets ready to be accepted and committed! And finally, my first patch in this project has been accepted and the changes has been reflected in the module.
At the end of these two weeks, I learnt about services and dependency injection which prove to be very useful concepts implemented in Drupal 8. I also had experiences of writing tests to check the runtime functionality of the module.
May 31 2016
May 31
TL;DR I have started working on the project Integrate Google Cloud Vision API to Drupal 8. As of the task of this week, the common functions have been moved to services. The crucial concepts involved Dependency Injection and use of Guzzle over curl.

My mentors suggested me to create issues for my tasks in the issue queues of the module Google Vision API, where they would track my progress and also discuss and leave suggestions for the improvement of the module. Thus, starting with my task for the first week, I created the issue Moving the common functions to services in the issue queue of the module and started coding the functions into services and injecting them as and when needed. I started the week by learning the concepts of services and containers, and gradually learnt about Dependency Injections in Drupal 8. The post on Services and dependency injection in Drupal 8 and the videos of Drupalize.me were of great help to understand the concept of services and implement dependency injection.

After completing this part, I put the patch for review, and there followed the next part- Use of Guzzle over curl in the services and injecting the httpClient service. I spent significant time learning the concept of Guzzle which was quite new for me. My mentors Naveen Valecha and Christian López Espínola helped me a lot to understand Guzzle, and subsequently this task was completed with Guzzle replacing curl and injection of httpClient. In addition, the present code made use of concatenated strings for sending the data during API call. I changed the code to make use of arrays and Json utility class and its static functions to send the data instead of strings. When the code seemed perfect, my mentors suggested me to perform clean up along with proper documentation.
At the end of the week, I successfully uploaded the patch with all the suggestions implemented, clean up done and documentation added, thereby completing the task for my first week.
May 23 2016
May 23
TL;DR I’ll be working on Integrating Google Cloud Vision API to Drupal 8 this summer as part of Google Summer of Code programme. During my community bonding period I got the chance to work with lots of grey shavers in drupal community.

I have been selected as one of the eleven students for the Google Summer of Code, who would be coding their summer away for Drupal. This summer I will be working on project Integrate Google Cloud Vision API to Drupal 8. The API detects objects ranging from animals to famous monuments and landmarks. Not only that, it supports the recognition of a wide range of famous logos/brands. Face detection, text detection within an image and safe searches are among the other features which this API provides. It is really cool for a Content Management System to have such a powerful feature, so I would be working this summer to implement this feature in Drupal.

Before jumping directly into the coding phase, the students selected for the Google Summer of Code, were given a month to bond with the community, their mentors and other friends with whom they share the same platform. This phase is known as “Community Bonding Period”, which lasted from April 23, 2016 to May 22, 2016.

Everyday brings up new opportunities to do something new, meet like-minded people across the globe, and contribute to awesomeness!! This is the simplest way in which I can put the "Community Bonding" period.

This is the ideal time when the students get closer to the community, interact with their mentors, learn more about the organization codebase and have a more detailed discussion on the project and its timelines before actually start the coding.

To mentor me through the project, I have been blessed with two experienced mentors- Naveen Valecha and Christian López Espínola. I have been discussing with them on the project, its execution and the tough areas for last few weeks. I was also guided to learn some basic methodologies, including the services and configuration management which would be of great help in my project. Meanwhile, I also devoted my time to the community by contributing in the core issues, providing suggestions, developing patches and reviewing the project applications, including Field Group Table Component and AddressField Copy of the newbies in the Drupal world, highlighting how can they develop it in a better way, and thus lightening the burden of the community as far as I could.

In this one month, I met a lot of like-minded across the globe who have gathered together to take the community to greater heights, and I felt myself lucky enough to get an opportunity to work with them. I tried developing good bonding with my mentors, being in touch with them and bringing to their notice all my actions and tracks. 
This I suppose is the most essential purpose of this one month pre-coding phase.

Here, I would also like to share a strange experience of mine during the community bonding period. During this phase, I came to know that the project on which I am working on, was already initiated by Eugene Ilyin. Here lies the bond strength of the Drupal Community. On having the discussion on this with Eugene, he not only agreed to let me work on the project, but has also agreed to mentor me on my project. It once again proved the awesomeness of the community, and how all of us come together, collaborate and help each other to grow as well as take the community to greater heights. This was all about my month of Community Bonding.

The month ends, and now we jump into coding the summer away.
Apr 30 2016
Apr 30
TL;DR My first contribution to Drupal was a small contributed module named Image CircleSlider. It had gone through different stages of examination by different reviewers, and during this process, I came to know about the work flow of a sandbox project to a full project.

Since the release of my first module Image Circle Slider, I have been asked this question at several instances that what did I do to get my module to be accepted.

So here I am discussing my experience of the first module I contributed. I was an intern at Innoraft Solutions Pvt. Ltd., where I came across this jQuery plugin, and was asked to implement it in Drupal. Though it was a very simple plugin, yet was an innovative one, and brought the images in a very modern way, instead of traditional sliding system. After developing the code of the module, I created it as sandbox project, and queued it to get it reviewed by the community.
P.S. A useful tip in order to go to the top of the priority list- Take part in the review process yourself, and help others to get their review done. It earns a special tag to your project: "PAReview Bonus".  Drupal is a friendly community, and we here believe in joining hands and working, instead of the stiff competition which exists in today's world, and hence this review process.
Once the reviewers start reviewing your project, they would be providing their valuable suggestions, and each step would take your project to its completion and acceptance. And when sufficient reviews have been done, and there would be no application blockers, it would be marked as "Reviewed and Tested by the Community", which the pre-final step and ultimately the git administrator would give you the access to release your project as Full Project.
This is a one-time process, and once you have been given git access, you can directly release your projects, and give your service to the community.
Another important point: I have seen many a times that the contributors gets restless during the process, and want to get it done within a single day. However, things does not go this way. Please be patient during the review process. These things does not happen overnight, and you need to be patient and coordinate with the community.

Hope this helps to all the new comers. Happy contributing!!

Mar 15 2016
Mar 15
TL;DR I'll be working on the project Integrate Google Cloud Vision API to Drupal 8 this summer as part of the Google Summer of Code programme. This API offers automated content analysis of the images, including landmark detection, logo detection, face detection, text detection in images among others.

The Google Summer of Code results are out!! And I have been one of those few students who have successfully been able to get their names on the list of selected students, who would be working for the next three months on the Open Source Projects.

Now, let me describe here the project and its applications.

Well, Google Cloud Vision API bring to picture the automated content analysis of the images. The API can not only detect objects ranging from animals to famous monuments, but also detects faces on emotions. In addition, the API can also help censor images, extract text from images, detect logos and landmarks, and even the attributes of the image itself, for instance the dominant color in the image.

What is so special about this project idea? This is the first question which comes in every mind regarding any GSOC project. So here is the answer. This feature has not been implemented in the Content Management Systems(CMS), and its integration in Drupal can give the users, a powerful tool which carries out automated content analysis of the images. Integration of this API would not only add to Drupal characteristics, but also would open up a new horizon for Drupal. Won't it be great if a CMS can detect logo of famous brands? Or some explicit contents not fit to be displayed? Or the detection of natural as well as artificial structures in an image?
So, to summarize the aspects of Vision API, it offers the following features:
  1. Label Detection- It detects broad set of categories in an image, ranging from living animals to non-living places.
  2. Explicit Content Detection- Detects and notifies about adult or violent content within an image.
  3. Logo Detection- Detects the logos of popular brands.
  4. Landmark Detection- Detects natural as well as artificial structures within an image.
  5. Image Attributes- Detects general attributes of an image, such as dominant color and background shades.
  6. Optical Character Recognition(OCR)- Detects and extracts texts within an image. Vision API supports a broad range of languages along with the support for automatic language identification.
  7. Face Detection- Detects multiple faces within an image, along with associated key facial attributes like emotional state. However, it still does not embed the support of facial recognition.
  8. Integrated REST API- Access via REST API to request one or more annotation types per image.

For more details on Google Cloud Vision API, please have a look at Vision API 
Jan 15 2016
Jan 15

This is the first time that I am working with automated tests. I have written tests before and and of course I believe that tests improve projects dramatically, but I have never had (/made) the time in a project to actually do it. So I was quite excited to get started with travis.

First things first, what do you test!?

I was terribly excited to start working with tests, but yesterday we didn't actually have any code to test against yet, so instead we decided, we actually want to run the drupal core tests. We want our module to be enabled, and to test know if any of our code causes a Drupal core regression, so I started with that.

Trial and Error (a lot)...

Travis build history logs showing eventually passing tests ​Travis build history logs showing eventually passing tests.

The documentation is pretty good and there are LOADS of example .travis.yml files floating around, so I started to just get an active environment build working and focused on getting php unit tests working first. I only wanted a couple of tests to run and the main thing I cared about was that drupal installed properly and that the environment was up and running. Eventually I got a passing build with the following .travis.yml

While I was really quite happy with this, I wanted more. Simpletests to say the least, and maybe even behat tests. I found a lot of inspiration when looking at BartFeenstra's (Xano on d.o) Currency module: https://github.com/bartfeenstra/drupal-currency/blob/8.x-3.x/.travis.yml
Most importantly, I found Drupal TI - https://github.com/LionsAd/drupal_ti

Drupal TI

This project makes it almost trivial to integrate Drupal Projects with Travis. It handles a lot of the setup in terms of MYSQL, Drupal download and how to start running tests. It managed to reduce my .travis.yml to the following:

This is quite a bit of code, but it is really clean and well commented so we can clearly see what is going on.

When to test?

So now there are tests... Great! But when do we run them? Running all of Drupal's tests can take quite some time, but we do want to have a check on development to catch things. So in short we need testing on pull requests between individual development repos (eg https://github.com/yanniboi/decoupled_auth) and the main project repo (https://github.com/FreelyGive/decoupled_auth) but when we are doing our development we only really care about project specific tests.

Seeing tests on GitHub

When I create a pull request now, I automatically have the tests run and the results on the pull request:

GitHub pull request GitHub pull request with Travis test results

Travis summary GitHub pull request test results in Travis CI

Eventually what we really want to do, is have a way of checking inside our .travis.yml, if the test has been started by a pull request, or by a code push/merge and run different parameters depending on that. But more about that next time. There will be a blog post on that soon...

In the meantime, use Travis and write tests :)

References:

Jan 13 2016
Jan 13

Developer experience is our primary concern with this Drupal 8 version of doing CRM.

We thought we could improve the experience of helping developers contribute to the project. We noticed that for Drupal 8 all the cool kids were moving to hosting their development to github, such as with Drupal Commerce, but even core bits of Drupal.

So we did some investigating and decided to join them. We thought it would be helpful to share a couple of our thoughts and reasons, we are by no means authorities on this!

Getting Started

Being able to work with Github is really nice. Someone can come along to github and easily fork the main repository which is possible on Drupal.org but much easier on github.

No Special Users

We have a principle where “no individuals is special”. On drupal.org the module maintainers get access to more tools then everyone else. On github everyone is basically the same. In theory someone’s fork may become a bigger deal than the original. This means everyone has the same tools and so the things we do to make lives for our developers easier, everyone else gets to share.

We found that when some developers were maintainers and had access to drupal.org’s git they had a much nicer experience than the people who had to just download the source code or set up their own git experiences.

Pull Requests

Pull Requests are really nice. We think pull requests are pretty much a nicer way of doing patches as you can just click a few buttons and copy and paste it into the issue queue. With Dreditor it is not a big deal but github keeps track of minor changes to a patch much more effectively especially if multiple people are working on it.

  • Although it does require giving others access to my fork of a project and so we have found that sometimes patches are easier
  • Although if multiple people are working on a pull request, they can do it by forking the pull request owner’s repository and do a pull request with that first!

Drupal.org

We definitely still use Drupal.org as the issue queue and turn off all of github’s issue tracking features. We then reference issue numbers in as many commits as possible and certainly all pull requests (We post pull requests in their issue).

One of the committers can, every so often push the “main repository” or any repository to the git repo on drupal.org

TravisCI

We also use travis-ci to handle tests and will follow up with a more detailed post about how we handle testing.

Jan 13 2016
Jan 13

Short history of Tests in Drupal

In Drupal 7 it was fairly straight forward what kind of a test you should write, especially for core. It would basically be a simpletest. If you wrote in contrib, you might write a few Behat or PHPUnit tests for your module, but you would either run these tests locally of on some remote testing server such as Jenkins or Travis.

While this was quite good for doing webtests along the lines of ‘create user’, ‘log in’, ‘go to page x’, ‘click on button y’, it was also pretty slow. Every patch on the issue queue could be delayed as much as 3 hours while all the tests ran and during sprint weeks we end up with huge queues of patches waiting to be tested.

Drupal 8 has really stepped up its game. In 2013 it separated our testing into two categories. one for web tests that require a complete or partial Drupal environment - Simpletests, and another for everything else - PHPUnit :) PHPUnit Change Notice

It didn’t take long for there to be several different subclasses for PHPUnit tests in Drupal:
Unit Tests
Kernel Tests
Functional Tests

What now?

So now the question is, “Which test should I be writing”. Here is a quick explanation of the different Test Base classes.

Drupal Test Structure

- TestCase (PHPUnit_Framework_TestCase)
  - \Drupal\KernelTests\KernelTestBase
  - \Drupal\simpletest\BrowserTestBase
  - \Drupal\Tests\UnitTestCase
- \Drupal\simpletest\TestBase
  - \Drupal\simpletest\KernelTestBase
  - \Drupal\simpletest\WebTestBase
    - \Drupal\simpletest\InstallerTestBase

TestCase and TestBase should never be extended directly. They are just structure that the remaining tests extend.

UnitTestCase is what you want if you don’t care about bootstrapping Drupal. All you want to do is test Class methods and make sure that they behave expectedly. Eg. If your method required the first argument to be a string and you pass it an array you get an exception, etc.

Next up we have KernelTestBase, WebTestBase and BrowserTestBase. These all do something towards setting up a Drupal Environment and allowing you to test your code in the wild. KernelTestBase is the simplest as it bootstraps Drupal and sets up a Database, etc. but it doesn’t explicitly install any modules, and so is similar to the environment in the early installer. Any modules that you require are loaded in setUp() and you can perform schema installation as required.

Notice: there are two KernelTestBase classes. \Drupal\simpletest\KernelTestBase has been deprecated for \Drupal\KernelTests\KernelTestBase as part of Drupals continuing battle to modernise its testing and has moved from simpletest to PHPUnit. KernelTestBase Change Notice

Then we get to WebTestBase and BrowserTestBase. They do basically the same thing, as they allow a full installation of Drupal and modules and are webtests so you test the UI through a browser. The general rule of thumb is use WebTestBase. BrowserTestBase is the newest move to modernize testing as it is an attempt to move Browser Testing from simpletest to PHPUnit and Mink BrowserTestBase Change Notice. So if you fancy it, you can give the new framework a go, but it is still work in progress and not going to be used extensively in core until 8.1.x.

And finally there is InstallerTestBase which is useful testing the installation of Drupal based on changes in configuration.

Dec 09 2015
Dec 09
DrupalCamp is the distribution, for spinning up camp sites.  Lately, as always :), I was part of the site building team at a local camp in India and after working with the team, found that all the sites were/are having similar content architecture e.g. content types, listing pages etc., except their themes and designs.
While sprinting at DrupalCamp Pune, we decided to build a site for DrupalCamp Delhi. While sprinting we realised we need  a common code/feature base for camp sites. And there we started building distribution in D8. The time we started making this distribution profile Drupal 8 was in beta 16 and we are hoping to make first release candidate soon.
Drupalcamp comes up with the 2 contributed modules fb_likebox, twitter_block for the facebook and twitter share blocks in sidebar.
Configurations that this installation profile provides are:
  1. Content types
    1. Basic Page :  For creating basic pages like about us,etc. on the site.
    2. Session : This is used for creating/submitting sessions.
    3. Sponsor : This is used to keep information about the sponsors.
  2. Listings (Views):
    1. Accepted Sessions
    2. Proposed Sessions
    3. Sponsors
    4. Students.
  3. Social sharing buttons:
    1. facebook
    2. twitter
  4. Blocks:
    1. facebook
    2. twitter

We need to just use the drupalcamp profile and theme the site.
Challenges that came up while building this profile :
  1. Contributed modules : The contributed modules stable release was not out so ported the fb_likebox, twitter_block module to drupal 8 and created the stable release of fb_likebox. Special thanks to the maintainer baekelandt for quick promptness.
  2. There was no tool available which will provide the boilerplate code, so we started with forking the standard profile and used the config-export to export the configurations. Now with the drupalconsole 0.9.9 we have generate profile command available to  generate the boilerplate code. Used the phing to automate the testing as sam specified in its blog.

Thanks to the entire Drupalcamp Distribution Team!
What's Next ?
Please join hands to release the stable version for twitter_block and include this stable release in the drupalcamp distribution.

Jul 06 2015
Jul 06

July 06, 2015

If you are using a site deployment module, and running simpletests against it in your continuous integration server using drush test-run, you might come across Simpletest output like this in your Jenkins console output:

Starting test MyModuleTestCase.                                         [ok]
...
WD rules: Unable to get variable some_variable, it is not           [error]
defined.
...
MyModuleTestCase 9 passes, 0 fails, 0 exceptions, and 7 debug messages  [ok]
No leftover tables to remove.                                           [status]
No temporary directories to remove.                                     [status]
Removed 1 test result.                                                  [status]
 Group  Class  Name

In the above example, the Rules module is complaining that it is misconfigured. You will probably be able to confirm this by installing a local version of your site along with rules_ui and visiting the rules admin page.

Here, it is rules which is logging a watchdog error, but it could by any module.

However, this will not necessarily cause your test to fail (see 0 fails), and more importantly, your continuous integration script will not fail either.

At first you might find it strange that your console output shows [error], but that your script is still passing. You script probably looks something like this:

set -e
drush test-run MyModuleTestCase

So: drush test-run outputs an [error] message, but is still exiting with the normal exit code of 0. How can that be?

Well, your test is doing exactly what you are asking of it: it is asserting that certain conditions are met, but you have never explicitly asked it to fail when a watchdog error is logged within the temporary testing environment. This is normal: consider a case where you want to assert that a given piece of code logs an error. In your test, you will create the necessary conditions for the error to be logged, and then you will assert that the error has in fact been logged. In this case your test will fail if the error has not been logged, but will succeed if the error has been logged. This is why the test script should not fail every time there is an error.

But in our above example, we have no way of knowing when such an error is introduced; to ensure more robust testing, let’s add a teardown function to our test which asserts that no errors were logged during any of our tests. To make sure that the tests don’t fail when errors are expected, we will allow for that as well.

Add the following code to your Simpletest (if you have several tests, consider creating a base test for all of them to avoid reusing code):

/**
 * {inheritdoc}
 */
function tearDown() {
  // See http://dcycleproject.org/blog/96/catching-watchdog-errors-your-simpletests
  $num_errors = $this->getNumWatchdogEntries(WATCHDOG_ERROR);
  $expected_errors = isset($this->expected_errors) ? $this->expected_errors : 0;
  $this->assertTrue($num_errors == $expected_errors, 'Expected ' . $expected_errors . ' watchdog errors and got ' . $num_errors . '.');

  parent::tearDown();
}

/**
 * Get the number of watchdog entries for a given severity or worse
 *
 * See http://dcycleproject.org/blog/96/catching-watchdog-errors-your-simpletests
 *
 * @param $severity = WATCHDOG_ERROR
 *   Severity codes are listed at https://api.drupal.org/api/drupal/includes%21bootstrap.inc/group/logging_severity_levels/7
 *   Lower numbers are worse severity messages, for example an emergency is 0, and an
 *   error is 3.
 *   Specify a threshold here, for example for the default WATCHDOG_ERROR, this function
 *   will return the number of watchdog entries which are 0, 1, 2, or 3.
 *
 * @return
 *   The number of watchdog errors logged during this test.
 */
function getNumWatchdogEntries($severity = WATCHDOG_ERROR) {
  $results = db_select('watchdog')
      ->fields(NULL, array('wid'))
      ->condition('severity', $severity, '<=')
      ->execute()
      ->fetchAll();
  return count($results);
}

Now, all your tests which have this code will fail if there are any watchdog errors in it. If you are actually expecting there to be errors, then at some point in your test you could use this code:

$this->expected_errors = 1; // for example

Please enable JavaScript to view the comments powered by Disqus.

Jun 10 2015
Jun 10

June 10, 2015

To me, modern code must be tracked by a continuous integration server, and must have automated tests. Anything else is legacy code, even if it was rolled out this morning.

In the last year, I have adopted a policy of never modifying any legacy code, because even a one-line change can have unanticipated effects on functionality, plus there is no guarantee that you won’t be re-fixing the same problem in 6 months.

This article will focus on a simple technique I use to bring legacy Drupal code under a test harness (hence transforming it into modern code), which is my first step before working on it.

Unit vs. functional testing

If you have already written automated tests for Drupal, you know about Simpletest and the concept of functional web-request tests with a temporary database: the vast majority of tests written for Drupal 7 code are based on the DrupalWebTestCase, which builds a Drupal site from scratch, often installing something like a site deployment module, using a temporary database, and then allows your test to make web requests to that interface. It’s all automatic and temporary environments are destroyed when tests are done.

It’s great, it really simulates how your site is used, but it has some drawbacks: first, it’s a bit of a pain to set up: your continuous integration server needs to have a LAMP stack or spin up Vagrant boxes or Docker containers, you need to set up virtual hosts for your code, and most importantly, it’s very time-consuming, because each test case in each test class creates a brand new Drupal site, installs your modules, and destroys the environment.

(I even had to write a module, Simpletest Turbo, to perform some caching, or else my tests were taking hours to run (at which point everyone starts ignoring them) – but that is just a stopgap measure.)

Unit tests, on the other hand, don’t require a database, don’t do web requests, and are lightning fast, often running in less than a second.

This article will detail how I use unit testing on legacy code.

Typical legacy code

Typically, you will be asked to make a “small change” to a function which is often 200+ lines long, and uses global variables, performs database requests, and REST calls to external services. But I’m not judging the authors of such code – more often than not, git blame tells me that I wrote it myself.

For the purposes of our example, let’s imagine that you are asked to make change to a function which returns a “score” for the current user.

function mymodule_user_score() {
  global $user;
  $user = user_load($user->uid);
  $node = node_load($user->field_score_nid['und'][0]['value']);
  return $node->field_score['und'][0]['value'];
}

This example is not too menacing, but it’s still not unit testable: the function calls the database, and uses global variables.

Now, the above function is not very elegant; our first task is to ignore our impulse to improve it. Remember: we’re not going to even touch any code that’s not under a test harness.

As mentioned above, we could write a subclass of DrupalWebTestCase which provisions a database, we could create a node, a user, populate it, and then run the function.

But we would rather write a unit test, which does not need externalities like the database or global variables.

But our function depends on externalities! How can we ignore them? We’ll use a technique called dependency injection. There are several approaches to dependency injection; and Drupal 8 code supports it very well with PHPUnit; but we’ll use a simple implementation which requires the following steps:

  • Move the code to a class method
  • Move dependencies into their own methods
  • Write a subclass replaces dependencies (not logic) with mock implementations
  • Write a test
  • Then, and only then, make the “small change” requested by the client

Let’s get started!

Move the code to a class method

For dependency to work, we need to put the above code in a class, so our code will now look like this:

class MyModuleUserScore {
  function mymodule_user_score() {
    global $user;
    $user = user_load($user->uid);
    $node = node_load($user->field_score_nid['und'][0]['value']);
    return $node->field_score['und'][0]['value'];
  }
}

function mymodule_user_score() {
  $score = new MyModuleUserScore();
  return $score->mymodule_user_score();
}

That wasn’t that hard, right? I like to keep each of my classes in its own file, but for simplicity’s sake let’s assume everything is in the same file.

Move dependencies into their own methods

There are a few dependencies in this function: global $user, user_load(), and node_load(). All of these are not available to unit tests, so we need to move them out of the function, like this:

class MyModuleUserScore {
  function mymodule_user_score() {
    $user = $this->globalUser();
    $user = $this->user_load($user->uid);
    $node = $this->node_load($user->field_score_nid['und'][0]['value']);
    return $node->field_score['und'][0]['value'];
  }

  function globalUser() {
    return global $user;
  }

  function user_load($uid) {
    return user_load($uid);
  }

  function node_load($nid) {
    return node_load($nid);
  }

}

Your dependency methods should generally only contain one line. The above code should behave in exactly the same way as the original.

Override dependencies in a subclass

Our next step will be to provide mock versions of our dependencies. The trick here is to make our mock versions return values which are expected by the main function. For example, we can surmise that our user is expected to have a field_score_nid, which is expected to contain a valid node id. We can also make similar assumptions about how our node is structured. Let’s make mock responses with these assumptions:

class MyModuleUserScoreMock extends MyModuleUserScore {
  function globalUser() {
    return (object) array(
      'uid' => 123,
    );
  }

  function user_load($uid) {
    if ($uid == 123) {
      return (object) array {
        field_score_nid => array(
          LANGUAGE_NONE => array(
            array(
              'value' => 234,
            ),
          ),
        ),
      }
    }
  }

  function node_load($nid) {
    if ($nid == 234) {
      return (object) array {
        field_score => array(
          LANGUAGE_NONE => array(
            array(
              'value' => 3000,
            ),
          ),
        ),
      }
    }
  }

}

Notice that our return values are not meant to be complete: they only contain the minimal data expected by our function: our mock user object does not even contain a uid property! But that does not matter, because our function is not expecting it.

Write a test

It is now possible to write a unit test for our logic without requiring the database. You can copy the contents of this sample unit test to your module folder as mymodule.test, add files[] = mymodule.test to your mymodule.info, enable the simpletest modules and clear your cache.

There remains the task of actually writing the test: in your testModule() function, the following lines will do:

public function testModule() {
  // load the file or files where your classes are located. This can
  // also be done in the setUp() function.
  module_load_include('module', 'mymodule');

  $score = new MyModuleUserScoreMock();
  $this->assertTrue($score->mymodule_user_score() == 3000, 'User score function returns the expected score');
}

Run your test

All that’s left now is to run your test:

php ./scripts/run-tests.sh --class mymoduleTestCase

Then add above line to your continuous integration server to make sure you’re notified when someone breaks it.

Your code is now ready to be fixed

Now, when your client asks for a small or big change, you can use test-driven development to implement it. For example, let’s say your client wants all scores to be multiplied by 10 (30000 should be the score when 3000 is the value in the node):

  • First, modify your unit test to make sure it fails: make the test expect 30000 instead of 3000
  • Next, change your code iteratively until your test passes.

What’s next

This has been a very simple introduction to dependency injection and unit testing for legacy code: if you want to do even more, you can make your Mock subclass as complex as you wish, simulating corrupt data, nodes which don’t load, and so on.

I highly recommend getting familiar with PHPUnit, which is part of Drupal 8, and which takes dependency injection to a whole new level: Juan Treminio’s “Unit Testing Tutorial Part I: Introduction to PHPUnit”, March 1, 2013 is the best introduction I’ve found.

I do not recommend doing away entirely with functional, database, and web tests, but a layered approach where most of your tests are unit tests, and you limit the use of functional tests, will allow you to keep your test runs below an acceptable duration, making them all the more useful, and increasing the overall quality of new and even legacy code.

Please enable JavaScript to view the comments powered by Disqus.

May 10 2015
May 10

After using the new gmail 'tabs' for a while, I began just ignoring anything not in the primary tab. It turns out, this included notifications from Godaddy regarding domain renewals. I neglected to renew devbee.com, and as sure as the sun rises, it got scooped up immediately by a ... person.

I've owned domains for the last twenty years, and I never thought it would happen to me :) At first I thought no big deal, I'll just use my backup devbee.net domain (which I registered a month after devbee.com and so was able to recover it before it expired). But I've been using [email protected] for my client work for about ten years now, and there's nothing that says amateur hour like bounced email.

I am in touch with the new owner of my domain, and I'll definitely be reporting back on the results of our discussion.  

Mar 07 2015
Mar 07

Previously we read how to be a webmaster at drupal.org , Now I became the git administer at drupal.org. So I think to write a blog post so that others can benefit and also get to know how to become git administer.

In simple words: "Start Contributing". Git Administer privileges are granted to the users with proven record of contributions in the Project Applications issue queue. A solid history of consistent contributions on drupal.org is a must to get consideration for an elevated role.

How to start contributing & where you can contribute :

  1. Join the code review group
  2. Read How to review full project applications
  3. Some helpful tools for reviewing :
  4. Learning Sources : 
  5. if you found any problem while contributing,just comment on the below post/ if you need immediate answer you can try and find one of the git administer on IRC - #drupal-codereview IRC channel on Freenode.

Benifits of becoming git administer :

  1. You will see a new challenging case in every new project application.
  2. Your drupal apis knowledge will become sharp.
  3. Many more....

I would encourage you to learn more about that process and join the group of reviewers.
Next article:  A guide to review project applications
Feb 23 2015
Feb 23

Security of the Drupal website is a important stuff for the site owners, site developers.This blog post has my presentation at the Drupal Camp Mumbai that  intended for Drupalers who want to avoid security loop holes while writing code or architecting solutions. We delved into common security issues that ails custom code and has both vulnerable and secure code snippets.This is mostly about my encounters and experience after doing 50+ project application reviews and also a good guideline for new contributors.

[embedded content]

Hack Proof Your Drupal Site from Naveen Valecha

Next article:  A guide to review Project Applications.


Feb 23 2015
Feb 23

February 23, 2015

Continuous integration (CI) is the practice of running a series of checks on every push of your code, to make sure it is always in a potentially deployable state; and to make sure you are alerted as soon as possible if it is not.

Continuous integration and Drupal projects

This blog post is aimed at module maintainers, and we’ll look at how to use CI for modules hosted on Drupal.org. I’ll use as an example a project I’m maintaining, Realistic Dummy Content.

The good news is that Drupal.org has a built-in CI service for hosted modules: to use it, project maintainers need to click on the “Automated Testing” tab of their projects, enable automated testing, and make sure some tests are defined.

Once you have enabled automated testing, every submitted patch will be applied to the code and tested, and the main branches will be tested continually as well.

If you’re not sure how to write tests, you can learn by example by looking at the test code of any module which has automated testing enabled.

Limitations of the Drupal.org QA system

The system described above is great, and in this blog post we’ll explore how to take it a bit further. Drupal’s CI service runs your code on a new Drupal site with PHP 5.3 enabled. We know this by looking at the log for a test on Realistic Dummy content, which contains:

[13:50:02] Database backend [mysql] loaded.
...
[simpletest.db] =>
[test.php.version] => 5.3
...

For the sake of this article, let’s say we want to use SQLite with php 5.5, and we also want to run checks from the coder project’s coder_review module. We can’t achieve this within the Drupal.org infrastructure, but it is possible using Docker, CircleCI, and GitHub. Here is how.

Step 1: get a local CoreOS+Docker environment

Let’s start by setting up a local development environment on which we can run Docker. Docker is a system which uses Linux containers to run your software and all its dependencies in an isolated environment.

If you need a primer on Docker, check out Getting Started with Docker on Servers for Hackers (March 20, 2014), and A quick intro to Docker for a Drupal project.

Docker works best on CoreOS, which you can install quite easily on any computer using Vagrant and VirtualBox, as explained at Running CoreOS on Vagrant.

Step 2: Add a Dockerfile to your project

Because, in this example, we want to run tests which require changing things on the server, we’ll use the Docker container management system to simulate a Ubuntu machine over which we have complete control.

To see how this works, download the latest dev version of realistic_dummy_content to your CoreOS VM, take a look at the included files ./Dockerfile and ./scripts/test.sh to see how they are structured, then run the test script:

./scripts/test.sh

Without any further configuration, you will see tests run on the desired environment: Ubuntu with the correct version of PHP, SQLite, and coder review. (You can also see the results on CircleCI on the project’s CI dashbaord if you unfold the “test” section – we’ll see how to set that up for your project later on).

Setting up Docker for your own project is just a question of copy-pasting a few scripts.

Step 3: Make sure there is a mirror of your project on GitHub

Having test results on your command line is nice, but there is no reason to run them yourself. For that we use continuous integration (CI) servers, which run the tests every time someone commits something to your codebase.

Some of you might be familiar with Jenkins, which I use myself and which is great, but for open source projects, there are free CI services out there: the two I know of, CircleCI and Travis CI, synchronize with GitHub, not with Drupal.org, so you need a mirror of your project on GitHub.

Note that it is possible, using the tool HubDrop, to mirror your project on GitHub, but it’s not on your account, whereas the CI tools sync only with projects on your own account. My solution has been to add a ./scripts/mirror.sh script to Realistic Dummy Content, and call it once every ten minutes via a Jenkins job on my personal Jenkins server. If you don’t have access to a Jenkins server you can also use a cron job on any server to do this.

The mirror of Realistic Dummy Content on GitHub is here.

As mentioned above, two of the CI tools out there are CircleCI and Travis CI. One of my requirements is that the CI tool integrate well with Docker, because that’s my DevOps tool of choice.

As mentioned in Faster Builds with Container-Based Infrastructure and Docker (Mathias Meyer, Travis CI blog, 17 Dec. 2014), it seems that Travis CI is moving towards Docker, but it seems that its new infrastructure is based on Docker, but does not let you run your own Docker containers.

Circle CI, on the other hand, seems to provide more flexibility with regards to Docker, as explained in the article Continuous Integration and Delivery with Docker on CircleCI’s website.

Although Travis is a great, widely-used tool (Drush uses it), we’ll use CircleCI because I found it easier to set up with Docker.

Once you open a CircleCI account and link it to your GitHub account, you will be able to turn on CI for your mirrored project, in my case Realistic Dummy Content.

Step 5: Add a circle.yml file to your project

In order for Circle CI to know what to do with your project, it needs a circle.yml file at the root of your project. If you look at the circle.yml file at the root Realistic Dummy Content, it is actually quite simple:

machine:
  services:
    - docker

test:
  override:
    - ./scripts/test.sh

That’s it! Commit your circle.yml file, and if mirroring with GitHub works correctly, Circle CI will test your build. Debug any errors you may have, and voilà!

Here is the result of a recent Realistic Dummy Content build on CircleCI: unfold the “test” section to see the complete output: PHP version, SQLite database, coder review…

Conclusion

We have seen how you can easily add Docker support to make sure the tests and checks you run on your code are in a controlled environment, with the extensions you need (one could imagine a module which requires some external system like ApacheSolr installed on the server – Docker allows this too). This is one concrete application of DevOps: reducing the risk of glitches where “tests pass on my dev machine but not on my CI server”.

Please enable JavaScript to view the comments powered by Disqus.

Feb 18 2015
Feb 18

February 18, 2015

I recently added Docker support to Realistic Dummy Content, a project I maintain on Drupal.org. It is now possible (with Docker installed, preferably on a CoreOS VM) to run ./scripts/dev.sh directly from the project directory (use the latest dev version if you try this), and have a development environment, sans MAMP.

I don’t consider myself an expert in Docker, virtualization, DevOps and config management, but here, nonetheless, is my experience. If I’m wrong about something, please leave a comment!

Intro: Docker and DevOps

The DevOps movement, popularized starting in about 2010, promises to include environment information along with application information in the same git repo for smoother development, testing, and production environments. For example, if your Drupal module requires version 5.4 of PHP, along with a given library, then that information should be somewhere in your Git repo. Building an environment for testing, development or production should then use that information and not be dependent on anything which is unversioned. Docker is a tool which is anchored in the DevOps movement.

DevOps: the Config management approach

The family of tools which has been around for awhile now includes Puppet, Chef, and Ansible. These tools are configuration management tools: they define environment information (PHP version should be 5.3, Apache mod_rewrite should be on, etc.) and make sure a given environment conforms to that information.

I have used Puppet, along with Vagrant, to deliver applications, including my Jenkins server hosted on GitHub.

Virtualization and containers

Using Puppet and Vagrant, you need to use Virtualization: create a Virtual Machine on your host machine.

Docker works with a different principle: instead of creating a VM on top of your host OS, Docker uses containers, so resources are shared. The article Getting Started with Docker (Servers for Hackers, 2014/03/20) contains some graphics which demonstrate how much more efficient containers are as opposed to virtualization.

Puppet and Vagrant are slow; Docker is fast

Puppet and Vagrant together work for packaging software and environment configuration, but it is excruciatingly slow: it can take several minutes to launch an environment. My reaction to this has been to cringe every time I have to do it.

Docker, on the other hand, uses caching agressively: if a server was already in a given state, Docker uses a cached version of it to move along faster. So, when building a container, Docker goes through a series of steps, and caches each step to make it lightning fast.

One example: launching a dev environment of the Jenkins Vagrant project on Mac OS takes over five minutes, but launching a dev environment of my Drupal project Realistic Dummy Content (which uses Docker), takes less than 15 seconds the first time it is run once the server code has been downloaded, and, because of caching, less than one (1) second subsequent times if no changes have been made. Less than one second to fire up a full-fledged development environment which is functionally independent from your host. That’s huge to me.

Configuration management is idempotent, Docker is not

Before we move on, note that Docker is not incompatible with config management tools, but Docker does not require them. Here is why I think, in many cases, config management tools are not necessary.

The config management tools such as Puppet are idempotent: you define how an environment should be, and the tools run whatever steps are necessary to make it that way. This sounds like a good idea in theory, but it looks like this in practice. I have come to the conclusion that this is not the way I think, and it forces me to relearn how to think of my environments. I suspect that many developers have a hard time wrapping their heads around idempotence.

Docker is not idempotent; it defines a series of steps to get to a given state. If you like idempotence, one of the steps can be to run a puppet manifest; but if, like me, you think idempotence is overrated, then you don’t need to use it. Here is what a Dockerfile looks like: I understood it at first glace, it doesn’t require me to learn a new way of thinking.

The CoreOS project

The CoreOS project has seen the promise of Docker and containers. It is an OS which ships with Docker, Git, and a few other tools, but is designed so that everything you do happens within containers (using the included Docker, and eventually Rocket, a tool they are building). The result is that CoreOS is tiny: it takes 10 seconds to build a CoreOS instance on DigitalOcean, for example, but almost a minute to set up a CentOS instance.

Because Docker does not work on Mac OS without going through hoops, I decided to use Vagrant to set up a CoreOS VM on my Mac, which is speedy and works great.

Docker for deploying to production

We have seen that Docker can work for quickly setting up dev and testing environments. Can it be used to deploy to production? I don’t see why not, especially if used with CoreOS. For an example see the blog post Building an Internal Cloud with Docker and CoreOS (Shopify, Oct. 15, 2014).

In conclusion, I am just beginning to play with Docker, and it just feels right to me. I remember working with Joomla in 2006, when I discovered Drupal, and it just felt right, and I have made a career of it since then. I am having the same feeling now discovering Docker and CoreOs.

I am looking forward to your comments explaining why I am wrong about not liking idempotence, how to make config management and virutalization faster, and how and why to integrate config management tools with Docker!

Please enable JavaScript to view the comments powered by Disqus.

Feb 09 2015
Feb 09

February 09, 2015

To get the most of this blog post, please read and understand Getting Started with Docker (Servers for Hackers, 2014/03/20). Also, all the steps outlined here have been done on a Vagrant CoreOS virtual machine (VM).

I recently needed a really simple non-production Drupal Docker image on which I could run tests. b7alt/drupal (which you can find by typing docker search drupal, or on GitHub) worked for my needs, except that it did not have the cUrl php library installed, so drush en simpletest -y was throwing an error.

Therefore, I decided to create a new Docker image which is based on b7alt/drupal, but with the php5-curl library installed.

I started by creating a new local directory (on my CoreOS VM), which I called docker-drupal:

mkdir docker-drupal

In that directory, I created Dockerfile which takes b7alt/drupal as its base, and runs apt-get install curl.

FROM b7alt/drupal

RUN apt-get update
RUN apt-get -y install curl

(You can find this code at my GitHub account at alberto56/docker-drupal.)

When you run this you will get:

docker build .
...
Successfully built 55a8c8999520

That hash is a Docker image ID, and your hash might be different. You can run it and see if it works as expected:

docker run -d 55a8c8999520
c9a98bdcab4e027e8571bde71ee92b4380247a44ef9314749ef5680864de2928

In the above, we are telling Docker to create a container based on the image we just created (55a8c8999520). The resulting container hash is displayed (yours might be different). We are using -d so that our containers runs in the background. You can see that the container is actually running by typing:

docker ps
CONTAINER ID        IMAGE               COMMAND...
c9a98bdcab4e        55a8c8999520        "/usr/bin/supervisor...

This tells you that there is a running container (c9a98bdcab4e) based on the image 55a8c8999520. Again, your hases will be different. Let’s log into that container now:

docker exec -it c9a98bdcab4e bash
[email protected]:/#

To make sure that cUrl is successfully installed, I will figure out where Drupal resides on this container, and then try to enable Simpletest. If that works, I will consider my image a success, and exit from my container:

[email protected]:/# find / -name 'index.php'
/srv/drupal/www/index.php
[email protected]:/# cd /srv/drupal/www
[email protected]:/srv/drupal/www# drush en simpletest -y
The following extensions will be enabled: simpletest
Do you really want to continue? (y/n): y
simpletest was enabled successfully.                   [ok]
[email protected]:/srv/drupal/www# exit
exit

Now I know that my 55a8c8999520 image is good for now and for my purposes; I can create an account on Docker.com and push it to my account for later use:

Docker build -t alberto56/docker-drupal .
docker push alberto56/docker-drupal

Anyone can now run this Docker image by simply typing:

docker run alberto56/docker-drupal

One thing I had a hard time getting my head around was having a GitHub project and Docker project, and both are different but linked. The GitHub project is the the recipe for creating an image, whereas the Docker project is the image itself.

One we start thinking of our environments like this (as entities which should be versioned and shared), the risk of differences between environments is greatly reduced. I was used to running simpletests for my projects on an environment which is managed by hand; when I got a strange permissions error on the test environment, I decided to start using Docker and version control to manage the container where tests are run.

Please enable JavaScript to view the comments powered by Disqus.

Pages

About Drupal Sun

Drupal Sun is an Evolving Web project. It allows you to:

  • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
  • Facet based on tags, author, or feed
  • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
  • View the entire article text inline, or in the context of the site where it was created

See the blog post at Evolving Web

Evolving Web