Feb 06 2019
Feb 06

Mass.gov dev team releases open source project

Go to the profile of Moshe Weitzman

The Mass.gov development team is proud to release a new open source project, Drupal Test Traits (DTT). DTT enables you to run PHPUnit tests against your Drupal web site, without wiping your database after each test class. That is, you test with your usual content-filled database, not an empty one. We hope lots of Drupal sites will use DTT and contribute back their improvements. Thanks to PreviousNext and Phase2 for being early adopters.

Mass.gov is a large, content-centric site. Most of our tests click around and assert that content is laid out properly, the corresponding icons are showing, etc. In order to best verify this, we need the Mass.gov database; testing on an empty site won’t suffice. The traditional tool for testing a site using an existing database is Behat. So we used Behat for over a year and found it getting more and more awkward. Behat is great for facilitating conversations between business managers and developers. Those are useful conversations, but many organizations are like ours — we don’t write product specs in Gherkin. In fact, we don’t do anything in Gherkin beside Behat.

Meanwhile, the test framework inside Drupal core improved a lot in the last couple of years (mea culpa). Before Drupal Test Traits, this framework was impossible to use without wiping the site’s database after each test. DTT lets you keep your database and still test using the features of Drupal’s BrowserTestBase and friends. See DrupalTrait::setUp() for details (the bootstrap is inspired by Drush, a different open source project that I maintain).

Zakim Bridge at Night, North End Boston. Photo by David Fox.

Using DTT in a Test

  • Our test cases extend ExistingSiteBase, a convenience class from DTT that imports all the test traits. We will eventually create our own base class and import the traits there.
  • Notice calls to $this->createNode(). This convenience method wraps Drupal’s method of the same name. DTT deletes each created node during tearDown().
  • Note how we call Vocabulary::load(). This is an important point — the full Drupal and Mink APIs are available during a test. The abstraction of Behat is happily removed. Writing test classes more resembles writing module code.

More Features


  • See the DTT repo for details on how to install and run tests
  • Typically, one does not run tests against a live web site. Tests can fail and leave sites in a “dirty” state so it’s helpful to occasionally refresh to a pristine database.

If you have questions or comments about DTT, please comment below or submit issues/PRs in our repository.

More from Moshe: Our modern development environment at Mass.gov

Interested in a career in civic tech? Find job openings at Digital Services.
Follow us on Twitter | Collaborate with us on GitHub | Visit our site

Nov 28 2018
Nov 28

I recently worked with the Mass.gov team to transition its development environment from Vagrant to Docker. We went with “vanilla Docker,” as opposed to one of the fine tools like DDev, Drupal VM, Docker4Drupal, etc. We are thankful to those teams for educating and showing us how to do Docker right. A big benefit of vanilla Docker is that skills learned there are generally applicable to any stack, not just LAMP+Drupal. We are super happy with how this environment turned out. We are especially proud of our MySQL Content Sync image — read on for details!

Pretty docks at Boston Harbor. Photo credit.

Docker compose

The heart of our environment is the docker-compose.yml. Here it is, then read on for a discussion about it.

Developers use .env files to customize aspects of their containers (e.g. VOLUME_FLAGS, PRIVATE_KEY, etc.). This built-in feature of Docker is very convenient. See our .env.example file:

MySQL content sync image

The most innovative part of our stack is the mysql container. The Mass.gov Drupal database is gigantic. We have tens of thousands of nodes and 500,000 revisions, each with an unholy number of paragraphs, reference fields, etc. Developers used to drush sql:sync the database from Prod as needed. The transfer and import took many minutes, and had some security risk in the event that sanitization failed on the developer’s machine. The question soon became, “how can we distribute a mysql database that’s already imported and sanitized?” It turns out that Docker is a great way to do just this.

Today, our mysql container builds on CircleCI every night. The build fetches, imports, and sanitizes our Prod database. Next, the build does:

That is, we commit and push the refreshed image to a private repository on Docker Cloud. Our mysql image is 9GB uncompressed but thanks to Docker, it compresses to 1GB. This image is really convenient to use. Developers fetch a newer image with docker-compose pull mysql. Developers can work on a PR and then when switching to a new PR, do a simple ahoy down && ahoy up. This quickly restores the local Drupal database to a pristine state.

In order for this to work, you have to store MySQL data *inside* the container, instead of using a Docker Volume. Here is the Dockerfile for the mysql image.

Drupal image

Our Drupal container is open source — you can see exactly how it’s built. We start from the official PHP image, then add PHP extensions, Apache config, etc.

An interesting innovation in this container is the use of Docker Secrets in order to safely share an SSH key from host to the container. See this answer and mass_id_rsa in the docker-compose.yml above. Also note the two files below which are mounted into the container:

Configure SSH to use the secrets file as private key Automatically run ssh-add when logging into the container


Traefik is a “cloud edge router” that integrates really well with docker-compose. Just add one or two labels to a service and its web site is served through Traefik. We use Traefik to provide nice local URLs for each of our services (www.mass.local, portainer.mass.local, mailhog.mass.local, …). Without Traefik, all these services would usually live at the same URL with differing ports.

In the future, we hope to upgrade our local sites to SSL. Traefik makes this easy as it can terminate SSL. No web server fiddling required.

Ahoy aliases

Our repository features a .ahoy.yml file that defines helpful aliases (see below). In order to use these aliases, developers download Ahoy to their host machine. This helps us match one of the main attractions of tools like DDev/Lando — their brief and useful CLI commands. Ahoy is a convenience feature and developers who prefer to use docker-compose (or their own bash aliases) are free to do so.

Bells and whistles

Our development environment comes with 3 fine extras:

  • Blackfire is ready to go — just run ahoy blackfire [URL|DrushCommand] and you’ll get back a URL for the profiling report
  • Xdebug is easily enabled by setting the XDEBUG_ENABLE environment variable in a developer’s .env file. Once that’s in place, the PHP in the container will automatically connect to the host’s PHPStorm or other Xdebug client
  • A chrome-headless container is used by our suite which incorporates Drupal Test Traits — a new open source project we published. We will blog about DTT soon

Wish list

Of course, we are never satisfied. Here are a couple issues to tackle:

Sep 29 2017
Sep 29

Drush 9 features a deep rewrite of our app, both user facing and internals. We created and open sourced AnnotatedCommand (example), OutputFormatters, and Config. We leveraged Symfony Console for our CLI fundamentals. For details on Drush9, see the video or slides from our Drupalcon Vienna presentation.

Annotated Command example

Unfortunately, old commandfiles such as example.drush.inc no longer load in Drush 9 (since beta5). We’ve made it relatively painless to port this code to Drush 9. The video below shows the drush generate command porting the migrate commands. Detailed instructions are below the video.

  1. Using Drush 9 on a working site, run drush generate drush-command-file. generate is a wrapper for the Drupal Code Generator library.
  2. You will be prompted for 2 pieces of information:
    1. Module name: example
    2. Absolute path to legacy Drush command file: /path/to/example.drush.inc
  3. Drush writes 2 files to the Example module:
    1. drush.services.yml. No edits are needed unless you want to inject Drupal dependencies into your class (e.g. yml, class.
    2. ExampleCommands.php: Each item in example_drush_command() in your old commandfile has been transformed into an Annotated method in a new ExampleCommands file. Copy the body of your command callback functions into the corresponding method in ExampleCommands. Then modernize the code as below. Compare the Drush9 commands versus the Drush8 commands for guidance.
      1. Replace drush_log() with $this->logger()->info() or $this->logger()->warning() or similar.
      2. Replace drush_get_option('foo') with $options['foo'']
      3. Replace drush_set_error() with throw new \Exception()
      4. Replace drush_print() with $this->output()->writeln()
      5. Replace drush_sitealias_get_record() as with $this->siteAliasManager()->getSelf(). From there you can call getRoot(), getUri(), legacyRecord(), and other handy methods. In order for this to work, edit your class to implement SiteAliasManagerAwareInterface (LoginCommands is an example).
      6. Optional - move user interaction to a @hook interact. Also, note the new $this-io()->confirm() and $this->io()->ask() methods.
  4. Run drush cr to add your commandfile to the Drupal container.
  5. Congrats - your commands are now runnable in Drush9!
    1. Please post a patch if Example is a Contrib module.
    2. Please leave example.drush.inc in your module so that Drush 8 users may still use it.

I’m in #drush on IRC and Slack in case anyone wants help with porting.

Note that drush generate knows how to generate controllers, plugins, config, services, libraries, etc. A blog post about generate is coming soon.

Moshe is available for hire as a solo consultant or as a duo with the inimatable Campbell Vertesi. Moshe has worked on Drupal core since 2001, and loves mentoring Drupal teams. Moshe loves to set up CI and developer workflows for PHP/JS teams.

Aug 14 2017
Aug 14

With pride and pleasure, the Devel maintainers have released 1.0 for Drupal 8. Even at the ripe age of 14, Devel is an active, popular, and nimble project. Devel has been downloaded 3.5M times, and 200,000 sites currently report Devel as an enabled project. Devel’s whole codebase was deeply improved in this release. A few highlights are below, with annotated screenshots and gifs below each section. Please upgrade and report your success or failure using the new features!


  • A new submodule in Devel that reports all sorts of useful details abut a Drupal response.
  • Dive into Cache-hit ratio, DB queries, Events, Forms, Session, Assets, and much more.
  • Add your own data collectors.
  • Learn more at our demo video.

WebprofilerWebprofiler toolbar expandedWebprofiler toolbarWebprofiler databaseWebprofiler widgets list

Devel Module

  • Integrated into the new toolbar as a top level menu (configurable).
  • New pages and Drush commands list available Services, Routes, Events, etc.
  • New State and Configuration editors
  • 3 Twig extensions for debugging
  • A new plugin system for showing dumps like dpm(). Enable and configure the new Kint submodule for pretty dumps.
  • Supports Drush 8 and Drush 9.

Devel configurationDevel drush servicesDevel drush uuidDevel dumpersDevel state editorDevel toolbarDevel toolbar items list

Devel Generate

  • We moved the generation of text/int/bool/etc. for Fields into Drupal 8 core. Now all field types, even custom and obscure ones, are supported.
  • Supports Drush 8 and Drush 9.

Moshe is available for hire as a solo consultant or as a duo with the inimatable Campbell Vertesi. Moshe has worked on Drupal core since 2001, and loves mentoring Drupal teams. Moshe loves to set up CI and developer workflows for PHP/JS teams.

Jul 26 2017
Jul 26

It is a repeating need for me to debug Drush commands that run at Acquia Cloud. When I say “debug”, I refer to step-wise debugging via Xdebug. This is the best and only way to slay hard bugs. Down with print_r() debugging!

This blog is a terse “how-to”. Refer to Acquia - Configuring Xdebug with PhpStorm for more detail.

  1. SSH with a tunnel: drush @massgov.prod -v --ssh-options="-R 9000:localhost:9000" ssh
    1. Replace @massgov.prod with your own site alias
    2. This command connects to Acquia via SSH AND opens a tunnel from Acquia’s port 9000 to your laptop’s port 9000. This allows PHP on Acquia to connect to your PHPStorm no matter what firewall separates you. If your site alias (e.g. @massgov.prod) already defines ssh-options option, then above needs to be changed as follows:
      1. If using Drush9, you may make a CLI request with --ssh-options='-R 9000:localhost:9000 ADD STUFF FROM ALIAS HERE'.
      2. If using Drush8, you must edit your alias to add the -R 9000:localhost:9000. You can’t add that from the CLI.
    3. To verify that the tunnel is working, you can run netstat -nlt | grep 9000 in same terminal window. If you see output, you are good.
  2. In your local PHPStorm project, Run => Start listening for PHP Debug Connections
  3. In your local PHPStorm project, Run => Break at the first line in PHP scripts
  4. Back in your terminal window that is ssh-ed into Acquia, run XDEBUG_CONFIG= php -dzend_extension=xdebug.so -dxdebug.remote_enable=1 ../vendor/bin/drush core-status
    1. The environment variable XDEBUG_CONFIG= has no value because none is needed. This nudges XDebug to connect back to PHPStorm.
    2. The two -d options enable and configure XDebug extension for the current request.
    3. The /vendor/bin/drush path points to the site-local Drush. Older Drupal sites that are not built with Composer can just use drush instead.
    4. core-status is just an example command. Replace with your own command and arguments/options.
  5. If all went well, your PhpStorm should be debugging at line 1 of drush. If PHPStorm has a red warning about Path Mappings, click that link to set them up.

Debugging Drush commands at Acquia

May 17 2017
May 17

I’ve long preferred doing web development on my local laptop. Unfortunately, my current consulting gig has strict requirements. Specifically, I’m again migrating content from an Oracle database into Drupal 8. This time, the Oracle database is behind an IP based VPN - I can only connect to it via a specific AWS instance that’s running our Drupal 8 site. My PHP has to run at AWS, and can’t run locally. Happily, PHPStorm thrives in this situation. PHPStorm Deployments sync code from my laptop to AWS via SSH. Further, I’ll show how to configure the Debugger to step through remote execution of PHP CLI scripts. Remember kids, the debugger is your friend.

This blog is a terse “how-to”. Refer to Sync changes and automatic upload to a deployment server in PhpStorm and Remote Debugging in PHPStorm via SSH Tunnel for more detail.

Remote Deployment

  1. Start creating a new Deployment configuration: Tools => Deployment => Configuration
  2. Enter a name and pick SFTP for the type.
  3. On the connection tab, enter the SSH details for remote server.
  4. On the Mappings tab, tell PHP storm how the remote path maps to the local path. In my case:
    Local path: /Users/moshe.weitzman/reps/b4
    Remote path: /var/www/drupal
  5. Test by browsing the remote host: Tools => Deployment => Browse Remote Hosts
  6. You may now upload from local to remote via Tools => Deployment => Upload To [name]. Explore other options in this menu that may better suit your use case.

Remote Deployment example

Remote CLI Debugging over an SSH Tunnel

On your local machine:
  1. Run => Start listening for PHP Debug Connections
  2. Run => Break at the first line in PHP scripts
  3. In a terminal on local machine, run a command like ssh -R 9000:localhost:9000 -A [email protected]. The -R arguments creates a SSH tunnel such that port 9000 on the remote server forwards all traffic to port 9000 on your local machine. This neatly bypasses any networking challenges between you and the remote machine. Customize [email protected] for your own needs.
On the remote server:
  1. Enable and configure the Xdebug PHP extension on the remote server. Use php --ini to figure out which ini file to edit. Then add:
  2. Use php -i | grep -i xdebug to verify that your edits are working (i.e. remote_enable is enabled).
  3. In a terminal on the remote server, run a PHP CLI script such as XDEBUG_CONFIG= vendor/bin/drush st. The first clause sets a required environment variable to null value. We only need this variable to be defined. It triggers XDebug to connect back to PHPStorm.
  4. If prompted about Path Mappings, choose to pull them from the Deployment specified above.


If you get stuck, I refer you again to these longer documents: Sync changes and automatic upload to a deployment server in PhpStorm and Remote Debugging in PHPStorm via SSH Tunnel.

Mar 01 2016
Mar 01

Drupal.org uses a patch based workflow in order to improve its software. Today I present a concrete instance where this significantly slowed us down. Views performance work is blocked when it didn’t have to be.

Today, Dries committed a fine patch by Sun which moved a lot of test files into a happier place. After that commit, Dries reviewed Daniel’s Views patch which changed a couple of those same test files. Daniel’s patch failed to apply so Dries changed the issue to Needs Work. This doesn’t need to happen anymore. Git knows about file moves and can overcome this problem when we use it fully. If Daniel’s change was a merge instead of a patch, git uses its ancestry to apply changes to the moved test files. Try for yourself - the commands I ran are below.

We could all increase our velocity by using a merge workflow. Think of all the patches that need reroll currently which would not need reroll. Think of all the patches considered disruptive when they don’t need to be. I know that we can’t avoid all ‘Needs reroll’ events, but we can avoid a lot.

For now, I encourage folks to use d.o. sandboxes or Github to host their forks, and to post git merge commands into the issue queue instead of uploaded patches. Discussion of these changes should remain in the main issue queue. I think we should document this workflow at Drupal.org Git tutorials.

Feedback welcome.

Start from a clean 8.0.x:

# Create  branch whose tip is the commit before Sun's
$ git checkout -b daniel ebace9e
#Apply Daniel's patch
$ wget -O tmp.patch https://www.drupal.org/files/issues/1849822-view_render-19.patch; git apply tmp.patch;
$git commit -am "Fix #1849822. Convert (HTML) view rendering to a render array"
#Switch to main branch
$ git co 8.0.x
#Merge Daniel's patch and watch git's work its magic
$ git merge daniel
Auto-merging core/modules/views/tests/src/Unit/Routing/ViewPageControllerTest.php
Auto-merging core/modules/views/tests/src/Unit/Plugin/Block/ViewsBlockTest.php
Merge made by the 'recursive' strategy.
 13 files changed, 147 insertions(+), 32 deletions(-)
Mar 19 2015
Mar 19

The Drush team is thrilled to announce the release of Drush 6. Full release notes are there. The highlight of Drush 6 is the new outputformat system. You can now request Drush output in easily parsed formats like json, csv, yaml etc.

Please come to our lab at Drupalcon Prague for in-depth discussion and hand-on learning about Drush 6.

Note: if you are using Drupal 8, you should stick with a frequently updated clone of master branch. Drush 6 supports Drupal 6+7 but not Drupal 8.

Drush Project Moves to Github

The Drush project has moved to Github. Update your clones!

See our wiki page which details the motivations and details surrounding this move.

Drush loves Github

Oct 16 2014
Oct 16

Yesterday, the Drupal Security team announced that all Drupal 7 sites are highly vulnerable to attack. Acquia deployed a platform-wide "shield" which protects all our customer sites, while still keeping them 100% functional for visitors and content editors. These sites can now upgrade to 7.32 in a more calm, controlled timeline.

Collaboration - 7 days before release

The Drupal Security team includes representatives from several Drupal hosting companies and Drupal distributions. This is really important, since those folks are pivotal in updating sites when a Security Advisory is released. While discussing the best fix for Drupal, many of us made plans to protect the websites and products under our care.

A week before the announcement, I started collaborating with fellow Security Team members, Damien Tournoud (Platform.sh) and David Strauss (Pantheon). We weighed possible platform-wide remedies and shared our views on the benefits and risks of each. This isn't novel; we recently collaborated on CMI implementations and Drush improvements. I'm proud to report that the Open Source way is embraced even among commercial competitors! Ultimately, Acquia, Platform.sh, and Pantheon all implemented different platform-wide protections for our respective customers. This blog details Acquia’s strategy.

Acquia pursued two independent strategies for mitigating the problem. That way, if one solution didn't pan out, our customers would still be protected. As it turned out, this redundancy paid off handsomely. The two strategies were:

The Acquia MySQL Database Driver

This strategy takes advantage of a seldom used feature of Drupal - custom database drivers. We wrote an Acquia driver that inherits all the behavior of the core MySQL driver, and implements a fixed expandArguments() method. Our expandArguments() is identical to the fixed one in 7.32. The advantage of this strategy is that it fixes the root problem, just as Drupal itself has done. When you use this driver, you are safe.

In order to coax Drupal to use this driver, we take advantage of the fact that Acquia Cloud controls settings.php for its clients in order to inject database details. So we change the ‘driver’ item in $databases to reference our custom driver and then Drupal carries on with serving the request. For the code monkeys out there:

* Use Acquia's Drupal MySQL database driver if it exists, but only for
* web requests, and only up to Drupal 7.31.
function acquia_hosting_use_db_driver($databases) {
  global $conf;
  if (empty($conf['acquia_hosting_disable_sa_2014_005_fix']) && defined('VERSION') && version_compare(VERSION, '7.32', '<') && !acquia_hosting_is_cli()) {
    $driver_src = '/path/to/drupal-db-drivers/ahmysqldseven';
    $driver_dst = DRUPAL_ROOT . '/includes/database/ahmysqldseven';
    $incs = array("$driver_dst/database.inc", "$driver_dst/install.inc", "$driver_dst/schema.inc");

    // In Live Development mode, create a symlink to the driver, since the
    // the system won't.
    if (is_writeable(DRUPAL_ROOT) && is_dir(dirname($driver_dst)) && !is_link($driver_dst)) {
      @symlink($driver_src, $driver_dst);

    // If all the include files exist, use the driver.
    if (count(array_filter($incs, 'is_readable')) == count($incs)) {
      foreach ($databases as $name => $info) {
        if ($databases[$name]['default']['driver'] == 'mysql') {
          $databases[$name]['default']['driver'] = 'ahmysqldseven';

  return $databases;

Sanitizing the HTTP Request

Our initial idea with this strategy was to block incoming web requests that contained malicious keys in the GET or POST data. Doing this requires knowing what valid, non-malicious keys can look like, so we could avoid false positives which would block a valid request, breaking the website. We could not just consider what keys Drupal core uses, because our customers run an enormous variety of contributed and custom modules. We therefore deployed some additional logging across our entire platform to log all unusual-looking keys for later human inspection.

We had hoped that there would be some fairly simple rules we could apply, such as blocking requests with keys containing certain characters. As soon as we started looking at the data, we realized that would never work. One site, for example, had POST keys that looked like taxonomy terms or perhaps page titles, such as Real estate news and advice. So much for disallowing whitespace! What about just blocking MySQL keywords? Nope - we found sites with POST keys such as select all.

This meant that if we wanted to block MySQL keywords without false positives, we'd have to construct regular expressions matching more of SQL's syntax, such as SELECT\s.*FROM or INSERT\s.*INTO. However, SQL has a rich syntax, with many potentially dangerous commands: TRUNCATE, GRANT, CHANGE MASTER, etc. We also found sites with legitimate POST keys containing semicolons (the MySQL query separator) and double-hyphen (the MySQL comment character). Bottom line, there was just no reasonable way to correctly identify attacks by inspecting the GET/POST data without a lot of false positives.

Finally, we realized that even if we could correctly detect attacks, it would be insufficient. GET/POST keys were only one possible vector to exploit the underlying vulnerability. For all we knew, there were other vectors that had nothing to do with the payload of web requests at all.

In the end, we decided that a mitigation strategy based on blocking web requests was worse than useless---it would provide a false sense of security while also breaking customer web sites. We abandoned the approach entirely.

Go Live - Wednesday

On Wednesday, we brought up our shields by enabling the database driver. Since the code was already deployed, we just ran chmod 755 /path/to/db-driver on all our servers. This command makes the DB driver readable, and causes us to pass the final is_readable() check in acquia_hosting_use_db_driver() [see code above]. Running this command is a very quick process, as opposed to a full Puppet deployment which takes a few hours, giving hackers a brief window for attack.

Acquia Customer Success then mobilized, sending out emails, tickets, tweets, and so on. The Acquia Remote Administration team kicked off its mass notification and updating of codebases.

Arrival of the attacks - Wednesday night

So far, the Acquia MySQL driver is performing flawlessly. Our customer sites are protected and still functional for all site visitors and content editors. Our logging showed no malicious requests before the SA was released, but within 12 hours we were seeing boatloads of them. For the safety of non-Acquia hosted sites, I won't share the malicious requests. But I can share their payloads. Here are some clever queries that attackers are trying to run.

update users set name='admin' , pass = '$S$CTo
9G7Lx2rJENglhirA8oi7v9LtLYWFrGm.F.0Jurx3aJAmSJ53g' where uid = '1';

INSERT INTO `menu_router` (`path`, `load_functions`, `to_arg_functions`, `description`, `access_callback`, `access_arguments`)
VALUES ('mziogj', '', '', 'mziogj', 'file_put_contents',’[TROJAN]’);

update {users} set mail='[EMAIL_ADDRESS]' where uid=1;

set @a=(SELECT MAX(uid) FROM users)+1;INSERT IN
TO users set [email protected],status=1,name='n0n0x' , pass = '$S$CTo9G7Lx2jmHrpHDdKDR0R8X/
q4H9PXo02REYap3z2t8UE3F0DfC';INSERT INTO users_roles set [email protected],rid=3;


Many thanks to the all-volunteer Drupal Security Team. A big part of the web is dependent on your generosity, skill, and professionalism.

About Drupal Sun

Drupal Sun is an Evolving Web project. It allows you to:

  • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
  • Facet based on tags, author, or feed
  • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
  • View the entire article text inline, or in the context of the site where it was created

See the blog post at Evolving Web

Evolving Web