Jun 21 2018
Jun 21

At 12:27pm, our alerts started firing. Multiple ones -- website down, server down, secondary monitoring -- one of our client's servers had completely disappeared off the Internet.

I confirmed that I could not reach the site, or the server, and then looked up our AWS credentials for that customer. They didn't work. Then I tried to call two different contacts at our client, leaving messages, and sent an email asking if they needed assistance restoring their AWS account.

The answer came back 20 minutes later, after our client checked their email account associated with AWS.

Subject: Irregular activity in your AWS account: xxx
Severity: Urgent
Correspondence: Dear AWS Customer,

Your AWS Account may be compromised! Please review the following notice and take immediate action to secure your account.

It turns out that another vendor had posted an AWS API key to a public Github server! This is like putting your password out on the Internet for anybody to see and use.

Worst Case Scenario

A malicious attacker, with an API key granting sufficient access to an AWS account, can shut down and delete your servers, your other accounts, your storage volumes. Worse, they could easily delete your backups, if your backups are managed in the same account.

This is why you should always have backups somewhere else entirely.

In our client's case, the worst case would not have been business-threatening -- we had rolled out an update to their site just last Friday, and our deployment system automatically takes a backup snapshot before each release. So we had all their code, and a relatively recent database. We did not have all of their assets (uploaded images, photos, etc) so there would be some loss, but we had some from months ago, and this is not an asset-heavy site, so they could have recovered.

But if they had taken advice we had given them just a couple months ago and set up a secondary backup, we could have restored them with only a few hours of missing data.

A stroke of luck

It turns out the other vendor had taken the site down. They had gotten the notice about the published key, went into the account and shut off every server they didn't recognize, disabled (or in our case, deleted) the IAM accounts they didn't recognize. And taken out the public website as a result.

Fortunately, the only thing deleted was our IAM account. This was easy enough to recreate, and everything else we were able to turn back on. Other than a little over an hour of downtime, and some emergency service, they escaped unscathed.

Are all your eggs in one basket?

If you don't have a full backup at an entirely different location, using an entirely different provider, you may have no backup at all if an attacker gains access to your hosting credentials.

If you don't have historical backups, or nightly checks on the integrity of your website and assets, you could lose data through corruption or infection, and if you don't notice before your backups rotate through to deletion, your backup strategy is not robust enough for the real world.

If you're not sure, we can answer that for you. Our Site Appraisal with a Security Assessment will answer these questions, and more! Give us a call and we'll get you squared away.

Mar 30 2018
Mar 30

No, you should not. You should let us worry about them, and go back to your business.

Seriously, we're getting questions from all kinds of people about whether this matters. I'm a bit surprised that there is any question about that. Would you be concerned if your top salesperson was selling for somebody else? If your cashiers were jotting down credit card numbers when they charged a card? If your office became a well-known spot for illicit drug or gun dealers? If your office had a bunch of scammers squatting and running a pyramid scheme? If your confidential client information could be revealed as easily as using a bic pen on an old Kryptonite lock?

[embedded content]

Bic Pen vs Kryptonite Lock

We've seen some variation of every single one of those scenarios. And all of them are possible with a remote code execution flaw in a web application, like yesterday's Drupal security vulnerability.

And yet people still don't take website security seriously.

If you have any question about whether it's important to keep your site up-to-date, review that second paragraph again. And then give us a call. Today.

Back up a second. What's remote blah-di-dy-blah blah code execution?

A "Remote Code Execution" vulnerability is a flaw that allows somebody bad to run whatever code they want inside your website. If this happens to you, it is like having a bad employee with keys to anything in your business. Only you don't even know who the employee is, and they are probably sitting in an entirely different country.

Remote because you don't need to have authorized access to the site or server -- they can get in without a key.

Code is what runs your Drupal or WordPress website, and you can find code snippets to do huge amounts of things without even having to write that code -- code can intercept data going in or out of the website, can attack other sites, can change data, can even plant your favorite celebrity's face on a porn video or generate crypto-cash.

Execution means the bad guy can run the code they put up there remotely, and potentially execute your company. Via lawsuits. From your customers whose data you failed to protect. Or the people you defrauded (or rather, your squatters who now control your website defrauded).

Drupalgedden 2 - Wait, didn't they learn the first time?

Yesterday we patched all of our Drupal maintenance customers for a new security vulnerability, dubbed Drupalgeddon 2, within a few hours of the disclosure.

Not only did we patch all the sites, but we did so automatically, with full backups taken immediately before the patch was applied, across over a dozen different hosting companies, 3 different Drupal versions, scads of different clients. We rocked it -- we had only 3 failures across the entire portfolio, all of which were easily dealt with manually. We can easily handle 4 times the number of sites we currently manage.

We did learn from the first DrupalGeddon, at least how to do it automatically. But then, DevOps is what we do best. Enough bragging. Let's look at the vulnerabilities.

DrupalGeddon the First happened in October 2014. That was a remote code execution vulnerability that at its heart had to do with form parameters posted with malicious keys. When you post data from a form, each field has a key (like username or password) and a value (like "John Doe" and "MyPassword123"). Web developers have long known not to trust any of the values coming from browsers, because those are very easy to fake. Keys? Most web applications look for specific keys and ignore keys they don't recognize.

A huge number of attacks target forms of one sort or another, simply because that's one place where you're meant to put stuff into an application. There's a lot of ways to inject stuff the developer doesn't expect, and if an attacker can figure out some way of getting past the protections used by the web developer, she can trick the application into doing something it's not supposed to do. Like doing things to your data (SQL Injection), adding malicious Javascript (Cross-Site-Scripting, or XSS), getting an admin to submit a form for you (Cross-site request forgery, or CSRF), or a variety of other attacks.

Drupal provides a "Form API" that blocks most of these attacks by design, using a lot of security "best practices".

Drupal has automatic "sanitization" of all form values, if the developer uses the core "Form API" it provides. The Form API is extremely powerful, but adds a fair amount of complexity under the hood -- and complexity means more places a vulnerability can creep in and get overlooked. The Form API uses arrays, nested inside other arrays, nested inside more arrays. Going back to the fundamental web form, this means many keys in the form themselves use an array notation, such as "name[0]" "name[1]". The "name" part of that has gotten correctly sanitized (cleaned to prevent any possible injection) -- but what's inside the brackets is a bit harder (because the PHP language itself helpfully converts these to array keys).

With the first DrupalGeddon, they discovered that an attacker could inject nasty stuff into the brackets, and own the site, along the lines of "name[--"put bad stuff here]".

This time around, the attack is on the Form API itself. The Form API tracks lots of stuff in arrays with keys that start with "#". Of particular concern: "#validate", which designates what function to run when the form is submitted. And happens to share the same function signature as "#submit" -- which gets all the other form values, but assumes the form has already been validated. Let's see. What form might I be able to hijack by bypassing validation? What other function might I be able to run that would load my code instead of what's expected? (This is left as an exercise for the reader...)

This has now happened twice to Drupal. Should I move to WordPress?

Gawd, no. Compared to Drupal, WordPress is a shit-show.

WordPress doesn't even have a Form API. Which means there is no standard way for developers to create forms. Which means if you use code from a developer who isn't extremely competent in how to block CSRF, XSS, SQL Injection, and other far "easier" attacks who rolled their own form, chances are your WordPress site has more security holes than swiss cheese.

WordPress core has a reasonably competent security team watching over it. But the > 54,000 plugins available in the WP repository? You're on your own. The security team might handle working with a plugin author if they get notified about a security issue, but they're not out their poring over plugin code looking for vulnerabilities. And with countless proprietary plugins in widespread use getting little to no external review, it's no wonder we see so many hacked WordPress sites.

There are even some accounts of WordPress sites getting broken into using the same exploit code used in Drupalgeddon the First!

If you have a WordPress site at a regular webhost, and don't know what to do if you went to it one day only to find a blank screen, get ahold of us now so we can make sure you're properly backed up and have a recovery plan! You are at far greater risk than you know...

If you're on a reasonably well-built Drupal site, have this vulnerability patched, have reasonably secure hosting, and strong passwords, you're way better off than most WordPress sites. Drupal has a really strong security record, and the vast majority of security issues reported (and fixed) are "privilege escalation" bugs. The typical security update fixes problems like your janitor who has keys to all of your office can find the combination to your safe written in a folder labeled "Insurance contracts." First off, you need to have a sketchy janitor up to nefarious things (why would you hire somebody like that?). And then they would need to go searching in all the most boring places imaginable.

Is this a Zero-day exploit?

No. And that is exactly why Drupal has such a strong reputation for security.

A "Zero-day exploit" is when there is code that exploits a vulnerability before there is a patch for it -- the code to fix a vulnerability was released after it was getting used in the wild. At worst, this is a One-day exploit -- it's been one day since the fix was released, and we haven't yet seen it exploited in the wild -- but that could change today.

Drupal Security coverage shield Drupal Security coverage shield

This vulnerability was discovered by a developer doing security research and audits on Drupal code. At this writing, it is still considered "theoretical" but that is not going to last long.

Disclosing possible issues, and patching them before they are exploited in the wild, is a hallmark of a project that takes security seriously.

Compared to WordPress, the Drupal security team covers not just Drupal core, but also thousands of contributed modules hosted on Drupal.org -- look for the security badge for any module you use and make sure it has security team coverage if you're concerned.

Jan 15 2018
Jan 15

It's only taken two years since the release of Drupal 8 for us to get our own site updated... Cobbler's children and all. But finally, we are proud to unveil our shiny new site!

But wait, don't you tell your clients you don't need a new site?

Take a look around... all our old content is here, most of it in the same place it has always been. In fact, we fixed some things that were missing from our last site -- several pages that had broken videos are now fixed. All in all, our site has right around a thousand pages -- and somewhere between 1/4 to 1/2 of our clients use it to pay their invoices, so Commerce is a critical piece. It turns out our site has a lot more going on than many of our clients, with some content going back over 18 years.

We see our content and the website as our biggest asset. Much of our new business comes through our website -- less now than in the past, but it still plays a vital role in helping new visitors get to know us, learn our strengths, and ultimately develop enough trust to become a client.

In the past couple years, we have added WordPress to our maintenance arsenal, provide regular maintenance, updates, enhancements and improvements. Working with WordPress is easy -- it's like working with a toy, it really does not do much for you. This frees up a web designer to do whatever they want with the look and feel, and the system does not get in the way.

But we're not web designers -- we are developers, system administrators, business consultants, information architects, and we think Drupal 8 is the best system out there. We love working with it -- it's great for marketing people, it's great for developers, it's great for managing data and information, it's great for integrating with other systems, and it's great for the future. So that's what we chose for our site.

A simple example: most WordPress sites we've been seeing have somewhere between a dozen and 50 database tables. This site has over 800 tables (ok yeah, maybe we experiment a bit much) and most of our Drupal sites have somewhere between 100 and 500 database tables. That's just one indication of how much more Drupal is doing for you. Overkill? Maybe, if you just want a blog. But if you're doing e-commerce, customer management, membership management, complex layouts, scheduling, event registration, publishing workflows, you end up with a lot more sophistication under the hood.


The initial migration was easy. We sucked over all our content very quickly, right from the start. But... that's just getting content into the site. There ends up being tons of issues to resolve going forward. Things like:

  • Converting embedded media to the new Drupal media format
  • Finding the current location of videos that were no longer where they used to be
  • Consolidating tags into our current set we want to make a bit cleaner
  • Customer payment profiles, to continue charging our clients who auto-pay their bills as seamlessly as possible
  • Supporting images/media that were previously migrated into Drupal 7

Part of the complexity of this for us was that our site has gone through many versions. First it was entirely custom PHP. Then it was Joomla. Then it was Drupal 6 -- and we folded in a separate MediaWiki site. Then it was Drupal 7, and we folded in a WordPress site. And without a person dedicated to going through the old content and bringing it up to date, we've just accumulated that content and brought it forward, fixing the issues so it continues to look ok (actually better than it ever has before!)

The more we looked around nearing launch, the more stuff we found that needed fixing, so it was a huge push on the week before we pulled the trigger to get that all squared away.


We're really impressed with Drupal Commerce 2, in Drupal 8. It seems very robust, and so much of it "just works" out of the box with very little configuration. We had to create two plugins -- one to support our payment gateway, and one for Washington State tax collection -- and we had to do some tweaks to get migrations from our old Ubercart 7 store for customers, products, and orders -- but otherwise we spent very little time on the Commerce. And we had a new customer successfully make a payment the very next day after we turned it all on!

We did write another custom module to support our payment links. Way back in Ubercart 6, we became early adopters of "cart links", which allows us to send a link to a customer that populates their cart with what they are buying. This sets up our automatic payments for hosting with tax properly calculated, and our monthly maintenance plans. Our bookkeeping system also sends out a payment link that allows people to pay invoices through our site.

We created a custom Product Variation for our invoice payment now that makes this process easier, and so while we were at it, we simplified our cart links to make them easier to figure out on the fly (just sku and qty, and for invoices, invoice # and amount) and also made them "idempotent" (a computer science term meaning you can click the link over and over again and you get the same result -- it won't keep adding more items to the cart).

Front End

Yes, it's Bootstrap. (Caution: that link is not exactly... kind... or appropriate for work) Bootstrap seems to be what everybody wants these days, it's a decent looking theme that we use on almost everything (contributing to that problem!)

The thing is, it looks nice, it works great in mobile, and it lets us focus more on what we want to get across -- our content -- and not spend much time with design. And frankly, that's what we advise for our clients, too -- start with your content, what you're trying to get across, what makes you special. If design is your thing, great! Go out and get a really top notch, custom design. But if it's not... well, just use Bootstrap. And try to use some unique photography, the difference between a great bootstrap site and one that's Meh is just photography.

It's not that we don't think design is important. The key point here is that design should be directed to support some goal you have for the website -- and if your goal is a company brochure or any of a number of different purposes, well, Bootstrap has solved a lot of those basic design problems. Spend your time on your content.

Once you have a really clear idea of what you want your users to do on your site, then bring in a designer to optimize for those goals.

With all that said, we're really excited to have our site current so we can start experimenting with entirely new web UIs. We've particularly been delving into Vue.js, React, and GraphQL, and have some demos we've built and integrated into a couple sites we can't wait to roll out here!

Here's to 2018!

We did launch the site early. There are still layout glitches here we're quickly fixing, in between client work (If you're on Safari, sorry!). But we feel a huge sense of relief to be fully up-to-date on a new site, which gives us so many opportunities to try out new things for ourselves, and then can share what works with our clients.

Need a web partner to bring you up to date? Let us know how we can help!

Jan 12 2018
Jan 12

The news was supposed to come out Tuesday, but it leaked early. Last week we learned about three variations of a new class of attacks on modern computing, before many vendors could release a patch -- and we come to find out that the root cause may be entirely unpatchable, and can only be fixed by buying new computers.

Today Microsoft released a patch -- which they had to quickly pull when they discovered that it crashed computers with AMD chips.

Essentially Spectre and Meltdown demonstrate a new way of attacking your smartphone, your laptop, your company's web server, your desktop, maybe even your tv and refrigerator.

Meltdown - Animated Meltdown in Action

This all sounds dreadfully scary. And it is... but don't panic! Instead, read on to learn how this might affect you, your website, and what you can do to prevent bad things from getting worse.

How will this affect you?

All of these attacks fall into a class of "Information Disclosure." A successful Spectre attack can reveal information you want to keep secret -- mainly your passwords, and security keys widely used to protect information and identity.


Have any Bitcoin lying around? Your wallet could get compromised with this type of attack. Visit any SSL sites? A secure SSL certificate on a server might have its private key stolen, and incorporated into a fake certificate -- which would make "Man in the middle" attacks from wifi hotspots a lot more effective -- a phisher could set up a fake copy of your bank's website, and there would be no way to tell it apart from the real website, because it has a copy of the real certificate. Use a password manager to keep track of all those different passwords you need for each site? Spectre can make those secrets -- not so secret. This is far worse than Heartbleed.

Over the coming months and years, this will be a headache. Security updates on your phone and all of your computers will be more important to apply promptly than ever before -- because a new 0-day attack could give the attacker the keys to your online kingdom.

The good news is, browsers have already updated with fixes that make a Spectre attack from remote Javascript much more difficult, and Meltdown is nearly patched.

The bad news is, patching for Meltdown means slowing down your computers substantially -- reports suggest by somewhere between 5% and 30%, depending on the types of computing being done. And there isn't really a way of patching Spectre -- it's a design flaw having to do with how the processor caches what it's working on, while using something called "Speculative Processing" to try to speed up its work -- fully preventing a Spectre attack means deploying new hardware processors that manage their low-level caching in a different way.

So preventing Spectre attacks falls more into the realm of viruses -- blocking specific attacks, rather than stopping the vulnerability entirely, at least as I understand the problem. For more, ZDNet has a pretty understandable explanation of the vulnerabilities.

How can they attack me?

To exploit any of these attacks, an attacker needs to get you to run malicious code. How can they do this? Well, for some Spectre attacks, through Javascript running in your browser. Firefox and Safari released updates that make the Javascript timer not so accurate -- having accurate timing to detect the difference in speed for loading particular caches is a critical part of how the currently identified attacks work. But it's scary that this level of attack could be embedded in Javascript on any website you visit...

Browsers are changing faster than ever, though, and I wonder if this will set back some proposed browser features like WebAssembly, which could be a field day for attackers wanting to deliver nasty code to you through innocuous web pages. It's relatively easy for a browser maker to make the Javascript execution environment fuzzy enough to defeat the precision needed to carry out these attacks. WebAssembly? The entire goal of that is to get programmers "closer to the metal" which is going to make it easier to get creative with exploiting side-channel vulnerabilities.

Browser extensions, mobile apps, anything you download and install now have far more opportunity to steal your secrets than ever before.

How will this affect your website?

Your website's host is almost certainly vulnerable. If you are not hosting on dedicated hardware, Spectre basically means that somebody else hosting on the same physical hardware can now possibly gain access to anything in your hosting account.

There are basically 3 "secrets" in nearly every website that's built on a CMS (like Drupal or WordPress) that might be a target:

  1. Your FTP, SSH, or other hosting account logins -- this could give them full access to your site, allow an attacker to upload malicious code, steal data, damage your site, whatever they want.
  2. The private key for your SSL certificate -- this would allow them to create a fake SSL certificate that looks legitimate to anybody visiting their copy of the site. This is particularly a problem for financial institutes, but it could happen to anyone -- this can lead to fake sites being set up under your domain name, and combined with a "man in the middle" used to phish other people, smear your reputation, or a variety of other things.
  3. Any secrets in your CMS -- your login, your passwords, any passwords of users that log into your site, ecommerce data, whatever there is to steal.

If you're on a "virtual private server" or a "shared hosting account", there will be exploits coming for years, until we've all replaced all the computer hardware that has come out in the last 20 years -- and another tenant on your hardware can potentially attack your site.

And those are just the universally available targets. You may have other things of value to an attacker, unique to you.

"Meltdown" got its name because it melts down the security boundaries between what you can do as a user, and the underlying system that has full access to everybody.

Meltdown does have patches available, and these are getting deployed -- at the cost of disabling CPU features built to make them perform quickly. Which means if you're nearing the limits of what your currently provisioned hosting provides, patching for Meltdown may push you over, and force you into more costly hosting.

What should you do now to make things better?

What you can do now really isn't much different than it was a month ago -- but the consequences of failing to use best security practices have gotten a lot higher. You could stop using technology, but who is really going to do that? And who already has all your data, who might get compromised anyway?

We think there are two main things to think about when it comes to this type of security planning:

  1. Make sure you are doing all you can to avoid an attack, and
  2. Have a plan for what to do if you fall victim to an attack.

Avoid an attack

To avoid an attack, a little paranoia can go a long way. Get something in email you don't recognize, with an attachment? Don't open the attachment. On a public wifi network? Don't go anywhere well known, like a banking website -- wait until you get home or on a network you trust.

Apply all security updates promptly, and verify that you're getting them from the real source. Pay attention to anything that looks suspicious. Expect more phishing attacks for the foreseeable future (as if we didn't have enough already...) Regularly check that any sites or servers you have do not show any signs of unexpected activity, changed files, etc.

It might be hard to detect an intrusion, because if they've hacked you, they will likely be connecting as you -- so set up any 2-factor authentication you can, consider getting security dongles/hardware tokens, and just think about security before clicking that link.

Plan for disaster

Nobody can be perfect. There is no such thing as "secure" -- there's always a way in. The practice of security is really one of risk management -- identifying what the consequences of a given security breach is, what the costs to avoid that breach are, and finding a balance between the cost of securing something and the cost of suffering a breach.

That equation varies for everybody -- but some key values in that equation just shifted -- now the consequences of a minor breach can lead to a much bigger problem than before. Or, perhaps more accurately, now we know about some ways to make these breaches worse, and thus the likelihood of them happening have become higher.

When it comes to a website, the three main risks to consider are:

  • Loss of service (Denial of service -- your website goes away)
  • Loss of data (you lose access to something you need -- e.g. ransomware, hardware failure without sufficient backups, etc)
  • Information Disclosure (revealing secrets that can cost you something)

What has changed now is that these new information disclosure attacks can reveal your keys and passwords, and then an attacker can conduct the other kinds of attacks impersonating you. It used to be that information disclosure was a bigger concern for data you stored in a database, because the operating system takes such special care of your passwords and keys -- but now we've learned the operating system protections can be bypassed with an attack on the CPU. And that this has been the case for the past 20 years.

Do you have a disaster recovery plan that outlines what steps to take if you discover you've been hacked? If not, we can probably help, at least for your website and server. We've written several disaster recovery plans for ourselves and our clients -- reach out if we can help you create one. We can also do the day-to-day work on your Drupal or WordPress site and server to keep them fully up-to-date, with a full testing pipeline to detect a lot of things that can break in an update.

Let us know if we can help!

Dec 29 2017
Dec 29

We're nearing launch of two new Drupal Commerce sites, one of them being this one. It turns out Freelock.com has some relatively sophisticated commerce needs: some taxable products, some non-taxable products. Recurring subscriptions. Arbitrary invoice payments.

We previously blogged about Commerce 2 Price Resolvers. Now, let's get into some of the details of payment gateways and taxes.

This post is going to be more high-level, less code, because there are some critical concepts that are fundamental to how Commerce 2 currently works that site owners need to understand.

Onsite Payment Gateways always store credit card info

Payment gateways have different capabilities. Some support storing a user's card in the gateway. Others have an "offsite" flow, sending you to their site to collect card info and then returning you back to the commerce site when done.

An "Onsite" payment gateway is one that keeps the visitor on the site, collects the card numbers and posts it from the site to the gateway. In Drupal Commerce 2, the payment flow for onsite gateways is completely different than in Commerce 1, Ubercart, or many other shopping carts: it now stores the credit card in the gateway first, and creates a "Payment Method" associated with the user that contains the token from the gateway. It can then use that token for any future charges (as long as the gateway continues to honor it).

This is a fundamental change that makes recurring transactions and "card on file" functionality pretty much built into Commerce 2!

This means that the Commerce EPN Gateway module we created has full support for recurring functionality out of the box.

One other thing of note: Each gateway can declare its capabilities in its code, and Commerce then automatically exposes that functionality to the store admins. One thing that confused me slightly was "Voids" -- in most payment gateways, you can void a transaction up until the batch closes (which usually happens once per day). However, in Commerce 2, "Void" is used to cancel an authorization, not void a transaction. If you use "authorization only" transactions, and capture the payment after the order is fulfilled, then you can use Void to cancel the authorization (or Capture to process it).

The Tax system needs help to recognize taxable vs non-taxable products

Out of the box, the tax system is extremely smart about zones and tax registrations. You can define, per store, which taxes you need to collect, and Commerce will automatically charge tax to customers in a taxable zone. However there is no built-in distinction for taxable vs non-taxable products.

There are tax rules coming, but in the meantime if you have both taxable and non-taxable sales, you need a custom tax resolver to do the right thing. These are relatively easy to create. We borrowed heavily from another tax module the configuration settings that allow you to define on a tax type which product variation(s) it applies to.

One other gotcha we found while testing recurring payments -- if you use the "Include tax in product price" option, recurring billing adds the tax again each time the purchase recurs -- be sure to uncheck these boxes and keep the tax separate, and then it works fine.

If you're in Washington State and need to collect sales tax, this module works fine today -- check it out here. However note that it's not yet possible to create a tax report -- we do store the location data with each transaction, and will be sure to have this report available by April, when we need it!

Working with Drupal Commerce 2 is really refreshing. It feels like you're working on a robust, well-oiled system with a very nice user experience right out of the gate. It is still missing some bits and pieces around the edges, and customizing it does involve turning to code more quickly than previous versions, but overall we're finding it really straightforward to work with. Stay tuned for our coming sites, and if you need a Commerce site, reach out to us!

Dec 21 2017
Dec 21

Drupal security updates generally come out on Wednesdays, to try to streamline everybody's time. WordPress security notices come out... well, whenever whichever feed you subscribe to bothers to announce something.

Today's notices showed some striking differences between the two communities.

Drupal module vulnerabilities

There were 4 Drupal contributed modules flagged with security bulletins. Of these 4, 3 of these were not fixed -- the module code was yanked from Drupal.org, and now any site that uses any of these modules has a big red warning "Unsupported module in use!" These were all modules I had never heard of, are not in widespread use, and now have been clearly marked as dangerous.

Unsupported Release message

The 4th security update turns out had actually been fixed over 2 years ago, but the fix had not been released in a "stable" release. The vulnerability did look like a ridiculously easy-to-exploit, dangerous chunk of code, and it only affected the module in Drupal 7.

Searching our sites, I found we did not have any Drupal 7 sites using this module, but we did have 2 Drupal 6 sites that actively used it. So I rolled up my sleeves and looked at the code to find that the affected sections were not present in the Drupal 6 version, which solved the problem in a different way.

WordPress vulnerabilities

Unlike Drupal, there is no single source of notifications of WordPress plugin vulnerabilities. There are multiple companies that do security assessments, each with their own conclusions, and none able to signal to the wider WordPress community about a problematic plugin.

We subscribe to several of these community feeds -- and the tale we get is full of drama, conflicting stories, firms calling out each other for misleading information... and basically you're on your own when it comes to determining whether you're using a safe set of plugins.

But today took the cake:

We recommend that you uninstall the Captcha plugin immediately from your site. Based on the public data we’ve gathered, this developer does not have user safety in mind and is very likely a criminal actor attempting yet another supply chain attack.

... that's from a WordFence blog post outlining a security vulnerability they've highlighted in a module that was removed from the WordPress Plugin Repository -- not due to the security hole, but rather because of a trademark infringement issue. The WP Vulnerability Database shows it had a back door in it until release 4.4.5. WordFence is apparently unconvinced that the plugin remains trustworthy at all -- because the new maintainers have included similar backdoors in other plugins they manage.

Just to clarify what these backdoors do: they allow anybody with the "secret handshake" method to knock on your WordPress site's door to log in as a site admin, and remove the evidence they have done so.

It does get worse... Another of our regular sources, Plugin Vulnerabilities, has found over 6000 current installations of an actively attacked plugin that was removed from the Plugin Registry a year and a half ago, and abandoned 4 years ago.

No wonder WordPress has such a bad security reputation!

If you're running a WordPress site, and not actively using a security service or watching the security lists, your site is at risk! Our WordPress protection plan is a must for making sure site updates get done, with testing on development copies, and with solid backup management so we can recover if you have any issues.

Of course, we have a Drupal plan too...

Dec 12 2017
Dec 12

In the previous post on A custom quantity price discount for Drupal Commerce we created a compound field for price breaks, which was composed by two subfields. One for a quantity threshold, the other for the price at that threshold.

That post covered everything needed to get quantity discounts working within Drupal Commerce, but for this particular project, we also had to find a way to populate these price breaks through the integration with their Sage accounting system.

Source Data

We're starting with their existing CSV exports, and one of them is a file that provides up to 5 pricebreaks for each project. These files get copied up to the webserver each hour, and then the migration into Drupal will get triggered.

Here's what the data looks like in the source CSV file:

Pricebreak CSV Data

... This file has other pricing data, for customer-specific pricing, so our migration skips those rows. The key thing is that there are 5 pairs of columns for the threshold and pricing, called "BreakQuery1" - "BreakQuery5" and "DiscountMarkup1" - "DiscountMarkup5". They use a very large number (99999999) for the largest tier.

So our migration needs to match these up into pairs, and add each pair as a delta in the price break field for that product variation.

Migration Configuration

Setting up migrations is covered elsewhere, as is importing/exporting configurations. For a project like this, we simply copy an existing migration configuration to a new file, edit it with a text editor, and import it into the site's configuration. For this migration, that was all we needed -- there is no supporting code whatsoever, the entire migration is handled with just a YAML file.

We're going to name this migration after the source file, im_pricecode, so the basic setup goes like this:

  1. Export the site configuration.
  2. Copy another CSV migration file to a new "migrate_plus.migration.im_pricecode.yml" file.
  3. Inside the file, delete the uuid line, and set the id to "im_pricecode". When the configuration is imported, Drupal will assign a new UUID, which will get added when you next export the configuration, and the id needs to match the filename.
  4. Add migration tags/group as appropriate.

That's the basic setup. There are 3 crucial parts to a migration configuration: source, process, and destination. All of the heavy lifting here is in the process block, so we'll tackle that last.

Here's our source section:

  plugin: csv
  path: /path/to/IM_PriceCode.csv
  delimiter: ','
  enclosure: '"'
  header_row_count: 1
    - ItemCode
    - PriceCodeRecord
  track_changes: true

This section needs to get set to the data source. (We already have plans to swap this for a webservice call down the road). In this case, we're using the "csv" source plugin provided by the Migrate Source CSV module. Of particular note here:

  • header_row_count here indicates that the first line of the file contains column headers, which will be automatically available on the source $row object in the migration, for processing.
  • keys need to specify a field or set of fields that uniquely identify the row in the CSV file. In our case, the file also contains customer-specific pricing data -- this makes the ItemCode appear multiple times in the file, so we need to add the second column to get a unique key.
  • track_changes stores a hash of the source row in the migration table. On future migration runs, the source row is checked against this hash to determine if there's a change -- if there is, the row will get updated, if not, it will be skipped.

Next, the Destination:

  plugin: 'entity:commerce_product_variation'
  destination_module: commerce_product
  default_bundle: default
    - ci_item_variation

The destination is pretty self-explanatory. However, there's one crucial thing here: we are running this migration on top of entities that have already been created by another migration, "ci_item_variation". We need to declare that this migration runs after ci_item_variation, by using migration_dependencies.

Finally, we get to our "process" block. Let's take this a field at a time:

      plugin: migration_lookup
      migration: ci_item_variation
      source: ItemCode
      no_stub: true
      plugin: skip_on_empty
      method: row

variation_id is the primary key for commerce_product_variation entities. We are running this through two plugins: migration_lookup (formerly known as just migration) and skip_on_empty.

The migration_lookup loads up the existing product_variation, by looking up the variation_id on the previous ci_item_variation migration. We use "no_stub" to make sure we don't create any new variations from possible bad data or irrelevant rows. The second plugin, skip_on_empty, is another safety check to be sure we're not migrating bad data in.

    plugin: static_map
    source: PricingMethod
      O: O
    default_value: ''
    plugin: skip_on_empty
    method: row

PricingMethod is what indicates the type of pricing each row in the source file contains. We don't need to migrate this field, but we do want to use it as a filter -- we only want to migrate rows that have this set to "O". So we chain together two more plugins to achieve this filtering, and we assign the result to a dummy field that won't ever get used.

The first static_map plugin maps the "O" rows to "O", and everything else gets set to an empty string.

The next plugin, skip_on_empty, simply skips rows that have the empty string set -- which is now everything other than rows with an "O" in this column.

Now we get to the fun stuff:

  plugin: get
    - BreakQuantity1
    - DiscountMarkup1
  plugin: get
    - BreakQuantity2
    - DiscountMarkup2
  plugin: get
    - BreakQuantity3
    - DiscountMarkup3
  plugin: get
    - BreakQuantity4
    - DiscountMarkup4
  plugin: get
    - BreakQuantity5
    - DiscountMarkup5

The whole trick to getting the 10 columns collapsed into up to 5 instances of our quantity price break field, each with two sub-fields, is to rearrange the data into a format that matches the field. This involves creating two layers of dummy fields. The first layer are these 5 dummy fields, break1 - break5, which group the BreakQuantityN and DiscountMarkupN fields together. The resut are these 5 dummy fields that each have a BreakQuantity and a DiscountMarkup.

One crucial point here is the order -- these are now set as an array with two values. If we take a look at the source data, the row for ItemCode 23311 will have an array that looks like:

$row->break1 = [
  0 => "9",
  1 => "5.1",
$row->break2 = [
  0 => "99999999",
  1 => "4.85",
$row->break3 = [
  0 => "0",
  1 => "0",

Remember that each of these is an indexed array, with 0 and 1 as keys.

Now we create our second layer -- a dummy field that combines the other 5 dummy fields:

  plugin: get
    - '@break1'
    - '@break2'
    - '@break3'
    - '@break4'
    - '@break5'

Prefixing a fieldname with "@" means use the row from the destination (the one we're building up) and not the original source row -- this lets us access the dummy fields we just created. The YAML parser requires them to be quoted.

So now we have a value built up that pretty much matches the entity field structure:

$row->breaks = [
  0 => [
    0 => "9",
    1 => "5.1",
  1 => [
    0 => "99999999",
    1 => "4.85",
  2 => [
    0 => "0",
    1 => "0",

Now we assign this to our real destination field:

  plugin: iterator
  source: '@breaks'
      plugin: skip_on_empty
      method: process
      source: '0'
    price: '1'

For the actual field values, we need to map the values to the actual subfields -- and we want to skip the empty pricebreak columns. The iterator plugin lets us loop through the breaks, and then assign the subfields with further processing.

The threshold sub-field is 0 if this break is unused, so we can use "skip_on_empty" to skip it -- this time we use the "process" method to only skip this field, and not the entire row. Because the threshold is the first value in the indexed array for each pricebreak, this is the "0" source (and needs to be quoted to be interpreted as an index).

The price sub-field is simply the second value in each of the pricebreak fields, with index "1".

Migrate is cool!

We love working with migrate module in Drupal 8 because of how powerful it is, and how easy to customize. We routinely create process plugins to quickly do some custom transformations, or create source plugins to connect to custom data sources, and I thought I would have to do one of those to solve this problem -- but it turns out some YAML and dummy fields worked with no additional code, pretty much the first time!

Hat tip to MTech for the basic technique! If you have an improvement or suggestion for other cool migrate tricks, leave a comment below! Or if you have a data integration you need done, reach out...

Dec 09 2017
Dec 09

We're in the midst of a Commerce 2 build-out for a client, and a key requirement was to preserve their quantity pricing rules. With thousands of products, and different pricing rules for each one, they need the price for each item in the cart adjusted to the appropriate price for the quantity purchased. When validating our plan for Drupal Commerce 2, we stumbled upon some examples of a custom price resolver, and knew this would work perfectly for the need.

Planning out the functionality

This past week was time for the implementation, and in addition to creating the new Price Resolver, we had to decide where to store the pricing data. The previous solution used a custom SQL lookup table, which they kept up-to-date from their back office Sage accounting system through an hourly CSV import.

We could, at worst, follow the same pattern -- query a table using the product SKU. But then we would have to maintain that table ourselves, have a bunch more custom code in place. Given that we were already using Drupal 8's awesome Migrate module to populate the site, we looked further for how we might best implement this using standard Drupal practices.

Commerce 2 has two levels of entities representing products -- the "Product" entity with the text, photos, description, title, and the "Product Variation" which contains the SKU, the price, and specific attribute values. Both use the standard Drupal Field API, so we can easily add fields to either. Why not add a multi-value field for the price breaks?

However, each price break does have two components: a "Threshold" value for maximum quantity valid for a price, and the price itself. We don't want something heavy like paragraphs or an entity reference here -- creating a custom field type makes the most sense.

So our task list to accomplish this became the following:

  1. Create a custom field type for the price break, along with supporting field widget and formatter.
  2. Add an "unlimited" value instance of that field to the default product variation bundle.
  3. Create a custom price resolver that acts upon this field.
  4. Migrate the CSV price-break data into the price break fields (covered in a later blog post).

There's a couple tools that make this process go very quickly: Drupal Console, and PHPStorm. Drupal Console can quickly generate modules, plugins, and more, while PHPStorm is an incredibly smart IDE (development environment) that helps you write the code correctly the first time, with helpful type-aheads that show function header parameters, automatically add "use" statements as you go, and more.

Create a custom Quantity Pricebreak field type

First up, the field type. We need a module to put our custom Commerce functionality, so let's create one:

> drupal generate:module

 This project's code name is dte, so we created a module called "dte_commerce", with a dependency on commerce_price.

Next, generate the field type plugins:

> drupal generate:plugin:field

... Enter the newly created module and "ProductPriceBreakItem" for the field type plugin class name. Give it a reasonable name, id, and description. We made the widget class "ProductPriceBreakWidget" and the formatter "ProductPriceBreakFormatter".

When we were done, we had a directory structure that looked like this:

Product Price Break Field Type

The field type plugin is the first thing to get in place. Drupal Console creates this file for you, you just need to fill in the necessary methods. It actually creates more than you need -- you can strip out several of the methods, because we don't need a field settings form for this custom field type. (If we wanted to publish the module on Drupal.org, we might want to create settings for whether the threshold for a price is the bottom value or the top value of threshold, perhaps -- but for the time being we're keeping this as simple as possible).

So we only need these methods:

  • propertyDefinitions()
  • schema()
  • generateSampleValue()
  • isEmpty()

There are a lot of examples of creating custom fields around the web, but very few were clear on what types were available for propertyDefinitions and schema -- it took a bit of digging and experimenting to find types that worked.

We ended up with:

  public static function propertyDefinitions(FieldStorageDefinitionInterface $field_definition) {
    // Prevent early t() calls by using the TranslatableMarkup.
    $properties['threshold'] = DataDefinition::create('integer')
      ->setLabel(new TranslatableMarkup('Threshold'));
    $properties['price'] = DataDefinition::create('string')
      ->setLabel(new TranslatableMarkup('Price'));

    return $properties;

... two fields, threshold and price. For this client, quantity is always an integer. We might've chosen numeric for price, but PriceItem in commerce_price uses strings, so we figured we would follow suit.

We considered extending commerce_price's PriceItem field type, but then we would end up with more configuration to do. We thought of trying to use a PriceItem field in the definitions, but it seemed like we would need to handle the schema ourselves anyway, and just seemed more complicated than needed -- if there is a simple way to do this, I would love to hear about it -- please leave a comment below!

Next up, the Schema:

  public static function schema(FieldStorageDefinitionInterface $field_definition) {
    $schema = [
      'columns' => [
        'threshold' => [
          'type' => 'int',
        'price' => [
          'type' => 'numeric',
          'precision' => 19,
          'scale' => 6,

    return $schema;

Simple enough schema -- an integer, and the same numeric field definition that commerce_price uses.

  public static function generateSampleValue(FieldDefinitionInterface $field_definition) {
    $values['threshold'] = rand(1,999999);
    $values['price'] = rand(10,10000) / 100;
    return $values;

Our compound field has two different values now, so for tests or devel generate to work, we need to populate random values for those fields.

Finally, isEmpty():

  public function isEmpty() {
    $value = $this->get('threshold')->getValue();
    return $value === NULL || $value === "" || $value === 0;

... if this method returns TRUE, the field is considered empty and that instance will get removed.

That's pretty much it! With the field type created, and this module enabled, you can add the field anywhere you add fields to an entity type.

On to the Widget...

Product Price Break Field Widget

Each field type needs at least one widget that can be used for editing values. Our custom widget needs to provide the form for filling out the two values in the custom field type -- threshold and price. Drupal Console provides a bunch of other method boilerplate for settings forms, etc. but most of these are optional.

We really only need to implement a single method: formElement(). This method returns the Form API fields for the editing form, so it ends up being just as simple as the field type methods:

public function formElement(FieldItemListInterface $items, $delta, array $element, array &$form, FormStateInterface $form_state) {
  $element += [
    '#type' => 'fieldset',
  $element['threshold'] = [
    '#type' => 'number',
    '#title' => t('Threshold'),
    '#default_value' => isset($items[$delta]->threshold) ? $items[$delta]->threshold : NULL,
    '#size' => 10,
    '#placeholder' => $this->getSetting('placeholder'),
    '#min' => 0,
    '#step' => 1,

  $element['price'] = [
    '#type' => 'textfield',
    '#title' => t('Price'),
    '#default_value' => isset($items[$delta]->price) ? $items[$delta]->price : NULL,
    '#size' => 10,
    '#placeholder' => $this->getSetting('placeholder'),
    '#maxlength' => 20,
  return $element;

This ends up being a simple widget that looks like this on the product variation form:

To finish up our custom price break field type, we need a field formatter.

Product Price Break Field Formatter

The Formatter is responsible for displaying the field. The only needed method is viewElements(), which renders all instances of the field. In this method we call another method to render each one. For now, this is really basic and ugly -- we'll come back and theme it later.

Here's the code:

public function viewElements(FieldItemListInterface $items, $langcode) {
  $elements = [];

  foreach ($items as $delta => $item) {
    $elements[$delta] = ['#markup' => $this->viewValue($item)];

  return $elements;

protected function viewValue(FieldItemInterface $item) {
  return nl2br('Up to: '. Html::escape($item->threshold). ' Price: '. Html::escape($item->price));

 ... and that's it! We have our custom field built. It currently looks like this:

Price Breaks on product

Next we add it to the product variation...

Add a product price break field to the product variation type

  1. Under Commerce -> Configuration -> Products -> Product Variations, on the Default row's operations button, go to "Manage Fields".
  2. Click "Add Field".
  3. In the "Add a new field" dropdown, under General select "Product price break".
  4. Add the label, and verify the machine name -- we chose "Qty Price Breaks", as field_qty_price_breaks.
  5. In "Allowed number of values", select Unlimited.

That's it for the widget, data storage, and everything! We don't even need to go near the database. The custom price break is now available on product variations.

Create the Custom Price Resolver

Now comes the fun part, the Price Resolver. The price resolver is implemented as a tagged Drupal Service, so the quick way to generate it is:

> drupal generate:service

We called it QtyPriceResolver, and put it in dte_commerce/src/Resolvers/QtyPriceResolver.php.

A Drupal service is registered in a modules' .services.yml file, so here's what we ended up with in dte_commerce.services.yml:

    class: Drupal\dte_commerce\Resolvers\QtyPriceResolver
    arguments: []
      - { name: commerce_price.price_resolver, priority: 600 }

The crucial part here is the tag -- by registering the service as a "commerce_price.price_resolver", this service will get called any time the price of a product is evaluated. The Priority is used to determine which order resolvers get called -- the first one to return a price sets the price for that product, and any other resolvers get skipped.

Our service class needs to implement \Drupal\commerce_price\Resolver\PriceResolverInterface. There is one crucial method to implement: resolve(). Ours looks like this:

public function resolve(PurchasableEntityInterface $entity, $quantity, Context $context) {
  if (isset($entity->field_qty_price_breaks) && !$entity->field_qty_price_breaks->isEmpty()) {
    foreach ($entity->field_qty_price_breaks as $price_break) {
      if ($quantity <= $price_break->threshold) {
        if (!isset($current_pricebreak) || $current_pricebreak->threshold > $price_break->threshold) {
          $current_pricebreak = $price_break;
    if (isset($current_pricebreak)) {
      return new Price($current_pricebreak->price, 'USD');

The key things to notice here is that we have hard-coded this to the field machine name we created in the product variation -- field_qty_price_breaks. If we were writing this more generally, we would iterate through the fields of the product variation looking for a ProductPriceBreak field type -- this is one shortcut we took. The other thing to notice is the return value -- we need to either return a \Drupal\commerce_price\Price object, or void. Because we're not storing the currency code, we always return "USD" as the currency code.

And with that, we have fully implemented the custom quantity price discount system! All that's needed is the actual pricing data.

In the next article, we'll show a slick trick for migrating a set of pricing thresholds from a CSV into multiple price break fields...

Dec 01 2017
Dec 01

I just read a quick post over on another Drupal shop's blog, Be a Partner, not a Vendor, and added a comment to the great point Dylan made about not limiting your project to the specs.

We've recently started asking our clients directly about their priorities for the project. Not just overall, but specifically for each one -- and particularly how they would rank these three aspects:

  • Timing
  • Budget
  • User Experience

It turns out the answer to this question can vary a lot! And if you're not on the same page with your client, you're probably going to disappoint them.

We find with most of our projects that delivering the base functionality is generally really straightforward, and we almost always nail the budget. But it turns out that the base functionality often is not as user-friendly as our clients pictured. Getting really nice, usable interfaces can take a lot more effort to get right, than just delivering a working, functional solution. This is the grey area where disagreements and missed expectations grow.

It's particularly a challenge for budget-sensitive clients. Lots of people come to us looking for Facebook-like user experiences, but they certainly don't have the budget to have a development team the size of Facebook working full time! We can provide an amazing amount of functionality on shoestring budgets, but that doesn't mean they're going to be as polished as what software-as-a-service providers build with multiple in-house developers.

Which are your priorities?

This model is not new. It's commonly called the Quality Triangle or Project Management Triangle -- "Fast, cheap, or good -- pick 2." However, instead of just identifying the lowest priority, we think ranking all three is more useful. If we take these 3 priorities, there are 6 different ways of ordering them:

  • Fast, within budget, cut scope
  • Strict budget, get launched, cut scope
  • Fast, high quality, add budget to get there
  • Premium user experience, get launched, ok to go over budget
  • Strict budget, high quality, take as long as you need
  • Premium user experience, keep to budget, willing to wait

Let's drill down into these project priorities just a little more, and give them names.

The Agile Entrepreneur - Fast and within budget

If you prioritize speed over all else, and don't mind reducing scope, you're probably into Agile. "Launch early and often", "if you're not breaking things you're not moving fast enough" and other slogans fit this type of prioritization, and it's pretty much the Silicon Valley mindset these days -- many of the most successful Internet companies started out this way.

The Hustler - Spend the least amount to get a result quickly

Starting with little or no budget, the hustlers get out and get to work, creating value out of not much resources. They may not have the budget to spend early on, but they emphasize getting something up and working so they can grow and add more functionality later, even if the first few releases are garbage -- having a presence is more important than getting it right.

The Mogul - Get there fast, don't care how much it costs

This clearly describes Amazon.com -- throw money at the problem with speed being the highest priority. Get something out the door and fix it later -- the biggest difference between the Mogul and the Agile entrepreneur is how much capital they have to throw at the problem.

The Visionary - It's got to be the best, get it going quickly

With more of an emphasis on quality, while still getting the project done and launched, these kinds of visionaries are among our favorites to work with. These are the sites that win awards, let us stretch our toolbox, and can be really fun to do.

The Main Street business - Will wait for a decent result, but it's got to be within my budget

We are quite accustomed to working with businesses with a very limited budget who want to get the best result possible. The web is developing so rapidly, things are changing so quickly, that cutting-edge sites of five years ago cost a quarter of what they used to cost. We can spin up basic sites in an afternoon. But if you want sophisticated user experiences, you might have to wait until there's a publicly-available module we can drop into your site, if you don't have the budget for us to develop it.

The Craftsman - Get the best result, however long it takes

We recently launched a site that was over 2 years in development. Most of the delay was our client going quiet on us, and not having their content ready to go, but part of it was the pixel-perfect, exacting design priorities they had, and not a lot of budget to spend on it. Eventually their priorities changed to more of the "Visionary" that needed to actually get it launched, they freed up some budget, and we got them live -- but if you want the best result on no budget, it's probably best to learn how to use the tools to build it yourself!

Or, another alternative these days is crowd-sourcing. We're quite interested in working with companies or organizations with a need, that many other organizations might share -- while you may not have the budget to get the job done, if you can help us reach out to other organizations who might be able to chip in, you could pool resources to make it something we can deliver. Obviously, this takes time, but it's a very interesting model...

Which one describes you -- today?

I can think of a client for every one of these categories -- we can usually find a way to work well with any of these. Where things go downhill is when we either misunderstand a client's priorities (which is why we now ask for these up front) or if we fail to manage to what we know are a client's priorities (allowing a "main street" customer to expand the scope beyond the budget they were willing to spend).

Internally, we definitely fall into the "Craftsman" category, building out our stuff, usually too slow, but always at the highest quality. Our favorite clients are the Visionaries -- the clients who want the highest quality, and don't mind spending to get it. But our successes go across the board.

Which one are you?

Feel free to drop us a line, or give us a ring at 206-577-0540.

Sep 26 2017
Sep 26

... that counts. Results matter. What results are important for you? What are you trying to accomplish with your website?

What matters for one organization may be irrelevant to another. What is the overarching biggest need you are trying to solve with your website? Here are a few we've seen:Graphing results

  • Sell more products
  • Establish credibility, be a trusted authority
  • Inspire interest in working for the company, recruiting
  • Persuade people to donate to a cause
  • Get people to sign up for a membership
  • Connect with other people in a neighborhood
  • Increase registrations for a class, program, session
  • Streamline online payments, make it easy for existing customers to self-register and pay
  • Reduce support costs, provide online chat and ticketing
  • Drive more traffic to support advertising
  • Provide a communications channel to a particular group
  • Unify information coming from a bunch of different sources into a single dashboard

You might have two or three goals for a website, but the more you try to do with it, the harder it is to make it fully effective.

We have so many tools at our disposal to measure things today. But there's a trap -- sometimes we all get focused on measuring things that don't matter at all, simply because they are easy to measure. Does the number of Twitter followers you have matter, if nearly all of them are bots who are never going to come to your site? Does the amount of traffic you get matter if nobody ever buys from you? Does the number of contact form submissions matter if it's all spam? Perhaps... but probably not.

There are definitely a few key metrics that are broadly useful, at least to see overall trends. Pretty much every site cares about unique visitors, number of page views, and bounce rate. Beyond that, though, there is a huge amount of variation around what metrics you should care about, and that depends far more on what challenges your business or organization currently has. Are you looking for new customers? Are you trying to increase engagement with your current customers? Are you trying to reduce costs? Are you trying to create something new? Are you trying to increase satisfaction? There's going to be some way to measure your effectiveness -- and what you pay attention to is generally where you'll get results.

Introducing the Freelock Vision Plan

We're applying this same concept to our own business, Basic web analyticswith this key question: How do we help our clients drive better results through their websites? And our answer is to develop automated tools that cover the basic needs across all sites, look at a single topic each month across our entire portfolio, and leave a little time to  focus on what makes your site unique.

We start with a site assessment, asking about your goals for the site and your organizational challenges, and then go through your site and rate it across some key areas: Effectiveness, Security, Performance, Scalability, Maintainability, Accessibility, Ease of Change, and Expected Lifespan. As part of this assessment, we uncover parts of your site that might cause problems when doing any work on the site, as well as providing independent feedback about what might be improved to make the site better accomplish its goals.

Once we have your site fully set up with analytics and an understanding of your key metrics, our vision plan delivers 3 things each month:

  1. Month-over-month changes on your key metrics, with some analysis to identify what looks to be working, and what's not so much.
  2. Topic of the month, and how your site stacks up. Example topics include: mobile performance, website accessibility, data security, SEO/SEM best practices.
  3. Recommendations. Each month we look for 2 or 3 improvements to make on your site, which we feel will help the site accomplish its goals, lower potential risks, or reduce costs.

Paying attention to your site can make a huge difference in how well it works for you. And we would love to partner with you to help drive results! We're currently pricing this at $799/month.

Introductory Special - $2400 value!

To kick off our new offering, we are offering 3 months of our Vision plan free, for the first 5 customers who sign up with a Site Assessment before October 15, 2017! Contact us if you would like to be one of them!

Aug 05 2017
Aug 05

The corners of Drupal 8 that aren't there are quickly dwindling, but there are still some that need to get worked out. While upgrading our internal issue tracker, we hit a new one -- getting a group context set via a URL alias, and generally keeping posts within a group.

In Drupal 6 and 7, we made extensive use of Organic Groups, Spaces, and Purl to make it easy to go straight to a project by URL. For example, with a project site of bugs.freelock.com, to get to the group for the "acme" client, we might have a URL of https://bugs.freelock.com/acme. Purl is a module that can take a specific part of a Drupal URL -- a subdomain, a query parameter, or the first part of the path -- and convert that part into a group context. It then rewrites the URL internally to a normal Drupal path where the rest of the system can do whatever needs to be done, and return a result. Finally, it inserts the group context back into links that Drupal generates. The end result? When you're in https://bugs.freelock.com/acme, views can be filtered to show only content inside the group, entity references can be filtered by the group context, new content gets automatically added to the group, you basically stay in the group until you follow a link out of it.

With our new PM site, we decided to give the new Group module a try, with its attempt to provide a simpler architecture than the previous Organic Group module. As best we can tell, Og is useful if you want to have different roles and permissions within specific groups, whereas with Group individual groups can only vary by creating a new type of group.

Challenges getting Group and Purl working in D8

We managed to get Group up and running, and all our old content migrated easily enough. But... then we ran into issue#2717981: Group ID unavailable when editing the referenced entity, which is one of the 3 blockers for a full release of the module. And... no progress on this for months.

I then checked out Purl to see where it stood -- which turned out to have an 8.x version started but not functional, with some progress made in a gitub repository. There was also some good starting documentation for implementing plugins for Purl. Given our recent under-the-hood Drupal 8 experience, we dove into the code and started working through issues. It probably ended up taking 5 or 6 hours to get it all working, but along the way we ended up fixing 7 different issues in Purl, in addition to creating a new Group Purl module and posting it to Drupal.

So all in all, I think we've arrived at a pretty slick solution! Here's how to get it up and running on your Group site...


These steps are sure to improve, but for the time being, there are a few hurdles to get the working code.

Two things in particular - getting a version of Purl with all the current patches, and installing the Purl fields on the various types. Here's the gist:

  1. On a working Drupal 8 site, download group and group_purl as usual from Drupal.org
  2. Until there is an Alpha release of Purl on Drupal.org, check to see if the ActiveLamp github repo has incorporated the pull requests -- if not, you can get a copy from the Freelock github repo. In either case, use the "master" branch.
  3. Enable Group, Purl, and Group Purl modules.
  4. At the current writing, you may get a stack trace/fatal error. To fix, you might need to use Drush -- running "drush entity-updates" will install the Purl fields appropriately that are the cause of this error.

Now, on to configuration...

Configuring Purl for Group module

At the moment, there are several "hidden" configurations you'll need to check out to get all of this working, so there's several basic steps to get to a working solution. The order you do these in don't particularly matter, once you have a group type set up you just need to get these other elements in place to get a working result:

1. Install the Group Purl provider plugin.

Under Configuration -> Search and Metadata -> Persistant Urls, Add a Purl Provider.

Group Purl provider

Give it a name, select "Group Provider" for the provider, and "Group Prefix" for the method. Then save.

The Group Purl provider loads the matched group into a Drupal context, which may then be used by views, blocks, etc.

The Group Prefix method creates a list of all groups using their currently configured URL alias, and matches when that alias matches the first part of the visited path. It strips off that first part, and then creates a new subrequest that it passes into the Drupal kernel for the actual processing. It also provides a URL helper that re-injects it into the start of URLs generated by the Url service.

At the moment, this prefix method does not match the actual group -- it only rewrites children paths. This allows you to access the Group entity itself, without causing redirect loops or taking you to a different page.

2. Create a group type and configure content types to be part of the group

Under Groups -> Group Type, add and configure a group type. Then, after you have the basic group type set up, you need to configure which content types go into the group. After you've saved the group type, go to its "Content" tab, which you can reach using the "Set available content" operation:

Group operations menu

When planning out your site, I would recommend having certain content types set to be in groups, and other content types outside groups. That's what you can configure on the "Content" page for a group -- which content types may be part of the group.

What you actually configure here is a special many-to-many relationship between content and groups. This relationship is an entity type called "group content", and is the fundamental way everything gets associated with a group. Group module provides a built-in Group Content configuration for users -- on this tab, you create the relationship for each content type you want in the group. Click "Install" for a content type, and then configure the nature of the relationship:

Configure a group content type

... Unless you have more sophisticated needs, I recommend setting Group Cardinality to 1 -- content of this type may only appear in a single group -- and uncheck the "use 2-step wizard" option.

3. Set the "Purl context" option on each content type that goes into a group

Under Structure -> Content Types, edit each content type you've configured as "group content". On the main edit page, go to the Purl tab and check the box to "Keep context of the node":

Purl content type settings

... if this option is set, then when you visit a node inside a group path, Purl keeps the group path active. Using our previous example, a node might be at https://bugs.freelock.com/group/2/node/234 -- this option keeps the URL the same.

If you uncheck this box, Purl will redirect out of the group when landing on a node of this type. So if node/5 is a basic page, and basic pages have this box unchecked, users visiting https://bugs.freelock.com/group/2/node/5 will get redirected automatically to https://bugs.freelock.com/node/5, and lose the group context.

Note: The Group Prefix method is aware of URL aliases -- if you set a pathauto alias pattern for group entities, the path prefix will use the alias instead of /group/xxx.

4. Configure views to use Group ID from Purl as a default argument.

The next bit to get in place are views -- views blocks, views pages, and entityreference views. There are two crucial parts to making this work: setting up the relationship to the Group Content, and providing the Purl default argument.

Starting with a view of content, under the Advanced section, click Add next to the Relationships header, and add a relationship to "Group content for content". If this view only uses a single content type, select it under "Filter by plugin".

Next, add a Contextual Filter of "Parent Group".

Finally, under "When the filter value is not available", select "Provide Default Value", and select "Group ID from Purl":

Group ID from Purl default argument

In the validation section you can define what to do with this view if you're outside a group -- generally you would either hide it (to make blocks only visible inside a group, for example) or show all values.

If you make an entityreference view, this becomes a great way to limit the list of available reference fields to other items in the group.

5. Menu links

The last thing to address are links to get in and out of groups. Purl adds a context to each menu -- you can use this to define a distinct menu for a particular group. More scalable than that is relying on the URL rewriting.

When you specify a menu link inside a view, Purl will rewrite the URL for that link, pretty much always. If you want to be able to control a whether a particular menu item keeps you in the group or takes it out, you need to add the menu link directly.

Maintain or strip Purl context in menu link

So on custom menu links, if you "maintain context", the links will get rewritten to include the group path. If you "Strip PURL context", the links will not get rewritten, and will take you out of the group. Or you can make a link go to a specific group right from this dropdown.

That's it! Those are the building blocks you need to make groups workable in Drupal 8. Leave any feedback below, and if we can help with your Group-based site, drop us a line!

Jul 24 2017
Jul 24

DevOps is the union of development, operations, and quality assurance -- but it's really the other way around. You start with the quality -- developing tests to ensure that things that have broken in the past don't break in the future, making sure the production environment is in a known, fully reproducible state, and setting protections in place so you can roll back to this state if anything goes wrong. Next comes operations automation -- building out operational tools that ensure snapshots are happening, deployments happen in an entirely consistent way, and environments are monitored carefully for performance degradation as well as simple uptime. Finally, you're in a safe place where you can develop new functionality without risking major breakage.

This is the philosophy we employed while building out our DevOps practice. We employ these principles to developing the very process we use to develop new sites as well as additions to existing sites -- as we flesh out one part of our pipeline, the next one becomes easier to get in place.

DevOps as a project

Our best example of a DevOps project is what we have developed internally to manage dozens of production Drupal and WordPress websites. At the center of it all is a chat bot we've written, named Watney. Watney gets input from all sorts of internal systems, including git repositories, configuration management tools, continuous integration tools, project management systems, and developers. It then enables/disabled pipelines and kicks off jobs as needed (and authorized), updates issues, logs activities, and sends notifications as jobs are completed.

Each day Watney triggers a job that checks the environment of every production website we manage. It alerts us of any changes that have not been properly tracked in configuration or code management. It also alerts us to any sites that have changes staged but not yet deployed. As new work is done, Watney kicks off Behavior-Driven Design tests automatically, and reports the results. If the tests pass, the site is automatically marked as "Ok to stage," otherwise it requires attention from a developer. When sites are staged, they are automatically run through visual regression testing, comparing stage sites with production. These often fail due to fresh content on production, different slides visible in carousels, different ads loading -- so we allow a person to approve a test run even if the number of pixels different exceeds the base threshold.

For the actual deployment, Watney assembles release notes from git commits, notes from the developers, cases/issues/user stories included in the release. It takes a fresh snapshot of the database, verifies that all tests have passed or been approved, tags and then rolls out the code. The next day, Watney creates a fresh sanitized copy of the production database to put on stage, resets the pipeline, and bumps the version number

In the process of fleshing out this pipeline, we had to address a lot of requirements and concerns, many of them particular issues around Drupal and WordPress sites:

  • How to prevent test sites that integrate with 3rd party APIs from polluting external production databases from test data
  • How to make sure we only work with sanitized data
  • For WordPress, how to consistently change domain names so the site functions on test and stage environments
  • How to deploy configuration changes for Drupal 8's new configuration management system
  • How to enable development-only modules on dev sites, and disable for production, and vice-versa
  • How to generate privileged logins while maintaining security around the sites
  • How to turn pipelines off and on when we reached some hard limits on scaling a pipeline across dozens of sites
  • How to improve pipeline performance to be able to push out a fully tested security update to all maintained sites within 4 hours
  • How to integrate our pipeline for a variety of hosting environments along with dedicated Drupal hosts, including Acquia, Pantheon, Amazon AWS, Microsoft Azure, Google Cloud Engine, Digital Ocean, and self-hosting

Other Operational systems

Operations is all about systems -- processes and tools to keep things running smoothly, without interruption. Surrounding our core chat bot and deployment process are more typical operations systems -- server monitoring, performance monitoring, ticketing systems, time tracking, billing/invoicing, phone systems, email, file sharing services. At Freelock we have rolled out self-hosted open source systems across the board, along with some custom glue holding it all together.

As a result, we sometimes provide consulting services to other organizations to help them build their own DevOps practices.

If we can help your organization build out a tailored Continuous Integration pipeline, or other DevOps process, let us know!

Jul 17 2017
Jul 17

Memberships are not all the same. Some memberships last a lifetime, others last a year, or a month. Some memberships are for an individual, others for a couple, others for an entire family.

Some memberships are for a particular time period -- the 2017 season, the 2017/18 school year, Summer -- others are for a particular length of time starting when you purchase it.

And then there's the behind the scenes stuff -- the "CRM" (Customer Relationship Management) systems of reminders to renew, lists of memberships up for renewal, grace periods between when a membership technically expires and the hard cut-off date, and renewal policies -- does it start on the day you pay for the renewal, or the day your previous membership expired?

What if a member wants to upgrade or downgrade their membership in the middle of their membership term? How do you account for a cost difference between membership levels? What if a student starts part way into a season, do they get a pro-rated price? What if one parent signs a child up for one session, and a different parent registers them for a later session?

In short, membership programs are complicated things. Every membership organization has its own business policies, its own rules, its own benefits.

These are a few of the variations we've run across so far, when implementing membership plans for a client.

Memberships in Drupal 8

Drupal has long been a great platform for a membership site, largely because it is so flexible. With Drupal, memberships can be easily configured to create roles, add users to groups, provide special pricing in a commerce site, grant access to content or the ability to create content, or whatever. Figure out your membership structure and the rest is easy.

However, until Drupal 8, creating memberships with the ability to provide the variations outlined above pretty much always involved a fair amount of custom work. The relatively new Membership Entity module, combined with Membership Entity Commerce goes a long ways towards making a flexible membership system, though there are still gaps, such as offering memberships that expire on a particular date.

We recently launched our first membership site in Drupal 8. We took our experience from nearly a dozen different membership sites over the years, with the full range of variations, and built a brand new membership term and role system with these variations in mind. We are working with the new maintainer of the Membership module to get is is all going to get published on Drupal.org, but for the time being, we've posted our code on Github.

Screenshot of edit membership_term screen, with reminder notification scheduleToday this system is built around 1-year membership terms, with a 60-day renewal grace period -- during that grace period, the member still has full access, but if they renew, the renewal starts on the expiration date of the previous term. This seems to be a very common pattern among associations.

The other things that work swimmingly well are email reminders that get sent as the membership term expires. Our client wanted a sequence of 4 different messages that get sent 4 weeks before the expiration date, 7 days before the expiration, 1 week after the expiration date, and a "last chance" message 1 week before the end of the grace period. We built a new Scheduled Message module to handle this, basing the architecture on the algorithm recently added to an experimental Webform submodule. The cool thing about the way we implemented this, is that these messages get suppressed if the member renews. And we can send a "thank you for renewing" message as this is confirmed.

Membership model

The membership module we fleshed out follows a similar architecture as the Drupal 7 membership_entity module. Memberships are the main entity, and can be active or expired. Memberships may be attached to multiple user accounts, and roles can be granted or revoked based on the status of a membership.

Membership DashboardThe "active" period of a membership is a separate entity, the "membership term". Each membership can have one active term, but any number of other terms in other states -- expired, expiring, renewed, pending, etc.

Many associations have a grace period -- a period of time where membership benefits are still active, but the term is effectively expired. And so this functionality is baked into the new membership_term module, with a corresponding "expiring" state.

The bulk of the work we did was all around managing what happens when a membership term expires, and gets renewed. For the first site, we set up rules that specify:

  • When a member renews a membership_term that is "active" or "expiring", the effective dates of the new term are calculated from the expiration date of the old term. For example, if a term expires on July 20, 2017, any renewal that is done before that date or up until September 19 extends the valid membership to July 20, 2018.
  • When a new membership_term is added, the old one is set to the "renewed" state, suppressing all future email reminders associated with that term.
  • When a member renews a membership_term that is expired, the new membership term starts on the date of purchase. (This assumes the member renews after the grace period, and has lost their membership privileges.)
  • An administrator can edit the active dates, and/or the revoke date of any membership_term, and the scheduled message send dates will adjust accordingly.

Module development in Drupal 8

This was our first foray into building a complex module for Drupal 8, and I found it to be a great learning experience. It exposed us to a large part of the Drupal API, including:

  • The plugin system - Creating plugins, plugin types
  • Dependency Injection, specifying interfaces rather than objects to facilitate tests with mock objects
  • The Event system
  • Creating Services and using the container
  • The state_machine module -- we still need to port to the newer "workflow" module that has been added to core
  • The Queue API

Next up, hooking up self-registration and e-commerce.

If your organization needs a custom membership solution, let us know! We'd love to help...

Jun 08 2017
Jun 08

Lots of stuff has been changing in Drupal 8 recently. In 8.3.0, a new experimental "layout discovery" module was added to core, which conflicted with the contrib "layout plugin" module. Now in 8.3.3, the two-column and three-column layouts had their region names changed, which hid any content dropped into those regions when those layouts were used.

In the past week, we've seen a couple issues upgrading a site from 8.2.x to 8.3.2, and now another issue with 8.3.2 to 8.3.3 that seem worth a quick mention.

All of these are issues we've caught before releasing to production, thanks to our visual regression testing.

Layout Plugin -> Layout Discovery

By now, this is quite an old one, and modules like Panels and Display Suite now do this for you in an upgrade script -- they disable the contributed Layout Plugin and enable the Layout Discovery module in core.

There are several different contributed modules that need to get updated to work with 8.3, but all the ones we've seen now have a major version dedicated to 8.3+. So not much of an issue anymore.

Twocol and Twocol_bricks layouts broken

In today's 8.3.3 release, a fixed issue in the release log with the innocuous title "Review layout CSS and markup" made a bunch of our content disappear. Reviewing the patch, it becomes obvious why -- the names of several regions in the layouts provided by layout discovery have changed. In general, regions with "left" in their name got changed to "first" and "right" to "second".

A quick way to find all of these is to export your configuration, and inside the config directory, use grep "id: layout_two" * and grep "id: layout_three" to find all the places using these layouts. Then go edit the displays to put your content back, export your updated config and deploy.

\Drupal\core\Template\TwigExtension has new arguments

On a site we just took over maintenance, the previous developers had declared a new custom Twig extension for some custom functionality, and they had extended \Drupal\core\Template\TwigExtension to do so. This pattern was used in core until 8.3.0, but this base class now has some services injected into its constructor. Which threw fatal PHP errors, leaving a blank site.

The fix? Change the custom class to extend \Twig_Extension directly. There is no need to extend the Drupal wrapper. One thing that caught us is the underscore -- you need to add that as well as the backslash to get the right class...

drupal_block() missing second argument

In the Twig Tweak module there's a useful {{ drupal_block('block-id') }} function. However, with the release of 8.3.0, blocks have a "status" of either enabled or disabled, and disabled blocks are the equivalent of unpublished. And so this function fails to load a block unless it's enabled in a region.

When you update from 8.2 to 8.3, you may have noticed that there's no longer a "Disabled" region for blocks -- blocks all end up in an actual block region, but are set to disabled status.

Twig Tweak has added a second parameter, "use access checks", to allow bypassing the disabled status -- this needs to get set to False to get public blocks to show up. I'm not sure whether you can actually use the block visibility rules for the block aside from this disabled/enabled status, but this is what we needed to set in the templates to make the blocks re-appear.

Upgrade in safety

Upgrades are a tricky thing, and it's not unusual for small things like this to crop up with any update. This is why we highly recommend having a testing pipeline set up to test upgrades, and not just applying upgrades on a production site. Check out our Drupal Protection plan if you'd like assistance testing all updates before pushing them live!

Jun 05 2017
Jun 05

As of today, the Drupal Matrix API module now supports sending messages to a room via Rules. Now you can automatically configure notifications to Matrix rooms without touching any code!

This is useful if you want to get notified in a Matrix room of some event on your website, such as a new comment, a user registration, updated content, etc.

Rules is still in Alpha, and has some UI quirks, but it works fine.

To set up a notification of new content in a Matrix Room:

1. Create a Matrix account

First, your site needs a Matrix account. Easiest is to set up an account using Riot.im for your site. You can also set up a site on your own homeserver using normal account creation processes.

2. Install the necessary modules

Next, install these modules on your site, using normal Drupal module installation steps:

  • matrix_api
  • rules
  • typed_data
  • token

3. Configure the Matrix API Settings

Then, go to Administration -> Configuration -> Web services -> Matrix API settings, and configure your account. Specify a custom homeserver if the account is not on matrix.org, and provide either an authentication token or a MX id and password.

If the Matrix API does not successfully authenticate, you will get an error message.

4. Configure the notification rulesConfigure a Matrix notify action

  1. Go to Administration -> Configuration -> Workflow -> Rules
  2. Add a new "Reaction Rule"
  3. Provide a label for the rule, and choose the event you want to receive a notification for -- e.g. "After saving new content"
  4. Under "Actions", click "Add Action" and fill out the following:
  • Room Alias: use any alias for the room you want to post this message in
  • Message: Enter the message to post. Tokens will be replaced, and you can pass a single Data entity into the token system to use for replacements.
  • Data: "Switch to data selection" and then you can select the entity that triggered the event (or any other entities you might add in other conditions). Whatever entity you provide here will be used for token replacement, with the first part of the token the entity type - e.g. "node", "user", "comment".
  1. Save the action.
  2. IMPORTANT! Save the rule. (If you don't do this, the action won't fire!)
  3. Clear caches -- the rule will not fire until after a cache rebuild.

That's it! You now have a rule that sends a Matrix message to a room when content is added to the site.


The site attempts to join the room before sending a message. If the room does not exist, or the account does not have permission to enter the room, you will get an error.

You can invite the site's account to a restricted room, and it will join when it attempts to send the first message to that room.

The Drupal Matrix API module does not currently support E2E encryption.

What's next?

The Rules UI still seems bare-bones -- I did not find a way to create a long-text field, or display the Token browser. You can go to the Token module help page to pull up a token browser for all entities on the site, and find the right tokens to use. As Rules develops, we will plan to provide better support for this.

We also plan to provide a message_notify plugin, so that users of the Message family of modules can easily send Matrix messages.

Can we help with your Matrix or Drupal integration? Contact us, or leave a comment below!

May 17 2017
May 17

As I write, we're in the midst of a big Ransomware attack. Millions of computers have been infected, with their data encrypted, held ransom pending an extortion payment or deleted. Supposedly.

There's a very simple way to avoid the catastrophe of losing everything due to an attack like this. And it's nothing new, it's something we've known to do all along: make good backups of everything, all the time.

Backups are only part of the puzzle, though. Because backups don't matter. Being able to recover from whatever risk you face is what matters -- backups just help you do that, if you have the right ones available. It doesn't help if your backups are all infected, encrypted, deleted... or unreachable because your provider had an outage.

What you really need is a "Disaster Recovery plan" that includes a risk matrix -- a summary of what risks you face, and steps to recover if one of these risks comes to pass.

But this attack affected computers, not websites!

We see hacked websites all the time. In many cases, the site owner isn't even aware they have been hacked! So what difference does it make? We've seen hacked websites that can be used to attack other sites in "Denial of Service" attacks, mine bitcoins, and game search engines. Even though you may not be a target, your website can be used to attack other targets -- your poor security opens the doors for cyber attacks against others.

Tomorrow? Might be a different story. I think we may have crossed some threshold where people are starting to realize that they can't just neglect keeping their computers up-to-date and not risk losing all of their data. Websites are next. We've seen so many out-of-date sites just asking for an attacker to hack... It's only a matter of time before there's a wave of Ransomware attacks on web hosts.

If your site gets encrypted by ransomware tomorrow, what would you do?

Do you have backups stored somewhere that won't get encrypted? Do you have a plan for getting back online? Would you have to start over? Would you pay the ransom? (would that even work?)

Do you have everything actually backed up? WordPress, Joomla, and Drupal sites (among others) all use databases to store the stuff you really care about -- your content. Creating a backup of your database is entirely different than copying over the files in your hosting account.

One backup isn't enough. One backup system isn't enough. If you can't pull up a copy of your site from a month ago, you don't have your risks covered.

Covering your bases

If you analyze the risks you face, most companies can provide decent coverage for most website risks with the following basic approach:

  • Keep your site code in version control, with other copies stored elsewhere -- this makes it easy to spin up a replacement as well as detect changed code.
  • Keep your site fully up-to-date with all updates available. Even if it's not a security update, it's much better to be caught up if a new security update comes out -- your site is far less likely to break.
  • Take a nightly snapshot of your database, and store on the same server for a few days to a week. This generally involves a script you need to set up yourself.
  • Take a nightly snapshot of the entire server, and keep several days worth of snapshots at the same datacenter as your web server. If your server is a VPS, most VPS hosts provide an automated backup system to do this -- make sure you're using it.
  • Once a day, back up your site, assets, and database dumps to some entirely different service. This protects you from risk associated with the primary host -- if they evaporate entirely, you don't lose everything.
  • Use a historical backup rotation system that keeps selected snapshots around for weeks, months, even a year or more. You would be surprised how often this proves useful.

Ok, John. Got it. You really expect me to do all of that? I don't know how to do all of that! I don't have time to do all of that!

Introducing the Freelock Protection Plan

Don't worry, this is what we do day in, day out -- keep our clients' sites safe, secure, and recoverable from virtually any risk. We make sure your WordPress or Drupal site is fully up to date. We make sure you have multiple, redundant, historical backups of your site code, assets, and database. We check your site configuration and code integrity every day to detect changes that might indicate you've been hacked. We apply critical security updates immediately, less critical security updates within a day or two, and all updates every month -- all while running tests to make sure nothing breaks as a result of an update.

And if something does happen, we've got you covered. We can promptly recover your site to any of the snapshot points we have -- typically a choice of 12 - 15 different dates going back up to 16 months. Full recovery is included in the plan -- we will get you back up and running.

In many cases, a client calls us up with accidental deletion of some data. With historical backups, we can usually accommodate these requests -- while the cost of doing partial recovery is not covered by the plan, we usually have the data available so it's possible to do!

What else should I think about?

Whenever you take a serious look at the risks you face, there are bunch of questions to consider. When considering a maintenance plan, here are the crucial ones to discuss with your provider:

  • How long can I live with my website being unavailable?
  • How much data can I afford to lose?
  • How much of a target is my website?
  • Do I care more about my website being offline and unavailable, or about potentially losing data?

We think our protection plan strikes a nice balance of affordability with reasonable answers to the above:

  • With major attacks that might affect all our customers, it could take us days to get everybody back online. We haven't had any recoveries take longer than 1 business day to resolve, and have done many recoveries after hours, the same day as we were brought in to recover.
  • Our backup strategy does risk up to 2 days of data -- if an off-site backup runs before a database dump. With more planning and budget, we can set up replication servers and hourly backups to reduce that time window to a much smaller amount.
  • If you're a target -- if you have secret information, store credit card data or health care data, you have a lot more work to do to fully lock down the environment. We have many e-commerce customers, but we make sure that the credit card data does not get stored on their servers and thus avoid becomng a target.
  • We think both availability and not losing data is important. We err on the side of not losing data -- you might see occasional blips on the site while we deploy updates, and a few minutes of downtime here and there to prevent your site from being vulnerable or losing data seems like a reasonable tradeoff to us. But whenever a site is down, you'll see us frantically acting to get it back up, whatever time it is...

Please. If our plan is not for you, please at least take the time to make sure your site is secure, backed up, and not making the Internet a worse place by being owned by an attacker. If you come asking us for help after you've been hacked, if you don't have a backup, we might not be able to help. It's far less expensive to prevent problems like this than to try to recover after the fact.

You can read more about our Drupal Protection plan or our WordPress Protection plan, and feel free to comment below or contact us if we can help in any way!

Apr 07 2017
Apr 07

Extreme irony: the person most responsible for making Drupal a mature, stable, long-term platform has been ejected from a leadership role for reasons that are not entirely clear. As a result, the Drupal community itself is going through a painful crisis.

The heart of the matter is painfully unclear: Is Larry "Crell" Garfield being ejected merely for "thought crimes?" Or is there actual evidence of some sort of abuse, that the extremely tolerant inner circle of the Drupal Association and leadership could not tolerate?

One thing is for sure: the way Dries Buytaert and the Drupal Association handled this situation has drawn the very negative attention they were trying to avoid by giving Crell the boot.

The story

My wife asked me what the short version of the story was. Here it is: Big contributor gets accused of a sexually deviant lifestyle, with no evidence presented to the public that he has brought this anywhere near his work in Drupal. Project leader terminates his role. Community is up in arms, canceling their Drupal.org accounts, protesting the dismissal, expressing lack of confidence in the Drupal Association and project leader.

But that simple story leaves out much of the complexity and background that makes this such a big deal. There's a lot more here:

  • Crell was the person who persuaded everybody to move Drupal 8 core to use Symfony, and convert to a fully object-oriented stack at its core. This is the single biggest change in Drupal, pretty much ever -- and I think it has made Drupal one of the best long term platforms out there.
  • Crell is not the first long-term technical leader ousted from the project this year -- "chx" was ejected just a couple months ago after a long history of abrasive communications but technical brilliance.
  • The lifestyle in question, for lack of a better word, is not just a BDSM subculture practiced by consenting adults, but one that promotes a misogynist viewpoint, overtly claims that females are inferior to males
  • Plenty of women in the Drupal community have come to Crell's defense, stating they have never felt any misogyny or discrimination from him in their interactions
  • The Drupal community has long taken pride in a spirit of tolerance and inclusivity, to an extreme I haven't witnessed elsewhere, so rejecting somebody due to their lifestyle and sexual preferences seems particularly ironic (and explains why there is so much outrage)
  • The explanations given for the ejection hint that there are incidents that are not in the public discussion that may indicate that Crell's beliefs may undermine the very tolerance and inclusivity the Drupal community prides itself on, and now that they are known, are entirely unacceptable to be associated with the Drupal community

There is so much here to delve into. I'm going to briefly touch on three: the technical contributions, the community ejections, and the community going forward.

Technical maturity

6 years ago I wrote one of the most popular posts on this site: Top 6 reasons Drupal really sucks -- Developer Edition. Of these 6 reasons, half are addressed by the architectural changes introduced in Drupal 8 -- especially the biggest, most important thing that has always made Drupal suck: the upgrade path.

Doing a major upgrade of Drupal has always been tremendously painful, because over and over again Drupal has broken backwards compatibility between major versions. Drupal 5 is quite a bit different than Drupal 6. Drupal 7 is drastically different, with entirely different data models. And Drupal 8 is really an entirely different application that resembles previous versions of Drupal -- but no code you wrote before is relevant anymore.

This policy has had a lot of consequences for Drupal. Major upgrades have been major projects, leading to maintenance headaches, site owner dissatisfaction, splits in the community and codebase, and much more.

It has also forged technical excellence, allowed the community to largely abandon messy, dangerous patterns in favor of solid architectural patterns, led to test-driven development and an extremely high bar for quality, security, and usability -- and with Drupal 8, we finally have a platform that will be maintainable and relevant for at least the next decade, with no more major upgrade projects ahead.

The fundamental change that Crell single-handedly introduced into Drupal 8 makes Drupal major versions pretty much irrelevant going forward.

That is a very big deal.

I don't think this has fully sunk in to the broader Drupal community. There are discussions around changing the version numbering scheme for contributed modules -- yet many seem to miss the point that thanks to object orientation and introspection, it's going to be possible for the same contributed module to support multiple major versions of Drupal. For example, the same version of an Address Field module could be made to work with Drupal 8, Drupal 9, and Drupal 10 -- at the same time.

Suddenly major Drupal versions no longer matter (once people have moved beyond Drupal 7, that is...), but minor versions do. For example, Drupal Commerce already depends on new APIs introduced in Drupal 8.2, and won't even run on 8.1.

So this means contrib modules are going to end up much more like libraries in most other languages and tool sets -- as a contrib coder, you will probably end up specifying minimum and maximum versions of core and other contributed modules that your module is currently compatible with. And almost certainly, after Druapl 9 comes out, you'll be able to be compatible with 8 and 9 at the same time.

For site owners, this is going to be an even bigger deal -- the fact that a major upgrade is no longer a big deal. They simply happen as part of routine maintenance, a module at a time, until you have no more modules incompatible with Drupal 9, and you update core. (I see Dries just blogged about this very improvement)

This is why I now consider Drupal technically mature. And I give Crell full credit for persuading the entire Drupal community to make this happen, and getting the first prototype working to show it was possible.

Community Ejections

This is the second major contributor to be ejected from the Drupal community in the past few months. Karoly (chx) Négyesi was removed last November, primarily due to his communication style. I met chx at a code sprint a long time ago, and found him to have a sharp technical mind, little patience for people who hadn't arrived at the same technical conclusions, and overall a sarcastic, funny guy.

In many ways, like Linus Torvalds, the creator of Linux, who has similar controversy brewing.

If you look around the issue queues for Drupal core, chx comes up again and again with code, solid patchs, strong arguments for particular technical solutions, and often rage-quits and other drama.

The greater Drupal community seems to pride itself on diversity, inclusiveness, and tolerance for a huge range of people, beliefs, and proclivities. However, there seems to be a growing intolerance for rudeness, misogyny, bigotry of any kind. Is this really a bad thing?

Peace, love and acceptance of everything and everyone is great until that runs up against philosophies opposed to that stance. And it's hard to reconcile the two.

There is no easy answer here. The community has grown to the point where some people are going to be unhappy, no matter what. As much as Crell and chx have had great contributions to the Drupal project as a whole, the community has matured to the point where it has had to make some tough decisions, where the ground rules end up murky and need a judgment call.

It's no longer black and white.

The Future

The future is here. Drupal is a better platform than it ever has been. The Drupal community has had to deal with some hard issues that don't have a clear correct answer. These are difficult milestones, but they are part of growing up.

I think the Drupal Association and the project leader did the best they could, and reached the only conclusion they could have. It's unfortunate that the story unfolded the way it did -- but does anybody want their platform associated with a misogynistic fringe philosophy? The leadership of a project reflects very directly on the project, and if the leadership did not eject Crell, would Drupal become known as the Gor platform? It would be irresponsible for any organization to let a highly visible leader remain in a leadership position when something like this comes to light. While I'm sad to see such a valuable contributor forced out of the project, I think letting him remain in such a visible role would not help the project broaden its reach, and may well make large numbers of potential users, developers, community members skip Drupal when considering a CMS.

It's time to put this issue behind us, and move on, older and wiser. I wish Crell well, and thank him whole-heartedly for his massive contributions to the project.

Mar 01 2017
Mar 01

Yesterday Amazon Web Services (AWS) had a major outage in their US-East datacenter, in Virgina. It made all sorts of national news, largely because it affected some major online services.

At Freelock, we were largely unaffected. Part of this was that while we are pretty heavy users of AWS, we use their Oregon datacenter. And we generally don't rely on outside "software as a service" systems -- unlike most other shops, we're pretty obstinately open source, and would rather deal with the inconvenience of running our own systems than the inconvenience of having our entire business at the mercy of somebody else's infrastructure.

We were working happily away without interruption all day yesterday, and across everything we run, only one system was affected at all -- our "Continuous Integration" system, Concourse CI. And even that was working fine -- it's just that one container image in our pipeline comes from the Docker Hub, which was down. And that container image is one we publish on the Docker hub for others to use, so it's still within our control -- if we had a big deployment to do yesterday, we could have easily changed our pipeline to get the image from our own private Docker registry, where all the other images come from.

Feel free to contact us here, emailing us directly at [email protected] or call us at (206) 577-0540!

But what if the AWS Oregon data center had gone down instead?

That would have been a much different story for us, and we likely would not have gotten much actual client work done yesterday. We would have about 1/3 of our production websites offline, and most of our larger clients' servers down. No email, no access to our LAN backups or secondary backups from other cloud providers. Our central git server and Docker registry would have been offline. Our secondary DNS would have gone down -- but our primary would still be up.

If the outage was prolonged, here's roughly what we would have needed to do to restore service:

  • Spin up a new secondary DNS server, and update our DNS registry to add it
  • Deploy a replacement Docker registry, rebuild our private images, and publish them in the registry
  • Deploy replacement web servers, perhaps at Google Cloud Engine (GCE), or in this case perhaps just a different AWS datacenter
  • Recover all production websites from our secondary backup system
  • Deploy a replacement mail server

Most of that would be pretty straightforward, though looking back at the nature of the Virginia datacenter outage, it highlights the need to complete some of our infrastructure backlog tasks. In particular, our secondary backup system is currently at AWS, and might have been affected by this outage. I was actually looking at spinning up another secondary backup system at GCE, specifically to back up the AWS servers, for exactly this type of risk.

Otherwise, the mail server would be the hardest thing to recover from, for two main reasons:

  • It takes time to establish a new mail server IP address and not have your email all rejected as Spam
  • We last deployed that system in 2014, about a year before we switched over most of our infrastructure to Docker containers -- much of the configuration is centrally managed by our configuration management system (Salt) but there is some custom configuration that may take some time to recover.

Everything else we could recover pretty quickly and easily, largely because nearly all of our infrastructure is centrally managed from a server sitting on our LAN (and that server has all of its configuration backed up to AWS).

What if our main LAN server went down?

If that happened, we would largely lose access to our development sites and our project management systems. Our developers could still work by pulling sites down locally and working on their own computers. Our chat system (Matrix) would still be up, along with our "newer" project management tool (Taiga) because we host those both out at Digital Ocean.

What if an outage lasted longer than a half-day?

We could move everything we have at any one cloud provider to another cloud provider in a matter of a few hours, and aside from needing to get a redundant secondary backup in place, we can do it without any access whatsoever to the old provider.

That's because the level of service we outsource is commodity infrastructure. We can spin up a replacement virtual server at any of a thousand competing cloud providers and be fully back in business in less than a day, with the complete failure of any single location. And that's because we've thought about the risks we face, and developed a "Disaster Recovery plan" for our internal systems. And... that's because as a rule we don't rely on "Software as a Service" applications that are out of our control.

Own your applications, don't rent

If there's a single big takeaway from a massive outage like this, it's the risk of renting the applications you use, and not owning them. For just about every SaaS out there you may want to use, there's a free, open source alternative you can install and run yourself (or hire someone like us to run it for you).

Instead of We use Gmail Our own self-hosted Postfix, Dovecot, Roundcube server Github Our own Gitolite server Docker Hub Our own private Docker registry TravisCI Our own Concourse.ci Jira, hosted PM tools Our own Atrium (Drupal) PM system Trello Our own Taiga install Slack, HipChat Our own self-hosted Matrix server Wix, Weebly, Squarespace, BigCommerce, other SaaS content management systems Drupal, Drupal Commerce/Ubercart, WordPress SalesForce, Zoho Our own Drupal-based CRM Office 365, Google Docs LibreOffice, with Collabora shared editing on NextCloud Dropbox, Box.com Our own NextCloud install Freshbooks Our own LedgerSMB install

There are some good reasons to use Saas applications:

  • Much lower cost than running yourself
  • Professionals presumably running their own infrastructure better than you can
  • "Best of breed" user experiences, companies trying to make systems that are easy to use

But in my opinion, these are all entirely negated by the risks associated with using them:

  • Vendor Lock-in -- you cannot take a proprietary SaaS system and run it yourself if you don't like the way you're treated by the vendor
  • Switching costs -- Once you start using a system, you cannot easily switch to a different one without losing data, time/cost involved in making the change, and a new learning curve for the new application
  • Dependence on the vendor -- if it's a critical business system, by using a SaaS application you now are directly dependent on the vendor. If they choose to go a different direction and change the way the app works, you're stuck. If they sell to a larger company, they might get shut down. If they go out of business, you might have to scramble to switch. If they don't do their backups right, you may lose data -- and you may not have any opportunity to check that they are doing things right in the first place.

The only place it can make sense to use SaaS services is when you have bigger risks and a short time frame to make it work -- you can likely get there faster and cheaper with SaaS, but you should pick vendors that will let you easily export your data, and preferably have open source alternatives you can jump to when you're ready to start stabilizing your company and mitigating risks.

Open Source outages

AWS isn't the only place to suffer severe outages recently. Two open source-based services had similar outages, one with some data loss.

Matrix.org outage

We use Matrix as our main chat system, primarily for communications across our team but also to interact with the broader community, clients, and soon, potential clients. The main Matrix.org server had a major 13-hour outage that brought down much of the service due to some disk space issues. Matrix.org has gone down a bunch of times.

But the distributed nature of Matrix means that even though we use it heavily, we didn't notice any outage at all. Our own server was entirely unaffected, and other than less communication from people who use matrix.org itself, we didn't notice a thing.

Gitlab data recovery issue

A month ago, Gitlab, an open source competitor to Github, had a major meltdown. They were apparently hit with a spam attack, possibly exacerbated by some other data deletion operations, and the service locked up. The whole gory story is worth a read, but in short there was some human error in restoring the wrong backup, and 5 different backup/redundency systems all were not operating correctly, so they ended up losing 6 hours of data.

The point is, stuff like this happens all the time in technology. Things fail. But as an open source company, you can run your own Gitlab server, and everybody who did was completely unaffected by Gitlab.com's outage. (And git being a "distributed system", the core code that people store in gitlab would always have multiple copies -- the data lost was really metadata -- issues, comments, pull requests, etc).

Freelock data recovery issue

Just last week, we had a minor incident with a similar "uh-oh" pit in the stomach. I was cleaning out some old email mailboxes and deleted our main sales mailbox entirely.

That same day, I noticed that our main backup system had not successfully backed up our mail server for a year -- it was getting stuck on a particular directory and failing to complete. I excluded that directory and let it run... but too late for the mailbox I had already deleted!

So there I was, thinking "crap! I just deleted it, and lost a year's worth of data!" I restored the year-old backup... and then went to our next backup system, nightly snapshots. That snapshot was fine. I mounted it and recovered the data from the previous night. Mail started delivering again, and the two legitimate mails that had been unable to deliver came through and all was well.

Outages are a basic fact of life

Freelock is an open source company through and through, and yesterday's massive outage illustrates a big reason why: instead of taking an "Internet Snow Day" we could continue working entirely as usual. And while there's always improvements to be made, any business that cares about staying in business needs to consider what risks they face with their IT systems, websites, communications systems, and project management tools, and have a plan for what to do if something major fails.

Feel free to contact us here, email us directly at [email protected] or call us at (206) 577-0540!

Feb 09 2017
Feb 09

[Update: It turns out the Drupal 7 site we tested had page caching disabled. With page caching enabled, the scalability of anonymous pages seems very comparable -- we're seeing between 30 and 83 requests per second on both Drupal 7 and Drupal 8 on our base server configuration, with caching enabled, varying by website.

However, page caching can be enabled on many more Drupal 8 sites than Drupal 7 -- many Drupal 7 sites cannot enable page caching due to content that varies by user -- whereas nearly every Drupal 8 site can use caching, thanks to Drupal 8's intelligent cache tagging and invalidation.]

The story

We had a client reach out about an error page they got on their site:

What? Normally we get alerted to this kind of thing. Sometimes it's a result of a "Denial of Service" attack, but we have had some stability trouble recently with a Linux kernel killing processes because it prematurely thinks it's out of memory -- when this happens we need to go into the server and restart the affected container.

Not this time -- the site loaded up and looked fine. What happened?

First, I checked our alerting system. No sign of any alerts sent, no indication of a recent change. The system does wait for 5 minutes before sending an alert, but nothing got sent.

Next, I checked our performance monitoring system, and it showed a big load spike during the time in question:

Load Spike

... and there was a similar spike in traffic. What happened, and why didn't we get alerted?

Well, we didn't get alerted because the entire event was over and done and back to normal in less than 3 minutes -- not long enough to trigger anything.

Next, I hopped on the server to see what was going on there. Looking through the Nginx error logs, I found a bunch of messages:

2017/02/09 16:47:56 [alert] 43#43: 768 worker_connections are not enough

... This was right in the affected time. A bit of log analysis later, and I had determined that all of this traffic was generated from a single IP address over the course of 2 minutes, and had resulted in over 2900 error responses. And these all came from "artillery 2.3.3 (https://artillery.io)".

What's this?

The tool

Artillery turns out to be a very cool website load testing tool. We've used jMeter and ab in the past for this kind of thing, though we haven't yet worked this into our regular routine. So I decided to check out Artillery. It's written in node.js, which is one of the technologies we use routinely here, and it seems to be very easy to get started and use. So, of course, I had to give it a quick spin.

Drupal 7 results - Nginx, PHP5.5, MariaDB - 2 requests per second

So I pointed it at a handy Drupal 7 site out of curiosity -- you can guess which one -- built up to handle a similar amount of load as our client. This is already a bit faster stack than most regular shared hosting, but starting with the example in the documentation for "artillery quick", 10 requests per second were enough to crush the server. We had to go down to 2 requests per second to have the server able to handle the traffic without starting to back up requests. At 3 per second, the latency started creeping up over 10 seconds, and any more we started getting gateway timeouts.

Drupal 8 results - Nginx, PHP7.0, MariaDB - 83 requests per second

Curious, I pointed artillery to a recent Drupal 8 site on pretty much the same infrastructure, the only other difference being PHP7. And kept throwing more and more requests at it -- the server kept turning them around in less than 3 tenths of a second, until we went all the way to 100! At 100 requests per second, we finally got some errors -- on about 0.5% of the requests (30 out of ~6000!)

To get a sustainable amount with the current relatively stock tuning we have on the server, 83 requests per second was the magic number -- at that rate, the maximum latency was still less than 0.3 seconds (which already beat the minimum latency on the D7 site at ~0.5 seconds!)

Drupal 8 is a speed demon.

I've heard people complain about Drupal 8 being bloated, slow, and unweildy. And... that turns out to be utter hogwash. Drupal 7 with some caching configuration can easily beat similarly complex WordPress sites, but it's not even in the same race as Drupal 8, handling 1/40th as much traffic on essentially the same hardware.

And, we haven't even optimized anything on the Drupal 8 configuration yet. We can add Redis for higher speed caching, a Content Delivery Network to speed up assets and allow multiple front ends, and much more.

If you're looking to have the fastest, most flexible, most powerful CMS out there, look no further than Drupal 8. And contact us if you need any help!

Dec 23 2016
Dec 23

Panacea, or disaster? Drupal 8 Configuration Management was supposed to solve all our woes when it came to dealing with deploying configuration. In many ways it's a vast improvement, but in some ways it has almost made matters worse.

Configuration Management has made it possible to store all Drupal configuration in code. This is a huge win, of course. But there are several gotchas that make it challenging to work with:

  • If you don't track individual changes, a configuration export/import is an all-or-nothing proposition. If you have people making changes in multiple environments, merging these changes takes manual effort and coordination.
  • It's easy to clobber changes made in one environment, when you synchronize changes.
  • It takes extra effort to maintain different settings in different environments.
  • Some configuration relies upon content entities that may only be present in one environment (for example, custom block placement, panels). This configuration does not deploy successfully, and can break sites if you're not careful.

The extra complexity of working with Drupal configurations might prove to be one more factor raising the bar for amateur web developers, making it useful primarily if you have other tools available to help manage it all, and a strong process that enforces best practices.


We've had a couple of issues catch us by surprise and affect sites in production, but for the most part, we've managed to avoid disaster. Here are some scenarios we see as challenges around configuration management. Most of these we've caught before they break stuff, but a couple have been "learning experiences".

Client makes a configuration change on production

Many of the larger enterprise users of Drupal may actually remove admin interfaces and lock down configuration so administrators can't make quick changes on production. That's too cumbersome for many of our clients -- one of the advantages of Drupal is how easy it is for non-programmers to make quick changes. We have clients who edit views headers or footers, or add blocks, or update panels, with the day-to-day operations of their organizations.

Then, when we go to roll out things we work on, if our dev database has gotten out of sync with production, when we deploy and import the changed configuration, we clobber all the client's work, gone, gone gone.

That does not make for a happy client.

Content does not deploy with new functionality

On a dev site, you might build out screens for new functionality, and drop in panes of content to support it. In Drupal 8, every block you create, or any other entity for that matter, gets created with a UUID. If you create a block and show it on a panel, it exports the panel configuration to code just fine -- but with the UUID of the block you created.

As it stands now, there's no way to deploy that block to production. If you try to create the content blocks on production first, they have different UUIDs, and so panels won't find them.

This one caught us on one site -- we deployed some changes to a panels page, and suddenly was greeted on production with "Block not found" errors visible right on the home page of a site!

Development modules and settings enabled on production

Deploying a new module to production has become quite easy in Drupal 8 -- turn it on on a development site, configure it, export the config, and then roll out and import the config on production, and presto, it's live. However, this also means that unless you take some extra measures, ALL of your configuration between production and development gets synchronized. Which means production sites end up with modules useful for development deployed, enabled, and configured -- consuming resources, potentially opening up attack vectors, allowing clients to muck around with stuff they should be leaving alone.

So how do we deal with these gotchas?

Strategies for managing configuration

We've largely lifted our process from managing dozens of Drupal 6 and 7 sites using features, and applied it to Drupal 8. The same basic conflicts and issues arise there -- the key difference is that with Drupal 8, ALL configuration gets exported/imported, so you need to be far more disciplined in following a process.

Here are some of our strategies.

Keep all configuration in sync with code

The challenge here is that nothing in Drupal complains loudly if you change any configuration in a site. You have to go looking for these changes.

We run a nightly check on all of our production and dev (integration) sites. If this check finds any configuration that is out of sync with the code of the currently checked out branch, it nags us in the project's chat room.

This check alerts us when a client has made a configuration change on production that we need to bring back to dev and merge into the next release. Our developers learn to commit all of their work, including configuration changes, each night, and keep the dev configurations clean.

Configuration that is stored in code can be tracked, much more easily merged, reverted, or managed -- but this takes discipline.

Create content entities on production, and pull down database

This can lead to a multi-step deployment process. For example, to set up some memberships on a new site, first we have to deploy code to support the new membership entity types and entities. Then we need to create some of the content that supports those screens on production, hidden from public view. Then we need to copy down the database to our development server, and then finally we can put the pieces together and do the final deployment manually.

It sounds to me like a lot of shops just build everything up on development and copy up the database -- the only time we allow this is for an entirely new site, otherwise our cardinal rule is the production database never gets overwritten, and all production data gets sanitized when we copy it down to development.

Use local configuration overrides, and exclusions

The D8 settings.php file comes with a commented out include to load a "settings.local.yml" file meant for environment-specific configuration. The great thing about the configuration management system is that you can override any configuration in the settings file, hard-coding it to whatever value you want in use without affecting the actual configuration that will get deployed. This is the most powerful, and most used way to manage environment-specific configurations.

Drush also supports excluding modules from the configuration when exporting or importing configuration.

Our setup

To support using configuration management effectively, you need to do a bit of setup on each site up front. After installing the site, we do the following:

  1. Edit the settings.php file to include settings.local.php
  2. Move the $database array and other environment-specific configuration into the settings.local.php file
  3. Set the $config_directories['sync'] variable to a directory inside sites/default but outside of sites/default/files, so that we can manage the site configuration in git
  4. Set .gitignore to track settings.php but ignore settings.local.php
  5. Set all the production settings for twig debugging, aggregation, etc using "drupal site:mode prod" and commit the services.yml file
  6. Set .gitignore to ignore the services.yml file -- this keeps the file in git but ignores any changes to it, because development settings change this file
  7. Add the list of modules we want to designate as "development" to an appropriate drushrc file's command-specific "skip-modules" list
  8. Export the site configuration and commit.

Supporting workflows

When you want to actually use the configuration management system, you end up having several different workflows to support different scenarios.

Handle nightly check warnings

Our Matrix bot will alert us each night to sites that have configurations that are out of sync.

If it's the production site, and we haven't deployed there in the past day, we can be pretty certain that it's a change from a customer (or possibly an attacker! Side benefit of a nightly check -- detect unexpected changes!) We export the production configuration, evaluate the nature of the changes, and if they are expected, commit and push back up to the master branch. Then, after verifying the development site is clean, merge those into the development site and import the configuration there.

If it's the dev site, we essentially do the same thing -- export the configuration, review for expected/desired changes, and commit.

If it's both sites, we export/commit both sites before attempting to merge -- this generally prevents us from clobbering any work done on development and makes the merges as simple as possible.

The longer a site goes out of sync, the harder it is to merge changes like these as people forget why a change might be made. We've found that adding these nightly checks to be a huge benefit!

Deploy configuration

Our process is pretty similar to what is written elsewhere. We export the configuration on dev, commit it, deploy the code, and import it on the target environment. Easy peasy.

In our case, we have a few extra steps that are mostly automated -- all having to do with quality checks. First we commit on a develop branch and push to our central repo. This triggers our Behat test runs, which let us know what passes/fails when testing the development site with all our "Behavior Driven Design" (BDD) tests.

Then we check out the "release" branch and merge in the develop code, and push. This triggers a deployment to our stage environment, and kicks off our visual regression testing, which highlights every pixel difference betwen stage and production.

Finally, when all that looks good, we ask our bot to deploy to production. It grabs a snapshot of the current production database, tags the release, pushes the code to the production environment, and applies the configuration.

On the next nightly check, our bot automatically does a fresh database copy from production to stage, to clean the decks and get ready for the next release.

Deploy a complex feature that depends upon content

We treat our stage site as an ephemeral testing location, routinely blown away and recreated from a fresh, sanitized copy of the production database, and the code/configuration to be rolled out in the next release. Our bot also maintains release notes, where we can add any manual steps needed for a successful deployment or to complete the job, as well as a list of commits in the release.

So when something needs to be deployed that will include content entities with unique UUIDs, it becomes a multi-release process.

To do this, we stage a release as usual with all the supporting code necessary to build up the content. Then after verifying that the changes are not "leaking" to people who shouldn't see them yet, we roll out to production. We create all the necessary supporting content, and then copy the database back down to stage, and then to development. Then we build out the remaining functionality and roll another release.

CMI is great, but watch out for dragons

All in all, we really like the new configuration management in Drupal 8, but we've found it essential to use other tools to manage the actual process, and make sure we're not destroying work along the way. Your team needs to understand how configuration management works, what it's capable of, where its shortcomings are, and how to merge and resolve conflicts. And if you don't enforce a "clean environment" policy, you will end up eventually spending more time sorting out what has changed where, in a nasty place where the wrong move will burn...

Nov 09 2016
Nov 09

Shan asks,

Hi folks- I am looking for an out of box eCommerce solution that meets the following criteria and hope to get some pointers on few solutions that's already available in the market.

There are certainly some decent self-hosted e-commerce solutions on the market, but I wouldn't really say any are "out of the box" -- all are going to require a fair amount of setup and configuration, not to mention provisioning a secure environment. A SaaS offering handles a lot of the hard work for you, and is as "out of the box" as you can get. Anything else opens you up to a lot more work to do it right, and not expose yourself to a bunch of risk.

That said, we do a lot of custom e-commerce, and our platform of choice is Drupal Commerce. The big news is, Drupal Commerce 2.0, for Drupal 8, just reached Beta last month, which means there is an supported upgrade path from here on. And it's looking like a great solution, though there's still quite a bit of work to be done.

First, to answer your questions:

  1. Visually appealing site. I have checked number of templates from themeforest/squarespace, etc but would love to see a fresh approach/look & feel beyond the traditional templates.

The flagship, launch customer for Drupal Commerce 2 is https://www.obermeyer.com/ .

Drupal has no limitations on how a site can be styled or themed. There are not that many custom themes for Drupal 8 yet available on the theme sharing sites, but Bootstrap is fully baked and ready, and we've deployed several Drupal 8 sites using a bootstrap-based custom theme. The theme system for Drupal 8 is much improved from previous versions, using the Twig template system...

Basically anything that can be done on the web can be done in Drupal. And for Commerce, you just have some more elements to theme (add-to-cart buttons, order templates, emails, etc).

  1. Hosted solution so we can expand the capabilities to meet our long term needs than a SaaS solution.

Drupal can be hosted at a huge number of hosts. For e-commerce, we would highly recommend hosting with a host that has appropriate PCI compliance, and use a dedicated virtual server. We put most of our e-commerce clients on a virtual server in Amazon AWS or Google GCE. (See our hosting guide for more...)

  1. Native App: This vertical doesn't need a mobile app yet for MVP but would love to keep that as an option and expand eventually beyond the desktop version. The platform should be scalable to build a native app eventually.

Drupal 8 has great mobile-responsiveness in its core -- it's a pretty decent experience to administer content from your cell phone. With a well-done theme, you may not need an app. If you want a native app, Drupal 8 has a decent web API for interacting with everything -- there are more and more "headless" Drupal sites that are simply a back end for a native app or an Angular/React/etc app.

Regarding scaling, Drupal 8 is blowing away other PHP-based systems on performance benchmarks, largely through some very intelligent caching. There's a new "BigPipe" module that loads "expensive" parts of the page after the rest of the page has rendered, allowing the site to feel really fast, while loading the user-specific shopping cart after the rest of the page has rendered. In addition to being extremely snappy, the caching layers greatly reduce server load, allowing it to handle a lot more traffic -- and the same caching mechanisms are used for web service calls as well as rendered HTML, and for authenticated users as well as anonymous. (Ok, not quite as well, but far better than competing systems).

  1. Product Images: Need ways to provide better viewing tools to look at the product images more clearly, true color, 3D views, drop the product to the room and experience it, etc.

Drupal 8's Media initiative provides some best-of-breed media management tools. See https://drupal-media.gitbooks.io/drupal8-guide/content/ for documentation of what's easy to do now -- slideshows, video, cropping, embedding any content in another page, etc.

For an example in action, check out https://www.obermeyer.com/catalog/womens/payton-down-jacket?v=6065 to see several different display options for photos. For True Color/3D, you're limited by what web browsers can do... if you have a plugin of some kind that will accomplish what you're after in a web browser, it can be used in Drupal.

Now, for the other stuff -

Why Drupal Commerce?

Drupal Commerce, compared to other shopping cart systems, is not a stand-alone e-commerce solution. Instead, it's basically a library that adds e-commerce support to Drupal, including: Shopping cart workflow, product entities that can be attached to any other Drupal entity, orders, payment gateways, shipping/tax/fulfillment modules, reports.

There are two big reasons why this is great:

  1. Commerce becomes a fully-integrated part of a much broader content management system. This means you can easily show products related to a blog post, product reviews, ratings, etc. And turn just about anything in Drupal to something that can be sold in the cart -- memberships (roles), event registrations, hotel bookings, physical goods, web service calls, digital downloads, content creation credits.
  2. Most other commerce packages may be great for the shopping cart and fulfillment side of things, but they end up not having a great content management user experience, and often are entirely separate applications that only share a theme with the rest of your site.

Flexibility and a great user experience is why you should consider Drupal Commerce, either 1 or 2.

Why Drupal 8?

If you haven't looked at Drupal recently, you should check out Drupal 8. While it shares a lot of concepts with earlier versions of Drupal, under the hood it's an entirely different system. It's now completely object-oriented, built upon the Symfony framework and uses modern dependency-injection patterns instead of the old mystifying hook system. If you like building on a modern framework, it's that -- but with a lot of the stuff you need already done, such as an authentication/authorization layer, a robust form API, a powerful theme/templating system.

Compared to earlier versions, the biggest win is the upgrade path -- Drupal has matured to the point where we're able to handle upgrades as a part of routine maintenance, instead of having to do a major upgrade project. We've already had significant new functionality added to 8.1 and 8.2, and if you stay current, the jump to 9.0 will not be drastic (just a housecleaning to remove stuff that's been deprecated along the way).

Stuff that used to be ridiculously hard in Drupal has gotten far easier, thanks to the OO framework. The UX is vastly improved. The mobile support is top notch. And there's a built-in migration framework, which, combined with the web services, makes it very easy to get any kind of data into or out of Drupal, when integrating with other systems.

Why Commerce 2?

Drupal Commerce 1 was built for Drupal 7, and while it epitomizes the benefits outlined above, the biggest downside was how much work you had to do to get it functional. There's a "kickstarter" that will get you up and running quickly, though it made some decisions for you that we found problematic later, so we generally had to put together everything from scratch (or our own base). This means defining all the order states and transitions, building all the email notifications, programming rules for what happens at checkout/payment fully received, creating report templates and sales dashboards, creating the cart displays, etc.

Drupal Commerce 2 is built for Drupal 8, and they have, much like Drupal 8 itself, started completely over with the benefit of experience. They took the opportunity to compare the user experience of all the other major shopping carts -- Magento, WooCommerce, Shopify, Squarespace, BigCommerce, etc -- and implemented a "best of breed" base user experience for the shopping cart, taking away a big chunk of the setup work necessary for Commerce 1.

But there's some other big improvements, too:

  • Multi-vendor/multi-store straight out of the gate. You can define a single store -- or a whole shopping mall -- out of the same Drupal instance.
  • Multiple checkout workflows. The Obermeyer site supports 3 different checkout user experiences -- retail customer, wholesale, and loyalty customers -- with entirely different shopping cart workflows, all under the same store.
  • Fully internationalized Address and Tax libraries. Instead of building these solely for Drupal Commerce, the Commerce team factored out this functionality into PHP libraries that are now used by other shopping carts to handle the needs of stores that sell internationally.
  • Migration support. You can migrate data from Ubercart and Commerce 1 today, and migrations from other popular carts such as Magento, OS Commerce, WooCommerce, Wix, etc are planned -- and not difficult for somebody experienced in Drupal migrations to build.

Why not Drupal Commerce?

The biggest reasons would be cost, and the amount of effort it takes to get a great user experience. Which is really the same thing.

For Commerce 1, pretty much everything you might need is there -- but you're building on a platform with no more than 3 or 4 years of remaining life before you would need to do a substantial upgrade (to Commerce 2). And there's quite a bit of setup to do to get a nice user experience...

For Commerce 2, the main thing is that not everything is quite there yet. Shipping, for example, is being worked on today, as I write this. There are only a couple payment gateways implemented just yet (though those are pretty easy to code). I don't think anyone is tackling fulfillment modules just yet (if you want to integrate with an external fulfillment vendor, for example), and a lot of the glue for memberships and event management still needs to be done.

If you are looking for the flexibility of a custom solution, need robust e-commerce, and want a solution that is going to be solid and current for the next decade, you won't find a better fit than Drupal Commerce, especially if you're willing to deal with some early implementation pain.

Let us know if you have any questions, or need a partner to help you get started with Drupal Commerce!

And kudos to the Commerce Guys for all their great work getting this out the door!

Oct 28 2016
Oct 28

WordPress versus Drupal. Republican versus Democrat. Two debates where the differences seem so broad, people can't even seem to agree upon fundamental facts. Why? Why is it so hard to find an objective, clear comparison of WordPress and Drupal? I've had several people ask this. And I think the difference comes from fundamentally different beliefs about what a website is, how it should work, and even how people can earn a living.

Caution! Extremely partisan views ahead! Gross generalizations, unfair characterizations, and mis-information follows. Do your own fact-checking, and your own analysis -- and please leave a comment below with what I've gotten wrong. Please keep it civil and on-topic, though, or it will be removed.

Where we are coming from

At Freelock, we've been working pretty much exclusively with Drupal for the past 8 years. We chose Drupal because, in spite of many flaws and shortcomings, we could deliver pretty much any functionality our clients wanted, faster, better, and with more flexibility and ability to adapt to change than any other platform.

We provide ongoing maintenance services for Drupal, and have a DevOps approach of continuous development, operations, and quality assurance. Recently several of our enterprise clients approached us asking to provide the same great maintenance service for some of their WordPress sites.

We took a look at our automated deployment and testing pipelines, dug into their concerns about having broken updates and solid backups, and realized that without much difficulty, we could build a mostly equivalent WordPress deployment and testing pipeline to our Drupal version, and provide pretty decent support.

Now that that is complete, we have some fresh hands-on experience with some current WordPress sites -- and with it, some unique perspectives on how the two compare.

What is a website?

Here come the gross generalizations!

If you're a WordPress developer, you think a website is a pretty brochure that advertises a business. WordPress is very designer-oriented, making it easy to create any crazy design you might want to implement. WordPress simply plugs in a blog engine and page editing to allow business users to easily create new content and edit page content across their sites. To a designer, everything you need is just another plugin you can deploy -- we don't really want to drill down into the details of anything inside your business beyond the brand, so just find a plugin somebody made that does what you want, install it, and you're off and running.

At least they think it is this easy. And if all you need is a brochure site, it can be this easy. Designers who love WordPress formerly built sites using Dreamweaver, Frontpage, tools that you can hand-edit HTML -- and so just adding the ability for clients to edit their own content is a huge leap forward.

If you're a Drupal developer, however, a website is an information management system. Sure you can do brochures, but what about your products? your events? your staff? customer testimonials? Twenty years ago lots of small businesses built out their operations on Filemaker Pro or Microsoft Access -- Drupal is a web-enabled, unlimited-user, tremendous upgrade of those systems. With an easily themeable face for the public.

If you're a WordPress developer, a website is an editable brochure that is meant to present the brand. If you're a Drupal developer, a website is a collection of content curated and presented to the public in an organized way, with a nice attractive look and feel, and possibly a much more business-critical application on the back end.

If you're a WordPress developer, Drupal is a bloated, complicated, confusing, obtuse monster of a platform, and you don't even know where to start -- where is the HTML that I can just edit?

If you're a Drupal developer, WordPress is an underpowered toy that is mystifyingly popular, with most sites a confusing jumbled poorly-thought-out hot mess under the hood -- what do you mean I have to pay for a module to have a current dev copy?

How should a website work?

I heard a WordPress shop owner tell a business prospect, right in front of me, that you should make an entirely new, fresh website every 4 or 5 years, and looked to me for corroboration. Websites are seen as disposible marketing vehicles that should be recreated regularly to keep current and fresh.

As a Drupal developer, I think the opposite -- a website is a valuable cornerstone of your business, should not be discarded but should be kept fresh all the time, periodically re-skinned, continually improved to provide your business with the most value possible.

If you treat your website as an asset, you can constantly improve it as part of your regular operations and always have something useful for your customers, employees, and other stakeholders, and not go through a big disruption every few years.

If you throw it away every few years, you're creating a nice income stream for your web developers, but you're constantly re-inventing the wheel, continually re-doing things you did the last time around, with major effort and attention needed all around -- by you (the business owner), your designers, and your staff as they have to learn their way around the new site.

Changing look and feel

A fundamental difference between WordPress and Drupal is how easy it is to entirely change the look and feel. In Drupal, a "theme" controls the look and feel, but all of the functionality is embedded in modules, and so when you switch themes, you simply drop the blocks into the appropriate region in the new theme, and you're done -- all your content and functionality is there, with an entirely new theme.

WordPress claims to have similar functionality with themes and plugins -- but in practice, a lot of functionality is embedded directly in the theme. You change themes, and a lot of what may be critical functionality of your site goes away and may need to be added to the new theme by a developer.

So this difference is not one of capability -- it's one of practice. Drupal developers isolate all functionality into modules so the theme is just a skin that can be easily changed. WordPress developers drop PHP snippets into their templates, because it's a fast, easy way to get that functionality there, but then you have to re-do that every time you want to change look and feel. (We do sometimes see Drupal sites apparently built by WordPress developers that follow the WP pattern...)

There are a lot more WordPress themes available, largely because designers are comfortable with WordPress, and that modular, information-architecture-driven approach of Drupal means that creating a nice Drupal theme is harder -- you have to cut everything up into smaller templates.

So at some level you do need to build a whole new site to change the look and feel of a WordPress site -- whereas in Drupal, you could choose a different theme every day without changing how your site works.

Ease of use

Ok, this is a big one. WordPress has a reputation for being "easy to use." After seeing a few WordPress admin areas, I'm left wondering why...WordPress Admin area

Sure, it's easy to install. So is Drupal. Sure, it's easy to write a blog post or a new page. And the core WordPress standard install does have nice media management built in, and all the basic every day tasks are easy enough. But sites in production with lots of plugins very quickly become extremely confusing, inconsistent, and puzzling -- very far from "easy to use".

Want to change some text that's in a sidebar? You might be able to do that, if it's in a plugin. But you probably need to go back to your designer because it's hard-coded in a template.

Want to slow down the jillions of spam submissions you're getting on your newsletter form? Well, there are dozens of CAPTCHA modules, but most only work on comment forms and user registration -- you need to find one that works with your newsletter plugin, or get a developer to integrate the one you want to use.

Want to create a custom content type? Your developer can do that, if they are actually a developer and know how to code. Or you can add a plugin to manage post types and fields -- but the better ones probably have a subscription, and you'll get nagged when it expires.

You can do it all in WordPress -- but the more you get beyond simple blog posts and pages, the worse the experience gets. You get confusing admin screens, stuff doesn't work across the board, approaches to building the site become fragmented and entirely custom, there are no universal "best practices."

In WordPress, beauty is skin deep -- delve beneath the skin and it gets ugly fast!Drupal 8 Admin area

Drupal, on the other hand, goes to great lengths to develop patterns that make everything work with everything else. Conflicts do arise, and can be challenging to resolve (just like any complex system) but for the most part, if you choose a captcha system, you can easily apply it to any form on the site, whether it's in core or a contributed module, or even an entirely custom form. When you add a date field to any entity in the system, you can now show it on a calendar. When you create a membership, you can add a price field to make it a product you can add to a cart, add a recurring field to make it auto-renew, add an address field and show it on a map. All without writing any code.

Editing a story in placeNo matter how many additional plugins or modules you add, the Drupal admin area is clean, well organized, and easy to figure out. If you have a complex, powerful site that is supporting a lot of functionality, it's far easier to manage in Drupal than it is in WordPress.

But even managing content on the front end is far easier, and with Drupal 8.2 you can edit most of what you see on any page, sometimes without even leaving the page!

The media management on Drupal 8 has also become excellent, and available right on the page as you edit.Set an image size, caption and alignment But more than anything, Drupal has always had great tools for organizing and managing content -- as you would expect a "Content Management System" to have. In the past, most of these were in add-on "contributed" modules, because many sites don't need them -- for example, scheduling posts to publish at a particular time, setting up a publishing workflow with proof-readers, copy-editors, and other reviews before publishing, arbitrary flags you can set on content to make it easy to keep track of what content has been reviewed and what you still need to go over, for keeping a large body of existing content up-to-date. Many of these are now in Drupal core.

WordPress starts out very easy to use, for very basic uses -- but pretty quickly devolves into a messy, confusing, hard-to-manage mess as you try to do more with the site. Whereas Drupal has historically started out very bare-bones, needing add-on modules to get decent usability, but making it easy to build out in a comparatively organized, consistent fashion, and everything works much more coherently with everything else. And even that has changed in Drupal 8, which ships a much nicer base user experience right out of the box.

If WordPress is easier to use, that's only for one group of users: designers. It's easy for them to get started. It's easy for them to add relatively simple code snippets. It's easy for them to plug it into a sophisticated design. It's easy for them to add plugins, especially if they're not the ones actually using the site.

For site owners and administrators, it's an entirely different story, especially when the site does more than simple blogging. Drupal can get messy if not planned out and built well -- but the standard practices and patterns of Drupal tend to make Drupal sites much better at doing complex things like event registration, e-commerce, CRM, and more than just simple blogging.

Security, Updates, and Reliability

Pasting random snippets of code from the Internet that you don't understand is not necessarily a safe practice. And yet this seems to be a very standard practice among a huge percentage of WordPress "developers". Even if the original author of those code snippets is perfectly honest, chances are good that it's not written in a secure, hardened way. This is part of why we see so many more hacked WordPress sites than Drupal sites.

When a Drupal site gets hacked, it's usually either due to a lame password, or an insecure hosting environment. WordPress is so much more interesting, in a depressing way.

WordPress core, and many plugins, are as secure as anything else, developed with solid programming practices, tested, and fine. The problem goes back to our first question, "what is a website?" and how much non-WordPress code is in so many WordPress sites.

Worse, it seems pretty common practice to develop on production sites, where you can easily break things. WordPress does not make it easy to keep a development or test copy of the site in sync -- when moving a site to another computer, path, or URL, you have to actually do a search and replace inside the database or your site may break in crazy ways.

Many WordPress developers are under the delusion that security updates can and should be done right on production, with the easy update buttons available right in the interface.

As a paranoid, tin-foil-hat-wearing security-conscious developer who expects things to fail now and then, I think that's crazy and foolhardy. What if something goes wrong during the update? What if the update source gets compromised? What if some feature your site depends upon changes? What if the change breaks the layout of your site? How do you roll back if something goes wrong? How do you detect what changed?

Worse, if the website can update its own code, that means an attacker exploiting your website can do that too.

In other words, when there are cyber attacks that hijack devices, websites, and home computers to use as weapons in cyberwarfare, updating websites this way is dangerous, reckless, and irresponsible.

Unfortunately, Drupal has built a similar mechanism for auto-updating contributed modules on a site, just as easily.

Fortunately, both WordPress and Drupal can be hardened so you can't update a site through itself, making it much, much harder for an attacker to break in and use your site as a weapon. (Not to mention what they can get from your site directly...)

It's harder to do in WordPress, but it can be done -- and we've done it with our WordPress Maintenance plan. We apply all updates on dev copies, we've automated the necessary search and replace so we can test changes effectively BEFORE they are put in production, and we've hardened the hosting environment to make it so an attacker can't put a file somewhere they can execute it. When we manage the environment, WordPress (or Drupal) cannot modify itself -- this hardening alone has entirely eliminated the risk of many of the recent dangerous vulnerabilities we've seen. And we can also sandbox each site in an isolated container so it can't even see or reach other sites on the same server.

The tools and techniques for keeping Drupal sites and WordPress sites secure and up-to-date are surprisingly similar. However, once again, it's the usual practice that is entirely different. And part of this is because it's much harder to run copies of a WordPress site than a Drupal site.

How do you make a living, as a developer?

Both Drupal and WordPress have a radical underbelly: The GPLv2 license. For a while, this was the most widely used Free, Open Source license, the very definition of copyleft, and it's still a broadly used standard.

A key tenet of the GPL is that you cannot restrict what people do with your code after you've given it to them. Anything that builds on (is "derived from") GPL code can only be used with the GPL -- which means if you try to restrict how other people distribute and use a module you create derived from Drupal or WordPress bases, you lose the right to use that code in the first place!

So what does that mean?

Apparently to many WordPress developers, it means you can't easily sue anybody who copies your commercial plugins -- but it seems to be widely acceptable to make commercial plugins and themes, and charge a license fee. I'm not sure how many commercial WordPress plugin creators there are, or how successful, but it's a huge market and it seems like the community largely supports these commercial plugins.

Which has led to the great fragmentation of the WordPress Plugin ecosystem. This is a large part of why it's such a mess -- commercial plugins are generally developed by a single company, they perhaps get fringe contributions from users, but largely the number of contributors to a commercial plugin is small. And plenty of competing plugins get developed, each with their own business model, and what plugin works with what other one is largely a question of how the various plugin writers form alliances, and what customers demand. This leads to... a huge mess. And it means there may be little consistency across sites that solve the same things.

On some WordPress blogs, I saw posts flaming a plugin developer for "stealing the code" from another plugin, and "harming the developer's ability to make a living" by developing a free, competing plugin. Well, that's exactly what the GPL explicitly makes not only possible, but desireable.

In the Drupal world, on the other hand, it would be the commercial plugin writer getting flamed for not working with the community to make a plugin that solves everybody's needs. There's pretty strong community pressure to make your plugins and modules open and free, so that many more developers can contribute the functionality they need.

This leads to Drupal modules becoming "best of breed" and generally ending up far more widely used, cover far more user scenarios really well, and far better quality. The number of contributed plugins you need for a sophisticated site has dropped substantially with each major new Drupal release, as more and more "power" modules like Views, Rules, Groups, Membership Entity, Message, Panels, and Commerce have emerged that are capable of replacing hundreds of individual single-purpose modules.

So if you're an ambitious WordPress shop, you aim to make a subscription plugin that thousands of sites will buy and give you an ongoing stream of income.

Whereas if you're an ambitious Drupal shop, that's not so much of a viable avenue -- but you can become a much bigger player by owning and leading the development of what becomes a critical module. Aside from Acquia, there are dozens of highly respected, multi-million dollar shops that are known leaders in Commerce, Organic Groups, Booking systems, Learning Management Systems, CRM, and more.

While both WordPress and Drupal are, technically, open source software, the Drupal ecosystem is a shining example of how successful that strategy can be. The WordPress ecosystem, on the other hand, seems to act more like the previous software generation's Shareware world of lots of tiny software companies selling one-off solutions of dubious quality. And many seem ignorant of the GPL, or apologetic about it...

Where does that leave us?

It's true that WordPress has an order of magnitude more sites deployed than Drupal. But the further up the value chain you go, the more you find Drupal. Drupal runs many top Internet sites -- and while WordPress appears in that rarified air, it needs to be heavily modified to handle the needs -- and at that point becomes pretty much custom software.

Compared to WordPress, Drupal:

  • Costs less to maintain (our Drupal maintenance plan is $100 less than WordPress, because we can automate quite a bit more, while WP involves many manual steps that cannot be easily automated).
  • Has far better tools to manage content
  • Can be easily re-skinned for a new look, usually with no change in functionality or architecture -- allowing us to change even old sites to a mobile responsive design with less effort
  • Can easily grow with your needs over time, plugging in new functionality that generally tends to work with the entire ecosystem, rather than taking you off to spaghetti-land
  • Has just finished modernizing their code and doing the hard work that will make future upgrades pretty much painless, while WordPress is rife with old code that may be in need of a big rewrite in the next year or two
  • Is a content management powerhouse that competes with, and effectively beats, the highest end commercial CMSs
  • Has drag-and-drop page layout managers that compare well to Software-as-a-Service platforms like SquareSpace, Wix, Weebly
  • Has a powerful migration utility in core, which can be leveraged for integration with back office systems
  • Tends to cost less if you're developing a complex site -- lots of users with lots of different user stories -- e-commerce, event registration, user-generated content, CRM, information database, course catalog, etc

Compared to Drupal, WordPress:

  • Is easier for most designers to theme
  • Has many more developers
  • Has many more canned themes available
  • Is easier to get going out of the box with little knowledge
  • Tends to cost less if you're developing a really simple site -- brochure site, blog

So why are you now working with WordPress?

As it turns out, it's not that hard to work on WordPress. After all, that's why it's so popular. The language is the same. It runs on the same technology stack, can be managed by most of the same tools. And most of the functionality is surface-level, easy to find, easy to debug.

For our customers who care about their website remaining up, protected, and regularly tested, we can apply the same process and automation to WordPress that we can with Drupal. There are still more things we need to do manually, but we can test all our changes safely away from production, do visual regression testing on all code changes, and provide a safe and secure hosting environment that we can easily recover if something happens.

What we're already finding is that we need to turn to code to fix a huge number of things that are simple to do in Drupal.

In Drupal, much of our support is providing help around how to accomplish a certain task -- a huge number of requests are things our customers can do themselves without HTML or coding knowledge. We're already finding that a lot more WordPress requests involve changing page templates or writing code.

You'll hear about the infamous steep learning curve of Drupal. It's true, there is so much power there that is hard to learn, hard to even know it exists -- but if you learn the tool, you see how powerful it is and how easy it is to accomplish things that are very difficult in WordPress.

And so our thinking is, there's got to be a ton of WordPress sites that would be far easier to grow and enhance in Drupal. We're happy to have the steady paycheck of doing lots of one-off patches on WordPress sites, but for those customers who would rather be able to do it themselves, we can provide an easy migration path into Drupal where they have that power!

For that matter, go ahead and build your site in WordPress. When you're ready to start actually making it a valuable asset for your business, we can port the design into Drupal, suck over all your content, and build out the database you need to make your business more effective!

Oct 07 2016
Oct 07

When choosing any service provider, a crucial question is, "What happens if something goes wrong?" When you're choosing a hosting provider, we like to dig a bit deeper, and ask what risks are likely to be an issue for you?

Here are some of our questions:

  • How easy is it to hack?
  • How easy is it to recover?
  • How much downtime might you have?
  • How much data might you lose?
  • How much traffic can you handle?
  • How much access do you have?
  • What steps are involved in handling more traffic than you can handle today?
  • What regulatory compliance needs do you have?
  • Who is maintaining the site, OS, and supporting packages?
  • How much support do you need?
  • What is your budget?

Before drilling into these areas, let's quickly go over the basic types of hosting:

  • Shared Hosting - Your site is one of many running on a large server. This is what you get when you go sign up for a "cheap" hosting plan. You generally get FTP access and a control panel -- if you're lucky there is "shell" access (SSH).
  • Dedicated Virtual Private Server (VPS, or Cloud Server) - This is generally what most of our customers use, a private server that only runs your stuff. Generally you have full root access and the ability to run whatever software you like.
  • Platform hosting - For Drupal this is hosting like Pantheon or Acquia -- infrastructure built up to allow you to manage your site, with tools to assist with deployment, multiple environments, and often abstracting away the hardware or virtual servers supporting the site.
  • Dedicated hardware ("Bare metal") - This is the "old school" co-located server hosting that was very common before the "Cloud" became such a big deal -- you lease a server in a datacenter, or purchase one and lease a connection in the data center.
  • Load-balanced, distributed system - Beyond a certain level of traffic, a single server cannot handle everything. Or, if any downtime means significant loss of business, you need redundant systems.

Yet another dimension to consider is the level of management you're getting, and the tools you have available to help you (or us) manage your site and server. This varies highly by the host.

Now let's take a look at how these stack up when we look at the various areas of risk:

How easy is it to hack?

We see hacked sites all the time. This single question basically eliminates most shared hosting from the discussion -- when the standard hosting package puts hundreds of sites on the same server, your site might get "infected" by a worm installed by any of those other sites. Or, if anyone on your team uses FTP to copy data up to the server, the username and password can be easily sniffed by any "man in the middle" or a script kiddie on that same coffeeshop wireless network.

Even if you take reasonable precautions, and have a security-conscious host, most shared hosts run your site with the same user account you upload files with -- meaning that somebody exploiting a vulnerability in Drupal or some other library can plant a malicious file and run it remotely, and from there attack the server itself, sniff out all traffic to your site, do whatever they want -- your site is owned by the attacker.

This really eliminates the vast majority of Shared Hosting for any site that you don't want to be easily hacked.

At Freelock, our shared hosting is not like this -- we don't run FTP anywhere, we don't even allow customers to log into production servers, and all executable code is locked down in a way that the web server cannot write to places where code can run -- an extra layer of security that hardens the site, makes it much harder for an attacker to get in.

This is the same configuration we put on all the servers we manage for our customers, too. So, scorecard:

Type Score Note Shared hosting Veto (-20) Most shared hosting has really bad security, and no way to harden your site. Managed VPS/Bare Metal 3 If you get a VPS managed by a host, typically they provide a control panel and stack that cannot easily be hardened. Platform Hosting 5 The Drupal-specific platform hosts are generally hardened. VPS, Bare Metal 5 Unmanaged VPSs or Bare Metal can be configured with a hardened stack and made secure. Freelock Shared Hosting 5 We harden our stack!

How easy is it to recover?

Even the most secure systems sometimes have outages. Hard drives fail. Power supplies fail. Controllers fail. Passwords get compromised. Data gets corrupted.

How easy is it to recover? Depends entirely on your backups -- and the nature of the incident.

We generally recommend having two entirely different backups to cover all the risks. And with Drupal, you have 4 different components to consider:

  1. Site code -- Drupal core and contrib modules.
  2. Site assets -- all user-generated files, uploaded photos/images.
  3. Site configuration -- in Drupal 8, exported configuration. In Drupal 6 - 7, generally features.
  4. Database -- your content, and generally a lot of configuration.

So being able to recover effectively means you need to prepare for all of the risks. Hardware failure is generally the easiest risk to recover from. A corrupt or accidentally deleted file that you only discover weeks or months later is perhap the hardest.

At a minimum, you should have a full server snapshot taken on a nightly basis, and keep a few of those around. We generally keep nightlies for 1 week.

A server snapshot can be spun up in a matter of minutes, to recover from the previous night. Extracting a single item starts to become a forensics activity, and is much harder to do, but if you have the data somewhere, it at least becomes possible.

So we couple our nightly snapshots with historical file-based backups that go back 16 months. And we keep this secondary backup system at an entirely different provider, so we have a copy of the data if a vendor shuts off your access entirely.

In short, most hosts (other than Shared Hosting) provide some sort of image snapshot capability -- make sure you configure and use them! Here's a quick run-down of ones we have current experience with, and a rating to reflect how easy to set up, easy to use, and how powerful/flexible the system is:

Host Provider Score Note Most shared hosts -5 Most shared hosting does not provide any auto-backup facility. Amazon EC2 (AWS) 4 Must be scripted -- Snapshots may be easily taken on demand at any time. We provide a custom script to take snapshots nightly. Can also leverage S3 versioned files to maintain historical backups, Content Delivery Networks, and more. Google Cloud Engine (GCE) 3 Like AWS, these backups must be scripted, and we have a custom script for doing GCE snapshots. Digital Ocean 3 Snapshots may be enabled when the image is created, for an extra charge. Automatic snapshots run on DO's schedule, you have no ability to trigger a snapshot on your schedule (unless you stop the server first). Rackspace Cloud 5 Easy automatic backups, easy on-demand backups Pantheon 5 Automatic and on-demand snapshots, very easy to configure. Can be downloaded with a shell tool. Acquia 5 Automatic and on-demand snapshots, very easy to configure. Can be downloaded with a shell tool. Pair.com 4 We have several clients at Pair, and they have a decent backup system, but generally requires interacting with their support to set up and actually use.

Generally with proper planning and extra work, any host can have proper backups set up. AWS is more work to set up properly, but extremely flexible with lots of options.

How much downtime might you have?

This roughly breaks down into several different kinds of downtime:

  • Downtime due to platform maintenance, upgrades
  • Downtime due to site maintenance
  • Downtime as a result of system failure, time to recover from backups

Platform maintenance

For regular maintenance, and upgrades, we have to give the crown to Pantheon. We have several clients at Pantheon, and never noticed any downtime at all.

Acquia, on the other hand, seems to have frequent maintenance windows that involve brief amounts of downtime.

Our own Freelock server management also generally involves server reboots every few weeks that lead to a minute or two of downtime, and Docker container restarts that take a few seconds.

Otherwise if you're on servers that never reboot, it is opening up the risk of running old server software that may contain more vulnerabilities.

Drupal site updates

Downtime due to Drupal site upgrades generally involves the time period from when the code is updated and the database update scripts applied -- and if something fails, the time it takes to recover. Our Drupal site maintenance provides testing of all updates before rolling out to production. Pantheon forces you through a deployment path that gives you the opportunity to test before rolling out, but doesn't test for you. Acquia also provides multiple environments for testing, but does not force you to use them.

Otherwise you're on your own.

Recovery from hardware failure

Ultimately a "Cloud Server" is a shorthand way of saying "Somebody else's computer." At some point, there's a hard drive, a controller, a powersupply, a motherboard, RAM, and physical network equipment. And hardware fails sometimes.

There are more serious risks to consider as well -- vendors go away without notice, accounts get shut down, stuff gets corrupted and hacked -- but hardware failure is relatively common and routine, and one of the first risks you need to have covered.

So what does recovery from a local snapshot backup look like?

Host type Score Time Note Most shared hosts -5 Never What snapshot backup? Freelock Shared Hosting 3 1 - 2 hours We use cloud VPS providers for all our production hosting, and configure multiple backups. Amazon EC2 (AWS), Google GCE, Rackspace, most cloud VPS providers 3 1 - 2 hours Create a new server, attach disks from old server, reassign IP address. Create new disks from snapshot if necessary. Pantheon, Acquia 4 < 1 hour Contact host, get them to do the steps above Dedicated hardware 1 1 -2 days, could be a week or more This is the big downside of dedicated hardware -- when it fails, it's not easy to migrate to new hardware. It needs to be replaced, and the site will be offline until replacements can be arranged. Distributed systems 5 Theoretically, no downtime

Everything's great, in theory. A properly architected system can be far more resilient to hardware outages, but costs a lot more to implement, and introduces a lot more complexity that can have its own risks. And a distributed system might also make you more vulnerable to outages, with more systems that have to operate correctly to maintain uptime.

The crucial point here is that with some effort ahead of time, and using a cloud server, you can generally recover from hardware failures within a matter of hours, but on dedicated hardware it could be days or weeks. Largely for this reason, we greatly prefer virtual private servers over dedicated hardware.

How much data could you lose?

Shared hosting, it's entirely up to you to create your own backup systems. Generally most platform and VPS providers make it easy to do nightly backups, and possible to do them more frequently -- there's no reason you can't do a snapshot every hour if it matters that much to you, though each snapshot tends to put a relatively heavy load on the site, which means it may impact the site performance and how much traffic it can handle.

So generally, if you're ok with losing any new data that was added today, most professional packages have you covered. That's what we cover with our server maintenance plans, and generally what most of our customers are comfortable with.

What if you're not okay with losing several hours of data?

Really the only answer is adding redundancy, with failover servers. Usually out of the components of the Drupal site, the data you need to back up is all in the database. You can set up master-slave replication, and get a live copy of all data going through the site. Then if something happens to the main database -- something other than data corruption that gets replicated, that is -- you have up-to-the minute data you can use when you get the service back up and running.

How much access do you have?

This boils down to what you can put and run on a server. For us to fully manage it, we need full root access, and the ability to install whatever software we need. Different providers will have their own management tools, and may dictate a particular stack of software that if you change, you break their tools and can no longer use their support (if you have the ability to change this at all). Here's our assessment of hosts we have current experience with:

Host Provider Score Note Digital Ocean, Google Cloud Engine, Rackspace, most unmanaged dedicated servers 5 Can install anything you want, large range of distributions, you have full root access, AND a remote console so you can log into the individual server if something goes wrong with the network stack or SSH. Amazon EC2, many other VPS providers 4 Full root access, even broader range of distributions available -- the one knock is no remote console, you have to have SSH/Networking running to administer a server. Recovering if you don't have that involves attaching the drive to another server. Pair, Acquia, most shared hosts with SSH support 3 No root access, no ability to install server software, but sufficient access to manage a Drupal site effectively. We cannot provide server maintenance on these plans, if we don't have root access. Pantheon 2 The only access you have is via Drupal itself, or the web interface/"terminus" shell tool. FTP-only shared hosts 0 FTP just plain sucks, you can't script deployments, we can't provide our site maintenance plan with only FTP access. Freelock hosting 1 Of course we have full access internally, but we don't provide access to clients. We will work with clients on arranging backup solutions, but we restrict access very tightly to maintain a very secure environment, and ensure we score highly everywhere else!

So the main questions with access are relevant to who's doing the work. We have a lot of flexibility over what we can install and manage, but if you have another company also managing the server, it usually greatly restricts what we can do, and usually conflicts with our management.

What steps are involved in handling more traffic than you can handle today?

This basically revolves around how much traffic are you provisioned to handle, and what is involved in making that more?

The Platform hosting providers -- Acquia, Pantheon. Platform.sh handle this scaling for you, and have the infrastructure built out to make this a really easy thing to do.

The larger infrastructure providers make this kind of growth possible -- Amazon, GCE, and Rackspace (among others) will let you secure a dedicated IP address you can assign to a load balancer or another server that can greatly ease a move when it's time, and also provide database-as-a-service ("RDS" for Amazon) and a Content Delivery Network (CDN) for your assets that provide a path to scaling up to a fully distributed system.

Digital Ocean, other smaller VPS providers, and dedicated hardware don't necessarily provide an easy upgrade path here -- so scaling up usually means getting a bigger VPS and transferring your site over, or building out your own distributed system from generic servers.

What regulatory compliance needs do you have?

The most common one we deal with is PCI Compliance -- the "Payment Card Industry" standards you need to adhere to if you're collecting credit card payments.

In our opinion no shared host adequately meets these requirements while having the ability to harden the site to the level we think necessary for any sensitive data. To be compliant, you either need to send the customer away to a completely separate payment site, or host at a host that has been certified for PCI Compliant hosting. There are a few hundred hosts on that list -- currently Amazon EC2, Google Cloud Engine, and Rackspace are on this list -- but Digital Ocean and many smaller providers, including Freelock, are not.

If you need PCI compliance, we generally turn to Amazon or Google -- or a local/regional datacenter that has undergone compliance.

Other kinds of compliance includes "PII" - Personally Identifiable Information, which is largely regulated state by state and not nationally. If you're storing PII in any way and get hacked, you could easily have a state Attorney General investigating you, and instituting large fines if you have not covered a whole range of best practices and measures to prevent compromise.

And the other big one: HIPAA, which covers health care information. Generally if you work with HIPAA-protected data, we would work with a data center that makes people available to help go through HIPAA compliance. We've spoken with several, though have not yet worked with any directly.

Who is maintaining the site, OS, and supporting packages?

Freelock stepped into this niche, because there is basically nobody else providing full management of the entire stack.

In the table below, the green cells show areas you don't have to even think about because they are handled for you. The orange ones illustrate areas where you might need to make some decisions or request support, but is largely on you -- or may have limited options. The red cells are areas that are completely up to you to figure out and manage on your own (unless you hire somebody else to manage these areas for you).

Host Provider Manages Hardware Manages OS Manages Servers (Solr, Redis, Nginx, Apache, PHP, Databases) Manages Drupal site updates Manages content/ effectiveness of your site Notes Shared Host Yes (unless reseller) Yes Some (though rarely up-to-date) You You   VPS Provider - Amazon, Google, Digital Ocean, etc Yes You You You You Everything is up to you above the hardware! Managed dedicated - e.g. Pair (without root) Yes Yes Yes - but cannot install extras beyond what they provide You You They manage the server, you manage the site Platform Hosting - Acquia, Pantheon Yes Yes Yes - and useful extras available Provides tools to make this easy, but you have to do it You Nice supporting tools if you want to be hands on managing your site Unmanaged Dedicated You (though with "remote hands" support) You You You You It's all you Freelock Server + Site maintainance on Cloud VPS Relies on Amazon/DO/Google for this Yes Yes Yes Provides consulting and support You get to focus on the content and goals for your site

Our Site Maintenance plan can provide everything necessary for the "Manages Drupal Site Updates" column, and some feedback on managing content and site effectiveness (though actual implementation is usually done as hourly retainer or budget).

Our server maintenance plan covers managing the OS and the servers, and we can provide all the same server add-ons that Pantheon and Acquia offer on top of a more bare VPS from elsewhere.

How much support do you need?

Taking a look at the previous question, there are lots of different levels where support may be needed. Most often, we get the question of, "Do you have 24x7 support?" And the direct answer is, no we don't, but do you really need that? And if so, what sort of issue are you concerned about?

The biggest one is downtime. Oh crap, my site is offline, and it's 9:00 on a Friday, am I going to be down all weekend?

Unless you've been hacked, or unless you've mucked around in a production site doing stuff you really should have done in a test site first, chances are you really only need 24x7 support from the actual host, whoever is managing the actual hardware. If you really need that, you will need more premium support, and it's worth looking at some of the support plans for Amazon, Rackspace, or Pair.

While we don't offer an SLA (a "Service Level Agreement" that basically refunds you if your site is unavailable for a specific amount of time) with support terms, we do monitor all of our hosting and server maintenance clients, and do our best to assist with any outages at any hour of the day. We've been doing server maintenance and hosting since 2003. Our longest outage in over 13 years has been one incident that took a server down for about 13 hours, and a handful of incidents that affected particular customers for 3 - 6 hours. Otherwise we may see up to 30 minutes of downtime if we screw something up, perhaps once or twice within 5 years, with upgrades we run well after regular business hours, and routine interruptions of a minute or two as we deploy updates.

If you need 24x7 Drupal support, Acquia is the only place I'm aware of that offers that, and then only on enterprise plans that cost tens of thousands per year.

Otherwise, when you're looking for support be sure to get support for the kinds of things you need -- help using your site, help fixing things that break, help restoring backups, help setting up new sites, help doing your own development, help managing server access, etc.

What is your budget?

Finally, the question most people start with, is where we're going to end. As you can see, there's a lot to consider when choosing a host, and an enormous range of what you get for your dollars. When it comes down to it, where are you going to get the most value for your money?

And this boils down to a huge number of factors we've only touched on in broad strokes.

< $75/month

At this level, you don't get full coverage or support. If you have the technical skills, you can certainly run your own sites and servers and find bargains here, but if you're reliant on others for skills, your best bets are probably either Pantheon Personal sites (at $25/month) or Freelock Multi-site hosting (at $50/month). These are low traffic options, minimal support, and often poor security options -- we focus on providing security and maintenance at the expense of flexibility in our multi-site option.

$75 - $300/month

At this level both Pantheon and Acquia have some options to consider, if you want to handle upgrades yourself, and Freelock hosting with maintenance enters the picture for low-traffic but highly customized sites.

$300 - $500/month

This is a reasonable level to spend to get decent support and maintenance for both a server and a site, with a dedicated VPS. At this level you should be able to handle a decent amount of traffic, conduct e-commerce, and generally have a well-secured server, full access and ability to run whatever you want.

> $500/month

Several requirements push you over $500/month:

  • High traffic levels, beyond what a single server can handle
  • Business critical need for protecting against data loss of even a single minute
  • Need for highly available, redundant systems
  • 24x7 support

So where should I host?

Our current top 3 recommendations are Amazon EC2, Google Cloud Engine, and Digital Ocean -- any of those combined with our site and server maintenance plans. These all are infrastructure-oriented cloud hosting services, and we generally provision virtual private servers at one of these three.

We don't tend to put people at Acquia or Pantheon mainly because there's a huge overlap with our services in terms of the deployment pipelines, server management and more -- and their toolchains tend to get in the way of ours (especially Pantheon's). But we do have active clients at both, and are very familiar with their offerings, and recommend both for people who aren't a good fit for our maintenance plans.

Digital Ocean used to be our "go-to" budget option, but Amazon and Google have been in a price war, and while the pricing might vary depending on exactly what you need, we're finding that for what we most commonly provision (2GB, 4GB, or 8GB servers) all three are right in the same ballpark.

Digital Ocean

We still like Digital Ocean as it's very easy to get started, very easy to use, and we like smaller startups. But the two downsides are:

  • Lack of PCI Compliance
  • No real control over timing of backup snapshots, without taking the server offline.

Sign up with this link and get a $10 credit.

Google Cloud EngineExcerpt from http://www.fredtrotter.com/2016/08/22/google-intrusion-detection-problems/

Google Cloud Engine seems to have it all -- a nice, streamlined management interface, console access to the servers, a range of server add-ons for persistent disk, database servers, and much more. Plus you get 2 months of free service, up to $300 of resources.

But the one big downside of GCE is the risk of going offline. We've had two accounts get suspended without warning or notice, over a weekend, taking hours to resolve, because of billing snafus at the end of the free trial. And this story scares the hell out of me (quoted on the right).

We like the Google service... we're just a bit unnerved by their "too smart" systems and lack of ability to reach a human to resolve any issues.

Amazon EC2 (AWS)

... That leaves Amazon. Amazon is the big behemoth of the "cloud" world, and tend to be innovators. We don't use most of what they offer as a platform -- but given their steadily sinking prices, extreme flexibility, their compliance story, and heavy reliance by most of the industry, it's the current "go-to" host and we recommend it more often than not.

Compared to GCE and Digital Ocean, AWS strangely lacks a direct console to any VPS you spin up, and sometimes it "feels" slower -- server restarts can take a couple minutes instead of a few seconds. Otherwise there really isn't a downside to using AWS.

Whew! I'm exhausted!

This is our best overview of how to pick a host, with notes based on our experiences as both a (small) host and a maintenance provider. I'd love to hear more about your favorite hosts, and why, below. There are plenty of other options, but some of what you need to know to provide a proper evaluation can only come after taking the time to put something into production there, working with the backup systems, upgrading servers.

If you want to advertise your own service in the comments below, please make sure you answer the questions as they apply to your service -- general spam is ruthlessly deleted. And if you'd like our assistance with server or site maintenance, contact us to get started!

Sep 05 2016
Sep 05

I learn best when I have a problem to solve, and with one of our D8 upgrade projects, we had a mess to clean up in the menu system. This provided an excellent oppportunity to get hands-on in Drupal core, learning some of the major differences from earlier versions, and three things in particular:

  1. How easy it is to use Drupal Console to create your own tools
  2. How much the menu system has changed
  3. How to load and save just about any entity

The problem

It seemed like an easy one: links on the menu were going to internal "node" paths instead of the aliased "friendly URL". For example, a link on a submenu went to "/node/494" instead of "/about/accreditation". In this particular project, there were several hundred links that got mangled in this way.

I'm not exactly sure how this happened, but it has something to do with starting the migration in Drupal 8.0, updating the migration to work with 8.1, and going back to clean up all the friendly URLs. Somewhere along the way, the menu system ended up with links to the node/[nid] path, and not the friendly URLs, even though the pages themselves worked fine with the URL aliases.

The Redirect module was already set up, and configured to redirect to the alias -- but this does not seem to work correctly yet.

Bulk operations were of no help -- and worse, the menu and parent items on the menu link edit form on most of the links did not show up correctly, even though the menu itself showed links in the correct place.

Creating a repair tool using Drupal Console

In earlier versions of Drupal, we've created dozens of little helper scripts using Drush. It's relatively easy to create a drush script to fix up data, set up integrations with other systems, diagnose issues, etc. Drush was our go-to tool for creating little helper tools.

In Drupal 8, Drupal Console has become a compelling alternative -- mainly due to its built-in code generation. Drush is still an important part of our kit, but Drupal Console has quickly become an important complement.

And it turns out, one of the things you can generate in Drupal Console is a Drupal Console command. First, create a module (if you don't want to add the command to an existing module):

[email protected] >git/test (develop)> drupal generate:module

 Enter the new module name:
 > Fix Menus

 Enter the module machine name [fix_menus]:

 Enter the module Path [/modules/custom]:

 Enter module description [My Awesome Module]:
 > Fix non-aliased menu link data

 Enter package name [Custom]:

 Enter Drupal Core version [8.x]:

 Do you want to generate a .module file (yes/no) [yes]:
 > no

 Define module as feature (yes/no) [no]:

 Do you want to add a composer.json file to your module (yes/no) [yes]:
 > no

 Would you like to add module dependencies (yes/no) [no]:
 > no

 Do you confirm generation? (yes/no) [yes]:

Generated or updated files
 Site path: /home/john/git/test
 1 - modules/custom/fix_menus/fix_menus.info.yml
[email protected] >git/test (develop)>

Next, generate your custom command:

[email protected] >git/test (develop)> drupal generate:command
 Enter the module name [ctools]:
 > fix_menus

 Enter the Command name. [fix_menus:default]:
 > fix_aliases

 Enter the Command Class. (Must end with the word 'Command'). [DefaultCommand]:
 > FixAliasesCommand

 Is the command aware of the drupal site installation when executed?. (yes/no) [yes]:

 Do you confirm generation? (yes/no) [yes]:

Generated or updated files
 Site path: /home/john/git/test
 1 - modules/custom/fix_menus/src/Command/FixAliasesCommand.php
[email protected] >git/test (develop)>

... And just like that, you have a module with a Drupal Console command that's all hooked up and ready for code, runnable as soon as you enable the new module.

In the new module, you primarily need to fill in the execute method in the generated file.

One thing I have not yet figured out is how to get PhpStorm to be intelligent about Drupal Console classes, like Drupal\Console\Style\DrupalStyle. But it's not hard to find info about the Symfony InputInterface and OutputInterfaces.

Menu System Changes

I've heard that the menu system is one of those areas that many Drupal folk think needs major improvement. While I'm not sure how improved it is in D8, it has certainly changed -- a lot.

In Drupal 7, menu links were basically rows in the menu_links table. Other than external links (which begin with "http://"), links in D7 and D6 pointed to specific items in the system -- views paths, nodes, taxonomy terms, etc. While you could enter an alias in a menu link, it got converted to the system path on save. And, as a result, a menu link pointing to a node always pointed to node/[nid] in the underlying table. When the menu was rendered, the system automatically replaced all system paths with the corresponding alias (friendly URL) if one exists.

This system is quite familiar to anyone with Drupal experience. Menu links pointing to unpublished nodes are completely invisible, always. Links to content you don't have access to are always hidden as well.

The Drupal 8 Menu System

Drupal 8 changes this entirely. You can still have links to entities, views, panels, terms, users, etc. just like D6 and D7 -- however, you can also have links to arbitrary paths inside the site, disconnected from whatever actually gets loaded at that path. These links may each be enabled or disabled, but they are otherwise disconnected from whatever loads at this URL.

Menu links are now content entities, like many other entities in the system, with an entity type of "MenuLinkContent." While there is no UI for attaching fields currently in Drupal 8 core, as a content entity you interact with it much the same as with other content entities, by loading and saving it.

So if you add a link using a regular entity edit form (e.g. by editing a node or a view), the menu link gets created with a direct reference to that content, and rewritten to whatever alias it has whenever the cache is rebuilt.

However, when you migrate in menu links, it uses the new "base" style.

In the menu_link_content_data table in the database, you will see most migrated links listed with a link__uri value beginning with "internal:/" followed by the migrated path. However, if you add a menu link to a node form, it saves as "entity:node/[nid]" instead.

On the MenuLinkContent entity, the "internal" links are considered Unrouted, while the entity links are Routed. These two varieties have quite different structure under the hood -- the question I had was how to change from one to another. Nobody on IRC seemed to have an answer, and the documentation seems pretty lacking in this area -- I found issues discussing the new structure, but no documentation for changing from one variety to the other.

So I rolled up my sleeves and debugged the menu link edit form, to see how this problem got solved.

And it turns out, it's pretty simple: just set the link item's link->uri to either 'base:node/[nid]' to get an unrouted link or to 'entity:node/[nid]' to get a routed one, and save it.

This still lost the menu's place in the overall hierarchy, but I was able to fix that by walking the menu tree, and setting the menu name and parent link on each child link.

Loading and Saving Entities

I've already spent quite some time in the D8 migration system, but this was really my first foray into the heart of Drupal 8's API, for far more usual things like updating an entity. How you do this seems to be in flux -- the approach I found to work, using \Drupal::entityManager(), is already marked deprecated.

When you walk the menu tree, you get link objects, but these are not actually entities and so you can't edit and save them directly. The link object does contain a UUID that can be used to load the actual entity, so you can update and save it.

So... here's the final code:

 * {}
protected function execute(InputInterface $input, OutputInterface $output) {
  $io = new DrupalStyle($input, $output);

  $menu_tree_parameters = new MenuTreeParameters();
  $tree = \Drupal::menuTree()->load('main', $menu_tree_parameters);

  foreach($tree as $id => $menu){
    $this->walk_link($menu, $io);

 *  $menu - Menu link to update, and load children
 *  $io - Drupal Console io object, to print results
 *  $parent - The parent menu link to set on this link
 * Recursive function to update all links on a menu
private function walk_link($menu, $io, $parent) {
  /**  \Drupal\menu_link_content\Plugin\Menu\MenuLinkContent $link */
  $link = $menu->link;

  if ($link->getProvider() == 'menu_link_content') {
    /**  \Drupal\Core\Url $url */
    $url = $link->getUrlObject();
    if ($url->isRouted()){
      // This is already a routed link, ignore
    } else {
      $path = $url->getUri();

      if (preg_match('/^base:node\/(\d+)$/', $path, $match)) {
        // We have a node path... ding ding ding!
        $nid = $match[1];
        // Plugin ID looks like: menu_link_content:29efb9ab-cde8-4f3b-8878-9065a5af79b1
        list($type,$uuid) = explode(':', $link->getPluginId());
        // entityManager has been split into many different services -- getStorage
        // is now on entityTypeManager.
        $storage = \Drupal::entityTypeManager()->getStorage('menu_link_content');
        $entities = $storage->loadByProperties(['uuid' => $uuid]);
        /**  \Drupal\menu_link_content\Entity\MenuLinkContent $entity */
        $entity = array_shift($entities);
        // Now update the entity:
        $entity->parent = $parent;
        $entity->menu_name = 'main';
        $entity->link->uri = 'entity:node/'.$nid;
        // Now print the change:
        $io->block($link->getTitle() . '('.$nid.'): ' . $path . ' -> entity:node/'.$nid);


  // Now update the child links
  if ($menu->subtree) {
    $parent = $link->getPluginId();
    foreach ($menu->subtree as $submenu) {
      $this->walk_link($submenu, $io, $parent);

The Takeaways

Going through this process, I learned a lot about the internals of Drupal 8:

  • Drupal Console is a great tool to quickly get started on any new functionality you might need, and can be used to extend itself, even easier than drush.
  • The Drupal 8 menu system is more sophisticated than earlier versions, with distinctly different Routed and Unrouted links -- this is an area you need to understand to build sites more effectively.
  • The current Drupal to Drupal migration creates unrouted links (probably so they can import them before the entities are imported) -- but older versions of Drupal are more comparable to routed links.
  • Routed links appear in the "menu settings" of content editing forms. Unrouted links do not.
  • When you update a URL alias, the system does attempt to update unrouted menu links -- however, sometimes this results in rewriting to the internal unaliased link, and not the correct alias. Routed links (presumably) always update correctly.
  • The current (as of 8.1.x) "correct" way to load an individual entity, of any kind, is using entityTypeManager->getStorage(). You can call it directly using \Drupal::entityTypeManager(), or better, use dependency injection to load it as a service.
  • Drupal entities provide a bunch of "magic" getters/setters that allow you to set properties directly on the entity, in relatively simple ways. Unlike earlier Drupal versions, you don't actually need to drill down into the nested arrays of language, delta, value, etc -- you can often just set a property directly and the magic setter will set the appropriate value.
  • As usual, finding related code examples is often the best way to get familiar with how to interact with a particular internal API. An IDE like PhpStorm can be immensely helpful for both searching for related code and debugging your own code.

Any tips you'd like to add to this, or share? Please leave a comment below!

Aug 04 2016
Aug 04

So there are definite "gotchas" to migrating content from Drupal 6 to Drupal 8, when you take away the assumption that the ids throughout the system will remain the same. We hit another one on a recent launch: the URL aliases imported from Drupal 6 did not get rewritten with new node ids, after the migration had started using a map.

The issue: When importing content that uses a new nid, you can simply add a "process" plugin to process that value through the migration map to get the correct nid in Drupal 8.

However, the upgrade_d6_url_alias does not deal in nids -- it simply adds a "/" character to the start of whatever it found in the Drupal 6 url_alias table, and inserts it into the corresponding table in the Drupal 8 url_alias table.

Here are some examples from a d6 url_alias table:

| pid | src | dst |
| 2208 | node/1850 | wisconsin-state-refund-policy |
| 2210 | node/1852 | what-does-it-cost-0 |
| 2213 | node/1855 | indonesian-phrases-study-tour-students |
| 2214 | node/1856 | aromatic-indonesia-study-tour-what-pack |
| 2215 | node/1857 | dos-and-donts-travel-indonesia |

In our D6 -> D8 migration, the url_alias migration is pretty simple. All migrations specify a source and destination -- the part we're most interested in is the "process" section, which looks like this:

plugin: concat
- constants/slash
- src
plugin: concat
- constants/slash
- dst
plugin: d6_url_alias_language
source: language

This section essentially processes 3 fields: source, alias, and langcode. It's the source field we need to rewrite with the correct nid mapped from the node migrations.

Fortunately in our case, users and taxonomy terms were not added on the Drupal 8 site before the final migration, we only needed to focus on nodes.

We could create our own process plugin to rewrite this all according to whatever rules we want -- but for expedience, we found in the Drupal 8 migration documentation there was a Callback process plugin that might make this a breeze.

This process function does not allow for any arguments to be passed to the callback, but it does allow you to use a static method on a class.

The solution? Add a class file inside the module, in src/. PHPStorm and Drupal Console greatly assisted here to make this a breeze to do.

Next up: How do we skip the rows we don't care about? The taxonomy/term, user/, paths? We only care about nodes. By viewing the source code of some other migration plugins, I found the MigrateSkipRowException exception, which presumably you would throw to skip processing this row.

So now we know how to get a function that will process the src URL, skip all the rows we don't care about, and return the nid so we can apply the migration plugin. After that, we need to rewrite the new nid into the new source pattern.

Unfortunately, I didn't see any way to use the previous Concat plugin to concatenate text with the nid returned by the migration plugin, so the answer was another Callback plugin, a new function I added to my little helper class.

Once I re-loaded that config, I reran the migration with the --update flag, and it worked perfectly!

The new YAML for the source process plugin:

      plugin: callback
        - '\Drupal\site_migrate\MigrateUtil'
        - filterint
      source: src
      plugin: migration
        - upgrade_d6_node_alumni_in_action
        - upgrade_d6_node_banner
        - upgrade_d6_node_blog
        - upgrade_d6_node_course
        - upgrade_d6_node_course_group
        - upgrade_d6_node_event
        - upgrade_d6_node_header_graphic
        - upgrade_d6_node_homepage_slide
        - upgrade_d6_node_job_post
        - upgrade_d6_node_landing_page
        - upgrade_d6_node_mock_panel
        - upgrade_d6_node_news
        - upgrade_d6_node_page
        - upgrade_d6_node_panel
        - upgrade_d6_node_program
        - upgrade_d6_node_quote
        - upgrade_d6_node_story
        - upgrade_d6_node_webform
      plugin: callback
        - '\Drupal\site_migrate\MigrateUtil'
        - nodepath

... And the class, in modules/custom/site_migrate/src/MigrateUtil.php:


namespace Drupal\site_migrate;

use Drupal\migrate\MigrateSkipRowException;

class MigrateUtil {

   *  $value
   *  mixed
   *  MigrateSkipRowException
   * Utility function to extract an integer from a string. Used to
   * extract the nid from a url_alias.src field for migration.
  function filterint($value) {
    if (!preg_match('/^node/', $value)) {
      throw new MigrateSkipRowException('Skip row that is not a node path');
    preg_match('/([\d]+)/', $value, $match);
    return $match[0];


   *  $value nid of node that needs to be written as an internal path
   *  string 'node/'.[nid]
   *  MigrateSkipRowException
   * Utility function to create a node path from a nid.
  function nodepath($value) {
    if (empty($value)) {
      throw new MigrateSkipRowException('Skip row if no mapped value');
    return '/node/' . $value;

And with just a bit of YAML and a helper class, we've fixed all the broken aliases from the previous migration!

Jul 14 2016
Jul 14

Richard asks:

Just a question after reading an article posted here back from January 21, 2016 on Drupal 8, why Freelock.com has not moved to Drupal 8?  Just wondering if there was a particular reason we should avoid before jumping in?  Thanks.

Ha! What a great question!

Three reasons: Time, requirements, priorities.

Time, Business Value

Major upgrades are time-consuming, especially if you have a non-trivial site. That translates directly to cost, even if the only cost is the time we spend on it -- that is time we don't have to spend on billable work or other priorities.

Any time you're considering an upgrade, you really need to weigh the cost against the expected "return on investment" -- any business person will tell you that. You also need to consider the risks associated with not upgrading.

For us, the current cost of upgrading is still high, and the added value is low. Our Drupal 7 site works perfectly fine for us. If we invest more time in it, we would improve the mobile experience first -- which we can do just as easily in Drupal 7 as in Drupal 8. There are years of support remaining for Drupal 7, so we don't really see much risk in staying put for now, other than the potential opportunity cost of not having our site be a shining example of Drupal 8 in all its glory, for the marketing benefit.

As it is, we think it's better to have really nice client sites to demonstrate our competence, and leave our site upgrade to another day.


Beyond that business assessment, there are two requirements we have that are not yet ready for Drupal 8: E-Commerce, and the Drupal 7 - 8 migration system.

Both are "close enough" that we could proceed, but we would have to invest quite a bit more time to make the move now, than if we wait another few months when these are completely ready.

Drupal Commerce for 8.x is still in Alpha, meaning they don't support upgrades to the next release (until they reach Beta status). We would feel comfortable building a new store in Drupal 8 now, given that it will likely be stable "enough" for use by the time the site is ready to launch -- but moving a store that is already in production? Not if it still has years of support remaining. Not yet.

The migration system, on the other hand, is getting very close. We've done several successful Drupal 6 - 8 migrations. There are still a few migrations that aren't there yet for Drupal 7 - 8 -- mainly entity references, some issues around taxonomies, and others.

In short, if you're coming from Drupal 6 it is time to move now -- you're on an unsupported platform now, and Drupal 8 is close enough just about everywhere, though if you're doing e-commerce, domains, organic groups, etc you need to be prepared for some upgrade pain...


As a business owner, there are always more projects you want to do than you can possibly get done. When looking across the portfolio of tools and systems we have and use, there are others that are in far more dire need of an upgrade than our main website. Our main client project management system, for example, is still running on an old OpenAtrium install on Drupal 6. We have 4 other internal sites that integrate with it, or we use alongside to compensate for its shortcomings.

We've limped along with these systems not quite up to snuff because we've been hard at work building out our DevOps systems, which are the backbone of our core products and services. Our priority, aside from customer work, has been building out the infrastructure to test and deploy critical security vulnerabilities across up to 200 Drupal sites in a matter of a few hours.

All the while doing our best to take care of our current clients, many of whom need their sites upgraded much more urgently than we need ours.

Our project management systems are next up on our internal queue for an upgrade... once those are dialed in, I'm sure we'll be migrating our public site soon thereafter!

Decision-making process

I think our decision-making process here is very sound, and I would highly recommend that other site owners use a similar process, asking some key questions:

  • Is the cost of doing an upgrade within my budget?
  • Is there a significant downside risk of not doing an upgrade now?
  • Is there a significant opportunity/benefit I can capture by upgrading now?
  • Is there some other business need I don't currently have met that might be possible to roll into this project?
  • Are all my requirements met by Drupal 8 today?
  • If my requirements are not met, can I live without them for the short term? Can I provide enough budget to get the missing pieces developed? Can I wait a little longer for them to get finished?
  • Out of all my priorities, is this currently the best use of my time/budget?

As always, let us know if we can help!

Jul 13 2016
Jul 13

Yesterday the Drupal security team gave a dire warning about extremely dangerous security vulnerabilities in multiple contributed modules. The fixes, and the details, would be released at 9am Pacific Time today.

I dropped what I was doing and started going through our customer sites, making sure they were all clean and ready for these updates when they were released.

When 9:00 this morning struck... it turned out to be completely anti-climactic. It wasn't the dozen widely used, highly vulnerable modules I was prepared to update on 60 different Drupal sites. It turned out to be... Three. None in our regular module selection.

Even so, I sprang into action. I went into our "configuration management" system, and did a search across the 39 servers and workstations we currently manage. Of the 3 modules that were affected, I found:

  • Coder, which had the most dangerous exploit, on 4 sites.
  • Restws, the other highly critical exploit, on 1 site.
  • webform_multifile nowhere on any site we've touched in the past two years.

Coder Exploit

The coder exploit is by far the most dangerous, as it can be exploited even if you have this module completely disabled. I found it on two OpenAtrium sites we manage, and quickly removed. I also found it on our main site, freelock.com -- where I could do a test to see if it was even exploitable.

And... due to the way we have our production web servers set up, it was not possible to exploit on our servers, because we do not allow any PHP execution in subdirectories on any Drupal sites. Which means that even if you had this kind of dangerous code present on your server, if we set it up for you in the past year, you're completely immune to not just this attack, but all attacks of this type!

How we can rush out an update to 60+ Drupal sites better than you can

It all starts with preparation. Every site we work on, we know what modules are running, we set up "visual regression tests" so we can see if a single pixel changes on a list of specific pages on your site after an upgrade, and we routinely check the production environment for anomolies that might cause problems when we need to rush something out in a hurry. We also make sure we have solid nightly backups of databases, files, and code if something truly goes awry (or your site gets hacked through some previously unknown vulnerability).

Before we can provide adequate maintenance

Most of the sites we maintain are not sites we built -- they are sites that others built, and the customers realized they need ongoing support, or the peace of mind that somebody has their back in situations like these. Most sites we build go straight on our maintenance plan -- with our Base Business Site this is included for the first year.

For other sites, we highly recommend our Drupal Site Assessment. When we do a site assessment, we compare every module in use on the site with the original state, so we can determine whether a prior developer changed it in ways that will likely break on an upgrade. Based on the site assessment, we will recommend a budget for getting the site fully cleaned up and up-to-date so that future security updates can be rolled out without catastrophe. We also set up the site in our "continuous integration" system, complete with setting up a base set of tests that will run on every update so we can catch things that break before they reach production.

Preparing for a big rollout

This is what I did to prepare for what could have been a much bigger deal today:

  • Made sure our inventory of maintenance sites was current, up-to-date
  • Checked each production site to make sure it is "clean" -- e.g. if there are any changes on production that might get clobbered with a release (we're on the verge of having this step automatically done, reporting unclean sites to us on a nightly basis)
  • Created a spreadsheet to index the current sites by version, server, server software, so I could keep track of sites that might have one-off affected modules, and could triage appropriately once I learned what the vulnerabilities were
  • Made sure our "continuous integration/deployment" systems were up and running and ready for a major workout!

Triage -- What happens when the vulnerabilities and fixes are announced

We have several tools all set up and ready to help us decide what or where we should focus our efforts first. These include:

  • Our "upstream" git repository, using gitolite. We curate a bunch of contributed modules here -- if any of these need an update, we can update it here and quickly pull it out to all of our sites, for Drupal 6, 7, or 8.
  • Salt, a server management tool that is the backbone of our server maintenance plan, and allows us to search across our entire managed infrastructure. This ended up being the only tool we needed today!
  • Matrix, a chat service, using Vector.im. This has become a crucial tool for us to manage all our projects, and we have our own custom bot that integrates it with our build system/test runner, Concourse CI. A full test run on any of our sites can take 10 - 15 minutes. Our bot tells us in Matrix when the tests are complete, provides a link to the test results, and lets us trigger the actual deployment to most of our production sites.
  • The spreadsheet we generated before this round of updates.

On this spreadsheet, I added a column for each vulnerable module, conducted a system-wide search for each module, and marked which sites had the affected module.

Then I reviewed the security notices themselves -- already able to ignore the modules we don't have anywhere.

For each vulnerability, my biggest concerns are potential data loss (SQL injection-type of vulnerabilities, like DrupalGeddon) and "Remote code execution" flaws that might allow an attacker to gain a foothold on the server.

For each of these, I then analyze whether the other security measures we have in place provide effective protection. On our servers, we have some very effective measures against remote code execution -- the web server itself cannot write to anywhere that code can execute, so if it's exploited via an upload, our servers protect for that. In the case of Coder, a different configuration we have in place prevents all direct access of PHP outside the webroot, and this proved to be effective on freelock.com.

Once I've done that analysis, I know which sites are most vulnerable, and to which vulnerabilities. At that point I order the list of sites for those I deem most at risk, and group them by version and production server environment.

Applying updates

List in hand, I can go on our dev servers and for any of the sites that are "clean" (e.g. don't actually have work in progress on them) I can apply the updates in about a minute, including creating the release and triggering the tests.

By the time I've applied all the updates, the test results start filtering in -- I go to Matrix, and review the results one at a time. If they look good, I add a note and trigger the deployment to production. If they don't look good, I send over to a developer to figure out why.

Then one more pass: I go back through the list in Matrix, where our bot kindly drops in the list of changes and whether or not it deployed successfully, paste the changelog into our Atrium project management system and notify the customer it's complete, or proceed with deploying manually if the automatic deployment failed (necessary for some sites hosted on shared hosts, or at Acquia or Pantheon which we haven't entirely scripted yet).

Scaling up

We were entirely ready for a dozen different security updates today, to deploy across 60 sites. All of the sites on our maintenance plan would have had the updates applied, with testing, within 3 - 4 hours of the announcement. We've spent a good part of the past couple years getting systems in place to make this rapid response a reality, and to make it so we can maintain this response time with 200 or more sites, with perhaps one or two more people on the team to help during these crunches.

Other companies provide tools so you can manage this type of upgrade on your own site, if you have the skills and knowledge. I don't know of any who offer the kind of testing we do with such a fast response time at such an affordable rate.

Let us know if you'd like us to handle your maintenance, keep your site protected and up-to-date so you can rest easy when other site owners are scrambling to make sure they are safe!

Jul 08 2016
Jul 08

Our branch strategy based on Git Flow did not survive. It was getting a bit old in the tooth, but the final blow was automation.

At Freelock, we've been hard at work building out automation so we can handle the maintenance on hundreds of websites with better test coverage and more confidence than ever before. Exciting news! It's all coming together, and we have it working across the board on ALL of our projects, now.

However, we've had to change our processes to support it. These really simplify what we need to do to roll out a tested release, so our partners and customers who work in our git repositories need to understand how it works now.


We still use git for deployment. And due to the nature of most of the Drupal sites we manage, we test against various copies of the production database -- which means the normal "continuous integration" (CI) and "continuous deployment" (CD) practices of building up a clean site just aren't very relevant -- the stuff that breaks is all related to things in the production database, so we have to test against copies of that.

Some of the tests we run use Selenium, PhantomJS, and direct access to resources like the various databases we have internally.

Site Copies

Based on these needs, we've decided to have two copies of every site running in our environment, mainly for testing purposes:

  • dev -- the development site, is where all new development happens, and most of the Behavior Driven Design (BDD) tests are run against this copy. Sometimes it gets stale -- generally we update this database from the stage copy before starting a chunk of work.
  • stage -- the stage site is where we test the actual deployment process, and run screenshot comparisons ("Visual Regression Testing"). It gets a fresh copy of the production database after every release, so its content should never be that far behind. The database gets sanitized to prevent accidental spamming of users and other mishaps.
  • prod -- Prod is what we call the main production site, and is compared to stage on each release candidate.

Site Release State and Versions

We've set up our continuous deployment to always have a particular status. What you can do depends upon the state of the current release of the site:

  • Released/Clean -- All code has been released to production. No changes currently exist on the dev site that have not been deployed.
  • Dev -- There are changes on the dev copy of the site that have not been added to a release. This is the state when there is work currently being done on the site.
  • Stage -- At least one "Release Candidate" has been created, and is being tested or is waiting to get deployed to production.

The CI system maintains a specific "version" for each site, and this also varies based on the release state:

  • Released: "final" version, for example: 7.1.5
  • Clean or Dev: "beta" version -- 7.1.6-beta.1
  • Stage: "rc" version -- 7.1.6.rc.2

On Drupal 6 and 7, the CI system writes the RC and Final versions to the settings.php file, in a $conf setting ("site_version"). This makes it easy to see what's actually running on various sites using "drush vget site_version". We are still working out how best to do this in Drupal 8 -- Drupal 8 does not care much for config objects that are not owned by a module...

Git Branches

We keep a branch that corresponds to each site copy. This makes it possible to do emergency releases that bypass current development if necessary, among other benefits.

  • feature/xxx -- These are arbitrary branches that the CI system entirely ignores. When working on something that is going to take more than a day to complete, the work should happen on a feature branch. If the Dev site is on a feature branch, it will block tests. Local development should be done on a feature branch, and merged into "develop" when ready to test and merge with other development.
  • develop -- This is the "home" branch for the Dev site. Whenever anything is pushed to develop, if the Dev site is otherwise clean the CI system will check out the develop branch on the Dev site, and merge in all changes from master and release branches, and kick off the Behat (BDD) tests on the Dev site. However, if the release state is currently "stage", or if the Dev site is on a feature branch, this won't happen -- the intent is to make us finish what we're doing before kicking off the next step.
  • release -- This corresponds to the Stage site. A push to the release branch will change the release state to "Stage", deploy everything to the Stage site, and kick off the visual regression tests. Generally if fixes need to happen, we check out the release branch on the Dev site, make the appropriate changes, and push them back in for a new deployment to stage.
  • master -- This corresponds to the Prod site. We restrict access to push here to designated release managers and the CI system.


The basic workflow we use is:

Feature/xxx -> develop -> release -> prod -> new development

We have a bot in a chatroom that facilitates moving stuff through this process, and triggering deployments and tests on demand.

Feature Development

When we're building out something major, work is done either on a local copy or on the Dev copy of the site. If the Dev copy is on a feature branch, it blocks all tests -- the assumption being we don't expect them to pass here.


When we're deploying updates, or merging in completed features, we set the Dev site to the develop branch, and/or push the develop branch (and have our bots set the dev site for us). This kicks off the Behat tests so we can see if anything breaks.

Site-specific behat tests are stored in tests/ under the root of the site. If there's no behat.yml file present, it runs a set of base tests we use across the board.


When the behat tests have passed to our satisfaction, and/or we're ready to roll stuff out, we kick off a release. Currently we do this by switching the Dev site to the release branch, merging in the develop branch, and pushing the release. We are thinking about having our bot do this for us on demand.

Whenever code is pushed to the release branch, it gets deployed on the stage copy. All features get reverted (in D6 and D7 sites) and configuratation applied (D8). For Drupal 8 sites, we also set the "site:mode" to production, turning on all caching functionality and disabling development/debugging settings.

After it's all deployed, our visual regression testing framework kicks in, using Wraith to compare Stage to Prod. The list of paths it compares is stored either in a .paths.yaml or a tests/paths.yaml file in the site, where we can easily update URLs to test.


Launching to production, for most of our sites, is easier than ever! A simple command to our bot will trigger the deployment on most of our sites. First the release code gets merged into master, and the final version number is applied and a matching git tag is created. Then the CI system looks up the deployment policy in ci/production, which can indicate whether to put the site in maintenance mode before deploying, whether this site needs manual deployment (e.g. on Acquia or Pantheon), if it should skip applying configuration, and other exceptions.

If it all goes cleanly, our bot reports the result back in the chat room. If any manual steps need to be done, it reports that.

On most of our sites, on most of our releases, we can now deploy with a single command in a chat room!

New Development

After a deployment is released, there's a cleanup phase to reset for new development. Currently this is manual, but our plan is to wait for the next production database backup to run to lessen the impact on the production server to do the cleanup/prep for the next round of development.

To clean up, this is what happens:

  • Version "patch" level gets bumped and set to "beta" - e.g. 7.1.4 gets set to 7.1.5-beta.1
  • New copy of the production database gets sanitized/imported on Stage
  • Dev copy gets reset to develop branch, all commits from release and master merged in, and for Drupal 8, site:mode gets set back up to the "develop" settings


This all certainly has gotten complex... and yet we're finding it's making our results far more reliable and consistent, and as a result, we're now able to focus on the quality of what we're doing. Now we're able to drop new Behat tests into place and have them run every time. Now we have systems to let us know when releases sit undeployed, allowing us to handle more sites, with more consistency than we've ever had before.

How do you manage your CI/CD projects? Anything obvious we've missed, or insight you've gained? Please comment below! And let us know if we can help you manage your Drupal sites...

Jul 01 2016
Jul 01

We have several Drupal 6 to Drupal 8 upgrade projects going on, which is particularly challenging given how quickly the Drupal Migration system is changing. Given that a couple of them are nearing launch, and were missing some node references, I set out to get the content updated from the production sites before launch.

When we started migrating these sites, you could run the Migrate Upgrade scripts passing the database credentials, and get a full list of what could be upgraded and what could not. And it would migrate all the stuff. The whole point of having a migration system like this is so that you can keep on incrementally migrating content until the new site is ready to go, do a final migration, and go live!

The first surprise: with migration in 8.1.x, that is no longer supported with the built-in automatic Drupal to Drupal upgrade -- it's a one-shot, one way deal.

Oh crap.

Fortunately, it's pretty easy to use migrate_upgrade to generate a custom migration, and with some database mangling, you can get your new migration to pick up the mappings from a previous auto-migration done in 8.0.x.

The main gotcha here is that node content still isn't showing up as migrated -- but for us, that was good enough -- that's what we wanted to update from production anyway. The big win here is that all of the rest of the dependencies for the migration -- users, input filters, taxonomy terms, blocks, menus, settings -- are all now mapped and so we can keep the changes we've made on the new site without clobbering with an entirely new migration.

Caution! Extremely technical backflips ahead...

Ok, not really... but this is tricky stuff, involving multiple wipes/reloading of your database. Make sure you do this all on a test copy!

Before starting: Gather your tools, backups

Before doing anything, I would recommend backing up your database. And make sure your tools are up to date. Here are the tools I use for this process:

  • drush 8.x
  • drupal console 1.0.0 (currently at beta3)
  • bash, sed
  • a working Drupal 8.0 site (it's ok if you've already upgraded it to 8.1)

Step 1: Back up database, upgrade all code

drush sql-dump > site-80.sql
(do whatever you usually do to upgrade)
drush dl --select migrate_tools migrate_upgrade migrate_plus # select the latest dev versions of these
drush updb
drush cr # cache rebuild
drush sql-dump > site-81.sql

Note: for some reason, drush dl migrate_upgrade wants to install in ~/.drush, and not in the site -- this caused me a bit of confusion, and lots of errors when trying to update! Be sure it installs in a location that doesn't get overridden by the 1.x version, which is not compatible with Drupal 8.1.

For changes as major as these, I like to have places I can roll back to. git tags can be useful, along with database snapshots at each step.

Step 2: Save your config

The config by default is stored under sites/default/files -- we exclude that path from git, and so we recommend moving it somewhere inside your git tree. You can set this in the settings.php file, to whatever path you would like:

$config_directories['sync'] = 'config/sync';

... the above configuration puts it under the site root, in config/sync.

Once you have that set:

drupal config:export
git add config
git commit

... and you have your current configuration exported and saved to code.

Step 3: Apply any core patches, prep the destination

In particular, we wanted node references migrated to D8 entity references: Add migrate support for Drupal 6 node & user reference

At this point you should enable all the modules that have migrations you want picked up, and generally make sure your target environment is ready.

If you make any further changes, export and commit your config again, and create a new database backup BEFORE creating your new upgrade.

Step 4: Create your custom migration module, and create the new migration

Here I borrowed heavily from other posts:

The gist of it:

drupal generate:module # Create a new module to contain the migration configuration
gedit sites/default/settings # Add another database credential array for the D6 database, using a key "upgrade"
drush migrate-upgrade --configure-only # Creates all the migration config objects it can find
drupal config:export

At this point you have a brand new migration in the config, and a module ready to store this config. Assuming your module is in module/custom/site_migrate:

mkdir -p module/custom/site_migrate/config/install
cp config/sync/migrate_plus.migration* module/custom/site_migrate/config/install/
rm module/custom/site_migrate/config/install/migrate_plus.migration_group.default.yml

... this exports your migration config to code, skipping the default migration_group that would conflict with migrate_plus.

Step 5: Mangle all the data

Ok. Now we're to the tricky part. We're essentially going to revert to where we were at the end of step 3, and then enable our migration module, with the mapping tables from the original migration.


git add modules/custom/site_module
git commit
git clean -df config/sync
git checkout HEAD config/sync

... to get your code base back to the right spot.

Now, the guts of the change. The new migrate_upgrade uses mostly the same migration names as the old, except with upgrade_ inserted at the beginning of the name. This means you need to rename your migrate_map and migrate_message tables to pick up those original mappings.

There's one other gotcha: at the moment, drush migrate-upgrade looks for a database key named "upgrade" but in the exported configuration, it changes to expect a database key named "drupal_6". So you'll need to edit your setting.php file to change that, after step 4.

Here's how I renamed the tables. Exactly what you need to change may vary based on the version of your mysql libraries and drush, but this worked for me:

cp site-81.sql site-mm.sql
sed -i "s/ALTER TABLE `migrate_message_/ALTER TABLE `migrate_message_upgrade_/g" site-mm.sql 
sed -i "s/ALTER TABLE `migrate_map_/ALTER TABLE `migrate_map_upgrade_/g" site-mm.sql 
sed -i "s/^LOCK TABLES `migrate_message_/LOCK TABLES `migrate_message_upgrade_/g" site-mm.sql 
sed -i "s/^LOCK TABLES `migrate_map_/LOCK TABLES `migrate_map_upgrade_/g" site-mm.sql 
sed -i "s/^INSERT INTO `migrate_message_/INSERT INTO `migrate_message_upgrade_/g" site-mm.sql 
sed -i "s/^INSERT INTO `migrate_map_/INSERT INTO `migrate_map_upgrade_/g" site-mm.sql 
sed -i "s/^CREATE TABLE `migrate_message_/CREATE TABLE `migrate_message_upgrade_/g" site-mm.sql 
sed -i "s/^CREATE TABLE `migrate_map_/CREATE TABLE `migrate_map_upgrade_/g" site-mm.sql 

Step 6: Run your migration!

Almost there...

drupal database:drop # this deletes any extraneous tables that might cause issues
drush sqlc < site-mm.sql  # reload the database with the changed tables, and without the second migration
drush en -y site_migrate
drush migrate-status

Presto! You should see most of your previous migrations, with most of the things already imported!

Now, if you migrate, you'll most likely get duplicate nodes of everything you previously imported. I would recommend bulk-deleting them, and then you should be able to proceed with:

drush migrate-import --all

... and get all node content migrated, along with any new users, menu items, files, etc.

But most importantly, you can now use the new migration improvements and same techniques as if you had started the migration in 8.1.x!

Thanks to Mike Ryan for assistance.

Jun 23 2016
Jun 23

It's really a shame. Drupal Gardens has announced to its users that it's shutting down completely on August 1, and users need to move away from the service before it disappears.

It's a shame because Drupal Gardens was the only low-cost way to run a Drupal site with somebody else handling maintenance for you.

But it's not really a surprise.

The Economics of Cheap Hosting

These days, servers are commodities. Disk space is cheap. Bandwidth doesn't cost that much unless you're doing video or get extremely popular. If you just put up a few static web pages, the cost is extremely minimal -- pennies a month -- which means that "cheap" hosts can do well charging you a few dollars a month.

But... Change out those HTML files for a content management system, a blogging platform, or anything that lets you go change content on your site without needing to actually know HTML, and there's a big, big, hidden cost -- the cost of keeping attackers from taking over your site.

The web is a nasty place. If you can log in to a site and change content, so can attackers, if they can find a way in. And... they do find ways in, all the time. It's a constant race to keep sites secure and easy for their owners to manage, while keeping the bad guys out.

"Cheap" hosts provide all sorts of easy installation options for Drupal, WordPress, and many other platforms -- but they leave it entirely up to you to keep those platforms up-to-date. And while all these platforms have mechanisms for keeping their platforms up-to-date, it's an extremely challenging problem to apply updates without stuff breaking, especially once you've started installing 3rd party add-ons that may depend on code not changing in substantial ways.

Maintenance is the big cost of running any website that is easy to update. No "Cheap" host does this maintenance for you -- it's expensive to do it right.

But what about SquareSpace, Wix, Weebly, etc? They are inexpensive?

"Software as a Service" (SaaS) is hugely popular right now. These are all proprietary hosts, which means they own the entire platform, and you rent a small piece of it. They are able to amortize the costs of keeping their platform maintained across all of their users. But even so, I think they often offer their platforms to end users at a loss.

There's a saying in the tech world -- "if you get it for free, you're the product!" If you're using a free service, you're either getting a bunch of ads, or the service is selling your information to other companies for one reason or another.

If you are paying for a service, the real question is, are you paying enough for it to last? Is it sustainable? For platforms that are extremely well designed and run, it very well could be the case that you are paying enough for the service to sustain itself -- but there is such heated competition in this space that companies backed by venture capital are under a lot of pressure to "capture market share" and so they almost always price the service below their cost in an attempt to get you to use their platform, instead of somebody else's.

Even the "successful" SaaS startups often get acquired by a larger player, and when this happens, the service often changes in ways its users don't like, and sometimes still get shut down. Google is famous for shutting off services because they only had a few million active users -- just not enough to be worth keeping going. At Google's scale, less than 10 million users can be considered "not successful" -- even though many SaaS services would be thrilled to get a few hundred thousand users.

What's different about Drupal Gardens (and WordPress.com)?

There is one huge difference between Drupal Gardens (and WordPress.com) compared to nearly all the other SaaS website builders: The code running DrupalGardens and WordPress is all open source, and so you can easily leave and run it on your own servers. That's something you just can't do on any of these low-cost proprietary SaaS platforms.

With Drupal Gardens, you can export your site and run it on pretty much any host that offers PHP and MySQL -- it's just a regular Drupal site. Same thing for WordPress.com sites -- you can export your content and run your own WordPress site without skipping a beat.

So with Drupal Gardens, the worst case has happened -- and there's a straightforward path to continue, you're not stranded. The one big thing that sucks is that now you have to take responsibility for doing your own maintenance, keeping the site up-to-date with security releases, because nobody at a "cheap" host will do this for you. And paying the actual cost of this will increase what you have to pay for the site substantially.

This is a risk you face with WordPress.com, and the SaaS providers, too -- with WordPress, you at least have the same way out -- with the SaaS providers, you're probably going to need to build a whole new site.

What about WordPress?

I don't have any inside knowledge here, but my guess is, similar economics apply to WordPress.com and Drupal Gardens. There's a big difference in scale here, in that WordPress.com has many more users than Drupal Gardens ever did, and there might be a bit more commitment on the part of Automattic (the company behind WordPress.com) to keep WordPress.com going as is, because it's a huge marketing value for the entire WordPress ecosystem.

Drupal Gardens was Acquia's attempt to do the same thing for Drupal, and that's why it's such a shame to see it go away -- it was a huge marketing benefit for Drupal to get users who could not afford to run and maintain their own Drupal sites, while providing an easy upgrade when they did reach that point.

So... Kudos to Automattic and WordPress.com for recognizing the value of a low-cost service for helping their ecosystem. I do truly hope they can continue -- I'm firmly convinced that the rash of SaaS providers and their entire short-term, venture-driven economics is harmful to the rest of the business community, and any open source-based offering hugely reduces the risk of catatrophe for your business.

Should I switch?

Well, any change of platforms incurs cost, and those development costs can be substantial. I like to take a longer term view of the value of a site to your business, and making sure you are budgeting appropriately to protect that asset.

If your website just isn't worth very much, you can probably put together a replacement relatively easily. Drupal may not be a good fit for you -- it may just cost too much to maintain for its value to your business.

Keep in mind two things though:

  1. You face the same risks with any platform change. If you go to any SaaS service, it could go away or change underneath you in ways that drive up your costs and no longer meet your needs. And this applies to WordPress.com as well, though as I mentioned there at least you have a way out.
  2. Drupal is growing in popularity for a reason: it can do extremely sophisticated things easily, that are much harder on other platforms. If you don't need anything special in a site, Drupal can do it but you may spend less getting there on other platforms. If you are managing any kind of data beyond blog posts and static web pages, you might find Drupal far more capable than other platforms -- you likely chose it for that reason in the first place!

What are my options if I want to stay on Drupal?

Acquia is steering people towards BlueHost for moving their sites, and they have decent documentation to help you make that move. But there's a huge difference between a Bluehost-hosted Drupal site and a Drupal Gardens site -- and that is that you are responsible for applying updates, maintaining backups, and everything else involved in running your site. There's a lot more to do than just log in and update your content.

If you neglect doing that work, you run the risk of getting hacked, losing data, hosting malware, or more. Drupal Gardens handled all this for you -- nobody else will do that for free, because it's a substantial cost. (Which is probably why Drupal Gardens is going away -- Acquia just did not see enough value from Drupal Gardens to justify the cost of doing all that work without charging sufficiently for it).

Freelock is in the site maintenance business, and if you have appropriate budget, we'd love to help. So here are the 4 options we can outline for moving forward:

  1. "Regular" hosting and maintenance. We provide a flat-rate maintenance plan that includes full testing of updates before they roll out to a production copy of your site. This is our main service offering, and we provide very inexpensive hosting if you want us to host it as well. (Note that hosting at this level requires a maintenance plan).
  2. Maintenance plan with site on your own server. We offer our maintenance plan for sites hosted elsewhere -- generally it's on a dedicated server or "virtual private server" that you host onsite or at a cloud provider. We also provide server maintenance if you need this as well.
  3. Freelock Shared Drupal hosting. This is our lowest-cost option that keeps your site safe. We do apply all updates, but we don't use our testing infrastructure for this -- which means things will break more often, you're responsible for fixing/working around broken features, and you don't get much of our attention -- this is basically the lowest rate we can offer that at least keeps your site secure, backed up, and running.
  4. Move to a basic host like Bluehost, and do your own maintenance. We do have customers who have hourly budget available with us, and they call on us when they need help with something. We're happy to assist on this basis, if you need a little extra, but need to keep your running costs down (as long as you're aware of the risks).

For any of these options, we will charge 3 hours at our regular rate to move your site from Drupal Gardens, wherever it's going.

Let us know if we can help!

Jun 18 2016
Jun 18

Its name is Watney. Watney lives in Matrix. Watney is a bot I created about 6 months ago to start helping us with various tasks we need to do in our business.

Watney patiently waits for requests in a bunch of chat rooms we use for internal communications about each website we manage, each project we work on. Watney does a bunch of helpful things already, even though it is still really basic -- it fetches login links for us, helps us assemble release notes for each release we do to a production site, reminds us when it's time to do a release, and kicks off various automation jobs.

Watney in action

Over the brief time of its existence, Watney continues to get smarter and do more for us. It listens in a special room for notices from a bunch of our systems, and when it hears something interesting from one of them, it can do some sort of privileged action. For example, when anyone pushes code into our central repository, it puts a message in the special deployment room, and Watney relays that to another system we use to manage all our servers... this makes it so that anyone on our team can change an alias to a site we manage, and within a minute or so that update is pushed out to all the development workstations, so our entire team gets the change and things "just work."


In the past year, Freelock has come to rely upon a bunch of new software systems I had never even heard of just 1 year ago. How we run all our work has changed pretty drastically over that time. Compared to how we worked just 3 - 4 years ago, what we do today is entirely different.Visual Regression Testing example

The goal is always to improve the quality of our work. The typical Drupal site we manage has over a 1/2 million lines of code, and there is a steady stream of security updates that change a bunch of different modules we use -- every now and then a fix to one module breaks another.

One of the biggest milestones towards that goal is getting a couple of entirely different kinds of tests running automatically, for every release. With 30+ sites that we currently work with, the big challenge is coming up with an automation system that can handle all the variations between these sites. If you're managing one site, and have a big team, this is pretty much a solved problem -- you deploy a "Continuous Integration" (CI) server like Jenkins, and set it to orchestrate all the tests you want to run -- but when you have active Drupal 6, 7 and 8 sites, some using "Features", some using Drupal 8's new Configuration Management tools, many not -- getting something that can run some standard tests across the entire set can be extremely difficult.

Especially when those tests become specific to the site, and not to the underlying code. We rely upon Drupal.org's unit testing to ensure that the core software operates the way it should -- but this doesn't cover whether the site looks the same after an update.

Continuous Integration

We had a single job set up to run with Jenkins, but basically all it did was attempt to copy the production database to our Stage environment, merge in the new code being released, sanitize the database so it doesn't accidentally spam (or charge users), and let us know when it's done so we could manually run tests. And that was proving very cumbersome to manage, daunting to improve to the level we wanted.

So a couple months ago I looked around for alternatives, and found Concourse. This CI system breaks up the jobs into much smaller units that can be organized into pipelines, flowing one into the next. The model for managing this seemed to be a much better fit for us than the old Jenkins system we had, so I dove into using it to build up our automations. And at the start it seemed to be a huge improvement, and I quickly got some of our more complicated tests kicked off and running, along with a bunch of different deployment jobs -- huge wins!

Concourse pipeline

I turned it on for a few projects. With 2 or 3 projects active, it worked fine, if a bit slow. But when I added a few more, it stopped working entirely.

The Rabbit-Hole

This led me down a rabbit-hole that took over a month to sort out. It turns out, Concourse was proving to be a huge resource-hog.

It works by defining a bunch of "resources" -- things like a particular branch of our git repository, a version number, a location to store test outputs, a notification -- and feeding those in and out of "tasks" that actually do stuff with them. Most of the resources are a git repository of some kind. And by default, Concourse would check every active resource for changes every single minute.

Our early pipeline had 5 - 6 of these git resources -- so when I turned on 6 pipelines I suddenly had 36 resources checking every minute. And I have over 30 of these to enable! And I hadn't even gotten that far on the pipeline getting built -- but we needed more hardware.

I ended up spinning up 7 different servers that act as "Workers", each one on beefier, more expensive cloud hosting than the last. I had to upgrade our main git server too, which couldn't handle all the requests. I dialed back the checking rate for most of these resources to once or twice an hour instead of per minute. I implemented a "pool" locking mechanism to prevent multiple jobs from running at once -- but even with all of these improvements and an expensive, high-powered server I could still only get to about 15 active pipelines before everything ground to a halt and lots of jobs started failing.

To get anything actually done, I had to "pause" most of the pipelines, and just turn on the ones for projects we were working on -- as long as there were only a handful of active pipelines, the system worked great.

I had one other problem -- I was developing a single template of a pipeline, but once that got loaded into Concourse, I had to remember to manually update it if I had changed the template (to run new deployment steps, for example).

The Insight

So there I was, thinking "This Concourse system is great, but I'm not going to be able to afford to run it." I was thinking it would probably take 4 or 5 expensive, active workers to handle the load for all our sites -- but these would sit idle most of the time, aggressively checking for changes that might happen a couple times a week for most of the projects we manage, only to benefit the handful we're actively working on any given day. Expensive to run, puts high demands on other systems we run, all for doing mostly nothing.

Hey. Watney already knows when something interesting happens. And Watney can already trigger jobs in Concourse. What if Watney turns on a pipeline when something interesting happens, and turns it off an hour later?

I checked some assumptions with Concourse -- jobs don't get interrupted when pipelines get paused, resources continue to get cached as usual when a pipeline is paused -- and I couldn't find any downside to this approach.

I implemented it that evening, taught Watney to activate AND update the pipeline whenever the git server reported a new commit on one of three managed branches... And overnight we had it running across ALL our sites!

The result

Today we have a full deployment pipeline that sanitizes and imports production databases to our stage environment, kicks off Behavior Driven Design and Visual Regression Tests for every release, deploys code and configurations to stage and production environments, manages a site verion number, and delivers test results to Matrix. It already handles a lot of variations of our sites -- it can grab databases from sites we manage on Pantheon and Acquia as well as self-hosted servers.

Now we get to focus on what counts -- what the site actually does, and the tests that verify it works. Now we can define tests specific to each site, and verify that those continue to work with each release.

Through automation, we're able to manage a lot more sites with fewer people, and deliver a better result. For the past 2 years we've been steadily automating a lot of the work we do, and now it's starting to pay off!

Apr 24 2016
Apr 24

I had the privilege of working with John and Freelock in launching a new Little League website. The process was flawless and the end product was magnificent exceeding our expectations. John knows his stuff! He had a wonderful ability to bring the perfect solution to our community based organization. Being volunteer run, we needed some special considerations in the way our website works, John understood this and delivered solutions that were perfect for us. We now have a cool website that also has the ability to grow with us into the future. I highly recommend John Locke and Freelock Computing.

Mar 20 2016
Mar 20

How quick do pages on your site load? How quick should they load? How do you make pages load quicker?
Why should you care?

The speed of pages loading on websites is often overlooked in web projects, and can be a major difference between two sites that otherwise look the same. Let's start with why you should care -- or rather, when you should care.

How fast should my Drupal site be, and why?

There is no universal answer to that question, but lots of generalities. The biggest question is, how motivated are your users to wait around? If they need the site to do their jobs, they might be willing -- although unhappy -- to wait around 30 or 40 seconds. But if your site is one option among many, and a primary sales tool, a site that slow will lose you tons of potential business. Search engines penalize sites that don't load within 10 seconds, but many quick browsers won't even wFreelock Performance Ratingsait that long -- if it takes longer than 3 or 4 seconds you might lose unmotivated surfers.

Do you care? How much are those surfers worth? If you have an ad-driven site, they might be a major source of your revenue, and that's critical. If your best customers are more patient, you might not care so much.

When we do a Drupal site assessment, we use a 1 - 5 scale based on what we think are substantial different tiers of performance. And these generally translate to how many potential visitors/users you have:

  • If your site loads in under 1 second, you're doing really well. Your site feels quick, nimble, not laggy at all. Performance is probably not an issue you need to spend more time or money on, unless you're in an intense competitive situation where 1/10th of a second (or less) differences count -- and then be prepared to spend a lot more budget to get those diminishing returns.
  • If your site loads in under 3 seconds, you're doing reasonably well. You're probably not losing site visitors due to having a slow site. There is room for improvement, but it's going to take some work to get there.
  • If your site loads in 3 - 6 seconds, it's middle-of the-road for performance. You're probably only losing the people with serious attention-deficits, but there's definitely room for improvement, and this is an area you may want to consider putting some budget.
  • If your site loads in 6 - 10 seconds, it's definitely on the slow side. If this is due to very large pages, you might especially be losing or frustrating some mobile users. This time frame is usual for processing-heavy web sites, e-commerce, custom portals, and other sites with a lot of content generated on the fly, and users who are already invested in what your site provides will stick around, but if you don't have a captive market you should at least get your landing pages responding quickly.
  • If your site takes longer than 10 seconds to load, you have a real performance problem, and you're almost certainly suffering due to that. You need to take action, unless your users have no choice but to wait around on your system -- and then you should take action anyway to avoid revolt!

How do you measure Performance?

We do a very quick benchmark using the built-in Firefox network profile tool. If you have Firefox, you can do this yourself:

  1. Right-click anywhere on a web page, and click "Inspect Element."
  2. Click the "Network" tab.
  3. Click Clock icon to start the profiling:
    Click the clock icon to measure page load time
  4. View the result:
    Slow page load

Note that the page load time can vary from page to page of your site -- what's most important are the pages your visitors first view on your site. For us, a blog page is far more important to improve than our home page, as we get far more traffic to our blog than our home page. Think about what is most important and start there.

Also see that there are two different sets of measurement: "Primed cache" and "Empty cache". A primed cache shows what it takes to load the second and later page views -- the empty cache is the one that counts for new visitors.

But... that is only part of the story, and only one crude measure. What that measures is the time until the browser says the page has completed loading, and stops spinning the "loading" icon. Your site may well be readable for visitors long before then.

Other performance measures

Performance is a complex subject, and there's lots of different parts to measure. And you might have different reasons for measuring, and attempting to improve, each:

  • Latency -- how long does it take from the time you click a link and your browser starts to get a response? This can be greatly affected by network conditions, including your local connection, as well as server issues, scalability, and other factors.
  • Time until page renders -- how long does it take before the site "looks" like your site? This might be a lot quicker than the actual measured page load, and may be "good enough" for your needs.
  • Time until a user can interact with your site -- once the page is visible, can your users click on links, open a menu, enter a term in a search box? Again, a user might find a portal site perfectly usable if they can click away to where they are going, before the page is completely loaded.
  • All resources loaded -- Drupal 8 with the "BigPipe" module has the capability to load some content into the page after the page is loaded. And lots of "single-page" applications load content through Ajax, a technique to get content from the server after the page is in the browser, which can make even slow web applications feel very responsive.

How can you improve performance?

We start by identifying what the underlying problem is, and the visible symptoms. Most often we're brought in to fix pretty egregious issues -- sites that are fine when traffic is light, but bog down at peak times, or particular pages that take a long time to load. And then many customers comment on how much faster their sites run when we migrate them to new infrastructure where we've baked in many system-level improvements.

Generally, these are the areas we most often dig in to improve performance:

  • System issues, hosting environment
  • Database slowness on particular pages
  • Caching
  • Code improvements to remove slow code
  • Browser-side improvements to make user interfaces "feel" quicker

System and hosting environment improvements

System-level improvements can greatly improve performance, along with scalability. These improvements become imperative on successful sites as they grow to support more traffic.

Most shared hosting uses the old "LAMP" stack -- Linux, Apache, MySQL, and PHP. While perfectly adequate to get Drupal sites loading in under 10 seconds, getting under 6 seconds can take some tweaking, and getting under 3 seconds gets near impossible for a non-optimized Drupal 6 or 7 site (Drupal 8 has some really smart caching that is optimized out of the box).

Our newer favored stack, Linux, Nginx, Docker, PHP-FPM, MariaDB, can more easily get you under 3 seconds, and with some optimizations, getting under 1 second is pretty doable on most sites.

Hardware itself counts -- the amount of RAM on a server directly translates to how many simultaneous requests the server can handle, beyond which performance tanks entirely. Solid-State Disks (SSDs) can noticeably improve performance too.

We just launched a site in a "regular" hosting environment -- a dedicated VPS with CPanel and Apache/MySQL -- and it turns out our slow, old development server has faster load times without optimizations than the Cpanel server configured for production. And our usual production servers are 5 - 6 times faster than our dev server!

And there are more tools available to speed up page load times, and other techniques -- moving to a dedicated database server, putting assets on a "Content Delivery Network" (CDN), adding a caching proxy server (Varnish), adding a database caching server (Memcached or Redis), or a search engine (Solr, specifically to speed up searches).

Database/Query issues

When particular pages are slow, the first place we look is a slow-query log. Drupal's Views module generally does a great job at writing database queries that perform just fine -- but now and then you'll put something together that the database just doesn't know how to execute quickly.

For MySQL and MariaDB, you can turn on the "slow_query_log" and see database requests that take longer than a few seconds, or run without using an index. On several sites we've seen queries that take 30 - 40 seconds to run, and some will run 5 minutes or more.

In some cases adding a database index can make a huge difference. The "EXPLAIN SELECT" syntax will break down how the database is putting together the query, which existing indexes its using, how many different rows it has to consider, and whether it needs to create and populate a temporary table to get a result.

Sometimes adding an index on some un-indexed column makes all the difference -- we've used this technique several times to shorten 40-second queries to 2 - 3 seconds, and sometimes to well under a second.

Caching Results

Sometimes you can't fix a slow query with an index. You might be able to rewrite the query, but sometimes it's much easier to just periodically run the slow query and cache the result for everyone else. And in certain cases, you might just be able to resolve it with a memory cache like Memcached or Redis.

One case we were working on this week has a view that generates a bunch of summary pages. On our dev environment, we run MariaDB, and this database had no trouble with the query that Views generated. However, the production server running MySQL was MUCH slower -- 35 seconds compared to 1.5 on our server. (This is due to the query having a "subselect", which MySQL has always handled poorly, and MariaDB has greatly improved). The quick fix: cache the view for 6 days, and expire the cache when new content is added.

Unfortunately in this case the strategy didn't work -- all in all this view generates ~100 individual pages, because some of the categories have several hundred items and only 20 are shown per page -- there were just too many pages that needed the cache regenerated while they were loading content.

But turning on a Memcached server may well solve a problem like this, because it can load the entire result set into RAM, and essentially provide a much faster temporary table than MySQL could do writing to disk.

Code improvements to remove slow code

This is where we get to break out all the fun tools!

If we still have a slow query, we might be able to rewrite it to take advantage of things we can index. Drupal's query objects can be manipulated to add and remove joins, change the structure, do things behind the scenes in code to improve what Views can generate. This technique can be very effective, but can limit your ability to edit the view in the future, so we try to avoid doing this for anything more than a band-aid fix.

In some cases we may need to re-structure the data itself. For one client, there was a webform with over 100 fields, and they needed a view of those results for their staff to download and process applications. To do this in Webform (at least in Drupal 6) this meant joining one webform table a hundred times -- something there was no way to effectively index! And while it worked for a few dozen submissions, by the time they had a few hundred the report became completely unusable.

We did find some indexes to add that helped, and we bumped the processing limits up to 5 minutes so they could at least get the job done, but as soon as their season was over we re-implemented the functionality using regular Drupal content objects (this was Drupal 6), which made handling this content far faster.

On another project there was a custom module that used a new (for Drupal 7) programmer interface called "EntityFieldQuery" and the Entity API module's object wrappers. This is a quite popular way for many developers to write custom code for Drupal 7 -- I think because they have the mistaken belief that Views has poor performance.

To solve this problem, we used a code profiling tool, something that analyzes every function call and reports how much time is spent in each function. Using this tool we drilled right in to find that a chain of entity wrappers ended up doing a single database call over 10,000 times in a single web request! Again, the number of queries was exponentially increasing based on the total number of items in the database -- so it wasn't noticed when the database was empty, but once in production it quickly became a huge problem.

For that case, we replaced the object wrapping with a regular view, and cut the page load time from 60 - 65 seconds down to 3 seconds.

Browser-side, UX performance improvements

With these, the fun really starts! Sometimes the web app is just the tip of a much bigger iceberg of data that simply cannot be processed quickly enough for a user's comfort level. While we have some tools like "BigPipe" and "Ajax Shopping Carts" that are already smart enough to load some slow parts of the page after the main page has been delivered, in many cases these end up involving some custom work.

Infinite Scrolling is one of my favorite user-side performance improvements. One example is Pinterest, which loads just a few cards at first, and as you scroll down, more cards load and you may never reach the bottom -- and yet the site loads quickly with just enough data for what you're looking at. This technique can be used with large tables of data, if you're working with some sort of data/information system.

Other techniques involve pre-processing or batch-processing information so it can be shown directly, instead of calculating on the fly.

This is one of the most exciting parts about Drupal 8 -- how much better suited it is than any previous version for loading bits of content as needed, and without needing to bootstrap all of Drupal for a simple request. This opens up a lot of opportunities to provide customizations of the web front-end, to hugely improve performance on complex web applications.

When should you care about performance?

I've spoken with very senior Drupal developers working on extremely large, high-traffic sites, and they claim to not care one bit about performance -- they care about Scalability. Performance is how quick a page loads for a single user -- Scalability is how many visitors can a site have before performance starts to suffer. The two are a bit intertwined.

So here are some indications that you should consider improving performance:

  • People are getting server timeouts.
  • Some pages never finish loading.
  • Users are frustrated because the site "feels" slow.
  • You know having a fast site is necessary for your brand, reputation, competitiveness, or other concern, and you have room (and budget) for improvement.

If you're looking to improve the performance of your Drupal site, contact us! We have a lot of experience solving performance issues of all kinds, and we'd love to help!

Jan 21 2016
Jan 21

Drupal 8 has been out for 2 months now, and there's never been a better time to choose Drupal for your website platform. Here are 8 reasons why!

1. Mobile Experience

Drupal 8 is mobile first. Every bit of Drupal 8 can be easily managed from your smartphone, with a responsive experience that works well right up to the widest displays.

You don't need a separate mobile site.

What's more, if you do have an app, Drupal 8's web services makes it a powerful platform for serving up content to mobile apps, other web applications, or any other place you want the data. "Headless Drupal" is fully supported, meaning whatever application your user interacts with can pull data from Drupal, without ever actually visiting the Drupal site itself.

2. Authoring Experience

In the past, authoring in Drupal has been extremely inconsistent, and entirely dependent on how much the initial developer put in place to make writing and editing easy. Out of the box, Drupal used to be very bare bones, but a large ecosystem of modules can be put together to make a really great editing experience, if the developer spent any time there.

Drupal 8 starts out with a very nice user experience right out of the box. The authoring environment has been revamped based on extensive usability testing, and now matches or beats anything on the market.

3. Expected Lifespan

Drupal has long been the go-to Content Management System for creating arbitrary databases of content online. For many smaller organizations, it easily replaces internal Access databases to become the primary business operating system. Larger organizations appreciate its strong group and role capabilities to manage access, permissions, and workflows.

Yet at its core it used to have structural problems that lead to mysterious, difficult-to-troubleshoot conflicts, extremely difficult upgrades, often incomprehensible behavior (to those not extremely experienced with the specifics of Drupal), and many wildly different ways of accomplishing the same task.

Drupal 8 changes all that. After 5 years of work, the core has been entirely replaced with an object-oriented system that sets it up for much easier upgrades going forward. All of the functionality is encapsulated and can be updated incrementally without breaking the rest of the system. There are strong guidelines that make it much more clear the best way to accomplish a given task, and far fewer conflicts.

Drupal 8 represents a maturing of the platform to a system that will be around for decades to come. Sites built in Drupal 8 today can expect to evolve with minor updates for at least a decade. Getting to Drupal 8 has been extremely painful, expensive, and a huge amount of work -- anyone using Drupal 8 now will gain the benefit of all of that hard work for years to come.

WordPress, in contrast, is in a time of great upheaval. At the start of 2016, there are forks of WordPress from developers who have lost confidence in the base platform, alternative admin interfaces that bypass WordPress core, and rumors of the entire core being rewritten in Javascript -- in short, WordPress is entering a phase of uncertainty while Drupal is emerging from one.

4. Performance/Scaling

Drupal performance is off the charts. Several groups are doing benchmarks related to various PHP versions -- the new PHP 7, Facebook's alternative "Hip Hop VM", and older PHP 5.6 -- but the key thing that stands out is how much more traffic Drupal can handle compared to WordPress.

Just picking out Drupal 8 with caching enabled on PHP7, compared to WordPress 4.4 with caching enabled on PHP7. One benchmark found Drupal handling 230 requests per second, compared with only 40 request per second for WordPress -- almost 6 times as much traffic. A slightly older benchmark found Drupal handling 917 requests per second compared to WordPress at 306 requests per second -- still 3 times as much traffic.

Drupal is a complex application that is capable of extremely interactive, customized functionality. Normally this would make it a performance hog, but due to extensive, smart caching, it runs on lower hardware specs than previous versions, and blows away the competition when handling large amounts of traffic.

5. Upgrade path

Drupal is a data processing engine. Drupal 8 includes a full data migration system in its core, with complete "ETL" capabilities (Extract, Transform, Load). This migration engine already migrates content from Drupal 6 and 7, and it's pluggable to migrate content from any data source.

We're finding that the data migration from Drupal 6 to Drupal 8 is much, much faster than the data migration from Drupal 6 to Drupal 7. To migrate to Drupal 7, first we had to build all the content types to contain all the data. Then we had to write migration scripts to map data from the old system to the new system. And only then could we actually pull over the data.

In Drupal 8, migration starts by pulling over your configuration. We skip the content type building, because the migration does this for us. The mapping then happens automatically. Within an hour we can have all the content migrated (or at least the script running that's doing all the hard work). And we can run an older Drupal site alongside a newer one, pulling over content changes as they're made in the old site until the new one is entirely ready to go.

This migration is extremely useful for integrating with other systems as well, not just one-time upgrades.

6. Growth Potential

The reason Drupal is such a popular platform is how much you can do with it. And one of its big hidden strengths is how easy it is to change and grow with your organization. You can start very small, with a basic brochure site, and gradually add community features, e-commerce, groups, workflows, publishing engines, rating systems, scheduling systems, mapping systems, or whatever else you can think of.

It does not all need to happen at the start.

As the business world changes to rapid delivery, speed and the ability to change your mind about what is important is what we think the killer feature of Drupal. Change always comes at a cost, of course, but that cost can be much lower with a system designed to accommodate change. And that's what Drupal does best.

How Drupal accomplishes this "under the hood" has changed entirely -- it now uses much more standard industry practices of object-orientation and dependency injection instead of a convoluted and confusing hook system. This just makes the end result far more dependable and reliable -- the promise of Drupal can now be delivered more reliably with fewer conflicts, and less need for Drupal specialists.

7. Quality results

As web applications get more complex, quality becomes a harder job. We used to target one size of screen, and now we have phones, tablets, notebooks, desktops, and widescreen tvs -- and what will come next? Voice interfaces, artificial intelligence agents, further personalization? The web world is exploding with complexity.

Fortunately, so are the tools we have available to both develop and test the results. Drupal 8 has an order of magnitude more testing happening than older releases, and not just functional tests of code. There's a "Behavior Driven Design" framework baked in, called Behat, that lets us write tests for how users interact with a site. We have visual regression testing tools to compare every pixel on each page before and after an update.

Drupal 8 is higher quality than ever before, and when managed by an experienced team, this leads to better results than ever before!

8. Ownership, and no vendor lock-in

Like other open source solutions, Drupal at its heart is completely free. It has no licensing costs whatsoever, and because of this there are thousands of vendors you may choose from when you need help. You control the pace of change, you have full access and control over your data, the timing of upgrades, and what your site does.

Compared to Software as a Service (SaaS) vendors, you are not stuck with an unresponsive vendor -- you own everything, instead of being at the mercy of a SaaS landlord who might make decisions that have a big impact on your site.

Bonus: Freelock support

At Freelock, we focus on ongoing "Dev-Ops" of Drupal sites. We provide all the technical support necessary to operate, grow, and get full value out of high-quality Drupal sites, from server and website maintenance plans to testing to creating new functionality to end user support and training.

We can keep your site up-to-date, secure, and mitigate the risks associated with hardware failure, vendor changes, hacking, and data corruption. We actively look for ways to make your Drupal site better support your organization's goals.

If you're looking for an experienced, reliable partner for your Drupal technical needs, you've come to the right place. Contact us and let's work together to make your organization more successful!

Jan 13 2016
Jan 13

We're digging into our first dry-run migrations from Drupal 6 to Drupal 8, and learning a lot about what's ready and what's not. And there's some surprises in there...

Surprisingly good

A few of the items migratedFirst of all, Drupal 8 is pretty fantastic out of the box for simple sites. And the amount of stuff that gets migrated from Drupal 6 to Drupal 8 is pretty astonishing -- the migrate devs have done a great job of getting a lot of configuration migrated cleanly in the upgrade.

Configuration management is also insanely great. A couple times we've deployed development things that blew up the site completely, to the point you could not load any page at all -- and a simple config edit let us turn those problem parts off and resume work in a matter of minutes, without even knowing how to recover from this minutes before.

And of course we already knew the editing, authoring, media management, mobile, theming, and overall user experience is a vast improvement over Drupal 7.

Surprisingly close

At DrupalCon Los Angeles last June, and even at this fall's NW Drupal Summit, it seemed that a bunch of large, critical modules were really far off, many not even begun. As I run down the list of modules needed to upgrade some of our Drupal 6 customers, I'm a bit surprised to see a lot of progress on many of these:


In the Drupal 6 to 7 upgrade cycle, the big missing module was Views -- without a stable Views, we didn't find a customer comfortable doing a new site in Drupal 7 until some 10 months after Drupal 7 first came out. Now that Views is in core, Drupal 8 is so much more usable right out of the gate. But over the Drupal 7 cycle we came to adopt Panels on a majority of our sites, and find it invaluable to replace hard-coding stuff in themes, greatly expediting development.

Because we do have to re-build themes for any new Drupal 8 site, migrated or not, tools like Panels are a mainstay.

Panels and Page Manager both have alpha releases available, and an expected milestone of mid-February for having something mostly useful here.

Drupal Commerce

Commerce for Drupal 8 sounded a year off the last I checked, but there's already an alpha release. I suspect it is still a long ways from having the support modules needed to having everything necessary to actually get a Commerce site up and running -- payment gateway modules, rules, and much more. But to have the core Commerce module there and available to start building with is extremely promising.


This is a huge surprise -- I was under the impression there would be no more Ubercart after Drupal 7. But there's an alpha release for Drupal 8 already, too!

I'm sure we'll see similar necessary modules not available for a while -- but for those people still on an older Ubercart, this right away promises to make a much more straightforward upgrade path.

Organic Groups

Organic Groups has a few companies providing part-time dedicated development help to get OG up for Drupal 8, and it looks like development is going very quickly. No releases yet, but it's looking like there will be something working much sooner than I expected.

Surprisingly not there yet

Now, for the let-downs. There's a couple really critical things that still look a ways off. If you need this functionality, you'll either need to wait, or provide budget to help get these across the finish line. (We'd be happy to develop some of these for you, if you have the budget to make it possible!)

Date API

This one is a bit shocking. Drupal core now has date-time fields right out of the box. But they're relatively simple, and missing a bunch from the former "date" contributed module. Specifically, a "to" date-time, and recurring dates.

This means if you want to create any kind of schedule of any kind of event that has any kind of duration, you're out of luck. Dates in Drupal 8 are a single point in time. No class calendars, no schedules, no repeating events, nothing.

Date has always been an extremely buggy, frustrating module to use -- but that largely reflects how tricky it is to program around dates and times and get it right. (I have a whole other blog post brewing on this topic).

The worst part of all this is that I don't see that anyone has stepped up to take ownership over this -- I don't see any sign it will get built any time soon.

Again, if you need this and have some budget to make it possible, drop us a line!


Rules is a fundamental module for any site with anything sophisticated going on. And I would think this would be a relatively easy module to port to Drupal 8, given how it's mainly a structure for stringing together conditions and actions. And yet there seems to be a lot of open issues before we even get to the point of being able to create rules.

We made extensive use of rules, and exporting them to features, to eliminate the need for a lot of one-off custom modules in Drupal 7. It's a bit disappointing to see that it still looks several months off before we'll have a working module here -- and this is going to hold up migrations of lots of complex sites, e-commerce, group sites, and more.

A lot of what rules does can be done in code, but this to some extent takes away some of the admin benefits of using Drupal for these kinds of things in the first place.


Another surprise -- we use Feeds on so many websites, I thought there would be enough demand that there would have been a Drupal 8 release months ago. A Drupal 8 version has been started, but it looks like there's a long way to go before it will be functional. I almost wonder if it's easier to build something on Migrate, now that migrate is in core?

Overall Status

We have yet to complete a migration, and only have a couple internal sites actually launched on Drupal 8 as of right now. But we've done some deep analysis on 3 different Drupal 6 sites so far, and we're figuring on a range of ~50 - 200 hours to do the full migration of some relatively simple Drupal 6 sites to Drupal 8, including building out a new Bootstrap-based theme and migrating all their content. And the high end of that is actually on some sites that were built using a huge number of redundant modules and poor site building practices, allowing budget to sort all of that out.

The biggest thing I'm noticing is that there are a lot of new modules specifically for Drupal 8 that replace different modules for Drupal 6 and 7. So much has changed under the hood, and it does look like there's been a lot of churn among module maintainers, so a lot of modules that aren't there yet for Drupal 8 have alternatives that are ready for use.

Things like:

  • Crop API, Image Widget Crop, EPSACrop for image manipulation all to be available and working, while Manualcrop and several others aren't even started
  • ng_lightbox is ready, Colorbox is in development, Lightbox2 appears to be stalled out
  • yamlforms is in production, Eform mostly works (replacing EntityForm) but Webform is a long ways from being there
  • Media modules are getting completely reorganized. Read this post from the Media folks for more details, much of which works now but needs to get built out.

The other big thing that has changed is the whole deployment process, the behind-the-scenes details of staging updates and moving them from development to stage to production.

We've been hard at work on our automated tools and processes to manage both deployments and testing, and these are going through an incredible amount of change right now. At the same time, we're trying to keep the billable work moving -- it's pretty astonishing how rapidly everything is changing right now.

It's a very exciting time, and there's still a ton of work to be done. If we can help with your web application needs, please reach out and let us know!

Nov 20 2015
Nov 20

Today marks the release of Drupal 8, and the birthday of its founder, Dries Buytaert. This release is more than just a new digit, it's an entirely new platform with something for everyone to love, but it's particularly big for web site owners.

What's the big deal? The biggest, most powerful, one of the most successful open source projects in the world has two major, fundamental changes that change everything you thought you knew about it.

Well, not everything. Lots of the things that people love about Drupal are getting some nice improvements:

  • Mobile, responsive support straight out of the box -- it's actually a nice experience administering a D8 site on your phone!
  • The information architecture is the same as always -- content types, vocabularies, comments, anything you've learned about how content is organized in Drupal is the same, but...
  • Lots of powerful modules like Views and CKEditor are now in core, and much less quirky than ever before
  • Deploying updates and configuration settings between multiple copies of the site is completely overhauled, and now very simple to do
  • Caching support is baked in, enabled by default, so in spite of doubling in code size it uses less computing resources and responds much faster, especially on busy sites
  • Loads of other improvements.

All of that is great, and we could go on for hours, days about how much of an improvement this is. And that is the stuff you will notice today, next week, next month. But that's not the big change, that's not the killer feature of this upgrade for site owners.

The killer feature is what happens in 6 months, in 1 year, in 5 years. And that is, the great big upgrade cost you don't have, when it's time to upgrade to 8.1.0, or 8.2.0, or even 9.0.0.

Two Fundamental Changes.

Ask any Drupal site owner where their biggest pain is. You'll run into two big complaints: How hard it is to find decent Drupal developer talent, and how painful it is to migrate to the next version. Those both change with Drupal 8.

The next version

The current version of Drupal 7 is 7.41. The new version of Drupal 8 is 8.0.0. The next version of Drupal 7 will be 7.42. The next version of Drupal 8 will be 8.0.1. Notice anything different? It's called "Semantic versioning," and yes, it's just one more number between the dots (or added to the end). But the change behind that simple little version number is enormous.

Drupal is changing its entire release process to have "minor" version releases every 6 months. That means 6 months from now will be 8.1.0, and in a year, 8.2.0. These are calendar-based releases that contain new functionality, AND maintain backwards compatibility with the previous minor version. Upgrades from 8.0.4 to 8.1.0 should be completely transparent, nothing breaking as a result -- but new stuff available.

Drupal has never maintained backwards compatibility like this before -- this is a fundamental change in the project, and it represents the maturity the platform has reached.

There will only be a Drupal 9 when there's enough changes that are not backwards compatible that it's time for a new major release. But this "minor release" plan provides plenty of notice of functionality being deprecated to allow people to transition away from those things that are going away, long before 9 arrives.

That means an update to Drupal 9, will mostly be a matter of making sure you've either moved away from stuff being changed in Drupal 8, or have added an alternative. And then update, potentially like any other minor release.

No more completely rebuilding your site in the new version! For the first time ever, major version updates in Drupal should be relatively painless, as long as you keep your site relatively current and pay attention to changes as they develop.

"Drupal Developers"

Drupal has always come with a steep learning curve, particularly for developers. This is because it has developed out of procedural code, with a "hook" system and naming conventions that make a lot of things happen "automagically". It takes a couple years to get your head around the many Drupalisms, code patterns, hooks, conventions that are not seen or used in most other projects. You need to be very proficient in coding, using a debugger, and having an open mind to be a good Drupal developer... until now.

"Object Oriented" is a term that came in vogue in development circles... in the 1960s. It became the dominant way of programming in the 1990s, particularly with the rise of Java in popularity, and it's at the heart of .NET as well as many open source practices. And while Drupal uses a lot of object-oriented concepts in its information architecture, it has never been fully object-oriented in its code... until Drupal 8.

Why should a site owner care about this? Two huge benefits -- the same two I'm talking about here:

  • Drupal development now shares the same programming architecture as 90% of the rest of the industry, instead of being its own thing. Now you don't need to find a good "Drupal developer" -- a good developer should be able to pick it up and figure it out without years of learning the specific incantations and magic charms of all those Drupalisms.
  • Updates. Because we now encapsulate all this code into objects that extend other classes, this allows for upgrading smaller bits of functionality without affecting the rest of the site. This means that it should be possible to upgrade some modules to Drupal 9, before the site itself.

I think a lot of people in the Drupal community don't fully realize how huge a change this is (and it is interesting to see some backlash to the changes from those who may fear some of this change).

In other words, when Drupal 9 eventually arrives, it won't be such a big deal -- it should be possible to run exactly the same contributed modules for Drupal 8 and Drupal 9, with no changes whatsoever -- and even if something important does need to change, it can be changed by inserting a "shim" class that translates the API changes as appropriate -- it will almost certainly be possible to run Drupal 9 modules in Drupal 8, and vice versa. And you won't have to find a Drupal-specific developer to do this for you, either.

The new world of web applications

Drupal has long been a compelling platform in terms of functionality, the speed that new functionality becomes available, and the power built into the system. Drupal 8 is not just another release -- it is the maturing of this platform into something that is completely up-to-date and capable of staying that way for at least the next decade, if not more.

If you are looking for a new content management system, a new project management system, a new platform for managing all kinds of communications between groups of people, you can't pick a better base for doing so than Drupal 8. Give us a call, and let's discuss what you want to build!

Oct 12 2015
Oct 12

When I first handed John an initial disaster- an unfinished website that had to be completely rebuilt, he took all in stride and developed our site from the ground floor up, with speed and expertise. In addition, John has continued to take initiative in making our site to be one that we are extremely proud of. John continues to be a collaborative partner in the maintenance and continued development of AnswersForElders.com. I highly recommend him and his team for all your open source web needs.

Jul 25 2015
Jul 25

Faster, more secure, more maintainable. Three nice benefits we get from our new standard Drupal server architecture.

This year we're replacing our old "traditional" LAMP stack with an entirely less pronounceable LNDMPS version. We still use Linux, MariaDB and PHP, of course, but instead of Apache we've moved to Nginx, and we've added Docker and Salt.

If you're a technical person and haven't heard of Docker, you must have been offline for a couple years. Docker is a system for managing Linux containers. You might think of Linux containers as a form of "cheap virtualization." However, the way the Docker community has come to use them is more like a chroot jail -- a way of isolating a single process into a container that protects the rest of the system if that process gets compromised, and not a full operating system with multiple processes. So the best practices are to have a different container for each necessary service, not a single container with the whole set.

Host layout, container linking

So far we have put PHP, MariaDB, Apache Solr, and a bunch of other supporting services into containers. On our production servers, we have kept Nginx, Postfix, and DNSMasq outside the containers and installed directly on the host.

When using Docker, each container has a different IP address bridged on the host, and can expose different TCP ports, or sockets in the filesystem, to allow connections. We configure DNSMasq on the host so that you can reach any container on the host by its name, and set each container to use the host's DNS. Using this pattern, we can link all the containers together by a simple hostname, without having to link specific containers together.

There are a couple gotchas with this approach:

  • There is a lag before the host DNSmasq populates new container addresses -- we currently have an update script that polls every 10 seconds for new changes, and Nginx needs a reload to see a change in a container IP address.
  • We ran into problems having a wildcard DNS entry set up (*.freelock.com) when the hostname was part of that domain -- most requests for a hostname that was not fully-qualified resolved the wildcard DNS entry instead of the local DNSMasq entry. We didn't end up solving this -- we just removed the wildcard DNS entry and then everything worked fine.

We also run a local Postfix instance on the host that relays mail to our mail server. Inside each container that might need to send email, we've added/configured SSMTP and pointed it at the host.

Also on the host: all data, databases, deployment tools (git, drush, composer, etc), cron jobs (using drush for Drupal cron).

Security improvements

Our philosophy on production Docker use is that containers are ephemeral -- they get destroyed and recreated all the time, and so there should not be anything in the container we care about losing.

Docker is still a pretty young technology, and it clearly opens up some new avenues of attack, which need to be carefully considered. For example, adding somebody to the "docker" group is quite effectively giving them full root access to the entire host. It is early enough days that I'm not entirely confident somebody gaining a shell on the host couldn't somehow attack the Docker process itself to gain root access and modify whatever they want on the host.

That said, I'm reasonably confident that Linux containers do an effective job of isolating processes running inside them to just what's visible to those processes. And the vast majority of attacked websites we see (5 so far this year) plant a malicious script in an executable environment and run it. So the first place to secure is the executable environment -- PHP itself.

In our former LAMP setup, we would use filesystem permissions to prevent the Apache web user from being able to write to the filesystem, anywhere other than the directories with assets Drupal manages on the disk (images, videos, aggregated CSS/JS, etc) and prevent execution in the directories where Drupal can write.

With Docker, we take this one step further: we mount the web root into the Docker container as a read-only volume -- so even if an attacker somehow gained root through the PHP interpreter itself, the attacker still cannot plant their executable code into the site.

A couple other big wins here: we support a few "legacy" installations of older Drupal and some non-Drupal sites -- we're now able to contain and isolate those from other sites on the same server, so a compromise on one site cannot hop over and infect another site. And we're able to easily run different versions of PHP on the same server.

Performance improvements

Docker gives us no performance improvements whatsoever, but it doesn't penalize us either.

Our new architecture, running Nginx and PHP-FPM, is providing very noticeable performance improvements. Sites that formerly took 3 seconds to load now take 0.3 seconds for the HTML, and by moving away from mod_php, the webserver can handle many more simultaneous connections for downloading assets.

The speed improvements are especially noticeable for admin users working on their sites.

Managing upgrades

I keep hearing people rave about how much easier it is to manage upgrades with Docker. And I pretty much completely disagree. In a production environment, if you want to upgrade a container for a new PHP release, for example, it's quite a bit more complex than a simple "apt-get dist-upgrade." You have to:

  1. Build a new Docker image.
  2. Stop the old docker container.
  3. Remove the old container.
  4. Create and run a new replacement container based on the new image.
  5. Fix all the links with the other parts of the system.

This is not simple.

Fortunately, Salt can manage this entire process!

Salt is a configuration management tool, very similar in purpose to Puppet, Chef, Ansible, CFEngine, and others. I think people who think Docker is a useful deployment/configuration tool don't have experience with one of these far more powerful configuration management tools.

These days Ansible seems to be getting a lot more attention than Salt -- my impression is Ansible is easier to learn, but Salt is more powerful. (That sounds familiar!) We've built up some Salt states to manage both containers and sites in containers.

So to upgrade a container, our process now looks like:

  1. Build a new docker image.
  2. Push into our private Docker registry.
  3. Run salt state.highstate and let Salt do the rest of the work!

Salt pulls the new image down to each host, and if it detects that the image a container is based upon has changed, it stops/removes the container and starts a new one in its place. Then the "update-dns" script we deployed earlier (also via salt) detects the new IP address for that container's name, and reloads Nginx.

This process has not been flawless, and so at this point we're running the "highstate" command that applies these updates manually, so we can address any issues that might arise -- so far we've had two failures, both of which I chalk up to the relative immaturity of Docker:

  • Docker container filesystem type -- this is configured when Docker is installed. On an Ubuntu host, it sounds like the current recommendation is the older AUFS filesystem, on other systems, Devicemapper seems to be the current standard. Our original systems ran AUFS, and AUFS upgrades have gone smoothly -- however, the systems we deployed with Salt ended up using Devicemapper, and that's broken multiple times when Docker itself came out with a new release, breaking all containers. We've eventually switched all our hosts to use AUFS and haven't seen further issues.
  • docker-py unable to pull from a V2 registry, errors on container creation -- You can (and should) run your own docker registry to store docker images. This allows you to build an image once and distribute it across your infrastructure. We only started using Docker a few months ago, so we never bothered to deploy a version 1 registry, went straight to V2. However, the Python docker-py library which Salt and Ansible use to manage Docker has lagged behind the Salt API, and again after various upgrades have suddenly stopped working, sometimes at the worst possible time.

We're nearly ready to turn the automatic "highstate" back on, but we want to go through a couple more upgrade cycles to make sure this goes smoothly first. As Docker and the tools mature, I'm sure these issues will be far less frequent.

The biggest improvement

Docker containers are cool. Docker is fun to work with. It's great to be able to quickly roll back to an earlier version of a container -- particularly some of the one-off servers we support, Docker makes us feel far more confident in the environment and in being able to quickly roll back the entire server to an older version.

But the biggest improvement we're seeing is in the process of creating the docker containers in the first place: the Dockerfile and the startup scripts. None of this is something that we couldn't do before -- it's just that we didn't do it before, we didn't map out the steps of creating our production environment. We had a bunch of ad-hoc scripts and miscellaneous Salt states to assist with adding sites to a server, tuning MariaDB, etc. But this was all jumbled, messy, and hard to maintain.

Now we're getting a much cleaner and well-defined separation of a standard environment build, and the run-time configuration. And that starts with the Dockerfile.

Docker images

You don't have to use a Dockerfile to create a docker image -- you can just start a container based on somebody else's image, make some changes, and "commit" it to have an image. But then you end up with an image that's hard to replicate if something upstream changes.

I've read about people using their configuration management system to update their containers, but this is backwards -- you need to install more software inside your container to make it capable of being managed!

The Dockerfile is a simple recipe for creating an image, and "docker build" is baked into docker itself, it's incredibly easy to use.

We see the Dockerfile not just as the recipe for creating a Docker image, but also a self-documenting map of the configuration itself. You do need to think about what variables need to change in different container instances -- for example, our PHP images can be tweaked by passing in different variables for max ram, max clients, and max execution time, and we declare these variables in the Dockerfile so it's easy to see what parameters you can pass at runtime, even though they're not necessary to build the image.

Even though we typically run Ubuntu servers, most of our Docker images end up based on Debian Jessie or other derivative images.

So we have a git repository of Dockerfiles for all our images. When doing an update, we "docker build" the new image and push it into our registry. This means we've been able to move a lot of the build information out of Salt into Docker.

Runtime configurations in Salt, Startup scripts

Inside a Docker container, you generally don't run any kind of init system -- you just run the service itself, in the foreground. So most images need to have the ability to set up the environment before starting the service, and we end up using a hand-built bash script for this. Most often, this script simply replaces values in configuration file from the values of environment variables provided to the container when it's started, and then starts the service.

You do need to consider that this script will get called whenever the container is started -- both the very first time it's launched, and also if the host gets rebooted or the container stopped/started for any other reason. Other than that, the startup scripts are very straightforward, because you don't need to consider shutdowns, status checks, or any of the other things you typically need in an init script.

The catch is, you can't change any of these environment variables after the container is created. So that means if you do need to change them, you need to create a new container and replace the old.

If you have a dozen variables to set at container creation, you need to keep track of those somewhere. We started by just putting it in our project management system, and cutting and pasting the startup line. Docker is developing "docker engine" for this purpose, to allow you to store these variables in a config file for easy startup, and orchestrating multiple containers.

But we've found Salt able to handle that task very, very well. We've built out a set of salt "states" that automatically provision the necessary containers, in a very elegant way, with the run-time values stored in salt "pillar".

Because we're primarily managing Drupal sites, we've set up pillar data to make it easy to focus on one layer at a time:

  • sites/sitename.sls -- contains information about a particular site: drush alias, git alias, public URLs, database credentials, site root path, assets path
  • server/servername.sls -- defines which containers to create on a particular host, based on which images, along with any changes to the runtime defaults and a list of sites to mount into that container. It also includes each site state file to provision on that server, and designates which container it runs in.

Once these pillars have been populated with appropriate data, Salt now ensures that a whole bunch of configuration is done on the server:

  • Latest site code is checked out of git
  • Permissions are set correctly for the site code and assets
  • Containers are running with the latest images
  • Site code is mounted as a read-only volume, and assets mounted as read-write inside the appropriate container
  • Nginx has a site configuration for each site, pointing to the appropriate PHP container
  • The Drupal settings file is written with appropriate database credentials and any other variables specific to the production environment

... and there's a couple easy next steps we haven't quite gotten to: scheduling the Drupal cron job in the host, and setting up the user account inside the database.

Other than that, completing a move of a site into this architecture consists of importing the production database, copying over the site assets from wherever they are, running "highstate" again to fix the permissions, and updating DNS to point to the new server!

From a disaster recovery standpoint, this is huge. We used to spend a couple hours dialing in the environment on a new server, working from checklists and error messages as we build it up. Now we simply copy the container configuration to the pillar for a new server, run highstate, and import a backup site database and assets and we're off and running.

This is a bit of a long and rambling post, but I hope it illuminates the big picture of how we use all these different systems to deliver what's really becoming a great result -- a fast, secure, replicable environment for running Drupal sites. We're still new to Docker, but happy to share our experiences further, and if you have any suggestions, questions, or obvious things we're missing, please comment below!

Jul 01 2015
Jul 01

We hear it all the time:

Why do you recommend a 6 hour budget for a simple integration? Here's an embed widget right here -- if this were WordPress I could do it myself!

Well, in Drupal you can do it yourself, exactly the same way you might in WordPress. Add a block, use an input format that doesn't strip out Javascript, paste in your code, put the block where you want it on your page, and away you go.

It's pretty spectacular what you can do by mashing up a bunch of widgets on a webpage these days. There are tens of thousands of "software as a service" providers that provide embeddable widgets you can put on your site, and have all kinds of cool things: Twitter feeds, Google Ads, Facebook Likes, analytics (a.k.a spying on your visitors), maps, upcoming event widgets, data collection forms, CRM capture forms, live cam feeds, games -- the list is endless.

But... this entire approach has a dark side. Several of them, in fact!

It hurts your page load speed, and potentially loses visitors

Google takes page load speed into account when indexing your page. Very slow-to-load pages are penalized in search engines, largely because they've learned that people don't stick around waiting for pages to load.

Sample page load from already-primed cacheWe check the page load times for our maintenance customers each month, checking times for both an "Empty cache" (e.g. the first time a visitor loads a page from your site) and a "Primed cache" (e.g. subsequent page views). We aim to get the empty cache load time under 6 seconds, and a primed cache load time under 1 second if possible, through a combination of tuning the server, minimizing the number of requests done on the page before it renders, and other optimizations. In the page loads of this particular site, you can see we're a very long ways from those targets!Empty cache page load

In this case, the page is full of images, and the server could use some tuning to deliver those faster. But a big part of the primed cache problem is how many different external services are getting loaded on the page. This is a substantial e-commerce site doing a healthy business, and it has scripts from 9 external services embedded on the page. Many of these load additional pages inside the main page, which then load more scripts and images... and usually all of this needs to load before the visitor sees the page as it's designed to look!

2 minute load times? I wonder how much business has been lost simply because people don't want to wait that long.

This problem gets far worse on mobile devices, with less powerful browsers, less RAM, sometimes spottier and slower connections...

Sometimes it breaks your site completely

We've had at least 4 or 5 calls in the past few months from customers saying "My site is down! Please get us back up right away!" And it turns out their site is just fine -- but some embedded widget they've added to their page is not loading, and it's blocking the rendering of the page entirely. And these aren't even unusual services -- 3 cases in particular come to mind from big, well known services: Disqus, Google Adwords, and a mobile app vendor.

It might not even be the remote service's fault, it could just be a network issue. But the simple fact is, by using a bunch of different embedded apps on your site, you're making your site dependent on all of them! If any one of them has a problem on a particular day, it can bog down your site entirely, and affect your business!

Do you trust all the sources of these widgets?

Anytime you use code from other sources, you're putting the security of your site at risk. Even reputable, widely used services can introduce malware onto your site. For example, last September there was malware loaded by Google ads that may have been shown to millions of computers, via sites that used Ad words, and that's not the first time. And these are major services run by major companies with a strong security record -- if they can't keep their embeds secure, can you really trust all those other services that don't have those resources?

When you load any script from a server that's not yours, you're trusting that that script is not going to hijack data sent through your page. If you run an e-commerce site, this means they could potentially sniff data like your customer's credit card numbers being typed on the checkout page, or their passwords entered into a login form. Even your password!

You can download and review exactly what the script is doing, but if you don't control the source of that script, it could get changed at some point and you'll never now.

It may not "degrade gracefully"

More and more people are becoming sensitive to their online privacy, and running ad-blockers/script blockers. Have you checked out what happens if you visit your site in a browser configured to block all 3rd party scripts? If you just embed a bunch of widgets on the page, it's quite likely that the experience is very bad... so you're forcing your visitors to give up their privacy to use your service. Do you want to be sensitive to your customers' privacy, or lose that business?

Argh! Enough already! What can we do instead?

First of all, you should probably have a reason for each one of the widgets you want to use on the page. Here is how I would go through and analyze, decide upon a course of action, for each one...

Do I really need this widget?

If you don't need it, ditch it. There's enough noise out there already. Think about what the goals are for each type of user on your site -- prospective customer, prospective employee, current customer, current employee, content editor, business owner, administrator, etc. If a widget does not support the goals of any of your users, get rid of it.

Can I provide a behind-the-scenes integration, instead of using the widget?

If the service in question has a straightforward API, it's often much better to have Drupal send information through the API, rather than having the user's browser load a bunch of uncontrolled Javascript.

For many data collection forms, this kind of approach can solve all of the issues identified above -- the entire interface gets aggregated and can use Drupal's caching to load quickly, nothing visibly breaks if the service is unavailable, you control all the data flow and don't expose anything unnecessary to the 3rd party, and done right it degrades gracefully.

This is why our first response when asked to integrate something is to go check to see if we already have a Drupal module available that we can just turn on to do the integration, or whether we need a few hours to create something new or custom.

This tends to be much more "the Drupal Way", compared to pasting in an embed code!

Can I "do it in Drupal?"

One step further than doing a server-side integration, is to not use the 3rd party service at all, and simply roll your own solution in Drupal.

For really complex integrations (like SalesForce) this can cost less than doing the integration, if you only need the other service for a couple of scenarios -- Drupal is a very powerful platform that can be easily configured to do a lot of CRM-kinds of activities. And there's a huge ecosystem of sometimes pretty complete solutions for event management, mapping, e-commerce, and much more, which put you more in an "Ownership" role rather than a "Renter".

If your needs are complex, this will cost more than using another already-existing service -- but you gain the benefit of complete control over the resulting solution.

How to embed "The Drupal Way"

Sometimes it does make sense to embed scripts and code from other sites. You don't necessarily want to run your own ad server, if Google can fill all your ad slots for you, and that's a big source of your revenue. Lots of specific services for loading fonts, analytics, and more really end up necessitating 3rd party scripts.

Even in those cases, manually embedding code snippets leads to all the issues we identified. By using a Drupal module (either one publicly available on Drupal.org, or a custom one we create for you) you can at least minimize some of those:

  • With many of these services, you can download the script to your server where it can be aggregated with the rest of your Javascript and not impact page load speed
  • If it's on your server, it's not necessarily subject to the same outages
  • If it's on your server you can detect changes to the script and trigger new audits
  • If you manage where and how the script is loaded, you can make it run after the page is rendered, and degrade gracefully

These changes aren't always possible, but you can almost always at least improve the amount of control you have over the user experience if something goes awry by bundling it into a Drupal module.

So... Yes, you can just paste in an embed code, really, really quickly. But do you really want to?

Jun 17 2015
Jun 17

More and more I keep running into assertions that Git is a version control tool, and that if you use it for deployment, you're doing it wrong.


At Freelock we find it to be a very effective deployment tool, and I'm not seeing a solution that meets our needs any better.

Two presentations in particular caught my attention recently mentioned this:

... but those are just two that come to mind. I feel like I've heard this message from lots of different directions, particularly among some of the broader PHP community.

But again, why? What is wrong with using git as a deployment tool?

Reasons to not use git as a deployment tool

As best I can tell, there are three main arguments against using git for deployment. These boil down to:

  1. Storing environment-specific information, including security risks around database credentials, API keys, etc.
  2. Storing object files, generated files, code unrelated to the development of the project.
  3. Disk usage, wasted space taken up by having the full history of a project in a production deployment.

I think the main thrust of this line of thinking is that "git is for source code management, you don't need that in delivered production code, and storing all of this other binary/environment stuff gets in the way of developing the upstream code for a project."

When we started using git for deployment, we ran into these issues, and lots of challenges particularly around managing multiple environments. But they are entirely solvable, and git brings with it a number of big positives as one part of a deployment toolchain.

Why I think git is a great deployment tool

Git brings several big benefits to deployment:

  1. It's extremely fast, can merge in a huge number of changes in a few seconds.
  2. It's very fast and easy to roll back changes that go awry. Especially if you use tags effectively.
  3. For managing lots of related websites, it's an extremely effective way to provide patches across a bunch of different sites.
  4. With a cryptographic hash of every code file, it's extremly simple to detect unauthorized modifications of a file -- e.g. detect if you've been hacked.

Git does not provide these benefits just by using it. For it to be an effective deployment tool, you need to have a very clear organization, relatively strict control over your processes, and some guiding principles. But with those in place, I haven't seen any tool that does a better job for deployment than git.

How we use git for deployment

We've spent years refining our git deployment process, and it actually feels very dialed in at this point. It's not the only tool we use -- we use a bunch of other tools and scripts to help automate and streamline this process, but here are a few of the ways we've addressed the problems with using git for deployment, and leveraged its benefits:

Keep environment-specific settings out of git

Acquia does this. So does Pantheon. Most hosts that rely on git either provide their own environments entirely, or leverage some sort of include structure.

We commit the settings.php file, and can add $conf variables there that need to be available in all environments. At the bottom of the settings.php file, we use a PHP include statement to include a local settings file that is excluded from git, and in that file we store database credentials, environment indicator settings, API keys (for production), enable/disable variables for things like securepages and reroute_email, and anything else that only applies to a single environment.

We catalog environments in drush alias files, one for each client site. These aliases include modules to enable/disable whenever a database sync is performed into an appropriate environment (e.g. into staging, sanitize emails, enable dev modules, disable payment modules, etc).

We distribute drush alias files using Salt Stack, a configuration management tool similar to Puppet, Chef, or Ansible. We're working on moving more environment-specific settings (mainly for provisioning Docker containers or kicking off test scripts) into Salt.

Add generated code to git

You'll find all sorts of guidance to exclude vendor code, things that can be pulled down with make files, things generated with Sass/Compass/other tools out of git.


Coming from an operations perspective, why on earth would you want build tools present on a production server? A huge amount of setting up tools like Compass and Composer can put you into dependency hell, even with composer.lock and gem.lock files. It seems Docker is what's in vogue these days for managing that -- run a bunch of build scripts in a Docker container, and then copy that up to production -- except that nobody who does that dares to run these Docker containers in production. And you've added a bunch of steps to getting something you can actually run on a server, requiring re-compiling and rebuilding everything for the tiniest change.

Git handles this way, way better. And for any scenario where you need tools to build something that you're actually delivering, why not commit that working CSS or whatever you're generating, into your build and pushing that out? No interim container builds necessary. No risky build tools lying around on your production server. No downtime waiting for builds to run on production -- just deploy and clear some caches.

Buy more disk space and grab a coffee

Having a copy of every single Drupal core commit ever in every single production copy may be unnecessary. It does make a clone of the git tree take a few minutes, and take up a couple hundred megabytes of disk.

While that doesn't really add any significant value to a deployment process, and while there's really no benefit to keeping all that around in so many copies, we've found that it's just not worth cleaning all that out.

First of all, these penalties really only make any difference whatsoever the very first time you create a new copy of the site. After that, updates are really really fast.

But the bigger reasons are all related to our core business: keeping dozens of Drupal sites up-to-date.

We've written about our branching and updating strategy before. In short, we maintain our own cloned copy of the main Drupal.org git repository, with branches for Drupal 6, 7, and Pressflow. Upgrading to a new point release of Drupal means merging the new tag into our branch. Then we've written a script to update all the contrib modules we curate across most of our sites. We apply patches to both Drupal core and contrib (for example, patching this core issue, and many changing patches for Media 2.x and other modules).

We call this clone our "upstream" and from there we can simply merge these updates into all of our client's dev sites, run update scripts, and test.

Trying to remove or squash the commit history of Drupal always seems to cause a lot more problems in this deployment process than it solves, so we just live with the extra disk usage and go refill the coffee cup.

Git is a "distributed version control system"

... This seems to be the biggest point that these "purists" who think git should only be used for version control miss: it's extremely cheap and easy to clone and maintain different branches with git. That means you can keep your upstream pristine clone for developing a project that you want to publish on Drupal.org, and very easily use a different git clone to manage it for deployment in a particular site. There is no reason you can't do both with git.

On to the stuff we leverage...

Git post-update hooks for automated deployment

We use a central git server running gitolite. In our gitolite server, we've added a post-update hook that looks for commits pushed that have a branch name matching "release/*" or "hotfix/*". If such a branch is pushed, it passes the repository name, branch name, user who pushed it, and a couple other details into Jenkins.

Jenkins runs a shell script that makes heavy use of drush to connect to the site's stage copy, and automatically deploy there. If the branch already exists on the stage site, it skips the database copy -- otherwise it imports a fresh copy of the production database, runs through some sanitization, checks out the new code, runs updates, reverts features, and notifies us in chat when it's done. Then we can review the Jenkins log and the state of the stage instance to determine if there are manual steps we've overlooked, how long the deployment will take (e.g. to run update scripts, apply features), and whether or not we should plan to take the production site offline for the upgrade (we lean towards keeping it online unless there will be many minutes of entirely bad user experience...)

Production tags and deployment

We don't automatically deploy to production, but we have written scripts to support our process. Our stage cleanup script merges the current release code to master and develop branches, cleans up the central repository, leaves the stage copy on master and the development copy on develop. And it created a git tag for the particular release.

On production, we have nightly backups of the database, and the ability to easily take ad-hoc database snapshots. Before deploying we do a few quick checks:

  • Is the database snapshot current enough? Take a new one if there are any update scripts.
  • git status - Is there any uncommitted code? (Might indicate a bigger problem).
  • drush fl - any features that are not up-to-date? If so, might need to update/pull back to development and re-roll the release.
  • git log/git tag -- identify the current state of the code base -- should already be tagged with the previous release version. If not, create a new tag so we have an easy restore point.

Once ready, we simply pull down the new tag from master, run the updates, apply the features, and then follow any manual deployment steps that need to be done.

Roll back when hell breaks loose

With git, deployment of new code is fast. Once you've fetched the code updates, the merge process only changes files that have changed in the new release, and this process takes fractions of a second for small updates, a few seconds for large ones.

Rolling back is just as fast. Find a bunch of substantial problems? Simply git checkout release-x-1. Find some individual problem in a particular module? git checkout <myoldrelease> path/to/module, and then you can use other git tools to bring that back to development for a proper fix.

What other tool solves these deployment challenges?

Rolling back seems to be the most critical, and that's something you want to do immediately.

How can you do this with other tools?

What other tools are out there that can help with these challenges?

The main ones I hear about: Docker, build scripts/make files, alternative images.


We've actually started using Docker in a pretty big way, but not for managing deployments of site code -- git does a far better job for that, as we've already seen. At least if you're managing a bunch of different but similar sites -- if you are trying to deploy exactly the same site to a bunch of different servers to handle high loads, Docker may well have some advantages.

But the main way to roll back with Docker is to stop the container running the new site, destroy it, then run a new container based on the old image. You end up with some downtime, and if you're using Docker to isolate processes, you now need all the other linked containers to get connected to the new one, possibly triggering a cascade of Docker restarts and far more downtime.

Build Scripts/Make Files

What, are you crazy? Since your site isn't in git, you now need to go back to your development machine, check out an old copy of your site with the old versions of the make files -- if those had any version tags to begin with. If not, now you've got to figure out what version to go back to to get to a working state.

Utter madness. You might as well go back to your previous night's backup, but now you risk losing a bunch of data, too!

Alternative production images

I don't know of anyone using this deployment strategy with Drupal. But while doing some consulting for Microsoft, I ran into this pattern as the preferred way of releasing on Azure. With this pattern, you have not one but two copies of your production site, with essentially a load balancer in front of them. You deploy to the offline one, get it all up-to-date, and then switch all traffic to it. If anything goes wrong, you simply switch back to the other one.

Database schema changes were similarly captured as they were made, using tools that could reverse those schema/data changes if necessary.

Now that sounds like a very intriguing deployment strategy, especially the database schema management. But that's one tool the Drupal community does not (to my knowledge) have at its fingertips... yet.

And even so, the git deployment strategy we use complements this approach very well -- you still need to get the code onto the production instances somehow, and git seems far better than FTP/SFTP... and all the build-script/make file security issues still apply.

What's your take?

Our approach to deployment comes from a Dev-Ops, risk management point of view. We've developed our practices through thousands of releases on hundreds of different sites, with no major issues. We strive to make our production servers extremely secure, easily recoverable from bad releases, and running with as little downtime as possible. From this perspective, I'm not seeing any viable alternative to git for deployment that does as good a job...

What do you use for deployment? Why shouldn't we use git this way? Is there some magical deployment tool we're missing from our arsenal?


About Drupal Sun

Drupal Sun is an Evolving Web project. It allows you to:

  • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
  • Facet based on tags, author, or feed
  • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
  • View the entire article text inline, or in the context of the site where it was created

See the blog post at Evolving Web

Evolving Web