Mar 04 2019
Mar 04


Bitbucket Pipelines is a CI/CD service, built into Bitbucket and offers an easy solution for building and deploying to Acquia Cloud for project’s whose repositories live in Bitbucket and who opt out of using Acquia’s own Pipelines service. Configuration of Bitbucket Pipelines begins with the creation of a bitbucket-pipelines.yml file and adding that file to the root of your repository. This configuration file details how Bitbucket Pipelines will construct the CI/CD environment and what tasks it will perform given a state change in your repository.

Let’s walk through an example of this configuration file built for one of our clients.

bitbucket-pipelines.yml

image: geerlingguy/drupal-vm:4.8.1
clone:
  depth: full
pipelines:
  branches:
    develop:
      - step:
         script:
           - scripts/ci/build.sh
           - scripts/ci/test.sh
           - scripts/ci/deploy.sh
         services:
           - docker
           - mysql
         caches:
           - docker
           - node
           - composer
    test/*:
      - step:
         script:
           - scripts/ci/build.sh
           - scripts/ci/test.sh
         services:
           - docker
           - mysql
         caches:
           - docker
           - node
           - composer
  tags:
    release-*:
      - step:
          name: "Release deployment"
          script:
            - scripts/ci/build.sh
            - scripts/ci/test.sh
            - scripts/ci/deploy.sh
          services:
            - docker
            - mysql
          caches:
            - docker
            - node
            - composer
definitions:
  services:
    mysql:
      image: mysql:5.7
      environment:
        MYSQL_DATABASE: 'drupal'
        MYSQL_USER: 'drupal'
        MYSQL_ROOT_PASSWORD: 'root'
        MYSQL_PASSWORD: 'drupal'

The top section of bitbucket-pipelines.yml outlines the basic configuration for the CI/CD environment. Bitbucket Pipelines uses Docker at its foundation, so each pipeline will be built up from a Docker image and then your defined scripts will be executed in order, in that container.

image: geerlingguy/drupal-vm:4.8.1
clone:
  depth: full

This documents the image we’ll use to build the container. Here we’re using the Docker version of  Drupal VM. We use the original Vagrant version of Drupal VM in Acquia BLT for local development. Having the clone depth set to full ensures we pull the entire history of the repository. This was found to be necessary during the initial implementation.

The “pipelines” section of the configuration defines all of the pipelines configured to run for your repository. Pipelines can be set to run on updates to branches, tags or pull-requests. For our purposes we’ve created three pipelines definitions.

pipelines:
  branches:
    develop:
      - step:
         script:
           - scripts/ci/build.sh
           - scripts/ci/test.sh
           - scripts/ci/deploy.sh
         services:
           - docker
           - mysql
         caches:
           - docker
           - node
           - composer
    test/*:
      - step:
         script:
           - scripts/ci/build.sh
           - scripts/ci/test.sh
         services:
           - docker
           - mysql
         caches:
           - docker
           - node
           - composer

Under branches we have two pipelines defined. The first, “develop”, defines the pipeline configuration for updates to the develop branch of the repository. This pipeline is executed whenever a pull-request is merged into the develop branch. At the end of execution, the deploy.sh script builds an artifact and deploys that to the Acquia Cloud repository. That artifact is automatically deployed and integrated into the Dev instance on Acquia Cloud.

The second definition, “test/*”, provides a pipeline definition that can be used for testing updates to the repository. This pipeline is run whenever a branch named ‘test/*’ is pushed to the repository. This allows you to create local feature branches prefixed with “test/” and push them forward to verify how they will build in the CI environment. The ‘test/*’ definition will only execute the build.sh and test.sh scripts and will not deploy code to Acquia Cloud. This just gives us a handy way of doing additional testing for larger updates to ensure that they will build cleanly.

The next section of the pipelines definition is set to execute when commits in the repository are tagged.

tags:
  release-*:
    - step:
        name: "Release deployment"
        script:
          - scripts/ci/build.sh
          - scripts/ci/test.sh
          - scripts/ci/deploy.sh
        services:
          - docker
          - mysql
        caches:
          - docker
          - node
          - composer

This pipeline is configured to be executed whenever a commit is tagged with the name pattern of “release-*”. Tagging a commit for release will run the CI/CD process and push the tag out to the Acquia Cloud repository. That tag can then be selected for deployment to the Stage or Production environments.

The final section of the pipelines configuration defines services built and added to the docker environment during execution.

definitions:
  services:
    mysql:
      image: mysql:5.7
      environment:
        MYSQL_DATABASE: 'drupal'
        MYSQL_USER: 'drupal'
        MYSQL_ROOT_PASSWORD: 'root'
        MYSQL_PASSWORD: 'drupal'

This section allows us to add a Mysql instance to Docker, allowing our test scripts to do a complete build and installation of the Drupal environment, as defined by the repository.

Additional resources on Bitbucket Pipelines and bitbucket-pipelines.yml:

Scripts

The bitbucket-pipelines.yml file defines the pipelines that can be run, and in each definition it outlines scripts to run during the pipeline’s execution. In our implementation we’ve split these scripts up into three parts:

  1. build.sh – Sets up the environment and prepares us for the rest of the pipeline execution.
  2. test.sh – Runs processes to test the codebase.
  3. deploy.sh – Contains the code that builds the deployment artifact and pushes it to Acquia Cloud.

Let’s review each of these scripts in more detail.

build.sh

#!/bin/bash
apt-get update && apt-get install -o Dpkg::Options::="--force-confold" -y php7.1-bz2 curl && apt-get autoremove
curl -sL https://deb.nodesource.com/setup_8.x | sudo -E bash -
apt-get install -y nodejs
apt-get install -y npm
cd hive
npm install
npm install -g gulp
cd ..
composer install
mysql -u root -proot -h 127.0.0.1 -e "CREATE DATABASE IF NOT EXISTS drupal"
export PIPELINES_ENV=PIPELINES_ENV

This script takes our base container, built from our prescribed image, and starts to expand upon it. Here we make sure the container is up-to-date, install dependencies such as nodejs and npm, run npm in our frontend library to build our node_modules dependencies, and instantiate an empty database that will be used later when we perform a test install from our codebase.

test.sh

#!/bin/bash
vendor/acquia/blt/bin/blt validate:phpcs --no-interaction --ansi --define environment=ci
vendor/acquia/blt/bin/blt setup --yes  --define environment=ci --no-interaction --ansi -vvv

The test.sh file contains two simple commands. The first runs a PHP code sniffer to validate our custom code follows prescribed standards. This command also runs as a pre-commit hook during any code commit in our local environments, but we execute it again here as an additional safeguard. If code makes it into the repository that doesn’t follow the prescribed standards, a failure will be generated and the pipeline will halt execution. The second command takes our codebase and does a complete Drupal installation from it, instantiating a copy of Drupal 8 and importing the configuration contained in our repository. If invalid or conflicting configuration makes it into the repository, it will be picked up here and the pipeline will exit with a failure. This script is also where additional testing could be added, such as running Behat or other test suites to verify our evolving codebase doesn’t produce regressions.

deploy.sh

#!/bin/bash
set -x
set -e

if [ -n "${BITBUCKET_REPO_SLUG}" ] ; then

    git config user.email "[email protected]"
    git config user.name "Bitbucket Pipelines"

    git remote add deploy $DEPLOY_URL;

    # If the module is -dev, a .git file comes down.
    find docroot -name .git -print0 | xargs -0 rm -rf
    find vendor -name .git -print0 | xargs -0 rm -rf
    find vendor -name .gitignore -print0 | xargs -0 rm -rf

    SHA=$(git rev-parse HEAD)
    GIT_MESSAGE="Deploying ${SHA}: $(git log -1 --pretty=%B)"

    git add --force --all

    # Exclusions:
    git status
    git commit -qm "${GIT_MESSAGE}" --no-verify

    if [ $BITBUCKET_TAG ];
      then
        git tag --force -m "Deploying tag: ${BITBUCKET_TAG}" ${BITBUCKET_TAG}
        git push deploy refs/tags/${BITBUCKET_TAG}
    fi;

    if [ $BITBUCKET_BRANCH ];
      then
        git push deploy -v --force refs/heads/$BITBUCKET_BRANCH;
    fi;

    git reset --mixed $SHA;
fi;

The deploy.sh script takes the product of our repository and creates an artifact in the form of a separate, fully-merged Git repository. That temporary repository then adds the Acquia Cloud repository as a deploy origin and pushes the artifact to the appropriate branch or tag in Acquia Cloud. The use of environment variables allows us to use this script both to deploy the Develop branch to the Acquia Cloud repository as well as deploying any tags created on the Master branch so that those tags appear in our Acquia Cloud console for use in the final deployment to our live environments. For those using BLT for local development, this script could be re-worked to use BLT’s internal artifact generation and deployment commands.

Configuring the cloud environments

The final piece of the puzzle is ensuring that everything is in-place for the pipelines to process successfully and deploy code. This includes ensuring that environment variables used by the deploy.sh script exist in Bitbucket and that a user with appropriate permissions and SSH keys exists in your Acquia Cloud environment, allowing the pipelines process to deploy the code artifact to Acquia Cloud.

Bitbucket configuration

DEPLOY_URL environment variable

Configure the DEPLOY_URL environment variable. This is the URL to your Acquia Cloud repository.

  1. Log in to your Bitbucket repository.
  2. In the left-hand menu, locate and click on “Settings.”
  3. In your repository settings, locate the “Pipelines section” and click on “Repository variables.”
  4. Add a Repository variable:
    1. Name: DEPLOY_URL
    2. Value: The URL to your Acquia Cloud repository. You’ll find the correct value in your Acquia Cloud Dashboard.

SSH keys

Deploying to Acquia Cloud will also require giving your Bitbucket Pipelines processes access to your Acquia Cloud repository. This is done in the form of an SSH key. To configure an SSH key for the Pipelines process:

  1. In the “Pipelines” section of your repository settings we navigated to in steps 1-3 above, locate the “SSH keys” option and click through.
  2. On the SSH keys page click the “Generate keys” button.
  3. The generated “public key” will be used to provide access to Bitbucket in the next section.

Acquia Cloud configuration

For deployment to work, your Bitbucket Pipelines process will need to be able to push to your Acquia Cloud Git repository. This means creating a user account in Acquia Cloud and adding the key generated in Bitbucket above. You can create a new user or use an existing user. You can find more information on adding SSH keys to your Acquia Cloud accounts here: Adding a public key to an Acquia profile.

To finish the configuration, log back into your Bitbucket repository and retrieve the Known hosts fingerprint.

Feb 01 2019
Feb 01

In this article we will see how to update data models in Drupal 8, how to make the difference between model updating and content updating, how to create default content, and finally, the procedure to adopt for successful deployments to avoid surprises in a continuous integration/delivery Drupal cycle.

Before we start, I would encourage you to read the documentation of the hook hook_update_N() and to take into account all the possible impacts before writing an update.

Updating the database (executing hook updates and/or importing the configuration) is a very problematic task during a Drupal 8 deployment process, because the updating actions order of structure and data is not well defined in Drupal, and can pose several problems if not completely controlled.

It is important to differentiate between a contributed module to be published on drupal.org aimed at a wide audience, and a custom Drupal project (a set of Drupal contrib/custom modules) designed to provide a bespoke solution in response to a client’s needs. In a contributed module it is rare to have a real need to create instances of configuration/content entities, on the other hand deploying a custom Drupal project makes updating data models more complicated. In the following sections we will list all possible types of updates in Drupal 8.

The Field module allows us to add fields to bundles, we must make difference between the data structure that will be stored in the field (the static schema() method) and all the settings of the field and its storage that will be stored as a configuration. All the dependencies related to the configuration of the field are stored in the field_config configuration entity and all the dependencies related to the storage of the field are stored in the field_storage_config configuration entity. Base fields are stored by default in the entity’s base table.  

Configurable fields are the fields that can be added via the UI and attached to a bundle, which can be exported and deployed. Base fields are not managed by the field_storage_config configuration entities and field_config.

To update the entity definition or its components definitions (field defintions for example if the entity is fieldable) we can implement hook_update_N(). In this hook don’t use the APIs that require a full Drupal bootstrap (e.g. database with CRUD actions, services, …), to do this type of update safely we can use the methods proposed by the contract EntityDefinitionUpdateManagerInterface (e.g. updating the entity keys, updating a basic field definition common to all bundles, …)

To be able to update existing data entities or data fields in the case of a fieldable entity following a modification of a definition we can implement hook_post_update_NAME(). In this hook you can use all the APIs you need to update your entities.

To update the schema of a simple, complex configuration (a configuration entity) or a schema defined in a hook_schema() hook, we can implement hook_update_N().

In a custom Drupal project we are often led to create custom content types or bundles of custom entities (something we do not normally do in a contributed module, and we rarely do it in an installation profile), a site building action allows us to create this type of elements which will be exported afterwards in yml files and then deployed in production using Drupal configuration manager.

A bundle definition is a configuration entity that defines the global schema, we can implement hook_update_N() to update the model in this case as I mentioned earlier. Bundles are instances that persist as a Drupal configuration and follow the same schema. To update the bundles, updated configurations must be exported using the configuration manager to be able to import them into production later. Several problems can arise:

  • If we add a field to a bundle, and want to create content during the deployment for this field, using the current workflow (drush updatedb -> drush config-import) this action is not trivial, and the hook hook_post_update_NAME() can’t be used since it’s executed before the configuration import.
  • The same problem can arise if we want to update fields of bundles that have existing data, the hook hook_post_update_NAME() which is designed to update the existing contents or entities will run before the configuration is imported. What is the solution for this problem? (We will look at a solution for this problem later in this article.)

Now the question is: How to import default content in a custom Drupal project?

Importing default content for a site is an action which is not well documented in Drupal, in a profile installation often this import is done in the hook_install() hook because always the data content have not a complex structure with levels of nested references, in some cases we can use the default content module. Overall in a module we can’t create content in a hook_install() hook, simply because when installing a module the integrity of the configuration is still not imported.

In a recent project i used the drush php-script command to execute import scripts after the (drush updatedb -> drush config-import) but this command is not always available during deployment process. The first idea that comes to mind is to subscribe to the event that is triggered after the import of the configurations to be able to create the contents that will be available for the site editors, but the use of an event is not a nice developer experience hence the introduction of a new hook hook_post_config_import_NAME() that will run once after the database updates and configuration import. Another hook hook_pre_config_import_NAME() has also been introduced to fix performance issues.

A workflow that works for me

To achieve a successful Drupal deployment in continuous integration/delivery cycles using Drush, the most generic workflow that I’ve found at the moment while waiting for a deployment API in core is as follows :

  1. drush updatedb
    • hook_update_N() : To update the definition of an entity and its components
    • hook_post_update_N() : To update entities when you made an entity definition modification (entity keys, base fields, …)
  2. hook_pre_config_import_NAME() : CRUD operations (e.g. creating terms that will be taken as default values when importing configuration in the next step)
  3. drush config-import : Importing the configuration (e.g. new bundle field, creation of a new bundle, image styles, image crops, …)
  4. hook_post_config_import_NAME(): CRUD operations (e.g. creating contents, updating existing contents, …)

This approach works well for us, and I hope it will be useful for you. If you’ve got any suggestions for improvements, please let me know via the comments.

Mar 20 2015
Mar 20

Not so long ago, many of us were satisfied handling deployment of our projects by uploading files via FTP to a web server. I was doing it myself until relatively recently and still do on occasion (don’t tell anyone!). At some point in the past few years, demand for the services and features offered by web applications rose, team sizes grew and rapid iteration became the norm. The old methods for deploying became unstable, unreliable and (generally) untrusted.

So was born a new wave of tools, services and workflows designed to simplify the process of deploying complex web applications, along with a plethora of accompanying commercial services. Generally, they offer an integrated toolset for version control, hosting, performance and security at a competitive price.

Platform.sh is a newer player on the market, built by the team at Commerce Guys, who are better known for their Drupal eCommerce solutions. Initially, the service only supported Drupal based hosting and deployment, but it has rapidly added support for Symfony, WordPress, Zend and ‘pure’ PHP, with node.js, Python and Ruby coming soon.

It follows the microservice architecture concept and offers an increasing amount of server, performance and profiling options to add and remove from your application stack with ease.

I tend to find these services make far more sense with a simple example. I will use a Drupal platform as it’s what I’m most familiar with.

Platform.sh has a couple of requirements that vary for each platform. In Drupal’s case they are:

  • An id_rsa public/private key pair
  • Git
  • Composer
  • The Platform.sh CLI
  • Drush

I won’t cover installing these here; more details can be found in the Platform.sh documentation section.

I had a couple of test platforms created for me by the Platform.sh team, and for the sake of this example, we can treat these as my workplace adding me to some new projects I need to work on. I can see these listed by issuing the platform project:list command inside my preferred working directory.

Platform list

Get a local copy of a platform by using the platform get ID command (The IDs are listed in the table we saw above).

This will download the relevant code base and perform some build tasks, any extra information you need to know is presented in the terminal window. Once this is complete, you will have the following folder structure:

Folder structure from build

The repository folder is your code base and here is where you make and commit changes. In Drupal’s case, this is where you will add modules, themes and libraries.

The build folder contains the builds of your project, that is the combination of drupal core, plus any changes you make in the repository folder.

The shared folder contains your local settings and files/folders, relevant to just your development copy.

Last is the www symlink, which will always reference the current build. This would be the DOCROOT of your vhost or equivalent file.

Getting your site up and running

Drupal is still dependent on having a database present to get started, so if we need it we can get the database from the platform we want by issuing:

platform drush sql-dump > d7.sql

Then we can import the database into our local machine and update the credentials in shared/settings.local.php accordingly.

Voila! We’re up and working!

Let’s start developing

Let’s do something simple: add the views and features modules. Platform.sh is using Drush make files, so it’s a different process from what you might be used to. Open the project.make file and add the relevant entry to it. In our case, it’s:

projects[ctools][version] = "1.6"
projects[ctools][subdir] = "contrib"

projects[views][version] = "3.7"
projects[views][subdir] = "contrib"

projects[features][version] = "2.3"
projects[features][subdir] = "contrib"

projects[devel][version] = "1.5"
projects[devel][subdir] = "contrib"

Here, we are setting the projects we want to include, the specific versions and what subfolder of the modules folder we want them placed into.

Rebuild the platform with platform build. You should notice the devel, ctools, features and views module downloaded, and we can confirm this by making a quick visit to the modules page:

Modules list page in Drupal

You will notice that each time we issue the build command, a new version of our site is created in the builds folder. This is perfect for quickly reverting to an earlier version of our project in case something goes wrong.

Now, let’s take a typical Drupal development path, create a view and add it to a feature for sharing amongst our team. Enable all the modules we have just added and generate some dummy content with the Devel Generate feature, either through Drush or the module page.

Now, create a page view that shows all content on the site:

Add it to a feature:

Uncompress the archive created and add it into the repository -> modules folder. Commit and push this folder to version control. Now any other team member running the platform build command will receive all the updates they need to get straight into work.

You can then follow your normal processes for getting modules, feature and theme changes applied to local sites such as update hooks or profile based development.

What else can Platform.sh do?

This simplifies the development process amongst teams, but what else does Platform.sh offer to make it more compelling than other similar options?

If you are an agency or freelancer that works on multiple project types, the broader CMS/Framework/Language support, all hosted in the same place and with unified version control and backups, is a compelling reason.

With regards to version control, platform.sh provides a visual management and record of your git commits and branches, which I always find useful for reviewing code and status of a project. Apart from this, you can create snapshots of your project, including code and database, at any point.

When you are ready to push your site live, it’s simple to allocate DNS and domains all from the project configuration pages.

Performance, Profiling and other Goodies

By default, your projects have access to integration with Redis, Solr and EntityCache / AuthCache. It’s just a case of installing the relevant Drupal modules and pointing them to the built-in server details.

For profiling, Platform.sh has just added support for Sensiolabs Blackfire, all you need to do is install the browser companion, add your credentials, create an account and you’re good to go.

Backups are included by default as well as the ability to restore from backups.

Team members can be allocated permissions at project levels and environment levels, allowing for easy transitioning of team members across projects and the roles they undertake in each one.

Platform.sh offers some compelling features over it’s closest competition (Pantheon and Acquia) and pricing is competitive. The main decision to be made with all of these SaaS offerings is if the restricted server access and ‘way of doing things’ is a help or a hindrance to your team and its workflows. I would love to know your experiences or thoughts in the comments below.

Apr 15 2014
Apr 15

Super Site Deployment with ctools exportable revert snippets

Sometimes when you are deploying new code to a production site you want to update views, panels, etc. with new code exports, but for one reason or another the defaults are overriden by the database.

Well with the following scripts you can stop worrying about that and just have an update hook take care of reverting (or deleting) the overriding database entries.

Improvements appreciated and feel free to comment!

Jul 16 2013
joe
Jul 16

It's been a really long time since I've worked on a Drupal build that didn't make use of the features module in some way or another. For better or worse it's turned into one of those modules like Views that's simply a part of the expected toolkit for many projects. Yet I still occasionally meet people who haven't heard of it yet. It's always fun to see the lightbulb over their head when you explain it to them and you can see the gears start churning as they immediately start imagining all the tedious box checking they can eliminate from their daily workflow.

The 2.x version of Features came out not too long ago and we are starting to make use of it in our work. It's a great update that's added a few new elements and greatly improved the usability of the the module. It reminds me again how important this module has become to solving so many different problems for us. So I want to take this opportunity to remind you that it exists. And for those of you who haven't used it before, hopefully this post will get the gears churning a little bit.

What exactly does Features do?

James Sansbury wrote an excellent retrospective on the Features module that does a good job of explaining the problem space and how we ended up with the Features module. In summary, one of the things that makes Drupal awesome is the ability to do so much site building without ever having to write any code. You can create content types, click together views, and change the default behavior of the user registration process all through your browser. The challenge in this is deploying those changes to other team members or from a development instance on your laptop to a live site.

Because content is likely continuing to be generated on the live site whilst you work away on your flashy new view you can't (???) simply replace the database on production with a copy from your laptop. We need a way to remember the configuration changes that we made on our local environment, so we can replicate them in the production environment. Enter Features. I often think of features as way of automating the things that I used to record in a checklist. Deploying meant going through my checklist and configuring those things on the live site really fast; hoping no one saw the site when things were broken for a few minutes. This presentation from DiWD 2008, while a little dated, does a good job of explaining the problem space.

Tell me more!

Want to learn more about using Features in your own workflow? Here are some good resources to help get you started:

In addition to the blog post linked above, James also has a recorded presentation that covers a lot of the high-level who, what, and why of features.

We have a complete series on creating features and deploying them with Drush that covers the majority of things you're likely to do with Features in your day-to-day workflow.

While Drush isn't required to use Features it certainly makes it more pleasant. If you haven't used Drush before or need some help we've got a series of videos covering Drush basics that will help you on that front.

I would also recommend taking a look at this blog post about naming things, since it's important to have a good naming convention when using Features.

Sometimes you're going to run into things within Drupal that can't be exported with the Features module. In this case, we still fall back on old reliable hook_update_N() and performing direct database queries. It's not always pretty, but it's a whole lot better than the checklist approach. If you find yourself needing to go this route, review this video on Altering the Database. It covers the process of creating your own update hooks, with a focus on altering the schema of an existing table. But you'll learn how to execute an update function and you can start adding your custom queries there.

Finally, what about writing your own modules and making the configuration data exportable via Features? We've got a short video about making things exportable via CTools. And in Drupal 7 you can make your custom entities built (using the Entity API)[] exportable with just a few extra tweaks to your controller.

Apr 05 2013
Apr 05

Listen online: 

Git is often touted as among other things being extremely flexible. It's a big selling point for the software. You're not throwing all your eggs in one basket and assuming that there is one singular workflow to rule them all. This flexibility can also be a challenge though. In this podcast we'll talk about the various ways that we at Lullabot use Git when working with teams of people. Both small and large. And the organizational tricks we've learned along the way to help make sure our projects continue moving forward and don't get to messy.

Some of the things discussed include designating someone as a branch manager, working in feature/task branches to keep your code organized and easy to review, and using pull requests. Spoiler alert! We end up boiling it all down to this. There is no one perfect way, but whatever way you decided to organize your team make sure you document it and that everyone on the team knows what you decided.

Podcast notes

Ask away!

If you want to suggest your own ideas for podcasts, or have questions for us to answer on a podcast, let us know:
Twitter
Facebook
Contact us page

Release Date: April 5, 2013 - 10:00am

Album:

Length: 42:35 minutes (24.85 MB)

Format: mono 44kHz 81Kbps (vbr)

Feb 22 2013
Feb 22

Listen online: 

In this week's episode Addison Berry is joined by Lullabots Joe Shindelar, Andrew Berry, and Ben Chavet to talk about automating server set up with tools like Puppet and Jenkins. Building a Drupal site is only part of the equation to a complete project, and setting up the servers you will need, and making sure that you can reproduce the site set up consistently is hugely important. It becomes even more important as your projects and teams grow in size. We cover the basics of what the goal is, and what tools we use at Lullabot, and why we use them. This is a great overview of the topic for people who keep hearing these terms thrown around, and would like to understand more of what it's all about.

Podcast notes

  • New videos out on Views Bulk Operations and Entity Views Attachment

Ask away!

If you want to suggest your own ideas for podcasts, or have questions for us to answer on a podcast, let us know:
Twitter
Facebook
Contact us page

Release Date: February 22, 2013 - 10:00am

Album:

Length: 38:50 minutes (22.4 MB)

Format: mono 44kHz 80Kbps (vbr)

Feb 07 2013
Feb 07

Lullabot Re-Launches a Responsive GRAMMY.com for Music’s Biggest Night

Lullabot is excited to announce its fourth annual redesign and relaunch of GRAMMY.com! Just in time for the 55th Grammy Awards this Sunday, February 10th, this year's site features a complete redesign, as well as an upgrade to Drupal 7 leveraging Panels and Views 3.x, some cool, fullscreen, swipeable photo galleries and other mobile-web bells and whistles. The site is also fully responsive, allowing the 40 million+ expected viewers to stream, share, and interact across any web or mobile device. Special congratulations to Acquia for hosting GRAMMY.com for the first time, as well as to our good friends over at Ooyala for once again delivering the site’s archive video content, and to AEG for delivering the live stream video via YouTube.

Simultaneously tune in to every aspect of music’s biggest night so you won’t miss a thing: awards news, video, pictures, artist info, Liveblog posts, thousands of tweets-per-minute and behind-the-scenes action.

The GRAMMY Live stream starts tomorrow, Friday February 8th at 5pm ET / 2pm PT, so you can spend all weekend with the website as you gear up for the CBS telecast on Sunday night. Don't miss any of the GRAMMY action this weekend!

We'll be watching and tweeting along from @lullabot. Follow us on Twitter and let us know what you think.

Feb 06 2013
Feb 06

This week I am continuing the trend of mini-series with some lessons on deploying your code, in the FREE Deploying Your Code Without a Terminal series. The reason behind this quick set of videos is that not everyone is command line savvy, and not everyone has to be. What is important though, is getting your code into version control, and there are plenty of tools that let you do that using a graphical interface. That is all good and well, but what do you do once your code is in version control and you need to get it live on the web? Using the web tool Beanstalk, you can deploy from version control and never open the terminal. This is a great lesson for those of us who want to do things properly, but don't have the time or desire to learn 20 commands that we will forget the next day.

In addition to being a cool service, the nice people at Beanstalk have given us a coupon code to get 50% off your first month. Just use the coupon code DRUPAL when you signup!

We hope you enjoy this mini-series, and we'll have more on the way next week.

Feb 02 2013
Feb 02

Episode Number: 

100

The big 100th episode! In this episode I go over my company's Drupal development environment (http://beginr.com). I discuss how we use Aegir (http://www.aegirproject.org/) to manage and deploy our Drupal websites. I also mention how the development process works using development and live environments spread across multiple servers dedicated to hosting Drupal websites. I mention what operating systems we use along with my person preference for browsers.

One thing I didn't mention is that I use Komodo Edit for my code editor. I think it is pretty awesome.

I also mention sponsorship opportunities, so if you want to become a sponsor of the Daily Dose of Drupal, here is some more information - http://codekarate.com/content/become-codekarate-sponsor

DDoD Video: 

Aug 29 2012
Aug 29

I have recently been wondering how a "perfect" development and deployment environment might look like for me if I had time to put the necessary pieces together. I am planning on working on this eventually, but I thought it might be wise to get my ideas out there to see if others had opinions or advice.

First, a little disclaimer. I am a developer. I live in PHP, Python, Java, etc. Since most of what I work on are Drupal based websites, I am almost always in Drupal module code (and occasionally in the theme layer). I have set up my fair share of Ubuntu based servers and have some experience with server virtualization, however I would not consider myself even close to a server expert or system administrator.

Since my company deals mostly with Drupal websites, this setup is based on developing and deploying Drupal 6 and Drupal 7 websites.

Using Aegir

I have been using Aegir for the past few years and have not looked back. Most of my development environment is going to be based on using Aegir for the heavy lifting. It will require some additional features to be added onto Aegir through a good deal of custom modules.

The Development Environment

My idea for setting up development environments on multiple developers computers is to build a virtualbox Ubuntu server image that any developer can deploy on their system. It would be great if Aegir could then deploy development sites to these developers servers on their local systems. The reason behind this is that some of the developers I work with are more front end designers who are not familiar with setting up and deploying Drupal websites on their local system. I would expect them all to have basic command line/drush skills, but I want to eliminate the process of having them need to set up the Drupal site, manage the database, etc, all from their local environment.

Obviously there are a lot of issues with this idea. First I think a verify from the Aegir admin interface on this development website would have the possibility of wiping out a developers uncommitted changes. This also forces the requirement that the developer is at minimum VPN'd into the development network. This also poses minor issues with Aegir trying to run cron on these sites that may not always be online. This is one area that I would need to research much more thoroughly before I make any decisions on how to proceed.

Website Creation

Website creation will be handled predominantly using Aegir. I would also need to build in the ability to allow non server users to import existing Drupal websites. Beginr Media does a lot of custom development for existing sites and often times we only receive FTP information for the live site. I would like a non system admin to be able to grab the Drupal files directory, get a dump of the database, and then be able to upload these directly from the Aegir interface. Depending on if a full Drupal install or multi-site is uploaded, Aegir will respond accordingly and build out the site and migrate it to the requested platform (assuming there are no errors or platform inconsistencies). This could probably pose a huge security threat because of the ability to upload this information directly to the server and then have it execute website building actions. However, since the only users of this will be fairly technical users (all of which are employees), I think I could make this work.

When creating a website, either in Aegir or through the import process described above, I would like the ability to either define an existing Git branch or have it create me a new branch for the project on Github using their API. This would be used to ensure that every website was tracked in Git.

I would need the ability to relate sites (Pantheon does this great with their Development -> Staging -> Live setup). For instance, after creating the original development site, I need the ability to clone this development site and drop it on another developers local system. This would allow multiple Drupal developers to work on the same website without running into the errors that often happen when working on the same Drupal site.

Automated deployments

Since the Aegir website knows the git branch for the website, along with the related development sites, it can monitor the status of the branch for changes. When a developer pushes up to the development version of the branch from their local environment, this should automatically trigger the development site to pull the updated codebase, run drush updatedb, clear the caches, and possibly even revert any enabled features. The developers would still need to pull the changes to their local environments (we wouldn't want to push anything automatically to them), or they could always just re-clone the site to their local environment after major changes.

Pushing to live

Production deployments would consist of cloning the development version of the site to a live server and somehow marking this as the live/production version of the site. This production version of the site would always monitor the master branch of the Git repo and automatically deploy any changes that make their way to master (while also updating the database, clearing the cache, and possibly reverting any features).

Handling change requests

This setup would always keep a development/staging site that would always remain connected to the live site. These sites could be located on different platforms, servers, etc, but would always be related. The development/staging site would be based on a Git development branch, while the live would be based on the Git master branch.

Change requests in the form of client requests, new features, bug fixes, etc, would follow a very simple but audit-able process. The change request would come in and be assigned to a developer. This developer would then create a clone of the development/staging version of the site and have Aegir automatically drop in on their local development environment. The developer would complete the change request and make a commit to the development branch in Git. This could automatically be pulled to the main development/staging site for testing. If all goes well, a merge request would be created and the code would be merged into the master branch by another developer. This would automatically update the code on the live site for a final round of testing. For relatively simple sites this does add some extra complexity, but for larger projects these techniques should already be in place anyway as it creates a defined and repeatable process.

Handling module and core updates

Module updates would be handled the same way the change requests were handled above. Core updates would follow the Aegir pattern of building a new development platform for the new version of Drupal, and migrating the development site to the platform. Testing would happen in this environment and if everything passed, a live platform would be created on the live server and the live site would be migrated over.

Additional developer tools

Additional development tools/options I would need to build into Aegir would be the ability to migrate data from the database backwards from the live environment to the development/staging site. At first this could be a simple database replacement and would just use the relationships that exist to show an extra Aegir task on the development/staging site that would allow pulling a fresh copy of the production database. This should employ some cleaning mechanisms to clear out or mask email addresses, passwords, etc. The reasons for this are two-fold. The first is to prevent sensitive data from entering the development environment, the second is to prevent live users receiving emails or notifications from the development site.

Drupal development and deployment environment conclusions

My current environment at Beginr Media is far from the ideal vision I discussed above, however I really want your opinion. If you have ideas or advice on things that you think should be added/removed/changed, let me know. I am planning on building out some of these modules as additional plugins for Aegir that I would then release on Drupal.org.

Before any of this can happen though, I want to make sure I am actually solving the problem in the most efficient way possible. Who knows, maybe there is something out there that will get me closer to my ideal environment with much less work (although I haven't found it yet).

I like the model of Pantheon, however it does not solve my developer/site import dilemma. Last time I checked there were still some things missing that I need such as SSL support (which Aegir has), etc. I would also like to keep things on servers that I have full control over. I plan on pushing a lot of sites through this environment and it will be much more cost effective for me to control the servers.

Another alternative/spin on the Aegir approach is to push everything through an installation profile for each site along with Drush make files for quickly building the website. I have done this in the past and it works pretty well, however in my case this would end up making pretty much every site its own platform (since almost every site we build ends up being significantly different). I may build a profile containing the modules/themes that we use on every site to get started, but building an installation profile/drush make setup for each site would be way to much overhead and seems to be a bad approach for my situation.

I need your help

If you made it this far, then I applaud your reading efforts and thank you for taking the time to look this over. However, I really would like your ideas or input in the comments below (or twitter @smthomas3). One of the greatest things about Drupal is the community and I am looking for some advice from some experienced Drupal deployment experts, or anyone with an opinion. If you know anyone that can help, please send them a link. Thanks for your help on this, who knows where this discussion may lead us!

May 10 2012
May 10

After months of site development, code, more code, and long hours, launch day arrives. A site launch can come as a relief, create a bittersweet moment, or one filled with pride and a sense of accomplishment, not unlike a parent sending a child off to their first day of kindergarten.

Every team wants their deployment process to be as close to a gentle push into deep waters as possible. At Zivtech we've launched many Drupal sites, and have compiled a checklist that we use to help things stay snag-free. We typically run through the checklist several days before launch, once the site is already on its production server(s). It ensures we didn't miss anything critical to a live site and reminds us to change settings that are appropriate only during the development cycle. This list is geared toward Drupal 7 sites but can be used for earlier versions with little difference. We hope this checklist helps prepare you and your site for the big day. Feel free to comment on items we should add or that you do differently.

Security and Spam Possibilities

File Permissions

  1. Make sure file permissions for file directories and code directory are set correctly: http://drupal.org/node/244924
  2. On /admin/modules

    The 'PHP Filter' module must be disabled.
  3. On /admin/config/content/formats
    Filtered HTML or plain text MUST be set as the default to prevent cross-site scripting attacks. You must also check the configuration of the Filtered HTML format to ensure that the HTML filter is enabled for it and that no tags are allowed such as img, embed or object.
 If the 'Better Formats' module is enabled, check the default formats assigned to the roles.
  4. On /admin/config/people/accounts
    
Check the registration settings are appropriate for the site.

Security Updates

On /admin/reports/updates/list

  1. Check for any security releases and run updates (testing locally first) if required.
  2. Drupal 7 will automatically alert you about security updates for enabled modules. If the client receives ongoing support, set the email to the proper notifications address.

Modules

On /admin/modules/list

  1. Make sure that only the modules that are being used are enabled. The more modules that are enabled, the slower the site will run and may be confusing to the administrators.
  2. Uninstall, then git remove any modules that are not being used and are not part of core.
  3. 'Devel' module and other similar tools should be disabled at launch, but not removed.

Email

  1. On /admin/config/system/site-information
    Make sure the email address and name are correct.
  2. On /admin/structure/contact and /admin/config/content/webform
    Make sure email addresses are set correctly for contact forms and webforms.
  3. Disable reroute_email module after informing all developers and QA users that email is going live.

Permissions

On /admin/people/permissions/list

  1. Ensure permissions are set appropriately and minimally.

  2. Any permission granted to anonymous users should generally also be given to authenticated users.

  3. Be careful with granting the administer users permission.

  4. The 'View Media' permission from the 'Media' module actually bypasses other file control and should only be for administrators.

  5. Be sure that if one or more node access modules is in use (workflow access, og access, content access etc) that node permissions are not given which would prevent access from being determined by these modules.
  6. Once permissions are set, create a test user for each role, log in as them and try using the site the way they would to see if all goes well. (Hopefully this and some of the other items in this list were done during QA rather than now, but this is your last reminder!)

Status Report

On /admin/reports/status

  1. Ensure that 'Cron' has been running.
  2. Make sure database updates are up to date.
  3. Take care of any other concerns listed on this page.

Pathauto Settings

  1. On /admin/config/search/path/patterns
    Make sure all paths are set properly, especially node path settings.
  2. On /admin/config/search/path/settings
    Change general settings for 'Update actions' to something other than 'Create a new alias.' and 'Delete the old alias.', which are only appropriate during development.

    In most cases 'Create a new alias but leave existing alias functioning' is appropriate for production. Alternatively or in combination, you can set redirects to be made when aliases are changed with the 'Redirect' module at /admin/config/search/redirect/settings.

Content types & Nodes

On /admin/structure/types

  1. Check to see if there are any content types not being used.
    If there are, confirm the type can be removed, and if so delete that content type.

  2. Delete any dummy nodes that were created during development for testing. Track these down by searching for 'test', 'dummy', and for content created by the super user.

Views

On /admin/structure/views

  1. Delete any views no longer in use.
  2. Ensure all views have been exported to Features and are not in an overridden state.
  3. Public facing views, especially on the home page and on sites with a lot of authenticated traffic, should be configured within each view for caching.

Error Reporting

On /admin/config/development/logging
Set the 'Error messages' display to 'None'.

Performance Settings

On /admin/config/development/performance

  1. Set 'Caching Mode' to normal.
  2. Enable
 'Page compression'.
  3. Enable
 'Optimize CSS' and 'Optimize JavaScript'.
  4. 
'Block caching' should be enabled, unless there is help text on the form warning otherwise.
 Minimum cache lifetime and expiration of cached pages should both be an hour, or as appropriate for the site.
  5. Alter the following lines in settings.php:

    ini_set('session.cache_expire', 200000);
ini_set('session.cache_limiter', 'none');
ini_set('session.cookie_lifetime', 2000000);
ini_set('session.gc_maxlifetime', 200000);

    
The default values result in the sessions table growing very quickly and unnecessarily. Changing the numbers to something reasonable will not hinder the user experience and will prevent database bloating.

  6. Check for full URLs in node bodies etc. The client may have put in full URLs that will not work when the site goes live. Search in PHPMyAdmin or by grepping a tab-separated database dump, and fix or flag to the client anything you find.

General Housekeeping

Remove .txt files from Core

  1. Get rid of CHANGELOG.txt etc (from git etc). Do NOT remove robots.txt!
  2. Edit robots.txt to be standard (in case it has been edited during dev to restrict search crawlers).

Do a click-through (for small sites without a formal QA process).

  1. In addition to checking the site as a test admin, also check the site logged out as an anonymous user.
  2. Click around and make sure there are no broken links and everything looks okay.

Update external services

  1. If using Google Analytics or Facebook integration, you need to update settings for the live domain. You may need to put these settings changes into a script to run at launch time.
  2. If using Commerce, ensure that the payment provider is set to live and has been tested.

Check SSL

If using SSL (you should be), change your local /etc/hosts to point the site to its live domain and ensure SSL redirection is working correctly.

Update features

Make sure all features are up to date so that the launch configuration is saved and can be reverted to.

May 01 2012
May 01

Devopsdays Mountainview sold out in a short 3 hours .. but there's other events that will breath devops this summer.
DrupalCon in Munich will be one of them ..

Some of you might have noticed that I`m cochairing the devops track for DrupalCon Munich,
The CFP is open till the 11th of this month and we are still actively looking for speakers.

We're trying to bridge the gap between drupal developers and the people that put their code to production, at scale.
But also enhancing the knowledge of infrastructure components Drupal developers depend on.

We're looking for talks both on culture (both success stories and failure) , automation,
specifically looking for people talking about drupal deployments , eg using tools like Capistrano, Chef, Puppet,
We want to hear where Continuous Integration fits in your deployment , do you do Continuous Delivery of a drupal environment.
And how do you test ... yes we like to hear a lot about testing , performance tests, security tests, application tests and so on.
... Or have you solved the content vs code vs config deployment problem yet ?

How are you measuring and monitoring these deployments and adding metrics to them so you can get good visibility on both
system and user actions of your platform. Have you build fancy dashboards showing your whole organisation the current state of your deployment ?

We're also looking for people talking about introducing different data backends, nosql, scaling different search backends , building your own cdn using smart filesystem setups.
Or making smart use of existing backends, such as tuning and scaling MySQL, memcached and others.

So lets make it clear to the community that drupal people do care about their code after they committed it in source control !

Please submit your talks here

Feb 28 2012
Feb 28

I have been following the Aegir project for some time now, almost 3 years. It’s great to see how far the project has come along, and how easy it is to get an Aegir instance up (it used to be very challenging to install). However, I haven’t really fully embraced Aegir (yet) into our current workflows at ActiveLAMP. I’m still pondering how exactly I want to use the tool. Allow me to describe our current process, before I elaborate on how I want to use Aegir for deployment.

Our current workflow.

Early in 2010 we adopted a workflow for managing and building sites using Drush Make. Several articles inspired this new workflow for us, so I won’t go into detail of why it’s a good idea to use Drush Make for doing your builds, rather than having one giant repository of code in which 80% of that code you don't touch (hack) anyway.

On each of our developers machines we have one drupal core (a platform) that runs any number of sites that we may be working on at the time. We may have several platforms (different core versions i.e. 7.9, 7.10, 7.12, etc...) on our development machines. All sites that we work on are essentially a multisite within the specific platform (drupal core version) that we're working within. With every site we have an aliases.drushrc.php within its sites directory. This aliases.drushrc.php file holds meta data regarding what platform the site is running on locally, as well as where the production server is, and where the dev server is.

I wrote a custom drush command that actually sets all this up, so we don't have to do too much work. When we start a new site we just type `drush ns mynewsite.com` (drush new-site) and that will fire off a number of tasks:

  • Check drupal.org to see what the latest version of core is
  • Check local platforms directory on developer machine to see if that core version is installed.
    • If the core version is not installed, it is downloaded, a vhost is setup in the apache config, and the /etc/hosts files is edited for the new platform.
  • Then it checks our git repository for a repo called newsite.com.git and checks that out into a separate sites directory (not within the platform just downloaded)
    • If the site repo doesn't exist, the drush new-site command creates a new site directory with a few files copied over from the drush new-site command for defaults (site.make, .gitignore, and rebuild.sh)
    • git init, git add, git commit, and git push origin master are all executed with the initial files at the very end of the drush command.
  • The new sites directory is symlinked to the platform
  • Two drush alias files are created, a global alias file placed in ~/.drush, and an aliases.drushrc.php in the actual site directory.
  • Finally the site is installed with `drush si`

For deployment we use Capistrano invoked with a custom Drush command. This drush command looks at parameters set in aliases.drushrc.php within the sites directory and then fires off a capistrano command passing in arguments it finds in aliases.drushrc.php. We're only deploying the sites directory to the new environment, and capistrano takes care of symlinking it to the correct platform on the dev or production servers.

I'm leaving out a lot of minor details, but in a nutshell that's how our current workflow works for developing sites and deploying.

Thoughts for using Aegir for deployment.

Most people (that I've read about in blog posts and the drupal.org issue queue) that are using Aegir for deployment, are using install profiles for their custom code, and are utilizing a recursive make file technique to handle the build of the platform, as well as the profile. The process seems to work well, and it makes sense, but I'm not sure I want to handle our deploys this way for a number of reasons:

  • A platform has to be built for every deployment, with the "new" profile.
  • Really only one site is going to run on this platform, unless you put multiple profiles in the platform make file (which then leads to more issues if you have different development cycles for the number of sites you're currently working on.)
  • A whole bunch of extraneous tasks are being fired to build this platform, when only site code could only be deployed.
  • Most important to me, you're getting away from what Aegir does well, host instances of sites on a common platform.


My initial thoughts for handling this with Aegir is just integrating Capistrano with Aegir with a few drush commands and then expose those drush commands to hostmaster. I would also add a sites make file field to the node add form for sites, so that when creating a new site, you can specify a site make file, just like you can when you build a platform.

The process of deployment would still be handled by Capistrano, while still utilizing Aegir for creating, managing, and migrating sites. I'm going to start developing this functionality, but I'm curious to hear others thoughts on this, and if there any holes in how I think this could work.

Feb 15 2012
Feb 15

Phing is a PHP tool that allows us to automate processes -- typically, it's used for building and deploying software. It's similar in functionality to the Apache Ant project and uses XML files to execute tasks that are defined as PHP classes. Best of all, it has a tonne of cool tasks already baked in! These include tasty things such as unit testing, file system operations, code sniffer integration, SQL execution, shell commands, 3rd party services and version control integration.

Ok, that sounds good. How do I use it?

You can install Phing easily through PEAR.

$> pear channel-discover pear.phing.info
$> pear install phing/phing

The default name for a Phing build file is build.xml. This file contains a number of targets, which are like different functions that you might find in a PHP file.

<?xml version="1.0" encoding="UTF-8" ?>

<project name="MyProject" default="hello">

  <!-- ============================================  -->
  <!-- (DEFAULT)  Target: hello                      -->
  <!-- ============================================  -->
    <target name="hello" description="Says Hello">
      <echo msg="Hello, world!" />
    </target>

  <!-- ============================================  -->
  <!-- Target: cheese                                -->
  <!-- ============================================  -->

  <target name="cheese" description="Expresses feelings about cheese">
    <echo msg="I like cheese" />
  </target>

</project>

Let's go ahead and run the above build, in which we've specified the default target to be "hello", by running the "phing" command from the directory where this file is saved,:

$ phing
Buildfile: /Users/sal/phing/build.xml

MyProject > hello:

     [echo] Hello, world!

BUILD FINISHED

Total time: 0.6856 seconds

Now, try and run the "cheese" target

$ phing cheese
Buildfile: /Users/sal/phing/build.xml

MyProject > cheese:

     [echo] I like cheese

Yo Dawg, I heard you like targets

Running multiple targets can be done by separating them with a space

$ phing cheese hello

We can also set targets to be dependencies of other targets. Let's modify our default.

<target name="hello" depends="cheese" description="Says Hello">
  <echo msg="Hello, world!" />
</target>

$ phing
Buildfile: /Users/sal/phing/build.xml

MyProject > cheese:

     [echo] I like cheese

MyProject > hello:

     [echo] Hello, world!

Tasks and Properties

Tasks are the actions that our targets are going to step through so we can get stuff done- we've already used the EchoTask above. We can pass them string, integer and boolean parameters as well as more complex data types such as lists of files. Tasks are defined as PHP classes and it's straightforward to roll your own should you need to, however most of what you'll need is probably already there. Documentation on all the built in Tasks can be found in the Phing User Guide. You can also check out the PHP source for all the tasks which will be located in your PHP installation's lib folder.

Phing tasks

Properties are the equivalent of variables and let us use and manipulate stored values in our tasks.

<property name="drupal.version" value="7.10" />

A simple Drupal deploy

We're going to clone the Drupal core repository and deploy it to a server. For this deploy, I'm going to grab Drupal from http://git.drupal.org/project/drupal.git (see http://drupal.org/project/drupal/git-instructions). We'll keep it simple for this example by having everything in one task, but you should split tasks into reusable chunks as you would with PHP code/functions.

<?xml version="1.0" encoding="UTF-8" ?>

<project name="DrupalDeploy" default="deploy">

  <!-- ============================================  -->
  <!-- (DEFAULT)  Target: deploy                     -->
  <!-- ============================================  -->
 
  <property name="drupal.workingdir" value="./drupal" />
  <!-- Get the full directory path -->
  <resolvepath propertyName="drupal.workingdir.resolved" file="${drupal.workingdir}" />
   
  <target name="deploy" description="Deploys a copy of Drupal Core">
    <mkdir  dir="${drupal.workingdir}" />
    <!-- Clone drupal core -->
    <gitclone repository="http://git.drupal.org/project/drupal.git" targetPath="${drupal.workingdir.resolved}" />
  </target>

</project>

This may fail for you if you don't have PEAR's VersionControl_Git package installed, go ahead and install that if you need to

$ pear install VersionControl_Git-alpha

This is going to take several minutes since cloning a repository with Git will retrieve the entire history of the Drupal project, so it's time to go make yourself a cup of tea. If you want to understand more about what exactly git is retrieving at this point, I recommend Blake Hall's excellent Vision for Version Control video on Drupalize.me. For the sake of saving us some time whilst we're learning, we're going to reuse this clone so we don't have to wait each time we run the Phing build file (you could try "git archive --remote" if your remote repository supports that). We'll do it by checking to see if the drupal/.git directory exists or not. For your own deployment scripts it's better to make no assumptions about files that are already on your system and start from a clean checkout of your code.

<target name="getDrupal" description="Clone the Drupal repository">
  <delete dir="${drupal.workingdir.resolved}" includeemptydirs="true" verbose="true" failonerror="false" />
  <mkdir  dir="${drupal.workingdir.resolved}" />
  <!-- Clone drupal core -->
  <gitclone repository="http://git.drupal.org/project/drupal.git" targetPath="${drupal.workingdir.resolved}" />
</target>

Now you can run "phing getDrupal" if you want to remake your git repository.

Steps for Deployment

It usually helps to write the steps down in a list before building the XML.

  1. Get Drupal
  2. Switch to the version of Drupal I want
  3. Add database credentials to settings.php
  4. Put the code in the webroot (and remove unwanted files)
  5. Make the code live

I then like to put these in as comments and fill them out. You can follow them in the full build file below.

<?xml version="1.0" encoding="UTF-8" ?>

<project name="DrupalDeploy" default="deploy">

  <!-- ============================================  -->
  <!-- (DEFAULT)  Target: deploy                     -->
  <!-- ============================================  -->
 
  <property name="drupal.version" value="7.10" />
  <property name="drupal.git" value="http://git.drupal.org/project/drupal.git" />
  <property name="drupal.workingdir" value="./drupal" />
  <!-- Get the full directory path -->
  <resolvepath propertyName="drupal.workingdir.resolved" file="${drupal.workingdir}" />

  <property name="server.webroot" value="/Users/sal/Sites/mysite" />
 
  <tstamp>
    <format property="build.time" pattern="%Y_%m_%d__%H_%M_%S" />
  </tstamp>
 
  <property name="drupal.settings" value="$databases = array (
    'default' =>
    array (
      'default' =>
      array (
        'database' => 'mysite',
        'username' => 'username',
        'password' => 'password',
        'host' => 'mydb',
        'port' => '',
        'driver' => 'mysql',
        'prefix' => '',
      ),
    ),
  );
  $drupal_hash_salt = 'abc123';" />
   
  <target name="deploy" description="Deploys a copy of Drupal Core">
    <!-- Switch to the version of Drupal I want -->
    <gitcheckout repository="${drupal.workingdir.resolved}" branchname="${drupal.version}" />
   
    <!-- Add database credentials to settings.php -->
    <delete file="${drupal.workingdir.resolved}/sites/default/settings.php" failonerror="false" />
    <copy file="${drupal.workingdir.resolved}/sites/default/default.settings.php" tofile="${drupal.workingdir.resolved}/sites/default/settings.php" />
    <append destFile="${drupal.workingdir.resolved}/sites/default/settings.php" text="${drupal.settings}" />
    <chmod file="${drupal.workingdir.resolved}/sites/default/settings.php" mode="0444" />
   
    <!-- Put the code in the webroot (remove unwanted files) -->
    <copy todir="${server.webroot}/drupal-${build.time}" >
      <fileset dir=".">
        <include name="drupal/" />
        <exclude name="drupal/.git" />
        <exclude name="drupal/.git/" />
      </fileset>
    </copy>

    <!-- Make the code live. -->
    <symlink target="${server.webroot}/drupal-${build.time}/drupal" link="${server.webroot}/live" overwrite="true" />

  </target>
  
  <target name="getDrupal" description="Clone the Drupal repository">
    <delete dir="${drupal.workingdir.resolved}" includeemptydirs="true" verbose="true" failonerror="false" />
    <mkdir  dir="${drupal.workingdir.resolved}" />
    <!-- Clone drupal core -->
    <gitclone repository="http://git.drupal.org/project/drupal.git" targetPath="${drupal.workingdir.resolved}" />
  </target>

</project>

$ phing
Buildfile: /Users/sal/phing/build.xml
[resolvepath] Resolved ./drupal to /Users/sal/phing/drupal

DrupalDeploy > deploy:

[gitcheckout] git-checkout command: /usr/bin/git checkout '7.10'
[gitcheckout] git-checkout: checkout "/Users/sal/phing/drupal" repository
[gitcheckout] git-checkout output:
[delete] Deleting: /Users/sal/phing/drupal/sites/default/settings.php
[copy] Copying 1 file to /Users/sal/phing/drupal/sites/default
[append] Appending string to /Users/sal/phing/drupal/sites/default/settings.php
[chmod] Changed file mode on '/Users/sal/phing/drupal/sites/default/settings.php' to 444
[copy] Created 120 empty directories in /Users/sal/Sites/mysite/drupal-2012_01_16__13_13_55
[copy] Copying 1035 files to /Users/sal/Sites/mysite/drupal-2012_01_16__13_13_55
[symlink] Linking: /Users/sal/Sites/mysite/drupal-2012_01_16__13_13_55/drupal to /Users/sal/Sites/mysite/live

Run it a few times and you should see your new code being deployed each time.

Terminal screenshot showing deployed site

Custom Tasks

One of the best things about Phing is that we can build our own custom tasks using PHP. See the Extending Phing docs for more information. Custom tasks can be placed in a folder called "tasks" next to your build XML.

Phing tasks subfolder

Here's an example of a task and build file that prints a random string.

<?php

require_once 'phing/Task.php';

class RandomStringTask extends Task
{
  private $propertyName;
 
  /**
   * Set the name of the property to set.
   * @param string $v Property name
   * @return void
   */
  public function setPropertyName($v) {
      $this->propertyName = $v;
  }
 
 
  public function main() {
    if (!$this->propertyName) {
        throw new BuildException("You must specify the propertyName attribute", $this->getLocation());
    }
   
    $project = $this->getProject();
   
    $c = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxwz0123456789";
    $length = 12;
    for(;$length > 0;$length--) $s .= $c{rand(0,strlen($c))};
    $random = str_shuffle($s);
    $this->project->setProperty($this->propertyName, $random);
  }
}

<?xml version="1.0" encoding="UTF-8" ?>

<project name="Random" default="random">

  <taskdef name="randomstring" classname="tasks.RandomStringTask" />
  <property name="mystring" value="null" />
 
  <!-- ============================================  -->
  <!-- (DEFAULT)  Target: random                     -->
  <!-- ============================================  -->
  <target name="random" description="Prints a random string">
    <randomstring propertyName="mystring" />
    <echo message="${mystring}" />
  </target>
   
</project>

$ phing
Buildfile: /Users/sal/Downloads/phing/build.xml

Random > random:

[echo] 0uH282IFqirG

Jenkins Usage

Jenkins is a very popular open-source continuous integration server that executes and monitors repeated jobs such as testing code, building software and running cron jobs. The good news is it has Phing support and you can enable it under the Plugin Manager.

Once you've enabled the plugin, you'll see an "Invoke Phing targets" build step option when you configure a job. From there you can specify which Phing targets to run and pass values of properties to your build.

Jenkings Phing configuration

You can pass a number of very useful things from the Jenkins build environment into your Phing script, such as the workspace location and build tag.

Jenkins Phing configuration build properties

A common use for this is deploying automated builds to test environments every time code is committed to version control.

Homework

  • Use an external property file to hold credentials or if using Jenkins use secret files
  • Use drush commands
  • Deploy code to remote servers
  • Switch the live symlink with an atomic operation by creating a temporary symlink and then using mv
  • Automate as much as possible! e.g. running database updates, getting contrib and custom modules
Dec 21 2011
Dec 21

Today is not your lucky day. The production server went down due to a hard disk error on the VM's host machine. You don't have a high availability setup because running additional servers 24x7 is quite costly. However, you need a new machine quickly.

In this blog post, I will tell you how we deploy brand new production machines in under 20 minutes. First, I'll talk about why it is so important to be able to deploy servers quickly.

Benefits of rapid deployment

  • Better testing environments. If you want to try Memcached out on the site, you want to reproduce the current production environment quickly. You are, after all, just trying it out, and you don't want to lose a day on it. If you can run one command and have the server running, the experiment is much more feasible during your busy work week.
  • Better development environments. If you just hired a new developer, and you want them to fix a minor bug on a site you're maintaining, they shouldn't have to waste time setting up the environment before fixing the bug. They shouldn't need to know about all the legacy baggage the site has, and they shouldn't have to pester you about how to set up every component of the site.
  • Better documentation. Instead of writing extensive documentation on each step of the deployment process, you can write a deployment script. Well-written code can rarely replace good documentation, but this is one case where relying on the code is suitable. With minimal documentation effort, you should be able to roll out of bed and deploy a new copy of your most mission-critical production sites in minutes.

Steps to Rapid deployment

Set up your Chef infrastructure

At Evolving Web, we use Chef, a configuration management tool. With Chef, you manage all your configurations in one git repository, and you write them in Ruby. This lets the Chef server build a new server or update stale configurations automatically. For example, Chef will occasionally add new public keys to our authorized_keys files as the team grows.

With Chef, your site's environment is reproducible and standardized. When you're attempting to reproduce bugs, it's good to know that your development VM is the same as the one running the production server.

Make a site-specific cookbook for launching the site

We use site-specific cookbooks to launch a specific site. This will use a rotating read-only SSH key to pull code and configuration from a git repository. Once Chef builds the site, you can sync a database from backups or the production server.

Document

Write a step by step guide that anyone can read without messing up. This takes clarity and detail. Having a reproducible process for creating the VM makes this much easier; you can document as you build a test VM with VirtualBox on your own desktop. If you find that a certain part of the process doesn't work properly, you'll notice while documenting. You can fix the Chef script and go through the process again. Since it's a short process (especially with VirtualBox templates), it's not a problem to start over.

Test the process with a beginner

Find the newest recruit on the team and send them a link to your documentation page. Tell them to set up a server in 30 minutes. Can they do it without your help? If so, you've succeeded. If not, it's back to the drawing board. Make those docs shine.

Having good documentation for this makes getting new team members up to speed much faster. Passing your deployment process down generation by generation through oral tradition is not how the professionals do it.

You're done!

Now you can rest assured that you can get that site running again with no hassle. The entire development process should be more pleasant too; you can boot up a development VM in half an hour, and make it a clean environment separate from all the experimental hacks you do on your desktop.

Resources

Jul 17 2011
Jul 17

For those who haven't noticed yet .. I`m into devops .. I`m also a little bit into Drupal, (blame my last name..) , so one of the frustrations I've been having with Drupal (an much other software) is the automation of deployment and upgrades of Drupal sites ...

So for the past couple of days I've been trying to catch up to the ongoing discussion regarding the results of the configuration mgmt sprint , I've been looking at it mainly from a systems point of view , being with the use of Puppet/ Chef or similar tools in mind .. I know I`m late to the discussion but hey , some people take holidays in this season :) So below you can read a bunch of my comments ... and thoughts on the topic ..

First of all , to me JSON looks like a valid option.
Initially there was the plan to wrap the JSON in a PHP header for "security" reasons, but that seems to be gone even while nobody mentioned the problems that would have been caused for external configuration management tools.
When thinking about external tools that should be capable of mangling the file plenty of them support JSON but won't be able to recognize a JSON file with a weird header ( thinking e.g about Augeas (augeas.net) , I`m not talking about IDE's , GUI's etc here, I`m talking about system level tools and libraries that are designed to mangle standard files. For Augeas we could create a separate lens to manage these files , but other tools might have bigger problems with the concept.

As catch suggest a clean .htaccess should be capable of preventing people to access the .json files There's other methods to figure out if files have been tampered with , not sure if this even fits within Drupal (I`m thinking about reusing existing CA setups rather than having yet another security setup to manage) ,

In general to me tools such as puppet should be capable of modifying config files , and then activating that config with no human interaction required , obviously drush is a good candidate here to trigger the system after the config files have been change, but unlike some people think having to browse to a web page to confirm the changes is not an acceptable solution. Just think about having to do this on multiple environments ... manual actions are error prone..

Apart from that I also think the storing of the certificates should not be part of the file. What about a meta file with the appropriate checksums ? (Also if I`m using Puppet or any other tool to manage my config files then the security , preventing to tamper these files, is already covered by the configuration management tools, I do understand that people want to build Drupal in the most secure way possible, but I don't think this belongs in any web application.

When I look at other similar discussions that wanted to provide a similar secure setup they ran into a lot of end user problems with these kind of setups, an alternative approach is to make this configurable and or plugable. The default approach should be to have it enable, but the more experienced users should have the opportunity to disable this, or replace it with another framework. Making it plugable upfront solves a lot of hassle later.

Someone in the discussion noted :
"One simple suggestion for enhancing security might be to make it possible to omit the secret key file and require the user to enter the key into the UI or drush in order to load configuration from disk."

Requiring the user to enter a key in the UI or drush would be counterproductive in the goal one wants to achieve, the last thing you want as a requirement is manual/human interaction when automating setups. therefore a feature like this should never be implemented

Luckily there seems to be new idea around that doesn't plan on using a raped json file
instead of storing the config files in a standard place, we store them in a directory that is named using a hash of your site's private key, like sites/default/config_723fd490de3fb7203c3a408abee8c0bf3c2d302392. The files in this directory would still be protected via .htaccess/web.config, but if that protection failed then the files would still be essentially impossible to find. This means we could store pure, native .json files everywhere instead, to still bring the benefits of JSON (human editable, syntax checkable, interoperability with external configuration management tools, native + speedy encoding/decoding functions), without the confusing and controversial PHP wrapper.

Figuring out the directory name for the configs from a configuration mgmt tool then could be done by something similar to

  1. cd sites/default/conf/$(ls sites/default/conf|head -1)

In general I think the proposed setup looks acceptable , it definitely goes in the right direction of providing systems people with a way to automate the deployment of Drupal sites and applications at scale.

I`ll be keeping a eye on both the direction they are heading into and the evolution of the code !

Feb 14 2011
Feb 14

In a previous post I announced a new module I had contributed back to Drupal.org, Incremental Deploy, which extends Deploy module in various ways, one of which is to make deployable certain items that couldn't previously be deployed. This follow-up post discusses some of the issues you can come up against when deploying certain types of content under certain circumstances and also what is involved in making something deployable. It assumes prior knowledge of Drupal's API hooks and functions and of the basic usage of Deploy module but not of its inner workings.

What's in a node?

As a starting point, let's look at what happens when you deploy a node from a source environment to a target environment. In a nutshell, here's how it works:

You send a representation of the node over xmlrpc to be processed by a service on a remote site. The xmlrpc call specifies the 'node.Save' method, and on the remote site the module responsible for the node.Save method (which is node_service module, part of the Services package) takes the node and passes it to the callback function it has associated with that method. The callback function saves the node.

Now, you might assume that what is happening here is that the node gets passed as a regular, fully-loaded node object and that it is simply passed to node_save() on the other end. But the essential thing is, what happens on the other end must replicate exactly what would have happened had the user created or edited the node directly in that environment. Think of the number of times you've hook_form_alter'd a node form; if there's no form involved, none of that code gets executed, and this could lead to discrepancies between the source and target environments.

So, what node_service module actually does when it receives a node from an xmlrpc call is, it simulates submission of the node form using drupal_execute. The node, rather than being sent as a regular node object, is sent as a set of form values ready to be passed to drupal_execute() on the other end. This is all explained very clearly in the excellent comments within the deploy hook in node_deploy module.

So why might this be a problem? Well, because it means we actually need to care about UI-level finnicky form widgets when all we're trying to do is save some content to the database programmatically.

Here are the steps involved in a node being created in a source environment and it being deployed to the target environment:

  1. User fills out node creation form and hits submit, node gets saved on the source.
  2. User goes to deploy the node, filling in credentials for the remote environment.
  3. Once deployment has been initiated, Deploy module checks for any dependencies, e.g. node references or files.
  4. Deploy module calls the deploy hook of node_deploy module, which is responsible for deploying this type of entity.
  5. The node is converted into a set of form values and sent, via xmlrpc, to the remote environment, specifying the node.Save() method to process it.
  6. The node_service_save() function receives the node, prepares the node form, and drupal_executes it with the values it has received.
  7. If all went well, it returns the nid of the newly created node, otherwise it returns an error, which will abort the rest of the deployment.

What could possibly go wrong?

For the most part, this will be a problem-free process - your deployment logs will show a "success" result for every item - but there are some not too off the wall scenarios where you'll see various errors being returned, or worse - silent failures.

Scenario #1

You're using a custom CCK field which includes a filefield, but nodes of this type get deployed without their files. See "How do you make something deployable?" below.

Scenario #2

Your node contains a CCK date field that uses the date popup widget. Unless you have applied the patch at http://drupal.org/node/945526, you'll only be able to deploy date field values if you're using the select lists widget for your date fields.

Scenario #3

You're using upload module to add file attachments to nodes, they seem to deploy fine but just don't show up on the remote site. This is the one scenario where you'll get an extremely frustrating silent failure - the deployment of the node returns success but the node has not been saved on the remote site. This is due to a conflict with how upload module alters the node form and the solution is simply not to use core upload module for file attachments but to use a CCK filefield instead (see http://drupal.org/node/459192)

Scenario #4

You have a multi-language site and are using i18n module to enhance Drupal's localization features. This in itself is not a problem but problems can indeed arise if, under Administer » Site configuration » Languages » Multilingual System, you have your content selection mode set to "Current language and language neutral". Imagine a German content producer, creating German content of a node type that has a nodereference field. The current language is set to German, which makes it easy for the producer to select from German nodes for the nodereference field. Once created, the node gets scheduled for deployment to the QA server, along with a whole slew of other content in various languages. The adminstrator doing the deployment enters the credentials for the remote environment, but the language for the user she's logging in as (a generic "Deploy" user) is set to en-US. Now, when the node form for the node in question is being prepared by node_service_save() module, a completely different set of referencable nodes are returned for the nodereference field (because they are being filtered by language=en-US, not by language="de"), and the one that's been passed is not among them. Fail. The whole deployment breaks. There is a workaround for this and I will gladly post it if anyone has actually read through this scenario and found that they've experienced the same problem. I am starting to feel like I'm going into too much detail about very edge-casey issues that nobody else will even come across.

How do you make something deployable?

If we're talking about a node that's just not fully deployable, e.g. because it has a custom CCK field with a nodereference field or a filefield, there are two main steps:

  1. Implement hook_node_deploy_check() to make sure the referenced node or file will get added to the deployment plan as a dependency of your node.
  2. Implement hook_node_deploy() to make sure the referenced node or file gets its nid or fid swapped out for its correct value in the remote environment.

For entirely custom entities that are the responsibility of your custom module, your module needs to implement hook_deploy() which should package up the information as necessary for sending along to the remote site. Then you'll need to add a service module for receiving it on the other end. Of course you'll also need some way of adding it to a deployment plan. There are plenty of examples of all of this in Incremental Deploy module and its submodules.

Feb 01 2011
Feb 01

I've spent a good portion of the past several months working on a content deployment solution for one of our clients at Work [at] Play. The system is built on top of heyrocker's deploy/services paradigm but needed to expand on it in two important ways:

  1. We needed the ability to deploy things other than core entities, e.g. nodequeue contents
  2. We needed the workflow to be: "Deploy everything that's changed since the last deployment"

What I came up with was Incremental Deploy module and I have just released a beta version at http://drupal.org/project/incremental_deploy.

The module automatically adds changes to the current active plan, as they happen in the source environment. This includes changes to variables and block configurations, as well as the items already taken care of by Deploy module, such as nodes, users, comments and taxonomy terms. There are submodules that deal with the deployment of the following:

  • Nodequeue contents (i.e. nodes added, removed or rearranged)
  • Global flags (such as when an editor marks an item as "featured"
  • Meta tags (as provided by the nodewords module

Knowing when content has changed

How can we tell when content has changed on a Drupal site? When a user submits a form? When a hook gets fired? When it changes in the database? Unfortunately, the answer varies depending on the type of entity involved as Drupal has as yet no standardized interface into pieces of content. So the various entities that get scheduled for deployment on change get triggered in one of three ways:

  • api hooks (this works for items like nodes, nodequeues, global flags)
  • hook_form_alter (for capturing when a new language has been added or enabled)
  • database triggers

This last method is a bit of a hammer approach and is currently used for the variables, blocks and nodewords tables. I'm not completely confident of the wisdom of this decision but it made sense at the time and, well, it works.

A note on Nodequeue deployment

We didn't feel it necessary to be able to deploy new nodequeues across environments, as we generally define our nodequeues in code. But deployment of Nodequeue contents from one environment to another depended on the ability to uniquely identify nodequeues across environments, and for this we needed machine names. My co-worker, Owen Loy (a.k.a. mundanity) submitted a patch to Nodequeue module, which was committed very recently and so machine names currently only exist in the 6.x-2.x-dev branch of the module. So, the set-up is: you use the latest dev version of Nodequeue, you have your nodequeues defined in code, with machine names of your choosing, replicated across all environments, and you can push the contents of your queues up the chain.

Post Deployment

Incremental Deploy automatically adds a "post deploy tasks" item to the end of every plan it deploys. What this does on the remote is invoke a hook that allows other modules to act on the fact that a deployment has just taken place. We used this for the purpose of clearing caches other than the Drupal cache, e.g. Varnish, xcache, akamai.

What about Drupal 7?

Um, good question. It does feel odd to be releasing a shiny new module to the community and for it only to be available for Drupal 6 when all anyone cares about these days is Drupal 7. Well, surely that's not really the case - many of us will have Drupal 6 sites to maintain for some time. But as Incremental Deploy depends on Deploy module, the question of whether this functionality will ever be available in D7 hangs on whether Deploy module gets ported. I'm not sure if anyone currently has plans to do so. I'd love to jump up and say "Sure, I'll do it!" but my gut feeling is that it will be a non-trivial undertaking, and so I'm afraid it would suck more of my time than I can realistically spare.

In the meantime...

But for those who are still interested in the whole area of content deployment, albeit only for Drupal 6, stay tuned for my next post, where I'll discuss some of the biggest gotchas we came up against in doing deployments involving things like multi-language content, complex custom cck fields and more.

Aug 29 2010
Aug 29

DrupalCon Copenhagen comes to an end, as does my blogging hiatus.

Two of my primary learning objectives here in Copenhagen were configuration management and deployment process. Historically, working with Drupal in these areas has been unpleasant, and I think that's why there is tons of innovation going on in that space right now. It needs to be fixed, and new companies are springing up to say "hey, we fixed it." Often, the people running the companies are the same people running the project that encapsulates the underlying technologies. I'm referring to:

  • The hyper-performant core distro, Pressflow
  • Distros with sophisticated install profiles, like OpenAtrium, ManagingNews and OpenPublish
  • Configuration externalization with Features
  • Development Seed's "for every site, a makefile" workflow using drush make
  • The different-yet-overlapping hosting platforms Pantheon and Aegir

Dries commented in his keynote that as Drupal continues to grow, it also needs to grow up. I think advances like these are part of the community's answer to that. I want to wrap my head around some of these tools, while continuing to watch how they progress. Others, I want to implement right now. What's perfectly clear though is that I have a lot of work to do to keep up with the innovation going on in this hugely powerful community. Which is actually nothing new, but reading a blog post about these technologies doesn't make my jaw drop the way that it does when I'm in the room watching Drupal advance.

About Drupal Sun

Drupal Sun is an Evolving Web project. It allows you to:

  • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
  • Facet based on tags, author, or feed
  • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
  • View the entire article text inline, or in the context of the site where it was created

See the blog post at Evolving Web

Evolving Web