Upgrade Your Drupal Skills

We trained 1,000+ Drupal Developers over the last decade.

See Advanced Courses NAH, I know Enough
Dec 18 2015
Dec 18

Drupal Content Hub

Creating a Content Hub using CouchDB and Drupal via the Deploy module.

[embedded content] Please enable JavaScript to view the comments powered by Disqus. blog comments powered by Disqus
Dec 18 2015
Dec 18

Deploy suite for Drupal 8

I just uploaded a screencast of how to use Deploy module in Drupal 8. It covers a few of the different possible use cases.

[embedded content] Please enable JavaScript to view the comments powered by Disqus. blog comments powered by Disqus
Oct 20 2015
Oct 20

Over the last 7+ years working with Drupal, one question always asked by clients is, how do I copy my content from staging to production? Generally the answer has been, “Don’t! Just add it on production”. Another option is to use the deploy module, which allows for the deployment of content between environments.

Drupal core has changed greatly in Drupal 8, the deploy module is following suit.

Multiversion
The multiversion module will be the foundation of all this change. It introduces three big changes to the way content is managed in Drupal.
Workspaces - All content is associated with a workspace, you can have multiple workspaces and switch between them. Think of them much like branches in version control systems such as git.
CRAP - Create, Read, Archive, and Purge. This is an alternative to CRUD (Create, Read, Update, and Delete). It means that in multiversion no content is deleted and everything is a new revision. Deleting an entity will just archive it, flagging it as deleted.
Revisions for everything - All content entities are revisionable with multiversion module. This means blocks, user, comments etc all have revisions.

Relaxed
The relaxed module is a RESTful API which is aligned to the CouchDB API. It exposes each workspace as a Couch database. This allows it to be replicated to another Drupal site running relaxed, to a native CouchDB database, or to something else that uses the same API, such as PouchDB.

Deploy
Finally the deploy module. This is now just a UI module as all the the content deployment is handed with external projects. The relaxed module exposes the API needed, a PHP based CouchDB replicator uses a PHP based CouchDB client to replicate / deploy the content.

The current plan is to have a “Deploy” button in the Drupal toolbar, this will open a modal where you can choose which site you want to deploy to and which workspace you want to merge into. It will be possible to deploy to a workspace on the current site. For example, a staging site may have two content editors, both working on different workspaces, then can then deploy / merge their content to the default workspace. This default workspace can then be merged up to the production site’s default workspace.

Tagging is also an idea being worked on. It will allow you to tag a workspace at a current point in time, then continue to edit content. When deploying from a tag it will only merge up changed until the tag was created.

These are big changes for the deploy module, but something that will make the content workflow a lot easier.

Please enable JavaScript to view the comments powered by Disqus.

blog comments powered by
Mar 10 2012
Mar 10

The lack of a robust content staging solution in Drupal has been long standing. In this post I won’t go into details of why this is such a difficult problem to solve. Instead, I’ll suggest a plan, to make a plan, to solve it! So, those of you that are interested in tackling a strategically important functionality for Drupal 8, please continue to read.

What’s being done right now?

Throughout the last 8-9 months I’ve been working on a rewritten version of UUID and Deploy for Drupal 7. It’s been moving slow for different reasons, one being because we’re very few people coding on it. But (dare I say it?) we are close to a release of Deploy!

Acquia started an initiative called Large Scale Drupal (LSD), focusing on identifying, planning and consolidating work on problems for large scale Drupal implementations. Content staging is one focus area, with wireframes and workflow discussions.

In parallel with all this, the Configuration Management Initiative landed the first UUID patch and the initial version of the file based configuration API. Both patches are important for improving the content staging situation in Drupal 8, but it’s not enough.

A plan, to make a plan

For those of you that are interested to work on this topic, here are some events you should participate in during DrupalCon Denver, and some points we need to discuss in order to get out of Denver with a solid plan on how to make content staging work better in Drupal 8!

Core conversation: Content Staging in Core, Tue 11:15am – 11:45am

Some points I will raise and we hopefully will discuss during my core conversation:

  • Why a file based configuration API isn’t the whole answer
  • What I think is needed in core to tackle content staging better (this will also be a separate post soon)
  • Next steps for the UUID implementation in core
  • Do we need some sort of documentation, group or sub-initiative to track this?

BoF: Content Staging in Drupal 7, Wed 11:45am – 1:00pm

  • Work being done in different projects for Drupal 7, like Deploy, State Machine, LSD etc.
  • How to consolidate work across those projects
  • Next steps moving towards a working solution in Drupal 7

BoF: Content Staging in Drupal 8, Wed 1:00pm – 2:00pm

  • Follow up on our discussion from the core conversation
  • Create issues, documentation or plans we find needed in order to move forward

Thought?

This blog post is a cross post of this g.d.o post, please discuss there.

Dec 28 2011
Dec 28

The next few days are like a timer counting down. Sometimes the conversations feel a bit like a NASA control room. We review and re-review our plans because this deployment isn't going to be a small one. It is likely in the top 3 in scope since I began over two years ago.

Day 12 - Testing

We started with Scrum, Scrum, Scrum, and Scrum of Scrums.

Testing continued on Day 12 - and all the while we continued to work through Product Manager user stories. Supporting artifacts were shared as well. This segued into the Project Managers reviewing points and starting to assign teams for the next timebox. For this cycle, we found that our story points were roughly double to the number of the points available for the next timebox. This requires a feedback loop with the Executive committee to prioritize completed set of user stories.

The Executive Technical Leadership Committee met where we discussed a variety of issues, successes, and follow ups from the prior week. This committee fulfills the role of a CTO.

Finally, I also reviewed a patent application.

Day 13 - Testing, Stories, and Horse Trading

We continued to work through user stories and test. We had a 90 minute meeting with the Executive Committee to discuss the priorities where some items were shifted lower on the priority list and, for intents, got knocked out of the upcoming timebox and into the next.

Testing is coming to an end at this point. Day 14 will be release day and so only the most critical items are prioritised to be addressed before we release. This means several status meetings throughout the day. In this timebox, we were still engaged with status meetings until later in the evening. This was partially due to a timebox that was 25% smaller than our normal 20 day period. It was also in part because we launched our new mobile experience transitioning from a third party, Verve, to an entirely Drupal based mobile site. This shift greatly enhances the user's experience on a mobile device by leveraging block logic from the desktop site and providing access to all Examiner.com content. The Verve experience was limited to the previous 30 days.

User stories and the supporting artifacts are being heavily examined at this point. We have to lock down our next timebox on Day 14 - and that is now less that a day away. We also anticipated that the morning release would be long based on the new features that have been made ready during the current timebox and previous ones. The good news is that the work of creating stories is largely done - there are a few holes that need to be filled - but there is also bad news. Even with the prioritization of scope, the points that will be needed for Timebox 6 exceed the points we have to expend. This will mean and tertiary review prioritisation prior to the planning days 1 and 2 for timebox 6. Fortunately the holidays have afforded us a week in which the team will be largely working on defects giving enough breathing room before the next timebox starting to do that evaluation.

Work went late into the evening. There was a 6 am MT release start time.

Day 14 - Deployment Day

We start at 6 am MT - a time which is a compromise between low traffic and a time when our entire distributed team can be available. Most of us work on deployment from our homes rather than in the office. We have several communication lines available during our deployments.

  • A logged "Live" IRC channel
  • A logged "Post Deployment Validation" IRC channel
  • This timebox we added a Voice line using Skype

We want everything logged during deployment. If we need to go back and see what someone was doing at any given time, the logged channels help in those forensics. Anything that is spoken in Skype is to be logged as well. The goal here is to allow for extremely rapid communication in an emergency that doesn't rely on the typing speed of any one person. One of the lessons learned in a previous timebox was just that - fingers, in a pinch, cannot trump the speed of someone saying out loud, "STOP".

Normally deployments, from start of code push to end of post deployment validation, take 2-3 hours. We were launching a new mobile site making this deployment a longer process. It was about 10 hours total if you include hot fixes that occurred after validation to mitigate defects that manifested after push. After the code push is complete and post deployment validation is done, team leaders huddle and discuss which issues that have arisen need to be addressed immediately. A rapid triage occurs were we identify what order the needed changes will be pushed. They are then fixed, pushed to staging, validated, pushed to production, and validated one at a time.

After deployment is complete, it is time to do the final lockdown of stories for the next timebox. Folks have one last opportunity to lobby for switching priorities. Stacey is busy preparing our demo plan for the next day and informing folks who will be presenting what order and for how long.

Day 15 - Demo and Retrospective

Demo Day - when we arrive we will start to prepare the Examiner lounge - it has a large screen that we hook laptops up to, a video camera so our virtual team can peek in and take part. We set up a Skype call and a video feed for folks to access. The demo lasts for about an hour with each Product Manager going over the work we've completed during the timebox. Generally several of the Development Team will present something as well. This time around quite a bit of time was spent on the new mobile experience and America Inspired which will, on the 9th of January, will boast a brand new flexible polling module with different states for roles and voting status.

The retrospective was held as a round table. It is designed to be a safe place where we can honestly discuss what worked and what didn't work in the timebox. It allows us to make adjustments to our process. We have changed things like timing for artifacts, length of our testing period, adding a environment synch and freeze between prod and staging for a few days after deployment, and adding voice to our deployments.

Rinse and Repeat

The process repeats itself. Currently we are moving into development for our 6th timebox, user stories and artifacts for timebox 7, and ideation for timeboxes 8 through infinity. We track our burn rates pretty closely and are able do a good job of estimating the time our stories will take. Every timebox we complete improves our team's efficiency. We learn new things about how to do our jobs better every 20 days. I am incredibly proud of our team and how our process has grown.

Image Credit from Flicker. "Stopwatch" by William Warby
sed under a Creative Commons License.

Bookmark/Search this post with:

Jun 28 2011
Jun 28

At drupal7camp in Leeds I presented about how we develop and deploy our sites using Git and various Drupal tools.

I talked about the pros and cons of using Features and various different tips and tricks that we find really useful when developing.

Here is a copy of my presentation uploaded to Slideshare.

Overall the weekend was a huge success with a good attendance and session list. Thanks to all those involved in making it a great get together!

Jul 31 2010
Jul 31

When creating a Drupal site for clients, many people use a temporary hostname for development purposes. What's more, that hostname is not generally on the same domain as the production site. I'm sure I'm not the only one to use client,mydomain.com to develop the www.client.com site.

[[wysiwyg_imageupload:21:]]

When you — like me — use Drupal in a multisite setup, where the same codebase is shared between many sites, you'll have per-site settings, files and images. These are located in a directory structure that's based on the hostname you're accessing the site under. Eg: the files and settings for client.mydomain.com live in the sites/client.mydomain.com/ directory.

However, when the time comes to deploy the site to its production hostname, those settings and files need to live somewhere else. Moving them can result in link and image breakage on the front-end, which is never a good look and means you might need to painstakingly update links and images on all your pages.

There is a simple way to avoid all that hassle by using a UNIX filesystem feature that's been around since the dawn of time: symbolic links. These are files that point at another file or directory, elsewhere on the system and — unlike Windows shortcuts — all applications will magically access whatever file or directory the symbolic link points at when trying to access the link.

By creating a settings directory named for the production site and creating a symlink to it, named for the development site, you can have Drupal use both, which can make life nice and easy down the line.

In my example, when first installing Drupal, you create a client.com directory for the settings files and copy the default settings into it. I don't use the default directory that comes with Drupal, so I always have a pristine copy of the default settings in there — much in the way that /etc/skel works on Linux.

# cd /path/to/drupal/
# cd sites 
# cp -a default client.com
# cd client.com 
# mv default.settings.php settings.php
# chmod 666 settings.php
# mkdir files
# chmod 777 files

And we can quickly check that the settings.php file and files directory exist with the correct permissions to let us install Drupal.

# ls -l
drwxrwxrwx 2 root www 4096 2010-07-31 12:25 files
-rw-rw-rw- 1 root www 9485 2009-09-14 22:59 settings.php

Excellent. Now for the symbolic link.

# cd ..
# ln -s client.com client.mydomain.com

And another check to see that we created the link correctly.

# ls -l
drwxrwxr-x 3 root www 4096 2010-07-30 13:50 all
drwxrwxr-x 3 root www 4096 2010-07-31 12:25 client.com
lrwxrwxrwx 1 root root   10 2010-07-31 12:27 client.mydomain.com -> client.com
drwxrwxr-x 3 root www 4096 2010-07-30 13:50 default

The l at the start of the line with "client.mydomain.com" on it indicates this is a link. The shell also shows what directory the link points at. You can now install Drupal normally by visiting http://client.mydomain.com/

When doing its bootstrap, Drupal will loads its settings from sites/client.mydomain.com/settings.php. Because  that directory is a symbolic link, it will load the same settings file when it bootstraps sites/client.com/settings.php.

There's one thing left to do now. Because we installed Drupal via client.mydomain.com, it will default to using the sites/client.mydomain.com/files directory to store uploads like images and attachments. To avoid this, you can set a different file system path.

Browse to Administration » Site configuration » File system and replace the file system path with the path that you will want to use on the production site, then click Save configuration.

[[wysiwyg_imageupload:25:]]

Because of the symbolic link, the files live in the same location on the disk, but Drupal will now use the name of the production site to create links. That means you no longer need to change them when you roll out the site on a new URL.

Unavoidable Caveats

Of course, there are caveats, what would life be without them. If you put themes any modules into a site-specific directory, Drupal will use the site-specific directory path to access them. In the example I used that means if you remove the symbolic link client.mydomain.com, Drupal may suddenly be unable to load modules or themes from sites/client.mydomain.com/modules and sites/client.mydomain.com/themes.

You fix this by logging in as Administrator and visiting the modules list page and the themes list page. Doing this will make Drupal re-scan all available directories for the module and theme files it needs to load.

If there are still files missing after that, you may need to update the system table in the database to manually update paths. Precisely how to go about doing that is material for a future blog.

About Drupal Sun

Drupal Sun is an Evolving Web project. It allows you to:

  • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
  • Facet based on tags, author, or feed
  • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
  • View the entire article text inline, or in the context of the site where it was created

See the blog post at Evolving Web

Evolving Web