Dec 21 2011
Dec 21

Today is not your lucky day. The production server went down due to a hard disk error on the VM's host machine. You don't have a high availability setup because running additional servers 24x7 is quite costly. However, you need a new machine quickly.

In this blog post, I will tell you how we deploy brand new production machines in under 20 minutes. First, I'll talk about why it is so important to be able to deploy servers quickly.

Benefits of rapid deployment

  • Better testing environments. If you want to try Memcached out on the site, you want to reproduce the current production environment quickly. You are, after all, just trying it out, and you don't want to lose a day on it. If you can run one command and have the server running, the experiment is much more feasible during your busy work week.
  • Better development environments. If you just hired a new developer, and you want them to fix a minor bug on a site you're maintaining, they shouldn't have to waste time setting up the environment before fixing the bug. They shouldn't need to know about all the legacy baggage the site has, and they shouldn't have to pester you about how to set up every component of the site.
  • Better documentation. Instead of writing extensive documentation on each step of the deployment process, you can write a deployment script. Well-written code can rarely replace good documentation, but this is one case where relying on the code is suitable. With minimal documentation effort, you should be able to roll out of bed and deploy a new copy of your most mission-critical production sites in minutes.

Steps to Rapid deployment

Set up your Chef infrastructure

At Evolving Web, we use Chef, a configuration management tool. With Chef, you manage all your configurations in one git repository, and you write them in Ruby. This lets the Chef server build a new server or update stale configurations automatically. For example, Chef will occasionally add new public keys to our authorized_keys files as the team grows.

With Chef, your site's environment is reproducible and standardized. When you're attempting to reproduce bugs, it's good to know that your development VM is the same as the one running the production server.

Make a site-specific cookbook for launching the site

We use site-specific cookbooks to launch a specific site. This will use a rotating read-only SSH key to pull code and configuration from a git repository. Once Chef builds the site, you can sync a database from backups or the production server.

Document

Write a step by step guide that anyone can read without messing up. This takes clarity and detail. Having a reproducible process for creating the VM makes this much easier; you can document as you build a test VM with VirtualBox on your own desktop. If you find that a certain part of the process doesn't work properly, you'll notice while documenting. You can fix the Chef script and go through the process again. Since it's a short process (especially with VirtualBox templates), it's not a problem to start over.

Test the process with a beginner

Find the newest recruit on the team and send them a link to your documentation page. Tell them to set up a server in 30 minutes. Can they do it without your help? If so, you've succeeded. If not, it's back to the drawing board. Make those docs shine.

Having good documentation for this makes getting new team members up to speed much faster. Passing your deployment process down generation by generation through oral tradition is not how the professionals do it.

You're done!

Now you can rest assured that you can get that site running again with no hassle. The entire development process should be more pleasant too; you can boot up a development VM in half an hour, and make it a clean environment separate from all the experimental hacks you do on your desktop.

Resources

Jan 21 2011
mg
Jan 21

Drupal is widely recognized as a great content management system, but we strongly believe that Drupal offers a lot more than that – a framework, a platform, and a set of technology – to build and run enterprise applications, specifically on the cloud. This post is an attempt to explore the benefits and potential of Drupal on the cloud.

Elasticity

One of the last things the customers should worry about their websites is the performance degradation due to sudden spike in the traffic. For years, the customers had to size their servers to meet the peak demand. They overpaid, and still failed to deliver on promise, at peak load. Cloud solves this elasticity problem really well, and if you are using Drupal, you automatically get the elasticity benefits, since Drupal’s modularized architecture - user management, web services, caching etc. - is designed for scale-up and scale-down on the cloud for elastic load.

PaaS

If Heroku’s $212 million acquisition by Salesforce.com is any indication, the future of PaaS is bright. Drupal, at its core, is a platform. The companies such as Acquia through Drupal Gardens are doing a great job delivering the power of Drupal by making it incredible easy for the people to create, run, and maintain their websites. This is not a full-blown PaaS, but I don’t see why they cannot make it one. We also expect to see a lot more players jumping into this category. The PaaS players such as phpfog and djangy have started gaining popularity amongst web developers.

Time-to-market and time-to-value:

Drupal has helped customers move from concept to design to a fully functional content-rich interactive website in relatively short period of time using built-in features and thousands of modules. Cloud further accelerates this process. Amazon and Rackspace have pre-defined high-performance Drupal images that the customers can use to get started. Another option is to leverage PaaS as we described above. The cloud not only accelerates time-to-market and time-to-value but it also provides economic benefits during scale-up and scale-down situations.

Management

The cloud management tools experienced significant growth in the last two years and this category is expected to grown even more as the customers opt for simplifying and unifying their hybrid landscapes. With Drupal, the customers not only could leverage the cloud management tools but also augment their application-specific management capabilities with Drupal’s modules such as Quant for tracking usage, Admin for managing administrative tasks, and Google Analytics for integration with Google Analytics. There is still a disconnect between the cloud native management tools and Drupal-specific management tools, but we expect them to converge and provide a unified set of tools to manage the entire Drupal landscape on the cloud.

Open source all the way

Not only Drupal is completely open source but it also has direct integration with major open source components such as memcached, Apache SOLR, and native support for jQuery. This not only provides additional scale and performance benefits to Drupal on the cloud but the entire stack on the cloud is backed by vibrant open source communities.

Security

It took a couple of years for the customers to overcome the initial adoption concerns around the cloud security. They are at least asking the right questions. Anything that runs on the cloud is expected to be scrutinized for its security as well. We believe that the developers should not explicitly code for security. Their applications should be secured by the framework that they use. Drupal not only leverages the underlying cloud security but it also offers additional security features to prevent the security attacks such as cross-site scripting, session hijacking, SQL injection etc. Here is the complete list by OWASP on top 10 security risks.

Search and Semantic Web

One of the core functionally that any content website needs is search. Developers shouldn’t have to reinvent the wheel. Integration with SOLR is a great way to implement search functionality without putting in monumental efforts. Drupal also has built-in support for RDF and SPARQL for the developers that are interested in Semantic Web.

NoSQL

The cloud is a natural platform for NoSQL and there has been immense ongoing innovation in the NoSQL category. For the modern applications and websites, using NoSQL on the cloud is a must-have requirement in many cases. Cloud makes it a great platform for NoSQL is so is Drupal. Drupal has modules for MongoDB and Cassandra and the modules for other NoSQL stores are currently being developed.

Drupal started out as an inexpensive content management system, but it has crossed the chasm. Not only the developers are trying to extend Drupal by adding more modules and designing different distributions, but importantly enterprise ISVs have also actively started exploring Drupal to make their offerings more attractive by creating extensions and leveraging the multi-site feature to set up multi-tenant infrastructure for their SaaS solutions. We expect that, the cloud as a runtime platform, will help Drupal, ISVs, and the customers to deliver compelling content management systems and applications on the cloud.

Jan 14 2010
Jan 14

We are in the middle of a content migration of 70 websites to Drupal per this page. We have been using the Rackspace Cloud's Cloud Sites hosting environment as opposed to procuring our own server. At first, we thought this couldn't be done. Then, after some testing and back-and-forth with Rackspace and some research, we discovered it could be done via aliasing in Cloud Sites. See my post here.

However, in the last few months, we've been plagued by intermittent "No suitable nodes are aviable to serve your request" messages on our sites in our multi-site Drupal install in Cloud sites. The Rackspace folks' "fanatical tech support" has been helpful, but, honestly, we have considered leaving a few times. What's kept us there is the tech support and the quality of the service - when it works!

During these last few months, when we had issues, I complained loudly on Twitter and subsequently met several other users in the Rackspace Cloud using Drupal who had similar issues. Thus, I learned a lot about the Rackspace and Drupal. Basically, in Drupal, everything is stored in a MySQL database. So, loading a Drupal site might, no matter how simple, still requires potentially hundreds of DB queries. If one fails or is hung up in the Cloud Sites' MySQL clusters, the page takes forever to load or returns a "No suitable nodes..." message.

Last week, we had more issues with our sites loading extremely slow (more than 30 seconds!) or the "No suitable nodes..." messages being returned. So, we jumped on Live chat in the Rackspace Cloud control panel and were connected with someone named Vincent W., who was very knowledgeable and helpful. He made an excellent suggestion - why not host our MySQL Databases on a separate, dedicated Cloud Server in the Rackspace Cloud? While this would mean a bit more money, it promised stability that couldn't be matched by remaining in the Cloud Sites' clustered environment - MySQL clusters where many DB queries are being fired simultaneously from many users.

So, yesterday, we deployed a Cloud Server (it only takes a few minutes in the Rackspace Cloud) and installed MySQL and migrated a few of our databases. What's genius about this solution is that our files and code (in Drupal, the php and the files) are still in Cloud Sites and don't need to be tweaked at all. We simply needed to point the MySQL database URL in the settings.php for the few sites we moved and voila!, they were now being served by the version of the database on the Cloud Server. And, I'm happy to report that the sites are loading quite consistently fast. (Note: it's hard to tell if they are loading better than Cloud Sites because the days and weeks after a major outage in Cloud Sites, everything loads fast - cache is cleared, etc.). But, they seem to work great and thus, so does this solution.

So, we'll monitoring these sites over the next few weeks and will likely migrate all of databases in this manner. This way, we can take advantage of the hosted environment that Cloud Sites offers, with the stability of having our own server for the databases Drupal needs to serve the pages. Not exaclty an elegant solution but not too bad either!

Share this

About Drupal Sun

Drupal Sun is an Evolving Web project. It allows you to:

  • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
  • Facet based on tags, author, or feed
  • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
  • View the entire article text inline, or in the context of the site where it was created

See the blog post at Evolving Web

Evolving Web