Creating the ideal Drupal development and deployment environment
I have recently been wondering how a "perfect" development and deployment environment might look like for me if I had time to put the necessary pieces together. I am planning on working on this eventually, but I thought it might be wise to get my ideas out there to see if others had opinions or advice.
First, a little disclaimer. I am a developer. I live in PHP, Python, Java, etc. Since most of what I work on are Drupal based websites, I am almost always in Drupal module code (and occasionally in the theme layer). I have set up my fair share of Ubuntu based servers and have some experience with server virtualization, however I would not consider myself even close to a server expert or system administrator.
Since my company deals mostly with Drupal websites, this setup is based on developing and deploying Drupal 6 and Drupal 7 websites.
I have been using Aegir for the past few years and have not looked back. Most of my development environment is going to be based on using Aegir for the heavy lifting. It will require some additional features to be added onto Aegir through a good deal of custom modules.
The Development Environment
My idea for setting up development environments on multiple developers computers is to build a virtualbox Ubuntu server image that any developer can deploy on their system. It would be great if Aegir could then deploy development sites to these developers servers on their local systems. The reason behind this is that some of the developers I work with are more front end designers who are not familiar with setting up and deploying Drupal websites on their local system. I would expect them all to have basic command line/drush skills, but I want to eliminate the process of having them need to set up the Drupal site, manage the database, etc, all from their local environment.
Obviously there are a lot of issues with this idea. First I think a verify from the Aegir admin interface on this development website would have the possibility of wiping out a developers uncommitted changes. This also forces the requirement that the developer is at minimum VPN'd into the development network. This also poses minor issues with Aegir trying to run cron on these sites that may not always be online. This is one area that I would need to research much more thoroughly before I make any decisions on how to proceed.
Website creation will be handled predominantly using Aegir. I would also need to build in the ability to allow non server users to import existing Drupal websites. Beginr Media does a lot of custom development for existing sites and often times we only receive FTP information for the live site. I would like a non system admin to be able to grab the Drupal files directory, get a dump of the database, and then be able to upload these directly from the Aegir interface. Depending on if a full Drupal install or multi-site is uploaded, Aegir will respond accordingly and build out the site and migrate it to the requested platform (assuming there are no errors or platform inconsistencies). This could probably pose a huge security threat because of the ability to upload this information directly to the server and then have it execute website building actions. However, since the only users of this will be fairly technical users (all of which are employees), I think I could make this work.
When creating a website, either in Aegir or through the import process described above, I would like the ability to either define an existing Git branch or have it create me a new branch for the project on Github using their API. This would be used to ensure that every website was tracked in Git.
I would need the ability to relate sites (Pantheon does this great with their Development -> Staging -> Live setup). For instance, after creating the original development site, I need the ability to clone this development site and drop it on another developers local system. This would allow multiple Drupal developers to work on the same website without running into the errors that often happen when working on the same Drupal site.
Since the Aegir website knows the git branch for the website, along with the related development sites, it can monitor the status of the branch for changes. When a developer pushes up to the development version of the branch from their local environment, this should automatically trigger the development site to pull the updated codebase, run drush updatedb, clear the caches, and possibly even revert any enabled features. The developers would still need to pull the changes to their local environments (we wouldn't want to push anything automatically to them), or they could always just re-clone the site to their local environment after major changes.
Pushing to live
Production deployments would consist of cloning the development version of the site to a live server and somehow marking this as the live/production version of the site. This production version of the site would always monitor the master branch of the Git repo and automatically deploy any changes that make their way to master (while also updating the database, clearing the cache, and possibly reverting any features).
Handling change requests
This setup would always keep a development/staging site that would always remain connected to the live site. These sites could be located on different platforms, servers, etc, but would always be related. The development/staging site would be based on a Git development branch, while the live would be based on the Git master branch.
Change requests in the form of client requests, new features, bug fixes, etc, would follow a very simple but audit-able process. The change request would come in and be assigned to a developer. This developer would then create a clone of the development/staging version of the site and have Aegir automatically drop in on their local development environment. The developer would complete the change request and make a commit to the development branch in Git. This could automatically be pulled to the main development/staging site for testing. If all goes well, a merge request would be created and the code would be merged into the master branch by another developer. This would automatically update the code on the live site for a final round of testing. For relatively simple sites this does add some extra complexity, but for larger projects these techniques should already be in place anyway as it creates a defined and repeatable process.
Handling module and core updates
Module updates would be handled the same way the change requests were handled above. Core updates would follow the Aegir pattern of building a new development platform for the new version of Drupal, and migrating the development site to the platform. Testing would happen in this environment and if everything passed, a live platform would be created on the live server and the live site would be migrated over.
Additional developer tools
Additional development tools/options I would need to build into Aegir would be the ability to migrate data from the database backwards from the live environment to the development/staging site. At first this could be a simple database replacement and would just use the relationships that exist to show an extra Aegir task on the development/staging site that would allow pulling a fresh copy of the production database. This should employ some cleaning mechanisms to clear out or mask email addresses, passwords, etc. The reasons for this are two-fold. The first is to prevent sensitive data from entering the development environment, the second is to prevent live users receiving emails or notifications from the development site.
Drupal development and deployment environment conclusions
My current environment at Beginr Media is far from the ideal vision I discussed above, however I really want your opinion. If you have ideas or advice on things that you think should be added/removed/changed, let me know. I am planning on building out some of these modules as additional plugins for Aegir that I would then release on Drupal.org.
Before any of this can happen though, I want to make sure I am actually solving the problem in the most efficient way possible. Who knows, maybe there is something out there that will get me closer to my ideal environment with much less work (although I haven't found it yet).
I like the model of Pantheon, however it does not solve my developer/site import dilemma. Last time I checked there were still some things missing that I need such as SSL support (which Aegir has), etc. I would also like to keep things on servers that I have full control over. I plan on pushing a lot of sites through this environment and it will be much more cost effective for me to control the servers.
Another alternative/spin on the Aegir approach is to push everything through an installation profile for each site along with Drush make files for quickly building the website. I have done this in the past and it works pretty well, however in my case this would end up making pretty much every site its own platform (since almost every site we build ends up being significantly different). I may build a profile containing the modules/themes that we use on every site to get started, but building an installation profile/drush make setup for each site would be way to much overhead and seems to be a bad approach for my situation.
I need your help
If you made it this far, then I applaud your reading efforts and thank you for taking the time to look this over. However, I really would like your ideas or input in the comments below (or twitter @smthomas3). One of the greatest things about Drupal is the community and I am looking for some advice from some experienced Drupal deployment experts, or anyone with an opinion. If you know anyone that can help, please send them a link. Thanks for your help on this, who knows where this discussion may lead us!