Sep 22 2013
Sep 22

The Drupal community is abuzz with talk of BackdropCMS, a recent fork of Drupal. In case you missed it, Lullabot has a great interview with "founding forkers" Jen Lampton and Nate 'quicksketch' Haug.

There have been previous attempts to fork Drupal, but apart from Pressflow, I can't think of any that gained any traction. Even there, the performance enhancements in Pressflow are mostly merged into core for the next version of Drupal. But its goal is to be a drop-in replacement, and is thus almost fully compatible with contrib modules. We use it ourselves as the core of many kPlatforms.

BackdropCMS seems to be a beast of an entirely different nature. A couple years ago, in response to another core developer asking "Is it time to fork Drupal?", Larry "Crell" Garfield, commented that core D8 initiatives were "effectively building a new framework and porting Drupal to it." So wanting to keep something recognizable as Drupal seems like a pretty reasonable response.

Will we use BackdropCMS?

It's hard to say, but it certainly looks promising. The stated motivations of the project address many of the concerns I've heard voiced about Drupal over the past few years. These boil down to essentially two challenges:

  • Having to re-learn a new CMS every few years
  • Requiring our clients with limited budgets to re-build their websites every few years.

We've been experimenting with using Wordpress as an alternative for simpler sites, as well as exploring Drupal distributions to accommodate these concerns. In particular, OpenOutreach fits our client segment well and is committed to long-term maintainability.

What about Drupal 8?

I can't imagine that we would shy away from using Drupal 8 when a project's goals call for it. While we make a concerted effort to support small non-profits with affordable websites, we also work with larger organizations who have more ambitious goals.

Koumbit is also the principal shop behind the Aegir Project. We've been planning to re-write lots of our front-end components for awhile, so we see moving to Drupal 8 as an opportunity to do just that.

What about Aegir?

In all likelihood, Aegir will support BackdropCMS. Unfortunately, it's not yet clear whether the Drush maintainers intend to support it.

This actually presents a unique opportunity for Aegir, as it might prove useful in abstracting our platform code. Without Drush's ability to bootstrap a Backdrop platform, we need to treat it as an alien platform. However, since it's a fork of Drupal 7, we should be able to re-use install procedures, and other operations.

Keep calm and carry on hacking

Any way that you look at it, these next few months (and years) should prove interesting. While I'm certain that emotions will run high at DrupalCon Prague this week, let's keep things in perspective. Remember that our community is based on consideration and respect. We'll all profit by assuming good faith from all concerned.

Jul 03 2013
Jul 03

Installing an SSL certificate on Aegir 1.x is a bit tricky, but definitely manageable. It is easy to bring all your sites down if you don’t follow these exact steps. Running a “verify” task on your site, either intentionally or inadvertently, will rewrite your SSL config and attempt to restart apache, which, if everything is not set up perfectly, will cause fatal errors upon restart and lock you out of your Aegir control panel until they are fixed.  Check out the latest documentation from the Aegir project here before following this documentation to ensure that everything matches, but do note the specifics pointed out in the steps below:

1.) Let Aegir generate its own self-signed certificate, which will create two files in /var/aegir/config/server_master/ssl.d/SITENAME.COM/:

[email protected]-prod:/var/aegir/config/server_master/ssl.d/sitename.com# ls 

openssl.crt <– Replace this Aegir-generated file with the X.509 cert file from the cert provider.

openssl.key <– Replace this with the .key file generated for the CSR. Or, even better, just use this file rather than generating the CSR yourself.

2.) Add the bundle / intermediate file to this same directory, but you must name it “openssl_chain.crt” in order for Aegir to recognize or “import” this file.

 openssl_chain.crt <– This is the bundle / intermediate cert file from the cert provider, which must be created manually. It *must* be named exactly like this.

3.) If you don’t follow these exact steps, or forget to add the bundle / intermediate cert file, when Aegir runs a “verify” task on your site again (or if you run it yourself), Apache will fail to restart, and all your sites will go down! So, before you run a “verify” task on the site on the front-end to complete the installation, be prepared to comment out the SSL settings for your site at /var/aegir/config/server_master/apache/vhost.d/SITENAME.COM, restart apache, fix your errors, and start over.

Feb 02 2013
Feb 02

Episode Number: 

100

The big 100th episode! In this episode I go over my company's Drupal development environment (http://beginr.com). I discuss how we use Aegir (http://www.aegirproject.org/) to manage and deploy our Drupal websites. I also mention how the development process works using development and live environments spread across multiple servers dedicated to hosting Drupal websites. I mention what operating systems we use along with my person preference for browsers.

One thing I didn't mention is that I use Komodo Edit for my code editor. I think it is pretty awesome.

I also mention sponsorship opportunities, so if you want to become a sponsor of the Daily Dose of Drupal, here is some more information - http://codekarate.com/content/become-codekarate-sponsor

DDoD Video: 

Aug 29 2012
Aug 29

I have recently been wondering how a "perfect" development and deployment environment might look like for me if I had time to put the necessary pieces together. I am planning on working on this eventually, but I thought it might be wise to get my ideas out there to see if others had opinions or advice.

First, a little disclaimer. I am a developer. I live in PHP, Python, Java, etc. Since most of what I work on are Drupal based websites, I am almost always in Drupal module code (and occasionally in the theme layer). I have set up my fair share of Ubuntu based servers and have some experience with server virtualization, however I would not consider myself even close to a server expert or system administrator.

Since my company deals mostly with Drupal websites, this setup is based on developing and deploying Drupal 6 and Drupal 7 websites.

Using Aegir

I have been using Aegir for the past few years and have not looked back. Most of my development environment is going to be based on using Aegir for the heavy lifting. It will require some additional features to be added onto Aegir through a good deal of custom modules.

The Development Environment

My idea for setting up development environments on multiple developers computers is to build a virtualbox Ubuntu server image that any developer can deploy on their system. It would be great if Aegir could then deploy development sites to these developers servers on their local systems. The reason behind this is that some of the developers I work with are more front end designers who are not familiar with setting up and deploying Drupal websites on their local system. I would expect them all to have basic command line/drush skills, but I want to eliminate the process of having them need to set up the Drupal site, manage the database, etc, all from their local environment.

Obviously there are a lot of issues with this idea. First I think a verify from the Aegir admin interface on this development website would have the possibility of wiping out a developers uncommitted changes. This also forces the requirement that the developer is at minimum VPN'd into the development network. This also poses minor issues with Aegir trying to run cron on these sites that may not always be online. This is one area that I would need to research much more thoroughly before I make any decisions on how to proceed.

Website Creation

Website creation will be handled predominantly using Aegir. I would also need to build in the ability to allow non server users to import existing Drupal websites. Beginr Media does a lot of custom development for existing sites and often times we only receive FTP information for the live site. I would like a non system admin to be able to grab the Drupal files directory, get a dump of the database, and then be able to upload these directly from the Aegir interface. Depending on if a full Drupal install or multi-site is uploaded, Aegir will respond accordingly and build out the site and migrate it to the requested platform (assuming there are no errors or platform inconsistencies). This could probably pose a huge security threat because of the ability to upload this information directly to the server and then have it execute website building actions. However, since the only users of this will be fairly technical users (all of which are employees), I think I could make this work.

When creating a website, either in Aegir or through the import process described above, I would like the ability to either define an existing Git branch or have it create me a new branch for the project on Github using their API. This would be used to ensure that every website was tracked in Git.

I would need the ability to relate sites (Pantheon does this great with their Development -> Staging -> Live setup). For instance, after creating the original development site, I need the ability to clone this development site and drop it on another developers local system. This would allow multiple Drupal developers to work on the same website without running into the errors that often happen when working on the same Drupal site.

Automated deployments

Since the Aegir website knows the git branch for the website, along with the related development sites, it can monitor the status of the branch for changes. When a developer pushes up to the development version of the branch from their local environment, this should automatically trigger the development site to pull the updated codebase, run drush updatedb, clear the caches, and possibly even revert any enabled features. The developers would still need to pull the changes to their local environments (we wouldn't want to push anything automatically to them), or they could always just re-clone the site to their local environment after major changes.

Pushing to live

Production deployments would consist of cloning the development version of the site to a live server and somehow marking this as the live/production version of the site. This production version of the site would always monitor the master branch of the Git repo and automatically deploy any changes that make their way to master (while also updating the database, clearing the cache, and possibly reverting any features).

Handling change requests

This setup would always keep a development/staging site that would always remain connected to the live site. These sites could be located on different platforms, servers, etc, but would always be related. The development/staging site would be based on a Git development branch, while the live would be based on the Git master branch.

Change requests in the form of client requests, new features, bug fixes, etc, would follow a very simple but audit-able process. The change request would come in and be assigned to a developer. This developer would then create a clone of the development/staging version of the site and have Aegir automatically drop in on their local development environment. The developer would complete the change request and make a commit to the development branch in Git. This could automatically be pulled to the main development/staging site for testing. If all goes well, a merge request would be created and the code would be merged into the master branch by another developer. This would automatically update the code on the live site for a final round of testing. For relatively simple sites this does add some extra complexity, but for larger projects these techniques should already be in place anyway as it creates a defined and repeatable process.

Handling module and core updates

Module updates would be handled the same way the change requests were handled above. Core updates would follow the Aegir pattern of building a new development platform for the new version of Drupal, and migrating the development site to the platform. Testing would happen in this environment and if everything passed, a live platform would be created on the live server and the live site would be migrated over.

Additional developer tools

Additional development tools/options I would need to build into Aegir would be the ability to migrate data from the database backwards from the live environment to the development/staging site. At first this could be a simple database replacement and would just use the relationships that exist to show an extra Aegir task on the development/staging site that would allow pulling a fresh copy of the production database. This should employ some cleaning mechanisms to clear out or mask email addresses, passwords, etc. The reasons for this are two-fold. The first is to prevent sensitive data from entering the development environment, the second is to prevent live users receiving emails or notifications from the development site.

Drupal development and deployment environment conclusions

My current environment at Beginr Media is far from the ideal vision I discussed above, however I really want your opinion. If you have ideas or advice on things that you think should be added/removed/changed, let me know. I am planning on building out some of these modules as additional plugins for Aegir that I would then release on Drupal.org.

Before any of this can happen though, I want to make sure I am actually solving the problem in the most efficient way possible. Who knows, maybe there is something out there that will get me closer to my ideal environment with much less work (although I haven't found it yet).

I like the model of Pantheon, however it does not solve my developer/site import dilemma. Last time I checked there were still some things missing that I need such as SSL support (which Aegir has), etc. I would also like to keep things on servers that I have full control over. I plan on pushing a lot of sites through this environment and it will be much more cost effective for me to control the servers.

Another alternative/spin on the Aegir approach is to push everything through an installation profile for each site along with Drush make files for quickly building the website. I have done this in the past and it works pretty well, however in my case this would end up making pretty much every site its own platform (since almost every site we build ends up being significantly different). I may build a profile containing the modules/themes that we use on every site to get started, but building an installation profile/drush make setup for each site would be way to much overhead and seems to be a bad approach for my situation.

I need your help

If you made it this far, then I applaud your reading efforts and thank you for taking the time to look this over. However, I really would like your ideas or input in the comments below (or twitter @smthomas3). One of the greatest things about Drupal is the community and I am looking for some advice from some experienced Drupal deployment experts, or anyone with an opinion. If you know anyone that can help, please send them a link. Thanks for your help on this, who knows where this discussion may lead us!

Jul 09 2012
Jul 09

Koumbit has been building services based on the Aegir Hosting System, in one way or another, since the project's inception. So it's with great pride (and a little relief) that we happily announce the public availability of our latest. Our AegirVPS services provide managed, dedicated virtual servers with Aegir fully installed, monitored, maintained and supported.

Aegir is the only fully free and open source distributed provisioning system for Drupal. It allows you to manage anywhere from a few sites for a single organization, to thousands of sites across as many concurrent instances of Drupal, on as many servers, and for as many clients as you need. Since it's all built on Drupal and Drush, it can be customized and extended using all the tools our community is already familiar with. We at Koumbit are fully committed to keeping software free (as in freedom), so not only have we been building a great service, we've been making sure to build it on an entirely open source software stack.

Koumbit first engaged in the Aegir project from its very beginning, over 4 years ago, as active users, developers and maintainers. Since then, we've come to run all of our Drupal development and hosting on Aegir, maintaining and contributing regularly both to the core project, as well as a number of extensions to, among other things:

Eventually, some larger clients warranted having dedicated Aegir servers setup for their own teams of developers, with custom platforms, high performance caching and high availability clusters. This led us to develop Debian packages and Puppet modules to make their maintenance and support more consistent, reliable, secure and all around easier.

From there, the natural evolution was to continue this trend of automation, giving clients control of Puppet-based configuration management, adding a helpdesk, dedicated client support tools, documentation, and automated platform maintenance. Almost one year later, Koumbit's AegirVPS services are now fully online and prepared for broader release.

To do real justice to the features and other aspects of our AegirVPS services, we've begun publishing a new series of articles. These are intended to be part tutorial, part requests for comment, and all shameless plugs for our new AegirVPSs. The first of these covers our innovative configuration deployment mechanism.

In a similar vein, I'll be presenting a session at DrupalCon Munich in August: Aegir-based Business Models. I'll also present a second session, Fearless development with Drush, Vagrant and Aegir, where I'll discuss some new tools and techniques designed to optimize Drupal development workflows around Aegir.

Most importantly though, this service begins and ends with the people involved. Our team comprises seasoned and talented sysadmins, designers, developers, themers, support and accounting staff. Years of experience on the part of this whole team, as well as our intrepid early-adopter clients, have helped shape the Aegir Project as a whole, from reporting and fixing bugs, to suggesting, building, testing and refining features. I'm very proud to be a part of it.

Want more information? Interested in a demo? Contact us today!

Apr 10 2012
Apr 10

Install Barracuda

Determine the url of the master Aegir instance and , if you use use Virtualbox/localhost, make sure it's added to the hosts file (if you're using remote server, it should be resolving). 

Go to the Barracuda project page and open the install link. Grab the command to get the latest stable release of Barracuda.

Barracuda project page

Paste this command into the command line to run it and then edit BARRACUDA.sh.txt with vim, nano or any other text editor. 

The file is well documented, feel free to read it to become more familiar with various options available.

Edit _MY_EMAIL to your own (you can use localhost). You can also turn on autopilog (_AUTOPILOT=YES) to speed up the installation process.

If you're installing Aegir on localhost, I recommend entering your ip, hostname and aegir front url (make sure they're in the /etc/hosts). You don't have to do this on a remote server.

Barracuda install script

Two more things you only need to do on the localhost are disabling _DNS_SETUP_TEST and _SMTP_RELAY_TEST

disabling _DNS_SETUP_TEST and _SMTP_RELAY_TEST

Save the script and run it

$ bash BARRACUDA.sh.txt

Please note, the install script is going to create a configuration file (.barracuda.conf) in the same directory and will be using it to rerun installation or upgrades in the future. So if for some reason installation fails and you'll have to change parameters, edit this file.

If you turned on autopilot, the script needs your input only a couple of time. The most important gotcha - and that's where Barracuda installation is different from standard Aegir - is setting up mysql root password. Barracuda script actually generates the password for you and that's what you need to use. 

So, the script pauses with detailed instructions. First, it's going to ask you for the current password - just hit enter without entering the password. Then you enter "Y" to the next question. Then you need to copy the password generated earlier (scroll up a bit if you don't see it) and enter it twice.

Setting up MySQL password

Follow the remaining prompts. When the script finishes the installation process, you'll see the first time login link for the master aegir instance. You can use it to reset password and check the installation.

Aegir login link

Install Octopus

After Barracuda has installed the master Aegir instance, it's time to install at least one satellite instance using the Octopus script. You can use the master Aegir to host sites, however, it's generally recommended by the maintainers to use Octopus instances for that purpose. First of all, it comes preinstalled with several popular platforms (distributions) of Drupal. It also has additional modules and caching improvements to all platforms, standard or custom. 

You can grab install script from the Octopus project page or from the same Barracuda INSTALL.txt file. Before running it, open and edit email and user. The Octopus instances are available at username.hostname (for example, o1.barracuda - so make sure they're added to the hosts file). The default username in the script is o1 and you can keep it because it has the same privileges as the 'aegir' user.

Edit Octopus script

You can edit other options, including PHP version, the list of platforms and autopilot.

Edit Octopus script

If you're using localhost, edit IP and turn off the DNS check.

Edit Octopus script

Save the script and run it.

$ bash OCTOPUS.sh.txt

When the installation is finished, you'll again get the link to the new Aegir instance - this time it's going to be Octopus.

Octopus login link

If you need to install another instance, change the username and run the script again.

Octopus is installed, what's next?

While Barracuda/Octopus is based on Aegir, there're numerous changes and improvements to make it a high performance stack. I recommend looking at the "welcome email" (you can find a copy of it in the user directory in /data/disk/username/log as setupmail.txt).

Octopus setup mail

You can also go directly to Omega8 site and read the posts (at least on caching and available modules). 

Mar 29 2012
Mar 29

Once again, we're using a fresh install of Ubuntu 11.10 on Virtualbox  for convenience  (it's even easier on a server with a properly resolving domain name).

Install Barracuda

Determine the url of the master Aegir instance and , if you use use Virtualbox/localhost, make sure it's added to the hosts file (if you're using remote server, it should be resolving). 

Go to the Barracuda project page and open the install link. Grab the command to get the latest stable release of Barracuda.

Barracuda project page

Paste this command into the command line to run it and then edit BARRACUDA.sh.txt with vim, nano or any other text editor. 

The file is well documented, feel free to read it to become more familiar with various options available.

Edit _MY_EMAIL to your own (you can use localhost). You can also turn on autopilog (_AUTOPILOT=YES) to speed up the installation process.

If you're installing Aegir on localhost, I recommend entering your ip, hostname and aegir front url (make sure they're in the /etc/hosts). You don't have to do this on a remote server.

Barracuda install script

Two more things you only need to do on the localhost are disabling _DNS_SETUP_TEST and _SMTP_RELAY_TEST

disabling _DNS_SETUP_TEST and _SMTP_RELAY_TEST

Save the script and run it

$ bash BARRACUDA.sh.txt

Please note, the install script is going to create a configuration file (.barracuda.conf) in the same directory and will be using it to rerun installation or upgrades in the future. So if for some reason installation fails and you'll have to change parameters, edit this file.

If you turned on autopilot, the script needs your input only a couple of time. The most important gotcha - and that's where Barracuda installation is different from standard Aegir - is setting up mysql root password. Barracuda script actually generates the password for you and that's what you need to use. 

So, the script pauses with detailed instructions. First, it's going to ask you for the current password - just hit enter without entering the password. Then you enter "Y" to the next question. Then you need to copy the password generated earlier (scroll up a bit if you don't see it) and enter it twice.

Setting up MySQL password

Follow the remaining prompts. When the script finishes the installation process, you'll see the first time login link for the master aegir instance. You can use it to reset password and check the installation.

Aegir login link

Install Octopus

After Barracuda has installed the master Aegir instance, it's time to install at least one satellite instance using the Octopus script. You can use the master Aegir to host sites, however, it's generally recommended by the maintainers to use Octopus instances for that purpose. First of all, it comes preinstalled with several popular platforms (distributions) of Drupal. It also has additional modules and caching improvements to all platforms, standard or custom. 

You can grab install script from the Octopus project page or from the same Barracuda INSTALL.txt file. Before running it, open and edit email and user. The Octopus instances are available at username.hostname (for example, o1.barracuda - so make sure they're added to the hosts file). The default username in the script is o1 and you can keep it because it has the same privileges as the 'aegir' user.

Edit Octopus script

You can edit other options, including PHP version, the list of platforms and autopilot.

Edit Octopus script

If you're using localhost, edit IP and turn off the DNS check.

Edit Octopus script

Save the script and run it.

$ bash OCTOPUS.sh.txt

When the installation is finished, you'll again get the link to the new Aegir instance - this time it's going to be Octopus.

Octopus login link

If you need to install another instance, change the username and run the script again.

Octopus is installed, what's next?

While Barracuda/Octopus is based on Aegir, there're numerous changes and improvements to make it a high performance stack. I recommend looking at the "welcome email" (you can find a copy of it in the user directory in /data/disk/username/log as setupmail.txt).

Octopus setup mail

You can also go directly to Omega8 site and read the posts (at least on caching and available modules). 

Mar 21 2012
Mar 21

So, why isn't Aegir at DrupalCon Denver?

I've been asked that question repeatedly over the past couple months, and so I figured I'd write up a little explanation for anyone else that's curious.

In case you don't know what Aegir is (nevermind how it's pronounced), I'll take a minute for a brief intro. The Aegir Project is, to our knowledge, the only fully free and open source Drupal provisioning tool available to the Drupal community. It allows for painless deployments, backups, migrations and updates of a whole network of Drupal sites. Being built exclusively on Drupal and Drush, it can easily be extended by anyone already familiar with the tools and techniques of Drupal development. We have a vibrant community that centers around our IRC channel (#aegir on freenode.net), our community site, and the various d.o issue queues that make up the project. There are also a number of shops offering professional services and hosted solutions (such as Omega8.cc and us at Koumbit).

After DrupalCon Chicago, this is what Dries had to say:

Aegir is an important tool in the toolbox of the Drupal community.

So, first off, it isn't that we didn't want or intend to be present at DrupalCon Denver. Two of Aegir's maintainers, Steven Jones (darthsteven) and Antoine Beaupré (anarcat) submitted sessions, as did I (ergonlogic) and at least one other community member. Koumbit (along with Steven) also submitted a pre-conference training proposal. Unfortunately, none of these were accepted. This came as a bit of a shock to us, as our sessions at DrupalCon London had been very well attended; standing-room only, and overflowing into the hall. The pre-conference training there also received mostly positive reviews.

Without such high-profile events, it was hard to justify our attendance. Most of the maintainers are spread out between the UK, Poland and Australia, and travel to the US can be prohibitively expensive. Koumbit, though based in Montreal, is a non-profit, and members here faced a similar dilemma.

Thankfully, some intrepid community members organized a BoF (thanks notzach & shrop, among others!), which seems to have been well attended and lively.

That said, we intend to re-double our efforts to ensure we have sessions at DrupalCon Munich, and future DrupalCons. Also, we're very active in regional DrupalCamps (Koumbit in the north-east US & Canada, Steven across the UK, etc.), and we can always be found on IRC, the community site or the d.o issue queues. So if you're interested in the project, feel free to reach out to us any time, not just during the big conferences. :)

Feb 28 2012
Feb 28

I have been following the Aegir project for some time now, almost 3 years. It’s great to see how far the project has come along, and how easy it is to get an Aegir instance up (it used to be very challenging to install). However, I haven’t really fully embraced Aegir (yet) into our current workflows at ActiveLAMP. I’m still pondering how exactly I want to use the tool. Allow me to describe our current process, before I elaborate on how I want to use Aegir for deployment.

Our current workflow.

Early in 2010 we adopted a workflow for managing and building sites using Drush Make. Several articles inspired this new workflow for us, so I won’t go into detail of why it’s a good idea to use Drush Make for doing your builds, rather than having one giant repository of code in which 80% of that code you don't touch (hack) anyway.

On each of our developers machines we have one drupal core (a platform) that runs any number of sites that we may be working on at the time. We may have several platforms (different core versions i.e. 7.9, 7.10, 7.12, etc...) on our development machines. All sites that we work on are essentially a multisite within the specific platform (drupal core version) that we're working within. With every site we have an aliases.drushrc.php within its sites directory. This aliases.drushrc.php file holds meta data regarding what platform the site is running on locally, as well as where the production server is, and where the dev server is.

I wrote a custom drush command that actually sets all this up, so we don't have to do too much work. When we start a new site we just type `drush ns mynewsite.com` (drush new-site) and that will fire off a number of tasks:

  • Check drupal.org to see what the latest version of core is
  • Check local platforms directory on developer machine to see if that core version is installed.
    • If the core version is not installed, it is downloaded, a vhost is setup in the apache config, and the /etc/hosts files is edited for the new platform.
  • Then it checks our git repository for a repo called newsite.com.git and checks that out into a separate sites directory (not within the platform just downloaded)
    • If the site repo doesn't exist, the drush new-site command creates a new site directory with a few files copied over from the drush new-site command for defaults (site.make, .gitignore, and rebuild.sh)
    • git init, git add, git commit, and git push origin master are all executed with the initial files at the very end of the drush command.
  • The new sites directory is symlinked to the platform
  • Two drush alias files are created, a global alias file placed in ~/.drush, and an aliases.drushrc.php in the actual site directory.
  • Finally the site is installed with `drush si`

For deployment we use Capistrano invoked with a custom Drush command. This drush command looks at parameters set in aliases.drushrc.php within the sites directory and then fires off a capistrano command passing in arguments it finds in aliases.drushrc.php. We're only deploying the sites directory to the new environment, and capistrano takes care of symlinking it to the correct platform on the dev or production servers.

I'm leaving out a lot of minor details, but in a nutshell that's how our current workflow works for developing sites and deploying.

Thoughts for using Aegir for deployment.

Most people (that I've read about in blog posts and the drupal.org issue queue) that are using Aegir for deployment, are using install profiles for their custom code, and are utilizing a recursive make file technique to handle the build of the platform, as well as the profile. The process seems to work well, and it makes sense, but I'm not sure I want to handle our deploys this way for a number of reasons:

  • A platform has to be built for every deployment, with the "new" profile.
  • Really only one site is going to run on this platform, unless you put multiple profiles in the platform make file (which then leads to more issues if you have different development cycles for the number of sites you're currently working on.)
  • A whole bunch of extraneous tasks are being fired to build this platform, when only site code could only be deployed.
  • Most important to me, you're getting away from what Aegir does well, host instances of sites on a common platform.


My initial thoughts for handling this with Aegir is just integrating Capistrano with Aegir with a few drush commands and then expose those drush commands to hostmaster. I would also add a sites make file field to the node add form for sites, so that when creating a new site, you can specify a site make file, just like you can when you build a platform.

The process of deployment would still be handled by Capistrano, while still utilizing Aegir for creating, managing, and migrating sites. I'm going to start developing this functionality, but I'm curious to hear others thoughts on this, and if there any holes in how I think this could work.

Dec 29 2011
Dec 29

First of all, check the requirements for Barracuda on the project page. You need a freshly installed server that matches one of the distributions listed (all of them are Debian or Ubuntu).

Open the terminal of the server (ssh to it if it's remote), (you might need to switch to root) and grab the script (open the Readme to make sure the link is correct):

wget&nbsp;-q&nbsp;-U&nbsp;iCab&nbsp;http://files.aegir.cc/versions/BARRACUDA.sh.txt

Then edit the IP of the server and hostname in the script (using vi or nano). You can use 127.0.0.1 or 127.0.11 for localhost. Check /etc/hosts for the hostname and corresponding IP. If you don't have public DNS setup, add the aegir frontend as a host.:

_MY_OWNIP="127.0.1.1"
_MY_HOSTN="yourhostname"
_MY_FRONT="aegir.yourhostname"

You can also change

_MY_EMAIL="youremailhere"

Try running the script now.

bash&nbsp;BARRACUDA.sh.txt

If you run into the "invalid DNS setup" error, you can try disabling the DNS check in the script. Note that after you run Barracuda installer for the first time, it creates a configuration file and will use the values from this files for the installation. So if you need to modify them, you can either delete this file and modify Barracuda script or edit the config file directly:

nano /root/.barracuda.cnf

Search (Ctrl+W in nano) for _DNS_SETUP_TEST and set it to NO:

_DNS_SETUP_TEST=NO

You might also need to disable _SMT_RELAY_TEST in the same manner if you're installing Barracuda on the local server.

Run the script again and wait for all the components to be installed. You'll get prompted about the MySQL passwords and optional components. Once you're done, you'll get the link to your Aegir frontend. Check the previous screencasts on how to use Aegir.

Optionally, you can try installing Octopus to create instances of Aegir on the same server with prebuilt platforms based on popular Drupal distributions.

Oct 12 2011
Oct 12

Assigning aliases to sites is convenient for site administration because you don't have to be in the site directory any more (it's especially helpful for multisites). In addition to that, it also helps with the interactive shell. For examples:

$ drush core-cli

$ use @yoursitealias

Drush interactive shell

After that, you can run commands on your site without using the word "drush" or the actual alias.

@yoursitealias&gt; $ pm-update

To see the aliases on your server, use 'drush sa'. Additionally, 'drush status' will show you the location of the aliases file if you have one set up.

If you change to the drush directory and list contents, you'll see the examples directory. Among other files, it contains the example aliases file. Let's copy it to the drush root and rename it (from drush directory):

$ cp examples/example.aliases.drushrc.php aliases.drushrc.php

Open the aliases file to edit it. It's well documented with various examples. For starters, we'll be using a very simple setup with root (the directory of your drupal installation) and uri (the url of the site).

Aliases file - add sites

You can run 'drush status' and 'drush sa' to check if drush has loaded the new file and site aliases.

At this point, you can use aliases to administer your site like this:

$ drush @alias command

You can use aliases file to set up default command options by using 'command-specific' array:

Specifying commands

Open interactive shell and choose one of the sites to use. If you run 'status' now, it should show the passwords:

Drush interactive shell

Drush interactive shell is actually a wrapper for the standard shell, so all the same commands apply. Sometimes bash commands would take priority (for example, status on some systems) - in that case, use the prefix drush.

A note on Aegir users (or anyone who uses root and then switches to the user who owns Drupal directory) - it's best practice to be logged in as the user before activating drush shell, or you're going to run into permission issues.

Interactive shell permissions issues

Check /etc/passwd to make sure the user has a shell and that the user has a password (you can, of course, use the RSA keys instead).

Sep 30 2011
Sep 30

Aegir is wizardry, pure and simple. Once you’ve got it set up and running there’s almost no excuse not to have all of your sites on it.

There are plenty of great tutorials on moving your existing site into Aegir like Dboettger’s tutorial or Aegir’s Community tutorial but they assume that you’re pretty handy with the command line.

If you don’t feel comfortable with the command line or you don’t want to mess with it you can still pull the migration off.

If your site uses a standard codebase like Drupal 6.22 and you already have an Aegir platform installed you can skip this part. Otherwise make sure you have the backup and migrate module installed on the standalone site. Then fire up your favorite SFTP program like Cyberduck for Mac or WinSCP for Windows.

Drupal Core Aegir platform

Login to your Aegir server as the aegir user. Create a directory for your new platform in /var/aegir/platforms. Transfer the whole codebase over. The permissions should stay the same because you’re transferring as the aegir user. You might have to go back and change the group of the files directory, the private directory and the settings.php to www-data if you run into permissions issues.

Now you have your files moved over go into Aegir, click Create Content in the bottom right menu and select Platform to set up a new platform.

Creating a new Aegir platform

Once the platform is set up and verified you can create a new site with it.

The site will be a new site on a blank database.

To transfer over all your database files use backup and migrate. Create a new quick backup from your standalone site and restore your blank Aegir site with it.

Syncing databases with backup and migrate

It might sputter a bit, make sure to clear the cache and run cron. Then there’s going to be a bit of clean up to do. Make sure there are no hard coded image or file paths in your template. Check all of your nodes for hard coded paths as well.

If the backup and migrate hopelessly bricked your Aegir site wipe it out and create a new one. This time before running the quick backup in your standalone site disable all the modules but backup and migrate and then create the quick backup.

Once you’ve restored your Aegir site from that backup you can start turning modules on one by one to see where the issue is.

Good luck, and remember don’t let a fear of the command line keep you from working with Aegir.

Sep 21 2011
Sep 21

In the previous installment, we created a drupal 7 platform and a pressflow 6 platform, and installed a site on each. You can see the overview of your sites and run tasks on them by going to hosting/sites

Site overview

Backup the site

To run backup, go to the site page and press the Run button next to Backup. An overlay window will appear where you can enter the description of the backup. Press the Backup button. I use Pressflow site for this example.

Backup the site

On the site page, there's also an overview of Packages (modules, themes and profiles) available on the site (enabled or disabled).

Packages on the sites

Now let's enable Views module. Open the terminal and run drush commands using the site alias. Aegir creates aliases for all sites, platforms and servers. The full list can be seen by running drush sa.

drush site alias

Note that servers and platforms would have special prefix. To enable views, run

drush @yoursite en views views_ui -y


enable views

If you go to the site now, views should be available (if you haven't logged in before, you'll see the reset password screen).

Let's try downloading the latest 6.3 version of Views:

drush @pressflow6 pm-releases views

drush @pressflow6 dl views-6.x-3.0-alpha3

drush @pressflow6 updb

Now go to your website page in Aegir, click verify and go to the packages tab. Views should be enabled and be version 6.3.

Packages update

Restore site

Views 3 has produced errors so now I'd like to revert back to views 2. It's as simple as going to the site page, running Restore and choosing the appropriate backup.

Restore from backup

Now first run Verify again and then check the packages. Views should be disabled but it's still version 3. The reason is because drush downloads to sites/all/modules directory, however aegir backs up the database and everything in sites/yoursite.com directory, including files (and that's how it should be, modules should belong to the platform, not sites).

Site packages

Migrate (upgrade) sites

Migrating means moving site to another platform. It's most often used for upgrading (core and packages). Let's upgrade pressflow6 to drupal 7. Click on Migrate and choose the platform.

Migrate site

You can see information on Upgrades, Errors and Warning and you can also compare the chosen platform with the current one.

Compare platforms

Click on Migrate when done. Aegir will move the site to the new platform and run the database updates. You can also migrate to the same platform which effectively means renaming the site (changing the domain name). Since my site is no longer on Pressflow 6, I'd like to rename it.

Let's click on Migrate again. This time, there will only be Drupal 7 platform. Aegir can't downgrade the sites, only upgrade them. Let's rename the site and run Migrate.

Rename the site by migrating

Before going to the site, add it to your /etc/hosts file. You can now login to your new D7 site and enable various modules and themes.

Clone sites

Clone task allows you to quickly create copies of the sites (usually used for testing or development purposes). Keep in mind that you'll probably need a different platform if you're going to test updates (remember, the modules are in sites/all directory so they'll affect other sites on the platform).

So I've created a copy of my D7 platform called drupal7dev (it's easy to do with the makefiles). Now I can go to drupal7 admin screen and click on Clone. You'll need to give the copy a new domain, but otherwise the screen is the same to Migrate. Choose the platform and click on Clone.

Clone the site

Thanks for watching/reading this tutorial, if you have any comments, don't hesitate to post them!

Sep 16 2011
Sep 16

Configure Aegir hosting options

When you first install Aegir, make sure that the task queues are running at admin/hosting/queues (use the top menu toolbar to navigate around the site).

Hosting Queue

Then go to Hosting->Features to enable Hosting Queue and other options as needed.

Aegir Hosting Features

Aegir Overview

There're three main components to Aegir:

  • Servers - web (currently Apache, nginx) and database (MySQL, pgSQL, etc.)
  • Platforms - the actual Drupal installation, such as Pressflow or Drupal 7
  • Sites - anything that lives in sites/yoursite.com directory

After the initial installation, Aegir will have one (sometimes two) server with local web server and database (Apache2 and MySQL in this example), one platform called hostmaster with one site (called localhost in this example, but it can have a different name on a VPS). It's a special distribution of Drupal that provides front end for Aegir administration.

You can see all the servers, platforms and sites you manage clicking on the convenient links in the primary menu (top right corner). The site name link will take you to the site administration page with information and available tasks. I recommend running Verify if it hasn't run yet.

Aegir site admin screen

Setting up platforms manually

You can use two methods to create platforms where your sites would be hosted: manually on the server or using drush make. Let's use the manual method first.

Open the terminal and become the aegir user.

su -s /bin/bash aegir

If you're using local server, you might need to become root first (sudo -i).

Got to aegir home directory (/var/aegir). It should already have platforms directory so change to it. It's going to be empty for now. You can use drush to download core.

drush dl -drupal

Download platform manually

To create a platform, go to Content Management->Create content->Platform. Give it a name (ex: Drupal 7). For the publish path, enter the path to the drupal installation you've just downloaded (/var/aegir/platforms/drupal-7.8). Save the platform.

Enter platform information

A new verify task would appear in Queues. Wait for it to finish running.

If you now go to drupal-7.8 directory, you'll see it's not different from the regular Drupal installation except for a file that Provision has added called drushrc.php.

Platform directory after verifying

If you read it, you'll see a long list of arrays describing the platforms (modules, themes, etc.) If you cd to sites directory, there won't be any sites installed yet. Don't install them manually because Aegir will do it for you.

Installing a site with Aegir

Go to Create Content->Site. Give it a name, then choose Install profile. Here's a tricky part: you need to choose the profile first and then you'll see any platforms where these profiles are available. However, the names of installation profiles have changed in Drupal 7. So in order to make the new Drupal 7 platform available, you need to choose Standard profile (or Minimal, or Testing).

We only have one database server right now. Note that there's no web server because you can only set it when creating a platform. Save the site.Install a new site

While you're waiting for the installation task to finish, you can add the new domain to your hosts files (/etc/hosts).

Open the new site page and click on Log in to your site link.

Go to your site

The first time you do it, you'll get a reset password page and then will be redirected to the admin user page where you can change the password. Congratulations: your Drupal 7 site is up and running!

If you now go back to the sites directory, the new site is going to be there. If you change to it and list the contents, you'll see files and other subdirectories.

Sites directory

Aegir has also added drushrc.php file where the site information (database, passwords, etc.) is stored (and not in settings.php). So if you need to add or override some settings, don't do it in settings.php. Check Aegir docs for sites overrides.

Using drush make to create a platform

I've created and uploaded a simple makefile that references Pressflow and a few other modules and themes. If you haven't used drush make before, you can get familiar with the documentation on Drupal.org.

Platform makefile

Open Create Platform page, enter the name and the path to the makefile on the system. Once again, specify the path to the platform. Although it doesn't exist right now, Aegir is going to create it for us.

Create Platform

Once the platform verified, you can create a new site using it.Create a new site

Coming soon: how to use migrate, clone and other common site tasks.

Sep 06 2011
Sep 06

This tutorial covers installing Aegir hosting system for Drupal websites on Ubuntu, including all the requirements such as Apache and Drush. If you administer more than a handful of Drupal websites and regularly use Drush and Git, check it out, it's definitely a lifesaver.

Watch the Screencast

Tutorial In a Nutshell

I'm not going over the details here as we mostly follow the manual instructions on the Aegir website. There're just a few things to keep in mind here.

First of all, make sure your hostname in /etc/hosts and /etc/hostname only has lower case letters (you can check this by running hostname and hostname -f commands in the terminal).

Additionally, if you're installing on the local server (such as using Virtualbox, for example), the IP for your domain name in /etc/hosts should be 127.0.0.1 (it's probably going to be 127.0.1.1 by default).

Related stories: 

Jun 23 2011
Jun 23

Thu, 06/23/2011 - 16:11

I'm always moving sites from a "dev" url to the production url using Aegir.  Something like dev.mysite.com will become mysite.com.

In doing this, drupal has image issues, but its easy to fix.  You just have to do a search and replace in your database for the urls of your multi-site install.

I usually do something like:

UPDATE node_revisions set body = replace (body, 'sites/dev.mysite.com/files', 'sites/mysite.com/files');
UPDATE files set filepath = replace (filepath, "sites/dev.mysite.com/files", "sites/mysite.com/files");
UPDATE variable set value= replace (value, 'sites/dev.mysite.com/files', 'sites/mysite.com/files');
UPDATE boxes set body = replace (body, 'sites/dev.mysite.com/files', 'sites/mysite.com/files');

I usually do this in phpmyadmin and it fixes everything up just fine.

Just remember to double check everything (backslashes, files not file, sites not sites, etc...) and make a database backup before you do this.

Jun 11 2011
Jun 11

We are currently deploying an Aegir based system for one of our clients. The client provides a Drupal ‘reporting’ system for their software. They have many customers around the country. Each customer has a separate Drupal Deployment.

We deployed a feature server 2 years ago to manage keeping all the different sites at the same level. This was a rewrite of a .NET 1.x based system. And the client had lost the original source code. So, every time we added a new feature for one of our clients customers, it would break someone else. This was very frustrating. So from the very beginning, this project was predicated upon ease of maintenance and updates.

While we felt very successful with that aspect of the project, our client has grown to over 40 deployments. So, the process of upgrading each site was simply taking too much time.

This is where Aegir comes in. With Aegir, we have been able to reduce the ‘upgrade’ time from 2-3 hours down to 2-3 minutes. But there were a few issues we had to resolve.

The first big issue for us was the fact that each of these sites was maintaining an inventory of 10-250K PDF files. This is the essence of the reports. The PDF files are actually copies from the internal servers where the reports are generated. Since the information contained on the PDF files was not only private, but required protections from competitors, we had to put them into the Drupal ‘Private’ file system.

Now we are working with Aegir and discover that setting the ‘private’ file system on each site gets overwritten by Aegir on save. It always goes back to the same value.

Call us stubborn, but that is simply not what we wanted. We had those files on specific SAMBA shares and didn’t want to have to changed all that architecture. So we did a little work inside Aegir and found that the simple answer was to use the local.settings.php file that is optional in sites/[sitename].

The essence of the fix was to use the Variable overrides section of settings.php. But settings.php is controlled by Aegir and tends to get re-written. So we could not make the ‘fix’ there. But in an Aegir system they have an ‘include’ for local.settings.php at the very end of settings.php.

So that was it. We managed $conf inside of local.settings.php as follows:

global $conf;
    $conf['file_directory_path'] = '/var/private/files/standard';
    $conf['file_directory_temp'] = ‘sites/[site]/files/tmp’;

Where [site] is normally replaced with the site name.

Apr 13 2011
Apr 13

Recently I’ve been working on an Aegir hosting system to handle the deployment and management of child Drupal sites. I’m impressed with Aegirs ability to handle the install, deploy and backup of an entire network of Drupal sites. The one thing I found to be lacking was the install documentation for Aegir. So, I’m just going to outline the steps I took to download and install Aegir on three virtual boxes.

read more

Apr 02 2011
Apr 02

Sat, 04/02/2011 - 14:25

Update 1 - April 5th 2011:  I modified some user creation items, they were completely wack.

Yesterday I updated my main aegir site to 1.0-RC3 and about 10 minutes after I was done I got my monthly bill from Linode.  I had forgotten how big its grown.  I've been adding more and more sites to my server and needed more memory and hard drive space.  Linode makes it sooo stink'n easy to add on more stuff that its grown to the point that I should probably be doing something different.

I decided it was time to take a look at Amazon EC2.  Now I don't have time to do a full tutorial on how to install aegir on EC2 but here are my notes:

Setting up the EC2 instance

  • Start here: https://help.ubuntu.com/community/EC2StartersGuide
  • Amazon AWS: http://aws.amazon.com - create an instance with the ami found at the link above.
    • Be sure to create a key pair, download the key and don't lose it.
  • Get your x.509 cert keys by going to:
    • http://aws.amazon.com/account/
    • then click on security credentials
    • there is a tab about half way down to download it.
    • you will get 2 keys a pk-XXXXXX.pem and cert-XXXXXX.pem
  • Back to ec2startsguide.
    • I only did step 1 - 4 on "Installing the API tools"since I already had my key's
    • NOTE: the "Installing the API tools" section is for your local computer, if your running ubuntu (I am...well linux mint anyways).
    • I skipped to 3 and 4 of "Using the Ubuntu Images" to login with ssh to my instance.
    • NOTE: you can get your "<external-host-name>" by clicking on your instance in the AWS console.  It will show down below where all the info shows.

Installing AEGIR

  • I followed the manual install here: http://community.aegirproject.org/installing/manual
  • You will have to use "sudo" in front of most of your commands to run them as root.
    • example: sudo apt-get install apache2 php5 php5-cli php5-gd php5-mysql postfix sudo rsync git-core unzip
  • At "3.2. DNS configuration" I could NOT get the stupid FQDN (fully qualified domain name) to stay after a reboot of the instance. 
    • I found this: http://www.lazytweet.com/post/6687864275
    • basically edit /etc/rc.local so that it says "hostname aegir.mydomain.com" (no quotes) before the exit 0.
    • Also you will need to edit your /etc/hosts file like normal
  • At "5.3. Running hostmaster-install" I had issues running hostmaster-install... I got the error "Dummy connection failed to fail".  I figured out I had to remove the anonymous users from mysql for some reason (I saw someone talk about it on drupal some place).  To do that I:
    • Added another user that I could login to webmin with (did I mension I installed webmin... its easy): 
      • Create the use and home directory: sudo useradd -d /home/myusername -m myusername
      • Set users password: sudo passwd myusername
      • add to sudo:  visudo
      • add this to the bottom of visudoers: myusername ALL=(ALL) ALL
    • Then I logined to webmin and enabled the mysql module
    • Then deleted the 2 anonymous users.I must have had my hostfile set funny because it created 2 servers: aegir.mydomain.com and localhost.
  • Lastly,  I must have had something funkey in my /etc/hosts file.  It showed "localhost" as a server having mysql only and aegir.mydomain.com as another server having onlyapache.  I set aegir.mydomain.com to have both server and mysql, verified it... then deleted localhost server.

Sorry its such a quick writeup but I hope it helps you out.

Dec 27 2010
Dec 27

So far in this series we have covered a potential target market and business plan, resources and infrastructure and the tools required to deliver Drupal sites with a sale price of $100 per site. In this post I'll be covering some of the considerations when building Drupal platforms or distributions.

The sites which customers deploy will need to be based on a custom Drupal distribution or "distro". The distro should be modular and primarily driven by Features.

Customers shouldn't have to know anything about administering Drupal when they first buy their site. A customer should be able to turn functionality on and off as they want, through a simple user interface.

Features

The platform should contain a good collection of Features. The following list is an example of what you might offer customers:

  • Contact Form
  • Image Gallery
  • Products
  • Services
  • "Static" Pages
  • Blog
  • News
  • Mailing Lists
  • Social Network Integration
  • Office / Store Locations
  • Staff Profiles

When developing your list of things to include in the site, think in terms of functionality a small business would want, not what modules you should be using. The list of modules should be derived from the functionality, not the other way around.

As the features included in the platform will be modular and generically useful, you should consider releasing them publicly, via your own features server or drupal.org as full modules.

On top of the features listed above you will probably need to include some custom glue code to enhance the user experience. In my first post in this series I discussed the target audience not having high level computer skills, so the user interface should take this into account. Some of the language might need to be changed or form options modified to use sane defaults and some might even be hidden from the user.

Security

As each server may have hundreds or even thousands of sites running on it, security will be an important consideration. Like with all servers you should ensure it is properly locked down and only running the services you need. Apache should be configured to block access to most things except index.php and relevant client side files (images, css, js) and the files directory. At the Drupal level you should make sure that things like the PHP module aren't enabled and secure coding practices are adhered to. The user account given to your customer shouldn't be user 1, they should be user 2, with restricted permissions that only gives them access to what they need.

I strongly recommend that you read Cracking Drupal by Greg Knaddison.

Sales and Support

In order to attract customers you will need a site to promote the service and allow customers to sign up and hand over their credit card details. Drupal now offers 3 ecommerce projects, Drupal e-Commerce, Drupal Commerce and ubercart, you should investigate which of these best suits your needs. The sales system will need some custom code to hook into Aegir, which will be managing the actual site deployments. The sales and support platform/s should be managed in a similar manner to the customer sites.

Once you have paying customers, you will also need to provide them with some resources such as detailed documentation, video walk throughs, forums and possibly a ticketing system. This site can either be part of the sales site or a separate site. In the next instalment I'll cover support in more detail.

Deploying Platforms

We need to keep the whole process very automated, CPU cycles are a lot cheaper than workers. Building and deploying platforms should involve a few clicks or tweaking a configuration file. For example platforms could be built as Debian (or Ubuntu) packages (aka debs) using an automated build process that runs a test suite against the code base before building a deb. The debs could then be deployed using puppet and a post installation script can notify Aegir that it has been installed successfully. The whole process could involve very little human interaction. Migrating client sites to upgraded platforms could also be automated using a simple script which adds a migrate task for each site.

What's Next?

Now that we have the service almost ready to go, we should look into how we are going to get customers to part with their cash and how we will support them once they have paid.

Dec 26 2010
Dec 26

In the previous instalment of my $100 Drupal site series I covered resources and infrastructure. In this post I will be covering the development tools I think you need in order to build and sell Drupal sites at the $100 price point. Given that Drupal 7 is close to release, it is assumed that the sites will be built using D7. I don't believe that it is smart to invest heavily in Drupal 6 for new long term projects, given D8 could be out in 18 months and D6 would then be unsupported.

git

You are going to need a version control system for storing all of the code. There are many different version control systems available, but I think git is the most flexible and powerful option. Git allows for distributed development, offline commits and best of all, Drupal is switching to git. Gitorious is an open source clone of github which allows you to browse your git repository and manage integration of code from all of your developers.

If you are new to git I strongly recommend you get a dead tree copy of the book Pro Git.

drush

Drush is the DRUpal SHell, a command line interface for Drupal. It is an invaluable tool for developing and maintaining Drupal sites. Simple things like running "drush cc all" to clear all caches when developing sites, through to being able to sync databases from one server to another, or one continent to another, save many hours. Even if you're only running a handful of sites, you should have drush installed on all of your servers.

Modules

There are a few modules which you should be using if you want to be able to create polished Drupal based platforms. My list includes:

  • module allows developers to package up configuration as a Drupal module, including access to all the normal hooks that a module can implement. Feature is version control friendly and includes the ability to reset to a known good state, either via the Drupal web GUI or by using drush on the command line.

  • exports values from Drupal's variables table, allowing sane defaults to be set. This means your site will always be configured the way you want it to be.

  • allows you to manage contextual conditions and reactions for different portions of your site. It gives you control over displaying blocks,hierarchy of breadcrumbs, themes and other context sensitive configuration options.

  • is a menu module which makes it quick and easy to access items. (Disclaimer: I co-maintain the module)

  • can perform static analysis of your code to ensure it is secure and complies with the Drupal coding standards. The module can also be used to upgrade code from one major version of Drupal to the next.

  • is a collection of tools to help developers create Drupal sites and modules. Devel integrates with Admin to provide easy access to the devel actions.

  • provides a pluggable framework for integrating rich text editors into a Drupal site. Given the target market for the $100 sites, we can't expect our users to hand craft HTML, they'll be expecting "something like Word".

  • provides a way for end users to apply predefined CSS to designated parts of a site, without the user having to understand CSS.

Installation Profiles

When most people install Drupal for the first time they are blissfully unaware that they are using an installation profile. An installation profile sets out a base configuration for a site that is installed by Drupal. Until recently installation profiles contained a list of modules to install and a large chunk of hand crafted PHP code which setup all the configuration for a site. These days most of the configuration can live in Features and so an installation profile can be very lightweight. It is worth reviewing some of the more popular installation profiles, such as Open Atrium, Managing News or Drupal Commons, for inspiration. Installation profiles have changed a lot in Drupal 7, so you will need to port the ideas to the new way of doing things.

Drush Make

Drush Make is a tool for building Drupal platforms. Drush Make allows developers to specify all the components of their site in a text file, then use the command line tool to "make" the site. It is possible to specify a version of Drupal core (including Pressflow), contrib modules and themes, third party components, patches and external libraries.

Aegir

Aegir is a Drupal site deployment and management tool built using a collection of Drupal modules. With Aeigr you can manage your DNS, http (web) and database servers from a common UI. It is possible to move a bunch of sites from one server to another in a matter of minutes, not hours and the downtime will be measured in seconds. When security fixes are released, testing can involve a few minutes of work then a few more clicks to deploy it to all of your sites.

Workflow

Instead of me explaining development workflows, I'll defer to Miguel Jacq (aka mig5). Miguel's blog post entitled "Drupal deployments & workflows with version control, drush_make, and Aegir" is considered by many to be the key work on modern Drupal development workflows.

What's Next

Many of the tools I've covered today should be in your Drupal developer's tool bag. My next post will cover what I think are the important considerations in building the platforms for the service.

Sep 24 2010
Sep 24

I just upgraded 2 of my systems to alpha 14 and here are a couple of things I stumbled with:

Rename your vhost.conf files

I copied my old vhost files from ~/config/vhost.d to ~/config/server_master/apache/vhost.d.  I'm not sure if I was suposed to do that or if I was to reverify all the platforms and it would maybe recreate them in the right place... I dont know.  But my way worked.

Since all the new sites come in as mysite.com rather than mysite.com_80 (old way), I wanted to rename everything so it looked the same.  I ran this command

rename "s/_80*//g" *

That removed all the "_80"s off the ends of the files.

Mysql stuff

GRANT ALL ON *.* to 'aegir_root'@$AEGIR_HOST IDENTIFIED BY 'xxxx' WITH GRANT OPTION;

This part confused the crap out of me since I've never really done any multi-server mysql crap.  Basically you need to replace the $AEGIR_HOST with every host and/or ip that you want to have access to mysql.  Note the xxxx is where your password should go that you want people to use to login to aegir_root.

Here is an example,  I have server1 and server2.  Server1 will be my main server and I will add server2's info to aegir on server2.  For server1 to access the mysql stuff on itself I need to add:

GRANT ALL ON *.* to 'aegir_root'@server1.mysite.com IDENTIFIED BY 'mYcrAzypassword' WITH GRANT OPTION;

(could have used the ip of server1 instead of the domain name)

On server2 I need to to the above for itself and for server1 to have access:

Give server2 self access

GRANT ALL ON *.* to 'aegir_root'@server2.mysite.com IDENTIFIED BY 'mYcrAzypassword' WITH GRANT OPTION;

Give server1 access...

I used the ip for this because mysql on server2 was resolving to the linode domain not all the way to my domain... weird but easily fixed just by using the ip.

GRANT ALL ON *.* to 'aegir_root'@64.255.255.1 IDENTIFIED BY 'mYcrAzypassword' WITH GRANT OPTION;

Aegir multiserver SSH stuff

Follow the instructions on "lesson 3" here: http://groups.drupal.org/node/90564

Also you will need to make sure that user "aegir" has a shell.  Normal install of aegir gives you a user with a shell of /bin/false...  we want it to be either /bin/sh or /bin/bash.  Login with root and run:

chsh -s /bin/bash aegir

Reverify stuff before migrating or cloning

Also, I had to re-verify sites before I could migrate or clone it.  Otherwhise I would get an error like:

Aug 29 2010
Aug 29

DrupalCon Copenhagen comes to an end, as does my blogging hiatus.

Two of my primary learning objectives here in Copenhagen were configuration management and deployment process. Historically, working with Drupal in these areas has been unpleasant, and I think that's why there is tons of innovation going on in that space right now. It needs to be fixed, and new companies are springing up to say "hey, we fixed it." Often, the people running the companies are the same people running the project that encapsulates the underlying technologies. I'm referring to:

  • The hyper-performant core distro, Pressflow
  • Distros with sophisticated install profiles, like OpenAtrium, ManagingNews and OpenPublish
  • Configuration externalization with Features
  • Development Seed's "for every site, a makefile" workflow using drush make
  • The different-yet-overlapping hosting platforms Pantheon and Aegir

Dries commented in his keynote that as Drupal continues to grow, it also needs to grow up. I think advances like these are part of the community's answer to that. I want to wrap my head around some of these tools, while continuing to watch how they progress. Others, I want to implement right now. What's perfectly clear though is that I have a lot of work to do to keep up with the innovation going on in this hugely powerful community. Which is actually nothing new, but reading a blog post about these technologies doesn't make my jaw drop the way that it does when I'm in the room watching Drupal advance.

Apr 16 2010
Apr 16

I am doing multi-server Aegir work and deployments were taking 10 seconds to get code to the servers. I have a hostmaster server and a web-only server, both need provision. I edit code on my laptop with my usual shell and editor. Now pushing 3 codebases takes less than 1 second. Use SSH master connections by adding

Host *
ControlMaster auto
ControlPath ~/.ssh/master-%[email protected]%h:%p

to ~/.ssh/config. When another SSH connection is open, new connections piggyback on it, saving connection time and getting straight to the rsync.

#!/bin/bash
( rsync -azC hostmaster/ [email protected]:hostmaster-HEAD/profiles/hostmaster; echo aegir-dev hostmaster ) &
( rsync -azC provision/ [email protected]:.drush/provision; echo aegir-dev provision ) &
( rsync -azC provision/ [email protected]:.drush/provision; echo web provision ) &
wait

The -C skips .git files. The & backgrounds each task and wait waits, so all three are done in parallel.

Now I have my familiar shell and editor with localhost speed. No more jumping around servers and forgetting to update one of the provisions. And updating code is fast enough to not slow me down.

Feb 16 2010
Feb 16

Tue, 02/16/2010 - 12:09

There are 2 errors I get all the time when importing existing sites.  I verify my platform and when the site is trying to import, I get this error:

This one is pretty easy to figure out, I have the wrong database info in my settings.php file.  I probably had my DB password wrong or forgot to capitalize some letter.

Ok so easy fix.  I correct the username/password/db name and click retry in Aegir.

.... wait ... wait....

Ahh failed again, new error:

This one was a a little trickier... but then I noticed it was talking about drush being in "sites/all/modules".  I was like "say wahhh?"  I realized that the old platform I had brought in had an older version of drush installed in the modules folder.

Ok so delete the drush folder... and it works...

Feb 14 2010
Feb 14

Since the release of Aegir 0.3 I've been using Drupals multisite configuration like crazy.  Mainly because Aegir uses it but also because it just makes sense.  Starting a site in multisite works great, but moving an existing site to multisite configuration can cause some problems.

For example, on a regular Drupal install the images and file uploads are stored in the "sites/default/files" directory.  On a multisite install, the files/images are stored in "sites/example.com/files" directory.  This causes a problem becuase the links to most images/files that are in content will point to the wrong location:

So you'll get a site that looks like so:

If I only have 4 images on the whole site thats not really a big deal but if I have hundreds of images in hundreds of nodes I don't really want to fix each one individually.

Using Search and Replace in phpMyAdmin to Quickly Fix Links

The command for mysql that allows you to a search and replace is pretty simple:

update [table_name] set [field_name] = replace([field_name],'[string_to_find]','[string_to_replace]')

This isnt completely dumby proof as you need to know what table and field name you want to search/replace.

Most of my images are in the node body and are stored in the table "node_revisions" in the field "body".  I am going to replace anytime there is a "sites/default/files" with "sites/example.com/files".

UPDATE node_revisions set body = replace (body, 'sites/default/files', 'sites/example.com/files');

Simple enough.  I also want to change where the main file system is set at:

UPDATE files set filepath = replace (filepath, "sites/default/files", "sites/example.com/files");

I can run both of these at the same time.  I first go to my database

Then click the sql tab and run the command in it:

Thats it for the basics.  You may need to do other tables besides these 2 depending on what else you've got going on, but this should help you out.

Imagecache Issues

To fix this, I had to do something like so:

WARNING, I have not tested this code on D5 (all the above I tested on D6).  But should be similar on D5

UPDATE files set filepath = replace (filepath, 'sites/old.example.com/file/', 'sites/new.example.com/file/');

I had to remove all the sites/example.com/file/ (dont forget the end slash on this one) from all the links.

Test this first

Always do this on a test site first and have backups.  Dont just start going around search replacing on your sites.  That = bad.

Other things to think about

  • site logo
  • images in comments
  • images in things other than comments and nodes

Am I missing anything?

Feb 10 2010
Feb 10

Wed, 02/10/2010 - 12:35

Aegir .4 Alpha 5 Came out last week and so I thought i would do a quick video for those VPS newbies out.  Aegir really is sweet and if you are wanting to ever just test it, this is a good starting point for you.

This can be done w/ almost any linux VPS host.  I chose to go with Burstnet for the screencast because they got decent reviews from web hosting forums and is DIRT cheap.

I would recommend Linode if you are looking for a good VPS host.  I have a node w/ them and they have always been wonderful.  Don't take my word for it, they have hundreds of reviews all over the web bragging them up.

So don't be scared... jump right in.

Part 1:

Part 2:

Here are my slides w/ links for the video:

Apr 24 2009
Apr 24

So, I have been inspired from a recent video to go play with AEgir. I thought I would try it on my laptop, so off to install DAMP.

I must say, I was very impressed with the DAMP install. Simple, straightforward, quick. Oh yea, and it 'Just Works'! Very impressive. I played with the Drupal install for a few minutes and it all seems in order.

So, a little light reading on the AEgir page and it seems I need to start with installing 'drush'. A quick read of the instructions and well, this is different. A download from d.o that is NOT installed as a module. No problem and off we go.
Run through the steps in the README and change to the drupal base directory. Run drush and there are a few problems. Some warning messages. I push a little deeper and find that DAMP doesn't include the 'mysql' client. The server is there, but no client command to run. It seems that drush needs this for some of its 'sql' features. Another thing that I noticed, but thought was related to the missing client was that the 'drush sql conf' command returned an empty array. We will come back to that in a moment.

Well, I thought, ok, let's install MAMP then and see if the client is there. Sure enough, the mysql client is part of MAMP. Got them both running. Ran an export from the DAMP side. Ran an import on the MAMP side. And then copied the Drupal install into the MAMP directories.

Back to that empty array. I don't have any idea how Drupal under DAMP was accessing the database. I really need to dig a little deeper. This could be interesting. The $db_url entry had a purely generic setting. No real userid, password, or database name. So, of course, after the copy, I still had that generic entry. I replaced that with $db_url = 'mysqli://root:[email protected]:8889/acquia_drupal'; (these are the defaults) and the acquia install came up just fine in the MAMP environment.

So now I get to go back and play a bit more with drush. The 'drush sql conf' command actually displays the expected output, so it seems that it must be reading the settings.php file.

So far, I would say that the DAMP installer could use:
#1) a mysql client
#2) a conventional settings.php file entry for $db_url

It should be noted that I have just started playing with the whole drush / [DM]AMP environment and have probably made some incorrect assumptions or outright mistakes.

I'll keep plugging away at it and see what else I can find.

First thing to look for is the right place to post this where the developers can see it.
So many different interacting parts though. Not sure if it should go on AEgir, drush, or acquia (DAMP).


Apr 08 2009
Apr 08

created on Tue, 2009-04-07 16:37

In November of 2007, Raincity Studios struck a deal to acquire, Bryght, the pioneering Drupal hosting solutions company. By combining the Raincity Studios development and design team with the hosting expertise of the folks at Bryght, we hoped to provide comprehensive online solutions to a range of clients. The last year and half has been a great learning experience, however it is with regret that today we must announce that effective April 30th, 2009 Bryght hosted services will be phased out with a plan to be closed completely by May 29, 2009.

Over the past five years, Bryght and Raincity Studios have forged many wonderful and productive relationships with partners and clients around the world. We pride ourselves in being able to create mutually beneficial partnerships.  As two companies that emerged at the same time and grew up together in Vancouver, we have had no relationship more meaningful than that with Bryght. The shared vision and complementary personalities of the two companies made for a excellent fit and we believe the cooperative efforts produced great results. 

Bryght ideas, ahead of their time

Bryght was an early player in helping to build the Web 2.0 industry and one of the first group of people to put Drupal publishing tools in the hands of the untrained public (a scary prospect at the time).  As the first commercial venture in the Drupal world, Bryght paved the way for all sorts of individuals who earn a living today using Drupal.  As early innovators and active contributors to the Drupal community, the team at Bryght worked tirelessly to develop the Hostmaster module and Aegir hosting system giving site administrators tools to help manage multiple sites and rapidly deploy new site installations.  Their services, Bryght VPS and Bryght Light, released in 2005 were the first hosted Drupal offerings and made signing up for a turn-key Drupal website simple and effortless for the end user.

Over the past year as we worked to improve our offering by relaunching Bryght.com, and adding new languages (French and Chinese), and upgrading our servers and technology, it became apparent to us that in order to offer competitive pricing and the best possible service to our clients, hosting must be carried out on a scale that is beyond our current capacity and in turn we could not provide you with the level of service that we would expect from anyone else.  

In the midst of an economic downturn we recognize the importance of keeping our company agile and innovative.  At the moment, we want to focus on doing what we're best at: developing and delivering innovative applications and software solutions to power online communities. 

Adrian Rossouw, the architect and father of Aegir, is now part of the great team at developmentseed.org. Together they are committed to continue the effort that he and Bryght put forth over the years. Combined with their internal projects, Context and Spaces, we feel this is a great use of Adrian's talents and opens new options to the technology we helped develop.

We also want to wish the best to our "coopetitors" in the hosting market, especially those offering turn-key hosted Drupal solutions, such as Acquia Gardens and Fields, Lullabot (with their forthcoming and still unnamed entry into the market) and Work Habit’s Elastic2. We recognize that the continued success of these companies is good for everyone in the community and industry.

Existing Bryght Customers

For our existing Bryght Light and Bryght VPS customers, we thank-you for your business and want you to know that we are committed to making sure your transition to a new hosting provider will be as smooth and painless as possible.  We've partnered with hosting providers Rackspace and Slicehost and will be able to offer you competitive affiliate pricing as well as facilitate the migration of your site to its new home.  Depending on the traffic your site is receiving, we can find the best solution for your needs.  As a Bryght customer, you will receive more information by email today. 

If you have any questions at all, please contact us at [email protected]

Tags:

About Drupal Sun

Drupal Sun is an Evolving Web project. It allows you to:

  • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
  • Facet based on tags, author, or feed
  • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
  • View the entire article text inline, or in the context of the site where it was created

See the blog post at Evolving Web

Evolving Web