Feb 09 2017
Feb 09

Not sure who's to blame, but we have a new HTML validation method from GoDaddy. It is an improvement from the "none HTML validation at all" phase they went through, but took me a while to make it work with apache. The problem was the dot/hidden directory they request to put your validation code in: /.well-known/pki-validation/godaddy.html

In my case there were a couple of reasons why this was difficult:

  • I didn't know about the hidden directory (.) block in Apache.
  • In my case some domains run the whole site over HTTPS, so I needed to make the new rules co-exist with the old HTTPS redirection rules.
  • I have a mixture of hostings. For some sites I control apache, so I could use Virtual Host configurations. But for others (like the ones running on Acquia) I need to create .htaccess rules.

The solution was much simpler than I anticipated, but quite difficult to debug. Finally I made it work for both environments.

I could have used the DNS ownership verification method, but in my case that means I would need to involve the people owning the domain. In my experience that takes longer and it can become really involved when owner doesn't know anything about DNS.

Using Virtual Host config (possible on self hosted sites)


RewriteEngine  on
RewriteRule    "^/\.well-known/pki-validation/godaddy\.html/" "/godaddycode.txt" [PT]
RewriteRule    "^/\.well-known/pki-validation/godaddy\.html$" "/godaddycode.txt" [PT]
    

If the site is only running on HTTPS and I have a redirection rule I'll work around these URLs. The rules below will work together with the one above:


RewriteCond %{REQUEST_URI} =!/.well-known/pki-validation/godaddy.html
RewriteCond %{REQUEST_URI} =!/.well-known/pki-validation/godaddy.html/
RewriteRule ^(.*)$ https://www.mydomain.com/ [R=permanent,L]
    

Using only .htaccess rules (and with no HTTPS redirection):


# GoDaddy verification rewrite rules
<IfModule mod_rewrite.c>
  RewriteRule    "^.well-known/pki-validation/godaddy.html/" "/godaddycode.txt" [PT,L]
  RewriteRule    "^.well-known/pki-validation/godaddy.html$" "/godaddycode.txt" [PT,L]
</IfModule>
    

Using .htaccess rules when site is only running over HTTPS:


# GoDaddy with HTTPS redirection rules
<IfModule mod_rewrite.c>
  # GoDaddy PassThrough rules
  RewriteRule    "^.well-known/pki-validation/godaddy.html/" "/godaddycode.txt" [PT,L]
  RewriteRule    "^.well-known/pki-validation/godaddy.html$" "/godaddycode.txt" [PT,L]
  # Set "protossl" to "s" if we were accessed via https://.  This is used later
  # if you enable "www." stripping or enforcement, in order to ensure that
  # you don't bounce between http and https.
  RewriteRule ^ - [E=protossl]
  RewriteCond %{HTTPS} on
  RewriteRule ^ - [E=protossl:s]
  # Redirect HTTP to HTTPS
  RewriteCond %{HTTP:X-Forwarded-Proto} !=https
  RewriteCond %{REQUEST_URI} !=/godaddycode.txt
  RewriteRule ^(.*)$ https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301]
</IfModule>
    

And to make this work on Acquia I had to borrow some rules from D8 .htaccess

So I replaced these sections/rules:


# Protect files and directories from prying eyes (D7)
<FilesMatch "\.(engine|inc|info|install|make|module|profile|test|po|sh|.*sql|theme|tpl(\.php)?|xtmpl)(|~|\.sw[op]|\.bak|\.orig|\.save)?$|^(\..*|Entries.*|Repository|Root|Tag|Template)$|^#.*#$|\.php(~|\.sw[op]|\.bak|\.orig\.save)$">
  Order allow,deny
</FilesMatch>

# Block access to "hidden" directories whose names begin with a period... (D7)
RewriteRule "(^|/)\." - [F]
    

With these D8 sections/rules:


# Protect files and directories from prying eyes (D8)
<FilesMatch "\.(engine|inc|install|make|module|profile|po|sh|.*sql|theme|twig|tpl(\.php)?|xtmpl|yml)(~|\.sw[op]|\.bak|\.orig|\.save)?$|^(\.(?!well-known).*|Entries.*|Repository|Root|Tag|Template|composer\.(json|lock))$|^#.*#$|\.php(~|\.sw[op]|\.bak|\.orig|\.save)$">
  <IfModule mod_authz_core.c>
    Require all denied
  </IfModule>
  <IfModule !mod_authz_core.c>
    Order allow,deny
  </IfModule>
</FilesMatch>

# Block access to "hidden" directories whose names begin with a period... (D8)
RewriteRule "(^|/)\.(?!well-known)" - [F]
    

I hope this helps someone else. I know it took me some time to figure it out and couldn't find an specific blog post about it.

Note: Just to be super clear, you should put the code given by GoDaddy into a file called godaddycode.txt on your docroot directory.

Dec 29 2014
hw
Dec 29

Before we start, I should say that this is a heavily delayed post, sitting in my queue for a few months now. The overall process to install the server on virtual machine and configure it for Drupal development is still valid and you only need to change the OS version from Ubuntu Server 14.04 to the latest one now (Ubuntu Server 14.10 at this time). Also, I have moved on to using vagrant for my setups and will hopefully write a post on that soon. I think this post will still serve some value if you are looking at setting up a virtual machine (or even a physical machine) from scratch.

A bit of history first. I bought a new laptop in November 2013 and decided not to run Linux on that. I was using Ubuntu on my previous machine for all my development work but I had a lot of problems with Chrome eating up all the memory, even for a few tabs. Comparatively, Chrome on Windows ran fine with a lot more tabs. I tried a few things but never solved the problem. The only workaround that worked for a while was using an extension called The Great Suspender. When I got my new machine, I just decided to stick to Windows and run Ubuntu Server in a VM inside VirtualBox. Here is a complete list of steps to set up a machine.

Tools on Windows

Console2 running cygwin on Windows

Console2 running cygwin on Windows

Download VirtualBox, Console2 and cygwin and set them up. VirtualBox and Console2 are simple and you would install them just like any other Windows program. Cygwin is a little bit more complicated. Follow the instructions in cygwin installation section to get started. Note: Console2 and cygwin are not strictly required but highly recommended. Console2 gives you a nice terminal interface with tabs and cygwin lets you directly use ssh and other Unix commands right from Windows. It is very convenient.

Ubuntu Server

Create a new Virtual Machine and install Ubuntu Server 14.04 on it. I have written a separate post which shows the steps to configure the virtual machine and install Ubuntu Server in detail. Once installed and configured, you can proceed with the next steps here.

Samba

First, we need a way to share our files. The model we will use is to store all the files in the virtual machine and access them in Windows via shares. This, of course, means that we can’t access the files if the virtual machine is not running. This is inconvenient but it is the faster option. NOTE: With the vagrant setup, we avoid this but run into the performance problem. More about this in my post on a vagrant setup.

To make the files in the virtual machine accessible to Windows, we need to install samba server. The guide helps you along but basically you just run this command.

sudo apt-get install samba

Next, configure samba by editing /etc/samba/smb.conf. There are more examples in the guide and in another community post. These are the settings in my smb.conf.

From above, the option ‘force user’ and ‘force group’ are particularly important. This makes sure that the files you create from Windows get the right user/group and permissions as well.

ssh-agent

If you are using public-private key authentication, as you should, and you have encrypted your keys, you would be asked for password everytime you connect to the machine. You just need to install ssh-agent to fix this. Follow the steps in this Stackoverflow answer and this blog post to enable ssh-agent in cygwin.

Adding more partitions

As a best practice, I mounted my document root (/var/www) on a different virtual drive. This way, you can reuse the same virtual disk on another system if you wanted to try something out. There is a detailed guide on how to set this up by creating a new virtual disk in VirtualBox and then configuring it in Ubuntu.

Other configuration on the Windows host

In some cases, the virtual machine cannot access Internet. Run the command below in your Windows host to modify VirtualBox settings. Replace “VM Name” with the actual virtual machine name.
VBoxManage modifyvm "VM name" --natdnshostresolver1 on

For nicer colors in the console, see this post. This has to be done only once, not for both cygwin’s bash and after ssh. It basically changes the palette for 16 color codes used in the terminal.

Other configuration in the virtual machine

Set UseDNS to no in sshd_config in the virtual machine for quicker ssh when logging in to your virtual machine. The setting basically disables reverse lookup of the client. There is more information in this StackExchange answer and this article.

Setup drush, php, etc… in the Ubuntu Server (virtual machine). Install packages like php5-gd, php5-apcu, php5-mysqlnd, php5-memcache, php5-gd, php5-curl, php5-xdebug. For drush, follow the instructions at the drush documentation pages.
sudo apt-get install php5-gd, php5-apcu, php5-mysqlnd, php5-memcache, php5-gd, php5-curl, php5-xdebug

Enable rewrite module for Apache. Without this, the .htaccess rewrites might work but might cause weird errors, such as File not found 404 error for virtual paths.
sudo a2enmod rewrite

If you want to access the MySQL server in the virtual machine from tools like MySQL Workbench or similar tools, give access to root (or other user) with GRANT PRIVILEGES and comment out bind-address setting in /etc/mysql/my.cnf. Replace ‘password’ with the actual password you would like. More details in this answer. The SQL for GRANT PRIVILEGES statement looks like this:
GRANT ALL PRIVILEGES ON *.* TO [email protected]'%' IDENTIFIED BY 'password';

Set up a ~/.my.cnf file to store your mysql users and passwords for various commands. This is great to access commands like mysqldump and mysqladmin from the command line without worrying about the username and password all the time. This is what the file should look like:

Conclusion

You should have a decent machine for Drupal development, or regular PHP development by now. There are more steps, of course, like installing composer but I think that could be subject of a different post. Like I mentioned earlier, I have moved on to using vagrant and this post is mainly for reference. I will be writing a post on that soon. Follow me on twitter for updates and subscribe to RSS for staying updated.

Aug 20 2013
Aug 20

Robert M. White
TL;RD

  1. Performance matter for all websites
  2. Performance is not just (80%) frontend
  3. SPDY kills 80% of your frontend problems

What
In the Drupal and broader web community, there is a lot of attention towards the performance of websites.

While "performance" is a very complex topic on its' own, let us in this posting define it as the speed of the website and the process to optimize the speed of the website (or better broader, the experience of the speed by the user as performance.

Why
This attention towards speed is for two good reasons. On one hand we have the site that is getting bigger and hence slower. The databases get bigger with more content and the the codebase of the website is added with new modules and features. While on the other hand, more money is being made with websites for business even if you are not selling goods or run ads.

Given that most sites run on the same hardware for years, this results in slower websites, leading to a lower pagerank, less traffic, less pages per visit, lower conversion rates. And in the end, if you a have a business case for your website, lower profits. Bottemline: If you make money online, you are losing this due to a slow website.
UFO's
When it comes to speed there are many parameters to take in to account, it is not "just" the average pageloading time. First of all the average is a rather useless metric without taking the standard deviation into account. But apart from that, it comes down to what a "page" is.

A page can be just the HTML file (can be done in 50ms)
A page can be the complete webpage with all the elements (for many sites around the 10seconds)
A page can be the complete webpage with all elements including third party content. Hint: did you know that for displaying the Facebook Like button, more Javascript is downloaded then the entire jQuery/backbone/bootstrap app of this website, non cacheable!
And a page can be anything "above the fold"

Moon Retro future
And then there are more interesting metrics then these, the time to first byte from a technologic point of view for example. But not just technical PoV. There is a website one visits every day that optimzes its' rendable HTML to fit within 1500 bytes.
So ranging from "First byte to glass" to "Round trip time", there are many elements to be taken into account when one measures the speed of a website. And that is the main point: webperformance is not just for the frontenders like many think, not just for the backenders like some of them hope, but for all the people who control elements elements in the chain involved in the speed. All the way down to the networking guys (m/f) in the basement (hint sysadmins: INITCWND has a huge performance impact!) Speed should be in your core of your team, not just in those who enable gzip compression, aggregate the Javascript or make the sprites.

Steve Souders (the webperformance guru) once stated in his golden rule that 80-90% of the end-user response time is spent on the frontend.

Speedy to the rescue?
This 80% might be matter of debate in the case of a logged in user in a CMS. But even if it is true. This 80% can be reduced by 80% with SPDY.
SPDY is an open protocol introduced by Google to overcome the problems with HTTP (up to 1.1 including pipeling, defined in 1999!) and the absence of HTTP/2.0. It speeds up HTTP by generating one connection between the client and the server for all the elements in the page served by the server. Orginally only build in chrome, many browsers now support this protocol that will be the base of HTTP/2.0. Think about it and read about it, a complete webpage with all the elements -regardless of minifying and sprites- served in one stream with only once the TCP handshake and one DNS request. Most of the rules of traditional webperf optimalisation (CSS aggregation, preloading, prefetching, offloading elements to different host, cookie free domains), all this wisedom is gone, even false, with one simple install. 80% of the 80% gone with SPDY, now one can focus on the hard part; the database, the codebase. :-)

The downside of SPDY is however that is is hard to troublshoot and not yet avaliable in all browsers. It is hard to troubleshoot since most implementations use SSL, the protocol is multiplexed and zipped by default and not made to be read by humans unlike HTTP/1.0. There are however some tools that make it possible to test SPDY but most if not all tools you use every day like ab, curl, wget will fail to use SPDY and fallback like defined in the protocol to HTTP/1.0

Measure
So can we test to see if SPDY is really faster and how much faster?
Yes, see Evaluating the Performance of SPDY-Enabled Web Servers (a Drupal site :-)
SPDY performance

So more users, less errors under load and a lower page load time. What is there not to like about SPDY?

Drupal
That is why I would love Drupal.org to run with SPDY, see this issue on d.o/2046731. I really do hope that the infra team will find some time to test this and once accepted, install it on the production server.

Performance as a Service
One of the projects I have been active in later is ProjectPAAS, bonus point if you find the easteregg on the site :-) . ProjectPAAS is a startup that will test a Drupal site, measure on 100+ metrics, analyse the data and give the developer an opinionated report on what to change to get a better performance. If you like these images around the retro future theme, be sure to checkout the flickr page, like us on facebook, follow us on twitter but most of all, see the moodboard on pinterest

Pinterest itself is doing some good work when it comes to performance as well. Not just speed but also the perception of speed.

Pinterest lazyloading with color
Pinterest does lazyload images but also displays the prominent color as background in a cell before the image is loaded, giving the user a sense of what to come. For a background on this see webdistortion

Congratulations you just saved 0,4 seconds
If you are lazyloading images to give your user faster results, be sure to checkout this module we made; lazypaas, currently a sandbox project awaiting approval. It does extract the dominant (most used) color of an image and displays the box where the image will be placed with this color. And if you use it and did a code review, be sure to help it to get it to a real Drupal module.

From 80% to 100%
Lazyloading like this leads to better user experience. Because even when 80% of the end-user response time is spent on the frontend, 100% of the time is spend in the client, most ofthen the browser. The only place where performance should be measured and the only page where performance matters. Hence, all elements that deliver this speed should be optimized, including the webserver and the browser.

Now say this fast after me: SPDY FTW. :-)

Jun 18 2013
Jun 18

In my new position at Drupalize.Me I have the luxury of helping a lot of projects in little ways. Being able to context switch quickly helps a lot. This means I've put a lot of time into how my workstation is setup so that I can easily move from one project to another. With the new job I also decided to add OSX to the mix of computers that I use on a daily basis. (Putting me at three platforms: Ubuntu Linux, Windows, and OSX.) There are certain tasks which I prefer doing on each of the three different systems, but I don't believe any one system is required to be an excellent Web developer. On the Lullabot side, most of our developers work either in OSX, or in Linux (usually Ubuntu). If you're "stuck" on a Windows machine, that doesn't mean you can't be great at your job, it just means there are fewer options available to you (and sometimes that's a good thing).

For each of my three different machines, I've got three slightly different developer environments.

Windows

My Windows machine has the easiest setup for Drupal development. I use the Acquia Dev Desktop. It's a one-ish-click install to get a web server stack working on your computer. There is a package for both Windows and OSX. Drupalize.Me has a free video on how to install the Dev Desktop on your computer. You'll want to watch Part 1 and Part 3 of the video to learn how to setup Drupal locally. Part 2 walks you through the specifics of how to install a different version of Drupal core. Obviously you only need the version of Drupal that you actually want to play with—this is probably Drupal 7. But maybe you're interested in trying out Drupal 8, or maybe you're working with a really old site and you need to have a local copy of Drupal 6 to mess around with?

Mac OS X

Next up is my OS X laptop. Most of the Drupalize.Me team uses a MAMP installation for local development. We have a free video that covers the installation of MAMP and a second video that covers the configuration of MAMP. For some reason I haven't entirely fallen in love with the idea of MAMP yet, so I actually have Vagrant installed on my OS X laptop with a "baby Ubuntu" server. If you're interested in learning more about how I set this up, please add it to our suggestion box! If you, like me, have limited hard drive space, you're probably smart to go the MAMP route instead of the Vagrant route. I'm constantly running out of hard drive space when I go to conferences and want to spin up a different environment for all the little projects I want to kick around. To save on hard drive space I'm leaning towards having only one (or maybe two) instances of Vagrant that I treat like my Linux laptop machine. Which leads nicely into my third and final machine.

Linux

Finally, my Linux laptop is a true LAMP stack with Apache, MySQL, and PHP installed directly onto the machine without any one-click-install wrappers. For over a decade Linux was my primary development environment (yup, even for front end work). My Linux laptop is pretty bare-bones. The most exciting developer tools that I've added to my dev stack are Drupal and Drush. (Not including browser plugins, like Firebug, of course.) Most of the work I do on this machine is light site building and theming. For as long as I can remember I've taken advantage of Drupal's built-in multisite capabilities, so I only have one installation of Drupal core, a bunch of common contributed modules in the "sites/all" folder, and per-site themes and custom modules in the appropriate directories for each different project. If you're lucky enough to be using Linux as your primary workstation, we've got a couple of videos to help get you started: installing a Web server on Ubuntu and configuring an Ubuntu Web server. If you'd like to learn more about Drupal's multi-site capabilities, there are a couple of older conference videos about how multisite works in our video archive.

And that's my setup in a nutshell. Most of these videos are also linked from our quick start page for recreating demo sites. Unless you're using the Dev Desktop, you'll definitely want to peruse the remainder of the instructions on this page to go from having a developer environment to having Drupal installed locally.

Oct 22 2012
Oct 22

I needed to locate current user’s home directory, to find and parse a config file like ~/config.ini. The current user is a user that is currently running a PHP CLI script. I have found a neat solution in drush, ready to copy & paste into your function:

  // getenv('HOME') isn't set on windows and generates a Notice.
  $home = getenv('HOME');
  if (empty($home)) {
    if (!empty($_SERVER['HOMEDRIVE']) && !empty($_SERVER['HOMEPATH'])) {
      // home on windows
      $home = $_SERVER['HOMEDRIVE'] . $_SERVER['HOMEPATH'];
    }
  }
  return empty($home) ? NULL : $home;
Mar 08 2012
Mar 08

Create your own personal website sandbox

Your personal computer can be turned into your very own local web server so that you can develop a website on your machine before pushing the code into a production environment. In this set of three free videos we show you just how to do it. We cover three major operating systems: Mac OS X Lion, Windows 7, and Ubuntu 11.10 (Oneiric Ocelot), so pick your OS and get started with your local web server:

Mac OS X

Windows

Ubuntu

Feb 01 2012
Feb 01

In this video we set up a LAMP stack web server for local development on an Ubuntu desktop (version 11.10). We will walk through installing and using tasksel to get the web server installed, then we'll add phpmyadmin to the mix. Once we have the server up and running, we'll look at where the web root is and how to add websites. Finally we wrap up with a quick look at how to start, stop, and restart our server when needed.

Nov 29 2011
ben
Nov 29

Drupal has the option of outputting its watchdog logs to syslog, the file-based core Unix logging mechanism. The log in most cases lives at /var/log/messages, and Drupal's logs get mixed in with all the others, so you need to cat /var/log/messages | grep drupal to filter.

But then you still have a big text file that's hard to parse. This is probably a "solved problem" many times over, but recently I had to parse the file specifically for 404'd URLs, and decided to do it (partly out of convenience but mostly to learn how) using Node.js (as a scripting language). Javascript is much easier than Bash at simple text parsing.

I put the code in a Gist, node.js script to parse Drupal logs in linux syslog (and find distinct 404'd URLs). The last few lines of URL filtering can be changed to any other specific use case you might have for reading the logs out of syslog. (This could also be used for reading non-Drupal syslogs, but the mapping applies keys like "URL" which wouldn't apply then.)

Note the comment at the top: to run it you'll need node.js and 2 NPM modules as dependencies. Then take your filtered log (using the greg method above) and pass it as a parameter, and read the output on screen or output with > to another file.

Oct 25 2011
Oct 25

Let me introduce myself – I'm Steph, the newest member of the Ixis team. I've now completed 6 weeks in my new role as Business Development Manager at Ixis.

The directors have taken a leap of faith employing a non-techie to drive forward their thriving Drupal development, hosting and support business.

Although I was familiar with open source systems in a previous life I had only really dipped my toe into the world of Drupal. I must admit I was a little anxious that everyone would laugh at my amateur hour attempts to converse in Drupal speak, especially as my initiation test was to “go to London, set up and run an the Ixis sponsored exhibition stand at Drupalcon” – the national Drupal conference attended by 1500 people from 54 countries - an easy first day on the job then?!

However, I spent 4 days in the company of some of the most awe inspiring and creative people I could have hoped to meet, which was probably the best training I could have wished for and the whole event was a great success for the business and raising our brand profile.

I'm now picking up new techie terms every day, from Drush to Puppet and not forgetting the cURLS

Most recently, I've been working on proposals for our larger clients including the British Council, the British Epilepsy Association and launching TES Growing Ambitions.  

Its also been a hugely productive month in the hosting department, despite being a member of staff down there have been several new launches and lots of progress towards automating provisioning of new servers (hat tip to Peter L .)

Finally, I'm preparing the office ready for our new starters as the recent recruitment drive has been going well.  We are always on the look out for new staff members though so do keep in touch.

Here are some highlights of my first 6 weeks living in Drupal world:

  • Learning what its like to share an office with 8 techies

  • Meeting and speaking to our lovely clients

  • Seeing that despite the doom and gloom spread about the recession, that businesses who are ripe and ready for work can thrive

  • Meeting the lovely folk at Daresbury Science and Innovation Campus who constantly strive to support our businesses

  • Meeting Prince Andrew HRH who came to launch the new Enterprise Zone building Vanguard House

  • Attending T-Shirts and Suits coffee morning hosted by David Parish at the beautiful new Leaf Cafe on Bold Street

  • Seeing Mike C – one of our directors being interviewed for BBC Granada news

  • Rallying the team to take part in “organised fun” including quiz nights and our contribution to the British Epilepsy Association tea party charity fundraiser

  • Eating my body weight in biscuits whilst introducing the team to my weird and wonderful tea collection

I'm now looking forward to planning how I can continue to drive forward the Ixis business plan and get more enthusiastic developers, site technicians and linux sys administrators through the door as we try to keep up with demand.

So this time in 6 weeks whilst you are warming your mince pies by the fire getting ready for Santa to bring you one of our Druplicon Stress Balls, I'll still be dreaming of a Drupal Christmas.

Sep 13 2011
Sep 13

New Relic is the all-in-one web application performance tool that lets you see performance from the end user experience down to the line of application code. Check out New Relic at http://newrelic.com/

Step 1: Setup repository and install packages

First become root

[[email protected] ~]$ sudo su -

First you need  to add New Relics repositries to your server

32 bit

[[email protected] ~]# rpm -Uvh http://yum.newrelic.com/pub/newrelic/el5/i386/newrelic-repo-5-3.noarch.rpm

64bit

[[email protected] ~]# rpm -Uvh http://yum.newrelic.com/pub/newrelic/el5/x86_64/newrelic-repo-5-3.noarch.rpm

Next install the yum package

[[email protected] ~]# yum install newrelic-php5

Last we need to run the New Relic install script

[[email protected] ~]# newrelic-install

Enter '1' and press enter.

Lastly you will need to edit the file /etc/newrelic/newrelic.cfg and add you api key fron newrelic.cfg

[[email protected] ~]# nano /etc/newrelic/newrelic.cfg 

Lastly restart New Relic Deamon

[[email protected] ~]# /etc/init.d/newrelic-daemon restart

Step 2: Install and Configure New Relic Drupal Module

Download and enable the New Relic module from here

Next navigate to Admin > Site Configuration > New Relic RPM Settings

Fill out your Application name, You may not know this until New Relic picks up your site. Login to New Relic and it will automatically assign an Application name.

Next fill out your api key, you can find this in your New Relic account.

From the looks of it at this point the New Relic drupal module is a bit unfinished. But once you install the yum package and click through your site, New Relic will automatically start tracking your site.

Mar 22 2011
Mar 22

One of the tasks that delayed my recent effort to migrate completely off using the Windows operating system was that I needed a new system for tracking my time and generating invoices. I used to use a custom Microsoft Access database that I created quite a few years ago, called Poplar Solo, which is available for free download on poplarware.com (and it works fine, if you have Microsoft Access available). But obviously, if I was not going to use Windows, I needed a different solution.

Being a devoted Drupal programmer and site builder, I of course thought first of Drupal as the platform for the new solution. My first step was to research the available Drupal modules for invoicing and time tracking, and I found a few, but none exactly fit my needs (which I think are fairly standard for a freelance Drupal contractor):

  • Track both overhead and billable hours, by client and job (including some overhead hours, such as time spent on marketing and education).
  • Track certain overhead expenses that I generally don't pay out of my business checking account (miles driven for business purposes, parking, bus fare, and food items) along with hours.
  • Invoice some clients for billable hours (possibly using different billing rates for different jobs), some clients for fixed dollar amounts, and some clients for both hours and amounts.
  • Choose when to invoice and which items to include on each invoice.
  • Generate reports on hours and overhead expenses.

So, given that no Drupal module I found did exactly what I wanted, I had to build it myself. I used Drupal 6.x, CCK, Flag, Views 6.x-2.x, Views Bulk Operations, and the existing Time Tracker module I found to build an invoicing and time tracking system, which I named "PS Time and Invoice" (PS is in homage to its predecessor, Poplar Solo). Because I thought it might be useful to others, I've made it available on Drupal.org as a "sandbox" project.

Besides solving my invoicing problem, building this system was also an exercise in Views programming. So, being a Drupal documentation fanatic (I'd have to be to have taken on the job as co-lead of the Documentation group!), I took the opportunity to add copious quantities of documentation to this module. You might want to check it out if you need a model for Views field handlers, Views join handlers, Views style plugins, defining CCK fields programmatically, or Flag programming.

One other reason someone might find this module useful is that it contains a Views style plugin for a table with column sums (you can configure which columns to sum). I was trying to use the existing Views Calc module to do this, but it doesn't currently work with table grouping (which I needed for some of my reports and invoices), and it also didn't work with some of the calculated fields I defined in my module. I found it easier to define my own table-with-column-sums style plugin than to fix the Views Calc table plugin, because the way the Views Calc plugin was set up, getting it to work with my fields and grouping was not going to be easy.

So, if you want to try out this module for time tracking and invoicing, use it as an example for your module, or grab the table-with-sums style Views style plugin, you can download PS Time and Invoice on drupal.org. (And by the way, if you find problems with this module or have suggestions for improvements, I'd appreciate it if you would use the issue queue there to report them, rather than commenting here.)

Jan 06 2011
Jan 06

Congratulations to the Drupal community for getting version 7.0 released! This is a major mile stone and an excellent reason to celebrate!

If you want to give Drupal 7 a try without having to install anything, I've now updated my Drupal 7 appliances on SuSE Studio to the latest release. The appliance is based on openSUSE Linux 11.3 and is available in two variants:

  • A text-mode only appliance to which you connect using your local web browser via the network.
  • A GUI version that starts up the Firefox browser in a minimalistic GNOME desktop to perform the installation locally. Remote network access is available, too.
The database backend is MySQL 5.1, with the InnoDB plugin and strict mode enabled. phpMyAdmin has been added to support web-based administration of the MySQL server. You can access it via http://localhost/phpMyAdmin/. I also added drush, the Drupal command line shell and scripting interface and some additional packages (yast2-http-server, bind-utils, php5-ctype, patch). I also bumped up the appliance's version number to 7.0.0, to match the version number of Drupal included.

The appliance is available in various formats:

  • A live raw disk image, ready to be written to an USB stick or flash drive
  • A live ISO image, to be burned on a CD or used in a virtual machine
  • A hard disk image, to be dumped on a hard disk drive
  • Various virtual disk formats, e.g. OVF, VMWare/VirtualBox/KVM and Xen
Please see the installation instructions provided on the download pages for details on how to use the various image formats.

So congratulations to the Drupal developer community for reaching this goal and thanks to SuSE/Novell/Attachmate for providing the infrastructure for creating such appliances. I also would like to especially thank Richard Bos for the testing and many suggestions for improvement of these appliances!

Nov 01 2010
Nov 01

FOSDEM banner

It's that time of the year again — the nice folks at FOSDEM have granted us a developer room at their upcoming conference (February 5+6 2011 in Brussels, Belgium)!

As usual there were more applications than they were able to accommodate, so we are very grateful for this opportunity for collaboration. Titled "MySQL and Friends", our room next year will be H.2213 with a capacity of 100 seats. It will be at our disposal on Saturday 5th, from 13:00 till 19:00. Like last year, we would like to set up a schedule of talks related to the MySQL server and the various projects that surround it. Each talk will last 20 minutes, plus 5 minutes of Q&A and a 5 minute break for switching speakers, giving us 12 slots in total to fill with excellent tech talks. Take a look at this year's schedule for some examples! I hope we can assemble an even more exciting and interesting schedule for next year.

Quoting from my last year's call for papers:

We are looking for covering a wide range of topics that attract both MySQL DBAs as well as application developers that work with MySQL as their database of choice. Are you developing a new storage engine or other plugin? Do you want to share your experiences and best practices in administering or deploying MySQL servers? Did you develop a new method to scale a MySQL setup? Let us and the audience know about it! You can submit your talk proposal via this submission form.

The deadline for turning in your proposal is Sunday, 26th of December, 2010, after which there will be a voting and rating period to identify the most interesting and attractive topics.

Please check the FOSDEM 2011 information page on the MySQL Forge Wiki for more details and don't hesitate to contact me directly, if you have any questions or suggestions. I look forward to your proposals!

Oct 31 2010
Oct 31

Eclipse for Drupal in Ubuntu

October 31, 2010

It took me a while to finally break down and install Eclipse and actually use it for Drupal development. In my mind it was one of those daunting tasks that I never thought I'd complete. Once you start using it for real, though, it feels like you finally got the Swiss Army knife you've always wanted, but now it's too big to fit in your pocket.

Very recently I got assigned to a new project for HealthMedia, Inc. (owned by corporate giant Johnson & Johnson). The specifics of the project are not up for discussion, but it turns out that it's a Drupal front-end linked to an in-house web services backend. The complexity of the system is such that a middleware had to be developed to take care of a lot pre-processing before Drupal can consume the serviced data in a usable fashion.

Using Eclipse, I'm able to quickly traverse the code and set breakpoints all over the place. There are clear visual cues coming up in Eclipse's several panels alerting you of wrong code, suspicious statements, search results, etc. Multiple views allow you browse the code tree, debug, compare versions of the code, and print out tasks and server output.

This all comes at a cost. The versatility and power of Eclipse, and the fact that it works equally well in any platform, means that you have to put up with a lot of overhead. Eclipse is written in Java; there is no native version of any other graphical toolkit. No GTK+, no QT. That, in itself, means that you have to load the entire Java Virtual Machine and it's graphical toolkit (along with all of its required libraries) just for that one program. The way Eclipse works, and one of the reasons why it's so good, means that everytime you change your code, it re-indexes related functions and variables. It is a great advantage, but it eats up a lot of CPU cycles. When you are running your enitre development environment locally, CPU cycles can be precious and your computer might be brought down to a crawl.

In any case, if you're reading this far, it probably means that you are either very interested, or have no choice. Let's move on to actually setting it up.

For this system I'm assuming Ubuntu 10.10.

Install the LAMP stack

The easies way is to do this through the terminal. Go ahead and click on Applications -> Accessories -> Terminal and paste in the following command:

sudo apt-get install apache2 php5 libapache-mod-php5 mysql-server libapache2-mod-auth-mysql php5-mysql phpmyadmin

This will install Apache, MySQL, PHP, and phpmyadmin (a web-based database administration tool). I could go into details on how to configure each one of these, but that would be a subject for another day. For now, visit the URL above to read up on that.

Install Xdebug

Same here. Most of the commands for installing anything will be using apt-get, since it's the easiest way. Run this command:

sudo apt-get install php5-xdebug

Now, to configure it, open the xdebug configuration file (nano /etc/php5/conf.d/xdebug.ini) and paste the following:

[debug]
; Remote settings
xdebug.remote_autostart=off
xdebug.remote_enable=on
xdebug.remote_handler=dbgp
xdebug.remote_mode=req
xdebug.remote_host=localhost
xdebug.remote_port=9000

; General
xdebug.auto_trace=off
xdebug.collect_includes=on
xdebug.collect_params=off
xdebug.collect_return=off
xdebug.default_enable=on
xdebug.extended_info=1
xdebug.manual_url=http://www.php.net
xdebug.show_local_vars=0
xdebug.show_mem_delta=0
xdebug.max_nesting_level=100
;xdebug.idekey=

; Trace options
xdebug.trace_format=0
xdebug.trace_output_dir=/tmp
xdebug.trace_options=0
xdebug.trace_output_name=crc32

; Profiling
xdebug.profiler_append=0
xdebug.profiler_enable=0
xdebug.profiler_enable_trigger=0
xdebug.profiler_output_dir=/tmp
xdebug.profiler_output_name=crc32

I originally got this code from http://www.apaddedcell.com/easy-php-debugging-ubuntu-using-xdebug-and-vim, which explains how to setup xdebug for vi. For all practical purposes, the configuration is the same.

Let us leave Xdebug for now and install other things. We'll go back to it later.

Install Drush

Drush will allow you to perform Drupal maintenance tasks from the command line. Pretty nifty utility. If you use Ubuntu, you're in luck! Run this command:

sudo apt-get install drush

If not, you will have to make sure you have CVS installed:

sudo apt-get install cvs

Then, you have to run the following set of commands to install Drush:

cd /usr/local/share
sudo cvs -z6 -d:pserver:anonymous:[email protected]:/cvs/drupal-contrib checkout -d drush contributions/modules/drush/
cd /usr/local/bin
sudo ln -s /usr/local/share/drush/drush .

That will install Drush on your /usr/local/share/drush folder and create a symbolic link on /usr/local/bin so that you can run Drush from wherever you are in the terminal.

Install jQuery libraries

This is not strictly necessary, but Drupal uses jQuery, so it could be nice to link the libraries into Eclipse, independent from any Drupal installation, so that they are available in your project.

sudo apt-get install libjs-jquery libjs-jquery-ui

Downgrade PHP to 5.2

Now, this part is tricky. PHP 5.3 changed a few things in the language construct that Drupal modules are not necessarily aware of. I won't go on about the details of these changes. Suffice it to say that not all Drupal modules out there are fully compatible with PHP 5.3. This is not strictly necessary either, but if you see that your web server starts complaining, you can use this to downgrade to PHP 5.2.

I found that this script works better when you run it after installing all your PHP stuff. If you ever need to install anything else that depends on PHP, you should re-run the script to make sure it's all part of the same version.

#!/bin/bash
# store list of installed PHP packages
php_installed=`dpkg -l | grep php| awk '{print $2}' |tr "\n" " "`
# install Aptitude
sudo apt-get install aptitude
# remove all PHP packages
sudo aptitude purge $php_installed
# use PHP packages from Karmic
# pin-params: a (archive), c (components), v (version), o (origin) and l (label).
echo -e "Package: php5\nPin: release a=karmic\nPin-Priority: 991\n"  | sudo tee /etc/apt/preferences.d/php > /dev/null
apt-cache search php5-|grep php5-|awk '{print "Package:", $1,"\nPin: release a=karmic\nPin-Priority: 991\n"}'|sudo tee -a /etc/apt/preferences.d/php > /dev/null
apt-cache search -n libapache2-mod-php5 |awk '{print "Package:", $1,"\nPin: release a=karmic\nPin-Priority: 991\n"}'| sudo tee -a /etc/apt/preferences.d/php > /dev/null
echo -e "Package: php-pear\nPin: release a=karmic\nPin-Priority: 991\n"  | sudo tee -a /etc/apt/preferences.d/php > /dev/null
# add Karmic to source list
egrep '(main restricted|universe|multiverse)' /etc/apt/sources.list | grep -v "#" | sed s/lucid/karmic/g | sed s/maverick/karmic/g | sudo tee /etc/apt/sources.list.d/karmic.list > /dev/null
# update package database
sudo apt-get update
# install original PHP packages
sudo apt-get install $php_installed
# make sure they don't get updated
sudo aptitude hold `dpkg -l | grep php5| awk '{print $2}' |tr "\n" " "`

Copy-paste that code into a text editor, then save as php-downgrade.sh and run the following command to make it executable:

chmod +x php-downgrade.sh

Then run the file with: ./php-downgrade.sh

Install Eclipse

I know, the whole purpose of this post is to install Eclipse, yet we're a few pages down and now is when I'm actually starting to do that.

Up until now, we were just satisfying some dependencies and setting up the entire development environment. This is the last actual program to install. The rest are plugins for Eclipse, as well as configuration.

Let's start!

You will need to install several components of Eclipse:

sudo apt-get install eclipse-platform eclipse-pde eclipse-plugin-cvs eclipse-platform This the base platform for Eclipse. Nothing more.
eclipse-pde This is the Plug-in Development Environment. It's not strictly necessary for Drupal, but you will need it to install other Eclipse plug-ins. Also, you might be interested in Apache ANT for building, deployment, or packaging. That is also here.
eclipse-plugin-cvs This is just one of the many plug-ins for Eclipse. This one just happens to be part of the Ubuntu repositories. So go ahead and install it. It provides CVS integration with Eclipse.

Apart from the CVS plug-in, there are several others that we will need. Eclipse is by default a Java IDE, but its extensibility has allowed the community to make it into pretty much anything with the proper plug-ins.

Here's the entire list:

To install them, open Eclipse and go to Help -> Install New Software and click on the Available Software Sites link to add new URLs. Make sure that the following URLs are there:

Then select All Available Sites from the drop-down list and select the previous plug-in list from the window. Follow the prompts to finally install the plug-ins.

Assuming the plug-in installation went well, all you have left is to configure Eclipse with the proper Drupal-friendly settings. Onwards!

Configure Eclipse

Open the Preferences windows and you will be able to browse and change the following settings. Match them as closely as you can.

  • General > Editors > Text Editors:
    • Displayed tab width: 2
    • Insert spaces for tabs: Yes
  • General > Web Browser: New...
    • Name: Firefox
    • Location: /usr/bin/firefox
    • Parameters: --new-window %URL%
  • PHP > Code Style > Formatter:
    • Tab policy: Spaces
    • Indentation size: 2
  • PHP > Debug
    • PHP Debugger: XDebug
  • Team > CVS: Files and Folders
    • Convert text files to use platform line ending: No
    • Default text mode: ASCII with keyword expansion (-kkv)
  • Web > CSS Files > Editor
    • Indent using spaces
    • Indentation size: 2
    • Capitalization style: Lowercase
  • Web > HTML Files > Editor
    • Indent using spaces
    • Indentation size: 2
  • Web > JavaScript > Code Style > Clean Up: Edit...
    • Use blocks in if/while/for/do statements: Always
  • Web > JavaScript > Code Style > Formatter: Edit...
    • Tab policy: Spaces only
    • Indentation size: 2
    • Tab size: 2
    • Statements within 'switch' body: Yes
  • Web > JavaScript > Include Path > User Libraries: New...
    • User library name: jQuery
    • Add .js file: /usr/share/javascript/jquery/jquery.js
    • Add .js file: /usr/share/javascript/jquery-ui/jquery-ui.js
Configure Debugging
  • PHP > PHP Servers
    • Name: Apache
    • URL: http://localhost
    • Path Mapping: Add...
    • Path on Server: /
    • Path in Workspace: /%PROJECT_NAME%
  • PHP > PHP Executables: Add...
    • Name: PHP
    • Executable path: /usr/bin/php
    • PHP ini file: /etc/php5/apache2/php.ini
    • SAPI Type: CLI
    • PHP debugger: XDebug

This configuration now should be done on the Run window, instead of the Preferences. Where it says %PROJECT_ROOT%, substitute for the path to your Drupal website (e.g. /var/www/example/index.php).

  • Run > Debug Configurations...
    • PHP Web Page: New
    • Name: Drupal
    • Server Debugger: XDebug
    • PHP Server: Apache
    • File: /%PROJECT_ROOT%/index.php
    • Common: Display in favorites menu: Debug

Yay! You're done with all the configuration! Now, to actually use your system...

Use your new development environment

Debugging
  1. Set your breakpoints
  2. Click on the bug tool: Drupal
  3. PHP Debug perspective should open
  4. Click on the Resume button (F8) every time you perform an action, so that the debugger runs through the entire page load.
Checking out CVS modules
  1. Right-click on /%PROJECT_ROOT%/sites/all/modules: Import... (substitute %PROJECT_ROOT% for your web root)
    • CVS > Project from CVS
    • Host: cvs.drupal.org
    • Repository path: /cvs/drupal-contrib
    • User: %USERNAME% (substitute %USERNAME% for your CVS account username)
    • Password: %PASSWORD% (substitute %PASSWORD% for your CVS account passowrd)
    • Connection type: pserver
  2. Browse to /contributions/modules/%MODULE_NAME% (substitute %MODULE_NAME% for your module name)
    • Check out into an existing project
    • Checkout subfolders: Yes
  3. Select /%PROJECT_ROOT%/sites/all/modules (substitute %PROJECT_ROOT% for your web root)
  4. Select Tag: Refresh Tags
    • Branches > DRUPAL-X--Y
Using Remote File System
  1. Open Remote System Explorer perspective
  2. Right-click > New Connection...
  3. Select Linux and enter the Host name
  4. From the following prompts, select ssh.files, processes.shell.linux, ssh.shells, ssh.terminals
  5. From the newly created connection, browse to your webroot or project folder
  6. Right-click > Create Remote Project
  7. Go back to the PHP perspective

Play around and have fun! Debugging can be a very satisfying experience, if done well and with purpose. If you managed to read all the way down to the end of this article: wow! I'm impressed! If not, it's okay, someone, somewhere, will find it useful.

Oct 25 2010
Oct 25

Drupal logoOver the weekend I updated my Drupal 7 test appliance in SUSE Studio to the Drupal 7.0-beta2 release, which was released on Oct. 23rd. I also added phpMyAdmin upon a user request, to provide a web-based method to work with the MySQL instance, if needed.

In addition to the lightweight "headless" appliance (which can only be accessed and configured via a remote network connection), I've now also created a GUI-based version. This appliance starts a minimal GNOME desktop and a Mozilla Firefox browser, which in turn opens the Drupal installation page by default. I hope you will find this useful if you want to toy around and test Drupal 7 without having to go through the entire OS and LAMP stack configuration yourself. In fact, you can even test this appliance via the recently added test drive option from right out of your web browser!

The appliance is now also available in OVF format. SuSE Studio now also builds Amazon EC2 images, which don't seem to be available for download from the SUSE Gallery yet. I assume this is a recent addition to the continuously improving SUSE Studio functionality, hopefully these images will be made available soon.

Oct 02 2010
Oct 02

Drupal development in Linux

October 2, 2010

I am a happy Linux user. I have been using different flavors of Linux for almost a decade now, and haven't seen Windows since the days of XP Service Pack 1. All my coworkers use Apple products (am I the only web developer that has never owned an Apple product, ever?)

I eventually settled on Ubuntu for desktop use, although my web server runs CentOS.

In any case, I swear by vi. And for most tasks, it's all I need for Drupal module development.

However, I'm always on the look for awesome development tools or different ways of doing development. I keep going back and forth between the *nix way (one tool per task) and the IDE (one tool for all tasks).

The one-tool-per-task approach

This one is easy: fire up a terminal and here are the tools of the trade...

  • Directory browsing: cd, ls
  • File manipulation: cp, mv, rm
  • Searching: find . -name "*" -print0 | xargs -0 grep -iIns "KEYWORD" (I have this whole command aliased to "search")
  • Source version control: cvs, svn, git
  • Text editor: vi

There's nothing else to this. I don't really do a lot of debugging, apart from dpm($var);. And the rest is, of course, a web browser.

Middle-tier

There is always the "halfway through" approach. Back in my KDE days, I used to love Quanta, but the project has been abandoned. Now that I use GNOME, Bluefish tends to be my editor of choice.

Even with these tools, I still revert to the terminal to execute remote commands and do source control tasks (apart from Giggle, which actually works very well).

Here I should note that debugging is still done through a combination of the Devel module and WebKit's Web Inspector in Chromium and Epiphany.

One tool to rule them all

So far, no one has sat down to write a good native IDE for web development that's as feature-complete as Eclipse. The emphasis is on the word native... and that means either QT for KDE, or GTK+ for GNOME.

Since such a tool doesn't exist, I have to suck it up and install über-bloated Java libraries to run Eclipse. Sigh.

There's nothing in Eclipse that I cannot do using a terminal and the proper commands. However, there are two features that really make it worthwhile to use Eclipse:

  • The Open Declaration functionality, where you Ctrl+click on a function to take you to its declaration; and
  • Integrated debugging, where you don't have to leave the comfort of your IDE to see what's going on during runtime.

Truth is, with Eclipse, you don't have to leave the comfort of your IDE for any task, apart from opening a web browser.

As I write, I'm installing all the Eclipse plugins I will need to accomplish this. In my next post I will walk through everything I did to set it up for doing Drupal development in pretty much any situation.

Sep 17 2010
Sep 17

Drupal logoThe Drupal community just recently released another alpha test release of their upcoming Drupal 7 version, to shake out the remaining bugs and to encourage more users to test it.

If you would like to give it a try, but you don't have a free server handy, how about using a virtual machine instead? Using the fabolous SuSE Studio, I've created an appliance based on openSUSE 11.3, Drupal 7.0-alpha7 and MySQL 5.1 with the InnoDB plugin and strict mode enabled (both for the SQL mode and InnoDB mode. Using this configuration helps to ensure that Drupal works well with the current version of MySQL/InnoDB and does not use any "questionable" SQL statements. This might be especially interesting for additional modules - Drupal core did not reveal any problems using strict mode so far.

You can download disk images for VMware/Virtualbox/KVM or XEN from the SUSE Gallery (free login required). Just boot the appliance in your virtualization application of choice, choose your keyboard layout and step through the network configuration and Time Zone selection. Once the appliance has booted up and the login: prompt appeared, point your web browser to the appliance's IP address to start the Drupal installation/configuration. MySQL has been pre-configured, there is an empty database named "drupal" and a user "drupal" with the same password to access it. You just need to enter this information in the Drupal Database configuration dialogue during the installation. Anything else can be configured to your liking.

After you have finished the installation, you can toy around with a fresh Drupal 7 installation! Install additional modules, change the themes, add content. And make sure to report all bugs that you run into while doing so! Have fun.

Feb 04 2010
Feb 04

Most often we work on our local database for Drupal projects and thus mysql host in settings.php would likely be 'localhost' as in the line: $db_url = 'mysql://db_user:[email protected]/db_name'; However, occasionally we may want to connect to a remote mysql server directly (eg. on a dev server). If you have full control over the remote mysql server, here's what you can do to achieve that:

1) Log onto the remote server and edit mysql server configuration file
Suppose the remote server is Fedora with IP address 192.168.1.150.
vim /etc/my.cnf

read more

Nov 05 2009
Nov 05

This blog post is a by-product of my preparation work for an upcoming talk titled "Why you should be using a distributed version control system (DVCS) for your project" at SAPO Codebits in Lisbon (December 3-5, 2009). Publishing these thoughts prior to the conference serves two purposes: getting some peer review on my findings and acting as a teaser for the actual talk. So please let me know — did I cover the relevant aspects or did I miss anything? What's your take on DVCS vs. the centralized approach? Why do you prefer one over the other? I'm looking forward to your comments!

Even though there are several distributed alternatives available for some years now (with Bazaar, git and Mercurial being the most prominent representatives here), many large and popular Open Source projects still use centralized systems like Subversion or even CVS to maintain their source code. While Subversion has eased some of the pains of CVS (e.g. better remote access, renaming/moving of files and directories, easy branching), the centralized approach by itself poses some disadvantages compared to distributed systems. So what are these? Let me give you a few examples of the limitations that a centralized system like Subversion has and how these affect the possible workflows and development practices.

I highly recommend you to also read Jon Arbash Meinel's Bazaar vs Subversion blog post for a more elaborate description of the limitations.

  • Most operations require interaction with the central repository, which usually is located on a remote server. Browsing the revision history of a file, creating a branch or a tag, comparing differences between two versions — all these activities involve communication via the network. Which means they are not available when you're offline and they could be slow, causing a slight disruption of your workflow. And if the central repository is down because of a network or hardware failure, every developer's work gets interrupted.
  • A developer can only checkpoint his work by committing his changes into the central repository, where it becomes immediately visible for everybody else working on that branch. It's not possible to keep track of your ongoing work by committing it locally first, in small steps, until the task is completed. This also means that any local work that is not supposed to be committed into the central repository can only be maintained as patches outside of version control, which makes it very cumbersome to maintain a larger number of modifications. This also affects external developers who want to join the project and work with the code. While they can easily obtain a checkout of the source tree, they are not able to put their own work into version control until they have been granted write access to the central repository. Until then, they have to maintain their work by submitting patches, which puts an additional burden on the project's maintainers, as they have to apply and merge these patches by hand.
  • Tags and branches of a project are created by copying entire directory structures around inside the repository. There are some recommendations and best practices on how to do that and how these directories should be arranged (e.g. by creating toplevel branches and tags directories), but there are several variants and it's not enforced by the system. This makes it difficult to work with projects that use a non-standard way for maintaining their branches and can be rather confusing (depending on the amount of branches and tags that exist).
  • While creating new branches is quick and atomic in Subversion, it's difficult to resolve conflicts when merging or reconciling changes from other branches. Recent versions of Subversion added support for keeping better track of merges, but this functionality is still not up to par with what the distributed tools provide. Merging between branches used to drop the revision history of the merged code, which made it difficult to keep track of the origins of individual changes. This often meant that developers avoided developing new functionality in separate branches and rather worked on the trunk instead. Working this way makes it much harder to keep the code in trunk a stable state.

Having described some downsides of the centralized approach, I'd now like to mention some of the most notable aspects and highlight a few advantages of using a distributed version control system for maintaining an Open Source project. These are based on my own personal experiences from working with various distributed systems (I've used Bazaar, BitKeeper, Darcs, git, Mercurial and SVK) and from following many other OSS projects that either made the switch from centralized to distributed or have been using a distributed system from the very beginning. For example, MySQL was already using BitKeeper for almost 2 years when I joined the team in 2002. From there, we made the switch to Bazaar in 2008. mylvmbackup, my small MySQL backup project, is also maintained using Bazaar and hosted on Launchpad.

Let me begin with some simple and (by now) well-known technical aspects and benefits of distributed systems before I elaborate on what social and organizational consequences these have.

In contrast to having a central repository on a single server, each working copy of a distributed system is a full-blown backup of the other repository, including the entire revision history. This provides additional security against data loss and it's very easy to promote another repository to become the new master branch. Developers simply point their local repositories to this new location to pull and push all future changes from there, so this usually causes very little disruption.

Disconnected operations allow performing all tasks locally without having to connect to a remote server. Reviewing the history, looking at diffs between arbitrary revisions, applying tags, committing or reverting changes can all be done on the local repository. These operations take place on the same host and don't require establishing a network connection, which also means they are very fast. Changes can later be propagated using push or pull operations - these can be initiated from both sides at any given time. As Ian Clatworthy described it, a distributed VCS decouples the act of snapshotting from the act of publishing.

Because there is no need to configure or set up a dedicated server or separate repository with any of today's popular DVCSes, there is very little overhead and maintenance required to get started. There is no excuse for not putting your work into revision control, even if your projects starts as a one-man show or you never intend to publish your code! Simply run "bzr|git|hg init" in an existing directory structure and you're ready to go!

As there is no technical reason to maintain a central repository, the definition of "the code trunk" changes from being defined by a technical requirements into a social/conventional one. Most projects still maintain one repository that is considered to be the master source tree. However, forking the code and creating branches of a project change from being an exception into being the norm. The challenge of the project team is to remain the canonical/relevant central hub of the development activities. The ease of forking also makes it much simpler to take over an abandoned project, while preserving the original history. As an example, take a look at the zfs-fuse project, which got both a new project lead and moved from Mercurial to git without losing the revision history or requiring any involvement by the original project maintainer.

Both branching and merging are "cheap" and encouraged operations. The role of a project maintainer changes from being a pure developer and committer to becoming the "merge-master". Selecting and merging changes from external branches into the main line of development becomes an important task of the project leads. Good merge-tracking support is a prerequisite for a distributed system and makes this a painless job. Also, the burden of merging can be shared among the maintainers and contributors. It does not matter on which side of a repository a merge is performed. Depending on the repository relationships and how changes are being propagated between them, some DVCSes like Bazaar or git actually provide several merge algorithms that one can choose from.

Having full commit rights into his one's own branch empowers contributors. It encourages experimenting and lowers the barrier for participation. It also creates new ways of collaboration. Small teams of developers can create ad-hoc workgroups to share their modifications by pushing/pulling from a shared private branch or amongst their personal branches. However, it still requires the appropriate privileges to be able to push into the main development branch.

This also helps to improve the stability of the code base. Larger features or other intrusive changes can be developed in parallel to the mainline, kept separate but in sync with the trunk until they have evolved and stabilized sufficiently. With centralized systems, code has to be committed into the trunk first before regression tests can be run. With DVCSes, merging of code can be done in stages, using a "gatekeeper" to review/test all incoming pushes in a staging area before merging it with the mainline code base. This gatekeeper could be a human or an automated build/test system that performs the code propagation into the trunk based on certain criterions, e.g. "it still compiles", "all tests pass", "the new code adheres to the coding standards". While central systems only allow star schemas, a distributed system allows workflows where modifications follow arbitrary directed graphs.

Patches and contributions suffer less from bit rot. A static patch file posted to a mailing list or attached to a bug report may no longer apply cleanly by the time you look into it. The underlying code base has changed and evolved. Instead of posting a patch, a contributor using a DVCS simply provides a pointer to his public branch of the project, which he hopefully keeps in sync with the main line of development. From there, the contribution can be pulled and incorporated at any time. The history of every modification can be tracked in much more detail, as the author's name appears in the revision history (which is not necessarily the case when another developer applies a patch contributed by someone else).

A DVCS allows you to keep track of local changes in the same repository, while still being able to merge bug/security fixes from upstream. Example: your web site might be based on the popular Drupal CMS. While the actual development of Drupal still takes place in (ghasp) CVS, it is possible to follow the development using Bazaar. This allows you to stay in sync with the ongoing development (e.g. receiving and applying security fixes for an otherwise stable branch) and keeping your local modifications under version control as well.

I've probably just scratched the surface on what benefits distributed version control systems provide with this blog post. Many of these aspects and their consequences are not fully analyzed and understood yet. In the meanwhile, more and more projects make the switch, gather experiences and establish best practices. If you're still using a centralized system, I strongly encourage you to start exploring the possibilities of distributed version control. And you don't actually have to "flip the switch" immediately — most of the existing systems happily interact with a central Subversion server as well, allowing you to benefit from some of the advantages without you having to convert your entire infrastructure immediately.

Here are some pointers for further reading on that particular subject:

Oct 29 2009
Oct 29

So you're a small startup company, ready to go live with your product, which you intend to distribute under an Open Source License. Congratulations, you made a wise decision! Your developers have been hacking away frantically, getting the code in good shape for the initial launch. Now it's time to look into what else needs to be built and setup, so you're ready to welcome the first members of your new community and to ensure they are coming back!

Keep the following saying in mind, which especially holds true in the Open Source world: "You never get a second chance to make a first impression!". While the most important thing is of course to have a compelling and useful product, this blog post is an attempt to highlight some other aspects about community building and providing the adequate infrastructure. This insight is based on my own experiences and my observations from talking with many people involved in OSS startups and projects.

First of all, realize that your community is diverse. They have different expectations, skills and needs. Pamper your early adopters. They are the multipliers that help you to spread the word, if they are convinced and excited about what you provide. Put some faith and trust in them and listen to their input. In the beginning, you might want to focus on your developer community and the tech-savvy early adopters, but this of course depends on the type of product you provide and on what your target audience looks like. In any case, make sure that you provide the necessary infrastructure to cater the respective needs of these different user bases.

Also remember that you can not overcommunicate with your community. Blog heavily, write documentation/FAQs/HOWTOs, build up Wiki content and structure, create screencasts. Don't rely on the community to create any of this in the early stages. But be prepared to embrace and support any activities, if they arise. Solicit input, provide opportunities and guidelines for participation!

While it's tempting to do: don't establish too many communication channels in the beginning. Keep it simple and reduce the different venues of communication to an absolute minimum at this point. A new forum with many different topics but no comments looks like an art gallery with a lot of rooms, but they are either empty or there's just a single picture hanging at the wall. Nobody wants to visit that, he'd feel lost in the void. At the early stage of a project, I think it's essential to keep the discussions in as few places as possible. This helps you to identify your key community contributors (the "regulars" aka the "alpha geeks") and to build up personal relationships with them (and among themselves).

Consider establishing a forum with only a few topics, start with one or two mailing lists. Also make sure that these are actively being followed (e.g. by yourself or your developers) and that questions are being answered! I personally prefer mailing lists over forums, but I'm probably not representative. Ideally, it would be nice if there would be a unified communication hub that supports both posting via the web site like a forum, or via email or NNTP (similar to Google Groups). This keeps the discussions on one central place (which eases searching for specific keywords/topics) and still allows users to choose their preferred means of communication. Unfortunately, I haven't really found any suitable platform for this approach yet — suggestions are welcome! And once your community grows and people start complaining about too many or off-topic discussions, you can think about further separation of the discussion topics.

Allow your users to submit and comment on issues and feature requests by providing a public bug/feature tracking system. Use this system for your release tracking and planning as well, to give your users a better insight into what they can expect from upcoming versions. Also, make it very clear to your users where bug reports and feature requests should be sent to! Should one use the Forums or the bug tracker for that? A mailing list or forum makes it easier for users to participate in these discussions, but makes it more difficult to keep track of them and to ensure they are being followed up on. For the sake of simplicity, I would actually suggest to remove any separate forums about these topics. Instead, educate your community early about which is the right tool and venue to use for such requests. This saves time and resources on your side and helps to build up an initial core of community members that can then educate others about "the ropes". Otherwise you end up with the burden of keeping track of every feature request or bug report that was posted somewhere, ensuring it has been added to the bug tracker...

If your community infrastructure consists of separate building blocks to provide the required functionality (e.g. forums, bug tracking, wiki), consider setting up a single-sign on (SSO) technology and establish a unified look and feel between these applications. Your users should not be required to log in with more than one username and password, and every application should share the same login and profile data. However, only require a login, if absolutely necessary! Many users feel alienated by having to enter their personal data, even if they only want to lurk around or browse through existing discussions or documentation. As an additional benefit, it helps you to quickly identify your "community stars" in the various sections of your site: Who reports the most bugs? Who is the most helpful person on our Forums? This information could also be published on your community site, giving users the opportunity to build up reputation and karma. Community infrastructure sites like Drupal or Joomla provide an excellent foundation to get you started, while offering enough room for improvement and additional functionality at a later point.

Lower the entrance barrier and make it as easy as possible for people to get started with your application. Don't just throw a source archive at them, hoping that someone else will take care of doing the binary builds. Put some effort into building and providing binary, ready-to-install packages for the most popular platforms that your target audience is likely to use. The three most important platforms to cover are Microsoft Windows, Mac OS X and Linux. While users of the latter usually have the required tools and experience in building stuff from source, Windows and Mac users are usually "spoiled" and don't want to be bothered with having to install a full-fledged development environment before they could eventually evaluate your application.

When it comes to Linux distributions, you should look into building distribution-specific packages. This heavily depends on the requirements for external libraries that your application is using, which might differ on the various flavours of Linux. Depending on the purpose of your application, you may either focus on the more desktop/developer-centric distributions like Mandriva, openSUSE, Ubuntu, or on the distributions commonly used in server environments, e.g. Debian, CentOS, Fedora, RHEL, SLES (Yes, I am aware that most distributions are multi-purpose and serve both tasks equally well, and it's of course possible to use each of them to get the job done — it's a matter of taste and preference). If possible, make use of existing build infrastructure like Fedora's Koji build system, Launchpad's Personal Package Archives (PPA) or the openSUSE Build Service (which even allows you to build RPMs and DEBs for non-SUSE distributions) to automate the building and provisioning of distribution-specific packages for a wide range of Linux flavours. If your application is slightly complicated to install or set up, consider providing a live demo server that people can access via the Internet to give it a try. Alternatively, create ready-to-run images for virtual machines like Parallels, VirtualBox or VMWare. Everything that makes it easier to access, install and test your software should be explored.

In closing, make community involvement a part of your company culture and make sure that you preserve enough time to take care of it. Community engagement has so many different aspects, you don't necessarily have to be a developer or a very technical person to get involved. I'm aware that doing community work can be seen as a distraction and definitely takes away time from other tasks. But community involvement should become a habit and a well-accepted part of everyone's job — this is much easier to establish while you're still small and growing.

Sep 14 2009
Sep 14

I had to move 264 databases from one Linux server (Fedora) to another (CentOS), hard copying /var/lib/mysql wasn't a good idea because version of two MySQL databases were different and i had to repair all the tables after move so i decided to write a very short and usefull bash script.

It dumps all the databases but each database into a different file :

for I in `echo "show databases;" | mysql -u myuser --password="" | grep -v Database`; do mysqldump -u myuser --password="" $I > "$I.sql"; done

As you can see you can put each .sql file into a different folder if you want, and it's possible to compress each file using piping :)

Jul 13 2009
Jul 13

Recently, we have a need for optimizing the performance of our drupal deployments, beyond the conventional mysql and apache optimization.

The first and easiest thing is to install a PHP Opcode Cache, and after some investigation I decided to try APC. Installing the version in PECL worked well on the staging environment.

pecl install apc
echo "extension = apc.so" > /etc/php5/apache2/conf.d/apc.ini
/etc/init.d/apache2 reload

On the live server it died with an out of memory error, although memory limits were set higher than the staging environment. To fix this I installed the version in apt and enabled a workaround in the suhosin patch configuration.

pecl uninstall apc
apt-get install php-apc
echo "suhosin.apc_bug_workaround = on" >> /etc/php5/apache2/conf.d/suhosin.ini
/etc/init.d/apache2 reload

Feb 18 2009
Feb 18

TTLUG
The Trinidad and Tobago Linux Users Group (TTLUG) is now using Drupal. Well seeing that I promised to have a website up and running after becoming President of the TTLUG, this was the least I could do on my own company's time.
It took me about 8 straight hours to set everything up from scratch. This 8 hours involved transferring nameservers, setting up hosting account, setting up Drupal, installing some modules, picking a theme, troubleshooting OpenID and creating some content to get things going....oh yeah somewhere in between I watched an episode of Scrubs, opps my bad.

Why was Drupal chosen?

Well I hate to keep repeating myself (!!) but after all, it is a community driven website and what other CMS out there does that sort of thing? It's great to know that it runs Drupal 6. It was built as a wiki type site so all of the content will come from the end users. Actually, the site should build and moderate itself (even spam), you'll see why after reading about some of the modules I used below.

The Modules

CCK and Views - Well hey hey lookie lookie, you didn't actually expect I wouldn't use these modules right?

Tagadelic - This would explain the tag cloud which would eventually show up in the footer region when there is enough content on the site.

Comment Notify - Used to notify users who have previously left comments on a node so that they will know that there is a new comment to their comment. This fosters back and forth discussions and all that good stuff.

Diff - Shows revisions between edits of a node so that a node can be reverted just in case an error was made.

Flag Content - Users can flag content just in case spam slips through.

Mollom - The major spam fighting ninja.

Service Links - provides those small image links that add some spice to the site...oh yeah and it can be used to spread the word about important nodes.

The site is a bit bare at the moment but I would expect to evolve over time. I'm looking forward to the having this site become an invaluable resource to IT professionals in Trinidad and Tobagp in the Linux/Open Source arena.

Sep 10 2008
Sep 10

MySQL UniversityTomorrow (Thursday, 11th of September) at 9:00 PST/16:00 UTC/17:00 GMT/18:00 CET, there will be an new free MySQL University Session. MySQL University started as an internal training program for MySQL engineers, to share and spread knowledge about their areas of expertise and has been available to the public for quite some time now. It covers a wide range of technical topics around the MySQL Server and usually takes place once per week.

For the first time, the presentation will not be performed by (former) MySQL employees/developers, but by two of our "Sun Classic" colleagues: Jyri Virkki (OpenSolaris Web Stack community lead) and Murthy Chintalapati (Sr Engineering Manager, Web Stack development) will talk about the OpenSolaris Web Stack:

OpenSolaris Web Stack is an OpenSolaris project and community building an integrated stack of popular open source web tier infrastructure technologies such as Apache HTTP server, MySQL, memcached, PHP and Ruby On Rails optimized for Solaris platform. This session introduces OpenSolaris Web Stack, its status and future development including addition of newer technologies such as lighttpd, Varnish etc., as well as the ease of use features for developers and deployers. We will also be discussing an experimental web stack IPS package repository and it could be leveraged to build and make available popular end user applications such as Drupal.

MySQL University sessions are free to attend - all you need is an IRC client (to post your questions and comments) and an audio player capable of playing back an OGG audio stream, so you can listen to what is being said. See the Instructions for Attendees on the MySQL University pages for more information on how to log in and attend. The audio stream will be recorded and published on the MySQL University pages for later consumption, in case you can't make it or want to listen to a previous session.

Sep 01 2008
Sep 01

I'm back home from DrupalCon 2008 now - it has been a great event! I met a lot of nice people from the Drupal Community and learned a lot about this CMS. I've been very busy in uploading the remaining pictures from the event to my gallery - so here's for your viewing pleasure:

I also gave two talks and held a BoF there - the slides have now been attached to the session nodes, one of them (the HA session) even includes a video recording:

I've also uploaded some pictures from FrOSCon to my Gallery now, hope you enjoy them! The slides of my FrOSCon talks are now uploaded to the conference system as well:

Aug 28 2008
Aug 28

Hello and greetings from DrupalCon 2008 in Szeged, Hungary!

We (Thierry Manfé, Scott Mattoon and myself) are having a great time manning our booth and talking about Drupal, MySQL and Open [email protected] with the nice crowd of Drupal Users and Developers here. Sun is a gold sponsor of the event and we're giving a number of sessions as well.

Today I gave my first presentation about MySQL Backup and Security - Best practices - unfortunately I ran a tad bit out of time at the end... The slides have already been attached to the session page, so you can read up on the last few things I was going to talk about. Feel free to contact me, if you have further questions!

Tomorrow I'll be talking about High availability solutions for MySQL: An Overview and practical demo, which will also include a practical demonstration of a two-Node Linux Cluster, performed by Jakub Suchy. In the afternoon, I will also hold a BoF about bzr - The Bazaar source revision control system

I've also uploaded some pictures from the event (and some impressions from the city) on my gallery (more will follow later). Enjoy!

Jul 31 2008
Jul 31

I've been using ActiveState's Komodo for a few weeks now in response to my frustrations with Eclipse. As with any IDE there was a short relearning of my ways that was necessary, but I am happy to report that I really like this IDE. The last bit of functionality I've been wanting to tie together is directly related to debugging Drupal in Komodo. Here's the steps that I took to get a Drupal development environment with debugging in Komodo on a clean installation of a 64bit Ubuntu 8.04.

A good deal of this may be redundant if you already have your environment established, but it is recorded here nonetheless to help others become proficient at what we do. While I prefer command line interfaces I am going to lean towards providing gui tools to allow newer devs options. If you don't need or want all of the minute details skip to step 6 and get straight into configuring for debugging.

Install the LAMP stack. LAMP is the abbreviation for Linux Apache MySql and PHP. We are going to go with the latest versions that are available in the Ubuntu repositories which means that, as of this writing, we will be running php 5.2. In addition to the traditional LAMP integration, Komodo also supports Perl, Python, Ruby, Tcl, XSLT and Javascript as well as xDebug, CVS, and SVN integration. Let's get all of this installed at once so we don't have to bother with it later.

There are two primary methods for installing the necessary packages. The first is by using the Synaptic Package Manager install the following packages (I've included descriptions for those not familiar with the packages):

apache2 - Apache v2 is the next generation of the omnipresent Apache web server. This version - a total rewrite - introduces many new improvements, such as threading, a new API, IPv6 support, request/response filtering, and more.
mysql-server-5.0 - MySQL is a fast, stable and true multi-user, multi-threaded SQL database server. SQL (Structured Query Language) is the most popular database query language in the world. The main goals of MySQL are speed, robustness and ease of use.
mysql-client - This is an empty package that depends on the current "best" version of mysql-client (currently mysql-client-5.0), as determined by the MySQL maintainers. Install this package if in doubt about which MySQL version you need. That will install the version recommended by the package maintainers.
mysql-gui-tools-common - Icons and GUI XML files used by MySQL Administrator, Query Browser and other tools in the MySQL GUI Tools suite. This package also includes documentation files.
phpmyadmin - phpMyAdmin is intended to handle the administration of MySQL over the WWW.
php5-common - This package contains the documentation and example files relevant to all the other packages built from the php5 source.
php5-cgi - This package provides the /usr/lib/cgi-bin/php5 CGI interpreter built for use in apache 2 with mod_actions, or any other CGI httpd that supports a similar mechanism.
php5-cli - This package provides the /usr/bin/php5 command interpreter, useful for testing PHP scripts from a shell, or perhaps even performing general shell scripting tasks, if you're frightened of perl and python.
php5-curl - CURL is a library for getting files from FTP, GOPHER, HTTP server.
php5-gd - This package provides a module for handling graphics directly from PHP scripts. It supports the PNG, JPEG, XPM formats as well as Freetype/ttf fonts.
php5-mcrypt - This package provides a module for MCrypt functions in PHP scripts.
php5-mysql - This package provides modules for MySQL database connections directly from PHP scripts. It includes the generic "mysql" module which can be used to connect to all versions of MySQL, an improved "mysqli" module for MySQL version 4.1 or later, and the pdo_mysql module for use with the PHP Data Object extension.
php5-tidy - This package provides a module for tidy functions in PHP scripts. Tidy is an extension based on Libtidy (http://tidy.sf.net/) and allows a PHP developer to clean, repair, and traverse HTML, XHTML, and XML documents -- including ones with embedded scripting languages such as PHP or ASP within them using OO constructs.
php5-xdebug - Provides extensive debugging utilities for PHP.
ruby - Ruby is the interpreted scripting language for quick and easy object-oriented programming. It has many features to process text files and to do system management tasks (as in perl). It is simple, straight-forward, and extensible.
cvs - CVS is a version control system, which allows you to keep old versions of files (usually source code), keep a log of who, when, and why changes occurred, etc., like RCS or SCCS. Unlike the simpler systems, CVS does not just operate on one file at a time or one directory at a time, but operates on hierarchical collections of directories consisting of version controlled files.
subversion - Subversion, also known as svn, is a version control system much like the Concurrent Versions System (CVS). Version control systems allow many individuals (who may be distributed geographically) to collaborate on a set of files (typically source code). Subversion has all the major features of CVS, plus certain new features that CVS users often wish they had.
scim-qtimm - SCIM (Smart Common Input Method) is an input method (IM) platform. This package brings functionality similar to the immodule for GTK+ to the Qt library. The main goal is to extend and enhance the input method support in the Qt library, in order to provide a modern and powerful multi-language input system.
scim-tables-additional - This package contains SCIM input method data tables for non-CJK languages. The currently supported languages are Amharic, Arabic, Nepali, Russian, Thai, Vietnamese, and several Indic languages (Bengali, Gujarati, Hindi, Kannada, Malayalam, Punjabi, Tamil and Telugu). It also provides support for X-SAMPA and LaTeX style input.
tcl8.3 - Tcl is a powerful, easy to use, embeddable, cross-platform interpreted scripting language. This package contains everything you need to run Tcl scripts and Tcl-enabled apps. This version does not support threads; if you need thread support, please use tcl8.4.

The second method can be initiated from a command line prompt with the following and selecting Yes where prompted:

sudo apt-get install apache2 mysql-server-5.0 mysql-client mysql-gui-tools-common phpmyadmin php5-common php5-cgi php5-cli php5-curl php5-gd php5-mcrypt php5-mysql php5-tidy php5-xdebug ruby cvs subversion scim-qtimm scim-tables-additional tcl8.3

During the install process you will be presebted with a screen asking you to pick the Web Server to configure for use with phpMyAdmin. Selecting apache2 by hitting the space bar and tabbing the ok button is the way to go. You will also be prompted to enter a mysql root password in a similar fashion.

At this point you can navigate to http://localhost/ to see a lovely It works! on the screen. At this point, I'd like to challenge windows to make this process easier ... ok enough sarcasm for now.

Configure Apache - This step is largely personal preference. I wrote a blog posting a few days ago entitled Dev Friendly Apache Configuration for Drupal. You can perform this step any way you want as long as it works. For the purposes of this tutorial I'm going to replace the existing conf file at /etc/apache2/sites-enabled/000-default with the conf file in that tutorial. I'm going to leave the file as is which means that our drupal installation from this tutorial will be available at http://mysite/

Next we will want to adjust the permissions on the /var/www. You will need to know your username for this step. Simply type whoami at a command prompt then use that response in this command:
sudo -R chown username.username /var/www
Replace username in the command with your username.

Reading the virtualhost entry shows that we have our host pointing to a specific directory. You can create that directory by using the following:
mkdir -p /var/www/mysite/trunk/htdocs

Next we need to edit the host file to include a mysite entry as outlined in the Apache configuration tutorial.

From there we need to enable the mod_rewrite module for Apache and reload the configuration by issuing the following commands:
a2enmod rewrite
sudo /etc/init.d/apache2 reload

*whew* You can now access http://mysite

Install Komodo - Activestate's Komodo is an awesome tool. The trial version will last for 3 weeks and should be sufficient for you to determine if you agree with me or not.

When you download the trial version of the software for Linux you will be presented with two choices depending on the libstdc libraries installed. The version of Ubuntu I'm using has the libcpp6 version so I downloaded the package labelled AS package (libcpp6) . In order to find out which version you will need run the following command:
find /usr/lib -name libstdc*

If you find entries for "/usr/lib/libstdc++.so.6", use libcpp6. If you find entries for "/usr/lib/libstdc++.so.5", use libcpp5. If you have both entries you can choose either one, but the suggestion is the more recent libcpp6.

Activestate provides comprehensive installation instructions ... so we'll skip over the details and rely on you to follow them. If you've been following this tutorial from the beginning you will already have everything necessary for the installation.

Check Your Configuration - Start the Komodo IDE and read the Start Page you are presented with. You should see a link titled Check Configuration. Click the Check Configuration link and you should see that each of the supported feature sets are ready for use with the exception of Perforce. If you want Perforce working then feel free to write your tutorial. If something is not working, use the help links in the window to find additional information and double check that you've followed everything in this tutorial so far.

One of the first things I like to do with Komodo is enable the sidebar by going to View >> Tabs >> Projects

Install Drupal - here's where I get lazy ... Drupal has an excellent documentation team that has been working hard to provide you what this tutorial would normally need to cover. http://drupal.org/getting-started/6/install is an excellent resource for a Drupal 6 installation while http://drupal.org/getting-started/5/install is good for a Drupal 5 installation. I installed my version of Drupal in /var/www/mysite/trunk/htdocs so that when I browse to http://mysite/ I see a Drupal website.
Configure Debugging - This is the bit that took me a little while to figure out. Each set of instructions I had found seemed to walk you from the very beginning all the way up to the debugging bit. However, because of the packages we just installed and the nature of Ubuntu we can skip a lot of what they describe because it is already in place and configured. The "It just works" mentality can be applied here. The xdebug configuration file is located at /etc/php5/apache2/conf.d/xdebug.ini and we need to add the following lines to the end of it:

xdebug.remote_enable=1
xdebug.remote_handler=dbgp
xdebug.remote_mode=req
xdebug.remote_host=127.0.0.1
xdebug.remote_port=9000

Restart Apache to enable your changes:

sudo /etc/init.d/apache2 restart

By default Komodo will listen for debugging requests on the lowest available port so we need to tell it to stay listening to the port we designated in xdebug.ini. In Komodo select the menu item Edit >> Preferences to open the preferences window. Expand the meu tree on the left so that you are looking at the options for Debugger >> Connection and select the radio box to specify a specific port. The number 9000 should already be entered, but if it is not go ahead and enter it. Click OK to exit this window saving your changes. You now have working debugging set up. For those of you that lack patience you can simply append the XDEBUG_SESSION_START variable to a local url and continue on your own ... an example url would be:

http://example.org/sample.php?XDEBUG_SESSION_START=1

Configure URL Mappings - The example dev site we set up in this tutorial is visible at http://mysiste/ ... we now need to tell Komodo to map the local files we work with to that url so that as we debug we can edit the files. If we do not perform this step Komodo will still display the files, but they will be read only versions. Open the menu item Edit >> Preferences again and this time select the Mapped URIs option. Click the Add Button which is a green circle with a plus sign. Following our tutorial, you will enter the URI as http://mysite/ and the Maps To can be selected by entering /var/www/mysite/trunk/htdocs/. Clicking OK twice exits the preferences window saving your changes. Using Mapped URIs you can have as many virtual hosts to develop against as your machine will hold simply by following the instructions in this tutorial.
Cool Tricks - This part isn't necessary, but if your using Firefox the Xdebug Helper Addon is an excellent resource for those that, like me, appreciate not having to remember everything. You simply click it to append or remove the XDEBUG_SESSION_START=1 variable from your requests.

This blog post has reached my limit for overly lengthy posts, but I think it covers good information and was worth the effort. The next post I write will be about how to actually use the debugger to resolve a specific problem in Drupal that we will create. Oh yeah and if anybody from ActiveState actually reads this I sure wouldn't mind if there was some way to slip me a license for all the blogging < grin >

Jul 26 2008
Jul 26

So this week I had to get search_files module for Drupal 6 running on a shared host, Bluehost, for one of my customers. I had promised that we would be able to search his PDF files, but I didn't realize that search_files, as well as search_attachments modules require a Linux command line utility named pdftotext to be installed.

I did request that Bluehost install it on my box and was told that anything requiring root access wasn't going to happen.

Fine, maybe I can run it myself, after all, I did get SVN running on Bluehost, how hard could it be?

Thanks to rasc on the Sphider PHP Search Engine forums for the solution, which I catered to work for search_files.

Setting up Bluehost

  • First, go to foolabs and grab yourself a hot copy of XPDF
  • Un-archive it, and get get rid of everything other than the pdftotext file
  • Rename this file to pdftotext.script
  • Create a shell script (flavor of your choosing) that calls this pdftotext.script file, for example:
#!/bin/sh
/home/YOURBLUEHOSTUSERNAME/bin/pdftotext/pdftotext.script $1 -
  • As you can tell from the code there, you will need to change YOURBLUEHOSTUSERNAME to the correct name.
  • Log in via FTP or SSH, and create a /bin and a /bin/pdftotext directory from your home directory, NOT FROM public_html, but one directory above that
  • Make /bin/pdftotext writable by all, this is where pdftotext will save the temporary files it creates
  • Upload both pdftotext and pdftotext.script to the /bin/pdftotext directory, and make them executable (chmod 755 should work)
  • If you don't have one already, in your home directory (not public_html) create a .bashrc file, and add the following so that the web server knows where your executable files are:
export PATH=$PATH:$HOME/bin:$HOME/bin/pdftotext:.
export pdftotext_path=/home/YOURBLUEHOSTUSERNAME/bin/pdftotext/pdftotext

Setting up search_files in Drupal 6

  • Go to the search_files project page and download the module
  • Upload the module
  • Go to Admin - Site Building - Modules and activate the module
  • You may need to adjust your permissions to let you use the module, do that if needed
  • Go to /admin/settings/search_files/helpers page and click "PDF"
  • In the Helper Path* box, put in:
/home/YOURBLUEHOSTUSERNAME/bin/pdftotext/pdftotext %file% -
  • Click Update. Please notice the - at the end, it's needed. I have it in both the helper line and the script, it's probably not needed in both, but it DOES work this way
  • Now, find the Directories page (/admin/settings/search_files/directories) and start adding directories where you have those PDF files that you would like to have indexed, such as /home/YOURBLUEHOSTUSERNAME/public_html/files - making sure that you use the full server path

Run it!

I took advantage of this time as an opportunity to setup my cron job on Bluehost, and then cleared my cache in Drupal, ran cron, and watched it find hundreds of files for me.

You MAY need to create a new custom php.ini file for your Drupal installation (Bluehost has a utility to create a default one in c-panel) and increase the limits so that you don't run into memory allocation or timeout issues.

Hope this helps, if you have any questions, go ahead and leave a comment!

Jul 17 2008
Jul 17

I like to develop against local virtual hosts when I work with Drupal. Here is the apache httpd.conf configuration that has served me the best so far. The first VirtualHost is the default .. .simply replicate the second VirtualHost entry and edit accordingly. A simple edit of /etc/hosts to enter the virtual host name that matches the virtual host entries in the httpd.conf and a quick reload by apache and you are good to go. I particularly enjoy the independent logging for each site.

This is likely a no-brainer for the server admins out there but I remember a time when this type of information would have been useful to me ... so here it is. This is not a production level configuration. Have suggestions for improving it?

httpd.conf

NameVirtualHost *
ServerName 127.0.0.1
<VirtualHost *>
  ServerAdmin [email protected]
  DocumentRoot /var/www/
  <Directory />
    Options FollowSymLinks
    AllowOverride all
  </Directory>
  <Directory /var/www/>
    Options All Indexes FollowSymLinks MultiViews
    AllowOverride all
    Order allow,deny
    allow from all
    RewriteEngine on
    RewriteBase /
    RewriteCond %{REQUEST_FILENAME} !-f
    RewriteCond %{REQUEST_FILENAME} !-d
    RewriteRule ^(.*)$ index.php?q=$1 [L,QSA]
  </Directory>
  ErrorLog /var/log/apache2/error.log
  LogLevel warn
  CustomLog /var/log/apache2/access.log combined
  ServerSignature Off
</VirtualHost>

<VirtualHost *>
  ServerName mysite
  DocumentRoot /var/www/mysite/trunk/htdocs
  ErrorLog /var/log/apache2/mysite_error.log
  CustomLog /var/log/apache2/mysite_access.log combined
</VirtualHost>

/etc/hosts

127.0.0.1     localhost.localdomain localhost mysite mysite1 mysite2

May 30 2008
May 30

The SUSE Linux guys just launched a new service called SUSE Studio. With it, you can create an operating system running SUSE, and your own checklist of packages. If you watch the screencast, you see that he creates a Linux distribution that has PHP and MySQL running out of the box.

This seems like the perfect solution for creating a Drupal Operating System that's ready for both Drupal development and runtime right when you install it on your machine. All we'd have to do is have the Drupal packages ready!

May 15 2008
May 15

Last night we migrated our Linode for Gamers With Jobs to a new Xen VPS and we've noticed a significant performance boost. We did, however, start encountering a random issue with segmentation faults in Apache. If you haven't seen this happen before, it tends to begin innocently with one Apache process dying, and therefore giving errors (usually WSOD), but quickly balloons into dozens of dead processes. It essentially hoses Apache.

Apparently the issue is due to eAccelerator, so I reinstalled it and cleared its caches in the hope that it might limit its occurrence. Just in case, though, 2bits has a great fix for it, using the logwatcher script by Firebright, Inc. I was able to quickly get it going, and the only difference is that I used the Debian init.d script provided by Derek Laventure to run it.

Apr 28 2008
Apr 28

This article describes how to install the Drupal 6.2 CMS on MySQL 6.0, using the Falcon Storage Engine. The operating system is a default Ubuntu 8.04 "Hardy Heron" (x86) installation.

I will make a few assumptions here, in order to keep the instructions simple: a fresh OS install, no other MySQL databases or web services are running or have already been installed. Both MySQL and the web server are installed on the same host. You should be able to become root to install packages and to have access to the local file system and the system configuration.

This article will explain how to install and configure Apache/PHP, MySQL 6.0 and Drupal 6.2.

Prerequisites

Running Drupal requires a web server (e.g. Apache) and PHP. We will use the packages as shipped with the distribution and will then install a MySQL 6.0 preview binary from http://dev.mysql.com. Other web servers like lighttpd will work equally well, but this article focuses on using the Apache web server.

Fortunately the MySQL 5.0 client applications as shipped with Ubuntu Linux are compatible with the MySQL 6.0.x client/server protocol, so we only make use of the 6.0 server and will use the installed, pre-compiled client applications and libraries to connect to it - there is no need to recompile PHP or anything to get going!

First of all you have to make sure the following packages have been installed (e.g. by using a package management tool like the Adept Package Manager, synaptic, aptitude or apt-get):

  • apache2
  • libapache2-mod-php5
  • php5
  • php5-common
  • php5-mysql
  • php5-gd
  • mysql-client-5.0

To enable the mod_rewrite Apache module (as recommended for Drupal), you need to enter the directory /etc/apache2/mods-enabled and create a symlink to the module loading instructions:

cd /etc/apache2/mods-enabled/
sudo ln -s ../mods-available/rewrite.load .

This will ensure, that mod_rewrite will be loaded when Apache starts up.

Additionally, you have to edit the file /etc/apache2/sites-available/default and make one change. In the directives for the Directory /var/www, change AllowOverride from "None" to "All". This will make sure that Drupal can enable the rewrite engine to allow nicer looking URLs.

Now restart the Apache server to apply the changes:

sudo /etc/init.d/apache2 restart

To verify that Apache is up and running, try opening http://localhost/ in a browser on the same machine that runs the web server. You should get a simple page, stating that "It works!".

Installing the MySQL 6.0 Falcon preview

Now that the web server is up and running, we need to install a MySQL database server that the Drupal installation can use. Download mysql-6.0.5-alpha-pb87-linux-x86.tar.gz (or any newer package, if available) from http://downloads.mysql.com/forge/falcon_feature_preview/

Create a /etc/mysql/my.cnf file with the following content (replacing the existing file, if necessary):

[client]
socket=/var/run/mysqld/mysqld.sock

[mysqld]
basedir=/usr/local/mysql
datadir=/usr/local/mysql/data
socket=/var/run/mysqld/mysqld.sock
default-storage-engine=falcon

The default-storage-engine option will make sure that every CREATE TABLE statement will default to using the Falcon storage engine. Now extract the binary tarball distribution into /usr/local and perform the following steps to finalize the installation/configuration:

$ sudo groupadd mysql
$ sudo useradd -g mysql mysql
$ cd /usr/local
$ sudo tar zxvf ~/mysql-6.0.5-alpha-pb87-linux-x86.tar.gz -C /usr/local
$ cd /usr/local
$ sudo ln -s mysql-6.0.5-alpha-pb87-linux-x86 mysql
$ cd mysql
$ sudo chown -R mysql .
$ chgrp -R mysql .
$ scripts/mysql_install_db --user=mysql
$ sudo chown -R root .
$ sudo chown -R mysql data
$ sudo ./bin/mysqld_safe --user=mysql &

The installation procedure is outlined in more detail in the reference manual at http://dev.mysql.com/doc/refman/6.0/en/installing-binary.html

If you want to enable the automatic startup of MySQL at system bootup time, you need to install an init script in /etc/init.d/ - follow the instructions as outlined in the reference manual. Note that the mysql.server script has been moved from the directory support-files to share/mysql for the binary tarball distributions and that the current 6.0 documentation has not been updated yet (I filed BUG#36382 about this).

Now start the server using the mysqld_safe script:

$ sudo /usr/local/mysql/bin/mysqld_safe &

Next you should verify that you can connect to the server and that the Falcon storage engine is enabled:

$ mysqladmin version
mysqladmin Ver 8.41 Distrib 5.0.51a, for debian-linux-gnu on i486
Copyright (C) 2000-2006 MySQL AB
This software comes with ABSOLUTELY NO WARRANTY. This is free software,
and you are welcome to modify and redistribute it under the GPL license

Server version 6.0.5-alpha-pb87
Protocol version 10
Connection Localhost via UNIX socket
UNIX socket /var/run/mysqld/mysqld.sock
Uptime: 43 min 6 sec

Threads: 1 Questions: 6 Slow queries: 0 Opens: 15 Flush tables: 1 Open tables: 8 Queries per second avg: 0.2

$ mysql -u root
[email protected]:(none) > SELECT * FROM information_schema.engines WHERE engine='Falcon';
+--------+---------+-----------------------+--------------+----+------------+
| ENGINE | SUPPORT | COMMENT | TRANSACTIONS | XA | SAVEPOINTS |
+--------+---------+-----------------------+--------------+----+------------+
| Falcon | DEFAULT | Falcon storage engine | YES | NO | YES |
+--------+---------+-----------------------+--------------+----+------------+
1 row in set (0.12 sec)

Before performing the actual Drupal installation, you need to create a Drupal Database and a user account. I chose "drupal" as the user name and password, please use some more sensitive values for your own setup!

[email protected]:(none) > CREATE DATABASE drupal;
Query OK, 1 row affected (0.01 sec)

[email protected]:(none) > GRANT ALL ON drupal.* to 'drupal'@'localhost' IDENTIFIED BY 'drupal';
Query OK, 0 row affected (0.00 sec)

Now MySQL 6.0 is installed and ready!

Installing Drupal 6.2

Now that the web and database server have been set up and configured, it's time to perform the installation of our application, the Drupal content management system. Start by downloading drupal-6.2.tar.gz from http://drupal.org/ (by clicking on the Drupal 6.2 download link on the front page).

Remove the default start page /var/www/index.html and extract the content of the drupal tarball into this directory. Then change the ownerships of these files to the user that apache runs under (www-data by default):

$ sudo rm /var/www/index.html
$ sudo tar --strip-components=1 -zxvf drupal-6.2.tar.gz -C /var/www
$ sudo chown -R www-data /var/www

The Drupal package installation itself is now done, the remaining installation and configuration steps are performed in a browser by opening http://localhost/ in your browser (reload the page if you still see the "It works" default page).

DrupalInstall_1

Follow the instructions in the Drupal installation manual on how to perform the actual installation. In the "Database Configuration" dialogue, use the MySQL database username and password that you created earlier.

DrupalInstall_2

Now the Drupal installer should perform its duty and you should see you fresh Drupal installation up and running!

DrupalInstall_3

Once the installation has finished, let's verify that we're really running on Falcon by running the following query in a MySQL command line client:

mysql> SELECT TABLE_NAME, ENGINE from information_schema.tables WHERE TABLE_SCHEMA='drupal';
+-------------------------+--------+
| TABLE_NAME | ENGINE |
+-------------------------+--------+
| access | Falcon |
| actions | Falcon |
| actions_aid | Falcon |
| authmap | Falcon |
| batch | Falcon |
| blocks | Falcon |
| blocks_roles | Falcon |
| boxes | Falcon |
| cache | Falcon |
| cache_block | Falcon |
| cache_filter | Falcon |
| cache_form | Falcon |
| cache_menu | Falcon |
| cache_page | Falcon |
| cache_update | Falcon |
| comments | Falcon |
| files | Falcon |
| filter_formats | Falcon |
| filters | Falcon |
| flood | Falcon |
| history | Falcon |
| menu_custom | Falcon |
| menu_links | Falcon |
| menu_router | Falcon |
| node | Falcon |
| node_access | Falcon |
| node_comment_statistics | Falcon |
| node_counter | Falcon |
| node_revisions | Falcon |
| node_type | Falcon |
| permission | Falcon |
| role | Falcon |
| sessions | Falcon |
| system | Falcon |
| term_data | Falcon |
| term_hierarchy | Falcon |
| term_node | Falcon |
| term_relation | Falcon |
| term_synonym | Falcon |
| url_alias | Falcon |
| users | Falcon |
| users_roles | Falcon |
| variable | Falcon |
| vocabulary | Falcon |
| vocabulary_node_types | Falcon |
| watchdog | Falcon |
+-------------------------+--------+
46 rows in set (0.00 sec)

Looks like we were successful - all Drupal tables are using the Falcon storage engine! Congratulations.

From here on, you can configure and change Drupal to your heart's content. Note however, that additional Drupal modules may contain code that is specific to the MyISAM or InnoDB storage engine, your mileage may vary. In that case it would be great to notify the module developers about these incompatibilities.

If you want to quickly populate a Drupal installation with content for testing, you can use the "Devel" module. I used it to create 500 users and 10.000 test pages on my demo installation. Even though this was performed within a virtual machine running VirtualBox, the system still was very responsive and the creation of the content proceeded amazingly fast! But I did not perform any serious benchmark or load tests (it would not make much sense in a VM anyway).

Dec 18 2007
Dec 18
#!/bin/bash

# guardian - a script to watch over application system dependences, restarting things
#            as necessary:  http://www.johnandcailin.com/john
#
#            this script assumes that at, logger, sed and wget are available on the path.
#            it assumes that it has permissions to kill and restart deamons including
#            mysql and apache.
#           
#            Version: 1.0:    Created
#                     1.1:    Updated logfileCheck() not to assume that files are rotated
#                             on restart.

checkInterval=10                         # MINUTES to wait between checks

# some general settings
batchMode=false                          # was this invoked by a batch job
terminateGuardian=false                  # should the guardian be terminated

# setting for logging (syslog)
loggerArgs=""                            # what extra arguments to the logger to use
loggerTag="guardian"                     # the tag for our log statements

# the at queue to use. use "g" for guardian. this queue must not be used by another
# application for this user.
atQueue="g"

# the name of the file containing the checks to run
checkFile="./checks"

# function to print a usage message and bail
usageAndBail()
{
   cat << EOT
Usage:guardian [OPTION]...
Run a guardian to watch over processes. Currently this supports apache and mysql. Other
processes can be added by simple modifications to the script. Invoking the guardian will run
an instance of this script every n minutes until the guardian is shutdown with the -t option.
Attempting to re-invoke a running guardian has no effect.

All activity (debug, warning, critical) is logged to the local0 facility on syslog.

The checks are listed in a checkfile, for example:

   #check type, daemonName, executableName, checkSource, checkParameters
   logfileCheck, apache2,       apache2,        /var/log/apache2/mainlog, "segmentation fault"

This checkfile specifies a periodic check of apache's mainlog for a string containing
"segmentation fault", restarting the apache2 process if it fails.

This script should be run on each host running the service(s) to be watched.

  -i        set the check interval to MINUTES
  -c        use the specified check file
  -b        batch mode. don't write to stderr ever
  -t        terminate the guardian
  -h        print this help

Examples:
To run a guardian every 10 minutes using checks in "./myCheckFile"
$ guardian -c ./myCheckFile -i 10

EOT

   exit 1;
}

# parse the command line arguments (l,s and t, each of which take a param)
while getopts i:c:hbt o
do     case "$o" in
        i)     checkInterval="$OPTARG";;
        c)     checkFile="$OPTARG";;
        h)     usageAndBail;;
        t)     terminateGuardian=true;;
        b)     batchMode=true;;        # never manually pass in this argument
        [?])   usageAndBail
       esac
done

# only output logging to standard error running from the command line
if test ${batchMode} = "false"
then
   loggerArgs="-s"
fi

# setup logging subsystem. using syslog via logger
logCritical="logger -t ${loggerTag} ${loggerArgs} -p local0.crit"
logWarning="logger -t ${loggerTag} ${loggerArgs} -p local0.warning"
logDebug="logger -t ${loggerTag} ${loggerArgs} -p local0.debug"

# delete all outstanding at jobs
deleteAllAtJobs ()
{
   for job in `atq -q ${atQueue} | cut -f1`
   do
      atrm ${job}
   done
}

# are we to terminate the guardian?
if test ${terminateGuardian} = "true"
then
   deleteAllAtJobs

   ${logDebug} "TERMINATING on user request"
   exit 0
fi

# check to see if a guardian job is already scheduled, return 0 if they are, 1 if not.
isGuardianAlreadyRunning ()
{
   # if there are one or more jobs running in our 'at' queue, then we are running
   numJobs=`atq -q ${atQueue} | wc -l`
   if test ${numJobs} -ge 1
   then
      return 0
   else
      return 1
   fi
}

# make sure that there isn't already an instance of the guardian running
# only do this for user initiated invocations.
if test ${batchMode} = "false"
then
   if isGuardianAlreadyRunning
   then
      ${logDebug} "guardian invoked but already running. doing nothing."
      exit 0
   fi
fi

# get the nth comma seperated token from the line, trimming whitespace
# usage getToken line tokenNum
getToken ()
{
   line=$1
   tokenNum=$2

   # get the nth comma seperated token from the line, removing whitespace
   token=`echo ${line} | cut -f${tokenNum} -d, | sed 's/^[ \t]*//;s/[ \t]*$//'`
}

# check http. get a page and look for a string in the result.
# usage: httpCheckImplementation sourceUrl checkString
httpCheck ()
{
   sourceUrl=$1
   checkString=$2

   wget -O - --quiet ${sourceUrl} | egrep -i "${checkString}" > /dev/null 2>&1
   httpCheckResult=$?
   if test ${httpCheckResult} -eq 0
   then
      ${logDebug} "PASS: found \"${checkString}\" in ${sourceUrl}"
   else
      ${logWarning} "FAIL: could NOT LOCATE \"${checkString}\" in ${sourceUrl}"
   fi

   return ${httpCheckResult}
}

# check to make sure that mysql is running
# usage: mysqlCheck connectString query
mysqlCheck ()
{
   connectString=$1
   query=$2

   # get the connect params from the connectString
   userAndPassword=`echo ${connectString} | sed "s/.*\/\/\(.*\)@.*/\1/"`
   mysqlUser=`echo ${userAndPassword} | cut -f1 -d:`
   mysqlPassword=`echo ${userAndPassword} | cut -f2 -d:`
   mysqlHost=`echo ${connectString} | sed "s/.*@\(.*\)\/.*/\1/"`
   mySqlDatabase=`echo ${connectString} | sed "s/.*@\(.*\)/\1/" | cut -f2 -d\/`

   mysql -e "${query}" --user=${mysqlUser} --host=${mysqlHost} --password=${mysqlPassword} --database=${mySqlDatabase} > /dev/null 2>&1
   mysqlCheckResult=$?
   if test ${mysqlCheckResult} -eq 0
   then
      ${logDebug} "PASS: executed \"${query}\" in ${mysqlHost}"
   else
      ${logWarning} "FAIL: could NOT EXECUTE \"${query}\" in database ${mySqlDatabase} on ${mysqlHost}"
   fi

   return ${mysqlCheckResult}
}

# check to make sure that a logfile is clean of critical errors
# usage: logfileCheck errorString logFile
logfileCheck ()
{
   logFile=$1
   errorString=$2
   logfileCheckResult=0
   marker="__guardian marker__"
   mark="${marker}: `date`"

   # make sure that the logfile exists
   test -r ${logFile} || { ${logCritical} "logfile (${logFile}) is not readable. CRITICAL GUARDIAN ERROR."; exit 1; }

   # see if we have a marker in the log file
   grep "${marker}" ${logFile} > /dev/null 2>&1
   if test $? -eq 1
   then
      # there is no marker, therefore we haven't seen this logfile before. add the
      # marker and consider this check passed
      echo ${mark} >> ${logFile}
      ${logDebug} "PASS: new logfile"
      return 0
   fi

   # pull out the "active" section of the logfile, i.e. the section between the
   # last run of the guardian and now i.e. betweeen the marker and the end of the file

   # get the last marker line number
   lastMarkerLineNumber=`grep -n "__guard" ${logFile} | cut -f1 -d: | tail -1`

   # grab the active section
   activeSection=`cat ${logFile} | sed -n "${lastMarkerLineNumber},$ p"`

   # check for the regexs the logFile's active section
   echo ${activeSection} | egrep -i "${errorString}" > /dev/null 2>&1
   if test $? -eq 1
   then
      ${logDebug} "PASS: logfile (${logFile}) clean: line ${lastMarkerLineNumber} to EOF"
   else
      ${logWarning} "FAIL: logfile (${logFile}) CONTAINS CRITICAL ERRORS"
      logfileCheckResult=1
   fi

   # mark the newly checked section of the file
   echo ${mark} >> ${logFile}

   return ${logfileCheckResult}
}

# restart deamon, not taking no for an answer
# usage: restartDamon executableName, initdName
restartDaemon ()
{
   executableName=$1
   initdName=$2
   restartScript="/etc/init.d/${initdName}"

   # make sure that the daemon executable is there
   test -x ${restartScript} || { ${logCritical} "restart script (${restartScript}) is not executable. CRITICAL GUARDIAN ERROR."; exit 1; }

   # try a polite stop
   ${restartScript} stop > /dev/null

   # get medieval on it's ass
   pkill -x ${executableName} ; sleep 2 ; pkill -9 -x ${executableName} ; sleep 2

   # restart the deamon
   ${restartScript} start > /dev/null

   if test $? -ne 0
   then
      ${logCritical} "failed to restart daemon (${executableName}): CRITICAL GUARDIAN ERROR."
      exit 1
   else
      ${logDebug} "daemon (${executableName}) restarted."
   fi
}

#
# things look good, let's do our checks and then schedule a new one
#

# make sure that the checkFile exists
test -r ${checkFile} || { ${logCritical} "checkfile (${checkFile}) is not readable. CRITICAL GUARDIAN ERROR."; exit 1; }

# loop through each of the daemons that need to be managed
for daemon in `cat ${checkFile} | egrep -v "^#.*" | cut -f2 -d, |  sed 's/^[ \t]*//;s/[ \t]*$//' | sort -u`
do
   # execute all the checks for the daemon in question
   cat ${checkFile} | egrep -v "^#.*" | while read line
   do
      getToken "${line}" 2 ; daemonName=${token}

      if test ${daemonName} = ${daemon}
      then
         # get the check definition
         getToken "${line}" 1 ; checkType=${token}
         getToken "${line}" 3 ; executableName=${token}
         getToken "${line}" 4 ; checkSource=${token}
         getToken "${line}" 5 ; checkParams=${token}

         # remove quotes
         checkSourceQuoteless=`echo ${checkSource} | sed "s/\"//g"`
         checkParamsQuoteless=`echo ${checkParams} | sed "s/\"//g"`

         # call the appropriate handler for the check
         ${checkType} "${checkSourceQuoteless}" "${checkParamsQuoteless}"

         if test $? -ne 0
         then
            ${logCritical} "CRITICAL PROBLEMS with deamon (${daemonName}), RESTARTING."
            restartDaemon ${executableName} ${daemonName}
         fi
      fi
   done
done

# delete all at jobs (race conditions)
deleteAllAtJobs

# schedule a new instance of this sucker
${logDebug} "scheduling another check to run in ${checkInterval} minutes"
at -q ${atQueue} now + ${checkInterval} minutes > /dev/null 2>&1 << EOT
$0 $* -b
EOT

Dec 17 2007
Dec 17

I am almost sorry to see winter break drawing near. For the past two weeks, I've had the privilege of introducing an amazing group of kids to open source software. Inspired by GHOP, Google's pilot Highly Open Participation contest, I've put together an extracurricular computer club for interested students at nearby Sandridge Elementary. We meet twice per week after school for an hour and a half.  I came into this with the slim hope that the school's new administration would let me shepard a couple of students through GHOP. Mr. Hollingsworth's (Sandridge's principal) and Dr. Sawyer's (Sandridge's superintendent) enthusiasm took me by surprise, and became a catalyst for the growth of a program that I hope will someday serve as a model for other schools.

Of the club's eight active participants (not counting occassional attendees), seven are trying their hands at GHOP projects, alongside high school students from around the world. Most have chosen to work on projects for Drupal, which makes for good crossover activities with our two under-13s who have taken charge of creating the club's web site.

So far, the biggest challenge for me is leading a group like this in an essentially unwired community. Students generally only have computer access at school. and few parents have regular access to email. Simple things like password resets can take days, and my students are at a disadvantage for things like GHOP. I must give them all credit, though -- despite never before being exposed to basics like FTP, they are all jumping in the deep end and making amazing progress.

I have been lucky enough to round up some great speakers and corporate donations. We are still looking for a speaker in a graphic design field (preferably someone familiar with GIMP, Inkscape, Blender, or other open source tools), and donations of hardware (especially laptops, thin clients, servers, USB keys, blank CDs, and a tablet). If you can help with any of this, please drop me a line.

The open source world is as much a true meritocracy as I think I'll ever see. No one cares who you are, where you are from, or what you have -- they just want to see your code. Anyone can do anything.

Jul 10 2005
Jul 10

The Drupal website has been down for two days now. They haven't received a response from support and hence cannot fix it (sounds familiar to me).

On the long run, Drupal will migrate to the Open Source Lab (OSL) which offers lots of services and already hosts many popular Open Source projects like Mozilla, Apache, and Debian.

To be able to do this, they need a new server (free rack space and bandwidth are provided by OSL) for which they are seeking donations now.
It's also planned to create a non-profit organization which will hold the funds, so the donations will be tax-deductible...

About Drupal Sun

Drupal Sun is an Evolving Web project. It allows you to:

  • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
  • Facet based on tags, author, or feed
  • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
  • View the entire article text inline, or in the context of the site where it was created

See the blog post at Evolving Web

Evolving Web