Apr 01 2013
Apr 01

Ubuntu Server 12.04 LTS finally provides a stable long term support server distro that has a recent version of Varnish in its repositories.

Trouble is, the repository provided package of Varnish has some issues. Specifically, the command line tools, such as varnishhist, varnishstat, ...etc. do not report anything. Therefore one cannot know the hit/miss rates, hits per second, or other useful information. Moreover, monitoring Varnish using Munin for such statistics does not work either.

There are two ways you can overcome this, both are described below.

Use the Varnish provided packages from their repository

The Varnish project provides an Ubuntu repository that contains Ubuntu packages. This is the recommended way, because you get updates for security if and when they come out. As well, you will get startup scripts installed and configured automatically, instead of having to create them from scratch, as in the second option.

First, install curl, if it is not already installed on your server:

# aptitude install curl 

Then, add the repository key to your server:

# curl http://repo.varnish-cache.org/debian/GPG-key.txt |
    sudo apt-key add -

And then add the repository itself.

# echo "deb http://repo.varnish-cache.org/ubuntu/ lucid varnish-3.0" | 
    tee /etc/apt/sources.list.d/varnish.list

Now, update the list of packages from the new repository:

# aptitude update

And then install Varnish

# aptitude install varnish 

Configuring Varnish listening port and memory

Then, you need to configure Varnish to front end your web server, so that it runs on port 80 (regular HTTP traffic) and 443 (SSL secure traffic, if your site uses it).

Also, increase, or decrease the amount of memory allocated to Varnish if your server has memory to spare.

We also change the name of the .vcl file, so when upgrading, Ubuntu will not ask about two files changed, and rather one file only (/etc/default/varnish).

So, go ahead and edit the /etc/default/varnish file, and this to the end, if you have a small server (e.g. 1G RAM).

DAEMON_OPTS="-a :80,:443 \
             -T localhost:6082 \
             -f /etc/varnish/main.vcl \
             -S /etc/varnish/secret \
             -s malloc,128m"

If you have a larger server, e.g. 8GB, you would allocate 2G of RAM for varnish.

DAEMON_OPTS="-a :80,:443 \
             -T localhost:6082 \
             -f /etc/varnish/main.vcl \
             -S /etc/varnish/secret \
             -s malloc,2048m"

Then restart Varnish:

service varnish restart

In an upcoming article, we will discuss the details of configuring Varnish's VCL.

Compiling from source

The second option is to download the source code, and compile it yourself.

This means that you are responsible for upgrading if there is a security fix.

You first need to install the C compiler, and some other libraries like curses development, PCRE, as well as the make utility.

You do this via the command:

# aptitude install libncurses5-dev libpcre3-dev pkg-config make 

Which will pull the C compiler if it is not already installed.

Then, download the source:

# wget http://repo.varnish-cache.org/source/varnish-3.0.3.tar.gz

Extract the source from the archive

# tar xzf varnish-3.0.3.tar.gz

Change the directory to what you just extracted

# cd varnish-3.0.3

Run the configure tool. Make sure there are no errors.

# ./configure

Build the binaries

# make

And install the source.

# make install

Varnish will be installed in /usr/local.

You will need to create start scripts for Varnish.
You should use a different name other than what would have been installed by the repository packages, so that it would not clash with the same file names, e.g. /etc/default/varnish-local. This would hold the DAEMON_OPTS mentioned above. You also need to create an /etc/init.d/varnish-local script for startup. I then use the following command to make Varnish run at run level 2.

update-rc.d varnish-local start 80 2 .

Monitoring Varnish with Munin

We assume that you have Munin installed and configured and is already monitoring other things on your server.

We need to install a perl library that will be able to pull the statistics info from Varnish

aptitude install libnet-telnet-perl

Then, we need get the monitoring scripts

cd /usr/share/munin/plugins/

git clone git://github.com/basiszwo/munin-varnish.git

chmod a+x ./munin-varnish/varnish_*

Then create symbolic links for the monitoring scripts.

cd /etc/munin/plugins

ln -s /usr/share/munin/plugins/munin-varnish/varnish_allocated
ln -s /usr/share/munin/plugins/munin-varnish/varnish_cachehitratio
ln -s /usr/share/munin/plugins/munin-varnish/varnish_hitrate
ln -s /usr/share/munin/plugins/munin-varnish/varnish_total_objects

Then edit the file /etc/munin/plugin-conf.d/munin-node, and add the following to the end.

[varnish*]
user root

And restart Munin for the changes to take effect.

service munin-node restart

Configuring Varnish for Drupal

As mentioned above, in an upcoming article, we will discuss the details of configuring varnish for Drupal.

May 01 2012
May 01

Devopsdays Mountainview sold out in a short 3 hours .. but there's other events that will breath devops this summer.
DrupalCon in Munich will be one of them ..

Some of you might have noticed that I`m cochairing the devops track for DrupalCon Munich,
The CFP is open till the 11th of this month and we are still actively looking for speakers.

We're trying to bridge the gap between drupal developers and the people that put their code to production, at scale.
But also enhancing the knowledge of infrastructure components Drupal developers depend on.

We're looking for talks both on culture (both success stories and failure) , automation,
specifically looking for people talking about drupal deployments , eg using tools like Capistrano, Chef, Puppet,
We want to hear where Continuous Integration fits in your deployment , do you do Continuous Delivery of a drupal environment.
And how do you test ... yes we like to hear a lot about testing , performance tests, security tests, application tests and so on.
... Or have you solved the content vs code vs config deployment problem yet ?

How are you measuring and monitoring these deployments and adding metrics to them so you can get good visibility on both
system and user actions of your platform. Have you build fancy dashboards showing your whole organisation the current state of your deployment ?

We're also looking for people talking about introducing different data backends, nosql, scaling different search backends , building your own cdn using smart filesystem setups.
Or making smart use of existing backends, such as tuning and scaling MySQL, memcached and others.

So lets make it clear to the community that drupal people do care about their code after they committed it in source control !

Please submit your talks here

Mar 10 2011
Mar 10

A couple of weeks ago I noticed a weird drop in web usage stats on the site you are browsing now. Kinda weird as the drop was right around Fosdem when usually there is a spike in traffic.

So before you start.. no I don't preach on practice on my own blog, it's a blog dammit, so I do the occasional upgrades on the actual platform , with backups available, do some sanity tests and move on, yes I break the theme pretty often but ya'll reading this trough RSS anyhow.

My backups showed me that drush had made a copy of the Piwik module somewhere early february, exactly when this drop started showing. I verified the module , I verified my Piwik , - Oh Piwik you say .. yes Piwik, if you want a free alternative to Google Analytics , Piwik rocks .. - I even checked other sites using the same piwik setup and they were all still functional happily humming and being analyzed.... everything fine ... but traffic stayed low ..

This taught me I actually had to upgrade my Piwik too ...

So that brings me to the point I`m actually wanting to make...
as according to @patrickdebois in his chapter on Monitoring "Quis custodiet ipsos custodes?" who's monitoring the monitoring tools, who's monitoring the analytics tools,

So not only should you monitor the availability of yor monitoring tools, you should also monitor if their api hasn't changed in some way or another.
Just like when you are monitoring an web app you shoulnd't just see if you can connect to the appropriate http port, but you should be checking if you get sensible results back from it , no gibberish.

But then again ... there's no revenue in my blog or its statistics :)

About Drupal Sun

Drupal Sun is an Evolving Web project. It allows you to:

  • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
  • Facet based on tags, author, or feed
  • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
  • View the entire article text inline, or in the context of the site where it was created

See the blog post at Evolving Web

Evolving Web