May 06 2013
May 06

We have mentioned before that both Pressflow 6.x and Drupal 7.x (but not core Drupal 6.x), disable page caching when a session is created for an anonymous user.

An extreme case of this happened recently, because of a perfect storm.

Symptoms

The client sends a newsletter to site users, be they who have accounts on the site, or others who just entered their email to get the newsletter.

After a recent code change, when a newsletter was sent, suddenly, we found saw a very high load average, very high CPU usage, and because we plot the number of logged in and anonymous users too, we found around 800 anonymous users on the site, in addition to the peak of 400 logged in users!

Since this is Pressflow, anonymous users are mostly served by Varnish, and they are not supposed to have sessions.

Investigation

So, we started to investigate those anonymous sessions in the database, in the sessions table.

Indeed, there are lots of anonymous sessions.

SELECT COUNT(*) 
FROM sessions 
WHERE uid = 0;
+----------+
| count(*) |
+----------+
|     6664 |
+----------+

And upon closer investigation, most of those sessions had a message in them saying "Comment field is requried".

SELECT COUNT(*) 
FROM sessions 
WHERE uid = 0 
AND session LIKE '%Comment field is required%';
+----------+
| count(*) |
+----------+
|     5176 |
+----------+

And just to compare the day the newsletter was sent to other days, we confirmed that indeed, that day had many multiples of any other day in terms of sessions.

In fact, more than 5X the highest day prior, and up to 55X higher than more typical days.

SELECT DATE(FROM_UNIXTIME(timestamp)) AS date, COUNT(*) 
FROM sessions 
WHERE uid = 0 
GROUP BY date;

+------------+----------+
| date       | count(*) |
+------------+----------+
| .......... |       .. |
| 2013-04-19 |       55 |
| 2013-04-20 |       81 |
| 2013-04-21 |       66 |
| 2013-04-22 |      115 |
| 2013-04-23 |       99 |
| 2013-04-24 |      848 |
| 2013-04-25 |       72 |
| 2013-04-26 |     4524 |
| .......... |       .. |
+------------+----------+

Graphs show the magnitude of the problem

Look at the graphs, to the very right of each one, after Friday noon.

You can see how the load shot up after the newsletter was sent:

The number of anonymous sessions shot up from only a handful to around 800!

The number of logged in users had a spike to 400, up from the 300 or above.

The number of SQL queries also shot up.

And so did the MySQL threads too.

And the CPU usage was very high, with the server trying to serve around 1200 users with no caching for them.

Root Cause Analysis

So, it turns out that the recent code change was done to encourage more people to sign up for an account on the site. This form alters the comment form and adds extra fields to prod the visitor to register for an account, including the email address. Another form above the node also captures the email address.

If people clicked on the button to add their email, Pressflow complained about the missing comment field. And since any message, be it for a logged in users or an anonymous one, is stored in a session, all users who tried to register for an account were treated as logged in users in that they bypass the page cache for Pressflow. This effectively tripled the number of logged in users (from 400 to 1200), who all have to execute PHP and MySQL queries and not being served from Varnish.

Hence the high load and high CPU usage.

Solution

The fix was to revoke the post comment permission for anonymous users, and therefore, remove the comment form from the bottom of every node.

After that, the newsletter was sent without increasing the load the server at all.

Although this problem was on Pressflow 6.x, it should apply to Drupal 7.x as well, since it also disables sessions for anonymous users.

Mar 12 2013
Mar 12

Over the past few years, we were called in to assist clients with poor performance of their site. Many of these were using Pressflow, because it is "faster" and "more scalable" than Drupal 6.x.

However, some of these clients hurt their site's performance by using Pressflow, rather than plain Drupal, often because they misconfigured or misused it in some way or another.

Setting cache to "external" without having a caching reverse proxy

We saw a couple of cases where clients would set the cache to "External" in admin/settings/performance, but they are not running a reverse proxy cache tier, such as Varnish or Squid.

What happens here is that Pressflow will not cache pages for anonymous users, and just issue the appropriate cache HTTP headers, assuming that a caching reverse proxy, e.g. Varnish, will cache them.

Performance of the site will suffer, since it will be hit by search engine crawlers.

The solution is simple: either configure a reverse proxy, or set caching to "normal".

Setting Page Cache Maximum Age too low

In admin/settings/performance, there is a configuration parameter called "Page cache maximum age" (in Pressflow, which is called "Expiration of cached pages" in Drupal 7.x). This value should not be left set to "none", because that means items will not be left in the cache for sufficient time for them to be served for subsequent users. Setting it too low (e.g. 1 minute) has the same effect too.

Do set this parameter to the highest time possible if you have an external cache like Varnish or Squid.

Enabling modules that create anonymous sessions

Both Pressflow 6.x and Drupal 7.x disable page caching for anonymous users if a session is present.

This means that if you have a module that sets a cookie, caching will be disabled, because a cookie needs a session to store it.

This means that code like this will disable page caching for anonymous users:

  $_SESSION['foo'] = 'bar';

The Pressflow Wiki started an effort to list such modules here: Modules that break Pressflow 6.x caching and how to fix them and here: Code that sets cookie or session, but with so many modules being written, it is virtually impossible to have a complete list.

Also, novice Drupal developers will not know this, and write modules that use cookies, and therefore prevent page caching for anonymous users.

We have seen such cases from such developers where a site that was perfectly working previously is rendered fragile an unstable via one line of code!

Not that this fault applies to Pressflow 6.x, and to Drupal 7.x as well.

If you are using the former, then you can solve the problem temporarily by switching to Drupal core 6.x instead of Pressflow 6.x. Drupal code 6.x does not mind cookies for anonymous users.

Using Varnish with hook_boot() or hook_exit() modules

When using an external cache, like Varnish, all anonymous requests do not hit Drupal at all. They are served from Varnish.

So if you have modules that implement hook_boot() or hook_exit(), then the code that is there will not be triggered at all. If you rely on it for some functionality, then it will be hit only the first time the page is accessed.

For example, the core statistics module hook_exit() increments the view count for the node. If you enable this module with this functionality, then these figures will be far lower than the real numbers, and you are better of disabling this module rather than having inaccurate numbers.

Sep 22 2011
Sep 22

Tomorrow is the last day of Summer but the Drupal training scene is as hot as ever. We’ve scheduled a number of trainings in Los Angeles this Fall that we’re excited to tell you about, and we’re happy to publicly announce our training assistance program.

First, though, we’re sending out discount codes on Twitter and Facebook. Follow @LarksLA on Twitter, like Exaltation of Larks on Facebook or sign up to our training newsletter at http://www.larks.la/training to get a 15% early bird discount* toward all our trainings!

Los Angeles Drupal trainings in October and November, 2011

Here are the trainings we’ve lined up. If you have any questions, visit us at http://www.larks.la/training or contact us at trainings [at] larks [dot] la and we’ll be happy to talk with you. You can also call us at 888-LARKS-LA (888-527-5752) with any questions.

Beginner trainings:

Intermediate training:

Advanced trainings:

All our trainings are $400 a day (1-day trainings are $400, 2-day trainings are $800, etc.). We’re excited about these trainings and hope you are, too. Here are some more details and descriptions.

Training details and descriptions

   Drupal Fundamentals
   October 31, 2011
   http://ex.tl/df7

Drupal Fundamentals is our introductory training that touches on nearly every aspect of the core Drupal framework and covers many must-have modules. By the end of the day, you’ll have created a Drupal site that looks and functions much like any you’ll see on the web today.

This training is for Drupal 7. For more information, visit http://ex.tl/sbd7

   Drupal Scalability and Performance
   October 31, 2011
   http://ex.tl/dsp1

In this advanced Drupal Scalability and Performance training, we’ll show you the best practices for running fast sites for a large volume of users. Starting with a blank Linux virtual server, we’ll work together through the setup, configuration and tuning of Drupal using Varnish, Pressflow, Apache, MySQL, Memcache and Apache Solr.

This training is for both Drupal 6 and Drupal 7. For more information, visit http://ex.tl/dsp1

   Drupal Architecture (Custom Content, Fields and Lists)
   November 1 & 2, 2011
   http://ex.tl/ccfl1

Drupal Architecture (Custom Content, Fields and Lists) is our intermediate training where we explore modules and configurations you can combine to build more customized systems using Drupal. You’ll create many examples of more advanced configurations and content displays using the popular Content Construction Kit (CCK) and Views modules.

This training is for Drupal 6. For more information, visit http://ex.tl/ccfl1

   Developing RESTful Web Services and APIs
   November 3, 4 & 5, 2011
   http://ex.tl/dwsa1

Offered for the first time in Southern California, Developing RESTful Web Services and APIs is an advanced 2-day training (with an optional third day of additional hands-on support) for those developers seeking accelerated understanding of exploiting Services 3.0 to its fullest. This is THE training you need if you’re using Drupal to create a backend for iPad, iPhone or Android applications.

This training covers both Drupal 6 and Drupal 7. For more information, visit
http://ex.tl/dwsa1

Training assistance program

In closing, we’d like to tell you about our training assistance program. For each class, we’re setting aside a limited number of seats for students, unemployed job seekers and people in need.

For more details about the program, contact us at trainings [at] larks [dot] la and we’ll be happy to talk with you. You can also call us at 888-LARKS-LA (888-527-5752) with any questions.

* Our early bird discount is not valid toward the Red Cross First Aid, CPR & AED training and 2-year certification that we’re organizing. It’s already being offered at nearly 33% off, so sign up today. You won’t regret it and you might even save someone’s life. ^

Jan 11 2011
Jan 11

Recently I started to notice I very high number of LRU Nuked Objects on my websites, which was essentially wipping the entire Varnish Cache. I run Varnish with 4GB File cache and site ocntent is mostly served by external "Poor Man CDN". So, in theory my site content should not be anything near 4GB, however, Varnish Cache was running out of memory and "Nuking" cached objects.

Sizing your cache

Here is what Varnish Cache man pages have to say:

Watch the n_lru_nuked counter with varnishstat or some other tool. If you have a lot of LRU activity then your cache is evicting objects due to space constraints and you should consider increasing the size of the cache.

Either I can increase the Varnish Cache size or be a little smarter in handling the resource.

I am using the Drupal's Compress Page feature along with Aggrigate JS and CSS features. However, Drupal will only compress the HTML output, which can result to quite large addgiated CSS and JS files. If Varnish is caching the uncompressed output, this will result in considerably more memory usage.

A much better and effective solution is to let Apache's mod_deflate handle the compression instead of Drupal. Drupal compression is oftenten a prefered choice as Drupal would compress>cache once compared to Apache compressing the files on every request. However, if you are using Varnish, which will handle the cache element, Apache would only have to do the hard work once. Also, the added advantage is that you have a granular control over which files are getting compressed.

So, did that make a difference?

LRU Nuked Objects vs Expired Items in Varnish Cache

Varnish Cache Memory Usage

Number of objects in Varnish Cache

Varnich Cache Hit Rate

Bandwidth Usage

Conclusion:

  • Varnish Cache now uses less memory
  • Handles more cached objects
  • No nuking cached items
  • Higher cache hit rate
  • Site is using less bandwidth to serve the same content.

Simplez ;)

Related blog posts: 


Bookmark and Share
Oct 04 2010
Oct 04

Following on from previous post about VCL tweaks to improve hitrate; there are occasions when a website should not be served from both www.foobar.com and http://foobar.com. In some instances Google will deem the content to be duplicate copy of each other and a website can suffer from dupe content penalty.

In such cases, it is often best to redirect (301) the incoming request to either automatically add or remove www to the domain name. So, if a request comes in for http://foo.com, you can get Apache to redirect to http://www.foo.com. Something like:

Apache redirect example

<IfModule mod_rewrite.c>
   Options +FollowSymLinks
   Options +Indexes
   RewriteEngine On
   RewriteBase /
   RewriteCond %{HTTP_HOST} ^foo\.com$
   RewriteRule ^(.*)$ http://www.foo.com/$1 [R=301,L]
</IfModule>

This works great BUT it requires a redirect at the Apache level, wasting precious Apache resources. Wouldnt it be great if Varnish could do the redirect instead and lookup the requested page in cache without waking Apache? Here is how:

Concept is simple:
- varnish checks incoming request.
- match criteria for host
- throw an error
- catch the error
- redirect

Varnish 301 Redirect VCL example

sub vcl_recv {
 
  // rediercts for subdomain, add www
  if (req.http.host == "foo.com") {
    error 301;
  }
 
  // rediercts for subdomain, remove www
  if (req.http.host == "www.barbaz.com") {
    #set req.http.host = "barbaz.com";
    error 301;
  }
}
 
 
sub vcl_error {
 
  // 301 if the domain does not contain www
  if (obj.status == 301 && req.http.host == "foo.com") {
    set obj.http.Location = "http://www.foo.com" req.url;
    set obj.status = 301;
    return(deliver);
  }
 
  // 301 redirect if domain contains www
  if (obj.status == 301 && req.http.host == "www.barbaz.com") {
    set obj.http.Location = "http://barbaz.com" req.url;
    set obj.status = 301;
    return(deliver);
  }
}

Related blog posts: 


Bookmark and Share
Sep 30 2010
Sep 30

The default Varnish config for Pressflow by Four Kitchens is an excellent starting point and gets you up and running with relatively little pain and effort. Having done a fair amount of Varnish tweaking for my personal and work websites, I came across a couple of varnish tweaks that resulted in a phenominal improvement in Varnish Hit rate.

Vary User Agent

It seems Varnish by default will cache every page for every user agent if a Vary: User-Agent header was sent. Unless the website is designed to behave differently for each user agent, this is a wasted resource and will result in a very HIGH miss rate. Most sites will only include style sheets for IE and every other browser.

Unless the sites are optimised for mobile versions, we only need to configure varnish to cache for IE and every other agent, 2 caches.

Varnish can be configured to something like this to reduce number of unique cache hashes:

sub vcl_recv {
  if (req.http.user-agent ~ "MSIE") {
    set req.http.user-agent = "MSIE";
  } else {
    set req.http.user-agent = "Mozilla";
  }
}

Normalizing Vhost namespace

Since a website can be accessed from http://www.foo.com and http://foo.com Varnish will generate a separate page for each one of these urls. This combined with the above User Agent issue, number of hash combinations would be fairly high:

If you need to serve the same web site from multiple URLs, Varnish can be configured like this:

if (req.http.host ~ "^(www.)?foo.com") {
  set req.http.host = "foo.com";
}

Further Reading

Related blog posts: 


Bookmark and Share
Sep 25 2010
Sep 25

After having run janaksingh.com website on Drupal 6 with Apache+PHP+MySql, I wanted to move to Pressflow so I could harness the added advantages of Varnish.

This is not an in-depth installation guide or a discussion about Varnish or Pressflow, but quick setup commands for my own reference. Please see links at the end if you wish to explore Varnish or Pressflow setup in greater depth.

I wanted:
Varnish > APC > Apache > Pressflow setup..

Here is what I did:

Install Apache

yum install httpd
For a detailed installation tutorial, please see: http://janaksingh.com/blog/drupal-development-osx-vmware-fusion-and-cent...

Configure Apache to run on port 8080

sudo nano /etc/httpd/conf/httpd.conf
edit the httpd.conf file to allow apache to run on port 8080

Listen 8080
NameVirtualHost *:8080

Reconfigure all my VHosts to run on port 8080

nano /etc/httpd/conf.d/1_varnish.conf

Disable proxy_ajp file

mv /etc/httpd/conf.d/proxy_ajp.conf /etc/httpd/conf.d/proxy_ajp.conf_dis
<VirtualHost *:8080>
 DocumentRoot /var/www/html/varnished
 ServerName varnished.local
 
 ErrorLog /var/log/httpd/varnished_com_error_log
 LogFormat Combined
 TransferLog /var/log/httpd/varnished_com_access_log
</VirtualHost>

Install EPEL Repo

rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/x86_64/epel-release-5-4.noarch.rpm

Install Varnish

Varnish is bundled in EPEL Repo
yum install varnish

Auto start varnish

sudo /sbin/chkconfig varnish on --level 2345

Configure Varnish to run on port 80

Varnish setup and config

nano /etc/sysconfig/varnish
DAEMON_OPTS="-a :80 \
             -T localhost:6082 \
             -f /etc/varnish/varnished.vcl \
             -u varnish -g varnish \
             -s file,/var/lib/varnish/varnish_storage.bin,1G"

Setup Varnish VCL File for Pressflow

Use example VCL config file from https://wiki.fourkitchens.com/display/PF/Configure+Varnish+for+Pressflow and modify to your needs

nano /etc/varnish/varnished.vcl
backend default {
  .host = "127.0.0.1";
  .port = "8080";
  .connect_timeout = 600s;
  .first_byte_timeout = 600s;
  .between_bytes_timeout = 600s;
}
 
sub vcl_recv {
  if (req.request != "GET" &&
    req.request != "HEAD" &&
    req.request != "PUT" &&
    req.request != "POST" &&
    req.request != "TRACE" &&
    req.request != "OPTIONS" &&
    req.request != "DELETE") {
      /* Non-RFC2616 or CONNECT which is weird. */
      return (pipe);
  }
 
  if (req.request != "GET" && req.request != "HEAD") {
    /* We only deal with GET and HEAD by default */
    return (pass);
  }
 
  // Remove has_js and Google Analytics cookies.
  set req.http.Cookie = regsuball(req.http.Cookie, "(^|;\s*)(__[a-z]+)=[^;]*", "");
 
  // To users: if you have additional cookies being set by your system (e.g.
  // from a javascript analytics file or similar) you will need to add VCL
  // at this point to strip these cookies from the req object, otherwise
  // Varnish will not cache the response. This is safe for cookies that your
  // backed (Drupal) doesn't process.
  //
  // Again, the common example is an analytics or other Javascript add-on.
  // You should do this here, before the other cookie stuff, or by adding
  // to the regular-expression above.
 
 
  // Remove a ";" prefix, if present.
  set req.http.Cookie = regsub(req.http.Cookie, "^;\s*", "");
  // Remove empty cookies.
  if (req.http.Cookie ~ "^\s*$") {
    unset req.http.Cookie;
  }
 
  if (req.http.Authorization || req.http.Cookie) {
    /* Not cacheable by default */
    return (pass);
  }
 
  // Skip the Varnish cache for install, update, and cron
  if (req.url ~ "install\.php|update\.php|cron\.php") {
    return (pass);
  }
 
  // Normalize the Accept-Encoding header
  // as per: http://varnish-cache.org/wiki/FAQ/Compression
  if (req.http.Accept-Encoding) {
    if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") {
      # No point in compressing these
      remove req.http.Accept-Encoding;
    }
    elsif (req.http.Accept-Encoding ~ "gzip") {
      set req.http.Accept-Encoding = "gzip";
    }
    else {
      # Unknown or deflate algorithm
      remove req.http.Accept-Encoding;
    }
  }
 
  // Let's have a little grace
  set req.grace = 30s;
 
  return (lookup);
}
 
 sub vcl_hash {
   if (req.http.Cookie) {
     set req.hash += req.http.Cookie;
   }
 }
 
 // Strip any cookies before an image/js/css is inserted into cache.
 sub vcl_fetch {
   if (req.url ~ "\.(png|gif|jpg|swf|css|js)$") {
     // This is for Varnish 2.0; replace obj with beresp if you're running
     // Varnish 2.1 or later.
     unset obj.http.set-cookie;
   }
 }
 
 sub vcl_error {
   // Let's deliver a friendlier error page.
   // You can customize this as you wish.
   set obj.http.Content-Type = "text/html; charset=utf-8";
   synthetic {"
   <?xml version="1.0" encoding="utf-8"?>
   <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
   <html>
     <head>
       <title>"} obj.status " " obj.response {"</title>
       <style type="text/css">
       #page {width: 400px; padding: 10px; margin: 20px auto; border: 1px solid black; background-color: #FFF;}
       p {margin-left:20px;}
       body {background-color: #DDD; margin: auto;}
       </style>
     </head>
     <body>
     <div id="page">
     <h1>Page Could Not Be Loaded</h1>
     <p>We're very sorry, but the page could not be loaded properly. This should be fixed very soon, and we apologize for any inconvenience.</p>
     <hr />     <h4>Debug Info:</h4>
     <pre>
 Status: "} obj.status {"
 Response: "} obj.response {"
 XID: "} req.xid {"
 </pre>
       <address><a href="http://www.varnish-cache.org/">Varnish</a></address>
       </div>
     </body>
    </html>
    "};
    deliver;
 }

Restart Apache and Varnish

/etc/init.d/httpd restart
/etc/init.d/varnish restart

Further reading:

Configure Varnish for Pressflow
Modules that break caching, and how to fix them
Using Pressflow behind a reverse proxy cache
Example Varnish VCL for a Drupal / Pressflow site
Varnish configuration to only cache for non-logged in users
Varnish best practices
Drupal guide to Managing site performance

Related blog posts: 


Bookmark and Share
Aug 29 2010
Aug 29

DrupalCon Copenhagen comes to an end, as does my blogging hiatus.

Two of my primary learning objectives here in Copenhagen were configuration management and deployment process. Historically, working with Drupal in these areas has been unpleasant, and I think that's why there is tons of innovation going on in that space right now. It needs to be fixed, and new companies are springing up to say "hey, we fixed it." Often, the people running the companies are the same people running the project that encapsulates the underlying technologies. I'm referring to:

  • The hyper-performant core distro, Pressflow
  • Distros with sophisticated install profiles, like OpenAtrium, ManagingNews and OpenPublish
  • Configuration externalization with Features
  • Development Seed's "for every site, a makefile" workflow using drush make
  • The different-yet-overlapping hosting platforms Pantheon and Aegir

Dries commented in his keynote that as Drupal continues to grow, it also needs to grow up. I think advances like these are part of the community's answer to that. I want to wrap my head around some of these tools, while continuing to watch how they progress. Others, I want to implement right now. What's perfectly clear though is that I have a lot of work to do to keep up with the innovation going on in this hugely powerful community. Which is actually nothing new, but reading a blog post about these technologies doesn't make my jaw drop the way that it does when I'm in the room watching Drupal advance.

About Drupal Sun

Drupal Sun is an Evolving Web project. It allows you to:

  • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
  • Facet based on tags, author, or feed
  • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
  • View the entire article text inline, or in the context of the site where it was created

See the blog post at Evolving Web

Evolving Web