Jul 16 2014
Jul 16
teaser image for blog post

Since we integrated the EdgeCast CDN for one of our clients, and released a related EdgeCast Drupal module we have been encouraging more and more clients to consider a CDN layer to accelerate performance to multiple geographic locations and maintain an excellent uptime even during site maintenance periods.

A recent international client who is running many domains with federated content using the Domain module needed to make use of the content delivery network to improve performance and resiliance for their sites.

The use of the Domain module created a problem with the purge process employed in the EdgeCast v1.x module. As with most of the cache purging modules available for Drupal it sent each URL individually and waited for confirmation back to say it the request had been successful.

With a large multi domain site this multiplied the page load time upon submitting a node edit form to the point of a PHP timeout and Ngnix HTTP 500 errors. Increasing the timeouts was a short term fix, but it still meant content editors were slowed down waiting for forms to submit.

There was also another effect cause by external feed aggregation which created nodes, which in turn triggered URL purge requests, and slowed down each node creation step.

We toyed with the idea of using a Drupal batch queue to process the purge requests during the regular cron call but this delayed the refreshing of content pages far too long for the content editors.

We settled on what appears to be the little used Multi Curl facility. This allowed us to build up an array of all the URLs using the Expire module and then send a single HTTP request to the EdgeCast API followed by ignoring the response back from the API. This created a much faster content editing experience for the users.

Note: The Expire module doesn't currently implement Domain module purging - the code class is empty. See issue #2293217 for details. We worked around this using the Cache Expiration Alias module.

The side effect of the Multi Curl development work was the speeding up of the single domain sites purge requests too!

The v2.0 EdgeCast module is available now for Drupal 7.

Find out more about the content delivery network from Edgecast.

Oct 07 2013
Oct 07

Using a Reverse Proxy and/or a Content Delivery Network (CDN) has become common practice for Drupal and other Content Management Systems.

One inconvenient aspect of this is that your web server no longer gets the correct IP address, and neither does your application. The IP address is that of the machine that the reverse proxy is running on.

In Drupal, there is code in core that tries to work around this, by looking up the IP address in the HTTP header HTTP_X_FORWARDED_FOR, or a custom header that you can set.

For example, this would be in the settings.php of a server that runs Varnish on the same box.

$conf['reverse_proxy'] = TRUE;
$conf['reverse_proxy_addresses'] = array('127.0.0.1');

There is also this setting for Drupal 7.x in case your CDN puts the IP address in some other custom header:

// CloudFlare CDN
$conf['reverse_proxy_header'] = 'HTTP_CF_CONNECTING_IP';

Only for the application, what about the web server?

But, even if you solve this at the application level (e.g. Drupal, or WordPress), there is still the issue that your web server is not logging the correct IP address. For example, you can't analyze the logs to know which countries your users are coming from, or identify DDoS attacks.

Apache RPAF module

There is a easy solution to this though: the Reverse Proxy Add Forward (RPAF).

What this Apache module does is extract the correct IP address, and uses that for Apache logs, as well hand over the correct IP address of the client in PHP's variable: $_SERVER['REMOTE_ADDR']

To install RPAF on Ubuntu 12.04 or later, use the command:

aptitude install libapache2-mod-rpaf

If you run the reverse proxy (e.g. Varnish) on same server as your web server and application, and do not use a CDN, then there is no need to do anything more.

However, if you run the reverse proxy on another server, then you need to change the RPAFproxy_ips line to include the IP addresses of these servers. For example, this will be the addresses for your Varnish servers which are front ending Drupal, then they are front ended by the CDN.

You do this by editing the file /etc/apache2/mods-enabled/rpaf.conf.

For example:

RPAFproxy_ips 10.0.0.3 10.0.0.4 10.0.0.5

CDN Client IP Header

If you are using a CDN, then you need to find out what HTTP header the CDN uses to put the client IP address, and modify RPAF's configuration accordingly.

For example, for CloudFlare, the header is CF-Connecting-IP

So, you need to edit the above file, and add the following line:

RPAFheader CF-Connecting-IP

Drupal Reverse Proxy settings no longer needed

And finally, you don't need any of the above Reverse Proxy configuration in settings.php.

// $conf['reverse_proxy'] = TRUE;
// $conf['reverse_proxy_addresses'] = array('127.0.0.1');
// $conf['reverse_proxy_header'] = 'HTTP_CF_CONNECTING_IP';

Now, you have correct client IP addresses in Apache's logs, and inside Drupal as well.

What If RPAF Does Not Work?

If you have RPAF front ended directly by a CDN, without Varnish, then RPAF may not work for a yet unknown reason.

To overcome this, you have several other options.

Apache mod_remoteip

There is a small Apache module called mod_remoteip. This basically does the same thing as RPAF, but with simpler configuration.

Use the download link and save the file to the file named apache-2.2-mod_remoteip.c.

apxs2 -i -a -c apache-2.2-mod_remoteip.c

This should create the module's .so file in Apache's modules directory. It should also add the LoadModule directive in mods-available/remoteip.load, which should look like so:

LoadModule remoteip_module modules/mod_remoteip.so

Now add the RemoteIPHeader directive in a new file called mods-available/remoteip.conf

RemoteIPHeader X-Forwarded-For

If you are using CloudFlare CDN then you use:

RemoteIPHeader CF-Connecting-IP

Now, enable the module:

a2enmod remoteip

Then restart Apache:

service apache2 restart

If this does not work, then you can still do it using the next set of tricks:

Apache Access Log and Drupal Reverse Proxy Settings

We can force Apache to log the correct client IP address to the access log by adding this to the virtual host entry for your site (e.g. /etc/apache2/sites-enabled/example.com):

LogFormat "%{CF-Connecting-IP}i %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" proxied

This takes the CF-Connecting-IP header from CloudFlare, and uses that instead of the IP address, which is of the proxy, not the originating client.

Then, under the "VirtualHost" stanza, you add this to use the custom proxied format you created above:

CustomLog ${APACHE_LOG_DIR}/access-example.com.log proxied

Then you need to enable the Drupal reverse proxy setting in settings.php:

$conf['reverse_proxy'] = TRUE;
$conf['reverse_proxy_header'] = 'HTTP_CF_CONNECTING_IP';

You don't need to add the reverse_proxy_addresses variable, because for CloudFlare there are too many of them.

Mar 06 2012
Mar 06

Last Thursday we had the opportunity to meet up wth Bryan Ollendyke aka btopro to discuss his flagship product ELMS, as well as numerous other modules – Accessible Content for making more standards compliant sites, CDN for speeding up sites, Organic Groups for working with students and classes, Spaces, database tuning, and many more.  You’ll want to skip to ~minute 8 or 9 where we worked out our technical difficulties.  Ask any questions in the blog and I’ll do my best to help answer.  Specifically settings for CDN are covered as well as use cases and other general high level approaches to development.   Note that link goes to a JNLP blackboard elluminate session – you can exand the size of the window pretty easily in there

https://sas.elluminate.com/p.jnlp?psid=2012-03-01.1249.M.31E08D9EFCC408A0F784A02BEE522E.vcr&sid=2008268

apologies for the

 

Feb 11 2012
Feb 11
Beer and developer conferences go hand in hand.

A few weeks ago I presented “CDNs made simple fast and cheap” at the Drupal Downunder conference in Melbourne Australia.

The talk covered:

  • the importance of good client side performance,
  • how A CDN works,
  • recommended CDN providers (from an Australian’s perspective),
  • a demonstration of how to set up a CDN and
  • a summary of the results (better YSlow score and page download times).


Setting up a CDN is very easy to do and cost effective. If you want you users to have the best online experience then there is nothing stopping you!

The CDN presentation is available as PDF slides and a video.

Thanks to my employer PreviousNext who kindly sponsored my trip to Melbourne. Hats off to Wim Leers for contributing the CDN module.

[embedded content]

Be Sociable, Share!
Sep 26 2011
Sep 26

Recently I decided to check out Amazon Cloudfront to use as a cdn. I had a delighted experience with it. It was so easy to setup with drupal and configured so easy in just a few steps.

Cloudfront is a pull only cdn as opposed to a push cdn. A pull cdn is a cdn that grabs the files from your site on demand. The only thing you have to do to configure a pull cdn, is switch out the base url of the files you want to serve, to the cdn base url. When that file gets requested the cdn checks if it has it already, if not it downloads it from your server and servers it to the client.

In addition to the easy setup of Amazon Cloud front is it's cost. It is really really cheap. It costs about 12 cents per gigabite of transfer. 

Step 1: Create a Cloudfront distrabution 

I am not going to walk through the sign up process for amazon web services. It is pretty straight forward you can find it here. Once you have an account you need to sign in to the AWS Management Console. The link is found on aws.amazon.com website at top right hand side. 

Next you will get the Management Console for all AWS services. Click on the Cloudfront tab.

Next click the 'Create Distribution' button in the upper left hand side.

You will be prompted with a small configuration screen. Click on 'Custom Origin' and enter your website domain name in 'Origin DNS Name' field and click 'Continue'. NOTE: This is the most basic setup. This can definetly be more complex, but for most users this configuration will work.

On the next configuation screen you can leave everything blank. You can optionally configure a cname. This will allow you to control the url of your cdn. For example, if you wanted everything to come from cdn.example.com you would enter that in, but be aware that you need to configure your DNS appropiatly. Again for most setups you can leave this blank. Click 'Continue'.

Finally you will be brought to the review page, hit 'Create Distribution'.

Next make note of the 'Domain Name' that was assign to you. It should be something.cloudfront.net. It may take a few minutes for the distribution to create it self, but while that is going on, you can install the cdn modle.

Step 2: Install and configure the Drupal CDN module

Now go to drupal.org and download and enable the CDN module.

Once enabled, there is only two simple configuations that you need to do. Go to /admin/config/development/cdn and click on the details tab. In the 'CDN-mapping' textarea paste in the url you got from cloudfront and remeber to put http in front of it and click save.

Lastly, click the 'General' tab at the top right hand corner and check the box to enable CDN.

Thats it! Now all files on your site will be pulled automatically to the cdn and be servered to the world from the closed location.

NOTE: This will cause almost all files to be pulled from the CDN, even javascript files. If you have javascript files that are making AJAX requests, they will not work, because you can not do cross domain ajax calls. So make sure to black list those files. You can also choose to blacklist other files as well in the cdn configuration.

About Drupal Sun

Drupal Sun is an Evolving Web project. It allows you to:

  • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
  • Facet based on tags, author, or feed
  • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
  • View the entire article text inline, or in the context of the site where it was created

See the blog post at Evolving Web

Evolving Web