Upgrade Your Drupal Skills

We trained 1,000+ Drupal Developers over the last decade.

See Advanced Courses NAH, I know Enough
Jan 29 2017
Jan 29

You can now easily test your Drupal projects on AppVeyor. Currently, AppVeyor is the major player in CI regarding Windows Servers. On other CI systems (Travis, Bitbucket pipelines) you are limited to Docker containers for the *nix platform. (This will soon change as some CI will throw Windows containers into the mix).

Until then, the only tool to CI your Drupal (or any PHP project) on a Windows based environment using IIS is AppVeyor.

In this post I'll show you how we setup free (because AppVeyor is free for open source/public projects) for the MS SQL Server driver for Drupal.

After opening a windows account, go the the + New Project menu.

AppVeyor automatically integrates with major code repository providers such as GitHub, BitBucket, Gitlab and Visual Studio Online. In our case, we will use the "raw" Git option, to connect to any Git repository (in this case the drupal.org one):

All the configuration for the project is done through an appveyor.yml file that you commit to the root of your repository. The reference for this file can be found here:

https://www.appveyor.com/docs/appveyor-yml/

In this file you configure the build script, services and any other behaviour required for your CI process.

We prepared a completely automated build and test script to download drupal, install it on a local IIS site and run the complete core test suite:

version: 1.0.{build}
skip_tags: true
init:
- ps: 
services:
- mssql2014
- iis
install:
- cmd: >-
    net start MSSQL$SQL2014
    powershell -command "Set-Service 'SQLAgent$SQL2014' -StartupType Manual"
    net start W3SVC
    powershell -command "new-item c:\tools\php\ext -itemtype directory"
    powershell -command "new-item c:\tmp -itemtype directory"
    rmdir c:\tmp /s /q
    powershell -command "new-item c:\tmp -itemtype directory"
    powershell -command "(New-Object Net.WebClient).DownloadFile('https://github.com/Microsoft/msphpsql/releases/download/4.1.5-Windows/7.0.zip','C:\tmp\sqlsrv.zip')"
    powershell -command "(new-object -com shell.application).namespace('C:\tmp').CopyHere((new-object -com shell.application).namespace('C:\tmp\sqlsrv.zip').Items(),16)"
    copy /Y "c:\tmp\7.0\x64\php_pdo_sqlsrv_7_nts.dll" "c:\tools\php\ext\php_pdo_sqlsrv.dll"
    powershell -command "(Get-Item c:\tools\php\ext\php_pdo_sqlsrv.dll).VersionInfo"
    rmdir c:\tmp /s /q
    powershell -command "new-item c:\tmp -itemtype directory"
    powershell -command "(New-Object Net.WebClient).DownloadFile('http://windows.php.net/downloads/pecl/releases/wincache/2.0.0.8/php_wincache-2.0.0.8-7.0-nts-vc14-x64.zip','C:\tmp\wincache.zip')"
    powershell -command "(new-object -com shell.application).namespace('C:\tmp').CopyHere((new-object -com shell.application).namespace('C:\tmp\wincache.zip').Items(),16)"
    copy /-Y "c:\tmp\php_wincache.dll" "c:\tools\php\ext\php_wincache.dll"
    powershell -command "(Get-Item c:\tools\php\ext\php_wincache.dll).VersionInfo"
    cinst -y OpenSSL.Light
    SET PATH=C:\Program Files\OpenSSL;%PATH%
    sc config wuauserv start= auto
    net start wuauserv
    cinst -y --allow-empty-checksums php -version 7.0.9
    cd c:\tools\php
    copy php.ini-production php.ini
    echo date.timezone="UTC" >> php.ini
    echo extension_dir=ext >> php.ini
    echo extension=php_openssl.dll >> php.ini
    echo extension=php_wincache.dll >> php.ini
    echo extension=php_pdo_sqlsrv.dll >> php.ini
    echo extension=php_com_dotnet.dll >> php.ini
    echo extension=php_sockets.dll >> php.ini
    echo extension=php_mbstring.dll >> php.ini
    echo extension=php_soap.dll >> php.ini
    echo extension=php_curl.dll >> php.ini
    echo extension=php_gd2.dll >> php.ini
    echo extension=php_gettext.dll >> php.ini
    echo zend_extension=php_opcache.dll >> php.ini
    echo opcache.enable=1 >> php.ini
    echo opcache.enable_cli=1 >> php.ini
    echo opcache.memory_consumption=128 >> php.ini
    echo opcache.revalidate_freq=1500 >> php.ini
    echo opcache.max_accelerated_files=8000 >> php.ini
    echo wincache.ucenabled=1 >> php.ini
    echo wincache.ucachesize=128 >> php.ini
    echo wincache.fcenabled=0 >> php.ini
    echo realpath_cache_size=5M >> php.ini
    echo realpath_cache_ttl=1800 >> php.ini
    echo pdo_sqlsrv.client_buffer_max_kb_size=24480 >> php.ini
    echo Setup certificate store
    powershell -command "(New-Object Net.WebClient).DownloadFile('https://curl.haxx.se/ca/cacert.pem','C:\cacert.pem')"
    echo curl.cainfo="C:\cacert.pem" >> php.ini
    SET PATH=C:\tools\php;%PATH%
    powershell -Command ($env:Path)
    powershell -command "new-item c:\composer -itemtype directory"
    cd /d C:\composer
    php -r "readfile('http://getcomposer.org/installer');" | php
    powershell -command "(Get-Item C:\composer\composer.phar).length"
    powershell -command "'@php C:\composer\composer.phar ' + $([char]37) + '*' | Out-File C:\composer\composer.bat -Encoding ASCII"
    SET PATH=C:\composer;%PATH%
    cd /d C:\projects\
    composer create-project -n drupal-composer/drupal-project:8.x-dev
    cd /d C:\projects\drupal-project
    composer config repositories.drupal composer https://packages.drupal.org/8
    composer require drupal/sqlsrv:~2.0
    xcopy /S /I /E %cd%\web\modules\contrib\sqlsrv\drivers %cd%\web\drivers
    composer config repositories.1 git https://%GITLABUSERNAME%:%GITLABPASSWORD%@gitlab.com/david-garcia-garcia/mssql.git
    composer require david/mssql
    powershell -command "'@php %cd%\vendor\drupal\console\bin\drupal ' + $([char]37) + '*' | Out-File %cd%/web/drupal.bat -Encoding ASCII"
    cd /d C:\projects\drupal-project\web
    drupal about
    drupal site:install standard --langcode="en" --db-type="sqlsrv" --db-host="localhost\SQL2014" --db-name="mydrupalsite" --db-user="sa" --db-pass="Password12!" --db-port="1433" --site-name="SQL Server Drupal Site" --site-mail="[email protected]" --account-name="admin" --account-mail="[email protected]" --account-pass="admin" --no-interaction
    drupal about
    cd /d C:\projects\drupal-project
    powershell -command "(New-Object Net.WebClient).DownloadFile('https://www.drupal.org/files/issues/2294731-39-phpunit-windows.patch','%cd%\patch.patch')"
    git apply patch.patch --directory=web
    powershell -command "(New-Object Net.WebClient).DownloadFile('https://www.drupal.org/files/issues/use_the_php_binary-2748883-15.patch','%cd%\patch.patch')"
    git apply patch.patch --directory=web
    powershell -command "(New-Object Net.WebClient).DownloadFile('https://www.drupal.org/files/issues/simpletest_is_broken_on-2605284-61.patch','%cd%\patch.patch')"
    git apply patch.patch --directory=web
    powershell -command "(New-Object Net.WebClient).DownloadFile('https://patch-diff.githubusercontent.com/raw/hechoendrupal/drupal-console/pull/3134.patch','%cd%\patch.patch')"
    git apply patch.patch --directory=vendor/drupal/console
    cd /d C:\projects\drupal-project\web
    drupal module:install simpletest
    choco install -y urlrewrite
    powershell -command "New-WebSite -Name 'MyWebsite' -PhysicalPath 'c:\projects\drupal-project\web' -Force"
    echo 127.0.0.1 www.mysite.com >> %WINDIR%\System32\Drivers\Etc\Hosts
    powershell -command "Import-Module WebAdministration; Set-ItemProperty 'IIS:\Sites\MyWebsite' -name Bindings -value @{protocol='http';bindingInformation='*:80:www.mysite.com'}"
    SET PATH=%systemroot%\system32\inetsrv\;%PATH%
    echo Change default anonymous user AUTH to ApplicationPool
    appcmd set config -section:anonymousAuthentication /username:"" --password
    echo Setup FAST-CGI configuration
    appcmd set config /section:system.webServer/fastCGI /+[fullPath='C:\tools\php\php-cgi.exe']
    echo Setup FACT-CGI handler
    appcmd set config /section:system.webServer/handlers /+[name='PHP-FastCGI',path='*.php',verb='*',modules='FastCgiModule',scriptProcessor='C:\tools\php\php-cgi.exe',resourceType='Either']
    iisreset
    NET START W3SVC
    powershell -command "wget http://www.mysite.com/"
test_script:
- cmd: >-
    cd /d C:\projects\drupal-project
    mkdir c:\testresults\
    php web/core/scripts/run-tests.sh --php php --all --verbose --url "http://www.mysite.com/" --xml c:\testresults\
artifacts:
- path: c:\testresults
  name: Results

You can see the CI process here with build results:

https://ci.appveyor.com/project/david-garcia-garcia/sqlsrv

If you host your projects in Bitbucket or GitLab AppVeyor has out of the box integrations that will update build statuses for pull requests, branches, commits, etc.

You can use the above testing script as a base template to implement CI and testing in your Windows based Drupal projects (private or public) against a MS SQL Server.

In order for Javascript based tests (full browser intergation tests) to work with the drupal test suite, you need to install and setup PhantomJs on your testing machine. Use these commands:

- cmd: choco install phantomjs
- ps: Start-Job -Name "test" -ScriptBlock {cmd /c "phantomjs --ssl-protocol=any --ignore-ssl-errors=true vendor/jcalderonzumba/gastonjs/src/Client/main.js 8510 1024 768 false 2>&1 >> c:\gastonjs.log"}
Oct 30 2016
Oct 30

Being able to analyze and monitor - with ease - the performance of your application is a key part to the success of any web based project. Not only because a slow site might negatively affect conversions, but as Drupal 8 is shifting away from websites and more into complex web based applications, specific business transactions are becoming more important. Considering Drupal's historical poor performance when it comes to "backend" operations, having proper profiling tools is a must.

The RPS (requests per second) metric becomes meaningless in applications where user interaction is the norm, and is only relevant now for brochureware.

 

In this post I will be discussing the Tideways Profiler: a FREE and OPEN SOURCE maintained drop-in replacement for XHProf and UProfiler that has support for PHP7, Windows and Linux and is actively maintained. But this is not just a drop-in replacement for XHprof and UProfiler.... to keep the business running - Tideways - the company behind the free and open source PHP extension provides a cloud service (very much like the proprietary blackfire.io) where you can collect, monitor and analyze your profiling sessions. If you want to stay on the cheap side, you can still use any of the free XHprof and UProfiler compatible tools such as XHGUI.

Why is Tideways important? Because if it was not for it there wouldn't be any available free and open source profiling tool that you can use with PHP7 and that is actively maintained. Worth mentioning here is that you can still profile with the XDebug extension (also free and open source) but it is not suitable for continuous performance monitoring neither for production usage. And to be honest, it is a pain to use even in development environments as the application becomes orders of magnitude slower when enabled.

At some point a big PHP "customer" called Facebook realized that they needed a tool to profile PHP software that could be deployed to production environments. Facebook developed such a tool in-house and open sourced it somewhere between the end of 2008 and the start of 2009. They worked on it for a couple of years until the project was abandoned in 2013. The original PECL extension can be found here

Then a fork of XHprof called UProfiler took the lead, being taken care after by Friends Of PHP, a small group of individuals that includes Fabien Potencier and Pierre Joye (among others).

During those years, integrations with XHprof/UProfiler poped for almost any platform (such as the Drupal XHProf module) as well as external tools to collect, analize and monitor profiling data such as XHgui.

But as projects mature and the individuals and companies behind them start to seek feasible business models to keep the business running (because just giving away simply won't work), things start to change. And so hapend with UProfiler. With Fabien being the main maintainer and seeking monetization on the hard built PHP ecosystem, they released the proprietary Blackfire.io knowing that the arrival of PHP7 would mean that - unless someone big took the lead and upgraded the current profiler - people would rather pay a handful of bucks for a proprietary solution (as PHP now had found it's way into more serious business where being cheap/free is not important anymore). It was obvious that they had no interest in anyone keeping up the work on UProfiler or forking it out, otherwise this thread would have not been locked.

Then Tideways was born. A small company with talented and dedicated individuals that forked the UProfiler extension, made it PHP7 compatible, added full Linux and Windows support and built a cloud based business model while keeping the PHP Extension free and fully backwards compatible with XHprof and UProfiler tools.

If you deploy the free PHP extension with the Tideways daemon, profiling data can be collected directly on your production environments (without affecting performance) and sent to the Tideways cloud service to be analyzed.

The Tideways website already contains detailed instructions on how to setup Tideways extension and Daemon on Linux. They also give instructions for Windows, but I feel those to be a bit lacking and will expand them here.

The first thing you need is the Tideways PHP extension that you download from the CI service APPVeyor:

https://ci.appveyor.com/project/tideways/php-profiler-extension/history

Look for the latest successful build (the most recent one with the "green light"):

Then choose which build you need for your specific platform (PHP Version, Thread Safety and x86/x64):

Then navigate to the artifacts tab, where you fill find the download link for the compiled PHP extension:

As usually, download the php_tideways.dll into your extension folder (/ext) and enable it in your php.ini file:

extension=php_tideways.dll

After enabling the PHP extension in PHP.ini you should see it in a phpinfo() dump:

Download the Tideways.php PHP library from here:

https://github.com/tideways/profiler/blob/master/Tideways.php

And store it somewhere in your system, such as d:\_Webs\xxx\Tideways.php

Now configure the extension in your PHP.ini file:

extension=php_tideways.dll
tideways.auto_prepend_library=0
auto_prepend_file = D:\_Webs\xxx\Tideways.php
tideways.connection=tcp://127.0.0.1:8136
tideways.api_key=MYAPIKEY
tideways.sample_rate=50
tideways.collect=profiling
tideways.monitor=full

The tideways.api_key value here should be the API key for an application that you have setup through the Tideways Cloud Service:

https://app.tideways.io

They offer 30 day trials so you can easily test this without commitment.

To get your environment to send data to the Tideways cloud service you need to deploy the Tideways daemon, that you can download from here.

s3-eu-west-1.amazonaws.com/qafoo-profiler/downloads/testing/tideways-windows-4.0.1p3.zip

If you are just trying out the profiler, you can start the daemon manually from the console using:

D:\your_path\daemon_windows_amd64.exe --address="127.0.0.1:8136"

 

But if you are going to deploy this on production environments you need to setup the daemon as a windows service.

Unfortunately, as the tideways daemon is not a .Net native windows service, the recommended way to deploy it as a service is to use the Non Sucking Service Manager:

The NSSM manager will ensure that the daemon is up and running, and to restart it in case it crashes or gets stuck.

Remember to let the daemon through your firewall, otherwise it won't be able to report data to the Tideways cloud service.

The rest of the Tideways setup instructions are much the same as the ones for Linux.

You should also get ahold of the Tideways Chrome Extension. Usually on production environments you don't capture full call traces (due to performance reasons) and you only profile a percentage of the requests (see the Tideways official setup guide for more details about this). Using the Chrome extension you can tell the profiler to capture full traces during a period of time directly from your browser.

After deploying the Tideways extension and daemon, you should install the Tideways Drupal module.

Once enabled, you will see a small report in your status page:

There is nothing more to do from Drupal. Under the hood what the module does is tell the Tideways extension meaningful information about each request, so that you can find useful information in the Tideways cloud reports.

Launch your website on Chrome and using the Tideways Chrome exension, start a full trace capturing session:

 

Then the most recent captured traces from the manual session will start showing (right above the regular continous monitoring data):

Choose any specific business transaction and access the full data:

Out of the box you get full traces, callgraph, timelines and much more. If you need anything custom, you can always add custom instrumentation.

To get a deeper look into the possibilities of Tideways, use the official documentation.

May 29 2016
May 29

On a previous post I explained how we are using BigPipe in Drupal 7 (Cheap Pipe (sort of BigPipe) in Drupal 7). Besides all the known benefits of big pipe, there is a less known side effect that might help you fight spam.

Those out there fighting for conversion optimization know that captcha based form protection is a conversion killer (even with the new NoCaptcha from Google). That's why you should always be looking for ways to protect your web application from bots without real users noticing or their workflows being affected.

Even non-intrusive method such as Honeypot or Hidden Captcha are becoming less effective as bots get more sofisticated.

For us, what has been working very well historically against SPAM is using real e-mail verification when capturing cold leads.This basically consists in trying to deliver a fake message to the e-mail address submited to the form. Addresses used by SPAM bots usually do not exist or have been throttled by the e-mail provider. Of course, e-mail verification is the last stand after being able to go through a set of non intrusive form protection techinques (such as honeypot and/or hidden captcha).

A recent change in one of our customer's applications resulted in an increased number of fake leads being sent to Sendy that were able to pass e-mail verification as they were using valid and not throttled e-mail addresses. If you don't know what Sendy is, it's a self-hosted "simple Mailchimp" that can bring massive savings to e-mail based marketing.

So we decided to  force BigPipe on the block that renders the new lead capturing form, and SPAM submissions stopped inmediately. This makes some sense. Content that is rendered using BigPipe is loaded into the DOM through AJAX. I guess that - for performance reasons - most bots probably don't use a full emulated browser and rely on parsing/scanning the HTML response from the server to perform automatic form submissions. What used to be a plain rendered form, is now this to the eyes of the bot:

As a final word, these are the steps you need to follow to completely stop spam:

  • Try with ordinary/site building methods (honeypot, captcha, etc..)
  • Code some custom integration of those methods with your public forms
  • Implement application specific validations
  • Hire someone that knows what they are doing
May 22 2016
May 22

The basics

If you run an online business you should take analytics very seriously. Improving sales, conversions and any other objectives your web application has is an iterative process that needs to be based on measurable and meaningful indicators.

Google Analytics is the most widely used tool to track user data and interactions with your web application, but if you don't have a clear strategy on what you expect and what you are going to do with this data you will easily see yourself failing to extract any value from the overwhelming amount of data that GA collects by default.

If your current digital presence provider is just "enabling" analytics for you - relying on the fact that GA captures a lot of data out-of-the-box - you are wasting your time because:

  • It is difficult - and sometimes not possible - to map the default collected data with the specific processes and interactions on your web application.
  • Although it might be "fun" to track page views, visitors and other global indicators, if you are not integrating this into a defined strategy with objectives it will be useless.
  • Conversions - something that you will always want to track in an e-commerce - do not work out of the box. Avoid "url based" conversion tracking or "configurable" conversion tracking through GA. Conversions should be always reported to GA with laser precision from within your application. Stay away from conversions measured through "thank you pages" and anything similar.

Although GA can get extremely complex, there are some basic steps you can take to ensure that you are at least extracting some real value from this tool:

  • Identify what are the key processes/interactions in your web application. Group them conceptually (i.e. the different pages during a checkout process are all part of the checkout process). Of course, this includes regular conversions (a sale, a subscription, etc..) but can also be extended to processes the user has to go through before performing the conversion and other non-conversion related stuff but that is related to global goals such as improve engagement, reduce bounces, etc.
  • Define meaningful indicators for this processes, when possible choose indicators or groups of indicators that answer a specific question and that can point to potential areas of improvement.
  • Ask your digital provider to track this data as GA events. No excuses here. Everything can be tracked, both from server side and client side with a couple of lines of code. Even e-mail views and opens (for example from autoresponders in your application) can be tracked as conversions or GA events
  • Track this information for a reasonable period of time, no need to seek for statisticall significance here, just use your common sense.
  • Go through the data to detect areas of improvement. If needed, define new indicators to expose an area or opportunity or collect additional data that will allow you to propose improvements.
  • Make your changes.
  • Wait.
  • Repeat.

Besides tracking specific events, you should always

  • Track conversions (sales, subscriptions, etc.)
  • Ensure that your web application is properly managing the User Id functionality of GA. The User Id funcionality allows google to track everything a visitor has done on a site, or even accross devices if you have an authentication system in place. Imagine that you could open a sale on your e-commerce, retrieve the UserId, go to the GA panel and see exactly everything that customer did prior to the purchase (how many visits, how many time, what pages, etc.). This is what the User Id is for.
  • If you are spending money on Google Adwords make sure that you are perfectly propagating conversion values to GA. If you can't perfectly mesure the ROI of what is being spent in Adwords you are throwing money away. Adwords now allows you to use any event from GA to track conversion values. 

Tools for doing this in Drupal

From a technical point of view you need the following to support the above strategies:

  • Embed the GA tracking script
  • Track client side events
  • Track server side events
  • Manage the lifetime of the UserId and integrate it with your application

Embed the GA tracking script

The first thing you need to do is to add the GA tracking script to your application. You can do so with the Google Analytics Drupal module or embed the script programatically. If you use the module there some tweaks you can do and some extended tracking you can setup.

Track client side events

There's the Google Analytics Event Tracking Drupal module that let's you define jquery selectors on a server side hook that will trigger GA events on the client side. This might be a good starting point, but as soon as you want to track interactions that cannot be declared through a selector (for example scrolled to end of page, or hovered over an area) or a combination of interactions, you should go manual. Don't worry, this is super easy.

To trigger an event client side just use this sample code:

ga('send', {
          'hitType': event.hit_type,
          'eventCategory': event.event_category,
          'eventAction': event.event_action,
          'eventLabel': event.event_label,
          'eventValue': event.event_value
        });

This is just a sample, you need to decide when to fire the event and consider situations such as an event being fired twice for the same user. You can do whatever you want here with some code.

Track server side events

The Google Analytic Events Drupal module exposes a small API that will let you trigger GA events during the execution of server side code. These events are sent to the client via Javascript on both page loads and Ajax calls.

Use the following code to trigger an event:

\Drupal\google_analytics_events\EventService::getInstance()->queueEvent(
          (new \Drupal\google_analytics_events\Event())
          ->setHitType('event')
          ->setEventCategory('checkout')
          ->setEventAction('personal_data_submit')
          ->setEventLabel($curso->titulo->value())
          ->setEventValue(0)
        );

Manage the lifetime of the UserId and integrate it with your application

This one is a little more tough to implement as you need to make some operational decisions that depend the nature of your web application.

For example, if you have a 100% anonymous checkout e-commerce where users never log-in (those exist and work quite well if properly crafted) you can manage the lifetime of the UserId using client side cookies, and then store this data server side to match a UserId with whatever you use to store conversions (i.e. a sale).

Here are Google's guidelines to implement this feature:

https://developers.google.com/analytics/devguides/collection/analyticsjs...

May 01 2016
May 01

This post is on how we implemented a simple (yet effective) BigPipe "like" rendering strategy for Drupal 7.

Why is big pipe so important?

Big pipe is a render strategy that assumes that not all the parts of your page are equally important, and that a loaded delay on some of them is acceptable as long as the "core" of the page is delivered ASAP. Furthermore, instead of delivering those delayed pieces with subsequent requests (AJAX) it optimizes network load by using a streamed HTTP response so that you get all those delayed pieces in a single HTTP request/response.

Big pipe does not reduce server load, but dramatically improves your website load time if properly integrated with your application.

Sounds great, and will work very well on some scenarios.

Take for example this landing page (excuses for the poor UX, that's a long story...):

This page has about 20 views and blocks. All of those views and blocks are cached, but can you imagine what a cold cache render of that page looks like? A nightmare....

What if we decided that only 4 of those views were critical to the page, and that the rest of the content could be streamed to the user after the page has loaded? It willl roughly load 70% faster.

UPDATE: Adding support for content streaming has oppened the door to awesome succesfull business strategies - without penalizing initial page load times - such as geolocalizing (or even customizing per user) blocks, advertising and others. All of that while keeping page cache turned on and being able to handle similar amounts of traffic on the same hardware, and without resorting to custom Ajax loading (and coding).

We decided to take a shot and try to implement a big-pipe like render strategy for Drupal 7. We are NOT trying to properly do BigPipe, just something EASY and CHEAP to implement and with little disruption of current core - that's why this is going to be dubbed Cheap Pipe instead of Big Pipe.

Furthermore, it was a requirement that this can be leveraged on any current Drupal application without modifying any existing code. It should be as easy as going to a block or view settings and telling the system to stream it's contents. It should also provide programmatic means of defining content callbacks (placeholders) that should be streamed after the page is served.

We made it, and it worked quite well!

Now every block has a "Cheap Pipe" rendering strategy option:

Where:

  • None: Block is rendered as usual.
  • Only Get: Cheap pipe is used only on GET requests
  • Always: Cheap pipe is used on GET/POST and other HTTP methods.

Cheap pipe is never used on AJAX requests no matter what you choose here.

Why these options? Because some blocks might contain logic that could missbehave depending on the circumstances, and we want to break nothing. So you choose what blocks should be cheap piped, how, and in what order.

What happens after you tell a block (the example is for blocks but there is an API to leverage this on any rendered thing) to be cheap-piped?

  • The block->view() callback is never triggered and the block is not renderd but replaced with a placeholder.
  • The page is served (flushed to the user) and becomes fully functional by artificially trigerring the $(document).ready() event. The </body></html> tags a removed before serving so that the rest of the streamed content is properly formed.
  • After the page has been served to the user, all deferred content is streamed by flushing the php buffer as content gets created and rendered.
  • This content is sent to the user in such a way that it leverages the Drupal AJAX  framework (although this is not AJAX) so that every new content that reaches the page gets properly loaded (drupal behaviours attached, etc...)

Take a look at this end-of-page sample:

The output markup even gives you some stats to see what time it took to render each Cheap Piped piece of content.

Because cheap piped elements are generated sequentially, if an element is slow, it will delay the rendering of the rest of the elements. That's why we implemented a "weight" property so that you can choose in what order elements are cheap-piped.

What kind of problems did we find?

  • Deferred content that called drupal_set_message() was, obviously, not working because the messages had already been processed and rendered. Solved by converting the messages rendering into Cheap Pipe and making it the last one to be processed (thanks to the weight property).
  • Deferred content that relied on drupal_goto() (such as forms in blocks) would not work because the page had already been served to the user. drupal_goto() had to be modified so that if cheap pipe rendering had already started, the redirection was done client side with javascript.
  • When fatals are thrown after the main content has been served your page gets stuck in a weird visual state. There is nothing we can do about this because after fatals you loose control of php output.
  • Server load sky rocketed. What used to be anonymous pages served from cache, now require a full Drupal bootstrap to serve out the streamed content.
Apr 14 2016
Apr 14

Drupal 8 performance: the Supercache module

Post date: 

Thu, 04/14/2016 - 00:00

Difficulty: 

Piece of Cake

The Supercache module is the result of an attempt to improve Drupal 8 efficiency when dealing with cache tag management and other design issues with several caching components that make it a pain to deal with Drupal 8 based applications that change a lot. 

An out of the box Drupal 8 install will issue about 2,100 database statements for a simple task such as performing a log in and creating two articles.

With a little setup and the Supercache module I was able to bring that down to 240 statements.

Here is a video proof that these numbers are real. The statement count is being measured real time thanks to the awesome SQL Server Profiler tool.

[embedded content]

The impact of the Supercache module - that was for a while a core patch - was benchmarked and proved to reduce wall times by about 25% and database queries by as much as 50% after things change (doing a cache write).

How does the Supercache module do this?

  • Drupal's cache system got heavier in Drupal 8 at the expense of making it more precise thanks to cache tags. But there are situations where you simply do not need all that bloatage. The Supercache module introduces a new and sleeker cache layer (of course without cache tags). A simple cache tag stored in Drupal's cache system takes up 196 bytes. The new caching system only uses 12 bytes. This does not seam like a big deal after all, it's just a few bytes difference. But it translates to being able to store 65,000 cache tags in 1MB of APCu/Wincache instead of just 5,000, But that is not the only advantage of this cache layer:
    • Reduced storage size, up to x12 less for small cache items.
    • Levarage native functionalities provided by most storage backends such as touch, counter, increment, etc.
    • Faster processing due to lack of cache tags and other extras.
    • Scalar types are stored natively so you can batch operate on the cache items themselves if the storage backend allows you to do so (database, couchbase or mongo)
  • Drupal 8 introduced the very useful ChainedFastBackend (that you can easily use in Drupal 7). But the current implementation of that backend has some design flaws - such as invalidating the whole fast backend when doing a cache write or not using cache tags in the fast backend. Supercache replaces the ChainedFastBackend implementation with one that solves those two issues improving hit rates in the fast backend on systems with lots of writes.
  • Replaces the default cache tag invalidator services (that works directly on the database) for one that leverages a similar concept to the ChainedFastBackend.
  • Introduces the ability for the key value storage to work similarly to the ChainedFastBackend.

To properly leverage what the Supercache module has to offer you should setup support for a centralized caching backend such as Couchbase

By: root Thursday, April 14, 2016 - 00:00

About Drupal Sun

Drupal Sun is an Evolving Web project. It allows you to:

  • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
  • Facet based on tags, author, or feed
  • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
  • View the entire article text inline, or in the context of the site where it was created

See the blog post at Evolving Web

Evolving Web