Feeds

Author

Upgrade Your Drupal Skills

We trained 1,000+ Drupal Developers over the last decade.

See Advanced Courses NAH, I know Enough
Jan 17 2014
Jan 17

In every Drupal project, it is crucial for your application to be fully defined in code at every time and every state. When working with a configuration management based upon Git (www.git-scm.com/), Features (www.drupal.org/project/features), Drush based shell scripts (www.undpaul.de/en/blog/2013/06/26/simplify-your-development-workflow-dr…) and Master (https://www.undpaul.de/en/blog/2013/06/28/introducing-master-drupal), it is possible to represent your whole Drupal 7 application's configuration state in a traceable and reproducible way.

Watch your Drupal modules

This way your team has a good tool to manage the development of the application and even manage the state of the Drupal modules in your application. Especially in large Drupal projects it is always important to know the modules you are dealing with, what modules to enable and to know what modules you can disable. Even already disabled Drupal modules influence your system and the overall development experience of your team.

For example, if you provide a large set of modules in your project, there will be a lot of noise for the developers when working on the sprint issues (e.g. a developer might find search results in the IDE for a code snippet in modules that are not meant to be enabled anymore). Therefore it is important all modules that should not be enabled are really disabled and uninstalled from the Drupal project. And, if possible, after that the modules should be removed from the file disk. As we are (hopefully) always dealing with a version control system like Git, we are safe to remove those modules without losing any information of the project's evolution.

Remove modules with Master

In projects that have grown for several months and no-one had a look at the modules directory, it will be very painful and time intense to remove those obsolete modules. To ease such tasks in the development workflow, we created the Master module some time ago. With the latest release of Master  (www.drupal.org/project/master), we have also introduced a new command to assist you in finding the modules you really don't need anymore in your directory.

With drush master-removables you will get a list of Drupal module directories that can be removed from the project's file system. When combining the command with the remove command, you can delete those modules from the file system:

rm -rf $(drush master-removables --absolute --pipe)

Make sure you checked the output of drush master-removables, especially when dealing with multiple environments (like develop, testing, live), you have to make sure all modules you are about to remove from the filesystem have been properly uninstalled before. For example, in Sprint 11, when your team decides they will not work with the Context module anymore, but it already has been rolled out and enabled in a previous sprint, they cannot simply remove Context from the Git repository in Sprint 11. That's because the module will be needed to successfully fire the uninstall procedure on the production-installation for that module in the next roll-out. So you may only fully delete the module after the roll-out was successful and make that change in the next sprint.

Tip: When you stumble upon such a situation, simply remind yourself for this task in the next sprint by creating a new related issue in the backlog for that upcoming sprint.

Absent modules

In some cases, e.g. when carelessly updating or removing modules, there might be modules missing, although the Drupal installation needs them to be enabled. For example some module was removed although the module was still enabled. This might be a heavy performance impact, as Drupal will scan the module directories several times each Drupal bootstrap. This would be not good. To make sure we do not run into this scenario, the newest Master release also contains a command to list those modules that are not in the installation but should be present. With `drush master-absent` you will get a list of those modules and can fix the problems.

The command will also list those modules that are part of the master configuration or an unmet dependency of the master modules and are not present in the module directory.

Master development

We are curious about your experience with Master or other approaches of managing your modules' state. As we are currently working on a stable release for Drupal 7 your feedback is very welcome in the Master issue queue on www.drupal.org/project/issues/master.

If you need any help in your Drupal development workflow, we will be happy to help with our professional Drupal service.

Jun 28 2013
Jun 28

Whenever working in a Drupal project you have to deal with a bunch of modules - core, contrib, custom and features. In the lifetime of a project you enable new ones and disable and uninstall modules you don't need anymore or replaced with a better one. But how do you track that, and how do you make sure that those modules that should not be active really are not?

I did not find any existing solution in the Drupal contrib modules that deals with this issue, so I wrote a small helper module: Master.

Master Yoda icon designed by Artua

Drupal module whitelists with master

You really should know and should define what modules have to be enabled to fulfill the project's functionality. Those modules that are meant to be active should be defined to be active in a way. With master you got a tool to whitelist those modules that have to be active. We call those modules master modules.

Configuration in settings.php

This is done by simply defining $conf['master_modules'] in your settings.php:

$conf['master_modules'] = array( 'myworld_news', 'myworld_contact', );

Module dependencies

Modules that are not in this whitelist may also be required by those modules. Therefore master also takes care of module dependencies. So if your module myworld_news needs node, features, strongarm and views enabled, you do not have to write those modules in your $conf['master_modules'], you simply place those as dependencies in your myworld_news.info.

name = "MyWorld News" description = "News section for my world" core = "7.x" dependencies[] = features dependencies[] = node dependencies[] = strongarm dependencies[] = views

As you might know, Views has a dependency to the Chaos Tools Suite - it is not listed in myworld_news.info as the dependency is already declared in the Views module. Master is aware of that. Modules that are a direct or indirect dependency of a master module are required modules.

Redundant modules

Modules that are no master modules and have no dependency to one of those master modules are meant to be redundant. They shall not be active in the given site. If you want a redundant module to be meant to be active, you have to define it as master module (add it to settings.php configuration) or add it as a direct or indirect dependency of one of the master modules.

Using master

So how can you use that? After you defined your modules in your settings.php, you can use drush commands to check the status of the module whitelist and to ensure the intended module status.

Status

drush master-status [--status=…] [--pipe] [--sort] [--scope=…] [--skip-scope-validation]

With the master-status command you will get an output of the current state of your master modules and their dependencies. So you can make sure certain modules are active or disabled.

The command lists all modules in specific categories:

  • Master modules: those defined in $conf['master_modules']
  • Required modules: dependencies of the master modules
  • Missing modules: modules that are either master or required, but are not enabled
  • Redundant modules: modules that are not master and not required, but are enabled
  • Uninstall modules: modules that are disabled, not master and not required, but not completely uninstalled

By default the list is printed to the shell in a human readable table with additional information. You can limit the output to specific groups of modules by using --status: e.g. drush master-status --status="master,required" for only master modules and their dependencies. You can use the --pipe option to only return a whitespace delimited list of module names or use the --sort option to sort the modules by name.

The --scope option is for using different sets for different environments. This functionality is described in detail below. By default a scope has to be given, unless you use the --skip-scope-validation option.

Ensure modules

drush master-ensure-modules [--no-disable] [--no-uninstall] [--scope=…] [--skip-scope-validation]

This is the heart of Master. With drush master-ensure-modules you can make sure all modules are enabled, disabled and uninstalled as they are meant to be - in one single command. The command retrieves the information from your master modules configuration. It enables all master and required modules. Additionally it disables and uninstalls all redundant modules.

Especially when using Master the first time, you first should check the status of your modules using drush master-status, because maybe you forgot to put some modules in your configuration or module dependencies. Without any additional option, Master would disable and uninstall all modules that are not defined to be active.

Master will make sure of the order modules are enabled, disabled and uninstalled. Dependency will be enabled first and disabled last. When modules are disabled they are uninstalled before the next module will be disabled (and uninstalled). This way we make sure dependent modules may still be active - just for a reason it might be needed.

Master also makes sure already disabled modules are really uninstalled: in most cases, uninstalling a module means removing tables and variables from the database - this is managed by hook_uninstall() and/or hook_schema().

By using the options --no-disable and --no-uninstall you can avoid disabling and uninstalling modules when using the master-ensure-modules command.

--scope and --skip-scope-validation fulfill the same purpose as in drush master-status.

Different modules on different environments

When you work with different staging environments or even different production environments you may have the need to enable an additional set of modules on these sites. For example you might have a language specific feature for the German website that is not part of the international one, or you simply want to make sure your developers have fieldui_ and devel enabled on the local development environment.

For that use case you can define scope specific additions to the global master module configuration. You will also do that by defining a variable in your settings.php. The variable is called $conf['master_modules:SCOPE'], where SCOPE is the name of the scope (e.g. local or live-de).

$conf['master_modules:local'] = array( 'devel', 'field_ui', 'stage_file_proxy', 'views_ui', ); $conf['master_modules:live-de'] = array( 'myworld_de', );

The listed modules will be merged with the modules defined in $conf['master_modules'] and therefore build a bigger set of master modules.

Note: Instead of listing additional separate modules it might be better to add a "dev-Feature", so you can provide additional configuration too (e.g. for devel).

When using the drush commands, you simply append a --scope=local to the command and the given set will be used:

drush master-status --scope=local drush master-ensure-modules --scope=local

Scope validation

As any accidental deactivation or uninstallation of a module might solve data loss, I implemented "scope validation" for any command. This means that a scope has to be defined for the command being executed.

You define a scope by simple setting an array (at least an empty one) for the scope specific module list:

$conf['master_modules:live'] = array();

When you do enter a scope that is not defined that way or enter no scope, the command will fail with an error The given scope "" is not valid!. This might happen if you accidentally type in drush master-ensure-modules --scope=locla or drush master-status. As said, the error is also triggered, when you enter no scope, so you do not accidently forget to enter your scope name.

You can disable scope-validation by using the --skip-scope-validation option in your drush commands. That way we make sure the user decides to do so.

Do not uninstall

In some cases it might be necessary to not completely uninstall the modules that shall be disabled. That might be the case when you only want to temporarily disable a module and leave content intact for the moment - e.g. you disabled the forum or just want to disable the migrate module for now but keep the migrate mapping tables.

For that case, you can fill your master_uninstall_blacklist in the settings.php with modules that shall not be uninstalled by Master when they are about to be.

$conf['master_uninstall_blacklist'] = array( 'myworld_module', ); $conf['master_uninstall_blacklist:live] = array( 'migrate', );

As you see, there is also a scope specific setting, using master_uninstall_blacklist:SCOPE.

Note: This functionality is no replacement for doing a database backups before changing your code. Be sure to make a backup before you use Master!

How we use it.

We at undpaul simply put the drush commands in one of our update.sh files:

040_modules.sh

################################################################################ # Enable our modules and features. ################################################################################ # We use the master module for controlling the module status. # @see http://drupal.org/project/master drush $DRUSH_PARAMS --yes pm-enable master # And finally execute master. drush $DRUSH_PARAMS --yes master-ensure-modules --scope=$STAGE_INDICATOR

This is located right before reverting features, so we do not revert old features.

What it (currently) does not do.

Currently Master does not have any UI for the admin backend. Everyone is welcome to contribute to a master_ui, but in our workflow with automated drush deployment scripts, we simply did not need it by now.

Master does not manage any dependencies by downloading missing components from drupal.org or any third party site. I guess this would be some part drush make might be good at.

Additionally I had in mind to provide a command that will delete all of the module projects, that are not in use in the current site, so we do not clutter up our modules directory - especially when we use a VCS.

I'm curious about your comments and maybe some contributions. The project is available on d.o. You're invited to p

Jan 10 2012
Jan 10

Setting up HTTPS configuration is easy ...

To build additional security on special pages on a website (e.g. a user form), the module Secure Pages has proved its worth. With it, you simply configure a domain within Drupal to be used for the secure connection, (provided that the server is configured correctly). With additional settings, you can specify several pages to be secured.

... and a mobile domain too, ...

To provide your page with a special theme for mobile devices, the module Mobile Tools is a good solution. You simply enter your mobile domain in the configuration and mobile devices are redirected to that page, with the theme switched to the selected one.

... but they don't want to work together!

But now, if you want to benefit from both modules, it comes to a small desaster, because we also have to care about mobile secure pages. Especially if a module like GlobalRedirect is activated, trouble occurs.

In such a situation, it will hapen that - on a mobile device - you get redirected from the desktop url to the mobile domain. But there securepages recognizes that the given page shall be delivered via https. So it redirects you to the configured https domain, which is a desktop url. But there mobile_tools wants us to go to the mobile domain. With all these redirects we end up with an error message like: Error 310 (net::ERR_TOO_MANY_REDIRECTS): There were too many redirects.. The settings caused a circular reference on the redirects.

In an example, we have the domain http://www.example.com and its secure page https://www.example.com, the domain configured in securepages. In the moment we set up the mobile site http://m.example.com, in most cases we also need the mobile https equivalent https://m.example.com. But neither Mobile Tools nor SecurePages offer us these settings - at least not in the configration forms.

We need a tamer!

This tamer we find  inside settings.php. Because as long as the configuration variables aren't stored in the database (more precisely: in the table {variable}), we simply can set them up within settings.php.

To avoid storing these values in the database I wrote a little sandbox and published it on http://drupal.org/sandbox/derhasi/1399172 (currently only for Drupal 6). It will delete the special variables from the database, after the Mobile Tools or SecurePages configuration form was sent.

But now let's take a look at the snippet for settings.php, with which we avoid the circular reference:

// Let mobile tools work with secure pages.

// We have to set the base_url too, so e.g. globalredirect

// will not force a wrong redirect.

$desktop_url = 'www.example.com';

$mobile_url = 'm.example.com';

$secure = (isset($_SERVER['HTTPS']) && $_SERVER['HTTPS'] == 'on');

// On mobile requests, mobile path provide the secure settings.

if ($_SERVER['HTTP_HOST'] == $mobile_url) {

  $conf['securepages_basepath'] = "http://$mobile_url";

  $conf['securepages_basepath_ssl'] = "https://$mobile_url";

  $base_url = ($secure) ? "https://$mobile_url" : "http://$mobile_url";

}

else {

  $conf['securepages_basepath'] = "http://$desktop_url";

  $conf['securepages_basepath_ssl'] = "https://$desktop_url";

  $base_url = ($secure) ? "https://$desktop_url" : "http://$desktop_url";

}

// On secure pages, mobile and desktop urls are https urls.

if ($secure) {

  $conf['mobile_tools_mobile_url'] = "https://$mobile_url";

  $conf['mobile_tools_desktop_url'] = "https://$desktop_url";

}

else {

  $conf['mobile_tools_mobile_url'] = "http://$mobile_url";

  $conf['mobile_tools_desktop_url'] = "http://$desktop_url";

}

With this configuration, SecurePages and Mobile Tools now react on the given circumstances (https or mobile) on each request, and set the correct base and configuration domains.

The snippet is available in default.settings.inc of the repository.

Extra hint

In addition to the MobileTools and SecurePages configuration, you also need to keep an eye on your custom redirects and rewrites. For example, if you let .htaccess redirect www.example.com to example.com, this will lead to a circular reference of redirects too (with the above settings).

Dec 21 2011
Dec 21

Who doesn't know the following scenario: one is on vacation, maybe on a farm. Letting it all go easy, the sun is shining on your head, cowbells are ringing all around, everything feels just wonderful.
And then ... there comes the inevitable call from a colleague. "Johannes, could you have a look at this thing on site xy and just fix it. We need it till tomorrow." I won't go into if that is a healthy idea on a vacation, but normally I'll dive headlong into the problem. Of course I took my laptop with me just in case something would come up ...

Creating a local development copy "in a minute"

With Git code is set up quickly. In many cases the problem is better testable with actual content of the site, so I also want that. For that task a database dump with Backup-Migrate is handy (configuration is of course bundled into features). Pulling the database is a quick job, as it normally makes up for a file size of a few megabytes. Now we basically got all the content, as far as it consists of text.

But our site also has a lot of images and preview images in views. To get that, the traditional way would be copying the entire sites/default/files-directory from the server (e.g. using FTP). With some projects this can amount to several gigabytes - so this can take long. But we only wanted to perform a quick fix on the site.

Stage File Proxy - in a minute, for real!

To do the sitebuilding work, we only need a few images that are in the subset of content we are working on locally. So it's a mess do really download all the files. This is where Stage File Proxy comes in.

The module saves the work of pulling all the images manually. Instead, with each request the local dev copy checks if it already has the desired file in the local file system. If not, it looks it up on the remote server and downloads it. For that, the module only needs to be installed on your local machine, not the remote one.

Setup is easy:

  1. Download the module and put it into the modules folder (drush dl stage_file_proxy)
  2. Activate the module (drush en stage_file_proxy)
  3. in settings.php enter the remote url: $conf['stage_file_proxy_origin'] = 'http://my-great-domain.com';

Of course the process is also documented in the INSTALL.txt of the module.

Not much to do after that - given the write permissions in the local installation for the files-directory are set correctly.

Stage File Proxy is also taking care of the Imagecache styles. Instead of downloading the derivates created by imagecache, it downloads the respective source iamge. So the derivates can be created locally. If you do not want that, switch off this behaviour by writing into settings.php $conf["stage_file_proxy_use_imagecache_root"] = FALSE;

By setting $conf['stage_file_proxy_hotlink'] = TRUE; you even can avoid downloading files completely. Instead the file requested will be served as a redirect 301 to the fle on the (remote) server.

If you placed your origin installation behind an access authentication (via .htaccess/.htpasswd), you simply have to add your authentication data as domain prefix like that: $conf['stage_file_proxy_origin'] = 'http://username:password@ my-great-domain.com';.

Wrap-up

Okay, the scenario above won't hit you that often on vacation, but almost every day at work if you are a developer. Since the Drupal 7 version of Stage File Proxy is out, I use it on almost every local site. Especially for working in a team it is great. You do not have to pass around big file dumps or tell everyone to get the files via FTP - the module does it by itself, nice and quickly. On a staging site before the go live, it also saves you a lot of time of tedious file copying.

So far I have only had one case where it did not work. We had a site with a lot of videos that were just too big to be downloaded by PHP. But maybe one of you readers already has a solution for that in mind.

You find the module - of course - on drupal.org http://drupal.org/project/stage_file_proxy, there is a dev version for Drupal 6 and Drupal 7. A big thanks to netaustin and robw for their great work!

Dec 12 2011
Dec 12

I guess a lot of people that have to deal with integrating videos on a website for iPhone and iPad, will  - at some point - run into the problem that some videos simply won't play. In most cases embedding the video is the simple part: either we use the fantastic HTML5 specifcation or use some weird snippet for loading a Flash Video Player. But I don't want to look at that in this post, I want to dig a little deeper into the encoding problematics. But this is not about dealing with whether to use OGG TheoraVP8 or h264, or how to use them all at once. In this case I stumbled upon an iOS specific problem. So h264 would be the codec to look at, as this is the favourite on Apple devices.

Differences for iOS devices

One might think it would be enough to simply convert any video to h264 format for it to run on iPhone and iPad. But it isn't that easy: when a video plays on an iPad, this does not mean it will play on an iPhone (in my case it was an iPhone 3GS). Instead of playing the video Safari would instead show a crossed out play button, with the Safari Debug Console displaying a "very helpful" message:

Quicktime: Movie could not be played.

The iPad played the same video just fine.

After some research on the web I came to the conclusion that it's not enough to choose the right codec. To a greater degree, you have to keep an eye on different parameters (as bitrate for audio and video). Sure, on other devices (not just Apple's) you have to deal with that too. A blog entry at Zencoder tries to summarize the settings for different devices.

Sure, you now could try to find the correct settings by trial and error. The iOS Simulator of XCode might help in that situation, as there the same problem occurs on the simulated iPhone, but not on the iPad. So you can debug on your local machine.

Use the Handbrake to quickly stop

With the hints from the zencode post I tried may way on the road, but of course the first attempt failed. The video conversion tool of my choice "Handbrake" (alternatives) luckily had a preset that would just fit the needs for Apple devices. So I tried it again with that preset - named "Apple - Universal", and, lo and behold, it worked. The video just played on both devices. I finally found the right conversion tool and settings.

Impacts your site

For site adminstrators this would mean to convert all videos they ever uploaded to the new format. This really only might be a viable solution for smaller sites, but in the current case it was fair enough to make the most important videos compliant for the iPhone.

Other solutions

Sure, converting the videos on your local machine is not feasible for every website. But it saves the need for server side processing (like ffmpeg) or implementing a third part tool (like Zencode). If you want to provide different sizes for different devices, the local conversion is no pragmatic solution. You should definitively grab an ffmpeg module or use a SaaS for video conversion for that use case.

Icons by: Dirceu Veiga

Nov 08 2010
Nov 08

On Oct 16/17th we held the Drupal 7 Sprintcamp in Hannover. We met with 40 Drupal Workers of all flavors to give a boost to Drupal 7. Beyond the main focus on D7 we also dived headlong into more general Drupal stuff.

Our beautiful venue was the Coworking Space Hannover. On three floors we had plenty of space. We worked on four topic groups: Coding, Documentation, Marketing and Translation. The official organizer was the Drupal Initiative Germany, which is our local mini-version of the Drupal Association. undpaul did all the practical planning and organizing. We managed to attract quite some sponsors, and thanks to them we could keep the camp completely free for attendants, including drinks, food and a saturday night event in a nearby pub. A big thank you to the sponsors!

We chose coordinators for each topic group to have better organization and to have someone to talk to for the sprinters at all times if they ran into problems or out of work.

Coding and translation

Stefan Borchert (stborchert) was responsible for Coding. The biggest chunk worked on was Views, mostly Views for Drupal 7, of course. As the co-maintainer of Views, Daniel Wehner (dereine) was there, we had fixed over 20 Views issues by the end of the weekend. Karsten Frohwein (Kars-T) did a little introduction to Simpletest so with the help of thus newly-trained people we were able to write some new tests for Views.

Besides Views we worked on some other modules:  Login Destination to Drupal 7 (though the patch is still pending for acceptance).  ThemeKey, which has an Alpha 2 now, and we made some serious progress on Content Access and Scheduler.

To sum up we found out that writing tests gets too little attention; especially for Views a lot of tests are missing. This includes writing Simpletests, but also just manually trying out Views and submitting patches. So anyone who is not a coder but wants to have Views ready with the Drupal 7 release: hop in and go for it. Install Drupal 7 with Views 7.x-3.x, create some content and a few Views, hit a wall and file some issues. This is much more valuable than you might think.

The Translation team used our great existing infrastructure on localize.drupal.org to translate Drupal 7 core and quite a few modules like Admin, Context, Pathauto, Features, Strongarm and more. The team was led by Anja Schirwinski (aschiwi), and besides translating they tested beta.drupal.org and helped improve http://drupal.org/features as well as the new website for the Drupal Initiative, which needed a new design, better structure and some text.

Documentation and Marketing

Floh Klare (SirFiChi) led the Documentation team that chose a very thorough approach with the status of the German documentation, which is hosted on drupalcenter.de. Not pleased with the lack of structure, they started over on tddp.de and rebuilt the thing from the ground up. Starting with a well thought out structure, most of the content is still lacking. But the plan is to migrate all the documentation content over time from drupalcenter.de, which was started on the weekend, as well as creating some great new docs. The main features are advanced filtering, which are found in the doc finder. Instead of relying on building the classical tree structure the team focussed on a user that would be likely to use search instead.

But it was not only the German docs that got some boost, also the English docs for Views 3 were extended, which went directly into the advanced help section of the module.

The Marketing team, led by Thomas Moseler (eigentor), faced a bit of a blank slate: we do not have any marketing for Drupal, especially in Germany. But especially over here it is badly needed, as Drupal is by far not as popular here as in the US. The strategy for the weekend was twofold: on one hand a presentation site, that is to be imagined like an extended drupal.com, that was continued to build. The idea is not new and Sebastian Daniel (s.Daniel) has already created a great design and thought out a basic structure. The site will serve only marketing and information purposes mostly for Drupal newcomers.

Similarly to Acquia's home page, this will be divided by four target audiences: decision maker, designer, coder and end user. These four personas also were the basis for the structuring of content that is going to be put on that site. A smaller group worked on creating and structuring the site, while the major part split into subgroups to collect the major advantages that could convince each of the personas to choose Drupal over another CMS.

There were lots and lots of discussions, sheets and whiteboards full of brainstorming results. In the end we managed to prioritize the results while not being finished. But we could be sure to have thought of really a lot, which is to be found in the sprintcamp forums, which document the progress of the other groups as well. What the marketing group must do now is still better prioritize the results, so we can put them on the presentation site that will hopefully launch in late 2010 or early 2011.

A busy and intense weekend, but we still managed to have nice evening events in the nearby pub "Spandau". The cosy and pleasant Coworking Space helped us enjoy it and not to get too much into a workday mood. From our side - we will definitely do it again :)

About Drupal Sun

Drupal Sun is an Evolving Web project. It allows you to:

  • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
  • Facet based on tags, author, or feed
  • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
  • View the entire article text inline, or in the context of the site where it was created

See the blog post at Evolving Web

Evolving Web