Upgrade Your Drupal Skills

We trained 1,000+ Drupal Developers over the last decade.

See Advanced Courses NAH, I know Enough
Jul 17 2012
rmo
Jul 17

Hello, Shadow is here!

Posted on: Tuesday, July 17th 2012 by Richard Mo

Have you ever felt that doing front-end web development for mobile web devices (i.e. Android, iOS) is a real pain, especially when you can’t do any inspections quickly? Designers or Themers would totally agree with me on this. Web developers, in general, would feel the same way but not as much as designers/themers do. And how about QA in general – manually pressing links or refreshing pages? It’s a lengthy process… Now, may I suggest to you that it is possible to shadow your actions across multiple devices?

I’ve recently been introduced to Adobe Shadow which addresses all the pain I mentioned earlier. A quote from Adobe Labs:

“Adobe® Shadow is a new inspection and preview tool that allows front-end web developers and designers to work faster and more efficiently by streamlining the preview process, making it easier to customize websites for mobile devices.”

Basically, it allows you to sync all URL changes to every shadowed device and inspect HTML and CSS on each and every one of them.

Here are some features Adobe® Shadow provides:

  • Synchronized Browsing and Refreshing (Device Pairing) — Wirelessly pair multiple iOS and Android devices to your computer.
  • Remote Inspection —Target a device for remote inspection and using familiar development tools, make changes to your HTML, CSS and JavaScript and see your device update instantly.
  • URL Monitoring — Shadow monitors the Address Bar in Chrome, and sends updates to Shadow devices as they happen.

Setting the system up isn’t hard at all; there are, however, 3 things you must download to get things up and running:

  1. You need to download the Adobe® Shadow service which installs onto your desktop. It acts as a server then communicates with all your devices.
  2. Download and install the Chrome browser extension which you would use to control your Shadow devices.
  3. Finally, each device (Android or iOS) must install their respective Shadow app.

Combining all 3 pieces of the puzzle, hook them up, and there you go!

Adobe® Shadow is excellent for traditional websites, where every page is uniquely named (i.e. each page has a unique URL), because Shadow devices follow URL paths sent by the host and interact with the native browser. Most AJAX elements on the page, however, will not be compatible with Shadow for that reason. Web applications built purely using JavaScript, Sproutcore per se, will not work.

Then, what would Adobe® Shadow be used for? Drupal! Shadow is excellent for Drupal front-end development! It really speeds things up for designers and themers. The built-in HTML CSS inspector will let you see what’s going on with each device and you can take screenshots for all of them at once!

For further information, please visit Adobe Shadow.

Jun 20 2012
Tom
Jun 20

In February 2012 our team at ConsumerSearch launched the ConsumerSearch Reviews iOS app. This handsome app helps you during your product purchase by providing extensive product reviews, comparisons, pricing information and online and local store product availability. Try it out during your next purchase -- you'll be surprised how much time you'll save in researching the best products.

Searching for products is possible by keyword search or barcode scanner. We are very proud of our first app that has already gone through several updates and has received positive user reviews.

cs-iphone-screen-1cs-iphone-screen-1
cs-iphone-screen-2

For a development team that historically had a huge focus on PHP and Drupal development, big technology changes were introduced that took us out of our comfort zone.
Of course we all know about the latest technologies, however making production ready code in a short amount of time is always a risk and a challenge (that we welcome :)).

I feel that many (Drupal) development teams are facing similar questions when moving to new technologies, so I hope this use case can give developers familiar with PHP and Drupal some quick guidance or fresh ideas. Feel free to ask me questions or provide feedback in the comments section.

ConsumerSearch is powered by Drupal

It's no secret that ConsumerSearch has been one of the earliest examples of highly trafficked Drupal sites since it switched to Drupal in 2008.

We manage to serve our huge traffic load by relying HEAVILY on caching (browser, Drupal, memcache, and Akamai).
This reliance on heavy caching in order to deliver super fast performance is one of the weaker points of our (and many other) websites, especially in a world where we aim for real time and personalized experiences.

While creating the architecture and technical specification for our API backend we realized that some drastic changes had to be introduced in order to create a highly performant and flexible API that could serve as a data source for the mobile application and for additional users.

We eventually settled on the following technology:

  • Drupal as our CMS which handles all the content creation. Drupal pushes updated and new content to MongoDB and Solr.
  • MongoDB as our intermediate storage layer to where we push all Drupal content in a heavily denormalized way. The API application layer queries MongoDB directly.
  • Acquia Search powered by Solr as the search index. This powers search related API calls.
  • Node.js as the API layer. We use Pronto.js for our lightweight, high-performance node.js framework.

This gives us a nice open source stack that plays very well together in creating a flexible and highly performant content delivery system.

CS API overview

As a side note, our team had limited to no experience with Node.js and MongoDB before working on the iOS app. Even today these technologies only account for a very small part of our day to day business. However both solutions proved to be relative easy to learn and master.

Speeding up and choosing the right technologies

We've been using Drupal 6 for the past couple of years and benefitted greatly from the flexibility of its content model (CKK), its presentation possibilities (views / templating engine) and the huge amount of contributed modules available (too many to name). However, where these tools have given us incredible speed during initial construction of the site, we have gradually experienced some of the unfortunate drawbacks of Drupal (which are also present in many other Content Management Systems). Some of these drawbacks include:

  • Speed and performance: We need to rely heavily on caching and scale out our web server infrastructure in order to provide speedy pages. Part of the speed problem is the heavy customization of our website (our fault), but also the nature of the Drupal architecture.
  • Module compatibility: As an example, we have a robust deploy process that depends on the deploy module. However, this module is not compatible with the new and recommended Services 3.x module.
  • Module scope: Very often, Drupal modules try to do too much. I have also been guilty of this while, for example, working on the first versions of the Mobile Tools module. If you want fast and lightweight, you want focused modules. Often you have to write the functionality yourself or find a different module/solution.

Eventually our research on using Drupal as a fast and lightweight API layer came down to the fact that Drupal was just not lightweight, fast or simple enough. It can, of course, be tuned so that it is some or even all of those things, but why choose the hard way when other technologies can give you those features out of the box? We value simplicity and focus within our development team and projects, but Drupal, unfortunately, is just not always simple or focused.

With speed, flexibility, and simplicity as our goals we made the following assumptions:

  • Storing our data in a denormalized fashion should speed up the the retrieval process.
  • Storing the data schema-less should result in shorter development time.
  • Using very dedicated code should make the application lightweight and fast.

Further, the API would have the following characteristics:

  • The database should store simple documents with no revisioning (Drupal keeps track of that) since our content doesn't frequently change.
  • The API should focus on getting data from the database -- no write operations needed.
  • The data should be replicated across multiple data centers.
  • The architecture should easily scale out across multiple data centers.

Denormalization of the data

Loading content stored in Drupal can be very DB intensive due to the underlying content storage model. Data is scattered in many tables, and when you rely on node references you end up making a lot of DB calls. So we decided to preprocess our data and store complete data objects in a denormalized way so retrieval becomes easy and fast.
For this we wrote a custom module, loosely based on the Apache Solr Drupal module that publishes some of our content types in a denormalized way in MongoDB. Some of the transformations we performed include:

  • removing fields that were only used internally or specific to Drupal.
  • giving fields Drupal-agnostic names.
  • collapsing some of the CCK arrays (e.g. "field_name[0][value] into "field_name[0]")
  • other content tweaks that made the resulting data object easy to understand.

We ended up with a series of collections for our primary objects: reports, products, channels, authors, etc.

MongoDB as a document store

It did not take much research to determine that MongoDB was an ideal choice as the database to store our denormalized data. Some of our key reasons for choosing MongoDB were determined by the fact that:

  • JSON is the native format to store and retrieve data. JSON is currently the DE FACTO standard for web APIs.
  • as a document store, it is natural for MongoDB to store denormalized nodes.
  • querying MongoDB is easy and intuitive. Although different than SQL it is very easy to dynamically create queries.
  • MongoDB has good driver support in many different languages and inv.
  • it provides easy replication using Replication Sets, resulting in read scalability and fail over.
  • reads and writes are scalable through Sharding.
  • it's lightning fast
  • and on and on.

While researching other storage databases, we learned that MongoDB isn't perfect at everything, but it is more than good enough for our needs. Some of its missing features include:

  • key-value stores like Cassandra and Reddis can reach much higher speeds.
  • no master-master replication. In our case that would mean having the possibility to have two active master data centers.
  • no document versioning like in CouchDB.

For each content type that we needed access to in our API we created a MongoDB Collection. We also setup a replication set across our our two data centers with an additional arbiter in a third datacenter.

The replication set gives us read operations across all the machines in the set, while writes still go to the primary database. In order to scale writing, sharding is the recommended technique.

Node.js as the API application layer

While it was clear that Drupal would never give us the required speed for our API (unless relying 100% on cached responses) and that MongoDB was an easy scalable solution to store our data, heavy debating occurred in choosing the application layer for our API. Would we stick with our familiar PHP environment, or would we explore other ways (Ruby, Node.js, etc.)?

The main high-level requirements for the API in our case were:

  • simple routing of API requests; no templating, CMS functionality, or heavy administration.
  • the ability to handle many I/O operations, including connecting to MongoDB and various third-party data providers.
  • the need to minimize dependencies on the content model in Drupal. For example, adding a field to a Drupal node should not result in changes to the API.
  • the ability to handle JSON well.
  • it needed to be as fast as possible.
  • it must be easy to learn, or at least minimize the learning curve for the team.

We narrowed the list down to PHP (we all know that one very well) and Node.js (new kid on the block, we heard many great things about it, etc.).
Node.js has been shown to be really fast thanks to the V8 JavaScript engine also used in Chrome. However, our researched showed that Node.js is not always the best solution for every problem.

Many of the Node.js related discussion in our team (and elsewhere) focused on the "blocking" versus "non-blocking" nature of Node.js and on its internals. I imagine that this discussion will be one of the most popular ones within teams that are new to Node.js. I'm going to try and summarize these discussions using the following examples:

1. Node.js

Node.js uses an event-driven, non-blocking I/O model. This means that I/O bound operations such as external connections, accessing a database and file operations do not block the execution of the main thread. Instead they are executed by a low-level Node.js thread pool and when completed are queued back in the main thread for further handling.
On top of that Node.js runs by default in one thread on the server that sequentially handles all incoming requests, returning callbacks and non I/O related processing logic. An example would be an algorithmic complex calculation (like a Fibonacci sequence).

In the following diagram you can see what is happening in a typical Node.js application where:

  • there is one main thread
  • there is a low-level (and very lightweight, low-overhead) thread pool that handles the non-blocking I/O operations.
  • incoming requests and finished I/O calls are put on the event queue and await processing by the Node.js main thread.

As a result, processing in the main thread can block your entire application (which is very bad). Luckily, workarounds are available to move complex operations out of the main execution thread.

2. Apache with PHP (Prefork)

In Apache, when using mod_php and assuming you use Prefork MPM (because PHP is not thread safe), every incoming request spins up a new process in a different thread. This means that if one thread hangs, it will not block the other threads. This is one of the strong benefits in using PHP and Apache.
However, I/O bound operations are blocking since the execution of further code waits until the I/O operation has finished (see dashed lines). This waiting reserves system resources that are heavily underused. Spinning up new processes also results in overhead on your system.

In a nutshell, you could say that Apache with PHP could be seen as a safe bet, but not the most performant one.

In the case of the API, we did not have complex calculations and the main part of the execution consisted of connecting to MongoDB for authentication and getting data back. Node.js was clearly a winner.

Pronto.js

For the API we settled on Pronto.js, a great and highly performant Node.js framework developed by long time Drupal developer Matt Butcher while he was a member of our team at ConsumerSearch.
Pronto.js is in many ways similar to the well known Connect framework. Pronto.js provides an easy interface to create chainable commands to execute on a request.

Many tricks are used to minimize the blocking nature of incoming requests. We've since made enhancements to Pronto.js that provide non-blocking execution. I'll cover that in a subsequent blog post if it warrants discussion.

Wrapping it all up

We have been running this setup now for almost 5 months with no problems. Development was fast and lightweight.
We experience super fast response times and very low server load. Benchmarks show that we can easily hold 1000 concurrent connections serving 4000 requests/second on two mid range servers. We actually need to scale up some of our testing infrastructure to explore the real limits :) Try our free app and see for yourself.

Further optimizations we are exploring include:

Please comment to share your experiences with these technologies or if you have any questions or comments. You can also follow me on Twitter: @twom.

Special thanks go out to Treehouse Mobile for collaborating with us for creating the iPhone app, and our development team for suggestions and corrections on this blog post!

View the discussion thread.

Jul 13 2011
Jul 13

Note: This blog post was posted on July 13, 2011. Youtube embeding may have changed since then, they claim to still be fixing it and in beta.

This suggestion is to provide the best user experience when watching a Youtube video on your mobile website. A lot of times it is done incorrectly and users are redirected to either m.youtube.com or it opens up your phone's native Youtube application, which basically minimizes your browser. This pushes users away from your site and basically requires users to hit the back button to get back to your site.

read more

Mar 26 2011
Mar 26
Drupad App and Module

I recently downloaded the Drupad module and purchased the Drupad App from the iTunes store. The two have changed the speed at which I can monitor, control, and promote my, and my client's Drupal sites. The module is a companion to the iPhone app (cost $4.99). The module was easy to install and configure the features you want your site to expose to the application. For example, you can select to manage comments, content, users, and more. Additionally you can choose which content types to exclude such as pages or Weblinks. You can even run cron with this handy set of tools. I highly recommend site users invest in the Drupad App and Drupad module.

Anyone else have experience with this module and App?

Aug 24 2009
Aug 24

Drupal 6 JavaScript and jQuery CoverI just finished reading a new book, Drupal 6 JavaScript and jQuery, by Matt Butcher. The book title makes it sound highly specialized, but in fact it's a great resource for a variety of readers, from Drupal beginners to JavaScript experts. Best of all, the book brings together two of my favorite open source technologies: Drupal and jQuery. If you aren't already a fan, I've written elsewhere about Drupal's benefits, and for jQuery, one statement should win you over: query HTML content using plain old CSS selectors!

Matt does a great job leading the reader from very basic principles to advanced concepts, thoroughly enough to initiate less experienced coders, but quickly enough to get everyone to real meat right away. You will get immediate value from this book whether you are a web designer just starting out with Drupal and/or JavaScript, an intermediate coder looking to expand your skills with either Drupal or JavaScript, or an advanced Drupalista or JavaScriptor looking to bring the two together.

Because I reach under the Drupal hood only sporadically, I love how how Matt quickly covers all the basic terminology and functionality of Drupal, JavaScript, and jQuery to remind me how they all work, and work together. I can see turning to this book as a basic reference again and again.

Best of all, in each chapter Matt provides a hands-on project worth doing for it's own sake as well as for the added learning. Trying it out, I modified the rotating sticky node teasers project in Chapter 3 to complete something I've been wanting to do here on my blog for some time: make my Flickr-stream header images rotate dynamically. Read more about exactly how I did it. If I can do something like this in just a few minutes, that tells you a lot about the power and simplicity of Drupal and jQuery together, and Matt's ability to make it all understandable.

Ready to dip your toes or delve deeper into how jQuery let's you "write less, do more" with Drupal? You can buy Matt's Drupal 6 JavaScript and jQuery book directly from the Packt website.

FYI: I drafted this entire review sitting on an Oregon coast beach using the Evernote iPhone app. Pretty nice working conditions ;)

Jul 21 2009
Jul 21

I'm a bit late in linking to a podcast interview that I had the pleasure of being on recently.

DrupalEasy, an organization focused on training and consulting around making Drupal easier to use, runs a podcast interviewing various folks in the Drupal community. My interview was about my work on Drupal, my business in general, Cocoa desktop and iPhone work, my Drupalcon presentation, my career history (in brief), and a lot more.

I had a great time being interviewed by Ryan and I hope you'll take a listen!

http://codesorcery.net/trackback/235

Mobile Theme 1.0 Released

May 11 2009
May 11
May 11 2009
May 11
If we are to believe in timestamps, on October 7th, 2006, I took over maintainership of the Mobile theme for the Drupal CMS. At the time there was no iPhone, and stripped down graphics-free versions of websites made it easier for people with small screens on their phones to get to the information quickly. Now, relatively larger screens coupled with effectively unlimited data plans make websites more consumable by tiny devices. My maintainership of the theme continues unabated, as today I (finally) released official 1.0 versions for Drupal 5 and Drupal 6, and dropped official support for the 4.7 version. m.justagwailo.com serves as the Mobile theme's demonstration site. Thanks to Bèr Kessels for originally writing the theme and webschuur.com for its original sponsorship. A note about version numbers: when the new system for CVS tagging came out way back when, somehow it occurred to me that "DRUPAL-6-4" was the correct version number to assign to a developmental release. What that meant in practices was the version number for the theme ended up as "4.x". Looking at the usage statistics, at this writing, the overwhelming majority of sites that have deployed the Mobile theme use that developmental release. It is my hope that they all move to the 1.x branch, either developmental or official, as that's where all development will happen from now on. As with any software, there are feature requests and bug reports, and I encourage anybody using the theme to give me feedback there. The theme is not to be confused with the Mobile Theme module, which cleverly detects whether a browser is a mobile device and serves up a "mobile" theme for that device. I have the combination of the Mobile Theme module and one of two iPhone-friendly themes at PDXphiles, my Portland-lovers blog.
Dec 31 2008
IO1
Dec 31

This is just a handy tip for anyone beginning to develop or test a Drupal IPhone theme when they don't have an IPhone and are not using a mac.

The first thing you need to do is add the User Agent Switcher FireFox addon - https://addons.mozilla.org/en-US/firefox/addon/59.

This module allows you to impersonate any browser when viewing a site (obviously this has lots of uses when doing drupal seo as well).  It will take a browser restart to get it working.

Then add in a new user agent using the user agent details below for the iPhone:

  • Description: iPhone
  • User Agent: Mozilla/5.0 (iPhone; U; CPU like Mac OS X; en)
  • App Name: AppleWebKit/420+ (KHTML, like Gecko)
  • App Version: Version/3.0
  • Platform: Mobile/1A542a Safari/419.3

In your FireFox tools menu, go to "user agent switcher" and choose iPhone (or whatever you named the user agent).

Then browse to http://www.testiphone.com/ and put in the URL you need to test. It should now show up as it will on an iphone.

Hope thats useful to someone.

Bookmark/Search this post with

Jul 15 2008
Jul 15

Our second episode features Nate Angell or @xolotl, a name that's hard to remember completely, but one that we will always remember. There's development talk later as he demonstrates the shiny, new iPhone app called iToony.

But first, we chatted about Drupal for large organizations and the relative livability of various European cities—all accompanied by extremely loud (but pleasant) French songs, one of which may or may not be a rendition of Cole Porter's Night And Day.

The image Nate made of Bram Pitoyo using the iToony app is here.

Just an iPhone

Jul 11 2008
Jul 11
May 19 2008
May 19

As more people use internet-enabled mobile phones, like the iPhone, the demand for mobile-enabled websites grows. Everyone likes to have their own information in their own hands, all the time. Creating these different interfaces can be a lot of work, but Joe Hewitt changed that on the iPhone front. The iUI User Interface Library was spawned and has popped up all over the place since (most notably on Facebook). The library just requires some simple HTML to create a very slick interface that mimics the iPhone interface.

Where was Drupal in all this though? There are a number of solutions for mobile-enabled Drupal websites, but none of them worked so well with the iPhone. So, I hacked together a iUI Drupal Theme, stuck it on Rob Loach .Mobi, and was surprised with the result. Go ahead, try it out! Aside from a few little things, it worked perfectly on the iPhone. Still needs lots of work, though.

So, if you want an iPhone-enabled version of your Drupal website, all you have to do is install the iUI Drupal Theme, and then switch to the iUI theme when the user visits your mobile-enabled website. You could do that by either a small hack in settings.php, or one of the many modules out there that changes the theme based on the user agent or the domain. Enjoy!

About Drupal Sun

Drupal Sun is an Evolving Web project. It allows you to:

  • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
  • Facet based on tags, author, or feed
  • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
  • View the entire article text inline, or in the context of the site where it was created

See the blog post at Evolving Web

Evolving Web