Aug 17 2018
Aug 17

In my previous blog post on managing microsites with Drupal 8 I promised to write something further and fuller about designing web APIs. This is less directly about Drupal 8, but I will comment on how to implement the recommendations here in Drupal 8.

These are the things that I take time to think about when building a web API.

Design the thing

As a developer, it’s all too easy, and too tempting, to just jump right into coding something. It’s certainly a weakness I suffer from and that I have to cope with.

Before putting the proverbial pen to paper, though, it’s really important to understand why we’re building an API in the first place. What are the problems we’re trying to solve? What do the users need or want?

With regard to building an API, that means thinking about the consumers of the data provided by your API. If you’re building a decoupled CMS, the main user is the frontend system. In other circumstances it may also mean other websites, embedded widgets, apps on mobile devices, and so on. Whatever it is, due consideration needs to be given to the needs of those consumers.

That means understanding your user’s needs, examining the patterns of behaviour of those users, and ultimately translating those into a design.

Sound like familiar language? Yes, that’s the language of visual designers and user experience specialists. In my books, I’d suggest that means you would do well to work closely with specialist design colleagues when designing and building an API.

Your web API needs to be designed: needs; behaviours; analysis; patterns; traits; design; feedback; improve.

Be an artisan with your API

Take time. Research. Think. Plan. Design.

Beware, Drupal

When you’re working with Drupal, it is too easy to jump over the design step. Drupal does so much out of the box that it’s too easy to start coding without thinking properly about what we’re coding.

The availability bias when you’re a specialist Drupal developer, having it as the go-to toolkit, is that we think about the solutions to the problems (if we’ve even got as far as articulating the problems) in a Drupally way. For instance, since Drupal has a menu system it’s easy to think about navigation in a decoupled CMS system in terms of the way Drupal handles the menu system, which prevents you from thinking about other ways of handling navigation.

The same is true with Drupal 8’s support for REST. Drupal 8 core includes REST resources for most entities in a Drupal installation. That’s very useful. But, it can also make you lazy, just using these core RESTful API endpoints for nodes or comments or whatever, with all the guff they include, without even thinking about whether they’re appropriate, whether all the guff they include is appropriate, whether it’s useful or formatted appropriately.

That goes also for REST exports from Views. They can be useful, giving you a quick way of creating a RESTful API endpoint. The problem is, thought, that also confines you to working with the way Views works and what it can produce. You may find that a problem if you want to support optionally requesting for additional objects to be embedded in the response, for instance (see below).

Resist the temptation! Instead, take the time to think from the other end first.

I’ll return to the question of designing your API below, but first we need to talk about documentation, since designing and documenting your API can be part of the same process.

Documentation

Yeah, I know. Most devs find this just the dullest thing in the world to write. With a web API, though, it’s incredibly important. If you want people to actually be able to use your API, they need to know how to work with it. It’s horrible trying to work with an undocumented or under-documented API.

So, what should go into the documentation for a web API? Here’s some pointers.

The basics:

API reference

Yeah, this is probably what everyone thinks of when they think of documentation for a web API, but it is in fact only part of the documentation—maybe the most important part, but only part.

There a plenty of good blog posts and descriptions of what your API reference should include, so there’s no need for me to reiterate that here.

The most important thing to say, though, is that, beyond identifying resource paths, actions and parameters, your reference should describe in full both what the request should and the response will look like.

Mock server

It is incredibly helpful to include a mock server with your API documentation. Preferably, your mock server will handle the documented requests and responses of each resource.

This will help those building apps and tools that will consume your API to get up-and-running quickly.

For gold stars and a round of applause:

Tutorials, guides, cookbooks

If your API gets to be any substantial scale then the developers who use your API will find it incredibly useful to have some tutorials and guides included in your documentation.

These should cover common tasks, or how to work with specific sections of your API. A guide to ‘best practices’ with your API may be appropriate to help people make the most out of your API.

Check out the guides in MailChimp’s API documentation for a good example. Twitter’s API docs ‘best practice’ section are great as well.

Quick start

One invaluable guide is the ‘getting started’ or ‘quick start’ guide. This can often be just a single page, with a succinct summary of the things you need to do to get going.

The YouTube API ‘getting started’ page is a useful example.

Useful tools

There’s lots of useful tools out there to help you get started when you document your API design. Here’s some suggestions.

API Blueprint is an open-source high-level API design language that is very useful for writing your documentation. The language is similar to Markdown, so it’s easy to work with. There are a number of SaaS tools offering services based on API Blueprint. One that I really like is Apiary.io (though they’ve recently been bought by Oracle so who know where that’ll take them), but there are others, like Gelato.

You might also consider Read the Docs and daux.io amongst others. There’s also the Open API Initiative, which is ‘focused on creating, evolving and promoting a vendor neutral API Description Format,’ though the initiative is ‘based on the Swagger Specification.’ Open API is an initiative of Swagger.io, and they have a list of tools and frameworks using the specification. The OpenAPI specification is on GitHub.

Whatever you use, your documentation should (probably) end up in a public location so that other developers can use it. (An exception might be for an API used in a secure decoupled system.)

Keep it simple

So, let’s return more directly to the question of designing your web API.

An important rule of thumb for me is to ‘keep it simple, stupid.’ There is no need to include anything more in the resources of your API than is necessary.

I say this as a long-time Drupal developer, knowing full well that we have a superpower in overcomplicating things, all those extra divs and classes all over the markup, all those huge arrays.

This is still true in the core REST resources of Drupal 8. For example, when GETting the core Content resource for node 10 /node/10?_format=json the response gives us …

{
"nid": [
{
"value": "10"
}
],
"uuid": [
{
"value": "6bfe02da-b1d7-4f9b-a77a-c346b23fd0b3"
}
],
"vid": [
{
"value": "11"
}
],

}

Each of those fields is an array that contains an array that contains the value name:value pair as the only entry. Whew! Exhausting. An array within an array, when there’s only one level-1 array ? Really? Maybe we could render that a little more simply as …

{
"nid": "10",
"uuid": "6bfe02da-b1d7-4f9b-a77a-c346b23fd0b3",
"vid": "11",

}

… which might help our API’s consuming applications to parse and use the JSON data more easily. Like I said above, I’d suggest that just using the core entity REST resources isn’t often the place to start.

The simplicity mantra should pervade your API design. Include only the data that is needed for the consuming apps. Pare it down, so it’s as easy to read as possible.

As a result, when you come to build that API in your Drupal 8 backend system, it will demand a good discipline on you of not just throwing out in the API resource responses what’s easiest but rather what’s best.

What’s in a name?

This is true in particular when it comes to your naming conventions and API resource paths.

Don’t just add root-level endpoints ad infinitum. Use well-structured paths for your resources where the depth of the path elements make sense together. The result should be that your resources are explorable via a browser address bar. E.g.

GET /articles/5/comments/19

… makes intuitive sense as a path: get comment 19 on article 5.

On the other hand, don’t just add depth to your resource paths unnecessarily. Separating things out with some logic will help make things intelligible for developers using your API. E.g.

GET /articles/comments

Umm? What’s that? The comments on articles — why would I want that? However …

GET /comments?contenttypes=articles

… is more obvious — a path to get comments, with a content types filter. Obvious. It also suggest we might be able to filter content types with a comma-separated list of types—nice!

Find a straightforward naming convention. Make the names of resource endpoints and data fields obvious and sensible at first glance.

Overall, make the way you name things simple, intuitive and consistent. If the title field of a data object in your resources is called ‘title’ in one place, ‘name’ in others and ‘label’ in still others, for instance, then it adds unnecessary complexity for writing reusable code.

Easy peasy, lemon squeezy

When designing your web API, it needs to be simple to use and work with. Help users to get just what they want from your API.

Support limiting response fields

You’ll make developers smile if you provide a way of limiting the fields that are returned in a response. You don’t always want to get everything from a resource. Being able to choose exactly what you want can help speed up usage of an API.

For example, consider supporting a fields parameter, that could be used like this:

GET /articles/5?fields=id,title,created

Support auto-loading related resources

The opposite might also be important, being able to load extra resources in the same request. If a request can combine related resources then fewer requests will need to be made, which again will help speed up using an API.

Supporting an embed query parameter could give you this. For example:

GET /articles/5?embed=author.name,author.picture,author.created

… would enable users to load also the article author’s name, their picture and the date their account was created. Note the dot syntax, which might be useful.

Flexible formats

Another way of making it easy for users is to support flexibility in the format of the data in the response. JSON is usually what people want to handle, but some do still prefer to use XML.

There’s also the problem that JSON has no support for hyperlinks, the building blocks of the web, which is a curiosity as the W3C admit. There are JSON protocol variants that attempt to address this, like HAL and JSON-LD, but I refer you to a fuller discussion of JSON and hypermedia and some useful resources on hypermedia and APIs from Javier Cervantes at this point.

Keep it steady, Eddy

When designing your API, you should expect it to have a certain lifetime. In fact, it’s bound to last long enough to need changing and improving. But what do you do about rolling out those changes?

Your devs will need the flexibility to change things, especially if they find bugs, and they’ll get frustrated if they can’t adapt the API to make improvements.

Your users need reliability and stability, though, and they’ll get frustrated if the API keeps changing and their consumer app dies without warning.

So, from the start, include versioning.

A pretty sensible thing is use a path element to specify the version number. E.g.

GET /api/v1/articles/5

You could use a query parameter instead, of course, though since query parameters are optional that would mean that without the version parameter your API would return the latest. Consumers who’d inadvertently missed including the version in their requests would be vulnerable to changes making their app die, which might result in some flame support emails.

Support that thing

Make sure there’s a way for your users to let you know when they have problems, there find a bug, or whatever.

If its an internal API, like with a decoupled CMS and frontend, then that is probably your bug tracker.

If it’s a public API, then you’ll need some public way for people to contact you. If you host your repository on e.g. GitHub then there’s support for issues baked in.

Respond.

Giant lists of bugs that never get addressed are soul-crushing.

Some other things to consider

Authentication and security

You’ll probably want to include some authentication to your API. You shouldn’t rely on cookies or sessions for your API as it should be stateless. Instead, by using SSL (you’re using SSL, right? yes, you’re using SSL.), you can implement a token-based authentication approach.

However, where a token approach is inappropriate, OAuth 2 (with SSL) is probably the best way to go. Here’s some further discussion on API security and authentication, if you’d like to read in more depth.

Caching

HTTP has a caching mechanism built in — woot! Just add some response headers and do some validation on request headers and it’s there.

I’ll point you elsewhere to read more about the 2 key approaches, ETag and Last-Modified.

Use HTTP status codes

HTTP defines lots of meaningful status codes that can be returned in your API responses. By using them appropriately, your API consumers can respond accordingly.

Useful errors

If a request has an error, don’t just return an error code. Your API should provide a useful error message in a format with which the consumer can work. You should use fields in your error message in the same way that a valid response does.

Healthy API design

In summary, when building an API it’s not healthy to just jump in and start writing the code for the API from a specification. Neither is it healthy to just rely on the default resources of CMS tools like Drupal 8. APIs always need to be tailor-made for the task.

APIs need to be designed.

If you can make your web API simple to understand and adopt, easy to work with, incorporating plenty of flexibility, if it’s stable and reliable and well-supported, then you’re well on your way to being the proud owner of a healthy API.

May 08 2018
May 08
May 8th, 2018

Over the past few months, Four Kitchens has worked together with the Public Radio International (PRI) team to build a robust API in PRI’s Drupal 7 site, and a modern, fresh frontend that consumes that API. This project’s goal was to launch a new homepage in the new frontend. PRI intends to re-build their entire frontend in this new structure and Four Kitchens has laid the groundwork for this endeavor. The site went live successfully, with a noticeable improvement in load time and performance. Overall load time performance increased by 40% with first-byte time down to less than 0.5 seconds. The results of the PRI team’s efforts can be viewed at PRI.org.

PRI is a global non-profit media company focused on the intersection of journalism and engagement to effect positive change in people’s lives. PRI’s mission is to serve audiences as a distinctive content source for information, insights and cultural experiences essential to living in our diverse, interconnected world.

Overall load time performance increased by 40% with first-byte time down to less than 0.5 seconds.

Four Kitchens and PRI approached this project with two technical goals. The first was to design and build a full-featured REST API in PRI’s existing Drupal 7 application. We used RESTFul, a Drupal module for building APIs, to create a JSON-API compliant API.

Our second technical goal was to create a robust frontend backed by the new API. To achieve that goal, we used React to create component-based user interfaces and styled them with using the CSS Modules pattern. This work was done in a library of components in which we used Storybook to demonstrate and test the components. We then pulled these components into a Next-based application, which communicates with the API, parses incoming data, and uses that data to populate component properties and generate full pages. Both the component library and the Next-based application used Jest and Enzyme heavily to create thorough, robust tests.

A round of well-deserved kudos to the PRI team: Technical Project Manager, Suzie Nieman managed this project from start to finish, facilitating estimations that led the team to success. Senior JavaScript Engineer, Patrick Coffey, provided keen technical leadership as well as deep architectural knowledge to all facets of the project, keeping the team unblocked and motivated. Engineer, James Todd brought his Drupal and JavaScript expertise to the table, architecting and building major portions of PRI’s new API. Senior Frontend Engineer, Evan Willhite, brought his wealth of frontend knowledge to build a robust collection of elegant components in React and JavaScript. Architect, David Diers created mechanisms that will be responsible for managing PRI’s API documentation that can be used in future projects.

Special thanks to Patrick Coffey and Suzie Nieman for their contributions to this launch announcement. 

Four Kitchens

The place to read all about Four Kitchens news, announcements, sports, and weather.

Sep 19 2017
Sep 19

Lately I have been hearing a lot about Laravel. This is a PHP framework to build web applications and that is quickly gaining popularity. I wanted to test it to keep up to date with this current technology. So I thought: I will build a concept in Laravel to see how it works and to compare it with Drupal 8.

My goals:

  • A static page in which the content is loaded from a local database.
  • Build a list of Blog items which is fed from a Drupal 8 RESTful API (which I had previously built for Node.js).

Overall content of this blog:

  1. Introduction to Laravel
  2. Laravel’s foundation
  3. Installing Laravel
  4. Routing in Laravel
  5. Laravel’s Migration: management of the database structure
  6. Eloquent ORM: query the database
  7. HTML templating in Laravel: Blade and Views
  8. Loading data from a RESTful Drupal 8 API

1. Introduction to Laravel

Required tools and knowledge

In order to participate you will need to have the following basic knowledge and tools:

  • PHP: intermediate knowledge
  • HTML/CSS basic knowledge
  • Good code editor (IDE), I am using PHPStorm
  • A browser, f.e. Firefox

What is Laravel

  • A PHP framework for web applications
  • Developed by Taylor Otwell
  • First release: February 2012
  • Open Source (MIT licence)

Why Laravel

According to the developers it is a ‘clean, modern web application framework’, which is built on other great tools. In addition, it is said that Laravel is ‘fun to code’.

Laravel is a MVC Framework

  • MVC = Model View Controller, a modern structure of web applications.
  • Model: application data and functionalities
  • View: the visible output, the HTML
  • Controller: interaction between user, model and view. Facilitated in Laravel by PHP.

Standard tools in Laravel

  • Routing of incoming requests
  • HTML templating (Blade)
  • Database definition and version control (Laravel’s Migrations and Eloquent ORM)
  • Authentication: login users and manage permissions.
  • Emailing with attachments

Laravel core is not a cms like Drupal 8

Laravel out of the box is not a cms. There is a cms component available (October cms), but in this regard Laravel cannot be compared with Drupal 8, which does offer in the core full blown cms functionalities. If you want to compare Laravel with Drupal, you will need to do this on the level of Drupal API and not Drupal cms.

2. Laravel’s foundation

Laravel’s foundation is built on strong components:

Laravel is a framework built on several other strong frameworks. Below an explanation of each component:

2.1 Laravel’s foundation: Symfony

Symfony is one of the main components in Laravel. The following Symfony components are, among others, used in Laravel:

Future releases of Laravel
 Laravel has announced to stay in sync with future releases of Symfony.

Symfony users
 When you are a user of Symfony you can also use Laravel components.

2.2 Laravel’s foundation: Composer

Laravel uses Composer, a PHP dependency manager. The main features are:

  • Works on a ‘per project’ basis, not global.
  • Works on Mac, Windows and Unix.
  • The dependencies are defined in a JSON file: composer.json. Composer can also be used in Drupal 8. This approach is also similar to the package.json in Node.js where NPM is ‘acting’ as Composer, see also.
  • See Packagist.org for 3rd party packages that can be used in Laravel by installing them through Composer.

2.3 Laravel’s foundation: Eloquent ORM

Eloquent ORM is Laravel’s database component, similar to the Database abstraction layer in Drupal 8. ORM is an acronym for Object Relational Mapper. It has been developed for Laravel, but can also be used outside Laraval. It is using an Active record pattern for database CRUD actions. It can facilitate one-on-one, one-on-many and many-on-many relations.

2.4 Laravel’s foundation: Migrations

Tables can be created, structured and (version) controlled through Laravel’s Migrations. All this is done via code, not configuration.

2.5 Laravel’s foundation: Blade

Blade is Laravel’s html templating machine. A Blade file is saved with the extension ‘.blade.php’. Variables in the template file can be placed as follows: {{variable}} (XSS filtered) or {!! variable !!} (unfiltered, [!] only use this when you know exactly what you are doing). You can also use PHP functionalities and codes in blade files. Blade also supports subtheming and conditional controls.

3. Installing Laravel

I am working on a Mac with OS X El Capitan. The current Laravel version is 5.1, and that is the version I am going to use. Go to Laravel docs and follow the instructions:

  • Make sure you have installed Composer.
  • Make sure the directory ~/.composer/vendor/bin is in your PATH, so that the laravel command is everywhere available. Learn here how.
  • Now you can install via the command laravel new a fresh Laravel installation. I am now going to my webroot and enter laravel new blog concept: a fresh Laravel installation will be created in the folder /blogconcept:

The created install:

  • You will get an ‘application key’. This will be used, among other things, to encrypt data, such as session data.
  • Go to the Laravel installation and run this command: php artisan serve to activate the Laravel server. Artisan is Laravel’s command line environment.

Go to your browser and navigate to http://localhost:8000. You should be seeing this:

4. Routing in Laravel

Routes are used to facilitate incoming page requests. This is similar to Drupal 7’s hook_menu() and Drupal 8’s routing system. The routes can be found in /app/Http/routes.php:

Static routes
 In routes.php you will see the default homepage defined, which you saw above in the browser. Here you can add your own routes. Below an example of a page with static information:

In a browser:

Dynamic routes
 Routes can also be built dynamically through working with variables:

Note the double quotes that are required to dynamically print out the variable. If you are using single quotes, Laravel will print literally {$person}.

In the browser:

5. Laravel’s Migrations: management of the database structure

First you will need a database, the standard used here is MySQL; I will make a database called ‘blog concept’. All the database settings are in config/database.php:

In this file you can set:

  • Fetch style
  • Type database: mysql is the standard, but Laravel also supports sqlite, pgsql and sql server.
  • The database connections, I am entering the following:

Tables can be managed manually through for example phpmyadmin, but that is not advisable. In Laravel the structure of tables/database can be programmed in code, resulting in flexibility and version control. ‘Database structure’ is also called ‘schema’.

By managing this in Laravel’s Migration developers can easily keep their databases in sync within a version control system, such as GIT. So you can also easily revert a database change.

Example: I am creating the initial migration file through command php artisan make:migration create_content_table. In the created file I am adding code that defines database tables:

This migration class exists of two methods: up and down. Up is used to create new tables, down is used to make the Up actions undone.

Execute command php artisan migrate, that will apply the migration code, and voila: the database tables are created:

More information: http://laravel.com/docs/5.1/migrations

6. Query the database with Eloquent ORM

For now I have manually filled the newly created tables with test content. Below a simple example to query this data:

  • Create a Model: php artisan make:model Content
  • Laravel creates the model in the /app folder:
  • The model is called ‘Content’ and not ‘Contents’, Laravel is smart enough to make that connection by itself.
  • Add the following code in routes.php:

$content = App\Content::find(1) => This is all it takes to query record with id=1 from the database table 'Contents', also here Laravel is smart enough to make the link with ‘Contents’. Then, all the fields from that record are stored in object $content, which can be assigned as variables to a blade template.

More information: http://laravel.com/docs/5.1/eloquent#defining-models

7. HTML templating in Laravel: Views & Blade

As previously indicated the HTML templating engine Blade is used in Laravel. The HTML files are called views, something else than Drupal Views: these are lists. An example of the use hereof can be seen immediately in the standard installation. In routes.php on line 15 the Laravel view ‘welcome’ is invoked: return view(‘welcome’);.

The views are included in the folder /resources/views, there you can also find the standard view ‘welcome.blade.php’:

An own dynamic view

Blade facilitates dynamically filling HTML pages. An example: I am creating a new view file called ‘about.blade.php’ and copy the HTML from welcome.blade.php:

I am adding the following code in routes.php:

You can see that the ‘about’ view is evoked through View::make() with an additional array included in which variables with content are defined that were previously loaded from the database.

Then I can use those variables in the Blade template:

In the browser:

FYI: more about Blade templates.

8. Data from a RESTful Drupal 8 API

As an example I am using the Drupal 8 API that I built earlier. The json output is looking like this:

First I played a bit around with Composer packages Buzz & Guzzle, both http clients. Those packages are facilitating much more than just retrieving data from a REStful API: POST requests, streaming large uploads, streaming large downloads, using HTTP cookies, uploading from JSON data, etc…

That is too much overhead, for now I can work with a standard php functionality: file_get_contents en json_decode:

  1. Create a new route: /blogs
  2. Query the json data from the external Drupal 8 RESTful API.
  3. Run through the json arrays data and create a new array where: key = url, value = blog title
  4. Render the Blade html view ‘blogs’.

Then I am copying an existing blade view and rename it to ‘blogs.blade.php’:

In this blade html view I am running through the associative array and am creating in this way a list with links:

Creating the detail page of a blog

Finally I would like to accomplish that when I click a link, the detail page of that blog appears:

  1. Create a new route with a variable
  2. Request the data from the Drupal 8 RESTful API.
  3. Look for a match in the url variable and a url in the json array from the API.
  4. When a match is found, create the variable: the title and body from the matched item.
  5. Render the html through a blade view.

In a browser:

As you can see, a lot still needs to be done concerning the styling. But as a purely functional concept, I am leaving it now the way it is.

Wrap up

Ok, that’s it for now. This is only an introduction in which I produced a concept. Many advanced components of Laravel have not yet been discussed, such as incorporating the logic code /routes.php that I placed to Models and Controllers. I would like to discuss this further in a next blog. Questions or feedback, let me know!

— Cheers, Joris

Dec 01 2016
Dec 01

dec16The twelfth issue of 2016 is now available! This month we look at how to write good tests with Behat and using Test Driven Development. This issue also includes articles on using HTTPlug to decouple your HTTP Client, Decoupled Blocks with Drupal and JavaScript. Our columnists have articles on writing a Chat bot, advice on securing your application’s secrets, making better bug reports, respecting diversity, and a look back at 2016.

Download your issue and read a FREE article today.

Oscar still remembers downloading an early version of the Apache HTTP server at the end of 1995, and promptly asking "Ok, what's this good for?" He started learning PHP in 2000 and hasn't stopped since. He's worked with Drupal, WordPress, Zend Framework, and bespoke PHP, to name a few. Follow him on Google+.
Jun 07 2016
Jun 07

Google summer of code (GSoC) seems to be a venue for students to get in touch with new technologies and be a part of many interesting open source organisations. Thanks to google for co- ordinating this initiative.

The last week was really a productive one for me in all aspects. I could manage time better to focus more towards my project. The climate here seems to have improved a lot. It’s now rainy here which has reduced the hot and humid climate to a large extent. My geographical location, Kerala, the southern part of India usually faces a fair climate.

If you are searching for a flashback of my previous GSoC’ 16 ventures, please have a look at these posts.

So, as you were expecting, now let’s talk about my activities in the second week of GSoC. The second week commenced with a bit more elaborative planning of the tasks to be carried out in the coming days. My main intention for the week was to discover more Drupal hooks and adapt it to my project code.

Wondering, what are hooks?

Hooks, in the simple definition, are PHP functions which have an inbuilt meaning given by Drupal to interact with modules. They also give us the freedom to extend the functionalities. The api.drupal.org gives wonderful explanations about the various hooks in action and their modifications that have come in the different Drupal versions.

Topics I have covered:

I would like to take this opportunity to share with you some of the concepts I could grasp from the previous week of GSoC.

  • hook_install
    • This performs the setup tasks when the module is installed.
  • hook_schema
    • This hooks the database schema for the module. This is invoked once the module is enabled. This resides in the .install file of the module.
  • hook_theme
    • This is for enhancing the module’s theme implementations.
  • hook_permission
    •  This hook defines the user permissions of the module; granting and restricting permission to the roles.
  • Configuration API
    • The Drupal variables are replaced by the configuration API.  You need to define the properties and default values in the new format.

Hoping to learn more Drupal concepts in the days ahead. I will be posting the updates regularly. Stay tuned for more Drupal concepts.

Feb 17 2015
Feb 17

Drupal is an excellent content management system. Nodes and fields allow site administrators the ability to create complex datasets and models without having to write a singe mysql query. Unfortunately Drupal’s theming system isn’t as flexible. Complete control over the dom is nearly impossible without a lot of work. Headless Drupal bridges the gap and gives us the best of both worlds.

What is headless Drupal?

Headless Drupal is an approach that decouples Drupal’s backend from the frontend theme. Drupal is used as a content store and admin interface. Using the services module in Drupal 7 or Core in Drupal 8, a rest web service can be created. Visitors to the site don’t view a traditional Drupal theme instead they are presented with pages created with Ember.js, Angular.js, or even a custom framework. Using the web service the chosen framework can be used to transfer data from Drupal to the front end and vice versa.

Pros

So what makes Headless Drupal so great? For one thing it allows frontend developers full control over the page markup. Page speed also increases since display logic is on the client side instead of the server. Sites can also become much more interactive with page transitions and animations. But most importantly the Drupal admin can also be used to power not only web apps but also mobile applications including Android and iOS.

Cons

Unfortunately Headless Drupal is not without its drawbacks. For one layout control for editors becomes much more difficult. Something that could be done easily via context or panels now requires a lot of custom fronted logic. Also if proper caching isn’t utilized and the requests aren’t batched properly lots of roundtrips can occur, causing things to slow down drastically.

Want to learn more about Headless Drupal?  Check out the Headless Drupal Initiative on Drupal.org.  And on a related topic, check out “Drupal 8 And The Changing CMS Landscape.

Feb 25 2013
Feb 25

In the Drupal community, we always recommend using the Drupal API, and best practices for development, management and deployment. This is for many reasons, including modularity, security and maintainability.

But it is also for performance that you need to stick to these guidelines, refined for many years by so many in the community.

By serving many clients over many years and specifically doing Drupal Performance Assessments, we have seen many cases where these guidelines are not followed, causing site slowdowns and outages.

Here are some examples of how not to do things.

Logic in the theme layer

We often find developers who are proficient in PHP, but new to Drupal misuse its API in many ways.

In extreme cases, they don't know they should write modules to house the application logic and doing data access, and leave only presentation to be done in the theme layer.

We saw a large site where all the application logic was in the theme layer, often in .tpl.php files. The logic even ended with an exit() statement!

This caused Drupal page caching mechanism to be bypassed, resulting in all page accesses from crawlers and anonymous users to be very heavy on the servers, and complicating the infrastructure by over-engineering it to compensate for such a development mistake.

Using PHP in content (nodes, blocks and views)

Another common approach that most developers start using as soon as they discover it, is placing PHP code inside nodes, blocks or views.

Although this is a quick and dirty approach, the initial time savings cause lots of grief down the road through the life cycle of the site. We wrote an article specifically about that, which you will find a link to below.

Heavy queries in the theme layer, when rendering views

In some cases, the logic for rendering individual nodes within a view is complex, and involves code in the view*.tpl.php file that has SQL queries, or calls to heavy functions, such as node_load() and user_load().

We wrote an article on this which you can find the link to below.

Conclusion

Following Drupal's best practices and community guidelines is always beneficial. Performance is just one of the benefits that you gain by following them.

Further reading

Jan 31 2013
Jan 31

Replicating module_invoke_all and drupal_alter in Javascript

If you've ever written a Drupal module before you're likely familiar with Drupal's hook system. I'm not going to go in to details about how the hook system works, or why this particular pattern was chosen by Drupal's developers. What's important here is what this systems allows module developers to accomplish.

At its most basic, the hook system is what allows me to write a module that enhances or extends Drupal -- without ever having to modify a line of someone else's code. I can, for example, modify the list of blocks that are available on a given page by simply implementing a "hook" function in PHP that modifies the information that was already set up. This approach is one of the things that makes Drupal incredibly flexible!

When you're writing your own custom modules, it is customary to expose these types of hooks for other modules, too. That way, other developers can come along and make minor modifications or feature enhancements to your module by "piggybacking" on your module's functionality, rather than hacking your code. It also means that you don't have to anticipate every possible use case for your code: by providing these extension points, you allow future developers to extend it.

Drupal makes it really easy for modules developers to do this, and it provides a set of helper functions that allow you to easily broadcast these "I have a hook! Who wants to tie into it?" announcements to the world.

The case for APIs

Now, that's all well and good, but what if the functionality I want people to be able to alter or events I want people to be able to react to are encapsulated in Javascript? This is where Drupal breaks down a bit and we're left to our own devices. Drupal provides a simple mechanism for modules to essentially register a bit of code that they would like to be executed whenever Drupal.attachBehavoirs is called. This happens when the DOM is fully loaded and Drupal's Javascript code has been properly initialized, and anytime new elements have been added to the DOM via AJAX. And that's about it.

For most cases where Javascript needs to interact with Drupal this works just fine. What you're likely really after is some element in the DOM anyway so you can do your sweet web 2.0 fadeIn().

Sometimes, though, your Javascript needs are more complex than adding visual pizzaz. Consider this; You've been asked to write a module that integrates a video player from a third party site into Drupal. The video service offers a straightforward Javascript based embed option. All you have to do is include their Javascript file on the page and call the player.setup() method, passing in an embed code to the player so that it knows which video to play. Easy enough, and a common pattern.

Let's say the setup() method takes not only an embed code but also an array of additional paramaters to configure how the player appears and behaves. Some of those paramaters are callback functions -- the name of an additional Javascript function that should be called when certian things happen. Some examples of this might be 'onCreate' when the player is embeded and ready to start playback, 'onPause' when someone clicks the player's play/pause button, and so on. For our example we'll assume that we're implementing an 'onCreate' callback. It should be triggered by the video player after it's been embedded, and is ready for playback to start. (Another common example of something like this the jQuery.ajax, which can take 'success' and 'error' callbacks. Which one gets called depends on the result of the Ajax request.)

This should be simple, right? Just set the callback to 'Drupal.myModule.onCreate' and write the corresponding function in your mymodule.js file!

Except... Later on in the project, Kyle comes along and is told to implement an unrelated piece of functionality that also fade a DOM element in on the page after the video player has been embeded. Now two different functions both need to fire when the Video player has been created. You can't just pass in a second 'onCreate' callback function to the player.setup() method -- it only allows one value! So now Kyle is stuck trying to jam his unrelated Javascript in to your Drupal.myModule.onCreate function. Blam! You've got a mess of unrelated, hard to maintain code!

A better way of handling this would be for your module to re-broadcast the 'onCreate' callback to give other code a chance to respond to it as well. You could take it one step farther and implement a system that sends out a notification when the 'onCallback' event occurs, and subscribe to it with any functions that need it. That approach would be a lot like the module_invoke_all() function in Drupal's PHP API.

Lucky for you, there are all kinds of ways to do this in Javascript! I'll outline two of them below.

The Drupal Way

One way of solving the problem is to replicate the Drupal.behaviors system provided by core. That's actually pretty straightforward. You need to:

  • Create a well known place for someone to register their objects or functions.
  • Write a short snippet of Javascript that will loop through and execute these registered functions.
  • Call this Javascript at the appropriate time.
  • Ensure that your module's Javascript is loaded before that of other modules.

In your javascript code, you'll need to create a standard object that other modules can go to when they register their functions. In core, this is Drupal.behaviors. We'll create our own new object for this example.

var MyModule = MyModule || {};
MyModule.callbacks = {};

Then you'll need an easy way to call and execute any registered callbacks.

MyModule.executeCallbacks = function(data) {
   $.each(MyModule.callbacks, function(key, callback) {
       if ($.isFunction(callback)) {
          callback(data);
       }
   });
}

What this code does is loop over all the functions collected in MyModule.callbacks and executes them. Pretty simple, really! It works well for notifying any code of some "event" as long as you remember to call the MyModule.executeCallbacks() method at the appropriate times.

Now, any other module can register callback functions that will be called by the MyModule.executeCallbacks() method:

MyModule.callbacks.theirModuleOnCreate = function() {
   // Do some sweet Javascript stuff here ...
}

Put it all together by implementing your onCreate callback (the code we wanted to implement at the very beginning of this exercise!) and call the new code.

MyModule.onCreate = function() {
   // Give all modules that have registered a callback a chance to respond.
   MyModule.executeCallbacks();
}

Pretty painless. Just make sure your module's Javascript file is loaded before any others: in Drupal, you can do that by changing the weight of your module to -10, or something similar. If you don't do that, you'll end up with warnings about "MyModule.callbacks being undefined" when someone else's Javascript is loaded first, and tries to register a callback with your object.

This approach is easy to implement, but it still has some problems.

  • It's a major "Drupalism." For anyone familiar with Javascript but not with Drupal's way of doing things, it's a conceptual hurdle that needs to be overcome before understanding how to add a new behavior.
  • If one behavior fails, the execution stops: anything that hasn't be executed will not get called, and you're dependent on others to write code that doesn't fail.
  • There is no easy way to remove a behavior added by someone else's code, or to overwrite the way that Drupal core does something. Don't like the table drag javascript? The only way around it is Monkey Patching.

An alternative way

Another approach that's a bit more "Javascripty" is to use the jQuery.trigger() and jQuery.bind() methods. With them, you can create custom events that other modules can listen for and react too. It's a lot like using jQuery to intercept the 'click' event on a link, perform some custom action, then allowing the link to continue with it's processing. In this case, though, we'll be triggering our own custom event on a DOM element. To do this you need to:

  • Call jQuery.trigger on an object or DOM element in order to broadcast an event.
  • Use jQuery.bind on an object or DOM element to register a listener for an event.
  • Wash, rinse & repeat ...

As usual, the code samples below would go inside of your module's mymodule.js file and be included on the page when necessary via the drupal_add_js() PHP function.

Inside of our module's .onCreate callback, we use the jQuery.trigger() method to trigger our custom event and alert all listeners that they should go ahead and do their thing. It's not necessary to prefix our event names with 'myModule.' but it does lead to cleaner code. (It also makes it easier to unbind all of the events associated with a particular module in one step.) This approach is functionally equivalent to calling the MyModule.executeCallbacks() method from the previous example. We're telling anyone that wants to participate that now is the time to do it!

MyModule.onCreate = function() {
   // Trigger an event on the document object.
   $(document).trigger('myModule.onCreate');
}

The second piece of this puzzle is using the jQuery.bind() method to add an event listener that will be triggered any time our custom event is triggered. Each event listener receives the jQuery.Event object as the first argument. The code below is equivalent to the bit above where we register our callback with MyModule.callbacks.theirModule = {}

$(document).bind('myModule.onCreate', function(event) {
   // Do my fancy sliding effect here ...
});

Any number of modules can bind to the custom event, and respond to the onCreate callback event, without ever having to modify your module's Javascript.

Another technique that I've used in the past is to create a drupal_alter() style functionality in Javascript. This would allow others to modify the parameters that my code passes to a third party's API. It's easy to do, so since you can pass an array of additional arguments to the jQuery.trigger() method. They'll be passed along to any listeners added with jQuery.bind(). And, since complex data types in Javascript are inherently passed by reference, the listener can make changes to the incoming parameters and they'll be reflected upstream. Something like the following would do the trick.

MyModule.createWidget = function() {
  var parameters = {width: 250, height: 100, onCallback: 'MyModule.onCreate'};
  // Allow other modules to alter the parameters.
  $(document).trigger('myModule.alterParameters', [parameters]);
  superAwesomeWidgetAPI().setup(parameters);
}

Then anyone else could bind to the new 'myModule.alterParameters' event and receive the parameters object as an additional argument. The first argument for any function using jQuery.bind() to listen to an event is always the jQuery.event object.

$(document).bind('myModule.alterParameters', function(e, parameters) {
  // Here I can change parameters and it will be reflected in the function that triggered this event.
  parameters.width = 350;
});

While this method isn't perfect either, I like that it's closer to the Javascript programming patterns used in the broader world outside of Drupal. This means it's easier for someone not familiar with Drupal to understand my code and to quickly figure out how to work with it.

It does, however, still exhibit some of the same problems as the Drupal.behaviors method. Notably the fact that if any one listener has code that fails the whole system breaks down. In addition, you have to trigger and bind to events on either a DOM element or other Javascript object.

Summary.

Drupal itself doesn't come with a Javascript equivalent to the module_invoke_all() function, but there are a lot of ways that we can implement a similar system ourselves. When you run in to this problem in your development, I encourage you to use the second approach outlined: it has all the same capabilities of the Drupal.behaviors approach, with less code and a shallower learning curve.

These are by no means the only methods for accomplishing this sort of task in Javascript. Another for example would be the popular publish/suscribe pattern, but we'll wait to explore those in another article! Whichever approach you choose, it's important to build for future flexibility, just as you would with your PHP code.

Oct 14 2012
Oct 14

When I was unable to post to Twitter from my blog tonight, I found a very new thread on drupal.org discussing the issue: the Twitter API changed again!

I am only using the Twitter module on Drupal 6 sites right n   ow, I took two of the fixes and combined them in a patch for the 6.x-3.0-beta9 version. Then MinhH submitted new code that only requires a change in one place.

here's my latest patch for the 6.x-3.0-beta9 version

since the Twitter API is constantly changing, and the module is not very stable, always read the thread carefully to make sure you are applying the latest patch! Others have cleaned up the code I added from MinhH, and created a patch for the 6.x-3.x-dev branch which makes sense (however I suspect that 6.x-3.0-beta9 may be ahead of the -dev branch, and it's not a branch provided in the project's Github). Anyway, Open Source rocks, Murray!

and because my Twitter RSS feed URL was also broken, I found this post

https://dev.twitter.com/discussions/844

which provides an update to the URL format:

https://api.twitter.com/1/statuses/user_timeline.rss?screen_name=decibelplaces

Aug 24 2012
Aug 24

For many years now I've written code to programmatically create nodes. I've done it for dozens of clients starting with Drupal 5. It has always looked the same, been very procedural and felt very wrong. Recently I started playing around with turning this chore into object oriented and started writing a module.

Well yet again it turns out there is already a module for that. The entity module. Not only did it do (almost) everything that I was trying to make my module do but it does far far more. In fact, it has totally revolutionized the way I write integration code.

Have you ever done something like this?

<br /> $node = node_load(1);<br /> $node-&gt;field_reference[LANGUAGE_NONE][0]['nid'] = '463';<br /> $node-&gt;field_text[LANGUAGE_NONE][0]['value'] = 'Some Value';<br /> node_save($node);<br />

If you have, don't ever do it again.

Many of you know that the Entity module is great for defining new entity types using the Entity API but it has another really great API within it. That is turning all the core entities into REAL objects, not just pretend objects. By this I mean that it adds a lot of methods to the objects as well as just properties.

The example above would be:

<br /> $node = new EntityDrupalWrapper('node', 1);<br /> $node-&gt;field_reference[0]-&gt;set(463); // Note that this is a multi value field.<br /> $node-&gt;field_text-&gt;set('Some Value'); // Note that this is a single value field.<br /> $node-&gt;save();<br />

Or if you want to get extra fancy, you can do the following:

class Node extends EntityDrupalWrapper {
public function __construct($data = NULL, $info = array()) {
parent::__construct('node', $data, $info);
}

}

$node = new Node(1);<br /> $node-&gt;field_reference[0]-&gt;set(463); // Note that this is a multi value field.<br /> $node-&gt;field_text-&gt;set('Some Value'); // Note that this is a single value field.<br /> $node-&gt;save();<br />

It makes me so excited to be able to interact with core entities in an entirely object oriented fashion. The Entity API module with the EntityDrupalWrapper will wrap most of the core drupal entities with plenty of useful helper methods.

It has revolutionized the way that I write integration code and hopefully it will yours as well. So next time you need to programmatically create nodes, check out the Entity API and the EntityDrupalWrapper.

Jul 08 2011
Jul 08

If you are a Drupal module developer, I have a request for you. A plea for you to help make my life easier, and likely the lives of many other site builders.

I'm currently building a website that uses a theme override to display a prettily formatted username. Drupal provides a way to do this via the theme_username() API call, so in all places where Drupal core displays a user's name, my pretty string now shows.

However, when I started adding contrib modules for extra functionality, I found that quite a few of these don't use the theme call to display a user's name. Instead, they simply print out the value of the name field in the user object. If I want to change that as a site builder, my only choice is to hack up the module so it displays what I need.

For example, a Drupal 6 module might set a page title on a tab somewhere in the user account like this:

drupal_set_title(t('Widgets for !username', array('!username' => $account->name)));

Whilst what it should be doing is passing the name through the theme layer first, like this:

drupal_set_title(t('Widgets for @username', array('@username' => strip_tags(theme('username', $account)))));

And the same goes for calls to drupal_set_message() for status or error notifications, or anywhere else a user's name might be displayed.

This way the site builder is in charge of how users are displayed and won't need to hack up any contrib modules. Though that's not hard, it makes updating them unnecessarily painful, as any changes would need to be re-applied.

So my plea is, if you're a module developer and your module displays $account->name somewhere, please change your module to use theme('username') instead.

Conversely, if you're using a contrib module and find that it prints $account->name when it really should use theme('username'), please open an issue and attach a patch if you can. I will :-)

Feb 28 2011
Feb 28

The ottomans brought many innovations, modern postal systems run by the various consulates, the use of the wheel for modes of transportation, stagecoach and carriage, the wheelbarrow and the cart and the oil lantern, among the first signs of modernization in the city. Thanks.Karpacz noclegi

Dec 23 2010
Dec 23

The Webform Drupal module is an amazing thing.  It lets you build all sorts of amazing forms.  Combined with jQuery you can construct dynamic forms that improve user quality of life.  However, Webform's export features are extremely limited.  You can get a run-of-the-mill exported CSV with every component includes with the generic field name.  What if you want or require a different format or a variable number of submissions only?  It seems like the only real solution right now is to write your own app or module that reads Webform's data.

I found a a new module called Webform MySQL Views that helps target data from Webform submissions.  This should help anyone writing something to interact with Webform submission data.

I recently wrote a backend PHP application that reads Webform data and exports it into a completely customized CSV.  It is pretty generic, though it requires manual setting updates to change what you're exporting into CSV (it needs component ID's from the targeted Webform).  The purpose of this customized CSV exporter is to import into The Raiser's Edge, a proprietary CRM that nonprofits use to track constituents.  Why not a module?  I definitely tried to make a module out of this, though having not authored a Drupal module yet my experience is somewhat limited in the Drupal API and it was more efficient to make an external app.  I learned a lot about module development though, so this could be a future endeavor (or maybe the Webform maintainers will expand on their export functionality).  Jumping straight into the Webform code and trying to understand all of it as a relative newcomer to module development isn't something I would recommend for everyone.

Oct 10 2010
Oct 10
Printer-friendly version

In part 1 of this series of articles, I discussed D7’s new database layer and we started looking at what to expect with examples of static and dynamic queries. Those queries are for fetching information from a database, but what’s changed from D6 to D7 when it comes to manipulating the information in our database? The answer: a lot.
 We’ll start with INSERT queries. In D6 we were relegated to two options when it came to INSERT queries. We could either write out an INSERT query the long way:

<?php
  $query = "INSERT INTO {node} (title, uid, created) VALUES ('%s', %d, %d)";
  db_query($query, 'Example', 1, $timestamp);
?>

Or, we could use the function drupal_write_record():

<?php
  $record = new stdClass();
  $record->title = 'Example';
  $record->uid = 1;
  $record->created = time();

  drupal_write_record('node', $record);
?>

This was a vast improvement over D5 certainly, but could still result in some cumbersome code. Looking back to the first article in this series I discussed how D7's database layer is now built on top of PDO. Let's see how we handle an INSERT query in D7 utilizing this new API.
 In D7, first we need to create an insert object:

<?php
  $query = db_insert('node', $options);
?>

Looking at the D7 handbook for INSERT queries we learn that this "query" object uses what is referred to as a "fluent API", meaning that all methods – with the exception of the execute() method – return the query object. That means we can chain our methods together and keep our insert queries compact:

<?php
  $nid = db_insert('node')
    ->fields(array(
      'title' => 'Example',
      'uid' => 1,
      'created' => REQUEST_TIME,
    ))
    ->execute();
?>

Nice. As you can see, we pass the db_insert() function a table name, then we chain the fields() method to it. We pass the fields() method an associative array (it can actually accept several different types of parameters, as we'll see shortly), where the array keys correspond to our table's column names. Finally, we chain the execute() method to perform the INSERT query. The execute() method, as mentioned earlier, does not return the query object, but instead returns the value of an auto-increment field in the table for the record just inserted. As the D7 handbook states, if the table does not have an auto-increment field the value that execute() returns will be undefined and "should not be trusted". Hence, in the example above, we assign the return value of the execute() method to a variable named $nid. Makes sense to me.

The last example is great for inserting a single record into a table. But, more often than not, we are going to want to insert multiple records into a table in one go. Remember that the fields() method can accept different types of parameters. We can pass the fields() method an indexed array of column names and then chain another method to it: values(), which takes an associative array. This method allows us to create several records at one time. Another example from the handbook should make this clearer:

<?php
  $values = array(
    array(
      'title' => 'Example',
      'uid' => 1,
      'created' => REQUEST_TIME,
    ),
    array(
      'title' => 'Example 2',
      'uid' => 1,
      'created' => REQUEST_TIME,
    ),
    array(
      'title' => 'Example 3',
      'uid' => 2,
      'created' => REQUEST_TIME,
    ),
  );

  $query = db_insert('node')->fields(array('title', 'uid', 'created'));

  foreach ($values as $record) {
    $query->values($record);
  }

  $query->execute();
?>

Pardon me while I pick my jaw up from the floor. How cool is this? We create a $values array that contains the records we want to insert, again with keys corresponding to column names. Then we create our query object and inform it of which table and fields we will be inserting data into. We then loop through the $values array, passing each record to our query object's values() method. Finally, with one call to the execute() method we insert three new rows into the node table. Sweet and simple.

We can also insert data based on a select query. For brevity's sake, I will refer you to the handbook page to see this in all its glory, but in essence you simply build a select query just how we did in the first article of this series, then we pass that select query object to another method of our insert query object: from(). http://drupal.org/node/310079

UPDATE queries follow a similar convention as INSERT queries in D7. As a quick aside, in D6, there is an "issue" with using drupal_write_record() in that you have to retrieve the value of the primary key of the table row you are wanting to update. Otherwise, the UPDATE query will silently fail. Let's assume we're performing an UPDATE query in D6 from within hook_nodeapi():

<?php
  // this will not work as expected
  $record = new stdClass();
  $record->title = 'New Title';

  drupal_write_record('node', $record, 'nid');

  // this does work, however
  $record = new stdClass();
  $record->title = 'New Title';
  $record->nid = $node->nid;

  drupal_write_record('node', $record, 'nid');
?>

I, like many other module developers, had to find this out the hard way. Not really a "big" issue, but an annoyance nonetheless in that the UPDATE query silently fails. Silent failures === beating your head against the wall until you trial-and-error out the issue, which all equates to a unneeded loss of time and productivity. But I digress.

As stated, D7 UPDATE queries are nearly identical to INSERT queries with a few caveats. As with INSERT queries, we begin with creating a new query object by calling the function db_update():

<?php
  $query = db_update('node', $options);
?>

This object, just like an INSERT query object is chainable, too. This object also has a fields() method where we set our column names and corresponding values, but unlike our INSERT query object fields() method, an UPDATE query object's fields() method can ONLY accept an associative array. We inform our UPDATE query of which row or rows to update by chaining the method condition() to the object, and finally perform the query with the execute() method. An example from the handbook:

<?php
  $num_updated = db_update('node')
    ->fields(array(
      'uid' => 5,
      'status' => 1,
    ))
    ->condition('created', REQUEST_TIME - 3600, '>=')
    ->execute();
?>

For more information on the awesomeness that is the condition() method, you can check out the handbook page on it here: http://drupal.org/node/310086

An UPDATE query object's execute() method in this case returns the number of rows affected by the query. However, this does not equate to the number of rows "matched" by the query. As the handbook tells us:

"The execute() method will return the number of rows affected by the query. Note that affected is not the same as matched. In the above query, an existing record that already has a uid of 5 and status of 1 will be matched, but since the data in it does not change it will not be affected by the query and therefore not be counted in the return value. As a side effect, that makes Update queries ineffective for determining if a given record already exists."

Thanks go out to the Drupal documentation team for saving me that potential future headache.

DELETE queries are nearly identical to UPDATE queries except we call the function db_delete():

<?php
  $num_deleted = db_delete('node')
    ->condition('nid', 5)
    ->execute();
?>

The execute() method for DELETE query objects returns the number of rows of deleted as a result. Not much else to cover on that front.

The last type of query I'll cover here are called MERGE queries. You may have also heard this out in the field referred to as "UPSERT" queries. These queries are a hybrid of INSERT and UPDATE queries, in that if a given condition is met either an INSERT query will be performed or and UPDATE query will be performed. This probably has me more giddy than any of the other type of queries because I can recount numerous times I have had to write long if/else statements for performing this task.

In its simplest form, MERGE queries look like the following:

<?php
  db_merge('example')
    ->key(array('name' => $name))
    ->fields(array(
      'field1' => $value1,
      'field2' => $value2,
    ))
    ->execute();
?>

We use the function db_merge() and pass it a table name. We chain the key() method which accepts an associative array whose key is the a key field in the table and we assign it a value of $name. We then chain the fields() method which, as in INSERT and UPDATE queries, contains the actual data we will be manipulating on which columns. And finally, we call the execute() method. In this example, if a row exists in our table whose 'name' column contains the value of $name, an UPDATE query will be performed setting 'field1' and 'field2' values appropriately. If there is NOT a 'name' field with the value of $name, a new row will be created with the value of $name we passed to the key() method.

For. Real. We can also get more granular with the conditions we want to apply to determine if we perform an UPDATE or INSERT query. Again, for brevity's sake (since this can get a little complicated) I will refer you to the handbook page on MERGE queries: http://drupal.org/node/310085

One final topic I would like to cover, as it is very relevant to performing these types of queries, is transactions. D6 and below had ZERO support for transactions and, as such, we module developers had to be very diligent in preventing data loss should a query fail halfway through execution. This typically meant lots of cumbersome error handling code, and even then the chance of data loss was still a very real risk.

In D7, transactions are very simple to implement. We start a new transaction by calling the function db_transaction(). Here's the example given in the handbook:

<?php
function my_transaction_function() {
  // The transaction opens here.
  $txn = db_transaction();

  $id = db_insert('example')
    ->fields(array(
      'field1' => 'mystring',
      'field2' => 5,
    ))
    ->execute();

  my_other_function($id);

  return $id;
  // $txn goes out of scope here, and the entire transaction commits.
}

function my_other_function($id) {
  // The transaction is still open here.

  if ($id % 2 == 0) {
    db_update('example')
      ->condition('id', $id)
      ->fields(array('field2' => 10))
      ->execute();
  }
}
?>

As you can see, when the variable $txn goes out of scope (or is destroyed as far as PHP is concerned), the transaction is committed. Until that happens, the transaction will not be executed. For the future of Drupal and module developers alike, this means we can rest a little easier at night knowing that data loss has been greatly minimized should things go wrong (like your DB server unexpectedly goes down).

I, for one, am just fine with that.

The next article in this series will cover some of the new hooks in D7 that have me very excited. Hooks like hook_page_alter() and hook_filetransfer_backends() are going to change the Drupal landscape greatly, I feel, and I can't wait to get my hands on them.

Until next time, happy coding.

Oct 06 2010
Oct 06
Printer-friendly version

With Drupal 7 right around the corner, I have recently put myself to the task of ramping up on what's new, what's changed, and what do I – as a module developer – need to know when I sit down to code my first D7 module (or upgrade one of my D6 modules *shameless plug*). I've spent the last few weeks scouring over the D7 core and API documentation, and let me be the first to tell you if haven't heard it yet (unlikely): better times are ahead. For all of us.

The amount of API changes from D6 to D7 are broad and sweeping. D7 is certainly still Drupal, meaning that it still feels and acts like Drupal. Nodes are still nodes that can have taxonomy and comments, users are still users with roles and permissions, so on and so forth. But nearly everything under the hood and in the UI, for that matter, has been improved upon and/or refactored. From a coder’s perspective, I am soon to be in developer’s heaven when D7 goes stable. From the standpoint of being a framework, D7’s API is more mature and modern. You could almost think of D6 as a kid in highschool and with D7 that kid has graduated and is now ready for the real world. The refactoring that has taken place over the last couple of years and all of the new features now available will be a boon for module and theme developers alike. This also explains just why we have had to wait so long for a stable release. It’s almost as if you took your grocery-getter car to Q from James Bond for an “upgrade”. But, I digress.

This article will be the first in a series exploring D7’s API. For this article (itself a two-part series), I want to talk about what I feel, as a developer, is the most substantial and important change to the Drupal core. The new database abstraction layer (or DBTNG Database: The Next Generation) is a complete overhaul of the database layer we have grown accustomed to in versions D6 and before. Built on PDO, the database layer in Drupal is now truly abstract, meaning that it is database agnostic. This gives Drupal the flexibility to run on any number of different platforms without having to write new code or change any existing code that interacts with the database. This also means that we can expect to see a lot more enterprise-level Drupal sites once D7 hits stable as it’s footprint in the corporate world will now be able to grow substantially. Gravy. So, let’s see what all this means.

First and foremost, gone are the days of having to write ludicrously long and complicated MySQL or Postgres dependent SQL statements to fetch or write data. Gone are the days of having to define your placeholders’ type. And gone are the days of having to worry about what would happen to your data if an insert or update query failed in the middle of executing (yes, we finally get to use transactions!).

Let’s jump right in and start with basic SELECT queries in D7’s new database layer. D7 introduces us to the concepts of static and dynamic queries. Static queries are queries that get passed to the database as is, meaning you query the database with a SQL string you write. Dynamic queries are built by Drupal using a query object and that query object’s methods. A few basic examples may help illustrate the difference.

Example of a static query in D7:

<?php
  $result = db_query(“SELECT nid FROM {node}”);
?>

Well, that should look familiar to anyone who has done any Drupal module development in the past. Our old friend db_query() is still here but is much more powerful now. How so? Glad you asked. First, the way we define placeholders has changed. We no longer have to type hint them with %d or %s, and we also don’t have to worry about remembering to surround string placeholders with quotes. Let’s see the difference:

Example of using placeholders in D6:

<?php
  $result = db_query(“SELECT mail FROM {users} WHERE name = ‘%s’”, “Bob”);
?>

Example of using placeholders in D7:

<?php
  $result = db_query(“SELECT mail FROM {users} WHERE name = :name”, array(‘:name’ => ‘Bob’));
?>

At first, this may seem a little convoluted as we have to write a little more code to accomplish the same task, but the benefits far outweigh the few extra characters I have to type. Placeholders, as seen above, are now defined using the convention :identifier. No need for quotes, no need to worry about the type. The function db_query()’s second parameter is an associative array where the keys are the placeholder identifiers and then assigned their corresponding values. It’s basically the same principle that the Drupal function t() uses with its placeholders. You can also assign a placeholder value to be an array.

An example taken from the official handbook page for D7’s database layer:

<?php
  // This code:
  db_query("SELECT * FROM {node} WHERE nid IN (:nids)", array(':nids' => array(13, 42, 144));

  // Will get turned into this prepared statement equivalent automatically:
  db_query("SELECT * FROM {node} WHERE nid IN (:nids_1, :nids_2, :nids_3)", array(
    ':nids_1' => 13,
    ':nids_2' => 42,
    ':nids_3' => 144,
  ));

  // Which is equivalent to the following literal query:
  db_query("SELECT * FROM {node} WHERE nid IN (13, 42, 144)";
?>

Sweet. Another goodie that db_query() now has thanks to PDO is the ability to define how our results will be retrieved. You can provide db_query() with a third parameter which is a PDO-specific constant that defines the retrieval type.

The types we can define are:

  • PDO::FETCH_OBJ — fetches results into an object
  • PDO::FETCH_ASSOC — fetches results into an associative array
  • PDO::FETCH_NUM — fetches results into an indexed array
  • PDO::FETCH_BOTH — fetches result into both an indexed and associative array

By default, results are fetched into an object. Here’s what an example of fetching our results into an associative array looks like:

<?php
  $result = db_query(“SELECT mail FROM {users} WHERE name = :name”, array(‘:name’ => ‘Bob’), array(‘fetch’=> PDO::FETCH_ASSOC));
?>

As you can see, db_query()’s third parameter is an associative array with the key ‘fetch’ where we assign the retrieval type. This array can also have another key defined: ‘target’, which tells Drupal where to retrieve the results from. As of the writing of this article, this is limited to only 2 possible values: “default” and “slave”. Still, though, the potential here makes me giddy.

Additionally, we can also pass the name of a class we want are results to be retrieved in. Say what?!? Oh yes. If we have defined this class, and we pass db_query() the name of our class, results will be retrieved and assigned as properties of that class. If that class has a constructor method, by default it will run after the results are retrieved and the properties defined, but we can also tell db_query() to run the class’ constructor method before the properties are assigned.

Another example from the official handbook page:

<?php
class exampleClass {
  function __construct() {
    // Do something
  }
}

  $result = db_query("SELECT id, title FROM {example_table}", array(), array(
    'fetch' => 'ExampleClass',
  ));
?>

Dynamic queries in D7, on the other hand, are a different creature. We have a new function that we will use for dynamic queries: db_select(). We can pass db_select() three parameters. The first is a database table name. The second is an alias for that table name. The third parameter is identical to db_query()’s third parameter. db_select() returns a query object which has some methods defined that we can use to construct our dynamic query.

An example of a dynamic query in D7:

<?php
  $query = db_select(‘users’, ‘u’);
  $query
    ->condition(‘u.name’, ‘Bob’)
    ->fields(‘u’, array(‘mail’));
  
  $result = $query->execute();
?>

Need to join in another table? No sweat. Like I mentioned earlier in this article gone are the days of having to write cumbersome SQL queries. Here’s how easy it is to do a join with dynamic queries in D7:

<?php
  $table_alias = $query->join('user', 'u', 'n.uid = u.uid AND u.uid = :uid', array(':uid' => 5));
?>

Our query object’s join method takes a table name as the first parameter, and an alias for that table as the second parameter. The third parameter is the “ON” condition of the join and the last parameter is for placeholder replacement. The join method returns the table alias of the table assigned. We can also pass SELECT query object to the join method as the first argument.

An example from the handbook:

<?php
  $myselect = db_select('mytable')
    ->fields('mytable')
    ->condition('myfield', 'myvalue');
  $alias = $query->join($myselect, 'myalias', 'n.nid = myalias.nid');
?>

I don’t have nearly enough room to go into all the features and capabilities of dynamic queries in this article, suffice to say as you can see from the few simple examples above they are extremely powerful and flexible and you’ll be using them in your own modules a lot very soon. Less code and more bang for the buck is always fine by me.

In the second part of the this article, I will go over insert, update, and delete queries and will introduce you merge queries.

Until next time, happy coding.

Further reading:

Jul 03 2010
Jul 03
Screenshot

The staging site for API.drupal.org is ready for testing. We have added the theme from Drupal.org’s redesign and documentation from classes. Behind the scenes there is a new PHP parser, project support, and automation with Hudson. This is a good test case for the latest Drupal.org infrastructure work, and has been great to work with. Please note problems in this issue queue.

Jun 21 2010
Jun 21

Here at Digg, we love open source. Many of our engineers are contributors back to projects like Cassandra, Puppet and Hudson and for the past couple of months we have been doubling down on our use of Drupal. Digg About and the forthcoming developers site are built upon Drupal. Drupal was choosen for its flexibility and active community allowing us to build sites quickly. We are confident that Drupal will continue to be a force in the future and are anxiously awaiting the release of Drupal 7. To those ends, we have decided to release a couple of brand new modules for Drupal.

  • Diggable module is written and maintained by Digg engineers and is the official way to integrate your Drupal site with Digg.
  • Digg Login is a module to give your site users the ability to log into your Drupal site using their Digg user name and password over OAuth. It is built using PEAR packages which are written and maintained by our own engineers.

The advantage of the Digg Login module is that your users do not have to maintain separate passwords providing them another easy and simple way to log into your site. It is written to ensure any customizations that have been added to the user registration form are preserved and to be filled out by your new users. All it takes to setup is to download the PEAR packages and setup a Digg Application.

For more on how we use open source Digg visit our Open Source page.

Apr 18 2010
Apr 18

The API module is almost ready to show classes and many more improvements on api.drupal.org. It is not quite ready yet; everything is in the database, but menu callbacks and template are needed to show the classes.

I’m looking for all bugs and criticals to be fixed and then, after a bit of testing, api.drupal.org can finally be upgraded. The configuration I’m using is

example configuration for API module

There is new support for projects. We will start off small, just Devel and Examples. The goal is all of contrib, but the module needs some changes to easily handle that volume.

The PHP parser was separated from documentation parsing and replaced with Grammar Parser, which goes toward fixing many bugs and the class & interface support.

There is a growing set of unit tests, which were unfortunately written after these refactorings. API is well-suited to testing since it is a parser with well-defined input and output. There are plenty of possibilities and testing will help make those happen. But not until the next release cycle, this one is already too long.

Thanks to

Nov 22 2009
Nov 22

How many times have you been coding something and it doesn't work even though you've used an api/function call hundreds of times before?  I ran into this the other day with node_load().  For those of you use the node_load function, can you tell me how many arguments it has and what those arguments do (without looking at api.drupal.org?  I know, some gurus can, but can the average module developer do it?)

In this instance, I created a module that was updating data from a MSSQL install on existing nodes.  After I did my node save I needed to do a node_load of the node I created/modified (some steps happened in between with other modules), modify some of the data and then save it again.  The only problem is, when I did my second node_save() my first save's data was wiped out.  After turning on debugging I found that my first (SQL) update worked properly but my second update was using data from before my first update.  How could that be, I did a second node_load (they were in different functions)?  It turns out that node_load was using a cached version of the node.

I'm not dogging the internals of node_load, it's a fine function and if I was a sailor on leave at a bar, I would ask it to dance.  In fact, I've had to employ it's caching abilities quite often so don't take this as some type of Drupal sucks post, I hate the core developers or cheesy link bait.

What I was doing isn't really the point of this post nor is how node_load works.  It's the argument order and the default argument that I'm against.  Yes, I'm writing a ~200 word post about three arguments in one function.  I don't like the fact that the third argument, "reset" is defaulted.  There, I've said it.  The entire problem I had (yes, it was my fault) would have been much easier to debug if the argument did not have a default value.  This way, when I called node_load(123) I would have received a PHP error stating that I was missing an argument for the function.  Think about it, if there is no default, you would have to put true or false there (in this case, it would have forced me to read the documents).

I am aware that having an API use sane defaults is SOP and this issue affects a fraction of a fraction of developers but I believe it's a bad practice that encourages programmers to not pay attention (I know, I know, I rush, I'm the king of this.  On the other hand don't most developers have some type of impending deadline?)

So what's the point of this rambling tirade?  In the small scope I want the order of node_load to be changed and one of the defaults to be removed (Drupal 8 perhaps).  For the big scope I would like to hear from the community about other functions that have defaults that might hide bugs.  I'm not just doing this because I'm too stupid to read the docs, I would like to move forward for D8 in reviewing the API for these types of gotchas that can seriously lower the frustration level of non-Drupal-guru developers.  Please let me know your thoughts.

Jan 30 2006
Jan 30

FINALLY finished the captcha.module for drupal. This is ONLY a first draft, lots of improvements to happen. Features:
* ability to protect any drupal form
* captcha API – make your own challenge response! (math and image are included in package)

(use cvs checkout to get)

About Drupal Sun

Drupal Sun is an Evolving Web project. It allows you to:

  • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
  • Facet based on tags, author, or feed
  • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
  • View the entire article text inline, or in the context of the site where it was created

See the blog post at Evolving Web

Evolving Web