Dec 13 2018
Dec 13
Mike and Matt talk with the team that helped implement content strategy on Georgia.gov.
Dec 05 2018
Dec 05

It's never too late to start thinking about user experience design when working on a project. To help ensure the project is a success, it's best to have a UX designer involved in the project as early as possible. However, circumstances may not always be ideal, and User Experience may become an afterthought. Sometimes it isn't until the project is already well on its way when questions around user experience start popping up, and a decision is made to bring in a professional to help craft the necessary solutions. 

What’s the best way for a UX designer to join a project that is well on its way? In this article, we will discuss some actions that UX designers can take to help create a smooth process when joining a project already in progress.

General Onboarding

Planning and implementing an onboarding process can help set the tone for the remainder of the project. If it’s disorganized and not well planned out, you can feel underprepared for the first task, which can lead to a longer design process. It’s helpful to designate a project team member to help with on-boarding. It should be someone who knows the project well and can help answer questions about the project and process. This is usually a product owner or a project manager but isn’t limited to either. If you haven’t been assigned someone to help you with the on-boarding process, reach out to help identify which team member would be best for this role. During the on-boarding process, discuss what user experience issues the team is hoping to solve, and also review the background of significant decisions that were made. This will help you to evaluate the current state of the project as well as the history of the decision-making process. You should also make sure you understand the project goals and the intended audience. Ask for any documentation around usability testing, acceptance criteria, competitive reviews, or notes for meetings that discuss essential features. Don’t be afraid ask questions to help you fully grasp the project itself. And don’t forget to ask why. Sometimes entertaining the mindset of a five-year-old when trying to understand will help you find the answers you’re seeking.

Process Evaluation

How you climb a mountain is more important than reaching the top. - Yvon Chouinard

Processes help ensure that the project goes smoothly, is on time, and on budget. They can also be a checkpoint for all those involved.  If a process doesn't already exist that includes UX Design, work together with the team to establish a process to discuss, track and review work. If you feel that a process step is missing or a current system isn't working, speak up and work with the team to revise it. Make sure to include any critical process that the team may be lacking. You also may want to make sure that discussions around any important features include a UX Designer. Ask if there are any product meetings that you should be joining to help give input as early as possible.

Schedule Weekly Design Reviews

One example of improving the process to include UX Design is scheduling weekly meetings to review design work that’s in progress. This also gives project members an opportunity to ask questions and discuss upcoming features and acceptance criteria.

Incorporate Usability Testing

Another suggestion is to include usability tests on a few completed important features before moving ahead. The results of the usability tests may help give direction or answer questions the product team has been struggling with. It can also help prioritize upcoming features or feature changes. The most important thing to remember is that usability testing can help improve the product, so it’s tailored to your specific users, and this should be communicated to the project team.

Collect General User Feedback

Establishing early on the best way to collect and give feedback on a design or feature can help streamline the design process. Should it be written feedback? Or would a meeting work better where everyone can speak up? Sometimes, when multiple people are reviewing and giving feedback, it’s best to appoint one person to collect and aggregate the input before it filters down to you.

Track Project Progress

You also want to discuss the best way to track work in progress. If your team is using an agile process, one idea is to include design tickets in the same software that you’re using to keep track of sprints such as Jira [link] or Trello [link]. Discuss the best way for summarizing features, adding acceptance criteria and tracking input in whatever system you decide to use.

Prioritization of Work

Efficiency is doing things right; effectiveness is doing the right things. - Peter Drucker

The team should be clear on priorities when it comes to work, features, and feedback. Joining a team that’s in progress can be very overwhelming to both the designers and stakeholders and creating clear priorities can help set expectations and make it clear to both sides on what the team should focus on first. If a list of priorities doesn't already exist, create one. It doesn't have to be fancy. A simple excel sheet or Google Sheets will do. You can create separate priority lists for things like upcoming features that need design, QA, or user feedback. You can also combine everything into a single list if that works better for your team. Just make sure that it links to or includes as much detail as possible. In the example below, a feature that has completed acceptance criteria is linked to a ticket in Jira that explains all of the details.

Google Sheets

It’s also helpful to group related features together, even though they may have different priorities. This will help you think about how to approach a feature without needing to reworking it later down the line. Be proactive. Ask questions around the priority of items if something doesn't make sense to you. If needed, volunteer to help prioritize features based on what makes sense for a holistic finished product or feature. Creating diagrams and flowcharts can help get everyone to understand how separate features can be connected and what makes the most sense to tackle first. Make sure that QA and user feedback is also part of the priority process.

Process flowchart

Summary

Having any team member join a project mid-process can be intimidating for all parties involved, but it’s important to be open and understanding. Improving the process and the end result is in everyone's interest, and giving and accepting feedback with an open mind can play an important role in ensuring that the project runs smoothly for everyone involved.

For User Experience Designers, it’s important to respect what’s already been accomplished and established with the idea that you should tread lightly to make small improvements at first. This will help gain confidence from the team, while also giving you time to learn about the project and understand the decisions that lead up to where it’s at today. For stakeholders involved, it’s important to listen with an open mind and take a small step back to reevaluate the best way to include UX in the process moving forward. The above suggestions can help both parties understand what actions they can take to help make the onboarding process for a UX Designer a smooth transition.

Nov 28 2018
Nov 28

We're excited to announce that 14 Lullabots will be speaking at DrupalCon Seattle! From presentations to panel discussions, we're looking forward to sharing insights and good conversation with our fellow Drupalers. Get ready for mass Starbucks consumption and the following Lullabot sessions. And yes, we will be hosting a party in case you're wondering. Stay tuned for more details!

Karen Stevenson, Director of Technology

Karen will talk about the challenges of the original Drupal AMP architecture, changes in the new branch, and some big goals for the future of the project.

Zequi Vázquez, Developer

Zequi will explore Drupal Core vulnerabilities, SA-CORE-2014-005 and SA-CORE-2018-7600, by discussing the logic behind them, why they present a big risk to a Drupal site, and how the patches work to prevent a successful exploitation.

Sally Young, Senior Technical Architect (with Matthew Grill, Senior JavaScript Engineer at Acquia & Daniel Wehner, Senior Drupal Developer at Chapter Three)

Discussing common problems and best practices of decoupled Drupal has surpassed the question of whether or not to decouple. Sally, Matthew, and Daniel will talk about why the Drupal Admin UI team went with a fully decoupled approach as well as common approaches to routing, fetching data, managing state with autosave and some level of extensibility.

Sally Young, Senior Technical Architect (with Lauri Eskola, Software Engineer in OCTO at Acquia; Matthew Grill, Senior JavaScript Engineer at Acquia; & Daniel Wehner, Senior Drupal Developer at Chapter Three)

The Admin UI & JavaScript Modernisation initiative is planning a re-imagined content authoring and site administration experience in Drupal built on modern JavaScript foundations. This session will provide the latest updates and a discussion on what is currently in the works in hopes of getting your valuable feedback.

Greg Dunlap, Senior Digital Strategist

Greg will take you on a tour of the set of tools we use at Lullabot to create predictable and repeatable content inventories and audits for large-scale enterprise websites. You will leave with a powerful toolkit and a deeper understanding of how you use them and why.

Mike Herchel, Senior Front-end Developer

If you're annoyed by slow websites, Mike will take you on a deep dive into modern web performance. During this 90 minute session, you will get hands-on experience on how to identify and fix performance bottlenecks in your website and web app.

Matt Westgate, CEO & Co-founder

Your DevOps practice is not sustainable if you haven't implemented its culture first. Matt will take you through research conducted on highly effective teams to better understand the importance of culture and give you three steps you can take to create a cultural shift in your DevOps practice. 

April Sides, Developer

Life is too short to work for an employer with whom you do not share common values or fits your needs. April will give you tips and insights on how to evaluate your employer and know when it's time to fire them. She'll also talk about how to evaluate a potential employer and prepare for an interview in a way that helps you find the right match.

Karen Stevenson, Director of TechnologyMike Herchel, Senior Front-end DeveloperWes Ruvalcaba, Senior Front-end Developer, & Ellie Fanning, Head of Marketing

Karen, Mike, Wes, and team built a soon-to-be-launched Drupal 8 version of Lullabot.com as Layout Builder was rolling out in core. With the goal of giving our non-technical Head of Marketing total control of the site, lessons were learned and successes achieved. Find out what those were and also learn about the new contrib module Views Layout they created.

Matthew Tift, Senior Developer

The words "rockstar" and "rock star" show up around 500 times on Drupal.org. Matthew explores how the language we use in the Drupal community affects behavior and how to negotiate these concepts in a skillful and friendly manner.

Helena McCabe, Senior Front-end Developer (with Carie Fisher, Sr. Accessibility Instructor and Dev at Deque)

Helena and Carie will examine how web accessibility affects different personas within the disability community and how you can make your digital efforts more inclusive with these valuable insights.

Marc Drummond, Senior Front-end Developer Greg Dunlap, Senior Digital Strategist (with Fatima Sarah Khalid, Mentor at Drupal Diversity & Inclusion Contribution Team; Tara King, Project lead at Drupal Diversity & Inclusion Contribution Team; & Alanna Burke, Drupal Engineer at Kanopi Studios)

Open source has the potential to transform society, but Drupal does not currently represent the diversity of the world around us. These members of the Drupal Diversity & Inclusion (DDI) group will discuss the state of Drupal diversity, why it's important, and updates on their efforts.

Mateu Aguiló Bosch, Senior Developer (with Wim Leers, Principal Software Engineer in OCTO at Acquia & Gabe Sullice, Sr. Software Engineer, Acquia Drupal Acceleration Team at Acquia)

Mateu and his fellow API-first Initiative maintainers will share updates and goals, lessons and challenges, and discuss why they're pushing for inclusion into Drupal core. They give candy to those who participate in the conversation as an added bonus!

Jeff Eaton, Senior Digital Strategist

Personalization has become quite the buzzword, but the reality in the trenches rarely lives up to the promise of well-polished vendor demos. Eaton will help preserve your sanity by guiding you through the steps you should take before launching a personalization initiative or purchasing a shiny new product. 

Also, from our sister company, Drupalize.Me, don't miss this session presented by Joe Shindelar:

Joe will discuss how Gatsby and Drupal work together to build decoupled applications, why Gatsby is great for static sites, and how to handle private content, and other personalization within a decoupled application. Find out what possibilities exist and how you can get started.

Photo by Timothy Eberly on Unsplash

Nov 28 2018
Nov 28

At Lullabot, we’ve been using GitHub, as well as other project management systems for many years now. We first wrote about managing projects with GitHub back in 2012 when it was still a bit fresh. Many of those guidelines we set forth still apply, but GitHub itself has changed quite a bit since then. One of our favorite additions has been the Projects tab, which gives any repository the ability to organize issues onto boards with columns and provides some basic workflow transitions for tickets. This article will go over one of the ways we’ve been using GitHub Projects for our clients, and set forth some more guidelines that might be useful for your next project.

First, let’s go over a few key components that we’re using for our project organization. Each of these will be explained in more detail below.

  1. Project boards
  2. Epics
  3. Issues
  4. Labels

Project boards

A project board is a collection of issues being worked on during a given time. This time period is typically what is being worked on currently, or coming up in the future. Boards have columns which represent the state of a given issue, such as “To Do”, “Doing”, “Done”, etc.

For our purposes, we’ve created two main project boards:

  1. Epics Board
  2. Development Board

Epics Board

ex: https://github.com/Lullabot/PM-template/projects/1

The purpose of this Project board is to track the Epics, which can be seen as the "parent" issues of a set of related issues. More on Epics below. This gives team members a birds-eye view of high-level features or bodies of work. For example, you might see something like “Menu System” or “Homepage” on this board and can quickly see that “Menu System” is currently in “Development”, while the “Homepage” is currently in “Discovery”.

The “Epics” board has four main columns. (Each column is sorted with highest priority issues at the top and lower priority issues at the bottom.) The four columns:

  • Upcoming - tracks work that is coming up, and not yet defined.
  • Discovery - tracks work that is in the discovery phase being defined.
  • Development - tracks work that is currently in development.
  • Done - tracks work that is complete. An Epic is considered complete when all of its issues are closed.

Development Board

ex: https://github.com/Lullabot/PM-template/projects/2

The purpose of the Development board is to track the issues which are actionable by developers. This is the day-to-day work of the team and the columns here are typically associated with some state of progression through the board. Issues on this board are things like “Install module X”, “Build Recent Posts View”, and “Theme Social Sharing Component”.

This board has six main columns:

  • To do - issues that are ready to be worked on - developers can assign themselves as needed.
  • In progress - indicates that an issue is being worked on.
  • Peer Review - issue has a pull request and is ready for, or under review by a peer.
  • QA - indicates that peer review is passed and is ready for the PM or QA lead for testing.
  • Stakeholder review - stakeholder should review this issue for final approval before closing.
  • Done - work that is complete.

Epics

An Epic is an issue that can be considered the "parent" issue of a body of work. It will have the "Epic" label on it for identification as an Epic, and a label that corresponds to the name of the Epic (such as "Menu"). Epics list the various issues that comprise the tasks needed to accomplish a body of work. This provides a quick overview of the work in one spot. It's proven very useful when gardening the issue queue or providing stakeholders with an overall status of the body of work.

For instance:

Homepage [Epic]

  • Tasks

    • #4 Build Recent Posts View
    • #5 Theme Social Sharing Component

The Epic should also have any other relevant links. Some typical links you may find in an Epic:

  • Designs
  • Wiki entry
  • Dependencies
  • Architecture documentation
  • Phases

Phases

Depending on timelines and the amount of work, some Epics may require multiple Phases. These Phases are split up into their own Epics and labeled with the particular Phase of the project (like “Phase 1” and “Phase 2”). A Phase typically encompasses a releasable state of work, or generally something that is not going to be broken but may not have all of the desired functionality built. You might build out a menu in Phase 1, and translate that menu in Phase 2.

For instance:

  • Menu Phase 1

    • Labels: [Menu] [Epic] [Phase 1]
    • Tasks
    • Labels: [Menu] [Phase 1]
  • Menu Phase 2

    • Labels: [Menu] [Epic] [Phase 2]
    • Tasks
    • Labels: [Menu] [Phase 2]
  • Menu Phase 3

    • Labels: [Menu] [Epic] [Phase 3]
    • Tasks
    • Labels: [Menu] [Phase 3]

Issues within Phase 3 (for example) will have the main epic as a label "Menu" as well as the phase, "Phase 3", for sorting and identification purposes.

Issues

Issues are the main objects within GitHub that provide the means of describing work, and communicating around that work. At the lowest level, they provide a description, comments, assignees, labels, projects (a means of placing an issue on a project board) and milestones (or a means of grouping issues by release target date).

Many times these issues are directly linked to from a pull request that addresses the issue. By mentioning the issue with a pound(#) sign, GitHub will automatically create a link out of the text and add a metadata item on the issue deep linking to the pull request. This is relevant as a means of tracking what changes are being made with the original request which then can be used to get to the source of the request.

For our purposes, we have two "types" of issues: Epics or Tasks. As described above, Epics have the "Epic" label, while all others have a label for the Epic to which it belongs. If an issue does not have a value in the "Project" field, then it does not show up on a project board and is considered to be in the Backlog and not ready for work.

Labels

Labels are a means of having a taxonomy for issues.

We have 4 main uses for Labels currently:

  1. Epic - this indicates the issue is an Epic and will house information related to the body of work.
  2. [name of epic] (ex: Menu) - indicates that this is a task that is related to the Menu epic. If combined with the Epic label, it is the Menu Epic.
  3. [phase] (ex: Phase 1) - indicates this is part of a particular phase of work. if there is no phase label, it's considered to be a part of Phase 1.
  4. bug - indicates that this task is a defect that was found and separated from the issue in which it was identified.
  5. Blocked - Indicates this issue is blocked by something. The Blocker should be called out in the issue description.
  6. Blocker - indicates that this issue is blocking something.
  7. front-end - indicates that an issue has the underlying back-end work completed and is ready for a front-end developer to begin working on it.

There are other labels that are used sometimes to indicate various meta, such as "enhancement", "design", or "Parking Lot". There are no set rules about how to use these sort of labels, and you can create them as you see fit if you think they are useful to the team. Though be warned, if you include too many labels they will become useless. Teams will generally only use labels if they are frictionless and helpful. The moment they become overwhelming, duplicative, or unclear, the team will generally abandon good label hygiene.

These are just some guidelines we consider when organizing a project with GitHub. The tools themselves are flexible and can take whatever form you choose. This is just one recommendation which is working pretty well for us one of our projects, but the biggest takeaway is that it’s versatile and can be adapted to whatever your situation may require.

How have you been organizing projects in GitHub? We’d love to hear about your experiences in the comments below!

Oct 11 2018
Oct 11
Mike and Matt interview members of the Drupal 8 JavaScript modernization initiative to find out what's going on, and the current status.
Sep 27 2018
Sep 27
Mike and Matt talk with the Drupal Association's Senior Events Manager, Amanda Gonser, about upcoming changes to DrupalCon events.
Sep 06 2018
Sep 06

Mike and Matt are joined by Joe Shindelar from Drupalize.Me and Baddý Breidert​ one of the organizers of Drupal Europe, a huge conference that's being billed as "A family reunion for the Drupal community."

Drupal Europe is put on by a huge group of community volunteers in collaboration with the German Drupal Association.

Aug 22 2018
Aug 22

I came across the following error the other day on a client's Drupal 8 website:

LogicException: The controller result claims to be providing relevant cache metadata, but leaked metadata was detected. Please ensure you are not rendering content too early.

Leaked? That sounded bad. Rendering content too early? I didn't know what that meant, but it also sounded bad. Worst of all, this was causing a PHP fatal error along with a 500 response code. Fortunately, I caught the error during development, so there was time to figure out exactly what was going on. In so doing, I learned some things that can deepen our understanding of Drupal’s cache API.

Down the rabbit hole

I knew that this error was being caused by our code. We were writing a custom RestResource plugin, which is supposed to fetch some data from the entity API and return that data, ready to be serialized and complete with cacheability metadata. This custom RestResource was the only route that would trigger the error, and it only started happening part way through development as the codebase grew complex. It had been working fine, until the error noted above, which I include here in full with a stack trace:

The website encountered an unexpected error. Please try again later. LogicException: The controller result claims to be providing relevant cache metadata, but leaked metadata was detected. Please ensure you are not rendering content too early. Returned object class: Drupal\rest\ResourceResponse. in Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->wrapControllerExecutionInRenderContext() (line 154 of core/lib/Drupal/Core/EventSubscriber/EarlyRenderingControllerWrapperSubscriber.php). Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->Drupal\Core\EventSubscriber\{closure}() (Line: 135)
Symfony\Component\HttpKernel\HttpKernel->handleRaw(Object, 1) (Line: 57)
Symfony\Component\HttpKernel\HttpKernel->handle(Object, 1, 1) (Line: 57)
Drupal\Core\StackMiddleware\Session->handle(Object, 1, 1) (Line: 47)
Drupal\Core\StackMiddleware\KernelPreHandle->handle(Object, 1, 1) (Line: 119)
Drupal\cdn\StackMiddleware\DuplicateContentPreventionMiddleware->handle(Object, 1, 1) (Line: 47)
Drupal\Core\StackMiddleware\ReverseProxyMiddleware->handle(Object, 1, 1) (Line: 50)
Drupal\Core\StackMiddleware\NegotiationMiddleware->handle(Object, 1, 1) (Line: 23)
Stack\StackedHttpKernel->handle(Object, 1, 1) (Line: 663)
Drupal\Core\DrupalKernel->handle(Object) (Line: 19)

I was confused that our code didn't appear in the stack trace; this is all Drupal core code. We need to go deeper.

As I do when this kind of situation arises, I took to the debugger. I set a breakpoint at the place in core where the exception was being thrown looking for clues. Here were my immediate surroundings:

// ...
elseif ($response instanceof AttachmentsInterface || $response instanceof CacheableResponseInterface || $response instanceof CacheableDependencyInterface) {
  throw new \LogicException(sprintf('The controller result claims to be providing relevant cache metadata, but leaked metadata was detected. Please ensure you are not rendering content too early. Returned object class: %s.', get_class($response)));
}
// ...

Foreign land. Knowing a smidgen about the Cache API in Drupal 8 and the context of what we were trying to do, I understood that we were ending up here in part because we were returning a response object that has cacheability metadata on it. That is, we were returning a ResourceResponse object that implements CacheableResponseInterface, including the relevant cacheability metadata with it. I could see from Xdebug that the $response variable in the snippet above corresponded to the ResourceResponse object we were returning, and it was packed with our data object and ready to be serialized. 

Xdebug screenshot showing expected domain object with cacheability metadata

So as far as I knew, I was playing nice and adding cacheability metadata like a good Drupal developer should. What gives?

Seeing the forest for the trees

It was at this point I felt myself getting lost in the weeds. I needed to take a step back and reread the error message. When I did, I realized that I didn't understand what “early rendering” was.

I knew it had some connection to caching, so I started by reading through all the Cache API docs on drupal.org. I’ve read these several times in the past, but it’s just one of those topics, at least for me, that requires constant reinforcement. Another relevant doc I found was CachebleResponseInterface. These provided a good background and laid out some terminology for me, but nothing here talks about early rendering. I also reviewed the Render API docs but again, no mention of early rendering, and nothing getting me closer to a resolution.

So then I zoomed back in a little bit, to the parent class of the code which threw the error: \Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber

As is often the case in Drupal 8 core code, there was an excellent and descriptive doc block for the class. I often find this to be key to understanding Drupal 8. Core committers take great care to document the code they write which makes it worth getting comfortable with reading through core and contrib code.

When controllers call drupal_render() (RendererInterface::render()) outside of a render context, we call that "early rendering." Controllers should return only render arrays, but we cannot prevent controllers from doing early rendering. The problem with early rendering is that the bubbleable metadata (cacheability & attachments) are lost.

At last a definition for early rendering! However, our code wasn't (at least directly) inside a controller, it never called drupal_render() that I could tell, and what in the world is a render context?

Nobody to blame

Still, in need of some context for understanding what was going on here, I looked at git blame to find out where the code that was throwing the error about early rendering came from. Ever since I started to do Drupal 8 development, I’ve always found it useful to use a clone of Drupal locally for such occasions. PHPStorm makes using git blame quite easy. In the file you’re interested in—opened in the editor—just right click the line numbers column and click Annotate. Once the annotations display, click the one that corresponds to the line that you’re interested in to see the commit message. 

Showing how to access git blame within PHPStorm

Most, if not all, Drupal core commits will have an issue number in the description, in this case, here is what I found:

Issue #2450993 by Wim Leers, Fabianx, Crell, dawehner, effulgentsia: Rendered Cache Metadata created during the main controller request gets lost

Loading up the issue, I’m faced with a wall of text, 159 comments. Although I did eventually wade through it out of morbid curiosity, what I immediately do when faced with a giant closed core issue, is check for a change record. The Drupal 8 dev cycle has been really excellent about documenting changes, and change records have really helped in the transition from earlier Drupal 7 concepts and explaining new concepts in Drupal 8. For any core issue, first, take a look in the right sidebar of the issue for “Change records for this issue”, and follow any that are linked to get a birds-eye view of the change. If you haven’t already, it’s also handy to bookmark the Change records for Drupal core listing as it's a great place to look when you're stuck on something Drupal 8.

Showing a Drupal core issue with links to the corresponding change record in the sidebar

The change record was very helpful, so if you’re interested, I recommend you definitely give it a read. In short, early rendering used to be rampant (in core and contrib), and this was a problem because cacheability metadata was lost. The change introduced a way to wrap all controllers, detect early rendering, catch and merge the cacheability metadata into the controllers' return (usually a render array). That’s all well and good, but wait! You might think, "If it’s handling the cacheabillity metadata from early rendering, why is it still throwing an error!?" Well, going back to the snippet where the exception is thrown from earlier:

// ...
elseif ($response instanceof AttachmentsInterface || $response instanceof CacheableResponseInterface || $response instanceof CacheableDependencyInterface) {
  throw new \LogicException(sprintf('The controller result claims to be providing relevant cache metadata, but leaked metadata was detected. Please ensure you are not rendering content too early. Returned object class: %s.', get_class($response)));
}
// ...

What this boils down to is if your controller is returning a response object of type, AttachementsInterfaceCacheableResponseInterface, or CacheableDependencyInterface, Drupal does not give you a pass, nor does it handle cacheability metadata from early rendering for you. Drupal takes the position that since you are returning this type of response, you should also be responsible, be aware of and handle early rendering yourself. From the change log:

Since you're returning responses, you want to fully control what is sent, so you should also be a responsible citizen and not do any early rendering.

I solemnly swear not to early render

Ok, so no early rendering, got it. But, what if it’s out of our control? In our case, the code we were working in didn't have any direct calls to drupal_render() (RendererInterface::render()). My next tactic was to understand more about what was triggering early rendering. 

To do this, I set a breakpoint in the sole implementation of RendererInterface::render() and then hit the REST endpoint that was triggering the error. Xdebug immediately broke at that line, and inspecting the stack trace, we saw some of our code! Proof that we broke it! Progress. 

Xdebug screenshot, stopped on the render method showing our code triggering early rendering

As it turns out, some code in another custom module was being called. This code is meant to wrap entity queries, massaging the return data into something more palpable and concise for the development team that wrote it. Deep in this code, in processing node entities, it was including a call to $node→url(), where $node is a \Drupal\node\Entity\Node object. Turns out, that triggers early rendering. To this, you might ask, "Why would something as innocuous as getting the URL for a node trigger early rendering?" The answer, and I’m only 80% sure after studying this for a while (do correct me if I’m wrong), is that URLs can vary by context based on language or the site URL. They can also have dependencies, such as the language configuration. Finally, URLs can have CSRF tokens embedded in them, which vary by session. All of which is important cacheability metadata that you want to be included in the response. OK, so what’s a responsible Drupal developer to do?

The complete (and verbose) solution, courtesy of ohthehugemanatee (indeed), is to replace your $node→url() call with something like:

// 1. Confusing: the method is called toString, yet passing TRUE for the first param nets you a \Drupal\Core\GeneratedUrl object.
$url = $node->toUrl()->toString(TRUE);
// 2. The generated URL string, as before.
$url_string = $url->getGeneratedUrl();
// Add the $url object as a dependency of whatever you're returning. Maybe a response?
$response = new CacheableResponse($url_string, Response::HTTP_OK);
$response->addCacheableDependency($url);
return $response;

That’s a lot, and it’ll be different depending on what you're doing. It’s broken down into 3 parts. First, you want to call $node->toUrl()->toString(TRUE);. This will essentially tell Drupal to track any cacheability metadata, which is part of generating the URL, and return an object from which you can get that cacheability metadata so you can deal with it. The second part is just getting the actual URL string, $url_string = $url->getGeneratedUrl();, to do with as you please. Finally, you need to account for any encountered cacheability metadata. In the context of a response as above, that means adding the $url object as a cacheable dependency. In the context of a render array, it might mean merging the $url cacheability metadata into the render array. (eg. CacheableMetadata::createFromObject($url)→applyTo($render_array))

Wrap it up

OK so now I understood where the exception was coming from and why. I also understand how I might change the code that is triggering an early rendering. But as I mentioned before, what if you don’t control the code that is triggering an early rendering? Is all hope lost? Not quite. What you can do is wrap the code triggering the early render in a render context. Let’s look at some code:

$context = new RenderContext();
/* @var \Drupal\Core\Cache\CacheableDependencyInterface $result */
$result = \Drupal::service('renderer')->executeInRenderContext($context, function() {
  // do_things() triggers the code that we don't don't control that in turn triggers early rendering.
  return do_things();
});
// Handle any bubbled cacheability metadata.
if (!$context->isEmpty()) {
  $bubbleable_metadata = $context->pop();
  BubbleableMetadata::createFromObject($result)
    ->merge($bubbleable_metadata);
}

Let’s break this down:

$context = new RenderContext();

Here, I instantiate a new render context. A render context is a stack containing bubbleable rendering metadata. It’s a mechanism for collecting cacheability metadata recursively, aggregating or “bubbling” it all up. By creating and passing it in the next line, the render context is able to capture what would have otherwise been lost cacheability metadata.

/* @var \Drupal\Core\Cache\CacheableDependencyInterface $result */
$result = \Drupal::service('renderer')->executeInRenderContext($context, function() {
  // do_things() triggers the code that we don't don't control that in turn triggers early rendering.
  return do_things();
});

Here I run some arbitrary code within the render context I created. The arbitrary code, somewhere along its execution path that we have no control over, triggers early rendering. When that early rendering occurs, since I’m wrapping the code in a render context, the cacheability metadata will bubble up to the render context I setup and allow me to do something with it.

// Handle any bubbled cacheability metadata.
if (!$context->isEmpty()) {
  $bubbleable_metadata = $context->pop();
  BubbleableMetadata::createFromObject($result)
    ->merge($bubbleable_metadata);
}

Now I check if the context is non-empty. In other words, did it catch some cacheability metadata from something that did early rendering? If it did, I get the captured cacheability metadata with $context→pop() and merge it with my \Drupal\Core\Cache\CacheableDependencyInterface object which will be returned. BubbleableMetadata is a helper class for dealing with cacheability metadata. This merge part may look different depending on your context, but the idea is to incorporate it into the response somehow. Take a look at the static methods in \Drupal\Core\Render\BubbleableMetadata  and its parent class for\Drupal\Core\Cache\CacheableMetadata some helpers to merge your cacheability metadata.

Really wrapping up

That was a heavy, long, complex debug session. I learned a lot digging into it and I hope you did as well. Let me know in the comments if you’ve ever run into something similar and came to a resolution in a different way. I’d love to continue furthering my understanding.

While it was great to figure this out, I was left wanting a better DX. In particular, improving the fact that Drupal auto-magically handles early rendering in some cases, but not others. There is also the odd workaround to capture cacheability metadata when cacheability metadata when calling $node→url() that could use some work. A quick search on the issue queue told me I wasn’t alone. Hopefully, with time and consideration, this can be made better. Certainly, there are good reasons for the complexity, but it would be great to balance that against the DX to avoid more epic debug sessions.

Acknowledgements

Aug 16 2018
Aug 16
Mike and Matt are joined by Lullabot John Hannah to talk with the creator of GatsbyJS
Aug 01 2018
Aug 01

BigPipe is a technique pioneered by Facebook that’s used to lazy-load content into a webpage. From the user’s perspective, the “frame” of a webpage will appear immediately, and then the content will pop in place when it’s ready. BigPipe has been included as a module in Drupal core since 8.1.x, and is very simple— just enable the module.

On my latest project, I'm using it to lazy-load content that’s generated from a very slow API call. The functionality works great out of the box, but we noticed a user-experience problem where the end-user would see a big blank area while the API call was waiting on a response. This behavior made the website seem broken. To fix this, we decided to implement a simple loading animation.

Finding the CSS selector to attach the animation to wasn’t as simple as I hoped it would be.

Spoiler: Let’s see the code

Looking for the code, and not the process? The CSS selector to target is below. Note that you’ll want to qualify this within a parent selector, so the loader doesn’t appear everywhere.


.parent-selector [data-big-pipe-placeholder-id] {
  /* Loading animation CSS */
}

BigPipe’s placeholder markup is only one <span> element, which makes styling tricky. Luckily, we can make use of CSS pseudo-selectors to make a Facebook-style throbber animation.

Here is some Sass with easy-to-use variables:


$pulse-duration: 0.2s;
$pulse-color: rebeccaPurple;

@keyframes pulse-throbber {
  0% { 
    opacity: 1;
    transform: scaley(1);
  }

  100% { 
    opacity: 0.2; 
    transform: scaley(0.5);
  }
}


[data-big-pipe-placeholder-id] {
  position: relative;
  display: block;
  margin: 20px auto;
  width: 6px;
  height: 30px;
  background: $pulse-color;
  animation: pulse-throbber $pulse-duration infinite;
  animation-delay: ($pulse-duration / 3);
  animation-direction: alternate;

  &:before,
  &:after {
    content: '';
    position: absolute;
    display: block;
    width: 100%;
    height: 100%;
    background: $pulse-color;
    top: 0;
    animation: pulse-throbber $pulse-duration infinite;
    animation-direction: alternate;
  }

  &:before {
    left: -12px;
  }

  &:after {
    left: 12px;
    animation-delay: ($pulse-duration / 1.5);
  }
}

Tracking down the placeholder’s CSS selector

Finding this selector wasn’t as simple as I initially hoped. The first technique that I tried was setting a DOM breakpoint in Chrome Developer Tools. This functionality allows you to pause the execution of JavaScript when a DOM element’s attributes change, the element gets removed, or any descendant DOM elements are modified.

In our case, we want to set a breakpoint when any descendant element is modified and then reload the page. Hopefully, when BigPipe inserts the rendered HTML, the breakpoint will trigger, and we can then inspect the placeholder HTML to find the appropriate CSS selector.

Setting DOM breakpoints within Chrome Developer ToolsSetting DOM breakpoints within Chrome Developer Tools

Unfortunately, this didn’t work. Why? I’m still not sure. This appears to be a bug within Google Chrome. I created an issue within the Chromium bug tracker and will update this article when there’s progress.

PHP Breakpoints to the rescue!

Because I know I’m using the BigPipe module to stream the content in, the next step is setting a PHP breakpoint within the BigPipe module within PHPStorm. I ended up setting a breakpoint within the sendContent() function within BigPipeResponse.php. This had the expected result of pausing the lazy-loading of the content, which enabled me to easily inspect the HTML (by halting the injection of the BigPipe content) so I could find the placeholder’s selector.

PHP breakpoint within the BigPipe modulePHP breakpoint within the BigPipe module Yay! We can finally see the placeholder HTMLYay! We can finally see the placeholder HTML

Conclusion

Sometimes a seemingly simple theming task ends up being tricky. It’s important to understand proper front-end and backend debugging techniques because you never know when you’re going to need them in a pinch. Hopefully, this article will save someone from having to go through this process.

Photo by Jonny Caspari on Unsplash

Jul 25 2018
Jul 25

If you build or manage public-facing websites, you've almost certainly heard the excited buzz around personalization technology. Content marketers, enthusiastic CEOs, and product vendors all seem to agree that customizing articles, product pitches, and support materials to each visitor's interests — and delivering them at just the right time — is the new key to success.

Content personalization for the web isn't new, and the latest wave of excitement isn't all hype; unfortunately, the reality on the ground rarely lives up to the promise of a well-produced sales demo. Building a realistic personalization strategy for your website, publishing platform, or digital project requires chewing on several foundational questions long before any high-end products or algorithms enter the picture.

The good news is that those core issues are more straightforward than you might think. In working with large and small clients on content tailoring and personalization projects, we've found that focusing on four key issues can make a huge difference.

1. Signals: Information You Have Right Now

A lot of conversations about personalization focus on interesting and novel things that we can discover about a website visitor: where they're currently located, whether they're a frequent visitor or a first-timer, whether they're on a mobile device, and so on. Before you can reliably personalize content for a given user, you must be able to identify them using the signals you have at your disposal. For example, building a custom version of your website that's displayed if someone is inside your brick-and-mortar store sounds great, but it's useless if you can't reliably determine whether they're inside your store or just in the same neighborhood.

Context

The simplest and most common kinds of signals are contextual information about a user's current interaction with your content. Their current web browser, the topic of the article they're reading, whether they're using a mobile device, their time zone, the current date, and so on are easy to determine in any publishing system worth its salt. These small bits of information are rarely enough to drive complex content targeting, but they can still be used effectively. Bestbuy.com, for example, uses visitor location data to enhance their navigation menu with information about their closest store, even if you've never visited before.

A picture of the Best Buy navigation bar, with the location of the closest store.

Identity

A picture of the Netflix login page, with two user profiles to choose from.

Moving beyond transient contextual cues requires knowing (and remembering) who the current visitor is. Tracking identity doesn't necessarily mean storing personal information about them: it can be as simple as storing a cookie on their browser to keep track of their last visit. At the other end of the spectrum, sites that want to encourage long-term return visits, or require payment information for products or services, usually allow users to create an account with a profile. That account becomes their identity, and tracking (or simply asking for) their preferences is a rich source of personalization signals. Employee intranets or campus networks that use single-sign on services for authentication have already solved the underlying "identity" problem — and usually have a large pool of user information accessible to personalization tools via APIs.

Behavior

Once you can identify a user reliably, tracking their actions over multiple visits can help build a more accurate picture of what they're looking for. Common scenarios include tracking what topics they read about most, which products they tend to purchase (or browse and reject), whether they prefer to visit in the morning or late at night, and so on. As with most of the building blocks of personalization, it's important to remember that this data is a limited view of what's happening: it tracks what they do, not necessarily what they want or need. Content Strategist, Karen McGrane sometimes tells the story of a bank whose analytics suggested that no one used their the site's "Find an ATM" tool. Further investigation revealed that the feature was broken; users had learned to ignore it, even though they wanted the information.

Consumer Databases

Some information is impossible to determine from easily available signals — which leads us to the sketchy side of the personalization tracks. Your current visitor's salary, their political views, whether they're trying to have a child, and whether they're looking for a new job are all (thankfully) tough to figure out from simple signals. Third-party marketing agencies and advertising networks, though, are often willing to sell access to their databases of consumer information. By using tools like browser fingerprinting, these services can locate your visitors in their databases, allowing your users to be targeted for extremely tailored messages.

The downside, of course, is that it's easy to slide into practices that unsettle your audience rather than engaging them. Increasingly, privacy-conscious users resent the "unearned intimacy" of personalization that's obviously based on information they didn't choose to give you. Europe's GPDR, a comprehensive set of personal data-protection regulations in effect since May 2018, can also make these aggressive targeting strategies legally dangerous. When in doubt, stick to data you can gather yourself and consult your lawyer. Maybe an ethicist, too.

2. Segments: Conclusions You Draw Based on Your Information

Individually, few of the individual signals we've talked about so far are useful enough to build a personalization strategy around. Collectively, though, all of them can be overwhelming: building targeted content for every combination of them would require millions of variations for each piece of content. Segmenting is the process of identifying particular audiences for your tailored content, and determining which signals you'll use to identify them.

It's easy to assume the segments you divide your audience into will correspond to user personas or demographic groups, but different approaches are often more useful for content personalization. Knowing that someone is a frequent flyer in their early 30s, for example, might be less useful for crafting targeted messages than knowing that they're currently traveling.

On several recent projects, we've seen success in tailoring custom content for scenarios and tasks rather than audience demographics or broad user personas. Looking at users through lenses like "Friend of a customer," "browsing for ideas" or "comparison-shopper" may require a different set of signals, but the usefulness of the resulting segments can be much higher.

Radical Truth

It's hard to overstate the importance of honesty at this point: specifically, honesty with yourself about the real-world reliability of your signal data and the validity of the assumptions you're drawing from it. Taking a visitor's location into account when they search for a restaurant is great, but it only works if they explicitly allow your site to access their location. Refusing to deal with spotty signal data gracefully often results in badly personalized content that's even less helpful than the "generic" alternative. Similarly, treating visitors as "travelers" if they use a mobile web browser is a bad assumption drawn from good data, and the results can be just as counterproductive.

3. Reactions: Actions You Take Based on Your Conclusions

In isolation, this aspect of the personalization puzzle seems like a no-brainer. Everyone has ideas about what they'd love to change on their site to make it appeal to specific audiences better, or make it perform more effectively in certain stress cases. It's exciting stuff — and often overwhelming. Without ruthless prioritization and carefully phased roll-outs, it's easy to triple or quadruple the amount of content that an already-overworked editorial team must produce. If your existing content and marketing assets aren't built from consistent and well-structured content, time-consuming "content retrofits" are often necessary as well.

Incentivization

The ever-popular coupon code is a staple of e-Commerce sites, but offering your audience incentives based on signal and segmenting data can cover a much broader range of tactics. Giving product discounts based on time from last purchase and giving frequent visitors early access to new content can help increase long-term business, for example. Creating core content for a broad audience, then inserting special deals and tailored calls to action, can also be easier than building custom content for each scenario.

Recommendation

Very little of the content on your site is meant to be a user's final destination. Whether you're steering them towards the purchase of a subscription service, trying to keep them reading and scrolling through an ad-supported site, or presenting a mall's worth of products on a shopping site, lists of "additional content" are a ubiquitous part of the web. Often, these lists are generated dynamically by a CMS or web publishing tool — and taking user behavior and signals into account can dramatically increase their effectiveness.

A picture of Amazon.com book recommendations based on user browsing history

The larger the pool of content and the more metadata that's used to categorize it, the better these automated recommendation systems perform. Amazon uses detailed analytics data to measure which products customers tend to purchase after viewing a category — and offers visitors quick links to those popular buys. Netflix hired taxonomists to tag their shows and movies based on director, genre, and even more obscure criteria. The intersections of those tags are the basis of their successful micro-genres, like "Suspenseful vacation movies" or "First films by award-winning directors."

Prioritization

One of the biggest dangers of personalization is making bad assumptions about what a user wants, and making it more difficult in the name of "tailoring" their experience. One way to sidestep the problem is offering every visitor the same information but prioritizing and emphasizing different products, messages, and services. When you're confident in the value of your target audience segments, but you're uncertain about the quality of the signal data you're using to match them with a visitor, this approach can reduce some of the risks.

Dynamic Assembly

A picture of a web page built from small components

Hand-building custom content for each personalization scenario is rarely practical. Even with aggressively prioritized audience segments, it's easy to discover that key pages might require dozens or even hundreds of variations. Breaking up your content into smaller components and assembling it on the fly won't reduce the final number of permutations you're publishing, but it does make it possible to assemble them out of smaller, reusable components like calls to action, product data, and targeted recommendations. One of our earliest (and most ambitious) personalization projects used this approach to generate web-based company handbooks customized for hundreds of thousands of individual employees. It assembled insurance information, travel reimbursement instructions, localized text, and more based on each employee's Intranet profile, effectively building them a personalized HR portal.

That level of componentized content, however, often comes with its own challenges. Few CMS's out-of-the-box editorial tools are well-suited to managing and assembling tiny snippets rather than long articles and posts. Also, dynamic content assembly demands a carefully designed and enforced style guide to ensure that all the pieces match up once they're put together.

4. Metrics: Things You Measure to Judge the Reactions' Effectiveness

The final piece of the puzzle is something that's easy to do, but hard to do well: measuring the effectiveness of your personalization strategy in the real world. Many tools — from a free Google Analytics account to Adobe Sharepoint — are happy to show you graphs and charts, and careful planning can connect your signals and segments to those tools, as well. Machine learning algorithms are increasingly given control of A/B testing the effectiveness of different personalization reactions, and deciding which ones should be used for which segments in the future. What they can't tell you (yet) is whether what you're measuring matters.

It's useful to remember Goodhart's Law, coined by a British economist designing tools to weigh the nation's economic health. "When a measure becomes a target, it ceases to be a good measure." Increased sales, reduced support call volume, happier customers, and more qualified leads for your sales team may be hard to measure on the Google Analytics dashboard, but finding ways to measure data that's closer to those measures of value than the traditional "bounce rate" and "time on page" numbers will get you much closer. Even more importantly, don't be afraid to change what you're measuring if it becomes clear that "success" by the analytics numbers isn't helping the bottom line.

Putting It All Together

There's quite a bit to chew on there, and we've only scratched the surface. To reiterate, every successful personalization project needs a clear picture of the signals you'll use to identify your audience, the segments you'll group them into for special treatment, the specific approaches you'll use to tailor the content, and the metrics you'll use to judge its effectiveness. Regardless of which tool you buy, license, or build from scratch, keeping those four pillars in mind will help you navigate the sales pitches and plan for an effective implementation.

Jul 19 2018
Jul 19

Though it seems like yesterday, Contenta CMS got the first stable release more than a year ago. In the meantime, Contenta CMS started using Media in core, improved Open API support, provided several fixes for the Schemata module, written and introduced JSON RPC, and made plans to transition to the Umami content model from Drupal core. A lot has happened behind the scenes. I’m inspired to hear of each new instance where Contenta CMS is being used both out-of-the-box and as part of a custom decoupled Drupal architecture. Both use cases were primary goals for the project. In many cases, Drupal, and hence Contenta CMS, is only part of the back-end. Most decoupled projects require a nodejs back-end proxy to sit between the various front-end consumers and Drupal. That is why we started working on a nodejs starter kit for your decoupled Drupal projects. We call this Contenta JS.

Until now, each agency had their own nodejs back-end template that they used and evolved in every project. There has not been much collaboration in this space. Contenta JS is meant to bring consistency and collaboration—a set of common practices so agencies can focus on creating the best software possible with nodejs, just like we do with Drupal. Through this collaboration, we will be able to get features that we need in every project, for free. Today Contenta JS already comes with many of these features:

  • Automatic integration with the API exposed by your Contenta CMS install. Just provide the URL of the site and everything is taken care of for you.
    • JSON API integration.
    • JSON RPC integration.
    • Subrequests integration.
    • Open API integration.
  • Multi-threaded nodejs server that takes advantage of all the cores of the server’s CPU.
  • A Subrequests server for request aggregation. Learn more about subrequests.
  • A Redis integration via the optional @contentacms/redis.
  • Type safe development environment using Flow.
  • Configurable CORS.
Diagram of Contenta JSDiagram of Contenta JS

Watch the introduction video for Contenta JS (6 minutes).

Videos require iframe browser support.

Combining the community’s efforts, we can come up with new modules that do things like React server-side rendering with one command, or a Drupal API customizer, or aggregate multiple services in a pluggable way, etc.

Join the #contenta Slack channel if this is something you are passionate about and want to collaborate on it. You can also create an issue (or a PR!) in the GitHub project. Together, we can make a holistic decoupled Drupal backend from start to end.

Originally published at humanbits.es on July 16, 2018.

Jul 12 2018
Jul 12

It may sound surprising to hear about brand consistency from a back-end developer. This is traditionally a topic for UX and marketing experts. Nevertheless, brand consistency is a powerful trend that’s affecting how we architect content APIs.

One of the ways I contribute to the Drupal API-First Initiative, aside from all the decoupled modules, is by providing my point of view from the implementation side. Some would call that real world™ experience with client projects. This means that I need to maintain a pragmatic point of view to make sure that we can do with Drupal what clients need from us. While being vigilant on the trends affecting our industry, I have discovered that there is a strong tendency for digital projects to aim for brand consistency. How does that impact implementation?

What I mean by brand consistency

When I talk about brand consistency, I only refer to a small part of it. Picture, for a moment, the home screen of Netflix on your TV. Now picture Netflix on your browser and on the app for your phone. They all look the same, don’t they? This is intentional.

The first time I installed Netflix on my wife’s iPad I immediately knew how to use the app. It took me about a second to learn how to use a complex and powerful application on a device that was foreign to me. I am an Android person but I was able to transition from using Netflix on my phone while on the bus to my wife's iPad and from there to the living room TV. I didn’t even realize that I was doing it. Everything was seamless because all the different devices running Netflix had a consistent design and user experience.

If you are interested in the concept of brand consistency and its benefits you can learn more from actual experts on the subject. I will focus on the implications for API design.

It changes the approach to decoupled projects

For the last few years, I have been speaking at events and writing about the imperious necessity for your back end to be presentation agnostic. Consumers can have radically different data needs. You don’t want your back end to favor a particular consumer because that will lead to re-coupling, which leads to high maintenance costs for the consumers that you turned your back on.

When the UX and designs are consistent across consumers, then the statement ‘the consumers can have radically different data needs’ may no longer apply. If they really are consistent, why would the data they need be radically different? You cannot be consistent and radically different at the same time.

Many constraints, API design tips, and recommendations are based on the assumption of presentation agnosticism. While this holds true for most projects, a significant number of projects have started to require consistency across consumers. So the question is: if we no longer need to be presentation agnostic in our API design, what can we optimize given that we have a single known presentation? We made many compromises. What did we give up, and how do we get it back?

How I approached the problem

The first time that I encountered this need for unified UX across all consumers in a client project my inherent pragmatism was triggered. My brain was flooded with potential optimizations. Together with the rest of the client team, I took a breath and started analyzing this new problem space. On this occasion, the client had suggested the BFF pattern from the start. Instead of having a general-purpose API back end to serve all of your downstream consumers, you have one back end per user experience. Hence the moniker ‘Backend for Frontend’ or BFF. This was a great suggestion that we carefully analyzed and soon embraced.

What is a BFF?

Think of a BFF as a server-side service that takes care of the orchestration and processing of the different interactions with the API (or even multiple APIs or microservices) on behalf of the consumers. In short, it does what each consumer would do against your presentation agnostic API, and consolidates it on the server for presentation. The BFF produces a render-ready JSON object.

In other words, we will build a consumer in the back end, but instead of outputting HTML, CSS, and JavaScript (using the web consumer as an example) we will output a JSON document.

BFF output example

You can see in the code above that the shape of the JSON response is heavily influenced by the single design and the components in the frontend. This implies some rigidness on front-end differences, but we agreed that’s OK for our case. For your completely different design, the JSON output would look completely different.

How we implemented BFFs

After requirements are settled, we decide that we will have a single Backend For Frontend that will power all the consumer applications. Instead of having one BFF for each consumer, as Netflix used to do it, we will only have one. The reason is that with one we ensure brand consistency. Also, as Lee Byron puts it:

The concern of duplicating logic across different BFFs is more than just maintaining two repositories of similar code rather than one. The concern is the endless fight against accidental divergence.

Additionally, we don’t have those requirements, but the BFF is also the best place to add global restrictions like authentication, request filters, rate limits, etc.

Our team decided to implement this as a set of rigid endpoints in a Serverless [LINK] application written in NodeJS. As you can imagine, you can implement this pattern with the tools and the stack you prefer. Since this will be so specific to your project’s designs you will likely need to start from scratch.

How consumers deal with BFFs

We create this consumer in the backend in order to simplify all the possible front ends. We move the complexity of building a consumer into a central service that can be reused by all the consumers. That way we can call the consumers, dumb clients. This is because the consumers no longer need to craft complex queries (JSON API, GraphQL, or whatever else); they don’t need to aggregate 3rd party services; and they don’t need to normalize the data from the different APIs, etc. In fact, all the data is ready to render.

In our particular case, we have been able to reduce the consumers to renderers. A consumer only needs to:

  1. Process an incoming request and then determine what screen to grab from the BFF. Additionally, extract any parameters from the request, like the entity ID. In addition to that any global parameters, like the user ID from the device, are added to the parameter bag.
  2. With the name of the screen and the extracted parameters the consumer makes a single HTTP request to the BFF.
  3. The BFF responds with all the data needed for rendering in a shape ready for rendering. The consumer takes that and renders all the components.
  4. The consumer finally adds all the business logic that is exclusive of the front end on top of the rendered output. This includes ads, analytics, etc.

Pros and cons

The pros of this approach are stated throughout the document, but to summarize they are:

  • Massive simplification of the consumers. Those complex interactions with the API are in a central place, instead of having each consumer team write them, again and again, in their native language.
  • Code reuse across consumers. Bug-fixes, changing requirements, improvements, and documentation efforts apply to all consumers since much of the logic lies in the BFF now.
  • Increased performance. The backend can be optimized in numerous ways since it does not need to enable every possible design. This can mean denormalized documents in Elastic Search with the pre-computed responses, increased cache hit ratios in calls to APIs now that we control how those are made, faster server-to-server communications for 3rd party API aggregation, etc.
  • Frontend flexibility. We can ship new features faster when front ends are dumb clients and just render the BFF output. Unless we need to render new components or change the way something is rendered there are few reasons to require an app update. Bear in mind that some platforms don’t support automatic updates, and when they do not all users have them turned on. With this re-coupled pattern, we can ship new features to old consumers.

On the other hand, there are some cons:

  • Requires a dedicated back-end team. You cannot just install an API generator, like Contenta CMS, that is configured in the UI and serves a flexible JSON API with zero configuration. Now you need a dedicated backend team to build your BFF. However, chances are that your project already has a dedicated back-end team.
  • Brings back the bikeshedding. In DrupalCon Baltimore, I talked about how the JSON API module stops the bikeshedding. In this new paradigm, we are back to discussing things like the shape of the response, the names in it, how to expose these responses, etc.
  • It requires cross-consumer collaboration. This is because you want to design a BFF that works well for all current consumers and future ones. Collaboration across different teams can be a challenge depending on the organization.

To summarize

An organization that can make the compromise of a consistent design across consumers can simplify their omni-channel strategy. One way to do that is to move the complexity from several consumers to a single one, that lives in the back end.

Some organizations have used the BFF pattern successfully to achieve these goals in the past. Using this pattern, the different consumers can be simplified to dumb clients, leaving the business logic to the BFF. That, in turn, will allow for better performance, less code to maintain, and smaller time to market for new features.

Photo by Andrew Ridley on Unsplash

Jun 27 2018
Jun 27

Note: This article was originally published on August 23, 2017. Following DrupalCon Nashville, we are republishing some of our key articles on decoupled or "headless" Drupal as the community as a whole continues to explore this approach further. Comments from the original will appear unmodified.

Decoupled Drupal has been well understood at a technical level for many years now. While the implementation details vary, most Drupal teams can handle working on decoupled projects. However, we’ve heard the following from many of our clients:

  1. We want a decoupled site. Why is this web project so expensive compared to sites I worked on in the past?
  2. Why do our decoupled projects seem so unpredictable?
  3. If we decide to invest in decoupled technologies, what can we expect in return?

Let’s dive into these questions.

Why Can Decoupled Sites Cost More?

Before getting too much into the details of decoupled versus full-stack, I like to ask stakeholders:

“What does your website need to do today that it didn't 5 years ago?”

Often, the answer is quite a lot! Live video, authenticated traffic, multiple mobile apps, and additional advertising deals all add to more requirements, more code, and more complexity. In many cases, the costs that are unique to decoupling are quite small compared to the costs imposed by the real business requirements.

However, I have worked on some projects where the shift to a decoupled architecture is fundamentally a technology shift to enable future improvements, but the initial build is very similar to the existing site. In those cases, there are some very specific costs of decoupled architectures.

Decoupling means forgoing Drupal functionality

Many contributed modules provide the pre-built functionality we rely on for Drupal site builds. For example, the Quickedit module enables in-place editing of content. In a decoupled architecture, prepare to rewrite this functionality. Website preview (or even authenticated viewing of content) has to be built into every front-end, instead of using the features we get for free with Drupal. Need UI localization? Content translation? Get ready for some custom code. Drupal has solved a lot of problems over the course of its evolution, so you don’t have to—unless you decouple.

Decoupling is shorthand for Service Oriented Architectures

For many organizations, a decoupled website is their first foray into Service Oriented Architectures. Most full-stack Drupal sites are a single application, with constrained integration points. In contrast, a decoupled Drupal site is best conceived of as a “content service,” accessed by many disparate consumers.

I’ve found that the “black-boxing” of a decoupled Drupal site is a common stumbling block for organizations and a driver behind the increased costs of decoupling. To properly abstract a system requires up-front systems design and development that doesn’t always fit within the time and budget constraints of a web project. Instead, internal details end up being encoded into the APIs Drupal exposes, or visual design is reflected in data structures, making future upgrades and redesigns much more expensive. Writing good APIs is hard! To do it well, you need a team who is capable of handling the responsibility—and those developers are harder to find and cost more.

Scalable systems and network effects

Once your team dives into decoupling Drupal, they are going to want to build more than just a single Drupal site and a single JavaScript application. For example, lullabot.com actually consists of five systems in production:

  1. Drupal for content management
  2. A CouchDB application to serve content over an API
  3. A second CouchDB application to support internal content preview
  4. A React app for the site front-end
  5. Disqus for commenting

Compared to the sites our clients need, lullabot.com is a simple site. In other words, as you build, expect to be building a web of systems, and not just a “decoupled” website. It’s possible to have a consumer request Drupal content directly, especially in Drupal 8, but expect your tech teams to push for smaller “micro” services as they get used to decoupling.

Building and testing a network of systems requires a lot of focus and discipline. For example, I’ve worked with APIs that expose internal traces of exceptions instead of returning something usable to API consumers. Writing that error handling code on the service is important, but takes time! Is your team going to have the bandwidth to focus on building a robust API, or are they going to be focusing on the front-end features your stakeholders prioritize?

I’ve also seen decoupled systems end up requiring a ton of human intervention in day-to-day use. For example, I’ve worked with systems where not only is an API account created manually, but manual configuration is required on the API end to work properly. The API consumer is supposed to be abstracted from these details, but in the end, simple API calls are tightly coupled to the behind-the-scenes configuration. A manual set up might be OK for small numbers of clients, but try setting up 30 new clients at once, and a bottleneck forms around a few overworked developers.

Another common mistake is not to allow API consumers to test their integrations in “production.” Think about Amazon’s web services—even if your application is working from a QA instance, as far as Amazon is concerned there are only production API calls available. Forcing other teams to use your QA or sandbox instance means that they won’t be testing with production constraints, and they will have production-only bugs. It’s more difficult to think about clients creating test content in production—but if the API doesn't have a good way to support that (such as with multiple accounts), then you’re missing a key set of functionality.

It’s also important to think about error conditions in a self-serve context. Any error returned by an API must make clear if the error is due to an error in the API, or the request made of the API. Server-side errors should be wired up to reporting and monitoring by the API team. I worked with one team where client-side errors triggered alerts and SMS notifications. This stopped the client-side QA team from doing any testing where users entered bad data beyond very specific cases. If the API had been built to validate inbound requests (instead of passing untrusted data through its whole application), this wouldn't have been a problem.

There's a lot to think about when it comes to decoupled Drupal sites, but it’s the only way to build decoupled architectures that are scalable and lead to faster development. Otherwise, decoupling is going to be more expensive and slower, leaving your stakeholders unsatisfied.

Why are decoupled projects unpredictable?

When clients are struggling with decoupled projects, we’ve often found it’s not due to the technology at all. Instead, poor team structure and discipline lead to communication breakdowns that are compounded by decoupled architectures.

The team must be strong developers and testers

Building decoupled sites means teams have to be self-driving in terms of automated testing, documentation, and REST best practices. QA team members need to be familiar with testing outside of the browser if they are going to test APIs. If any of these components are missing, then sprints will start to become unpredictable. The riskiest scenario is where these best practices are known, but ignored due to stakeholders prioritizing “features.” Unlike one-off, full-stack architectures, there is little room to ignore these foundational techniques. If they’re ignored, expect the team to be more and more consumed by technical debt and hacking code instead of solving the actual difficult business problems of your project.

The organizational culture must prioritize reliable systems over human interactions

The real value in decoupled architectures comes not in the technology, but in the effects on how teams interact with each other. Ask yourself: when a new team wants to consume an API, where do they get their information? Is it primarily from project managers and lead developers, or documentation and code examples? Is your team focused on providing “exactly perfect” APIs for individual consumers, or a single reusable API? Are you beholden to a single knowledge holder?

This is often a struggle for teams, as it significantly redefines the role of project managers. Instead of knowing the who of different systems the organization provides, it refocuses on the what - documentation, SDKs, and examples. Contacting a person and scheduling a meeting becomes a last resort, not a first step. Remember, there’s no value in decoupling Drupal if you’ve just coupled yourself to a lead developer on another team.

Hosting complexity

One of the most common technological reasons driving a decoupled project is a desire to use Node.js, React, or other JavaScript technologies. Of course, this brings in an entire parallel stack of infrastructure that a team needs to support, including:

  • HTTP servers
  • Databases
  • Deployment scripts
  • Testing and automation tools
  • Caching and other performance tools
  • Monitoring
  • Local development for all of the above

On the Drupal side, we’ve seen many clients want to host with an application-specific host like Acquia or Pantheon, but neither of those support running JavaScript server-side. JavaScript-oriented hosts likewise don’t support PHP or Drupal well or at all. It can lead to some messy and fragile infrastructure setups.

All of this means that it’s very difficult for a team to estimate how long it will take to build out such an infrastructure, and maintenance after a launch can be unpredictable as well. Having strong DevOps expertise on hand (and not outsourced) is critical here.

Decoupled often means “use a bunch of new Node.js / JavaScript frameworks”

While server-side JavaScript seems to be settling down towards maturity nicely, the JavaScript ecosystem for building websites is reinventing itself every six months. React of today is not the same React of 18 months ago, especially when you start considering some of the tertiary libraries that fill in the gaps you need to make a real application. That’s fine, especially if your project is expected to take less than 6 months! However, if your timeline is closer to 12-18 months, it can be frustrating to stakeholders to see a rework of components they thought were “done,” simply because some library is no longer supported.

What’s important here is to remember that this instability isn't due to decoupling—it’s due to front-end architecture decisions. There’s nothing that stops a team from building a decoupled front-end in PHP with Twig, as another Drupal site, or anything else.

If we invest in Decoupled Drupal, what’s the payoff?

It’s not all doom and decoupled gloom. I’ve recommended and enjoyed working on decoupled projects in the past, and I continue to recommend them in discoveries with clients. Before you start decoupling, you need to know what your goals are.

A JavaScript front-end?

If your only goal is to decouple Drupal so you can build a completely JavaScript-driven website front-end, then simply doing the work will give you what you want. Infrastructure and JavaScript framework churns are most common stumbling blocks and not much else. If your team makes mistakes in the content API, it’s not like you have dozens of apps relying on it. Decouple and be happy!

Faster development?

To have faster site development in a decoupled context, a team needs to have enough developers so they can be experts in an area. Sure, the best JavaScript developers can work with PHP and Drupal but are they the most efficient at it? If your team is small and a set of “full-stack” developers, decoupling is going to add abstraction that slows everything down. I’ve found teams need to have at least 3 full-time developers to get efficiency improvements from decoupling. If your team is this size or larger, you can significantly reduce the time to launch new features, assuming everyone understands and follows best development practices.

Multichannel publishing?

Many teams I’ve worked with have approached decoupled Drupal, not so much to use fancy JavaScript tools, but to “push” the website front-end to be equal to all other apps consuming the same content. This is especially important when your CMS is driving not just a website and a single app, but multiple apps such as set-top TV boxes, game consoles, and even apps developed completely externally.

With full-stack Drupal, it’s easy to create and show content that is impossible to view on mobile or set-tops apps. By decoupling the Drupal front-end, and using the same APIs as every other app, it forces CMS teams to develop with an API-first mentality. It puts all consumers on an equal playing field, simplifying the development effort in adding a new app or platform. That, on its own, might be a win for your organization.

Scaling large teams?

Most large Drupal sites, even enterprise sites, have somewhere between 5-10 active developers at a time. What if your team has the budget to grow to 30 or 50 developers?

In that case, decoupled Drupal is almost the only solution to keep individuals working smoothly. However, decoupled Drupal isn’t enough. Your team will need to completely adopt an SOA approach to building software. Otherwise, you’ll end up paying developers to build a feature that takes them months instead of days.

Decoupling with your eyes open

The most successful decoupled projects are those where everyone is on board—developers, QA, editorial, and stakeholders. It’s the attitude towards decoupling that can really push teams to the next level of capability. Decoupling is a technical architecture that doesn't work well when the business isn't buying in as well. It’s worth thinking about your competitors too—because if they are tech companies, odds are they are already investing in their teams and systems to fully embrace decoupling.

Jun 14 2018
Jun 14
Mike and Matt talk with Angie "Webchick" Byron on what she's been up to, various Drupal initiatives, and what Drupal needs to do to succeed.
Jun 13 2018
Jun 13

Our sales team often refers to our Hierarchy of Qualification when evaluating projects. This pyramid, inspired by Maslow’s hierarchy of needs, gives us the tools not just to evaluate the business needs of a project, but the human needs that are “encoded” in the project and team.

I’ve been thinking about how this applies to software development, particularly in the enterprise space. Enterprise implies that the software is maintained by large teams over long periods of time. Most developers have encountered internal enterprise software that leaves us shaking our heads, asking “how was this ever released?” Alternatively, we’ve used an internal tool that is quite good, but where the business has trouble repeating that success with new projects.

If we can describe the path towards self-actualizing software quality, we can elevate our teams from solving a series of one-off problems towards building value for our businesses and ourselves. It’s important to acknowledge that these steps are additive, meaning a team may benefit by mastering the lower rungs of the pyramid before moving on to the next.

Hierarchy of Software Quality Illustration

Describing software and how humans will use it

This is the base of the pyramid and undergirds everything else. Before writing a line of code, a team needs to have a good handle on who the audience is and how the software will affect them, along with the overall goals of the new project. Often, enterprise teams are given work to do by their customers or their management with the explanation: “a big internal department has told us this is a blocker, so don’t ask why and get to coding, so we aren’t late.”

This leaves project managers and developers in the dark, and unable to make reasonable guesses when requirements and technology conflict. Knowing the audience lets teams prioritize their work when time is short. Is the audience a group of editorial users? What’s their day-to-day workflow like? In that case, time may be best spent on testing and iterating the user interface, at the expense of implementing every last feature request. Is the audience another group of developers? Then perhaps the project doesn’t require a user interface initially, so developers can focus on great APIs and documentation. Not knowing the audience may lead to otherwise avoidable mistakes.

Like audience, project or product goals are another important piece of the puzzle. When goals are fuzzy, it is hard to know when a software project is done, or at least at a “1.0” release. Teams tend to leave projects at a nebulous “0.x” version, making future developers uncertain about the quality or robustness of a system. For example, perhaps a client asks to implement Apple News and Facebook Instant Articles on their website. It’s a reasonable request. Nevertheless, prematurely ending requirements gathering at “implement the feature” deprives your team of critical information.

There’s a business reason behind the request. Is the analytics team seeing declining traffic on their website, and worried the website audience is defecting to social networks? It may be good to suggest working on Facebook integration first, assuming you have some analytics data from existing Facebook posts to back it up. Or, perhaps the sales team is hearing from advertisers that they want to purchase ads against your content inside of the Apple News app. In that case, finding out some rough estimates for the additional ad revenue the sales team expects can help with qualifying estimates for the integration effort. If the estimated development budget eclipses the expected revenues, the team can investigate different implementation methods to bring costs down, or even rework the implementation as a one-off proof of concept.

Using audiences and goals to guide your development team makes it possible to discover opportunities beyond the immediate code, elevating the team from an “IT expense” to a “technical partner.”

Technical architecture and documentation

Writing the right software for today is one thing. Writing maintainable software that future developers can easily iterate on is another. Spaghetti architecture can leave you with software that works for now but is very expensive to maintain. For Drupal sites, that means avoiding god-objects (and services), writing true object-oriented code (and not procedural code that happens to live in classes), and avoiding side effects such as global and public variables. It’s important to document (preferably in code) not just what you built but why you built it, and how the architecture works to solve the original business problems. Think of it as writing a letter to your future self, or to whatever team inherits your code.

Determining the effort to put into this work is tricky, and time pressures can lead to quickly written, but poorly architected software. It’s useful to think about the expected life of the application. Look at similar applications in the company and ask about their initial and current development. For example, in our CMS projects, we often find we are replacing systems that are over a decade old. The code has gone through many teams of developers but is in a decrepit state as the architecture doesn’t follow common design patterns and the documentation is non-existent. No one wants to touch the existing application or infrastructure because experience has shown that may lead to a multi-day outage.

In cases like this, we know that following good design patterns and effectively documenting code will pay dividends later. A marketing site for an event in 3 months? You can probably take a lot of shortcuts without risk. Treat the project for what it is—a startup within an enterprise.

A culture of iterative feedback

Defining the software to write and following good architecture in its execution is important but we can do more. Now that our efforts are well documented, its time to create a proper harness for testing. Without one, it’s impossible to know if your team is meeting your quality goals. It’s easy for teams to fall into a fragile, untrustworthy process for testing. For sites, that typically means not defining and following deployment best practices or creating a QA bottleneck through infrastructure.

The typical git model of development implies not just a few branches that are merge destinations (like master and develop) but many feature branches, too. It’s not uncommon for developers to create multiple branches breaking a single ticket into smaller code components that can be independently reviewed and tested.

For developers, “waiting for a QA build” is one of the biggest motivation killers they face. For QA, trying to coordinate testing of various features onto a single QA server is complex and leaves them questioning the validity of their test plans.

Just as “smaller, chunkier” pull requests are a best practice for developers, a similar guideline helps QA analysts feel confident when certifying a feature or release.

Tugboat is one tool that lets teams implement automatic, lightweight, and disposable testing environments. When a developer opens a pull request, a complete working copy of the website is built and available for anyone with access to poke at. It’s easy to rebuild new testing environments because the setup is automated and repeatable—not manual. A process like this extends the effective QA team by including both developers (who can reproduce and take ownership of bugs “live” with QA, before code is merged), and the actual patrons of the build—project managers and business stakeholders—who will eventually need to sign off on the product.

These tools also change the culture of communication between stakeholders and implementation teams. Even with two-week sprints, there still can be an information gap stakeholders face between sprint planning and the sprint demo. Instead, getting their feedback is as frictionless as sending a link. There are fewer surprises at demos because stakeholders have already seen the constituent parts of the work that compose the whole demo.

Automated tests and quality metrics

Frictionless QA is great, but it’s still a manual process. It’s incredibly frustrating for an entire team to be going through manual QA for regression testing. Sometimes, regression tests uncover the sort of bugs that mean “rewrite the feature from scratch,” leading to failed sprints and missed demos. That’s especially likely when regression testing occurs at the end of a sprint, leaving precious little time for rework and another round of QA.

In contrast, developers love when automated tests alert them to issues. Not only does it save developers and QA time (they can work on a new ticket while waiting for a test suite to run), but it reduces the cognitive load of developing a new feature in a complex application. Instead of developers having to become the application experts, and maintain that knowledge in perpetuity, the automated tests themselves become the legal description of how the code should work.

Comprehensive test coverage, regardless of the actual tool used (like PHPUnit or Behat), also helps ensure that software quality remains constant as underlying dependencies are upgraded. For a Drupal site, teams should expect at least two significant upgrades of Drupal per year, along with an update to PHP itself. Automated testing means the initial work is as simple as “upgrade Drupal, run tests, and see what breaks,” instead of throwing a bunch of time and money at manual testing. It also means that testing of security patches is much faster, saving time between vulnerability disclosure and deployment.

Of course, there’s a cost to automated testing. Tests are still code that need to be maintained. Tests can be buggy themselves (like that time I accidentally tested for the current year in a string, which broke when 2018 rolled around). Someone has to pay for a continuous integration service or maintain custom infrastructure. Reducing these costs is a particular interest of ours, leading to templates like Drupal 8 CI and the Drupal Testing Container.

Again, knowing the expected life of the software you write helps to inform how deep into testing to go. I don’t think I’ve ever written automated tests for a microsite but for a multi-step payment form expected to endure for 5 years, they were critical.

Technical architecture, for all of its focus on “clean code,” can rapidly devolve into unresolvable conflicts within a team. After all, everyone has their idea of what “good design” looks like. While automated tools can’t tell you (yet) if you’re using the right design pattern to solve a given problem, they can tell you if your app is getting better or worse over time.

Start with enforcing basic code standards within your team. Code standards help team members to read code written by others, which is “step 0” in evaluating if a given codebase meets quality goals or not.

For example:

As a technical      architect, you       want reading the          code to feel as natural as             reading written text — so you            can focus on the meaning, and not       the individual words.

Think about how much harder it is to focus on the idea I’m trying to communicate. Even though you can understand it, you have to slow down and focus. Writing code without code standards sacrifices future comprehension at the altar of expediency. I’ve seen teams miss critical application bugs even though they were hiding in plain sight due to poor code formatting.

Luckily, while different projects may have different standards, they tend to be internally consistent. I compare it to reading different books with different fonts and layouts. You may have to context switch between them, but you would rarely have the paragraphs of the books interspersed with each other.

While code standards tend to either be “right” or “wrong,” code quality metrics are much more of an art than a science—at least, as far as interpreting and applying them to a project goes. I like to use PhpMetrics as it has a very nice user interface (and works perfectly within continuous integration tools). These reports inherently involve some subjective measure of what a failure is. Is the cyclomatic complexity of a method a fail at 15, or 50, or not at all? Sometimes, with difficult codebases, the goals come down to “not making things any worse.” Whatever your team decides, using these tools daily will help ensure that your team delivers the best product it can.

Continuous delivery

As a team addresses each new layer of the hierarchy, the lower layers become habits that don’t require day-to-day focus. The team can start to focus on solving business problems quickly, becoming an IT partner instead of an IT expense. Each person can fully participate in the business instead of the narrow field in front of them.

With all of these prerequisites in place, you will start to see a shift in how your team approaches application development. Instead of a conservative and fragile approach to development, your team will start to design and develop applications without fear. Many teams find themselves rotating between the bottom two levels of the pyramid. With careful planning and hard work, teams can work towards the upper tiers. The whole software delivery process—from ideas to releases to hotfixes to security releases—becomes a habit rather than a scramble every time.

Hero Image Photo by Ricardo Gomez Angel on Unsplash

Jun 06 2018
Jun 06

We’ve all been hearing a lot about JavaScript eating the web, but what does that mean for traditional content management systems like Drupal and Wordpress? In many ways, it’s a fatuous claim to say that any particular language is “winning” or “eating” anything, but if a different approach to building websites becomes popular, it could affect the market share of traditional CMS platforms. This is an important topic for my company, Lullabot, as we do enterprise software projects using both Drupal, a popular CMS, and React, a popular JavaScript framework.

At our team retreat, one of Lullabot’s front-end devs, John Hannah, who publishes the JavaScript Report, referred to Drupal as a “legacy CMS.” I was struck by that. He said that, for many in the JavaScript community, PHP CMSes like Drupal, WordPress, and Joomla are seen this way. What does he mean? If these “monolithic” CMS platforms are legacy, what’s going to replace them? Furthermore, PHP, the language, powers 83% of total websites and plays a large part in platforms like Facebook. That’s not going to change quickly.

Still, JavaScript’s surging popularity is undeniable. In part, this is due to its ubiquity. It’s a law-of-increasing-returns, chicken-and-the-egg, kind of thing, but it’s real. JavaScript is the only programming language that literally runs everywhere. Every web browser on every device has similar support for JavaScript. Most cloud providers support JavaScript’s server-side incarnation Node.js, and its reach extends far beyond the browser into the internet of things, drones and robots. Node.js means JavaScript can be used for jobs that used to be the sole province of server-side languages. And isomorphism means it can beat the server-side languages at their own game. Unlike server-side applications written in PHP (or Ruby or Java), an isomorphic application is one whose code (in this case, JavaScript) can run on both the server and the client. By taking advantage of the computing power available from the user’s browser, an isomorphic application might make an initial HTTP request to the Node.js server, but from there asynchronously load resources for the rest of the site. On any subsequent request, the browser can respond to the user without having the server render any HTML, which offers a more responsive user experience. JavaScript also evolved in tandem with the DOM, so while any scripting language can be used to access nodes and objects that comprise the structure of a web page, JavaScript is the DOM’s native tongue.

Furthermore, the allure of one-stack-to-rule-them-all may attract enterprises interested in consolidating their IT infrastructure. As Hannah told me, “An organization can concentrate on just supporting, training and hiring JavaScript developers. Then, that team can do a lot of different projects for you. The same people who build the web applications can turn around and build your mobile apps, using a tool like React Native. For a large organization, that’s a huge advantage.”

Node.js Takes a Bite Out of the Back-end

Tempted to consolidate to a single stack, one of Lullabot’s largest digital publishing clients, a TV network, has begun to phase out Drupal in favor of microservices written in Node.js. They still need Drupal to maintain metadata and collections, but they’re talking about moving all of their DRM (digital rights management) content to a new microservice. This also provoked my curiosity. Is the rapidly changing Node.js ecosystem ready for the enterprise? Who is guaranteeing your stack stays secure? The Node foundation? Each maintainer? Drupal has a seasoned and dedicated security team that keeps core and contrib safe. Node.js is changing fast, and the small applications written in Node.js are changing faster than that. This is a rate of change typically anathema to enterprise software. And it’s a significant shift for the enterprise in other ways, as well. While Drupal sites are built with many small modules that together create a unique application, they all live within one set of code. The Node.js community approaches the same problem by building small applications that communicate with each other over a network (known as microservices). The preferred approach of most within the Node.js community is to build applications using a microservices architecture. Does that work in the context of a major enterprise publisher? Is it maintainable? Airbnb, Paypal, and Netflix have proven that it can be, but for many of our clients in the digital publishing industry, I wonder how its use within these technology companies pertains. Arguably, Amazon pioneered the modern decouple-all-the-things, service-oriented architecture, and, well, you, dear client, are not Amazon.

In this article, I’ll explore this question through examples and examine how JS technologies are changing the traditional CMS stack architecture within some of the client organizations we work with. I should disclose my own limitations as I tackle an ambitious topic. I’ve approached this exploration as a journalist and a business person, not as an engineer. I’ve tried to interview many sources and reflect the nuances of the technical distinctions they’ve made, but I do so as a technical layperson, so any inaccuracies are my own.

The Cathedral and the Bazaar

How is the JavaScript approach different than that of a CMS like Drupal? In the blogosphere, this emerging competition for the heart of the web is sometimes referred to as “monolith vs. microservice.” In many ways, it’s less a competition between languages, such as JavaScript and PHP (or Ruby or Python or Java), but more the latest chapter in the dialectic between small, encapsulated programs with abstracted APIs as compared to comprehensive, monolithic systems that try to be all things to all people.

The JavaScript microservices approach—composing a project out of npm packages—and the comparatively staid monolithic approach taken by Drupal are both descendants of open source collaboration, aka the “bazaar,” but the way things are being done in Node.js right now, as opposed to the more mature and orderly approach of the Drupal community, make Node.js the apparent successor of the bazaar. Whereas Drupal—with 411,473 lines of code in the current version of core—has cleaned up its tents and begun to look more like a cathedral, with core commits restricted to an elite priesthood.

Drupal still benefits from a thriving open-source community. According to Drupal.org, Drupal has more than 111,000 users actively contributing. Meaning, in perhaps the most essential way, it is still benefiting from Linux founder Linus Torvald’s law, “Given enough eyes, all bugs are shallow.” Moreover, the Drupal community remains the envy of free software movement, according to Google’s Steve Francia, who also guided the Docker and MongoDB communities following Drupal’s lead.

Nevertheless, JavaScript and its server-side incarnation are gaining market share for a reason. JavaScript is both accessible and approachable. Lullabot senior developer Mateu Aguiló Bosch, one of the authors of Contenta CMS, a decoupled Drupal distribution, describes the JavaScript ecosystem, as follows: “there’s an open-script vibe in the community with snippets of code available everywhere. Anyone, anywhere, can experiment with these snippets using the built-in console in their browser.” Also, Bosch continues, “Node.js brings you closer to the HTTP layer. Many languages like PHP and Ruby abstract a lot of what’s going on at that layer, so it wasn’t until I worked with Node.js that I fully understood all of the intricacies of the HTTP protocol.”

Furthermore, thanks to the transpiler Babel, JavaScript allows developers to program according to their style. For instance, a developer familiar with a more traditional nomenclature for object-oriented classes can use TypeScript and then transpile their way to working JavaScript that’s compatible with current browsers. “Proposals for PHP have to go through a rigorous process, then go to binary, then the server has to upgrade, and then apps for Drupal have to support this new version,” says Sally Young, a Lullabot who heads the Drupal JavaScript Modernization Initiative. “With transpiling in JS, we can try out experimental language features right away, making it more exciting to work in.” (There is precedence for this in the PHP community, but it’s never become a standard workflow.)

However, JavaScript’s popularity is attributable to more than just the delight it inspires among developers.

Concurrency and non-blocking IO

I spoke with some of our clients who are gradually increasing the amount of JS in their stack and reducing the role of Drupal about why they’ve chosen to do so. (I only found one who is seeking to eliminate it all together, and they haven’t been able to do so yet.) Surprisingly, it had nothing to do with Hannah’s observation that hiring developers for and maintaining a single stack would pay dividends in a large organization. This was a secondary benefit, not a primary motivation. The short answer in each case was speed, in one case speed in the request-response sense, and in the other, speed in the go-to-market sense.

Let’s look at the first example. We work with a large entertainment media company that provides digital services for a major sports league. Their primary site and responsive mobile experience are driven by Drupal 8’s presentation layer (though there’s also the usual caching and CDN magic to make it fast). Beyond the website, this publisher needs to feed data to 17 different app experiences including things like iOS, tvOS, various Android devices, Chromecast, Roku, Samsung TV, etc. Using a homegrown system (JSON API module wasn’t finished when this site was migrated to D8), this client pushes all of their content into an Elasticsearch datastore where it is indexed and available for the downstream app consumers. They built a Node.js-based API to provide the middleware between these consumers and Elasticsearch. According to a stakeholder, the group achieved “single-digit millisecond responses to any API call, making it the easiest thing in the whole stack to scale.”

This is likely in part due to one of the chief virtues of Node.js. According to Node’s about page:

As an asynchronous event-driven JavaScript runtime, Node is designed to build scalable network applications…many connections can be handled concurrently. Upon each connection, a callback is fired, but if there is no work to be done, Node will sleep. This is in contrast to today's more common concurrency model where OS threads are employed. Thread-based networking is relatively inefficient and very difficult to use.

Multiple core CPUs can handle multiple threads in direct proportion to the number of CPU cores. The OS manages these threads and can switch between them as it sees fit. Whereas PHP moves through a set of instructions top to bottom with a single pointer for where it’s at in those instructions, Node.js uses a more complex program counter that allows it to have multiple counters at a time. To roughly characterize this difference between the Node.js asynchronous event loop and the approach taken by other languages, let me offer a metaphor. Pretend for a moment that threads are cooks in the kitchen awaiting instructions. Our PHP chef needs to proceed through the recipe a step at a time: chopping vegetables, and then boiling water, and then putting on a frying pan to sauté those veggies. While the water is boiling, our PHP chef waits, or the OS makes a decision and moves on to something else. Our Node.js chef, on the other hand, can handle multi-tasking to a degree, starting the water to boil, leaving a pointer there, and then moving on to the next thing.

However, Node.js can only do this for input and output, like reading a database or fetching data over HTTP. This is what is referred to as “non-blocking IO.” And, it’s why the Node.js community can say things like, “projects that need big concurrency will choose Node (and put up with its warts) because it’s the best way to get their project done.” Asynchronous, event-driven programs are tricky. The problem is that parallelism is a hard problem in computer science and it can have unexpected results. Imagine our cook accidentally putting the onion on the stove to fry before the pan is there or the burner is lit. This is akin to trying to read the results from a database before those results are available. Other languages can do this too, but Node’s real innovation is in taking one of the easier to solve problems of concurrent programming (IO), designing it directly into the system so it’s easier to use by default, and marketing those benefits to developers who may not have been familiar with similar solutions in more heavyweight languages like Java.

Even though Node.js does well as a listener for web requests, the non-blocking IO bogs down if you’re performing CPU intensive computation. You wouldn’t use Node.js “to build a Fibonacci computation server in Node.js. In general, any CPU intensive operation annuls all the throughput benefits Node offers with its event-driven, non-blocking I/O model because any incoming requests will be blocked while the thread is occupied with your number-crunching,” writes Tomislav Capan in “Why the Hell Would You Use Node.js.” And Node.js is inherently single-threaded. If you run a Node.js application on a CPU with 8 cores, Node.js would just use one, whereas other server-side languages could make use of all of them. So Node.js is designed for lots of little concurrent tasks, like real-time updates to requests or user interactions, but bad at computationally intensive ones. As one might expect, given that, it’s not great for image processing, for instance. But it’s great for making seemingly real-time, responsive user interfaces.

JavaScript Eats the Presentation Layer

There’s an allure to building a front-end with just HTML, CSS, and JavaScript since those three elements are required anyway. In fact, one of our largest clients started moving to a microservices architecture somewhat by accident. They had a very tight front-end deadline for a new experience for one of their key shows. Given the timeline, the team decided to build a front-end experience with HTML, CSS, and JavaScript and then feed the data in via API from Drupal. They saw advantages in breaking away from the tricky business of Drupal releases, which can include downtime as various update hooks run. Creating an early decoupled (sometimes referred to as “headless”) Drupal site led them to appreciate Node.js and the power of isomorphism. As the Node.js documentation explains, “after over 20 years of stateless-web based on the stateless request-response paradigm, we finally have web applications with real-time, two-way connections, where both the client and server can initiate communication, allowing them to exchange data freely. This is in stark contrast to the typical web response paradigm, where the client always initiates communication.”

After hacking together that first site, they found React and began to take full advantage of stateful components that provide real-time, interactive UX, like AJAX but better. Data refreshes instantly, and that refresh can be caused by a user’s actions on the front-end or initiated by the server. To hear this client tell it, discovering these technologies and the virtues of Node.js led them to change the role of Drupal. “With Drupal, we were fighting scale in terms of usage, and that led us to commit to a new, microservices-oriented stack with Drupal playing a more limited role in a much larger data pipeline that utilizes a number of smaller networked programs.” These included a NoSQL data-as-a-service provider called MarkLogic, and a search service called Algolia, among others.

So what started as the need for speed-to-market caused this particular client to discover the virtues of Node.js, and then, subsequently React. Not only can libraries like React provide more app-like experiences in a browser, but tools like React Native can be used to make native apps for iOS and Android. PWAs (progressive web apps) use JavaScript to make web applications behave in certain respects like native applications when they’re opened on a mobile device or in a standard web browser. If there was ever a battle to be won in making the web more app-like, JavaScript won that contest a long time ago in the days of jQuery and Ajax. Heck, it won that battle when we all needed to learn JavaScript in the late 90s to swap navigation images using onMouseOver.

Is JavaScript taking over the presentation layer? Only for some. If you’re a small to medium-sized business, you should probably use the presentation layer provided by your CMS. If you’re a large enterprise, it depends on your use case. A good reason to decouple might be to make your APIs a first-class citizen because you have so many downstream consumers that you need to force yourself to think about your CMS as part of a data pipeline and not as a website. Moreover, if shaving milliseconds off “time to first interactive” means earning millions of dollars in mobile conversions that might otherwise have been lost, you may want to consider a JS framework or something like AMP to maximize control of your markup. Google has even built a calculator to estimate the financial impact of better load times based on the value of a conversion.

That said, there are some real disadvantages in moving away from Drupal’s presentation layer. (Not to mention it’s possible to make extensive use of JavaScript-driven interactivity within a Drupal theme.) By decoupling Drupal, you lose many of Drupal’s out-of-the-box solutions to hard problems such as request-response handling (an essential component of performance), routing, administrative layout tools, authentication, image styles and a preview system. Furthermore, you now have to write schemas for all of your APIs to document them to consumers, and you need to grapple with how to capture presentational semantics like visual hierarchy in textual data.

As Acquia CTO Dries Buytaert wrote in The Future of Decoupled Drupal,

Before decoupling, you need to ask yourself if you're ready to do without functionality usually provided for free by the CMS, such as layout and display management, content previews, user interface (UI) localization, form display, accessibility, authentication, crucial security features such as XSS (cross-site scripting) and CSRF (cross-site request forgery) protection, and last but not least, performance. Many of these have to be rewritten from scratch, or can't be implemented at all, on the client-side. For many projects, building a decoupled application or site on top of a CMS will result in a crippling loss of critical functionality or skyrocketing costs to rebuild missing features.

These things all have to be reinvented in a decoupled site. This is prohibitively expensive for small to medium-sized businesses, but for large enterprises with the resources and a predilection for lean, specific architectures, it’s a reasonable trade-off to harness the power of something like the React library fully.

JavaScript frameworks will continue to grow as consumers demand more app-like experiences on the web and that probably means the percentage of websites directly using a CMS’s presentation layer will shrink over time, but this is going to be a long-tail journey.

JavaScript Eats the Admin Interface

PHP CMSes have a decade lead in producing robust editorial experiences for managing content with a refined GUI. Both WordPress and Drupal have invested thousands of hours in user testing and refinement of their respective user interfaces. Well, wait, you say, aren’t both Drupal and WordPress trying to replace their editorial interfaces with decoupled JavaScript applications to achieve a more app-like experience? Well, yes. Moreover, Gutenberg, the new admin interface in WordPress built on React, is an astonishing evolution for the content authorship experience, a consummation devoutly to be wished. Typically, editors generate content in a third-party application before moving it over to and managing it in a CMS. Gutenberg attempts to create an authorship experience to rival that of Desktop applications typically used for this purpose. At Word Camp 2015, Matt Mullenweg issued a koan-like edict to the WordPress community “learn JavaScript, deeply.” He was preparing the way for Gutenberg.

Meanwhile, on Drupal island, the admin-ui-js team is at work building a decoupled admin interface for Drupal with React, code-named the JavaScript Modernization Initiative. In that sense, JavaScript is influencing the CMS world by beginning to eat the admin interface for two major PHP CMSes. As of this writing, neither interface was part of the current core release.

JavaScript Replaces the Monolithic CMS?

Okay, great, developers love JavaScript, JavaScript devs are easier to hire (perhaps), Node.js can handle concurrency in a more straightforward fashion than some other languages, isomorphic decoupled front-ends can be fast and provide interactivity, and the Drupal and Wordpress admin UIs are being rewritten in React, a JavaScript library, in order to make them more app-like. But our original question was whether JavaScript might eventually eat the Monolithic CMS. Looking at the evidence I’ve produced so far, I think you’d have to argue that this process has begun. Perhaps a better question is what do we, the Drupal community, do about it?

A sophisticated front-end such as a single-page application, a PWA, or a React application, let’s say, still needs a data source to feed it content. And while it’s possible to make use of different services to furnish this data pipeline, editors still need a place to edit content, govern content, and manage meta-data and the relationship between different pieces of content; it’s a task to which the PHP monolithic CMS platforms are uniquely suited.

While some JavaScript CMSes have cropped up—Contentful (an API-first CMS-as-a-service platform), CosmicJS, Prismic, and ApostropheCMS— they don’t have near the feature-set or flexibility of a Drupal when it comes to managing content. As head of Drupal’s JavaScript initiative, Sally Young says, “the new JS CMSes do less than Drupal tries to do, which isn’t necessarily a bad thing, but I still think Drupal’s Field API is the best content modeling tool of any CMS by far.” And it’s more than the fact that these CMSes try to do less, it’s also an issue of maturity.

“I’m not convinced from my explorations of the JS ecosystem that NPM packages and Node.js are mature enough to build something to compete with Drupal,” says senior architect Andrew Berry. “Drupal is still relevant because it’s predicated on libraries that have 5-10 years of development, whereas in the Node world everything is thrown out every 6 months. In Drupal, we can’t always get clients to do major releases, can you imagine if we had to throw it out or change it every 6 months?” This was echoed by other experts that I spoke with.

Conclusion

The monolithic web platforms can’t rest on their laurels and continue to try to be everything to everyone. To adapt, traditional PHP CMSes are going to require strong and sensible leadership that find the best places for each of these tools to shine in conjunction with a stack that includes ever-increasing roles for JavaScript. Drupal, in particular, given its enterprise bent, should embrace its strengths as a content modeling tool in an API-first world—a world where the presentation layer is a separate concern. As Drupal’s API-First initiative lead, Bosch said at Drupalcon Nashville, “we must get into the mindset that we are truly API-first and not just API compatible.” Directing the formidable energy of the community toward this end will help us remain relevant as these changes transpire.

To get involved with the admin-ui-js initiative, start here.

To get involved with the API-first initiative, start here.

Special thanks to Lullabots Andrew Berry, Ben Chavet, John Hannah, Mateu Aguiló Bosch, Mike Herchel, and Sally Young for helping me take on a topic that was beyond my technical comfort zone.

May 30 2018
May 30

Note: This article was originally published on November 29, 2017. Following DrupalCon Nashville, we are republishing (with updates) some of our key articles on decoupled or "headless" Drupal as the community as a whole continues to explore this approach further. Comments from the original will appear unmodified.

As part of the Decoupled Hard Problems series, in this fourth article, I'll discuss some of the challenges surrounding routing, custom paths and URL aliases in decoupled projects. 

Decoupled Routing

It's a Wednesday afternoon, and I'm using the time that Lullabot gives me for professional development to contribute to Contenta CMS. Someone asks me a question about routing for a React application with a decoupled Drupal back-end, so I decide to share it with the rest of the Contenta Slack community and a lengthy conversation ensues. I realize the many tendrils that begin when we separate our routes and paths from a more traditional Drupal setup, especially if we need to think about routing across multiple different consumers. 

It's tempting to think about decoupled Drupal as a back-end plus a JS front-end application. In other words, a website. That is a common use case, probably the most common. Indeed, if we can restrict our decoupled architecture to a single consumer, we can move as many features as we want to the server side. Fantastic, now the editors who use the CMS have many routing tools at their disposal. They can, for instance, configure the URL alias for a given node. URL aliases allow content editors to specify the route of a web page that displays a piece of content. As Drupal developers, we tend to make no distinction between such pieces of content and the web page that Drupal automatically generates for it. That's because Drupal hides the complexity involved in making reasonable assumptions:

  •  It assumes that we need a web page for each node. Each of those has a route node/<nid> and they can have a custom route (aka URL alias).
  •  It means that it is okay to add presentation information in the content model. This makes it easy to tell the Twig template how to display the content (like field_position = 'top-left') in order to render it as the editor intended.

Unfortunately, when we are building a decoupled back-end, we cannot assume that our pieces of content will be displayed on a web page, even if our initial project is a website. That is because when we eventually need a second consumer, we will need to make amends all over the project to undo those assumptions before adding the new consumer.

Understand the hidden costs of decoupling in full. If those costs are acceptable—because we will take advantage of other aspects of decoupling—then a rigorous separation of concerns that assigns all the presentation logic to the front-end will pay off. It takes more time to implement, but it will be worth it when the time comes to add new consumers. While it may save time to use the server side to deal with routing on the assumption that our consumer will be a single website,  as soon as a new consumer gets added those savings turn into losses. And, after all, if there is only a website, we should strongly consider a monolithic Drupal site.

The cost of re-couplingThe cost of re-coupling

After working with Drupal or other modern CMSes, it's easy to assume that content editors can just input what they need for SEO purposes and all the front-ends will follow. But let's take a step back to think about routes:

  • Routes are critical only for website clients. Native applications can also benefit from them, but they can function with just the resource IDs on the API.
  • Routes are important for deep linking in web and native applications. When we use a web search engine in our phone and click a link, we expect the native app to open on that particular content if we have it installed. That is done by mapping the web URL to the app link.
  • Links are a great way to share content. We want users to share links, and then let the appropriate app on the recipient's mobile device open if they have it installed.

It seems clear that even non-browser-centric applications care about the routes of our consumers. Luckily, Drupal considers the URL alias to be part of the content, so it's available to the consumers. But our consumers' routing needs may vary significantly.

Routing From a Web Consumer

Let's imagine that a request to http://cms.contentacms.io/recipes/4-hour-lamb-stew hits our React application. The routing component will know that it needs to use the recipes resource and find the node that has a URL alias of /4-hour-lamb-stew. Contenta can handle this request with JSON API and Fieldable Path—both part of the distribution. With the response to that query, the React app builds all the components and displays the results to the user.

It is important to note the two implicit assumptions in this scenario. The first is that the inbound URL can be tokenized to extract the resource to query. In our case, the URL tells us that we want to query the /api/recipes resource to find a single item that has a particular URL alias. We know that because the URL in the React side contains /recipes/... What happens if the SEO team decides that the content should be under https://cms.contentacms.io/4-hour-lamb-stew? How will React know that it needs to query the /api/recipes resource and not /api/articles?

The second assumption is that there is a web page that represents a node. When we have a decoupled architecture, we cannot guarantee a one-to-one mapping between nodes and pages. Though it's common to have the content model aligned with the routes, let's explore an example where that's not the case. Suppose we have a seasonal page in our food magazine for the summer season (accessible under /summer). It consists of two recipes, and an article, and a manually selected hero image. We can build that easily in our React application by querying and rendering the content. However, everything—except for the data in the nodes and images—lives in the react application. Where does the editor go to change the route for that page?

On top of that, SEO will want it so that when a URL alias changes (either editorially or in the front-end code) a redirect occurs, so people using the old URL can still access the content. Note that a change in the node title could trigger a change in the URL alias via Pathauto. That is a problem even in the "easy" situation. If the alias changes to https://cms.contentacms.io/recipes/four-hour-stewed-lamb, we need our React application to still respond to the old https://cms.contentacms.io/recipes/4-hour-lamb-stew. The old link may have been shared in social networks, linked to from other sites, etc. The problem is that there is no recipe with an alias of /recipes/4-hour-lamb-stew anymore, so the Fieldable Path solution will not cover all cases.

Possible Solutions

In monolithic Drupal, we'd solve the aforementioned SEO issue by using the Redirect module, which keeps track of old path aliases and can respond to them with a redirect to the new one. In decoupled Drupal, we can use that same module along with the new Decoupled Router module (created as part of the research for this article).

The Contenta CMS distribution already includes the Decoupled Router module for routing as we recommend this pattern for decoupled routing.

Pages—or visualizations—that comprise a disconnected selection of entities—our /summer page example—are hard to manage from the back-end. A possible solution could be to use JSON API to query the entities generated by Page Manager. Another possible solution would be to create a content type, with its corresponding resource, specific for that presentation in that particular consumer. Depending on how specific that content type is for the consumer, that will take us to the Back-end For Front-end pattern, which incurs other considerations and maintenance costs.

For the case where multiple consumers claim the same route but have that route resolve to different nodes, we can try the Contextual Aliases module.

The Decoupled Router

Decoupled Router is an endpoint that receives a front-end path and tries to resolve it to an entity. To do so it follows as many redirects and URL aliases as necessary. In the example of /recipes/four-hour-stewed-lamb it would follow the redirect down to /recipes/4-hour-lamb-stew and resolve that URL alias to node:1234. The endpoint provides some interesting information about the route and the underlying entity.

Using the decoulped routerA request for /recipes/four-hour-stewed-lamb resolves into the JSON API resource resolving one redirection and one alias

In a previous post, we discussed how multiple requests degrade performance significantly. With that in mind, making an extra request to resolve the redirects and aliases seems less attractive. We can solve this problem using the Subrequests module. Like we discussed in detail, we can use response tokens to combine several requests in one.

Imagine that we want to resolve /bread and display the title and image. However, we don’t know if /bread will resolve into an article or a recipe. We could use Subrequests to resolve the path and the JSON API entity in a single request.

Subrequests using the decoupled routerWe use the decoupled router to resolve the JSON API URL and then we request that using the {{[email protected]$.jsonapi.individual}} token

In the request above, we provide the path we want to resolve. Then we get the following response.

Response to subrequests using the decoupled routerResponse to the subrequests blueprint that resolves the path and gets the JSON API data (with sparse fieldsets and includes) in a single request

To summarize, we can use Decoupled Router in combination with Subrequests to resolve multiple levels of redirects and URL aliases and get the JSON API data all in a single request. This solution is generic enough that it serves in almost all cases.

Conclusion

Routing in decoupled applications becomes challenging because of three factors:

  • Instead of one route, we have to think about (at least) two, one for the front-end and one for the back-end. We can mitigate this by keeping them both in sync.
  • Multiple consumers may decide different routing patterns. This can be mitigated by reaching an agreement among consumers. Another alternative is to use Contextual Aliases along with Consumers. When we want back-end changes that only affect a particular consumer, we can use the Consumers module to make that dependency explicit. See the Consumer Image Styles module—explained in a previous article—for an example of how to do this.
  • Some visualizations in some of the consumers don’t have a one-to-one correspondence with an entity in the data model. This is solved by introducing dedicated content types for those visualizations. That implies that we have access to both back-end and front-end. A custom resource based on Page Manager could work as well.

In general, whenever we need editorial control we'll have to turn to the back-end CMS. Unfortunately, the back-end affects all consumers, not just one. That may or may not be acceptable, depending on each project. We will need to make sure to consider this when thinking through paths and aliases on our next decoupled Drupal project.

Lucky for us, every project has constraints we can leverage. That is true even when working on the most challenging back-end of all—a public API that powers an unknown number of 3rd-party consumers. For the problem of routing, we can leverage these constraints to use the mitigations listed above.

Hopefully, this article will give you some solutions for your Decoupled Drupal Hard Problems.

Photo by William Bout on Unsplash.

May 25 2018
May 25

Earlier this year Lullabot was engaged to do a redesign of Pantheon’s homepage and several landing pages. If you’re not familiar with Pantheon, they’re one of the leading hosting companies in the Drupal and WordPress space. The timeline for this project was very ambitious—a rollout needed to happen before Drupalcon Nashville. Being longtime partners in the Drupal community, we were excited to take this on.

The front-end development team joined while our design team was still hard at work. Our tasks were to 1) get familiar with the current Drupal 7 codebase, and 2) make performance improvements where possible. All of this came with the caveat that when the designs landed, we were expected to move on them immediately.

Jumping into Pantheon’s codebase was interesting. It’s a situation that we’ve seen hundreds of times: the codebase starts off modularly, but as time progresses and multiple developers work on it, the codebase grows larger, slower, and more unwieldy. This problem isn’t specific to Drupal – it’s a common pandemic across many technologies.

Defining website speed

Website speed is a combination of back-end speed and browser rendering speed, which is determined by the front-end code. Pantheon’s back-end speed is pretty impressive. In addition to being built atop Pantheon’s infrastructure, they have integrated Fastly CDN with HTTP/2.

80-90% of the end-user response time is spent on the front-end. Start there. – Steve Souders

In a normal situation, the front-end accounts for 80% of the time it takes for the browser to render a website. In our situation, it was more like 90%.

What this means is that there are a lot of optimizations we can make to make the website even faster. Exciting!

Identifying the metrics we’ll use

For this article, we’re going to concentrate on four major website speed metrics:

  1. Time to First Byte – This is the time it takes for the server to get the first byte of the HTML to you. In our case, Pantheon’s infrastructure has this handled extremely well.
  2. Time to First Paint – This is when the browser has initially finished layout of the DOM and begins to paint the screen.
  3. Time to Last Hero Paint – This is the time it takes for the browser to finish painting the hero in the initial viewport.
  4. Time to First Interactive – This is the most important metric. This is when the website is painted and becomes usable (buttons clickable, JavaScript working, etc.).

We’re going to be looking at Chrome Developer Tools’ Performance Panel throughout this article. This profile was taken on the pre-design version of the site and identifies these metrics.

Devtools performance panel annotated

Front-end Performance with H2

HTTP/2 (aka H2) is a new-ish technology on the web that governs how servers transfer data back and forth between the browser. With the previous version of HTTP (1.1), each resource request required an entire TCP handshake that introduced additional latency per request. To mitigate this, front-end developers would aggregate resources into as few files as possible. For example, front-end developers would combine multiple CSS files into one monolithic file. We would also combine multiple small images, into one “sprite” image, and display only parts of it at a time. These “hacks” cut down on the number of HTTP requests.

With H2, these concerns are largely mitigated by the protocol itself, leaving the front-end developer to split resources into their own logical files. H2 allows multiplexing of multiple files under one TCP stream.  Because Pantheon has H2 integrated with its global CDN, we were able to make use of these new optimizations.

Identifying the problem(s)

The frontend was not optimized. Let’s take a look at the network tab in Chrome Developer Tools:

Network tab showing 1.4MB CSS file

The first thing to notice is that the aggregated CSS bundle is 1.4MB decompressed. This is extremely large for a CSS file, and merited further investigation.

The second thing to observe: the site is loading some JavaScript files that may not be in use on the homepage.

Performance tab showing flame chart of JavaScript execution

Let's optimize the CSS bundle first

A CSS bundle of 1.4MB is mongongous and hurts our Time to First Paint metric, as well as all metrics that occur afterward. In addition to having to download a larger file, the browser must also parse and interpret it.

Looking deeper into this CSS bundle, we found some embedded base64 and encoded SVG images. When using HTTP/1.1, this saved the round-trip request to the server for that resource but means the browser downloads the image regardless of whether it's needed for that page. Furthermore, we discovered many of these images belonged to components that weren’t in use anymore.

Similarly, various landing pages were only using a small portion of the monolithic CSS file. To decrease the CSS bundle size, we initially took a two-pronged approach:

  1. Extract the embedded images, and reference them via a standard CSS background-image property. Because Pantheon comes with integrated HTTP/2, there’s virtually no performance penalty to load these as separate files.
  2. Split the monolithic CSS file into multiple files, and load those smaller files as needed, similar to the approach taken by modern “code splitting” tools.

Removing the embedded images dropped the bundle to about 700KB—a big improvement, but still a very large CSS file. The next step was “code splitting” the CSS bundle into multiple files. Pantheon’s codebase makes use of various Sass partials that could have made this relatively simple. Unfortunately, the Sass codebase had very large partials and limited in-code documentation.

It’s hard to write maintainable Sass. CSS by its very nature “cascades” down the DOM tree into various components. It’s also next-to-impossible to know where exactly in the HTML codebase a specific CSS selector exists. Pantheon’s Sass codebase was a perfect example of these common problems. When moving code around, we couldn't anticipate all the potential visual modifications.

Writing maintainable Sass

Writing maintainable Sass is hard, but it’s not impossible. To do it effectively, you need to make your code easy to delete. You can accomplish this through a combination of methods:

  • Keep your Sass partials small and modular. Start thinking about splitting them up when they reach over 300 lines.
  • Detailed comments on each component detailing what it is, what it looks like, and when it’s used.
  • Use a naming convention like BEM—and stick to it.
  • Inline comments detailing anything that’s not standard, such as !important declarations, magic numbers, hacks, etc.

Refactoring the Sass codebase

Refactoring a tangled Sass codebase takes a bit of trial and error, but it wouldn’t have been possible without a visual regression system built into the continuous integration (CI) system.

Fortunately, Pantheon had BackstopJS integrated within their CI pipeline. This system looks at a list of representative URLs at multiple viewport widths. When a pull request is submitted, it uses headless Chrome to reach out to the reference site, and compare it with a Pantheon Multidev environment that contains the code from the pull request. If it detects a difference, it will flag this and show the differences in pink.

Visual Regression diff with pink highlighting changes

Through blood, sweat, and tears, we were able to significantly refactor the Sass codebase into 13 separate files. Then in Drupal, we wrote a custom preprocess function that only loads CSS for each content type.

From this, we were able to bring the primary CSS file down to a manageable 400KB (and only 70KB over the wire). While still massive, it’s no longer technically mongongous. There are still some optimizations that can be made, but through these methods, we decreased the size of this file to a third of its original size.

Optimizing the JavaScript stack

Pantheon’s theme was created in 2015 and made heavy use of jQuery, which was a common practice at the time. While we didn’t have the time or budget to refactor jQuery out of the site, we did have time to make some simple, yet significant, optimizations.

JavaScript files take much more effort for the browser to interpret than a similarly sized image or media file. Each JavaScript file has to be compiled on the fly and then executed, which utilizes up the main thread and delays the Time to Interactive metric. This causes the browser to become unresponsive and lock up and is very common on low-powered devices such as mobile phones.

Also, JavaScript files that are included in the header will also block rendering of the layout, which delays the Time to First Paint metric.

Devtools performance panel with main thread utilization hightlightedJavaScript main thread utilization

The easiest trick to avoid delaying these metrics is simple: remove unneeded JavaScript. To do this, we inventoried the current libraries being used and made a list of the selectors that were instantiating the various methods. Next, we scraped the entirety of the HTML of the website using Wget, and cross-referenced the scraped HTML for these selectors.

# Recursively scrape the site, ignore specific extensions, and append .html extension.
wget http://panther.local -r -R gif,jpg,pdf,css,js,svg,png –adjust-extension

Once we had a list of where and when we needed each library, we modified Drupal’s preprocess layer to load them only when necessary. Through this, we reduced the JavaScript bundle size by several hundred kilobytes! In addition, we moved as many JavaScript files into the footer as possible (thus improving Time to First Paint).

Additional front-end performance wins

After the new designs were rolled out, we took an additional look at the site from a performance standpoint.

Devtools performance tab showing a late layout operation* Note third party scripts (that contribute to more relayouts) are being blocked in this screenshot.

You can see in the image above that there’s an additional layout operation happening late in the rendering process. If we look closely, Chrome Developer Tools allows us to track this down.

The cause of this late layout operation down is a combination of factors: First, we’re using the font-display: swap; declaration. This declaration tells the browser to render the page using system fonts first, but then re-render when the web fonts are downloaded. This is good practice because it can dramatically decrease the Time to First Paint metric on slow connections.

Here, the primary cause of this late layout operation is that the web fonts were downloaded late. The browser has to first download and parse the CSS bundle and then reconcile that with the DOM in order to begin downloading the web fonts. This eats up vital microseconds.

Devtools network tab showing fonts downloading late

In the image above, note the low priority images being downloaded before the high priority web fonts. In this case, the first web font to download was the 52nd resource to download!

Preloading web fonts via resource hints

What we really need to do is get our web fonts loaded earlier. Fortunately, we can preload our web fonts with resource hints!

<link rel="preload" href="/sites/all/themes/zeus/fonts/tablet_gothic/360074_3_0.woff2" as="font" type="font/woff2" crossorigin>
<link rel="preload" href="/sites/all/themes/zeus/fonts/tablet_gothic/360074_2_0.woff2" as="font" type="font/woff2" crossorigin>
<link rel="preload" href="/sites/all/themes/zeus/fonts/tablet_gothic/360074_4_0.woff2" as="font" type="font/woff2" crossorigin>
<link rel="preload" href="/sites/all/themes/zeus/fonts/tablet_gothic/360074_1_0.woff2" as="font" type="font/woff2" crossorigin>
<link rel="preload" href="/sites/all/themes/zeus/fonts/tablet_gothic_condensed/360074_5_0.woff2" as="font" type="font/woff2" crossorigin>
Devtools network tab showing fonts downloading immediately after document

Yes! With resource hints, the browser knows about the files immediately and will download the files as soon as it is able to. This eliminates the re-layout and associated flash of unstyled content (FOUT). It also decreases the time to the Time to Last Hero Paint metric.

Devtools performance tab without a late layout operation

Preloading responsive images via resource hints

Let’s look at the Film Strip View within Chrome Developer Tools’ Network Tab. To do this, you click the toolbar icon that looks like a camera. When we refresh, we can see that everything is loading quickly—except for the hero’s background image. This affects the Last Hero Paint metric.

Devtools network tab showing hero background image loading late

If you we hop over to the Network tab, we see that the hero image was the 65th resource to be downloaded. This is because the image is being referenced via the CSS background-image property. To download the image, the browser must first download and interpret the CSS bundle and then cross-reference that with the DOM.

We need to figure out a way to do resource hinting for this image. However, to save bandwidth, we load different versions of the background image dependent on the screen width. To make sure we download the correct image, we include the media attribute on the link element.

We need to make sure we don’t load multiple versions of the same background image, and we certainly do not want to fetch the unneeded resources.

  <link rel="preload" href="/<?php print $zeus_theme_path; ?>/images/new-design/homepage/hero-image-primary--small.jpg" as="image" media="(max-width: 640px)">
  <link rel="preload" href="/<?php print $zeus_theme_path; ?>/images/new-design/homepage/hero-image-primary--med.jpg" as="image" media="(min-width: 640px) and (max-width: 980px)">
  <link rel="preload" href="/<?php print $zeus_theme_path; ?>/images/new-design/homepage/hero-image-primary--med-large.jpg" as="image" media="(min-width: 980px) and (max-width: 1200px)">
  <link rel="preload" href="/<?php print $zeus_theme_path; ?>/images/new-design/homepage/hero-image-primary.jpg" as="image" media="(min-width: 1200px)">

At this point, the correct background image for the viewport is being downloaded immediately after the fonts, and this completely removes the FOUT (flash of unstyled content)—at least on fast connections.

Conclusion

Front-end website performance is a constantly moving target but is critical to the overall speed of your site. Best practices evolve constantly. Also, modern browsers bring constant updates to performance techniques and tools needed to identify problems and optimize rendering. These optimizations don’t have to be difficult, and can typically be done in hours.

Devtools performance panel showing before/after optimizations

Special thanks

Special thanks to my kickass coworkers on this project: Marc Drummond, Jerad Bitner, Nate Lampton, and Helena McCabe. And to Lullabot’s awesome designers Maggie Griner, Marissa Epstein, and Jared Ponchot.

Also special thanks to our amazing counterparts at Pantheon: Matt Stodolnic, Sarah Fruy, Michelle Buggy, Nikita Tselovalnikov, and Steve Persch.

May 25 2018
May 25
Matt and Mike talk with Andrei Mateescu and Tim Millwood about Workspace, what it is, and including it in Drupal core.
May 23 2018
May 23

Note: This article was originally published on November 3, 2017. Following DrupalCon Nashville, we are republishing (with updates) some of our key articles on decoupled or "headless" Drupal as the community as a whole continues to explore this approach further. Comments from the original will appear unmodified.

The Schemata module is our best approach so far in order to provide schemas for our API resources. Unfortunately, this solution is often not good enough. That is because the serialization component in Drupal is so flexible that we can’t anticipate the final form our API responses will take, meaning the schema that our consumers depend on might be inaccurate. How can we improve this situation?

This article is part of the Decoupled hard problems series. In past articles, we talked about request aggregation solutions for performance reasons, and how to leverage image styles in decoupled architectures.

TL;DR

  • Schemas are key for an API's self-generated documentation
  • Schemas are key for the maintainability of the consumer’s data model.
  • Schemas are generated from Typed Data definitions using the Schemata module. They are expressed in the JSON Schema format.
  • Schemas are statically generated but normalizers are determined at runtime.

Why Do We Need Schemas?

A database schema is a description of the data a particular table can hold. Similarly, an API resource schema is a description of the data a particular resource can hold. In other words, a schema describes the shape of a resource and the datatype of each particular property.

Consumers of data need schemas in order to set their expectations. For instance, the schema tells the consumer that the body property is a JSON object that contains a value that is a string. A schema also tells us that the mail property in the user resource is a string in the e-mail format. This knowledge empowers consumers to add client-side form validation for the mail property. In general, a schema will help consumers to have a prior understanding of the data they will be fetching from the API, and what data objects they can write to the API.

We are using the resource schemas in the Docson and Open API to generate automatic documentation. When we enable JSON API and  Open API you get a fully functional and accurately documented HTTP API for your data model. Whenever we make changes to a content type, that will be reflected in the HTTP API and the documentation automatically. All thanks to the schemas.

A consumer could fetch the schemas for all the resources it needs at compile time or fetch them once and cache them for a long time. With that information, the consumer can generate its models automatically without developer intervention. That means that with a single implementation once, all of our consumers’ models are done forever. Probably, there is a library for our consumer’s framework that does this already.

More interestingly, since our schema comes with type information our schemas can be type safe. That is important to many languages like Swift, Java, TypeScript, Flow, Elm, etc. Moreover, if the model in the consumer is auto-generated from the schema (one model per resource) then minor updates to the resource are automatically reflected in the model. We can start to use the new model properties in Angular, iOS, Android, etc.

In summary, having schemas for our resources is a huge improvement for the developer experience. This is because they provide auto-generated documentation of the API and auto-generated models for the consumer application.

How Are We Generating Schemas In Drupal?

One of Drupal 8's API improvements was the introduction of the Typed Data API. We use this API to declare the data types for a particular content structure. For instance, there is a data type for a Timestamp that extends an Integer. The Entity and Field APIs combine these into more complex structures, like a Node.

JSON API and REST in core can expose entity types as resources out of the box. When these modules expose an entity type they do it based on typed data and field API. Since the process to expose entities is known, we can anticipate schemas for those resources.

In fact, assuming resources are a serialization of field API and typed data is the only thing we can do. The base for JSON API and REST in core is Symfony's serialization component. This component is broken into normalizers, as explained in my previous series. These normalizers transform Drupal's inner data structures into other simpler structures. After this transformation, all knowledge of the data type, or structure is lost. This happens because the normalizer classes do not return the new types and new shapes the typed data has been transformed into. This loss of information is where the big problem lies with the current state of schemas.

The Schemata module provides schemas for JSON API and core REST. It does it by serializing the entity and typed data. It is only able to do this because it knows about the implementation details of these two modules. It knows that the nid property is an integer and it has to be nested under data.attributes in JSON API, but not for core REST. If we were to support another format in Schemata we would need to add an ad-hoc implementation for it.

The big problem is that schemas are static information. That means that they can't change during the execution of the program. However, the serialization process (which transforms the Drupal entities into JSON objects) is a runtime operation. It is possible to write a normalizer that turns the number four into 4 or "four" depending if the date of execution ends in an even minute or not. Even though this example is bizarre, it shows that determining the schema upfront without other considerations can lead to errors. Unfortunately, we can’t assume anything about the data after its serialized.

We can either make normalization less flexible—forcing data types to stay true to the pre-generated schemas—or we can allow the schemas to change during runtime. The second option clearly defeats the purpose of setting expectations, because it would allow a resource to potentially differ from the original data type specified by the schema.

The GraphQL community is opinionated on this and drives the web service from their schema. Thus, they ensure that the web service and schema are always in sync.

How Do We Go Forward From Here

Happily, we are already trying to come up with a better way to normalize our data and infer the schema transformations along the way. Nevertheless, whenever a normalizer is injected by a third party contrib module or because of improved normalizations with backward compatibility the Schemata module cannot anticipate it. Schemata will potentially provide the wrong schema in those scenarios. If we are to base the consumer models on our schemas, then they need to be reliable. At the moment they are reliable in JSON API, but only at the cost of losing flexibility with third-party normalizers.

One of the attempts to support data transformations and the impact they have on the schemas are Field Enhancers in JSON API Extras. They represent simple transformations via plugins. Each plugin defines how the data is transformed, and how the schema is affected. This happens in both directions, when the data goes out and when the consumers write back to the API and the transformation needs to be reversed. Whenever we need a custom transformation for a field, we can write a field enhancer instead of a normalizer. That way schemas will remain correct even if the data change implies a change in the schema.

Field Enhancers in JSON API ExtrasField Enhancers in JSON API Extras are aware of schema changes.

We are very close to being able to validate responses in JSON API against schemas when Schemata is present. It will only happen in development environments (where PHP’s asserts are enabled). Site owners will be able to validate that schemas are correct for their site, with all their custom normalizers. That way, when a site owner builds an API or makes changes they'll be able to validate the normalized resource against the purported schema. If there is any misalignment, a log message will be recorded.

Ideally, we want the certainty that schemas are correct all the time. While the community agrees on the best solution, we have these intermediate measures to have reasonable certainty that your schemas are in sync with your responses.

Join the discussion in the #contenta Slack channel or come to the next API-First Meeting and show your interest there!

Hero photo by Oliver Thomas Klein on Unsplash.

May 16 2018
May 16

Note: This article was originally published on October 25, 2017. Following DrupalCon Nashville, we are republishing (with updates) some of our key articles on decoupled or "headless" Drupal as the community as a whole continues to explore this approach further. Comments from the original will appear unmodified.

As part of the API-First Drupal initiative, and the Contenta CMS community effort, we have come up with a solution for using Drupal image styles in a decoupled setup. Here is an overview of the problems we sought to solve:

  • Image styles are tied to the designs of the consumer, therefore belonging to the front-end. However, there are technical limitations in the front-end that make it impossible to handle them there.
  • Our HTTP API serves an unknown number of consumers, but we don't want to expose all image styles to all consumers for all images. Therefore, consumers need to declare their needs when making API requests.
  • The Consumers and Consumer Image Styles modules can solve these issues, but it requires some configuration from the consumer development team.

Image Styles Are Great

Drupal developers are used to the concept of image styles (aka image derivatives, image cache, resized images, etc.). We use them all the time because they are a way to optimize performance on our Drupal-rendered web pages. At the theme layer, the render system will detect the configuration on the image size and will crop it appropriately if the design requires it. We can do this because the back-end is informed of how the image is presented.

In addition to this, Drupal adds a token to the image style URLs. With that token, the Drupal server is saying I know your design needs this image style, so I approve the use of it. This is needed to avoid a malicious user to fill up our disk by manually requesting all the combinations of images and image styles. With this protection, only the combinations that are in our designs will be possible because Drupal is giving a seal of approval. This is transparent to us so our server is protected without even realizing this was a risk.

The monolithic architecture allows us to have the back-end informed about the design. We can take advantage of that situation to provide advanced features.

The Problem

In a decoupled application your back-end service and your front-end consumer are separated. Your back-end serves your content, and your front-end consumer displays and modifies it. Back-end and front-end live in different stacks and are independent of each other. In fact, you may be running a back-end that exposes a public API without knowing which consumers are using that content or how they are using it.

In this situation, we can see how our back-end doesn't know anything about the front-end(s) design(s). Therefore we cannot take advantage of the situation like we could in the monolithic solution.

The most intuitive solution would be to output all the image styles available when requesting images via JSON API (or REST core). This will only work if we have a small set of consumers of our API and we can know the designs for those. Imagine that our API serves to three, and only three, consumers A, B and C. If we did that, then when requesting an image from consumer A we would output all the variations for all the image styles for all the consumers. If each consumer has 10 - 15 image styles, that means 30 - 45 image styles URLs, where only one will be used.

many images

This situation is not ideal because a malicious user can still generate 45 images in our disk for each image available in our content. Additionally, if we consider adding more consumers to our digital experience we risk making this problem worse. Moreover, we don't want the presentation from one consumer sipping through another consumer. Finally, if we can't know the designs for all our consumers, then this solution is not even on the table because we don't know what image styles we need to add to our back-end.

On top of all these problems regarding the separation of concerns of front-end and back-end, there are several technical limitations to overcome. In the particular case of image styles, if we were to process the raw images in the consumer we would need:

  • An application runner able to do these operations. The browser is capable of this, but other more challenged devices won't.
  • A powerful hardware to compute image manipulations. APIs often serve content to hardware with low resources.
  • A high bandwidth environment. We would need to serve a very high-resolution image every time, even if the consumer will resize it to 100 x 100 pixels.

Given all these, we decided that this task was best suited for a server-side technology.

In order to solve this problem as part of the API-First initiative, we want a generic solution that works even in the worst case scenario. This scenario is an API served by Drupal that serves an unknown number of 3rd party applications over which we don't have any control.

How We Solved It

After some research about how other systems tackle this, we established that we need a way for consumers to declare their presentation dependencies. In particular, we want to provide a way to express the image styles that consumer developers want for their application. The requests issued by an iOS application will carry a token that identifies the consumer where the HTTP request originated. That way the back-end server knows to select the image styles associated with that consumer.

no consumer leaks

For this solution, we developed two different contributed modules: Consumers, and Consumer Image Styles.

The Consumers Project

Imagine for a moment that we are running Facebook's back-end. We defined the data model, we have created a web service to expose the information, and now we are ready to expose that API to the world. The intention is that any developer can join Facebook and register an application. In that application record, the developer does some configuration and tweaks some features so the back-end service can interact optimally with the registered application. As the manager of Facebook's web services, we are not to take special request from any of the possible applications. In fact, we don't even know which applications integrate with our service.

The Consumers module aims to replicate this feature. It is a centralized place where other modules can require information about the consumers. The front-end development teams of each consumer are responsible for providing that information.

This module adds an entity type called Consumer. Other modules can add fields to this entity type with the information they want to gather about the consumer. For instance:

  • The Consumer Image Styles module adds a field that allows consumer developers to list all the image styles their application needs.
  • Other modules could add fields related to authentication, like OAuth 2.0.
  • Other could gather information for analytic purposes.
  • Maybe even configuration to integrate with other 3rd party platforms, etc.

The Consumer Image Styles Project

Internally, the Consumers module takes a request containing the consumer ID and returns the consumer entity. That entity contains the list of image styles needed by that consumer. Using that list of image styles Consumer Image Styles integrates with the JSON API module and adds the URLs for the image after applying those styles. These URLs are added to the response, in the meta section of the file resource. The Consumers project page describes how to provide the consumer ID in your request.

{

   "data": {

       "type": "files",

       "id": "3802d937-d4e9-429a-a524-85993a84c3ed"

       "attributes": { … },

       "relationships": { … },

       "links": { … },

       "meta": {

           "derivatives": {

               "200x200": "https://cms.contentacms.io/sites/default/files/styles/200x200/public/boyFYUN8.png?itok=Pbmn7Tyt",

               "800x600": "https://cms.contentacms.io/sites/default/files/styles/800x600/public/boyFYUN8.png?itok=Pbmn7Tyt"

           }

       }

   }

}

To do that, Consumer Image Styles adds an additional normalizer for the image files. This normalizer adds the meta section with the image style URLs.

Conclusion

We recommend having a strict separation between the back-end and the front-end in a decoupled architecture. However, there are some specific problems, like image styles, where the server needs to have some knowledge about the consumer. In these very few occasions the server should not implement special logic for any particular consumer. Instead, we should have the consumers add their configuration to the server.

The Consumers project will help you provide a unified way for app developers to include this information on the server. Consumer Image Styles and OAuth 2.0 are good examples where that is necessary, and examples of how to implement it.

Further Your Understanding

If you are interested in alternative ways to deal with image derivatives in a decoupled architecture. There are other alternatives that may incur extra costs, but still worth checking: Cloudinary, Akamai Image Converter, and Origami.

Hero Image by Sadman Sakib. Also thanks to Daniel Wehner for his time spent on code and article reviews.

May 11 2018
May 11

Drupal 8.6 sees the addition of Node.js based functional browser testing with Nightwatch.js. Nightwatch uses the W3C WebDriver API to perform commands and assertions on browser DOM elements in real-time.

Up until now, the core method for testing JavaScript and browser interactions in Drupal has been to use either Simpletest or PHPUnit. For tests that require a simulated browser, these have been running on PhantomJS, which is now minimally maintained and has been crashing frequently on Drupal CI (as of 8.5, Drupal can now use Chromedriver for these tests, however the majority of tests are still running on Phantom). This situation requires Drupal themers and front-end developers, working primarily with HTML / CSS / JavaScript, to learn the PHPUnit testing framework and specifics of its implementation within Drupal, instead of being able to write tests for page interactions and JavaScript in JavaScript itself.

Nightwatch logo of an owl

Nightwatch is a very popular functional testing framework that has now been integrated into Drupal to allow you to test your JavaScript with JavaScript! Drupal's implementation uses Chromedriver out of the box, however it can also be used with many other browsers via Selenium. The choice to use Chromedriver was because it's available as a standalone package and so doesn't require you to install Java and Selenium, as well as running the tests slightly quicker.

module.exports = {
  before: function(browser) {
    browser
      .installDrupal();
  },
  after: function(browser) {
    browser
      .uninstallDrupal();
  },
  'Visit a test page and create some test page': (browser) => {
    browser
      .relativeURL('/test-page')
      .waitForElementVisible('body', 1000)
      .assert.containsText('body', 'Test page text')
      .relativeUrl('/node/add/page')
      .setValue('input[name=title]', 'A new node')
      .setValue('input[name="body[0][value]"]', 'The main body')
      .click('#edit-submit')
      .end();
  },

};

A very powerful feature of Nightwatch is that you can optionally watch the tests run in real-time in your browser, as well as use the .pause() command to suspend the test and debug directly in the browser.

Videos require iframe browser support.

Right now it comes with one example test, which will install Drupal and check for a specific string. The provided .installDrupal command will allow you to pass in different scripts to set Drupal up in various scenarios (e.g., different install profiles). Another available command will record the browser's console log if requested so that you can catch any fun JavaScript errors. The next steps in development for core are to write and provide more helpful commands, such as logging a user in, as well as porting over the 8.x manual testing plan to Nightwatch.

Jon Snow saying "I'm ready"

You can try Nightwatch out now by grabbing a pre-release copy of Drupal 8.6.x and following the install instructions, which will show you how to test core functionality, as well as how to use it to test your existing sites, modules, and themes by providing your own custom commands, assertions, and tests. For further information on available features, see the Nightwatch API documentation and guide to creating custom commands and assertions. For core developers and module authors, your Nightwatch tests will now be run by Drupal CI and can be viewed in the test log. For your own projects, you can easily run the tests in something such as CircleCI, which will give you access to artifacts such as screenshots and console logs.

Failed test in CircleCI Test artifacts in CircleCI
May 09 2018
May 09

It's now been over 10 years since Drupal started shipping jQuery as part of core, and although a lot of Drupal's JavaScript has evolved since then it's still based on the premise of progressive enhancement of server-side generated pages. Meanwhile, developing sites with JavaScript has made huge strides thanks to popular rendering frameworks such as React, Angular, and Vue. With some of the web's largest sites relying on client-side only JavaScript, and search engines able to effectively crawl them, it's no longer imperative that a site provides fallback for no JavaScript, particularly for specialised uses such as editorial tasks in a content management system.

At DrupalCon Vienna, the core JavaScript team members decided to adopt a more modern approach to using JavaScript in Drupal to build administrative interfaces, with efforts centred around the React framework from Facebook. To try this out, we decided to build a prototype of the Database Logging page in React, and integrate it into Drupal.

Why Not Vue or Angular?

We chose React because of its popularity, stability, and maturity. Many client projects built with Drupal have successfully leveraged the model of annotating Drupal templates with Angular or Vue directives for quite some time now, but we were wary of building something so tightly coupled with the Drupal rendering monolith. We're very aligned with the API-first initiative and hope that through this work we can create something which is reusable with other projects such as Vue or Angular. The choice of library which renders HTML to the browser is the easiest part of the work ahead, with the hard problems being on the bridge between Drupal and the front-end framework.

What We Learned From The Prototype

prototype database log built with React

As we anticipated, rendering something in React was the easy part— our early challenges arose as we realized there were several missing exposed APIs from Drupal, which we addressed with help from the API-first initiative. Having completed this, we identified a number of major disadvantages to our approach:

  • Although we were creating a component library that could be re-used, essentially we were embedding individual JavaScript applications into different pages of a Drupal theme
  • Our legacy JavaScript was mixed in on the page alongside the new things we were building
  • The build process we currently have for JavaScript can be frustrating to work with because Drupal doesn't have a hard dependency on Node.js (and this is unlikely to change for the time being)
  • A developer coming from the JavaScript ecosystem would find it very unfamiliar

The last point in particular touches on one of the biggest tensions that we need to resolve with this initiative. We don't want to force Drupal's PHP developers also to have to know the ins-and-outs of the JavaScript ecosystem to build module administration forms, but we also don't want developers coming from the world of JavaScript—with promises of "We use React now, so this will be familiar to you!"—to then have to learn PHP and Drupal's internals as well. Having discussed this at length during our weekly initiative meetings, we decided to build an administration interface that is completely separate from Drupal itself, and would only consume it's APIs (aka decoupled). The application would also attempt to generate administration forms for some configuration scenarios.

Hello, world!

We began building our decoupled administration application based on the Create React App starter-kit from Facebook. This is an extremely popular project that handles a lot of the complexities of modern JavaScript tooling and creates a very familiar starting point for developers from the JavaScript ecosystem. So far we haven't needed to eject from this project, and we're enjoying not having to maintain our Webpack configurations, so fingers crossed we can continue with this approach indefinitely.

The first page we decided to build in our new application was the user permissions screen so we could experiment with editing configuration entities. One of the challenges we faced in the implementation of this was the inability to get a list of all possible permissions, as the data we had available from Drupal's API only gave us permissions which had been enabled. We created an Admin UI Support module to start providing us with these missing APIs, with the intention to contribute these endpoints back to Drupal once we have stable implementations.

A demo of the decoupled administration UI for Drupal

We chose to use GitHub as our primary development platform for this initiative, again because we wanted to be in a place familiar to as many developers as possible, as well as to use a set of workflows and tools that are common across the greatest number of open source projects and communities. Using GitHub and CircleCI, we've created an automated build process which allows Drupal developers to import and try out the project with Composer, without requiring them to install and run a build process with Node.js. Over the long-term, we would love to keep this project on GitHub and have it live as a Composer dependency, however, there are logistical questions around that which we'll need to work through in conjunction with the core team.

A New Design

Having got the architectural foundations in place for our application, we then turned to the team working on redesigning the admin UI to collaborate more closely. One of the features we had already built into our application was the ability to fall-back to a regular Drupal theme if the user tried to access a route that hadn't been implemented yet. Using this feature, and trying to keep in mind realistic timelines for launching a fully decoupled admin theme, the team decided that our current path forward will involve several parallel efforts:

  • Create new designs for a refreshed Seven-like theme
  • Adapt the new designs to a regular Drupal theme, so the fallback isn't jarring for the user
  • Build sections of the administration interface in the new React application as designs become available, hopefully starting with the content modeling UI
Updated designs for the Drupal administration interfaceAdmin UI redesign explorations

Hard Problems

We still have a lot of issues to discuss around how the administration application will interact with Drupal, how extensible it can be, and what the developer experience will be like for both module authors and front-end developers. Some of the major questions we're trying to answer are:

  • How and what is a Drupal module able to do to extend the administration UI
  • We're not looking to deprecate the current Drupal theming system, so how can modules indicate which "mode" they support
  • If a module wants to provide its own React component, do we want to include this in our compiled JavaScript bundle, and if so how do we build it when Drupal has no requirement to include Node.js
  • How can modules provide routes in the administration UI, or should they become auto-generated
  • How do we handle and integrate site building tools such as Views, or do we replace this with the ability to swap in custom React components

Forms

How we're going to handle forms has been a big point of discussion, as currently Drupal tightly couples a form's schema, data, validation, and UI. Initially, we took a lot of inspiration from Mozilla's react-jsonschema-form project, which builds HTML forms from JSON schema and separated out these concepts. Nevertheless, a major issue we found with this was it still required a lot of form creation and handling to happen in Drupal code, instead of creating good API endpoints for the data we want to fetch and manipulate.

Code showing a form split into different data models and updating a Bootstrap-based form in real-timereact-jsonschema-form in action

The approach we're currently looking at is auto-generating forms from configuration entities and allowing a module to provide a custom React component if it wants to do something more complex than what auto-generation would provide. Here's a working (unstyled!) example of the site information form, which has been auto-generated from an API endpoint.

Auto-generated version of the Drupal site information form

We're now looking to augment configuration entities with metadata to optionally guide the form display (for example, whether a group of options should be a select dropdown or radio buttons).

Try It Out and Get Involved

We have a Composer project available if you want to check out our progress and try the administration interface out (no previous Drupal installation required!). If you're looking to get involved we have weekly meetings in Slack, a number of issues on GitHub, our initiative plan on drupal.org, and lots of API work. You can also join us in-person at our next sprint at Frontend United in June.

A very special thank you to my initiative co-coordinators Matt Grill and Angie Byron, as well as Daniel Wehner, Ted Bowman, Alex Pott, Cristina Chumillas, Lauri Eskola, the API-First initiative, and all our other contributors for everyone's stellar work on this initiative so far!

⬅️✌️➡️  

May 04 2018
May 04

Functional programming is all the rage, and for good reason. By introducing type systems, immutable values, and enforcing purity in our functions, to name just a few advantages, we can reduce the complexity of our code while bolstering our confidence that it will run with minimal errors. It was only a matter of time before these concepts crept their way into the increasingly sophisticated front-end technologies that power the web.

Projects like ClojureScript, Reason, and Elm seek to fulfill the promise of a more-functional web by allowing us to write our applications with functional programming restraints that compile down to regular ol’ JavaScript for use in the browser. Learning a new syntax and having to rely on a less-mature package ecosystem, however, are a couple roadblocks for many who might be interested in using compile-to-JS languages. Fortunately, great strides have been made in creating libraries to introduce powerful functional programming tenets directly into JavaScript codebases with a gentler learning curve.

One such library is Redux, which is a state-management tool heavily inspired by the aforementioned Elm programming language. Redux allows you to create a single store that holds the state of your entire app, rather than managing that state at the component level. This store is globally-available, allowing you to access the pieces of it that you need in whichever components need them without worrying about the shape of your component tree. The process of updating the store involves passing the store object and a descriptive string, called an action, into a special function called a reducer. This function then creates and returns a new store object with the changes described by the action.

This process is very reliable. We can be sure that the store will be updated in exactly the same way every single time so long as we pass the same action to the reducer. This predictable nature is critical in functional programming. But there’s a problem: what if we want our action to fire-off an API call? We can’t be sure what that call will return or that it’ll even succeed. This is known as a side effect and it’s a big no-no in the FP world. Thankfully, there’s a nice solution for managing these side effects in a predictable way: Redux-Saga. In this article, we’ll take a deeper look at the various problems one might run into while building their Redux-powered app and how Redux-Saga can help mitigate them.

Prerequisites

In this article, we’ll be building an application to store a list of monthly bills. We’ll focus specifically on the part that handles fetching the bills from a remote server. The pattern we’ll look at works just the same with POST requests. We’ll bootstrap this app with create-react-app, which will cover most of the code I don’t explicitly walkthrough.

What is Redux-Saga?

Redux-Saga is a Redux middleware, which means it has access to your app’s store and can dispatch its own actions. Similar to regular reducers, sagas are functions that listen for dispatched actions. Additionally, they perform side effects and return their own actions back to a normal reducer.

Redux Flow with Saga Middleware

By intercepting actions that cause side effects and handling them in their own way, we maintain the purity of Redux reducers. This implementation uses JS generators, which allows us to write asynchronous code that reads like synchronous code. We don’t need to worry about callbacks or race conditions since the generator function will automatically pause on each yield statement until complete before continuing. This improves the overall readability of our code. Let’s take a look at what a saga for loading bills from an API would look like.

 1   import { put, call, takeLatest } from 'redux-saga/effects';
 2   
 3   export function callAPI(method = 'GET', body) {
 4     const options = {
 5       headers,
 6       method
 7     }
 8   
 9     if (body !== undefined) {
10       options.body = body;
11     }
12   
13     return fetch(apiEndpoint, options)
14             .then(res => res.json())
15             .catch(err => { throw new Error(err.statusText) });
16   }
17   
18   export function* loadBills() {
19     try {
20       const bills = yield call(callAPI);
21       yield put({ type: 'LOAD_BILLS_SUCCESS', payload: bills });
22     } catch (error) {
23       yield put({ type: 'LOAD_BILLS_FAILURE', payload: error });
24     }
25   }
26   
27   export function* loadBillsSaga() {
28     yield takeLatest('LOAD_BILLS', loadBills);
29   }

Let’s tackle it line-by-line:

  • Line 1: We import several methods from redux-saga/effects. We’ll use takeLatest to listen for the action that kicks-off our fetch operation, call to perform said fetch operation, and put to fire the action back to our reducer upon either success or failure.
  • Line 3-16: We’ve got a helper function that handles the calls to the server using the fetch API.
  • Line 18: Here, we’re using a generator function, as denoted by the asterisk next to the function keyword.
  • Line 19: Inside, we’re using a try/catch to first try the API call and catch if there’s an error. This generator function will run until it encounters the first yield statement, then it will pause execution and yield out a value.
  • Line 20: Our first yield is our API call, which, appropriately, uses the call method. Though this is an asynchronous operation, since we’re using the yield keyword, we effectively wait until it’s complete before moving on.
  • Line 21: Once it’s done, we move on to the next yield, which makes use of the put method to send a new action to our reducer. Its type describes it as a successful fetch and contains a payload of the data fetched.
  • Line 23: If there’s an error with our API call, we’ll hit the catch block and instead fire a failure action. Whatever happens, we’ve ended up kicking the ball back to our reducer with plain JS objects. This is what allows us to maintain purity in our Redux reducer. Our reducer doesn't get involved with side effects. It continues to care only about simple JS objects describing state changes.
  • Line 27: Another generator function, which includes the takeLatest method. This method will listen for our LOAD_BILLS action and call our loadBills() function. If the LOAD_BILLS action fires again before the first operation completed, the first one will be canceled and replaced with the new one. If you don’t require this canceling behavior, redux-saga/effects offer the takeEvery method.

One way to look at this is that saga functions are a sort-of intercepting reducer for certain actions. We fire-off the LOAD_BILLS action, Redux-Saga intercepts that action (which would normally go straight to our reducer), our API call is made and either succeeds or fails, and finally, we dispatch an action to our reducer that handles the app’s state update. Oh, but how is Redux-Saga able to intercept Redux action calls? Let’s take a look at index.js to find out.

 1   import React from 'react';
 2   import ReactDOM from 'react-dom';
 3   import App from './App';
 4   import registerServiceWorker from './registerServiceWorker';
 5   import { Provider } from 'react-redux';
 6   import { createStore, applyMiddleware } from 'redux';
 7   import billsReducer from './reducers';
 8   
 9   import createSagaMiddleware from 'redux-saga';
10   import { loadBillsSaga } from './loadBillsSaga';
11   
12   const sagaMiddleware = createSagaMiddleware();
13   const store = createStore(
14     billsReducer,
15     applyMiddleware(sagaMiddleware)
16   );
17   
18   sagaMiddleware.run(loadBillsSaga);
19   
20   ReactDOM.render(
21     <Provider store={store}>
22       <App />
23     </Provider>,
24     document.getElementById('root')
25   );
26   registerServiceWorker();

The majority of this code is standard React/Redux stuff. Let’s go over what’s unique to Redux-Saga.

  • Line 6: Import applyMiddleware from redux. This will allow us to declare that actions should be intercepted by our sagas before being sent to our reducers.
  • Line 9: createSagaMiddleware from Redux-Saga will allow us to run our sagas.
  • Line 12: Create the middleware.
  • Line 15: Make use of Redux’s applyMiddleware to hook our saga middleware into the Redux store.
  • Line 18: Initialize the saga we imported. Remember that sagas are generator functions, which need to be called once before values can be yielded from them.

At this point, our sagas are running, meaning they’re waiting to respond to dispatched actions just like our reducers are. Which brings us to the last piece of the puzzle: we have to actually fire off the LOAD_BILLS action! Here’s the BillsList component:

 1   import React, { Component } from 'react';
 2   import Bill from './Bill';
 3   import { connect } from 'react-redux';
 4   
 5   class BillsList extends Component {
 6     componentDidMount() {
 7       this.props.dispatch({ type: 'LOAD_BILLS' });
 8     }
 9   
10     render() {
11       return (
12         <div className="BillsList">
13           {this.props.bills.length && this.props.bills.map((bill, i) =>
14             <Bill key={`bill-${i}`} bill={bill} />
15           )}
16         </div>
17       );
18     }
19   }
20   
21   const mapStateToProps = state => ({
22     bills: state.bills,
23     error: state.error
24   });
25   
26   export default connect(mapStateToProps)(BillsList);

I want to attempt to load the bills from the server once the BillsList component has mounted. Inside componentDidMount we fire off LOAD_BILLS using the dispatch method from Redux. We don’t need to import that method since it’s automatically available on all connected components. And this completes our example! Let’s break down the steps:

  1. BillsList component mounts, dispatching the LOAD_BILLS action
  2. loadBillsSaga responds to this action, calls loadBills
  3. loadBills calls the API to fetch the bills
  4. If successful, loadBills dispatches the LOAD_BILLS_SUCCESS action
  5. billsReducer responds to this action, updates the store
  6. Once the store is updated, BillsList re-renders with the list of bills

Testing

A nice benefit of using Redux-Saga and generator functions is that our async code becomes less-complicated to test. We don’t need to worry about mocking API services since all we care about are the action objects that our sagas output. Let’s take a look at some tests for our loadBills saga:

 1   import { put, call } from 'redux-saga/effects';
 2   import { callAPI, loadBills } from './loadBillsSaga';
 3   
 4   describe('loadBills saga tests', () => {
 5     const gen = loadBills();
 6     
 7     it('should call the API', () => {
 8       expect(gen.next().value).toEqual(call(callAPI));
 9     });
10   
11     it('should dispatch a LOAD_BILLS_SUCCESS action if successful', () => {
12       const bills = [
13         {
14           id: 0,
15           amountDue: 1000,
16           autoPay: false,
17           dateDue: 1,
18           description: "Bill 0",
19           payee: "Payee 0",
20           paid: true
21         },
22         {
23           id: 1,
24           amountDue: 1001,
25           autoPay: true,
26           dateDue: 2,
27           description: "Bill 1",
28           payee: "Payee 1",
29           paid: false
30         },
31         {
32           id: 2,
33           amountDue: 1002,
34           autoPay: false,
35           dateDue: 3,
36           description: "Bill 2",
37           payee: "Payee 2",
38           paid: true
39         }
40       ];
41       expect(gen.next(bills).value).toEqual(put({ type: 'LOAD_BILLS_SUCCESS', payload: bills }));
42     });
43   
44     it('should dispatch a LOAD_BILLS_FAILURE action if unsuccessful', () => {
45       expect(gen.throw({ error: 'Something went wrong!' }).value).toEqual(put({ type: 'LOAD_BILLS_FAILURE', payload: { error: 'Something went wrong!' } }));
46     });
47   
48     it('should be done', () => {
49       expect(gen.next().done).toEqual(true);
50     });
51   });

Here we’re making use of Jest, which create-react-app provides and configures for us. This makes things like describe, it, and expect available without any importing required. Taking a look at what this saga is doing, I’ve identified 4 things I’d like to test:

  • The saga fires off the request to the server
  • If the request succeeds, a success action with a payload of an array of bills is returned
  • If the request fails, a failure action with a payload of an error is returned
  • The saga returns a done status when complete

By leveraging the put and call methods from Redux-Saga, I don’t need to worry about mocking the API. The call method does not actually execute the function, rather it describes what we want to happen. This should seem familiar since it’s exactly what Redux does. Redux actions don’t actually do anything themselves. They’re just JavaScript objects describing the change. Redux-Saga operates on this same idea, which makes testing more straightforward. We just want to assert that the API was called and that we got the appropriate Redux action back, along with any expected payload.

  • Line 5: first we need to initialize the saga (aka run the generator function). Once it’s running we can start to yield values out of it. The first test, then, is simple.
  • Line 8: call the next method of the generator and access its value. Since we used the call method from Redux-Saga instead of calling the API directly, this will look something like this:
{ '@@redux-saga/IO': true, CALL: { context: null, fn: [Function: callAPI], args: [] } }

This is telling us that we’re planning to fire-off the callAPI function as we described in our saga. We then compare this to passing callAPI directly into the call method and we should get the same descriptor object each time.

  • Line 11: Next we want to test that, given a successful response from the API, we return a new action with a payload of the bills we retrieved. Remember that this action will then be sent to our Redux reducer to handle updating the app state.
  • Line 12-40: Start by creating some dummy bills we can pass into our generator.
  • Line 41: Perform the assertion. Again we call the next method of our generator, but this time we pass-in the bills array we created. This means that when our generator reaches the next yield keyword, this argument will be available to it. We then compare the value after calling next to a call using the put method from Redux-Saga with the action.
  • Line 44-46: When testing the failure case, instead of plainly calling the next method on our generator, we instead use the throw method, passing in an error message. This will cause the saga to enter its catch block, where we expect to find an action with the error message as its payload. Thus, we make that assertion.
  • Line 48-50: Finally, we want to test that we’ve covered all the yield statements by asserting that the generator has no values left to return. When a generator has done its job, it will return an object with a done property set to true. If that’s the case, our tests for this saga are complete!

Conclusion

We’ve achieved several objectively useful things by incorporating Redux-Saga into our project:

  • Our async code has a more synchronous look to it thanks to the use of generators
  • Our Redux reducers remain pure (no side effects)
  • Our async code is simpler to test

I hope this article has given you enough information to understand how Redux-Saga works and what problems it solves, and made a case for why you should consider using it.

Further Reading

Header photo by Becky Matsubara

May 01 2018
May 01

At this point, you may have read several DrupalCon retrospectives. You probably know that the best part of DrupalCon is the community aspect. During his keynote, Steve Francia, made sure to highlight how extraordinary the Drupal community is in this regard.

One of the things I, personally, was looking forward to was getting together with the API-First initiative people. I even printed some pink decoupled t-shirts for our joint presentation on the state of the initiative. Wim brought Belgian chocolates!

Mateu, Wim and Gabe in decoupled T-Shirts (with multiple consumers)Mateu, Wim and Gabe in decoupled T-Shirts (with multiple consumers)

I love that at DrupalCon, if you have a topic of interest in an aspect of Drupal, you will find ample opportunity to talk about it with brilliant people. Even if you are coming into DrupalCon without company, you will get a chance to meet others in the sprints, the BoFs, the social events, etc.

During this week, the API-First initiative team discussed an important topic that has been missing from the decoupled Drupal ecosystem: RPC requests. After initial conversations in a BoF, we decided to start a Drupal module to implement the JSON-RPC specification.

The decision log after finishing the BoF at DrupalConThe decision log after finishing the BoF at DrupalCon

Wikipedia defines RPC as follows:

In distributed computing, a remote procedure call (RPC) is when a computer program causes a procedure (subroutine) to execute in a different address space (commonly on another computer on a shared network), which is coded as if it were a normal (local) procedure call, without the programmer explicitly coding the details for the remote interaction.

The JSON API module in Drupal is designed to only work with entities because it relies heavily on the Entity Query API and the Entity subsystem. For instance, it would be nearly impossible to keep nested filters that traverse non-entity resources. On the other hand, core’s REST collections based on Views, do not provide pagination, documentation or discoverability. Additionally, in many instances, Views will not have support for what you need to do.

We need RPC in Drupal for decoupled interactions that are not solely predicated on entities. We’re missing a way to execute actions on the Drupal server and expose data that is not based on entities for read and write. For example, we may want to allow an authenticated remote agent to clear caches on a site. I will admit that some interactions would be better represented in a RESTful paradigm, with CRUD actions in a stateless manner on resources that represent Drupal’s internals. However because of Drupal’s idiosyncrasies sometimes we need to use JSON-RPC. At the end of the day, we need to be pragmatic and allow other developers to resolve their needs in a decoupled project. For instance, the JS initiative needs a list of permissions to render the admin UI, and those are stored in code with a special implementation.

Why the current ecosystem was not enough

After the initial debate, we came to the realization that you can do everything you need with the current ecosystem, but it is error-prone. Furthermore, the developer experience leaves much to be desired.

Custom controllers

One of the recommended solutions has been to just create a route and execute a controller that does whatever you need. This solution has the tendency to lead to a collection of unrelated controllers that are completely undocumented and impossible to discover from the front-end consumer perspective. Additionally, there is no validation of the inputs and outputs for this controller, unless you implement said validation from scratch in every controller.

Custom REST resources

Custom REST resources have also been used to expose this missing non-entity data and execute arbitrary actions in Drupal. Custom REST resources don’t get automatic documentation. They are also not discoverable by consumers. On top of that, collection support is rather limited given that you need to build a custom Views integration if it’s not based on an entity. Moreover, the REST module assumes that you are exposing REST resources. Our RPC endpoints may not fit well into REST resources.

Custom GraphQL queries and mutators

GraphQL solves the problem of documentation and discovery, given it covers schemas as a cornerstone of the implementation. Nevertheless, the complexity to do this both in Drupal and on the client side is non-trivial. Most important, bringing in all the might of GraphQL for this simple task seems excessive. This is a good option if you are already using GraphQL to expose your entities.

The JSON-RPC module

Key contributor Gabe Sullice (Acquia OCTO) and I discussed this problem at length and in the open in the #contenta Slack channel. We decided that the best way to approach this problem was to introduce a dedicated and lightweight tool.

The JSON-RPC module will allow you to create type-safe RPC endpoints that are discoverable and automatically documented. All you need to do is to create a JsonRpcMethod.

Each plugin will need to declare:

  • A method name. This will be the plugin ID. For instance: plugins.list to list all the plugins of a given type.
  • The input parameters that the endpoint takes. This is done via annotations in the plugin definition. You need to declare the schema of your parameters, both for validation and documentation.
  • The schema of the response of the endpoint.
  • The PHP code to execute.
  • The required access necessary to execute this call.

This may seem a little verbose, but the benefits clearly surpass the annoyances. What you will get for free by providing this information is:

  • Your API will follow a widely-used standard. That means that your front-end consumers will be able to use JSON-RPC libraries.
  • Your methods are discoverable by consumers.
  • Your input and outputs are clearly documented, and the documentation is kept up to date.
  • The validation ensures that all the input parameters are valid according to your schema. It also ensures that your code responds with the output your documentation promised.
  • The module takes care of several contrived implementation details. Among those are: error handling, bubbling the cacheability metadata, specification compliance, etc.

As you can see, we designed this module to help Drupal sites implement secure, maintainable, understandable and reliable remote procedure calls. This is essential because custom endpoints are often the most insecure and fragile bits of code in a Drupal installation. This module aims to help mitigate that problem.

Usage

The JSON-RPC module ships with a sub-module called JSON-RPC Core. This sub-module exposes some necessary data for the JS modernization initiative. It also executes other common tasks that Drupal core handles. It is the best place to start learning more about how to implement your plugin.

Let's take a look at the plugins.list endpoint.

/**
 * Lists the plugin definitions of a given type.
 *
 * @JsonRpcMethod(
 *   id = "plugins.list",
 *   usage = @Translation("List defined plugins for a given plugin type."),
 *   access = {"administer site configuration"},
 *   params = {
 *     "page" = @JsonRpcParameterDefinition(factory = "\Drupal\jsonrpc\ParameterFactory\PaginationParameterFactory"),
 *     "service" = @JsonRpcParameterDefinition(schema={"type"="string"}),
 *   }
 * )
 */
class Plugins extends JsonRpcMethodBase {

In the code, you will notice the @JsonRpcMethod annotation. That contains important metadata such as the method's name, a list of permissions and the description. The annotation also contains other annotations for the input parameters. Just like you use @Translation, you can use other custom annotations. In this case, each parameter is a @JsonRpcParameterDefinition annotation that takes either a schema or a factory key.

If a parameter uses the schema key, it means that the input is passed as-is to the method. The JSON schema will ensure validation. If a parameter uses the factory key that class will take control of it. One reason to use a factory over a schema is when you need to prepare a parameter. Passing an entity UUID and upcasting it to the fully-loaded entity would be an example. The other reason to choose a factory is to provide a parameter definition that can be reused in several RPC plugins. An example of this is the pagination parameter for lists of results. The class contains a method that exposes the JSON schema, again, for input validation. Additionally, it should have a ::doTransform() method that can process the input into a prepared parameter output.

The rest of the code for the plugin is very simple. There is a method that defines the JSON schema of the output. Note that the other schemas define the shape of the input data, this one refers to the output of the RPC method.

  /**
   * {@inheritdoc}
   */
  public static function outputSchema() {
    // Learn more about JSON-Schema
    return [
      'type' => 'object',
      'patternProperties' => [
        '.{1,}' => [
          'class' => [ 'type' => 'string' ],
          'uri' => [ 'type' => 'string' ],
          'description' => [ 'type' => 'string' ],
          'provider' => [ 'type' => 'string' ],
          'id' => [ 'type' => 'string' ],
        ],
      ],
    ];
  }

Finally, the ::execute() method does the actual work. In this example, it loads the plugins of the type specified in the service parameter.

  /**
   * {@inheritdoc}
   *
   * @throws \Drupal\jsonrpc\Exception\JsonRpcException
   */
  public function execute(ParameterBag $params) {
    // [Code simplified for the sake of the example]
    $paginator = $params->get('page');
    $service = $params->get('service');
    $definitions = $this->container->get($service)->getDefinitions();
    return array_slice($definitions, $paginator['offset'], $paginator['limit']);
  }

Try it!

The following is a hypothetical RPC method for the sake of the example. It triggers a backup process that uploads the backup to a pre-configured FTP server.

Visit JSON-RPC to learn more about the specification and other available options.

To trigger the backup send a POST request to /jsonrpc in your Drupal installation with the following body:

{
    "jsonrpc": "2.0",
    "method": "backup_migrate.backup",
    "params": {
        "subjects": ["database", "files"],
        "destination": "sftp_server_1"
    },
    "id": "trigger-backup"
}

This would return with the following response:

{
    "jsonrpc": "2.0",
    "id": "trigger-backup",
    "result": {
        "status": "success",
        "backedUp": ["database", "files"]
        "uploadedTo": "/…/backups/my-site-20180524.tar.gz"
    }
}

This module is very experimental at the moment. It’s in alpha stage, and some features are still being ironed out. However, it is ready to try. Please report findings in the issue queue; that's a wonderful way to contribute back to the API-First initiative.

Many thanks to Gabe Sullice, co-author of the module, and passionate debater, for tag teaming on this module. I hope that this module will be instrumental in coming improvements to the user experience both in core's admin UI and actual Drupal sites. This module will soon be part of Contenta CMS.

Header photo by Johnson Wang.

Apr 25 2018
Apr 25

In this episode, Matthew Tift discusses DrupalCon Nashville, the movie The Matrix, and various ways to understand the Drupal community. He plays clips from the Driesnote and Steve Francia's keynote, describes some of his experiences at DrupalCon, and offers ideas for what it might mean to understand "the real" Drupal.

Apr 25 2018
Apr 25

Note: This article was originally published on April 29, 2015. Following DrupalCon Nashville, we are republishing (with updates) some of our key articles on decoupled or "headless" Drupal as the community as a whole continues to explore this approach further. Comments from the original will appear unmodified.

One of the major topics of discussion in the Drupal community has been decoupled (or headless) Drupal. Depending on who you ask, it’s either the best way to build break-through user experiences or nothing short of a pandemic. But what exactly is a decoupled architecture?

A decoupled content store splits the content of a website from how it is displayed on multiple independent systems. Decoupled sites are the logical evolution of splitting content from templates in current CMSs. Decoupled architectures started to become mainstream with the publication of NPR’s Create Once, Publish Everywhere (COPE) series of articles. Other media organizations including Netflix have seen great benefits from a decoupled approach to content.

Like many other solutions in computer science, decoupling is simply adding a layer of technical abstraction between what content producers create and what content consumers see.

Technical decision makers face an important choice when evaluating Drupal 8. When an existing site is upgraded to Drupal 8, how do we decide if we should decouple the site or not? Before we decide to work on a decoupled implementation, it’s critical that everyone, from developers and project managers to content editors and business leaders, understand what decoupling is and how to ensure a decoupled effort is worth the technical risk.

Why Decouple?

I’ve seen many people jump to the conclusion that decoupling will solve problems unrelated to a decoupled architecture. Decoupling doesn’t mean a website will have a cleaner content model or a responsive design. Those are separate (though relevant) solutions for separate problem sets.

These are the specific advantages of a decoupled architecture for a large organization:

  • Clean APIs for mobile apps: Since the website front-end is consuming the same APIs as mobile apps, app developers know that they aren’t a second-tier audience.
  • Independent upgrades: When the content API is decoupled from the front-end, the visual design of a website can be completely rebuilt without back-end changes. Likewise, the back-end systems can be rebuilt without requiring any front-end changes. This is a significant advantage in reducing the risk of re-platforming projects but requires strict attention to be paid to the design of the content APIs.
  • APIs can grow to multiple, independent consumers: New mobile apps can be created without requiring deep access to the back-end content stores. APIs can be documented and made available to third parties or the public at large with little effort.
  • Less reliance on Drupal specialists: Drupal is a unique system in that front-end developers need to have a relatively deep understanding of the back-end architecture to be effective. By defining a clear line between back-end and front-end programming, we broaden our pool of potential developers.
  • Abstraction and constraints reduce individual responsibilities while promoting content reuse: Content producers are freed from needing to worry about an exact presentation on every single front-end that consumes content. Style and layout tweaks are solely the responsibility of each front-end. Meanwhile, front-end developers can trust the semantics of content fields and the relationships between content as determined by the content experts themselves.

Here Be Dragons

At the beginning of a decoupled project, the implementation will seem simple and straight-forward. But don’t be fooled! Decoupled architectures enable flexibility at the cost of simplicity. They aren't without risk.

  • One system becomes a web of systems: A decoupled architecture is more complex to understand and debug. Figuring out why something is broken isn’t just solving the bug, but sorting out whether the problem lies in the request or in the API itself.
  • Strict separation of concerns is required to gain tangible benefits: As front-end applications grow and change, care has to be taken to ensure that front-end display logic isn’t encoded in the API. Otherwise, decoupled systems can slowly create circular dependencies. This leads to systems with all of the overhead of a decoupled architecture and none of the benefits.
  • Drupal out-of-the-box functionality only works for the back-end: Many contributed modules provide pre-built functionality we rely on for Drupal site builds. For example, the Google Analytics module provides deep integration with Drupal users and permissions, "page not found" tracking, and link tracking. In a decoupled architecture, this functionality must be rewritten. Site preview (or even authenticated viewing of content) has to be built from scratch in every front-end, instead of using the features we get for free with Drupal. Need UI localization? Get ready for some custom code. Drupal has solved a lot of problems over the course of its evolution so you don’t have to—unless you decouple.
  • The minimum team size is higher for efficient development: A Drupal site with a small development team is not a good candidate for decoupling unless the content is feeding a large number of other applications. In general, decoupling allows larger teams to work concurrently and more efficiently, but doesn't reduce the total implementation effort.
  • Abstraction and constraints affect the whole business: The wider web publishing industry still has the legacy of the "webmaster". Editors are used to being able to tweak content with snippets of CSS or JavaScript. Product stakeholders often view products as a unified front-end and back-end, so getting the funding to invest in building excellent content APIs is an uphill battle. Post-launch support of decoupled products can lead to short-term fixes that are tightly coupled, negating the original investment in the first place.

The Heuristic

To help identify when decoupling is a good fit for a client, Lullabot uses the following guidelines:

Decoupled architectures may be appropriate when:

  1. The front-end teams require full freedom to structure and display the data.
  2. The front-end team does not have Drupal expertise.
  3. More than one content consumer (such as a website and multiple mobile apps) is live at the same time.
  4. Display front-ends combine data from multiple distinct API sources like CMSs, video management systems, and social media.
  5. A project consists of multiple development teams.

If a project meets some of these criteria, then we’ll begin a deep-dive into what decoupling would require.

  • Does decoupling also require a complete content rewrite, such as when migrating from legacy "full-page" CMSs? We’ve encountered sites that haven’t made the move to structured data yet and still consist primarily of HTML “blobs.” This scenario presents a significant hurdle to decoupling, though it’s a separate problem from decoupling.
  • Does the development team have the time needed to build and document a content API with something like Swagger? Or is using Drupal as a site building (but coupled) development framework a better fit?
  • Does the web team consist primarily of Drupal developers, and will those developers continue to support the website in the future? Would the front-end team be better served by Views, Panels and the theme layer, or by a pure front-end solution like React or Angular?
  • Is there enough value in decoupling that the business is willing to change how they work to see its benefits?

Decoupled architectures are a great solution - but they’re not the only solution. Some of the best websites are built with a completely coupled Drupal implementation. It’s up to us as technical leaders and consultants to ensure we don’t let our excitement over an updated architecture get in between us and what a client truly needs.

Header image by Daniel Schwen CC BY-SA 4.0, from Wikimedia Commons

Apr 18 2018
Apr 18

This first-ever Decoupled Summit at DrupalCon Nashville was a huge hit. Not only did it sell out but the room was packed to the gills, literally standing room only. Decoupled Drupal is a hot topic these days. The decoupled summit was an opportunity to look at the state of decoupled Drupal, analyze pros and cons of decoupling, and look at decoupling strategies and examples. There is lots of interest in decoupling, but there are still many hard problems to solve, and it isn’t the right solution for every situation. This summit was an opportunity to assess the state of best practices.

The summit was organized by Lullabot's Sally Young and Mediacurrent's Matt Davis, two of the innovators in this space.

What is “decoupled Drupal”? 

First, a quick explanation of what “decoupled Drupal” means, in case you haven’t caught the fever yet. Historically, Drupal is used to deliver all the components of a website, an approach that can be called “traditional,” “monolithic,” or “full stack” Drupal. In this scenario, Drupal provides the mechanism to create and store structured data, includes an editorial interface that allows editors to add and edit content and set configuration, and takes responsibility for creating the front-end markup that users see in their browsers. Drupal does it all.

“Decoupled”, or “headless” Drupal is where a site separates these website functions across multiple web frameworks and environments. That could mean managing data creation and storage in a traditional Drupal installation, but using React and Node.js to create the page markup. It could also mean using a React app as an editorial interface to a traditional Drupal site. 

Drupal tools and activity

Drupal core is enabling this activity through a couple of core initiatives:

Drupal and the Drupal community have numerous tools available to assist in creating a decoupled site:

  • Contenta, a pre-configured decoupled Drupal distribution.
  • Waterwheel, an emerging ecosystem of software development kits (SDKs) built by the Drupal community.
  • JSON API, an API that allows consumers to request exactly the data they need, rather than being limited to pre-configured REST endpoints.
  • GraphQL, another API that allows consumers to request only the data they want while combining multiple round-trip requests into one.

There’s lots of activity in headless CMSes. But the competitors are proprietary. Drupal and WordPress are the only end-to-end open source contenders. The others only open source the SDKs.

Highlights of the summit

The summit included several speakers, a business panel, and some demonstrations of decoupled applications. Participants brought up lots of interesting questions and observations. I jotted down several quotes, but it wasn't always possible to give attribution with such an open discussion, so my apologies in advance. Some general reflections from my notes:

Why decouple?

  • More and more sites are delivering content to multiple consumers, mobile apps, TV, etc. In this situation, the website can become just another consumer of the data.
  • It’s easier to find generalist JavaScript developers than expert Drupal developers. Decoupling is one way to ensure the front-end team doesn't have to know anything about Drupal.
  • If you have large teams, a decoupled site allows you to have a clean separation of duties, so the front and back end can work rapidly in parallel to build the site.
  • A modern JavaScript front-end can be fast—although several participants pointed out that a decoupled site is not automatically faster. You still need to pay attention to performance issues.
  • Content is expensive to create; decoupling is a way to re-use it, not just across platforms, but also from redesign to redesign.
  • You could launch a brand new design without making any changes to the back end, assuming you have a well-designed API (meaning an API that doesn't include any assumptions about what the front end looks like). As one participant said, “One day, React won't be cool anymore, we'll need to be ready for the next big thing.”

What are some of the complications?

  • It often or always costs more to decouple than to build a traditional site. There’s additional infrastructure, the need to create new solutions for things that traditional Drupal already does, and the fact that we’re still as a community figuring out the best practices.
  • If you only need a website, decoupling is a convoluted way to accomplish it. Decoupling makes sense when you are building an API to serve multiple consumers.
  • You don’t have to decouple to support other applications. Drupal can be a full-featured website, and also the source of APIs.
  • Some tasks are particularly tricky in a decoupled environment, like previewing content before publishing it. Although some participants pointed out that in a truly decoupled environment preview makes no sense anyway. “We have a bias that a node is a page, but that’s not true in a decoupled context. There is no concept of a page on a smartphone. Preview is complicated because of that.”
  • Many businesses have page-centric assumptions embedded deep into their content and processes. It might be difficult to shift to a model where editors create content that might be deployed in many different combinations and environments. One participant discussed a client that "used all the decoupled technology at their disposal to build a highly coupled CMS." On the other hand, some clients are pure Drupal top to bottom, but they have a good content model and are effectively already "decoupling" their content from its eventual display.
  • Another quote, “Clients trying to unify multiple properties have a special problem; they have to swallow that there will have to be a unified content model in order to decouple. Otherwise, you're building numerous decoupled systems.”
  • Once you are decoupled, you may not even know who is consuming the APIs or how they're being used. If you make changes, you may break things outside of your website. You need to be aware of the dependency you created by serving an API.

Speakers and Panelists

The following is a list of speakers and panelists. These are people and companies you could talk to if you have more questions about decoupling:

  • Sally Young (Lullabot)
  • Matt Davis (Mediacurrent)
  • Jeff Eaton (Lullabot)
  • Preston So (Acquia)
  • Matt Grill (Acquia)
  • Daniel Wehner (TES)
  • Wes Ruvalcaba (Lullabot)
  • Mateu Aguiló Bosch (Lullabot)
  • Suzi Arnold (Comcast)
  • Jason Oscar (Comcast)
  • Jeremy Dickens (Weather.com)
  • Nichole Davison (Edutopia)
  • Baddy Breidert (1xinternet)
  • Christoph Breidert (1xinternet)
  • Patrick Coffey (Four Kitchens)
  • Greg Amaroso (Softvision)
  • Eric Hestenes(Edutopia)
  • David Hwang (DocuSign)
  • Shellie Hutchens (Mediacurrent)
  • Karen Stevenson (Lullabot)

Summary

It was a worthwhile summit, I learned a lot, and I imagine others did as well. Several people mentioned that Decoupled Drupal Days will be taking place August 17-19, 2018 in New York City (there is a link to last year's event). The organizers say it will be “brutally honest, not a cheerleading session.” And they’re also looking for sponsors. I’d highly recommend marking those days on your calendar if you’re interested in this topic!

Apr 04 2018
Apr 04

This article is the second in our series on Continuous Integration tools for Drupal 8, which started with CircleCI. This time, we explore Travis CI.

Travis CI is the most well known CI tool for open source projects. Its setup process is straightforward and it offers a lot of flexibility and resources to implement Continuous Integration for any kind of project. In this article we will implement the same set of jobs that we did with CircleCI and then compare both tools.

Resources

This article makes references to the following resources:

Browse the demo project to discover where the CI components are placed, then use the one-line installer to add these components automatically to your project.

The goal

We want to run the following jobs in a Drupal 8 project when someone creates a pull request:

To accomplish the above, we will use the following tools in Travis CI:

  • Drush, Drupal’s command line interface, to perform Drupal-related tasks like installing Drupal or updating the database.
  • Docker Compose, via docker4drupal, to build the environment where Behat tests run.
  • Robo, a PHP task runner, to define a set of tasks for each of the above jobs.

Here is a screenshot of the Travis CI dashboard with the above setup in place:

Travis CI dashboard

Now, let’s see how this has been set up. If you want to dive straight into the code, have a look at the demo Drupal 8 repository.

Setting up Travis CI

Travis CI requires the presence of a .travis.yml file at the root of the repository that dictates how it will build and test the project. I have used this installer that adds the following files:

Additionally, a few dependencies are added via Composer, which are required for the CI jobs.

After adding the above files to the repository, it’s time to give Travis CI access to it. Open https://travis-ci.org and authenticate there with your GitHub account. Next, add the repository at the Travis CI dashboard as shown below:

Travis CI add project

That’s it! After this, future changes to the repository should trigger builds at Travis CI. If you create a pull request, you will see a status message like the following one:

Travis CI pull request

Seeing the jobs at work

Here is an excerpt of the .travis.yml file. We are leveraging Travis’ build matrix for spinning up three jobs that run in parallel:

env:
  matrix:
    - JOB=job:check-coding-standards
    - JOB=job:run-unit-tests
    - JOB=job:run-behat-tests

install:
  - composer --verbose install

script:
  - vendor/bin/robo $JOB

The script section is called three times: one for each value assigned to the $JOB variable. It calls a different Robo task each time. We decided to write the implementation of each job as Robo tasks because:

  • It makes the .travis.yml file easier to read and maintain.
  • It makes the job implementations portable between CI tools.
  • It gives developers an opportunity to run the jobs locally.

If you are curious what a Robo task looks like, here is the implementation of the one that runs Behat tests:

/**
 * Command to run behat tests.
 *
 * @return \Robo\Result
 *   The result of the collection of tasks.
 */
public function jobRunBehatTests()
{
    $collection = $this->collectionBuilder();
    $collection->addTaskList($this->downloadDatabase());
    $collection->addTaskList($this->buildEnvironment());
    $collection->addTask($this->waitForDrupal());
    $collection->addTaskList($this->runUpdatePath());
    $collection->addTaskList($this->runBehatTests());
    return $collection->run();
}

Building the environment with Docker Compose

The build environment task shown above, $this→buildEnvironment(), uses Docker Compose to build a Docker environment where the Drupal site will be configured, the database will be updated, and finally, Behat tests will run.

In contrast with CircleCI, where we define the mix of Docker images that the test environment will use to run the jobs, Travis CI offers two environments (Precise and Trusty) with common pre-installed services. Trusty has everything that we need for checking coding standards or running PHPUnit tests, but Behat tests require more setup which we find easier to manage via Docker Compose.

Here are the contents of the build environment task. For simplicity, we have removed a few unrelated lines:

/**
 * Builds the Docker environment.
 *
 * @return \Robo\Task\Base\Exec[]
 *   An array of tasks.
 */
protected function buildEnvironment()
{
    $force = true;
    $tasks = [];
    $tasks[] = $this->taskFilesystemStack()
        ->copy('.travis/docker-compose.yml', 'docker-compose.yml', $force);
    $tasks[] = $this->taskExec('docker-compose pull --parallel');
    $tasks[] = $this->taskExec('docker-compose up -d');
    return $tasks;
}

The above task uses this docker-compose.yml file to build the environment.

Generating and visualizing coverage reports

Travis CI does not support storing artifacts like CircleCI does. Therefore, we need to use a third-party service to host them. Travis documentation suggests either uploading them to an Amazon S3 bucket or using Coveralls, a hosted analysis tool. We chose the latter because it posts a summary in each pull request with a link to the full coverage report.

Setting up Coveralls is straightforward. Start by opening https://coveralls.io and then, after authenticating with your GitHub account, use their browser to find and connect to a repository, like this:

Coveralls add repository

Next, it is recommended to review the repository settings so we can customize the developer experience:

Coveralls settings

With that in place, new pull requests will show a status message with a one-line summary of the coverage report, plus a link to the full details:

Coveralls pull request

Finally, when we click on Details, we see the following coverage report:

Coveralls report

A comparison to CircleCI

CircleCI can do all that Travis CI does with less setup. For example, coverage reports and Behat screenshots can be stored as job artifacts and visualized at the CircleCI dashboard. Additionally, CircleCI’s Command Line Interface gives a chance to developers to debug jobs locally.

Travis CI shines on flexibility: for example, only the Behat job uses Docker Compose to build the environment while the rest of the jobs use the Trusty image. Additionally, there is a huge amount of articles and documentation, which you will surely find helpful when tweaking the jobs to fit your team's needs.

If you liked Travis CI, check out this installer to get started quickly in your Drupal 8 project.

What next?

We aren’t sure about which tool to pick for our next article in this series on CI tools for Drupal 8. Do you have a preference? Do you have feedback on what you’ve found relevant about this article? Please let us know by posting a comment.

Acknowledgements

Mar 28 2018
Mar 28

Every development team has a particular way of working. However, over the years I have noticed that there are a few common practices that, once documented and agreed upon, can help considerably in the team’s productivity. This article aims to serve as a template for new and existing projects.

Assumptions

Feel free to copy the following contents to an internal page in your wiki and start discussing and adjusting it with your team. It’s crucial that everyone in the team is willing to give these practices a go for a sprint and then evaluate and adjust during the next sprint retrospective.

The README and CHANGELOG documents

Each repository must have a README file and may have a CHANGELOG document.

The README describes the project’s nature, lists requirements, and contains installation instructions. This document should contain everything that a developer needs to set up the project locally. If you are looking for a Drupal 8 specific one, here is an example.

The CHANGELOG is the living Release Notes document that allows anyone (developers, product managers, etc.) to understand exactly what changes are affiliated with a specific version of the project. This document is used when creating new releases to describe how should other teams update their current installations.

Working with repositories

It’s useful to receive notifications every time that someone makes changes in a repository. We encourage you to subscribe to pull requests in order to start learning from everyone's code by clicking on the eye icon at the top right corner of each repository's homepage:

undefined

Make sure you have a GitHub account with an SSH key. You can add your SSH key here and, after doing that, you can share it with the team by appending .keys to your profile, like this.

Setting up your environment

Each project is different, so look at the repository’s README for instructions on how to set it up. If you find a repository which does not have a README, use this one as a template.

Setting up your editor

You can use any editor to work on projects, but the one that offers the best integration with Drupal 8 at the moment is PHPStorm. Please take a few minutes to adjust PHPStorm to work with Drupal by reading this guide.

Working on a ticket

Each ticket should have its own branch or branches. A branch name should be prefixed with the ticket number and a one or two-word description (for example, PROJECTNAME-123-foo). Here is how you can do it:

cd /var/www/projectname
git checkout master
git pull origin master  // Merge the current master into your branch.
git checkout -b PROJECTNAME-123-foo // create and switch to a new branch 'PROJECTNAME-123-foo'

Once you are ready to push your changes, commit them to the branch with the following commands:

git status // Shows your changes.
git add [file] // Add any new or modified files. Supports patterns.
git add --patch [file] // Better yet, review changes and stage your hunks.
git diff --cached // Shows what is about to be committed. Review it carefully!
git commit -m "PROJECTNAME-123 Description shorter than 50 characters"

Aim to make small commits which you can describe easily in your branch. The Erlang project has a good guide for writing good commit messages. If you want to add further info to the commit message, skip the -m parameter and use the default editor to set the title and the description of the commit message.

If your pull request alters the installation process or requires additional steps during the updating process, then adjust the README and CHANGELOG files accordingly.

Writing and updating tests

Bug fixes and new features will be easier to peer review if the project's tests pass and the new code is covered by the tests. To get an idea of what sort of tests you may find, check out this article.

Ok, ship it!

Once you have completed the above section, push the branch to Github:

git push -u origin PROJECTNAME-123-foo // pushes your branch to the origin repository.
PROJECTNAME-456 Fix video uploads only allowing mp4
undefined

Post a link to the pull request in the respective ticket and set it's status to "Needs Review" in order to track progress within the sprint.

Peer reviews

The rule of thumb of peer reviews is that nobody can push to master branch. Everything should happen in pull requests which are reviewed and merged by the rest of the team.

After creating the pull request, prepare it for code review by following this guide. Once the pull request is ready for review, add reviewers to it through the web interface. Here is an example:

undefined

When merging a pull request, follow the git standards with a short one-line message, a blank line, and paragraphs if needed. Here are two sample merge commits::

[#pr-number] PROJECTNAME-123: Summary of the work being merged
PROJECTNAME-123: Summary of the work being merged #pr-number

Finally, if this pull request completes the Jira ticket, then open it and change its status to the next one, which normally is "In QA", if you have a QA team, or "Done" if not.

How to create a release

If you are distributing code to be deployed by other teams, then your team needs to keep a CHANGELOG which evolves along with the code. Review the section "The README and CHANGELOG documents" for an introduction.

Once the sprint has completed, it is time to create a release. Here are the steps to follow:

  1. If there is Continuous Integration, check that all tests pass and that there is the same or higher coverage that by the end of the previous sprint. If not, request the development team to fix this.
  2. Clone the repository where you want to create the new release.
  3. Create a new branch where you update the module's CHANGELOG by setting a version number at the top. If the module does not have a CHANGELOG file yet, create one using the example further above as a template.
  4. Developers indicate the level bump needed for each change by indicating it with the word "PATCH", "MINOR" or "MAJOR". This takes into account semantic versioning. So, depending on the changes that this release includes you increase either the PATCH, MINOR or MAJOR number for the next version.
  5. Once the pull request has been approved by the team, merge it.
  6. Create a tag: git tag -a [VERSION]
  7. Enter a description of what is being released on this tag.
  8. Push the tag to the repository: git push origin [VERSION].

Note: If you need to double check anything stated in the CHANGELOG while preparing the release, you may use the following command to retrieve all commits since the last release:

git log --oneline --reverse --first-parent <insert last tag version here>.. | cut -c 9-
git log --oneline --reverse --first-parent 1.0.0.. | cut -c 9-

Ticket review guide

Here are a few tips for both sides: the individual working on the code and the people reviewing the pull request.

For the pull request authors

  • Make sure that the checked in code does what the ticket specifies.
  • Use code sniffer to check and fix violations against Drupal’s coding standards.
  • Add comments on the code as you see fit. This is useful not only for maintenance but also for peer reviewers.
  • Make sure that your set of changes are covered by reasonable PHPUnit or Behat tests.
  • Write a summary of what the code does and other related info such as screenshots. Include manual testing instructions if needed.
  • If there is Continuous Integration, verify that all jobs pass.

For the reviewers

  • Verify that all the jobs executed by the Continuous Integration tool are passing. If there are failures, inspect the logs and notify the pull request’s author.
  • Check that the code is easy to follow, respects Drupal’s Coding Standards, and is documented.
  • If there are no tests, consider suggesting how they could be written.
  • Look for performance and security risks. If you need tips, this guide offers great help.

Further tips can be found at The Peer Review How-To guide.

Acknowledgments

This guide is the result of working with the following clients, which we thank deeply:

Plus, the following folks at Lullabot:

Mar 26 2018
Mar 26
In this episode, Matthew Tift talks with Jen Lampton about the Backdrop project, the differences between the Drupal and Backdrop communities, helping people, organizing software communities, and much more.
Mar 21 2018
Mar 21

At Lullabot, we’ve been using Agile techniques for years to help run our software projects. We’ve learned what pieces of Scrum and Kanban seem to fit our teams and what parts are safe to ignore. We’ve learned not just how different Agile methodologies prescribe we should work, but we’ve uncovered what’s actually helpful to us. In other words, we have opinions.

One of those opinions really crystallized among our project managers over the last 12 months, and we want to share it with you. It has to do with the way we think and communicate about the work that our teams do.

User stories

Most of us in the software industry have encountered the idea of user stories. These minimalist requirement statements have been around for two decades, and are one of the fundamental ideas in Agile software development. As such, there’s been a lot of thinking and discussion around them.

Rather than being some definitive statement on user stories, this article is a reflection of the experience of working with this kind of artifact within the specific context of our clients, developers, and project managers here at Lullabot.

A user story is supposed to be a short statement about a task a user wants to do with a particular software. It briefly describes the user, the task, and what benefit the user gets from it.

On typical format of a user story looks like this:

As a <type of user> I want to <do something> so that I can <benefit>.

A more concrete example would be:

As a Night Owl, I want to drink multiple cups of coffee in the morning so that I can be even slightly functional before noon.

Or:

As a visitor to this website, I want to easily search for content so that I can find what I’m looking for quickly.

The brevity of this format is helpful to force people to have actual conversations about the feature they’re building. Once upon a time, development teams and business stakeholders needed to be encouraged to collaborate more, so this was a revolutionary tool.

So, obviously, this is a good idea. But In the course of the last year, we talked about user stories a lot and how they can help or hurt a development project.

Is everything really a story?

Stories are such an operating assumption that many of the software packages that help organize development projects start with ‘Story’ as a default issue type. They’re so normalized in modern software that we’ve even had clients direct us that “everything needs to be a story” in their projects.

Explicitly or implicitly, there’s an idea out there that since we’re Agile (whatever that really means), we have to do user stories for everything. I’ve found myself writing 'As a user' stories uncritically, out of habit, or because I was instructed to do so.

However, there’s often business-driven or system-level requirements, which are not user-focused at all, that have to be crammed into the user story format. It’s easy to apply the user story format to everything, for the sake of consistency, but it’s a bad fit in several common cases. Here’s an example of a technical requirement stuffed into a user story:

As the system, I want to verify a user's OAuth credentials before granting access so that I can ensure secure connections.

I’ve seen and written lots of stories like that. The problem is that this story personifies the system with wants and desires which it does not have. If it were truly a user story, the user would be the focus like in this example:

As an amateur chef, I want to log into the system because I want to access recipes behind the paywall.

You’ll note that the whole character of the story has changed. Security and technology standards are not the primary concern of the user, so they aren't reflected. It doesn't reflect the business requirement of HOW the authentication should happen, but then again the user doesn't care about that. It’s honest but less than effective as a requirement.

As a user, I want you to take my money

This is also a problem for business requirements that are not actually user-focused. They are similarly ill-suited to the user-story format:

As a site visitor, I want to see advertisements so I can know about products and services that might interest me.

We know that no user ever wanted that. They came to the website for the content, and the advertising was a distraction. So that user story is fundamentally NOT about the site visitor—it’s about the revenue model of the site.

User stories function as conversation starters about the value of a piece of work and the ways in which that value might be realized. Our examples above don’t need conversation or discussion. The imperative and authority to do the work come from the organization’s need to provide security or earn revenue.

So how do you express non-user requirements?

In general, it’s better to surrender to common sense and not put these kinds of technical requirements into the user’s voice. Instead, write simple, imperative statements that declare what must be done.

Integrate Google AdSense into article pages. Require a valid OAuth token for access to the system.

We like to surrender to the forces of common sense and call a user story that no longer involves a user what it actually is: a task for a developer to perform.

This might seem like a meaningless distinction—who cares if it’s a story or a task or whatever? But if your mental model of the user is demonstrably false, what else are you getting wrong?

It’s easy to write your biases into the user’s voice and finding yourself keeping the status quo instead of doing something new. If you insist on shaping the conversation around your product from a false premise, how can you spot your real business problems and innovate to solve them?  

To take our example from above, maybe traditional web advertising is a sub-standard way of generating revenue for your business, but you’ll never have that conversation if you paint it over with false user requirements and benefit statements.

What are user stories actually good at?

The beginning of our process is a well-written, truthful story about a feature with benefits for specific kinds of users, accompanied by clear acceptance criteria.

Ideally, we’re starting from a position where there’s been actual user research and one-on-one interviews during the discovery process. That ensures we’re building features that users are actually interested in.

As a content administrator, I want to be notified of new user account requests because I need to review and approve them quickly. As an authenticated user, I want to drill into search results using facets because I’m looking for something very specific. As a vacation planner, I want to visit a page that aggregates content about my country of interest, to help me decide what I might like to do while in that country.

The conversation around actual user-based stories builds understanding between the business and our development team, and it sets us on the right path. It just needs to be recognized that the user story format itself isn't magic.

The life-cycle of a story

It’s natural that stories would be initiated by the business stakeholders based on their knowledge of their users. In the same way, when the developers are done with their work, those same stakeholders will want to verify that the story is properly complete.

That makes stories an ideal artifact to drive QA and acceptance testing. Developers and QA team members both benefit from business-level acceptance criteria elaborating the story to help guide their work.

We like to use Gherkin as a way to write acceptance criteria using a particular format consisting of ‘Given, When, Then’ statements.

  • Given expresses the preconditions of the acceptance criteria. This might be authentication, the existence of some data, or the completion of a business step that must precede the feature under discussion.
  • When expresses the action a user takes.
  • Then expresses the result of the action.

All three of these keywords can be combined with conjunctions like ‘and’ or ‘but’ to layer conditions onto the acceptance criteria. Here’s a simple example:

Given a user has requested an account in the system, and the account request has been reviewed by an admin, and the admin wishes to approve the request. When the admin approves the request, then the account changes state from ‘pending’ to ‘active,’ and the user is notified by email that their account request has been approved. 

This kind of acceptance criteria is great to work up as the story is being discussed, where developers can ask questions of business stakeholders. Gherkin is actually used in software like Cucumber and Behat to drive automation of these test criteria.

Not every client is interested in this kind of test automation, but even if those tests will never end up in code, working with Gherkin to write acceptance criteria has a way of clarifying everyone’s thinking.

When the conversation is done

The story and the acceptance criteria are really valuable for everyone involved. But, when you’ve reached a stopping point and the story is ‘ready for development,’ there’s still more that has to happen. Even with great acceptance criteria, the story still might not express the level of detail that a developer needs.

Beyond the story and acceptance criteria, developers may rely on technical documentation, design artifacts, or architectural planning to fill in the gaps. This is especially true when a story is a single unit of value to the business, but the implementation crosses disciplines and teams.

Stories for everyone, subtasks for developers

It’s helpful at that point to break the story into subtasks which can be assigned and sized for an optimal developer workflow — bite-sized chunks that take no more than 1-2 days to accomplish, and which are often unit testable but aren't always good candidates for QA.

Our account approval example from above can be broken into several tasks. Most of these features are off-the-shelf with Drupal, but we’d probably want to make some adjustments and customize the language in the notification emails.

Again, our task format is short imperative statements, possibly with additional notes as needed in the body of a ticket.

Grant permission to site admins to allow account approvals. Build a list of open account requests. Customize account approval emails.

You may or may not need distinct acceptance criteria here. Some tasks are self-explanatory, while others have the need for more guidance as we describe above.

In any case, QAing these shorter tasks can be cumbersome and confusing for the QA team because the total work for that story is not sufficiently complete. Any single task is only part of the full acceptance criteria, and they’re often interdependent.

However, we’re not without validation for those subtasks. Unit testing and peer review between developers work very well for checking them along the way, especially when we write custom code where errors might be introduced. When all the tasks are done, the complete story can move forward to QA and acceptance testing for the complete feature.

All of this is to say that user stories are an important part of Lullabot’s software development process—but they’re not the only thing we need to get the job done.  

Get more background

I like to know the background and history of these techniques we use to manage our projects. The user stories and Agile development practices have been around for about 25 years, but it sits in the context of the whole history of software. Knowing the history and the thinkers behind it all set you up to use the practices selectively and use them well.

In that light, and to give credit where credit is due, these are some of the folks that did the big thinking around user stories and other Agile practices. If you’re a practitioner of Agile in some way, check out their history and opinions:

And, here a couple of books that have been helpful:

Mar 07 2018
Mar 07

There are many advantages to using SVG for the icons on a website. If you’re not familiar with SVG (aka Scalable Vector Graphics) it is an open-standard image format that uses XML to describe graphical images. SVG is great for performance, flexibility, and they look great on high-resolution displays. Though SVG can be really useful, one downside to implementing them can be the effort it takes to reuse them across a Drupal theme in various .html.twig files. SVG markup can be complex due to the number of attributes and, in some cases, there can also be a lot of markup. This issue can in part be mitigated by using inline SVG with the <use> element coupled with SVG Sprites. The combination of these two is a great solution—but unfortunately only gets part of the way to ideal reusability as the markup is still verbose for a single icon, and the number of SVG attributes needed can be hard to remember.

Wouldn’t it be nice if there was a way not to repeat all this code over again wherever we wanted to use an SVG? In this article we outline a way to reuse icons across template files in a Drupal 8 theme by creating a Twig helper function to reuse the markup.

Implementing a custom Twig custom function is fairly accessible to those comfortable with writing custom Drupal themes, but because of the way Drupal separates functionality between what can be defined in a module and what can be defined in a theme, we will need to create a module to contain our custom Twig function. To get started, create a new directory inside <drupalroot>/modules/custom with the following files and directories inside it. For the sake of this article I named the module svgy and the source is available on GitHub.

svgy
├── src
│   └── TwigExtension
│       └── Svgy.php
├── svgy.info.yml
└── svgy.services.yml

One important thing to note here is that we do not actually have a .module file in our directory. This is not required by Drupal, and the Twig extension which adds our function will be defined as a service and autoload the Svgy.php within the src/TwigExtension directory. Don’t worry if you are unfamiliar with defining custom services in Drupal 8, this article explains all you need to know to get going with what we need for our Twig function.

Now that the directory and file structure is in place, we first need to add the correct metadata to svgy.info.yml:

name: svgy
type: module
description: Add a Twig function to make using inline SVG easier.
core: 8.x

Next the necessary information needs to be added to svgy.services.yml. The contents of this file tells Drupal to autoload the Svgy.php file in the src/TwigExtension directory:

services:
  svgy.twig.extension:
    class: Drupal\svgy\TwigExtension\Svgy
    tags:
      - { name: twig.extension }

Now that Svgy.php is going to be loaded, add the code for our Twig function into it:

<?php

namespace Drupal\svgy\TwigExtension;

class Svgy extends \Twig_Extension {

 /**
  * List the custom Twig fuctions.
  *
  * @return array
  */
  public function getFunctions() {
    return [
      // Defines a new 'icon' function.
      new \Twig_SimpleFunction('icon', array($this, 'getInlineSvg')),
    ];
  }


  /**
   * Get the name of the service listed in svgy.services.yml
   *
   * @return string
   */
  public function getName() {
    return "svgy.twig.extension";
  }

  /**
   * Callback for the icon() Twig function.
   *
   * @return array
   */
  public static function getInlineSvg($name, $title) {
    return [
      '#type' => 'inline_template',
      '#template' => '<span class="icon__wrapper"><svg class="icon icon--{{ name }}" role="img" title="{{ title }}" xmlns:xlink="http://www.w3.org/1999/xlink"><use xlink:href="#{{ name }}"></use></svg></span>',
      '#context' => [
        'title' => $title,
        'name' => $name,
      ],
    ];
  }
}

More information regarding defining custom Twig extensions is available here. For the purpose of this article the most important part to explain is the inlineSvgMarkup. This function takes two arguments:

  • $name = the unique #id of the icon
  • $title = a string used as the title of the element for better accessibility

When invoked, this function returns a render array with the #type as an inline_template. We use this template type for two reasons: Twig in Drupal 8 has built-in filtering for security reasons and using inline_template allows us to get around that more easily. Since this markup is fairly small and not going to be changed often, we don’t need to create an extra .html.twig file to contain our SVG code.

Implementing the Custom Function In A Theme

Now that the custom twig extension for our icon function is created, how is this implemented in a theme? An example of the following implemented in a theme can be found on GitHub.

The first thing you have to do is add an inline SVG to your theme. The icons inside this SVG should have unique ids. In most situations it is best to generate this SVG “sprite” using something like svgstore as part of your build process. But to keep today’s example simple, I’ve created a simple SVG with two icons and placed it in a theme at <themeroot>/images/icons.svg.

After the icons.svg is in place in the theme you can include it in in the rendered page with the following in the themename.theme file:

function <theme_name>_preprocess_html(&$variables) {
  // Get the contents of the SVG sprite.
  $icons = file_get_contents(\Drupal::theme()->getActiveTheme()->getPath() . '/images/icons.svg');

  // Add a new render array to page_bottom so the icons
  // get added to the page.
  $variables['page_bottom']['icons'] = array(
    '#type' => 'inline_template',
    '#template' => '<span class="hidden">' . $icons . '</span>',
  );
}

This appends a new render array to the page_bottom variable that is printed out inside Drupal core’s html.html.twig template file. If you haven’t overridden this template in your theme, the icons will get printed out automatically. If you have overridden html.html.twig in your theme—just make sure you are still printing page_bottom inside it: ({{ page_bottom }} is all you need.

The .hidden class that is used in the wrapping <span> here is one provided by Drupal 8 core. It applies display: none; to the element. This results in hiding it both visually and from screen readers. Since the individual icons from the SVG will be referenced elsewhere in the page this is the desired outcome.

In the example theme on GitHub, this results in the following output on the page:

  <span class="hidden"><?xml version="1.0" encoding="UTF-8"?>
  <svg viewBox="0 0 130 117" version="1.1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
    <polygon id="down-arrow" fill="#50E3C2" points="64.5 117 0 0 129 0"></polygon>
    <polygon id="up-arrow" fill="#50E3C2" points="65.5 0 130 117 1 117"></polygon>
  </svg></span>

Now that the SVG “sprite” is included on the page we can start referencing the individual icons within our SVG sprite with the custom Twig function. After you have cleared Drupal’s cache registry, icons can be added with this:

{{ icon('up-arrow', 'Navigate up') }}

The first argument passed into the function here, up-arrow, is used to reference an existing id in the SVG example included above. The second argument, Navigate up the page, is used as the title of the SVG to better describe contents of the element to users navigating with screen readers.

The resulting markup of this implementation of the icon() function when rendered in a page looks like:

  <span class="icon__wrapper">
    <svg class="icon icon--up-arrow" role="img" title="Navigate up" xmlns:xlink="http://www.w3.org/1999/xlink">
      <use xlink:href="#up-arrow"></use>
    </svg>
  </span>

As you can see, the resulting markup is much larger than just the single line where we used the icon() Twig function. This makes it a lot cleaner to reuse icons throughout our .html.twig files. In addition to making templates easier to read and not having to remember all the necessary <svg> attributes, we also ensure consistency in how icons get included throughout the project.

As Drupal themers, it is great to have flexibility like this provided by Twig. If you’re interested in digging deeper into other types of Twig extensions, I encourage you to checkout the Twig Tweak module, which has examples of other helpful Twig extensions.

Have more questions about SVG? Here are some other helpful articles

Feb 28 2018
Feb 28

After my article about Drupal Development Environments, we had some discussions about the differences junior developers see when using Drupal and PHP applications locally, compared to React and other front-end tools. Someone mentioned how easy it was for them to get started by running yarn serve in a React project, and I was curious how close to that we could get for Drupal.

To make this a fair comparison, I’m not including managing MySQL databases and the like. Most React apps don’t use a database directly, and if you do need to run backend services locally, the complexity goes way up. In between writing and publishing this article, Stranger in a familiar land: Comparing the novice's first impression of Drupal to other PHP frameworks was published, which constrained itself to using the Drupal GUI installer. I think this guide shows that we can run Drupal locally in very few steps as long as we don't force ourselves to use a GUI.

I also wanted to see what the “new laptop” experience was like. I’ve been migrating my macOS setup between computers for nearly 15 years (if only Windows worked as well!), and it’s easy to forget what exactly is built in and what I’ve added over time. So, I installed a fresh copy of High Sierra in VirtualBox to ensure none of my terminal or Homebrew customizations were giving me a leg up.

Installing Composer

We need Composer to install Drush. Change to the drupal directory in the terminal (cd drupal), and run the Composer installation instructions to download composer.

When composer is done, you will have a composer.phar file in your Drupal directory.

undefined

Installing Drush and Drupal

Drush is what will let us easily run Drupal using the built-in PHP webserver. It’s also required to do the initial site installation. Pull Drush into your Drupal site by running:

$ composer require drush/drush

This will not only pull in Drush, but it will also install all of the other libraries Drush needs at the same time.

Once Drush is installed, we have to use it to install Drupal. Drupal does have a graphical installer, but Drush won’t run the PHP webserver unless Drupal is already installed. The most important parameter is the database URL, which tells Drupal what database server to use and how to connect to it. We’re going to use SQLite, which is a simple single-file database. We don’t want the database file itself to be accessible from the web server (in case it’s ever exposed to respond to external requests), so we tell Drupal to put the database in a directory above our Drupal document root.

$ vendor/bin/drush site-install --db-url=sqlite://../drupal.sqlite

undefined

When the installation is done, Drush will tell you the administrator password. If you ever forget it, you can reset it by generating a login link with drush user-login.

Running the Drupal Web Server

To start the web server, use the runserver command:

$ vendor/bin/drush runserver

By default, the server will listen on http://127.0.0.1:8888. Run vendor/bin/drush help runserver to see how to change these and other defaults.

Finally, open that URL in a browser. You should see the Drupal 8 home page and be able to log in with the administrator account shown earlier. Press CTRL-C in the terminal to shut down the web server.

undefined

The default macOS PHP configuration is pretty good, though it sets a very low limit of 2MB for uploaded files. If you need to raise it, copy /etc/php.ini.default to /etc/php.ini with:

sudo cp /etc/php.ini.default /etc/php.ini

Then, edit it with sudo nano /etc/php.ini to change settings as you see fit. You will need to restart the Drush web server after changing this file.

Bonus: Installing Git and Cloning Drupal

I like to use git even for basic testing because I can run git status at any time to see what files I’ve changed or added. I opened the Terminal and ran the git clone command copied from the Drupal project page.

$ git clone --branch 8.5.x https://git.drupal.org/project/drupal.git

The first run of this command prompts to install the developer tools:

undefined

After they install, you need to rerun the git command again (which is accessible by pressing the up arrow on your keyboard).

When this initial clone is done, you will have a Drupal 8.5.x checkout in a folder called “drupal,” and you can go back to the earlier steps to install and run Drupal.

Next Steps

Now that you have a running Drupal 8 site, it’s easy to try out contributed modules or new experimental modules without worrying about breaking a real site. It’s easy to run a “clean” instance of Drupal 8 later, by reinstalling the current site with drush site-install, or by creating a new Drupal git clone separate from the first one. And, if you are evaluating Drupal and decide to use it for a real website, you can set up a better development environment without having to learn Composer and Drush at the same time.

Feb 22 2018
Feb 22

As the digital landscape evolves, brands have to continuously give their website users richer experiences to stay competitive. They also have to consider their internal users who create and publish content to ensure their CMS actually does the job it was designed to do. This often requires migrating to new platforms that enable new functionality or redesigning a website so that it delivers the deep user experiences consumers expect.

For better or worse, website redesign combined with CMS re-platforming can be an enormous undertaking. As a result, one of the most common themes we hear during the sales process is the desire to change one major aspect of the site while leaving the others intact. This usually comes in one of two variants: 

  • “We want to migrate from [CMS A] to [CMS B] but we don’t want to change anything else on the site” (The “Forklift Migration”)
  • "We want to do a site redesign, but we don’t want to change anything on the backend” (The “Lipstick Redesign”)

With the instability of budgets and increasing pressure to show a return on investment visibly and quickly, it’s easy to understand why clients ask for these types of projects. Restricting scope means a faster turnaround and a smaller budget right? What’s not to love? Unfortunately, these projects rarely work out as planned. In this article, we will examine the problems with this approach and some solutions that may mean more upfront work but will result in a CMS implementation that pays greater dividends in the long term.

Redesigns Change Content

What can appear to be a simple design change can often have an enormous impact on back-end functionality and content structure. For instance, consider the following design change.

Two article screens with a small design tweakThe small addition of an author name to an article can have wide-ranging consequences.

In the example above, the decision was made to begin showing author credits on articles. While this appears to be a simple addition to the design, it has a ripple effect that spreads into every aspect of the system. Some examples:

  • Content created prior to this will not have an author attached, so either that content will remain un-attributed or those authors will have to be added retroactively.
  • Will this just be a text field, or will an actual author entity be created to centralize author information and increase consistency? If the latter, what will this entity look like and what other functions will it have?
  • Will this author name be a clickable link which leads to an author profile page? If so, what is the design of that page, what information will it show, and how will that information be gathered?
  • Will authors be able to edit their own information in the CMS? If so, what is the process for getting those logins created? What access rights will those authors have? If the CMS is behind a firewall, how will they access it?

This is just a small portion of the questions that such a small change might evoke, especially when adding items to a design. Another common example is when a site’s navigation is changed as part of a redesign. Once again, depending on the implementation, this can have several side effects.

  • For sections that are added to the navigation, there will need to be appropriate content added or moved so that clicking on the navigation item doesn't result in no content.
  • For sections that are removed, the content associated with those sections will have to be reviewed in order to determine what sections it should move to, or if it should be archived. 
  • For more complex changes, some combination of the above may need to happen, and the more complex the changes, the more far-reaching the content review behind them.

An existing design is the embodiment of a million decisions, and changing that design means revisiting many of those decisions and potentially starting up the entire process again. This is why a “lipstick redesign” is rarely successful. If you multiply the seemingly simple design choices described above by 100 or more, you can see that most clients quickly come to realize that a more extensive upgrade is necessary.

Migrations change design

Similarly, migration between CMSes (or between versions of the same CMS) often involves content changes that have an impact on the design. This is because a site’s design is often a careful balance between the look and feel, theming system, and CMS-provided functionality.

As an example, every CMS has its own layout and theming system, and the capabilities of these systems can be radically different from CMS to CMS. A design optimized for one system’s advantages may be extremely difficult to implement in another. Even if you could build it, it may be difficult to maintain over time because the new CMS isn't built around the same paradigms as the previous ones. In these cases, it is usually cheaper (in both time and money) to adjust the design than to rebuild the needed functionality in the new system. 

Migration spreadsheetField mappings for even a straightforward D7 to D8 migration can get complex quickly.

This can even be a problem when doing major upgrades from one version of the same CMS to the next. A theme or module you relied on for layout or theming in Drupal 7 may not be available for Drupal 8, or it may have been deprecated and added to Drupal core. This means that the affected functionality will either have to be replicated to meet the existing design, or the existing design will have to change to work with the new functionality. Either way, there’s work to do.

If, as we said above, an existing design is like the embodiment of a million decisions, then cloning that design onto a new system means you have answers to all your questions, but they may be suboptimal ones in the new context. Given that, once you open the door to tweaks, you are opening a world of new questions and starting that process again whether you want to or not.

Functionality happens

In addition to the technical issues behind the these projects, there are organizational pressures that are commonly brought to bear as well. In the end, we always find that stakeholders will receive or create pressure to add new functionality on top of the forklift. The reality is that spending real money on a project that results in no functional improvements for end users is a very tough sell in any organization. Even if that approval does come, it very rarely sticks, and once new functionality gets introduced into the middle stages of a project, the impact on schedule and budget are always far greater than if they had simply been accounted for from the start.

Additionally, most re-platforming projects require a significant amount of time to go through planning, architecture, design, and implementation. During that time your business is not at a standstill, it is moving forward and developing new needs every single day. It is unrealistic to think that your entire business can pause while a migration is performed, and this is another reason why new features have a tendency to creep into projects that were originally envisioned to just forklift from one platform to another. New business needs always pop up at the most inconvenient times in projects like this.

In almost all cases, this new functionality will impact the design and content in ways that will return the affected components back to the design/revise/approve stage. This can be extremely time-consuming, as it often takes months or years just to get the initial designs approved in the first place. Additionally, if development has already started, it may need to be redone or adjusted, wasting valuable developer hours and affecting the schedule even more.

I know what you’re thinking. You’ve got this planned out all the way down the line and everyone has signed off on it. We have heard it a hundred times, and in the end, these plans always fall apart because as everyone sees the time and money being spent, they just can’t resist getting something more out of it than they planned for. When your scope, budget, and schedule are as limited as these projects tend to be, those additions can end up being much more costly than on other projects.

How to avoid these pitfalls

Having shown that these forklift projects don’t always proceed as originally envisioned, what can you do to prepare? Thankfully, there is a lot you can do to avoid these problems. 

Get your goals and priorities straight.

While a project is in planning and development, it is too easy to be swayed into adding or removing features based on an assumed “need” when they don’t actually add a lot of value. If someone comes in with a new idea, ask yourself “How will this help the site achieve its goals? Where does this fit in my priorities?” Sometimes an addition to the site’s functionality can help increase ease of use, other times you may look to give up some design elements that don’t really add value but will increase scope. Always weigh these decisions against your goals and priorities to make sure the functionality you’re building will have the highest possible impact. 

Know your content.

Perform a full content inventory before you begin; it’s an essential step in planning your migration. Without a content inventory, you won’t have the answer to questions like “What is the longest article we need to allow for?” or “How are our articles allocated into different topics and tags?” These questions come up all the time in the migration and redesign process, and having the answers at your fingertips can really help move things along. An inventory will also help to highlight content that is miscategorized, unused, or out of date. This allows you to start planning for how to deal with these items early rather than trying to force them to fit somewhere at the last minute.

Plan for added functionality from the beginning of your project.

This is best achieved by bringing all your teams together, discussing priorities, and building a wish list. With this list in hand, you can now have a discussion with potential vendors about what is and isn't doable within budget and schedule constraints. Typically the list will be required items (“We have to migrate from Drupal 7 to Drupal 8”, “We have to refresh the site’s design to make it more modern and mobile-friendly”) as well as additional functionality you’ll want to add as part of that process (“We need to improve the editorial experience around media management”, “We need a better way to organize our content that reflects real-world use cases”). 

Work with a vendor that can provide both design and development services

Many companies will get their designs done at a standard design firm, then bring those designs to a development agency for their site build. Unfortunately, most design agencies are not well-versed in CMS implementation as a concept, and especially not for the specific quirks that each individual CMS might bring to the table. This often results in designs that are difficult or even impossible to implement. Working with an agency that does both design and development ensures that your new theme can stand up to real-world implementation, and streamlines the inevitable process of tweaking the designs as changes occur throughout the project.

Be agile and prepare for the unexpected.

While everyone wants to know exactly what they’re going to get at the end of a year-long project, the fact is that real life always intrudes at just the wrong moment. This can come in the form of unplanned functionality from higher up the ladder, technical issues that turned out to be more complex than predicted, or even personnel changes within the organization. While there are things you can do to streamline these situations and make them easier (such as choosing a full-service agency as described above) to some extent you just have to embrace the chaos and accept that changes will always be happening, and you need to deal with them and roll with the punches. Working within an agile methodology can help with this, but we find that more often than not the key is more in the state of mind of the stakeholders than it is in any structure that they might apply to the project.

Conclusion

Whether you are embarking on your own “Forklift Migration” or “Lipstick Redesign”, it’s critical to recognize the inherent connection between your front-end design and back-end CMS functionality. Navigating the interplay between the two of these concerns can be tricky, especially in organizations where product and technology are the responsibility of separate teams with discrete management. 

Feb 09 2018
Feb 09
Mike and Matt talk to Acquia's Tim Plunkett and Emilie Nouveau about the Layout Initiative, which aims to build layout building functionality into Drupal core similar to Panels / Display Suite. Note that the Layout Initiative did officially make it into Drupal 8.5!
Feb 07 2018
Feb 07

Yesterday, your CTO mandated that all new CMS builds would be on Drupal 8 and that all existing CMS platforms would be consolidated to Drupal within the next three years. Great! Time to start prioritizing sites, hiring teams, and planning development and launch schedules.

The old methods of release planning don’t work anymore

Most enterprise IT organizations are used to a “development” cycle followed by a “maintenance” cycle, at which point you start over again and rebuild with a new development cycle. This type of planning (which Dave Hansen-Lange called the “boom-bust cycle”) led to poor user experiences with Drupal 7 sites as they became stale with time and is impossible to use for a Drupal 8 site.

Drupal 8 has a different release cycle compared to previous Drupal releases. For example, Drupal 7 was released in January 2011, and ever since has only had bug fixes (with the occasional developer-only feature added). Drupal 5 became unsupported, and Drupal 6 entered it’s last phase of support (until Drupal 8 was released). Product owners of sites big and small found this release process stifling. The web is a very different platform today compared to January 2011—although Drupal 7 is pretty much the same. The slow release cycle also reduced willingness to contribute directly to Drupal. After all, why spend time writing a new feature you can’t use until you migrate to Drupal 8?

Part of why Drupal 8’s development took the time it did was to allow for a faster release cycle for features. Now, new features are added in point releases (while code compatibility with prior Drupal 8 releases is broadly maintained).

Furthermore, security updates are only reliable for the latest minor release. After a minor release, such as Drupal 8.4, security issues in the prior minor release (such as Drupal 8.3) will only be patched for a few weeks, if at all.  In other words, to keep your sites secure, you have to be on the latest point release.

Upgrading your site every six months is a cost of using Drupal 8.

However, it’s not just Drupal that requires more frequent upgrades. Docker, NodeJS, and PHP itself all have more aggressive support schedules than similar software may have had in the past. Here lies the core of release planning: synchronizing all of these release schedules with your business.

1. Build a schedule of key dates for your business

In the publishing industry, tentpole events such as tournaments, live events, or holiday specials drive production deadlines. In addition to tentpole events, identify conferences, retreats, and other dates where your team may not be available. If they overlap with an EOL of software in your stack, you can try to schedule an upgrade ahead of time.

There are two reasons to identify these early and socialize them with the rest of the business. First, these are times when you don’t want to schedule maintenance or deployments if you can help it. Second, these key dates are times to be prepared and aware of security release windows. For example, Drupal core security releases are usually on the third Wednesday of the month. If this overlaps with a high-traffic event, having a plan to deploy and test a security patch before the patch is issued will ensure your site is kept secure, and that technical teams aren’t stressed out during the process.

2. Build a schedule of your stack and its support windows

Too often, we see organizations mandate a specific tool (like Drupal or Docker) without budgeting time and money for recurring, required upgrades. Maintenance often means “respond to bug reports from our team and handle outages,” instead of “incrementally upgrade software to keep it in supported releases.”

Before development begins, teams should gather a list of all key software used in their expected production stack. Then, go to each software release page, and find out the start and end dates of support for the current and next few releases. Use this to create a Gantt chart of support windows.

Here’s an example of one I created for a client that was deploying Drupal 8, PHP, Nginx, nodejs, Docker, and Ubuntu for their sites. In their case, they weren’t using Ubuntu packages for any of the other software, so they had to track each component directly.

Gantt chart showing project software EOLs

Having such a chart prevents surprises. For example, if a site is using nodejs 6.x, we can see that it’s going to EOL just after Drupal 8.5 is released. It would make sense for a developer to work on upgrading to nodejs 8.x over the December holiday break so that the team isn’t juggling upgrading two core components at the same time.

This type of chart also gives teams the power to know if a given technology stack is maintainable for their organization. If it’s a small team maintaining a large Drupal site, they may not have the bandwidth or expertise to keep up with updates elsewhere in the stack. That may mean they have to stick with LTS releases, or avoid software without an LTS available.

For Drupal 8, that may mean that it is not a fit for your organization. If your budget only allows an initial build and then very minimal maintenance, Drupal 7 or another CMS may be a better fit (at least until Drupal 8 has an LTS). Or, for event and marketing sites, consider using a static site generator like Jekyll, so that when development ends the site can be “serialized” into static HTML and served with a basic stack using LTS software only.

3. Have a top-down process for handling security and feature updates

Many times we see software update requests coming from the development team, and then bubbling up to product owners and managers, meaning management will often prioritize new features with immediate business value before critical updates. Of course, this ignores the fact that a functioning and secure website is the foundation for addressing the needs of the business. And, it places developers in an awkward position where they are responsible for cleaning up any security breaches that occur.

Instead, software updates should be visible to and scheduled by the project managers, ensuring that time is available in the right sprint to test and deploy updates, preventing the development team from being overcommitted or opening your site up to a security breach. For example, let's say a team is using two-week sprints, with a sprint set to start on February 19, 2018. A quick look at the release cycle overview for Drupal 8 shows that 8.5.0-rc1 will be tagged on February 21st. Creating a ticket for that sprint ahead of time will ensure the upgrade won’t be forgotten.

It can be more difficult to handle software—like Drupal contributed modules or npm packages—that doesn't have a defined release schedule. In those cases, having a recurring “update everything” ticket can help. Assuming one team manages an entire deployment, a successful schedule might look like:

  • Sprint 1: Update all Drupal contributed modules
  • Sprint 2: Update all npm packages
  • Sprint 3: Update all system-level packages
  • Sprint 4: Update all Drupal contributed modules
  • … etc.

Remember, just because you schedule software updates doesn’t mean that you have to deploy them to production if QA goes poorly. But, if you do find issues, at least you can try to address them before, and not during a security patch.

4. Promote the new features you deploy to your colleagues and users

After doing all of this work, you should tell your end users and stakeholders about the gains you’ve made in the process. Show editorial users how Drupal 8.4’s Inline Form Errors gives them a better display of form errors on content they write. Tell your stakeholders how your API servers respond 15% faster to mobile clients. High-five your developers when they can remove a patch they’d been manually applying to the site.

For larger teams, newsletters and blogs are a great way to communicate these improvements. They are especially useful for core development teams, who write and maintain software shared throughout an enterprise.

With a solid grasp of how upstream release schedules affect your product, you will be able to reduce the need for unexpected delays and budget requests. If you’re interested in exploring this further with your team, we’d be glad to help.

Jan 31 2018
Jan 31

Whether you are familiar with Composer or not, using it to manage dependencies on a Drupal project entails its own unique set of best practices. In this article, we will start by getting a Composer-managed Drupal project set up, and highlight some common questions and issues you may have along the way.

Before we dive in, though, you may be asking yourself, “Why Composer? Can’t I just download Drupal and the modules I need without requiring another tool?” Yes you can, but you will quickly realize it’s not a simple task:

  1. Contributed modules or themes often depend on third-party libraries installed via Composer. Without using Composer for the project, you’ll need to manage these individually when downloading, which can be quite a chore.
  2. Some packages and modules only work with certain versions of PHP or Drupal. While Drupal core does help you identify these issues for modules and themes, it’s still a manual process that you’ll need to work through when choosing which versions to download.
  3. Some packages and modules conflict with other packages. You’ll need to read the composer.json files to find out which.
  4. When you upgrade a package or a version of PHP, you’ll need to do all the above over again.
  5. If you’re thinking you’ll use drush dl and friends, they’ve been removed in favor of Composer.

Dependency management is complicated, and it’s not going to get any easier. As Ryan Szrama put it , “if you’re not using a dependency manager [like Composer], then you’re the dependency manager, and you are unreliable.”

Where do I start?

To begin, it’s good to familiarize yourself with the fantastic documentation on Drupal.org . If you haven’t already, install Composer in your development environment. When getting started, it’s extremely important that your development environment is using the same version of PHP that the target production environment will use. Otherwise Composer may download packages that won’t work in production. (With larger development teams, using something like Vagrant or Docker for local development can help ensure compatibility between local development instances and the production stack, but that’s a separate topic).

To start with, you need to get Drupal core installed. If you are familiar with using starter-kits like Drupal Boilerplate in Drupal 7, there is a Composer template for Drupal projects called, well, drupal-project. It’s the recommended starting place for a Composer-based Drupal 8 installation. Before you create it—again—be sure you are using the production version of PHP. Once you’ve confirmed that, run:

$ composer create-project drupal-composer/drupal-project:8.x-dev \
example --stability dev --no-interaction

This will copy the drupal-project to the example directory, and download Drupal core and some handy packages. This is a good point to cd into the example directory, run git init , and to then create your initial commit for your project.

This also might be a good time to pop open the composer.json file that was created in there (which should look a lot like this). This file serves as your “recipe” for the project.

How do I download modules and themes?

If you are familiar with using drush dl in Drupal 7 to download contributed modules and themes, it can be a bit of a shift switching to a Composer workflow. There is a different syntax for selecting the version you’d like, and in some cases it’s best to not commit the actual downloaded files to your repository (more on that to follow).

For most instances, you can use composer require [vendor]/[packagename] to grab the package you’d like. Most Drupal modules and themes are hosted on a custom Composer repository, which drupal-project configures for us out of the box. That means we can use drupal as the vendor to download Drupal modules and themes that are maintained on drupal.org. If you aren’t using drupal-project, you may need to add the Drupal.org Composer repository yourself.

For example, if you were wanting to download the Devel module, you would run the following command:

$ composer require drupal/devel

For most cases, that would download the most recent stable version of the Devel module that is compatible with your version of Drupal and PHP. If you used drupal-project above, it would download the module to web/modules/contrib. If you want to change the download destination, you can alter the paths in your composer.json. Look for the installer-paths under extra and adjust to your liking.

Development Dependencies

The Devel module is a good example to use, as it brings up a few other things to note. For example, since the Devel module shouldn’t be used in a production environment, you may only want it to get downloaded on development environments. With Composer, you can achieve this by passing the --dev flag to the composer require command.

$ composer require --dev drupal/devel

This will ensure that the Devel module is available for your developers when they run composer install , but that it won’t ship with your production release (which should get built using composer install --no-dev ). This is just an example, and your project needs may differ, but the use of --dev to specify dependencies only needed during development is a recommended best practice.

As a side note, if you are committing your exported Drupal 8 configuration to code, you may want to use the Configuration Split module to ensure that the Devel module’s enabled status doesn’t get pushed to production by mistake in a release.

Nested Dependencies

Another thing to note is that the Devel module ships with its own composer.json file. Since we used Composer to download Devel, Composer will also read in Devel’s composer.json and download any dependencies that it may have. In this case (at the time of this writing), Devel doesn’t have any required dependencies, but if you were to use Composer to download, say Address module, it would pull in its additional dependencies as well, as Address module uses Composer to specify some additional dependencies .

Downloading specific versions

While it’s most often fine to omit declaring a specific version like we did above, there still may come a time when you need to narrow the range of possibilities a bit to get a package version that is compatible with your application.

Judging by the length of the Composer documentation on Versions and constraints, this is no small topic, and you should definitely read through that page to familiarize yourself with it. For most cases, you’ll want to use the caret constraint (e.g., composer require drupal/foo:^1.0 means the latest release in the 8.1 branch), but here’s some more details about versions and constraints:

  1. Read up on Semantic Versioning if you aren’t familiar with it. Basically, versions are in the x.y.z format, sometimes described as MAJOR.MINOR.PATCH , or BREAKING.FEATURE.FIX . Fixes increment the last number, new features increment the middle number, and refactors or rewrites that would break existing implementations increment the first number.
  2. Use a colon after the vendor/package name to specify a version or constraint, e.g. composer require drupal/foo:1.2.3 (which would require that exact version (1.2.3) of the foo package). If you’re used to drush syntax (8.x-1.2.3), the Composer syntax does not include the Drupal version (i.e., there's no 8.x).
  3. Don’t specify an exact version unless you have to. It’s better to use a constraint.
  4. The caret constraint (^): this will allow any new versions except BREAKING ones—in other words, the first number in the version cannot increase, but the others can. drupal/foo:^1.0 would allow anything greater than or equal to 1.0 but less than 2.0.x . If you need to specify a version, this is the recommended method.
  5. The tilde constraint (~): this is a bit more restrictive than the caret constraint. It means composer can download a higher version of the last digit specified only. For example, drupal/foo:~1.2 will allow anything greater than or equal to version 1.2 (i.e., 1.2.0, 1.3.0, 1.4.0,…,1.999.999), but it won’t allow that first 1 to increment to a 2.x release. Likewise, drupal/foo:~1.2.3 will allow anything from 1.2.3 to 1.2.999, but not 1.3.0.
  6. The other constraints are a little more self-explanatory. You can specify a version range with operators, a specific stability level (e.g., -stable or -dev ), or even specify wildcards with *.

What do I commit?

Now that we’ve got a project started using Composer, the next question you may have is, “What files should I commit to my repository?” The drupal-project will actually ship a default .gitignore file that will handle most of this for you. That said, there are a few things that may be of interest to you.

You may have noticed that after running some of the above commands, Composer created a composer.lock file in your repository. The composer.lock file is a very important file that you want to commit to your repository anytime it changes. When you require a new package, you may have omitted a version or a version constraint, but Composer downloaded a specific version. That exact version is recorded in the composer.lock file, and by committing it to your repository, you then ensure that anytime composer install is run on any other environment, the same exact code gets downloaded. That is very important from a project stability standpoint, as otherwise, Composer will download whatever new stable version is around based on what’s in composer.json.

Now that we understand more about what composer.lock is for, here’s a list of what to commit and what not to commit:

  1. Commit composer.json and composer.lock anytime they change.
  2. There’s no need to commit anything in ./vendor , Drupal core (./web/core), or contributed modules and themes (./web/modules/contrib or ./web/themes/contrib). In fact, it’s recommended not to, as that ensures the same code is used on all environments, and reduces the size of diffs. If you really want to (because of how your code gets deployed to production, for example), it is possible, you’ll just need to change the .gitignore, and always make sure the committed versions match the versions in composer.lock.
  3. Commit any other custom code and configuration as usual.

How do I keep things up to date?

Updating modules, themes, and libraries downloaded with Composer is similar to installing them. While you can run composer update to update all installed packages, it’s best to save that for the beginning of a sprint, and update individual projects as needed after that. This ensures you only update exactly what you need without risking introducing bugs from upstream dependencies. If you aren’t working in sprints, set up a regular time (weekly or monthly) to run composer update on the whole project.

To update the Devel module we installed earlier, you could run:

$ composer update --with-dependencies drupal/devel

You might be wondering what that --with-dependencies option is. If you run the update command without it, composer will update just the module, and not any packages required by that module. In most circumstances, it’s best to update the module’s dependencies as well, so get used to using --with-dependencies.

You can also update multiple packages separated by spaces:

$ composer update --with-dependencies drupal/foo drupal/bar

For a more aggressive approach, you can also use wildcards. For example, to update all drupal packages, you could run:

$ composer update --with-dependencies drupal/*

Once complete, be sure to commit the changes to composer.lock.

What about patches?

Inevitably, you’ll need to apply some patches to a Drupal module, theme, or possibly Drupal core. Since you aren’t committing the actual patched modules or themes with a Composer workflow, you’ll need some way to apply those patches consistently across all environments when they are downloaded. For this we recommend composer-patches. It uses a self-documenting syntax in composer.json to declare your patches. If you’re using drupal-project mentioned above, composer-patches will already be installed and ready to use. If not, adding it to your project is as simple as:

$ composer require cweagans/composer-patches

While it’s installing, take this time to go read the README.md on GitHub.

Once installed, you can patch a package by editing your composer.json . Continuing with our example of Devel module, let’s say you wanted to patch Devel with a patch from this issue. First step?—copy the path to the patch file you want to use. At the time of this writing, the most recent patch is https://www.drupal.org/files/issues/2860796-2.patch .

Once you have the path to the patch, add the patch to the extras section of your composer.json like so:

"extra": {
  "patches": {
    "drupal/devel": {
      "2860796: Create a branch of devel compatible with Media in core": "https://www.drupal.org/files/issues/2860796-2.patch"
    }
  }
} 

You can see that in the above example, we used the node id (2860796) and title of the Drupal.org issue as the key, but you may decide to also include the full URL to the issue and comment. It’s unfortunately not the easiest syntax to read, but it works. Consistency is key, so settle on a preferred format and stick to it.

Once the above has been added to composer.json, simply run composer update drupal/devel for the patch to get applied.

$ composer update drupal/devel
Gathering patches for root package.
Removing package drupal/devel so that it can be re-installed and re-patched.
Deleting web/modules/contrib/devel - deleted
> DrupalProject\composer\ScriptHandler::checkComposerVersion
Loading composer repositories with package information
Updating dependencies (including require-dev)
Package operations: 1 install, 0 updates, 0 removals
Gathering patches for root package.
Gathering patches for dependencies. This might take a minute.
- Installing drupal/devel (1.2.0): Loading from cache
- Applying patches for drupal/devel
https://www.drupal.org/files/issues/2860796-2.patch (2860796: Create a branch of devel compatible with Media in core)

And finally, commit the changes to both composer.json and composer.lock .

If any of your dependencies declare patches (for example, let’s say a module requires a patch to Drupal core to work properly), you’ll need to explicitly allow that in your composer.json. Here’s a quick one-liner for that:

$ composer config extra.enable-patching true

How do I remove a module or theme?

Removing a module or theme using Composer is fairly straightforward. We can use composer remove for this:

$ composer remove drupal/devel

If you added drupal/devel with the --dev flag, you might get a prompt to confirm the removal from require-dev . If you want to avoid that prompt, you can use the --dev flag in your remove command as well:

$ composer remove --dev drupal/devel

Once it’s done, commit your composer.json and composer.lock file.

Anything else to watch out for?

Removing patches

If you need to remove a module or theme that has patches, the patches don’t automatically get removed from the composer.json file. You can either remove them by hand in your editor, or run:

$ composer config --unset extra.patches.drupal/devel

Lock hash warnings

On a related note, any time you manually edit composer.json in this way without a subsequent composer update, the hash in composer.lock will be out of date, throwing warnings. To fix that without changing the code you have installed, run composer update --lock. That will just update the hash in the lock file without updating anything else. Once complete, commit the composer.json and composer.lock.

PHP Versions…again

As mentioned early on, it’s very important with a Composer-based workflow that the various environments use the same major and minor release of PHP. If you aren’t using Vagrant or Docker for local development and are finding it difficult to standardize, you can enforce a specific PHP version for Composer to adhere to in composer.json.

$ composer config platform.php 7.1

Once added, don’t forget to update your lock file and commit the resulting composer.json and composer.lock .

$ composer update --lock

Speeding things up

There are some Composer operations that can be slow. To speed things up, you can install Prestissimo, which will allow Composer to run operations in parallel. It’s best to do this globally:

$ composer global require hirak/prestissimo

It’s not perfect

You will inevitably run into some frustrating issues with a Composer workflow that will cause you to question its wisdom. While there may be scenarios where it might make sense to abandon it for other options, you will bear the burden of managing the complexities of your project’s dependencies yourself. If you stick with Composer, the good news for you is that you are not alone, and these issues promise to improve over time because of that. Think of it as an investment in the long-term stability of your project.

Collage of JoAnn Falletta by Mark Dellas.

Jan 27 2018
Jan 27

In this episode, Matthew Tift talks with Tom Grandy, who oversees websites for 23 school districts. Tom describes himself as a journalist, a teacher, and a non-coder who helps out with documentation and marketing for Backdrop. He describes his experiences using proprietary software, finding Drupal, his involvement with Backdrop, and the challenges of using free software in K-12 education. Tom shares why people working in schools make decisions about technology most often based on cost, but that he believes we should also considers software licenses, communities, and other more philosophical factors.

Pages

About Drupal Sun

Drupal Sun is an Evolving Web project. It allows you to:

  • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
  • Facet based on tags, author, or feed
  • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
  • View the entire article text inline, or in the context of the site where it was created

See the blog post at Evolving Web

Evolving Web