Apr 11 2019
Apr 11

The benefits of backwards compatibility (BC) are clear: no users are left behind. Which leads to higher adoption rates because you’re often getting new features and you always have the latest security fixes.

Of course, that’s easy when you have a small API surface (as Nate Haug once said: “the WordPress API has like 11 functions!” — which is surprisingly close to the truth). But Drupal has an enormous API surface. In fact, it seems there’s APIs hiding in every crevice!

In my job at Acquia, I’ve been working almost exclusively on Drupal 8 core. In 2012–2013 I worked on authoring experience (in-place editing, CKEditor, and more). In 2014–2015, I worked on performance, cacheability, rendering and generally the stabilizing of Drupal 8. Drupal 8.0.0 shipped on November 19, 2015. And since then, I’ve spent most of my time on making Drupal 8 be truly API-first: improving the RESTful Web Services support that Drupal 8 ships with, and in the process also strengthening the JSON API & GraphQL contributed modules.

I’ve learned a lot about the impact of past decisions (by myself and others) on backwards compatibility. The benefit of backwards compatibility (BC). But the burden of ensuring BC can increase exponentially due to certain architectural decisions. I’ve been experiencing that first-hand, since I’m tasked with making Drupal 8’s REST support rock-solid, where I am seeing time and time again that “fixing bugs + improving DX” requires BC breaks. Tough decisions.

In Drupal 8, we have experience with some extremes:

  1. the BigPipe & Dynamic Page Cache modules have no API, but build on top of other APIs: they provide functionality only, not APIs
  2. the REST module has an API, and its functionality can be modified not just via that API, but also via other APIs

The first cannot break BC. The second requires scrutiny for every line of code modified to ensure we don’t break BC. For the second, the burden can easily outweigh the benefit, because how many sites actually are using this obscure edge case of the API?

We’ll look at:

  • How can we make our modules more evolvable in the future? (Contrib & core, D8 & D9.)
  • Ideas to improve this, and root cause hypotheses (for example, the fact that we have API cascades and not orthogonal APIs)

We should be thinking more actively about how feature X, configuration Y or API Z might get in the way of BC. I analyzed the architectural patterns in Drupal 8, and have some thoughts about how to do better. I don’t have all the answers. But what matters most is not answers, but a critical mindset going forward that is consciously considering BC implications for every patch that goes into Drupal 8! This session is only a starting point; we should continue discussing in the hallways, during dinner and of course: in the issue queues!

Preview:

Slides: Slides with transcriptVideo: YouTubeConference: DrupalCon SeattleLocation: Seattle, WA, United StatesDate: Apr 11 2019 - 09:00Duration: 30 minutesExtra information: 

See https://events.drupal.org/seattle2019/sessions/backwards-compatibility-burden-benefit.

Mar 21 2019
Mar 21

The JSON:API module was added to Drupal 8.7 as a stable module!

See Dries’ overview of why this is an important milestone for Drupal, a look behind the scenes and a look toward the future. Read that first!

Upgrading?

As Mateu said, this is the first time a new module is added to Drupal core as “stable” (non-experimental) from day one. This was the plan since July 2018 — I’m glad we delivered on that promise.

This means users of the JSON:API 8.x-2.x contrib module currently on Drupal 8.5 or 8.6 can update to Drupal 8.7 on its release day and simply delete their current contributed module, and have no disruption in their current use of JSON:API, nor in security coverage!

What’s happened lately?

The last JSON:API update was exactly two months ago, because … ever since then Gabe, Mateu and I are have been working very hard to get JSON:API through the core review process. This resulted in a few notable improvements:

  1. a read-only mode that is turned on by default for new installs — this strikes a nice balance between DX (still having data available via APIs by default/zero config: reading is probably the 80% use case, at least today) and minimizing risk (not allowing writes by default)
  2. auto-revisioning when PATCHing for eligible entity types
  3. formally documented & tested revisions and translations support
  4. formally documented security considerations

Get these improvements today by updating to version 2.4 of the JSON:API module — it’s identical to what was added to Drupal 8.7!

Contributors

An incredible total of 103 people contributed in JSON:API’s issue queue to help make this happen, and 50 of those even have commits to their name:

Wim Leers, ndobromirov, e0ipso, nuez, gabesullice, xjm, effulgentsia, seanB, jhodgdon, webchick, Dries, andrewmacpherson, jibran, larowlan, Gábor Hojtsy, benjifisher, phenaproxima, ckrina, dww, amateescu, voleger, plach, justageek, catch, samuel.mortenson, berdir, zhangyb, [email protected], malik.kotob, pfrilling, Grimreaper, andriansyahnc, blainelang, btully, ebeyrent, garphy, Niklan, joelstein, joshua.boltz, govind.maloo, tstoeckler, hchonov, dawehner, kristiaanvandeneynde, dagmar, yobottehg, [email protected], keesee, caseylau, peterdijk, mortona2k, jludwig, pixelwhip, abhisekmazumdar, izus, Mile23, mglaman, steven.wichers, omkar06, haihoi2, axle_foley00, hampercm, clemens.tolboom, gargsuchi, justafish, sonnykt, alexpott, jlscott, DavidSpiessens, BR0kEN, danielnv18, drpal, martin107, balsama, nileshlohar, gerzenstl, mgalalm, tedbow, das-peter, pwolanin, skyredwang, Dave Reid, mstef, bwinett, grndlvl, Spleshka, salmonek, tom_ek, huyby, mistermoper, jazzdrive3, harrrrrrr, Ivan Berezhnov, idebr, mwebaze, dpolant, dravenk, alan_blake, jonathan1055, GeduR, kostajh, pcambra, meba, dsdeiz, jian he, matthew.perry.

Thanks to all of you!

Future JSON:API blogging

I blogged about once a month since October 2018 about JSON:API, to get more people to switch to version 2.x of the JSON:API module, to ensure it was maximally mature and bug free prior to going into Drupal core. New capabilities were also being added at a pretty high pace because we’d been preparing the code base for that months prior. We went from ~1700 installs in January to ~2700 today!

Now that it is in Drupal core, there will be less need for frequent updates, and I think the API-First Drupal: what’s new in 8.next? blog posts that I have been doing probably make more sense. I will do one of those when Drupal 8.7.0 is released in May, because not only will it ship with JSON:API land, there are also other improvements!

Special thanks to Mateu Aguiló Bosch (e0ipso) for their feedback!

Jan 21 2019
Jan 21

As promised 3 months ago, Gabe, Mateu and I together with twelve other contributors shipped support for revisions and file uploads today!

What happened since last month? In a nutshell:

JSON:API 2.1

JSON:API 2.1 follows two weeks after 2.0.

Work-arounds for two very common use cases are no longer necessary: decoupled UIs that are capable of previews and image uploads.

  • File uploads work similarly to Drupal core’s file uploads in the REST module, with the exception that a simpler developer experience is available when uploading files to an entity that already exists.
  • Revision support is for now limited to retrieving the working copy of an entity using ?resourceVersion=rel:working-copy. This enables the use case we hear about the most: previewing draft Nodes. Browsing all revisions is not yet possible due to missing infrastructure in Drupal core. With this, JSON:API leaps ahead of core’s REST API.

Please share your experience with using the JSON:API module!

Jan 07 2019
Jan 07

Mateu, Gabe and I just released JSON:API 2.0!

Read more about it on Mateu’s blog.

I’m proud of what we’ve achieved. I’m excited to see more projects use it. And I’m confident that we’ll be able to add lots of features in the coming years, without breaking backwards compatibility. I was blown away just now while generating release notes: apparently 63 people contributed. I never realized it was that many. Thanks to all of you :)

I had a bottle of Catalan Ratafia (which has a fascinating history) waiting to celebrate the occasion. Why Ratafia? Mateu is the founder of this module and lives in Mallorca, in Catalunya. Txin txin!

If you want to read more about how it reached this point, see the July, October, November and December blog posts I did about our progress.

Dec 31 2018
Dec 31

Last week was my twelfth Drupalversary!

The first half dozen years as a volunteer contributor/student, the second half as a full-time contributor/Acquia employee. Which makes this a special Drupalversary and worth looking back on :)

The d.o highlights of the first six years were my Hierarchical Select and CDN modules. I started those in my first year or so of using Drupal (which coincides with my first year at university). They led to a summer job for Mollom, working with/for Dries remotely — vastly better than counting sandwiches or waiting tables!

It also resulted in me freelancing during the school holidays: the Hierarchical Select module gained many features thanks to agencies not just requesting but also sponsoring them. I couldn’t believe that companies thousands of kilometers away would trust a 21-year old to write code for them!

Then I did my bachelor thesis and master thesis on Drupal + WPO (Web Performance Optimization) + data mining. To my own amazement, my bachelor thesis (while now irrelevant) led to freelancing for the White House and an internship with Facebook.

Biggest lesson learned: opportunities are hiding in unexpected places! (But opportunities are more within reach to those who are privileged. I had the privilege to do university studies, to spend my free time contributing to an open source project, and to propose thesis subjects.)

The second half was made possible by all of the above and sheer luck.

When I was first looking for a job in early 2012, Acquia had a remote hiring freeze. It got lifted a few months later. Because I’d worked remotely with Dries before (at Mollom), I was given the opportunity to work fully remotely from day one. (This would turn out to be very valuable: since then I’ve moved three times!) Angie and Moshe thought I was a capable candidate, I think largely based on the Hierarchical Select module.
Imagine that the remote hiring freeze had not gotten lifted or I’d written a different module? I was lucky in past choices and timing.
So I joined Acquia and started working on Drupal core full-time! I was originally hired to work on the authoring experience, specifically in-place editing.
The team of four I joined in 2012 has quadrupled since then and has always been an amazing group of people — a reflection of the people in the Drupal community at large!

Getting Drupal 8 shipped was hard on everyone in the community, but definitely also on our team. We all did whatever was most important; I probably contributed to more than a dozen subsystems along the way. The Drupal 8 achievement I’m most proud of is probably the intersection of cacheability and the render pipeline: Dynamic Page Cache & BigPipe, both of which have accelerated many billions responses by now. After Drupal 8 shipped, my primary focus has been the API-First Initiative. It’s satisfying to see Drupal 8 do well.

Biggest lessons learned:

  1. code criticism is not personal criticism — not feeling the need to defend every piece of code you’ve written is not only liberating, it also makes you immensely more productive!
  2. always think about future maintainability — having to provide support and backwards compatibility made me truly understand the consequences of mistakes I’ve made.

To many more years with the Drupal community!

Dec 13 2018
Dec 13

Gabe, Mateu and I just released the third RC of JSON:API 2, so time for an update! The last update is from three weeks ago.

What happened since then? In a nutshell:

RC3

Curious about RC3? RC2RC3 has five key changes:

  1. ndobromirov is all over the issue queue to fix performance issues: he fixed a critical performance regression in 2.x vs 1.x that is only noticeable when requesting responses with hundreds of resources (entities); he also fixed another performance problem that manifests itself only in those circumstances, but also exists in 1.x.
  2. One major bug was reported by dagmar: the ?filter syntax that we made less confusing in RC2 was a big step forward, but we had missed one particular edge case!
  3. A pretty obscure broken edge case was discovered, but probably fairly common for those creating custom entity types: optional entity reference base fields that are empty made the JSON:API module stumble. Turns out optional entity reference fields get different default values depending on whether they’re base fields or configured fields! Fortunately, three people gave valuable information that led to finding this root cause and the solution! Thanks, olexyy, keesee & caseylau!
  4. A minor bug that only occurs when installing JSON:API Extras and configuring it in a certain way.
  5. Version 1.1 RC1 of the JSON:API spec was published; it includes two clarifications to the existing spec. We already were doing one of them correctly (test coverage added to guarantee it), and the other one we are now complying with too. Everything else in version 1.1 of the spec is additive, this is the only thing that could be disruptive, so we chose to do it ASAP.

So … now is the time to update to 2.0-RC3. We’d love the next release of JSON:API to be the final 2.0 release!

P.S.: if you want fixes to land quickly, follow dagmar’s example:

If you don't know how to fix a bug of a #drupal module, providing a failing test usually is really helpful to guide project maintainers. Thanks! @GabeSullice and @wimleers for fixing my bug report https://t.co/bEkkjSrE8U

— Mariano D'Agostino (@cuencodigital) December 11, 2018
Nov 21 2018
Nov 21

Gabe, Mateu and I just released the second RC of JSON:API 2, so time for an update! The last update is from a month ago.

What happened since then? In a nutshell:

  • Usage has gone up a lot! 2.0-RC1: 0 → 250, 2.x: ~200 → ~330
  • 2.0-RC2 released :)
  • RC1 had about half a dozen issues reported. Most are fixed in RC2.
  • The file uploads feature: 80% → 100%, will ship in 2.1!
  • The revisions feature : 80% → 90%, will ship in 2.2!
  • New core patch to bring JSON:API to Drupal core: #2843147-78

RC2

Curious about RC2? RC1RC2 has four changes:

  1. Renamed from JSON API to JSON:API, matching recent spec changes.
  2. One critical bug was reported by brand new Drupalist Peter van Dijk (welcome, and thanks for noticing something we had gotten used to!): the ?filter syntax was very confusing and effectively required the knowledge (and use) of Drupal internals. Now you no longer need to do /jsonapi/node/article?filter[uid.uuid]=c42…e37 to get all articles by a given author, but just /jsonapi/node/article?filter[uid.id]=c42…e37. This matches the JSON:API document structure: the abstraction is no longer leaky! The updated ?filter syntax is still 95% the same.
  3. JSON:API responses were no longer being cached by Page Cache. This was of course a major performance regression. Fortunately long-time Drupalist yobottehg discovered this!
  4. Massive performance increase for requests using sparse fieldsets such as ?fields[node--article]=title. Thanks to long-time Drupalist jibran doing quite a bit of profiling on JSON:API, something which Mateu, Gabe and I haven’t gotten around to yet. Moving 2 lines of code made his particular use case 90% faster! We’re sure there’s more low-hanging fruit, but this one was so tasty and so low-risk that we decided to commit it despite being in RC.

So … now is the time to update to 2.0-RC2!

The JSON:API team: 150/150/150

In the previous update, I included a retrospective, including a section about how it’s been to go from one maintainer to three. A month ago this happened:

11:08:15 <e0ipso> https://usercontent.irccloud-cdn.com/file/REdyLpD6/2018-10-24%2011-07-42.png
11:08:27 <e0ipso> Let's get that to 150, 150 and 150
11:08:31 <WimLeers> :D
11:08:46 <WimLeers> that'd be pretty cool indeed
11:09:05 <e0ipso> COMEON gabesullice you're slacking!
11:09:08 <e0ipso> hehe

That image was: The screenshot Mateu shared a month ago. Showing Wim at 149 commits, Gabe at 147, Mateu at 150.

Well, as of yesterday evening, it looks like this: JSON:API committer statistics showing that the three main maintainers each have 150 commits!

On top of that, mere hours after previous update went live, Gabe became an official API-First Initiative coordinator, together with Mateu and I:

https://t.co/n9mB2BtKgn

This is up there at the top as one of my proudest Drupal moments :)

— Gabriel Sullice (@GabeSullice) October 25, 2018

Congrats Gabe, and thanks! :)

Nov 06 2018
Nov 06

I’ve been running a lot lately, and so have been listening to lots of podcasts! Which is how I stumbled upon this great episode of the Lullabot podcast recently — embarrassingly one from over a year ago: “Talking Performance with Pantheon’s David Strauss and Josh Koenig”, with David and Josh from Pantheon and Nate Lampton from Lullabot.

(Also, I’ve been meaning to blog more, including simple responses to other blog posts!)

Interesting remarks about BigPipe

Around 49:00, they start talking about BigPipe. David made these observations around 50:22:

I have some mixed views on exactly whether that’s the perfect approach going forward, in the sense that it relies on PHP pumping cached data through its own system which basically requires handling a whole bunch of strings to send them out, as well as that it seems to be optimized around this sort of HTTP 1.1 behavior. Which, to compare against HTTP 2, there’s not really any cost to additional cost to additional connections in HTTP 2. So I think it still remains to be seen how much benefit it provides in the real world with the ongoing evolution of some of these technologies.

David is right; BigPipe is written for a HTTP 1.1 world, because BigPipe is intended to benefit as many end users as possible.

And around 52:00, Josh then made these observations:

It’s really great that BigPipe is in Drupal core because it’s the kind of thing that if you’re building your application from scratch that you might have to do a six month refactor to even make possible. And the cache layer that supports it, can support lots other interesting things that we’ll be able to develop in the future on top of Drupal 8. […] I would also say that I think the number of cases where BigPipe or ESI are actually called for is very very small. I always whenever we talk about these really hot awesome bleeding-edge cache technologies, I kinda want to go back to what Nate said: start with your Page Cache, figure out when and how to use that, and figure out how to do all the fundamentals of performance before even entertaining doing any of these cutting-edge technologies, because they’re much trickier to implement, much more complex and people sometimes go after those things first and get in over their head, and miss out on a lot of the really big wins that are easier to get and will honestly matter a lot more to end users. “Stop thinking about ESI, turn on your block cache.”

Josh is right too, BigPipe is not a silver bullet for all performance problems; definitely ensure your images and JS are optimized first. But equating BigPipe with ESI is a bit much; ESI is indeed extremely tricky to set up. And … Drupal 8 has always cached blocks by default. :)

Finally, around 53:30 David cites another reason to stress why more sites are not handling authenticated traffic:

[…] things like commenting often move to tools like Disqus and whether you want to use Facebook or the Google+ ones or any one of those kind of options; none of those require dynamic interaction with Drupal.

Also true, but we’re now seeing the inverse movement, with the increased skepticism of trusting social media giants, not to mention the privacy (GDPR) implications. Which means sites that have great performance for dynamic/personalized/uncacheable responses are becoming more important again.

BigPipe’s goal

David and Josh were being constructively critical; I would expect nothing less! :)

But in their description and subsequent questioning of BigPipe, I think they forget its two crucial strengths:

BigPipe works on any server, and is therefore available to everybody, and it works for many things out of the box, including f.e. every uncacheable Drupal block! In other words: no infrastructure (changes) required!

Bringing this optimization that sits at the intersection of front-end & back-end performance to the masses rather than having it only be available for web giants like Facebook and LinkedIn is a big step forward in making the entire web fast.

Using BigPipe does not require writing a single line of custom code; the module effectively progressively enhances Drupal’s HTML rendering — and turned on by default since Drupal 8.5!

Conclusion

Like Josh and David say: don’t forget about performance fundamentals! BigPipe is no silver bullet. If you serve 100% anon traffic, BigPipe won’t make a difference. But for sites with auth traffic, personalized and uncacheable blocks on your Drupal site are streamed automatically by BigPipe, no code changes necessary:

[embedded content]

(That’s with 2 slow blocks that take 3 s to render. Only one is cacheable. Hence the page load takes ~6 s with cold caches, ~3 s with warm caches.)

Oct 25 2018
Oct 25

Mateu, Gabe and I just released the first RC of JSON API 2, so time for an update!

It’s been three months since the previous “state of JSON API” blog post, where we explained why JSON API didn’t get into Drupal 8.6 core.

What happened since then? In a nutshell:

  • We’re now much closer to getting JSON API into Drupal core!
  • JSON API 2.0-beta1, 2.0-beta2 and 2.0-rc1 were released
  • Those three releases span 84 fixed issues. (Not counting support requests.)
  • includes are now 3 times faster, 4xx responses are now cached!
  • Fixed all spec compliance issues mentioned previously
  • Zero known bugs (the only two open bugs are core bugs)
  • Only 10 remaining tasks (most of which are for test coverage in obscure cases)
  • ~75% of the open issues are feature requests!
  • ~200 sites using the beta!
  • Also new: JSON API Extras 2.10, works with JSON API 1.x & 2.x!
  • Two important features are >80% done: file uploads & revisions (they will ship in a release after 2.0)

So … now is the time to update to 2.0-RC1!

JSON API spec v1.1

We’ve also helped shape the upcoming 1.1 update to the JSON API spec, which we especially care about because it allows a JSON API server to use “profiles” to communicate support for capabilities outside the scope of the spec.

Retrospective

Now that we’ve reached a major milestone, I thought it’d be interesting to do a small retrospective using the project page’s sparklines:

JSON API project statistics, annotated with green vertical lines for the start of 2018 and the time of the previous blog post. The first green line indicates the start of 2018. Long-time Drupal & JSON API contributor Gabe Sullice joined Acquia’s Office of the CTO two weeks before 2018 started. He was hired specifically to help push forward the API-First initiative. Upon joining, he immediately started contributing to the JSON API module, and I joined him shortly thereafter. (Yes, Acquia is putting its money where its mouth is.)
The response rate for this module has always been very good, thanks to original maintainer Mateu “e0ipso” Aguiló Bosch working on it quite a lot in his sparse free time. (And some company time — thanks Lullabot!) But there’s of course a limit to how much of your free time you can contribute to open source.

  • The primary objective for Gabe and I for most of 2018 has been to get JSON API ready to move into Drupal core. We scrutinized every area of the existing JSON API module, filed lots of issues, minimized the API surface, maximized spec compliance (hence also minimizing Drupalisms), minimized potential for regressions to occur, and so on. This explains the significantly elevated rate of the new issues sparkline. It also explains why the open bugs sparkline first increased.
  • This being our primary objective also explains the response rate sparkline being at 100% nearly continously. It also explains the plummeted average first response time: it went from days to hours! This surely benefited the sites using JSON API: bug fixes happened much faster.
  • By the end of June, we managed to make the 1.x branch maximally stable and mature in the 1.22 release (shortly before the second green vertical line) — hence the “open bugs” sparkline decreased). The remaining problems required BC breaks — usually minor ones, but BC breaks nonetheless! The version of JSON API that ends up in core needs to be as future proof as possible: BC breaks are not acceptable in core. Hence the need for a 2.x branch.

Surely the increased development rate has helped JSON API reached a strong level of stability and maturity faster, and I believe this is also reflected in its adoption: a 50–70 percent increase since the end of 2017!

From 1 to 3 maintainers

This was the first time I’ve worked so closely and so actively on a small codebase in an open-source setting. I’ve learned some things.

Some of you might understandably think that Gabe and I steamrolled this module. But Mateu is still very actively involved, and every significant change still requires his blessing. Funded contributions have accelerated this module’s development, but neither Acquia nor Lullabot ever put any pressure on how it should evolve. It’s always been the module maintainers, through debate (and sometimes heartfelt concessions), who have moved this module forward.

The “participants” sparkline being at a slightly higher level than before (with more consistency!) speaks for itself. Probably more importantly: if you’re wondering how the original maintainer Mateu feels about this, I’ll be perfectly honest: it’s been frustrating at times for him — but so it’s been for Gabe and I — for everybody! Differences in availability, opinion, priorities (and private life circumstances!) all have effects. When we disagree, we meet face to face to chat about it openly.

In the end I still think it’s worth it though: Mateu has deeper ties to concrete complex projects, I have deeper ties to Drupal core requirements, and Gabe sits in between those extremes. Our discussions and disagreements force us to build consensus, which makes for a better, more balanced end result! And that’s what open source is all about: meeting the needs of more people better :)

API-first Drupal with multiple consumers @DrupalConNA :D pic.twitter.com/GhgY8O5SSa

— Gábor Hojtsy (@gaborhojtsy) April 11, 2018

Thanks to Mateu & Gabe for their feedback while writing this!

Oct 02 2018
Oct 02

Drupal 8’s REST API reached a next level of maturity in 8.5. In 8.6, we matured it further, added features and closed some gaps.

Drupal 8.6 was released with some significant API-First improvements!

The REST API made a big step forward with the 6th minor release of Drupal 8 — I hope you’ll like these improvements :)

Thanks to everyone who contributed!

  1. File uploads! #1927648

    No more crazy per-site custom REST resource plugins, complex work-arounds or base64-encoded hacks! Safe file uploads of any size are now natively supported!

    POST /file/upload/node/article/field_hero_image?_format=json HTTP/1.1
    Content-Type: application/octet-stream
    Content-Disposition: file; filename="filename.jpg"
    
    [… binary file data …]
    

    then, after receiving a response to the above request:

    POST /node?_format=json HTTP/1.1
    Content-Type: application/json
    
    {
      "type": [{"value": "article"}],
      "title": [{"value": "Dramallama"}],
      // Note that this is using the file ID we got back in the response to our previous request!
      "field_hero_image": [
        {
          "target_id": 345345,
          "description": "The most fascinating image ever!"
        }
      ]
    }
    

    If you’d like a more complete example, see the change record, which explains it in detail. And if you want to read about the design rationale, see the dedicated blog post.

  2. parent field on Term now is a standard entity reference #2543726

    "parent": []

    "parent":[{
      "target_id": 2,
      "target_type": "taxonomy_term",
      "target_uuid": "371d9486-1be8-4893-ab20-52cf5ae38e60",
      "url": "https://example.com/taxonomy/term/2"
    }]
    
    We fixed this at the root, which means it not only helps core’s REST API, but also the contributed JSON API and GraphQL modules, as well as removing the need for its previously custom Views support!
  3. alt property on image field lost in denormalization #2935738

    "field_image":[{
      "target_id": 2,
      "target_type": "file",
      "target_uuid": "be13c53e-7f95-4add-941a-fd3ef81de979",
      "alt": "Beautiful llama!"
    }]
    

    after denormalizing, saving and then normalizing, this would result in:

    "field_image":[{
      "target_id": 2,
      "target_type": "file",
      "target_uuid": "be13c53e-7f95-4add-941a-fd3ef81de979",
      "alt": ""
    }]
    

    Same thing for the description property on file and image fields, as well as text, width and height on image fields. Denormalization was simply not taking any properties into account that specializations of the entity_reference field type were adding!

  4. PATCHing a field → 403 response without with reason #2938035

    {"message":"Access denied on updating field 'sticky'."} {"message":"Access denied on updating field 'sticky'. The 'administer nodes' permission is required."}

    Just like we improved PATCH support in Drupal 8.5 (see point 4 in the 8.5 blog post), we again improved it! Previously when you’d try to modify a field you’re not allowed to modify, you’d just get a 403 response … but that wouldn’t tell you why you weren’t allowed to do so. This of course was rather frustrating, and required a certain level of Drupal knowledge to solve. Now Drupal is far more helpful!

  5. 406 responses now lists & links supported formats #2955383

    Imagine you’re doing a HTTP request like GET /entity/block/bartik_branding?_format=hal_json. The response is now more helpful.

    
    Content-Type: application/hal+json
    
    {"message": "No route found for the specified format hal_json."}
    
    

    
    Content-Type: application/hal+json
    Link: <http://example.com/entity/block/bartik_branding?_format=json>; rel="alternate"; type="application/json", >http://example.com/entity/block/bartik_branding?_format=xml>; rel="alternate"; type="text/xml"
    
    {"message": "No route found for the specified format hal_json. Supported formats: json, xml."}
    
    
  6. Modules providing entity types now responsible for REST tests

    Just like we achieved comprehensive test coverage in Drupal 8.5 (see point 7 in the 8.5 blog post), we again improved it! Previously, the rest.module component in Drupal core provided test coverage for all core entity types. But if Drupal wants to be API-First, then we need every component to make HTTP API support a priority.
    That is why in Drupal 8.6, the module providing an entity type contains said test coverage (A). We also still have test coverage test coverage (B). Put A and B together, and we’ve effectively made HTTP API support a new gate for entity types being added to Drupal core. Also see the dedicated blog post.

  7. rest.module is now maintainable!

    I’m happy to be able to proudly declare that Drupal 8 core’s rest.module in Drupal 8.6 can for the first time be considered to be in a “maintainable” state, or put differently: in a well-maintained state. I already wrote about this in a dedicated blog post 4.5 months ago. Back then, for the first time, the number of open issues fit on “a single page” (fewer than 50). Today, several months later, this is still the case. Which means that my assessment has proven true :) Whew!

Want more nuance and detail? See the REST: top priorities for Drupal 8.6.x issue on drupal.org.

Are you curious what we’re working on for Drupal 8.7? Want to follow along? Click the follow button at REST: top priorities for Drupal 8.7.x — whenever things on the list are completed (or when the list gets longer), a comment gets posted. It’s the best way to follow along closely!

The other thing that we’re working on for 8.7 besides the REST API is getting the JSON API module polished to core-worthiness. All of the above improvements help JSON API either directly or indirectly! I also wrote about this in my State of JSON API blog post. Given that the REST API is now in a solid place, for most of 2018 the majority of our attention has actually gone to JSON API, not core’s REST API. I expect this to continue to be the case.

Was this helpful? Let me know in the comments!

For reference, historical data:

Jul 10 2018
Jul 10

Quite a few people in the Drupal community are looking forward to see the JSON API module ship with Drupal 8 core.

Because:

  • they want to use it on their projects
  • the Admin UI & JS Modernization Initiative needs it
  • they want to see Drupal 8 ship with a more capable RESTful HTTP API
  • then Drupal will have a non-NIH (Not Invented Here) API but one that follows a widely used spec
  • it enables them to build progressively decoupled components

So where are things at?

Timeline

Let’s start with a high-level timeline:

  1. The plan (intent) to move the JSON API module into Drupal core was approved by Drupal’s product managers and a framework manager 4 months ago, on March 19, 2018!
  2. A core patch was posted on March 29 (issue #2843147). My colleague Gabe and I had already been working full time for a few months at that point to make the JSON API modules more stable: several security releases, much test coverage and so on.
  3. Some reviews followed, but mostly the issue (#2843147) just sat there. Anybody was free to provide feedback. We encouraged people to review, test and criticize the JSON API contrib module. People did: another 1000 sites started using JSON API! Rather than commenting on the core issue, they filed issues against the JSON API contrib module!
  4. Since December 2017, Gabe and I were still working on it full time, and e0ipso whenever his day job/free time allowed. Thanks to the test coverage Gabe and I had been adding, bugs were being fixed much faster than new ones were reported — and more often than not we found (long-existing) bugs before they were reported.
  5. Then 1.5 week ago, on June 28, we released JSON API 1.22, the final JSON API 1.x release. That same day, we branched the 2.x version. More about that below.
  6. The next day, on June 29, an updated core patch was posted. All feedback had been addressed!

June 29

I wrote in my comment:

Time to get this going again. Since #55, here’s what happened:

  1. Latest release at #55: JSON API 1.14
  2. Latest release today: JSON API 1.22
  3. 69 commits: ($ git log --oneline --since "March 30 2018 14:21 CET" | wc -l)
  4. Comprehensive test coverage completed (#2953318: Comprehensive JSON API integration test coverage phase 4: collections, filtering and sorting + #2953321: Comprehensive JSON API integration test coverage phase 5: nested includes and sparse field sets + #2972808: Comprehensive JSON API integration test coverage phase 6: POST/PATCH/DELETE of relationships)
  5. Getting the test coverage to that point revealed some security vulnerabilities (1.16), and many before it (1.14, 1.10 …)
  6. Ported many of the core REST improvements in the past 1.5 years to JSON API (1.15)
  7. Many, many, many bugfixes, and much, much clean-up for future maintainability (1.16, 1.17, 1.18, 1.19, 1.20, 1.21, 1.22)

That’s a lot, isn’t it? :)

But there’s more! All of the above happened on the 8.x-1.x branch. As described in #2952293: Branch next major: version 2, requiring Drupal core >=8.5 (and mentioned in #61), we have many reasons to start a 8.x-2.x branch. (That branch was created months ago, but we kept them identical for months.)
Why wait so long? Because we wanted all >6000 JSON API users to be able to gently migrate from JSON API 1.x (on Drupal ⇐8.5) to JSON API 2.x (on Drupal >=8.5). And what better way to do that than to write comprehensive test coverage, and fixing all known problems that that surfaced? That’s what we’ve been doing the past few months! This massively reduces the risk of adding JSON API to Drupal core. We outlined a plan of must-have issues before going into Drupal core: #2931785: The path for JSON API to core — and they’re all DONE as of today! Dozens of bugs have been flushed out and fixed before they ever entered core. Important: in the past 6–8 weeks we’ve noticed a steep drop in the number of bug reports and support requests that have been filed against the JSON API module!

After having been tasked with maturing core’s REST API, and finding the less-than-great state that was in when Drupal 8 shipped, and having experienced how hard it is to improve it or even just fix bugs, this was a hard requirement for me. I hope it gives core committers the same feeling of relief as it gives me, to see that JSON API will on day one be in much better shape.

The other reason why it’s in much better shape, is that the JSON API module now has no API surface other than the HTTP API! No PHP API (its sole API was dropped in the 2.x branch: #2982210: Move EntityToJsonApi service to JSON API Extras) at all, only the HTTP API as specified by http://jsonapi.org/format/.

TL;DR: JSON API in contrib today is more stable, more reliable, more feature-rich than core’s REST API. And it does so while strongly complying with the JSON API spec: it’s far less of a Drupalism than core’s REST API.

So, with pride, and with lots of sweat (no blood and no tears fortunately), @gabesullice, @e0ipso and I present you this massively improved core patch!

EDIT: P.S.: 668K bytes of the 1.0M of bytes that this patch contains are for test coverage. That’s 2/3rds!

To which e0ipso replied:

So, with pride, and with lots of sweat (no blood and no tears fortunately), @gabesullice, @e0ipso and I present you this massively improved core patch!

So much pride! This was a long journey, that I walked (almost) alone for a couple of years. Then @Wim Leers and @gabesullice joined and carried this to the finish line. Such a beautiful collaboration!

:)

July 9

Then, about 12 hours ago, core release manager xjm and core framework manager effulgentsia posted a comment:

(@effulgentsia and @xjm co-authored this comment.) It’s really awesome to see the progress here on JSON API! @xjm and @effulgentsia discussed this with other core committers (@webchick, @Dries, @larowlan, @catch) and with the JSON API module maintainers. Based on what we learned in these discussions, we’ve decided to target this issue for an early feature in 8.7 rather than 8.6. Therefore, we will will set it 8.7 in a few days when we branch 8.7. Reviews and comments are still welcome in the meantime, whether in this issue, or as individual issues in the jsonapi issue queue. Feel free to stop reading this comment here, or continue reading if you want to know why it’s being bumped to 8.7. First, we want to give a huge applause for everything that everyone working on the jsonapi contrib module has done. In the last 3-4 months alone (since 8.5.0 was released and #44 was written):

  • Over 100 issues in the contrib project have been closed.
  • There are currently only 36 open issues, only 7 of which are bug reports.
  • Per #62, the remaining bug fixes require breaking backwards compatibility for users of the 1.x module, so a final 1.x release has been released, and new features and BC-breaking bug fixes are now happening in the 2.x branch.
  • Also per #62, an amazing amount of test coverage has been written and correspondingly there’s been a drop in new bug reports and support requests getting filed.
  • The module is now extremely well-documented, both in the API documentation and in the Drupal.org handbook.
Given all of the above, why not commit #70 to core now, prior to 8.6 alpha? Well,
  1. We generally prefer to commit significant new core features early in the release cycle for the minor, rather than toward the end. This means that this month and the next couple are the best time to commit 8.7.x features.
  2. To minimize the disruption to contrib, API consumers, and sites of moving a stable module from core to contrib, we’d like to have it as a stable module in 8.7.0, rather than an experimental module in 8.6.0.
  3. Per above, we’re not yet done breaking BC. The mentioned spec compliance issues still need more work.
  4. While we’re still potentially evolving the API, it’s helpful to continue having the module in contrib for faster iteration and feedback.
  5. Since the 2.x branch of JSON API was just branched, there are virtually no sites using it yet (only 23 as compared with the 6000 using 1.x). An alpha release of JSON API 2.x once we’re ready will give us some quick real-world testing of the final API that we’re targeting for core.
  6. As @lauriii pointed out, an additional advantge of allowing a bit more time for API changes is that it allows more time for the Javascript Modernization Initiative, which depends on JSON API, to help validate that JSON API includes everything we need to have a fully decoupled admin frontend within Drupal core itself. (We wouldn’t block the module addition on the other initiative, but it’s an added bonus given the other reasons to target 8.7.)
  7. While the module has reached maturity in contrib, we still need the final reviews and signoffs for the core patch. Given the quality of the contrib module this should go well, but it is a 1 MB patch (with 668K of tests, but that still means 300K+ of code to review.) :) We want to give our review of this code the attention it deserves.
None of the above aside from the last point are hard blockers to adding an experimental module to core. Users who prefer the stability of the 1.x module could continue to use it from contrib, thereby overriding the one in core. However, in the case of jsonapi, I think there’s something odd about telling site builders to experiment with the one in core, but if they want to use it in production, to downgrade to the one in contrib. I think that people who are actually interested in using jsonapi on their sites would be better off going to the contrib project page and making an explicit 1.x or 2.x decision from there. Meanwhile, we see what issues, if any, people run into when upgrading from 1.x to 2.x. When we’re ready to commit it to core, we’ll consider it at least beta stability (rather than alpha). Once again, really fantastic work here.

Next

So there you have it. JSON API will not be shipping in Drupal 8.6 this fall.
The primary reason being that it’s preferred for significant new core features to land early in the release cycle, especially ones shipping as stable from the start. This also gives the Admin UI & JS Modernization Initiative more time to actually exercise many parts of JSON API’s capabilities, and in doing so validate that it’s sufficiently capable to power it.

For us as JSON API module maintainers, it keeps things easier for a little while longer: once it’s in core, it’ll be harder to iterate: more process, slower test runs, commits can only happen by core committers and not by JSON API maintainers. Ideally, we’d commit JSON API to Drupal core with zero remaining bugs and tasks, with only feature requests being left. Good news: we’re almost there already: most open issues are feature requests!

For you as JSON API users, not much changes. Just keep using https://www.drupal.org/project/jsonapi. The 2.x branch introduced some breaking changes to better comply with the JSON API spec, and also received a few new small features. But we worked hard to make sure that disruption is minimal (example 1 2 3).
Use it, try to break it, report bugs. I’m confident you’ll have to try hard to find bugs … and yes, that’s a challenge to y’all!

Jul 01 2018
Jul 01

Two weeks ago, I stumbled upon a two-part blog post by Alex Russell, titled Effective Standards Work.

The first part (The Lay Of The Land) sets the stage. The second part (Threading the Needle) attempts to draw conclusions.

It’s worth reading if you’re interested in how Drupal is developed, or in how any consensus-driven open source project works (rather than the increasingly common “controlled by a single corporate entity” “open source”).

It’s written with empathy, modesty and honesty. It shows the struggle of somebody given the task and opportunity to help shape/improve the developer experience of many, but not necessarily the resources to make it happen. I’m grateful he posted it, because something like this is not easy to write nor publish — which he also says himself:

I’ve been drafting and re-drafting versions of this post for almost 4 years. In that time I’ve promised a dozen or more people that I had a post in process that talked about these issues, but for some of the reasons I cited at the beginning, it has never seemed a good time to hit “Publish”. To those folks, my apologies for the delay.

Parallels!

I hope you’ll find the incredibly many parallels with the open source Drupal ecosystem as fascinating as I did!

Below, I’ve picked out some of the most interesting statements and replaced only a few terms, and tadaaa! — it’s accurately describing observations in the Drupal world!

Go read those two blog posts first before reading my observations though! You’ll find some that I didn’t. Then come back here and see which ones I see, having been a Drupal contributor for >11 years and a paid full-time Drupal core contributor for >6.

Standards Theory

Design A new Drupal contrib module is the process of trying to address a problem with a new feature. Standardisation Moving a contributed module into Drupal core is the process of documenting consensus.

The process of feature design Drupal contrib module development is a messy, exciting exploration embarked upon from a place of trust and hope. It requires folks who have problems (web developers site builders) and the people who can solve them (browser engineers Drupal core/contrib developers) to have wide-ranging conversations.

The Forces at Play

Feature Drupal module design starts by exploring problems without knowing the answers, whereas participation in Working Groups Drupal core initiatives entails sifting a set of proposed solutions and integrating the best proposals competing Drupal modules. Late-stage iteration can happen there, but every change made without developer site builder feedback is dangerous — and Working Groups Drupal core initiatives aren’t set up to collect or prioritise it.

A sure way for a browser engineer Drupal core/contrib developer to attract kudos is to make existing content Drupal sites work better, thereby directly improving things for users site builders who choose your browser Drupal module.

Essential Ingredients

  • Participation by web developers site builders and browser engineers Drupal core/contrib developers: Nothing good happens without both groups at the table.
  • A venue outside a chartered Working Group Drupal core in which to design and iterate: Pre-determined outcomes rarely yield new insights and approaches. Long-term relationships of WG participants Drupal core developers can also be toxic to new ideas. Nobody takes their first tap-dancing lessons under Broadway’s big lights. Start small and nimble, build from there.
  • A path towards eventual standardisation stability & maintainability: Care must be taken to ensure that IP obligations API & data model stability can be met the future, even if the loose, early group isn’t concerned with a strict IP policy update path
  • Face-to-face deliberation: I’ve never witnessed early design work go well without in-person collaboration. At a minimum, it bootstraps the human relationships necessary to jointly explore alternatives.

    If you’ve never been to a functioning standards Drupal core meeting, it’s easy to imagine languid intellectual salons wherein brilliant ideas spring forth unbidden and perfect consensus is forged in a blinding flash. Nothing could be further from the real experience. Instead, the time available to cover updates and get into nuances of proposed changes can easily eat all of the scheduled time. And this is expensive time! Even when participants don’t have to travel to meet, high-profile groups Drupal core contributors are comically busy. Recall that the most in-demand members of the group Drupal core initiative (chairs Drupal core initiative coordinators, engineers from the most consequential firms Drupal agencies) are doing this as a part-time commitment. Standards work is time away from the day-job, so making the time and expense count matters.

Design → Iterate → Ship & Standardise

What I’ve learned over the past decade trying to evolving the web platform is a frustratingly short list given the amount of pain involved in extracting each insight:

  • Do early design work in small, invested groups
  • Design in the open, but away from the bright lights of the big stage
  • Iterate furiously early on because once it’s in the web Drupal core, it’s forever
  • Prioritize plausible interoperability; if an implementer says “that can’t work”, believe them!
  • Ship to a limited audience using experimental Drupal core modules as soon as possible to get feedback
  • Drive standards stabilization of experimental Drupal core modules with evidence and developer feedback from those iterations
  • Prioritise interop minimally viable APIs & evolvability over perfect specs APIs & data models; tests create compatibility stability as much or more than tight prose or perfect IDL APIs
  • Dot “i”s and cross “t”s; chartered Working Groups Drupal core initiatives and wide review many site builders trying experimental core modules are important ways to improve your design later in the game. These derive from our overriding goal: ship the right thing.

    So how can you shape the future of the platform as a web developer site builder?

The first thing to understand is that browser engineers Drupal core/contrib developers want to solve important problems, but they might not know which problems are worth their time. Making progress with implementers site builders is often a function of helping them understand the positive impact of solving a problem. They don’t feel it, so you may need to sell it!

Building this understanding is a social process. Available, objective evidence can be an important tool, but so are stories. Getting these in front of a sympathetic audience within a browser team of Drupal core committers or Drupal contrib module maintainers is perhaps harder.

It has gotten ever easier to stay engaged as designs experimental Drupal core modules iterate. After initial meetings, early designs are sketched up and frequently posted to GitHub Drupal.org issues where you can provide comments.

“Ship The Right Thing”

These relatively new opportunities for participation outside formal processes have been intentionally constructed to give developers and evidence a larger role in the design process.

There’s a meta-critique of formal standards processes in Drupal core and the defacto-exclusionary processes used to create them. This series didn’t deal in it deeply because doing so would require a long digression into the laws surrounding anti-trust and competition. Suffice to say, I have a deep personal interest in bringing more voices into developing the future of the web platform, and the changes to Chrome’s Drupal core’s approach to standards adding new modules discussed above have been made with an explicit eye towards broader diversity, inclusion, and a greater role for evidence.

I hope you enjoyed Alex’ blog posts as much as I did!

Jun 02 2018
Jun 02

This is an ode to the Drupal Association.

DrupalCI

  1. Yesterday, I stumbled upon Customizing DrupalCI Testing for Projects, written by Ryan “Mixologic” Aslett. It contains detailed, empathic explanations. He also landed d.o/node/2969363 to make Drupal core use this capability, and to set an example.
  2. I’ve been struggling in d.o/project/jsonapi/issues/2962461 to figure out why an ostensibly trivial patch would not just fail tests, but cause the testing infrastructure to fail in inexplicable ways after 110 minutes of execution time, despite JSON API test runs normally taking 5 minutes at most! My state of mind: (ノಠ益ಠ)ノ彡┻━┻
    Three days ago, Mixologic commented on the issue and did some DrupalCI infrastructure-level digging. I didn’t ask him. I didn’t ping him. He just showed up. He’s just monitoring the DrupalCI infrastructure!

In 2015 and 2016, I must have pinged Mixologic (and others, but usually him) dozens of times in the #drupal-infrastructure IRC channel about testbot/DrupalCI being broken yet again. Our testing infrastructure was frequently having troubles then; sometimes because Drupal was making changes, sometimes because DrupalCI was regressing, and surprisingly often because Amazon Web Services was failing.

Thanks to those two things in the past few days, I realized something: I can’t remember the last time I had to ping somebody about DrupalCI being broken! I don’t think I did it once in 2018. I’m not even sure I did in 2017! This shows what a massive improvement the Drupal Association contributed to the velocity of the Drupal project!

drupal.org

Of course, many others at the Drupal Assocation help make this happen, not just Ryan.

For example Neil “drumm” Drumm. He has >2800 commits on the Drupal.org customizations project! Lately, he’s done things like making newer & older releases visible on project release pages, exposing all historical issue credits, providing nicer URLs for issues and giving project maintainers better issue queue filtering. BTW, Neil is approaching his fifteenth Drupal anniversary!
Want to know about new Drupal.org features as they go live? Watch the change recordsRSS feed available.

In a moment of frustration, I tweeted fairly harshly (uncalled for … sorry!) to @drupal_infra, and got a forgiving and funny tweet in response:

The system doesn't believe that a human could do as much as you do.

— Ryan Aslett (@ryanaslett) April 5, 2018


(In case it wasn’t obvious yet: Ryan is practically a saint!)

Thank you!

I know that the Drupal Association does much more than the above (an obvious example is organizing DrupalCons). But these are the ways in which they are most visible to me.

When things are running as smoothly as they are, it’s easy to forget that it takes hard work to get there and stay there. It’s easy to take this for granted. We shouldn’t. I shouldn’t. I did for a while, then realized … this blog post is the result!

A big thanks to everyone who works/worked at the Drupal Association! You’ve made a tangible difference in my professional life! Drupal would not be where it is today without you.

May 17 2018
May 17

Two big “maintainability” milestones have been hit in the past few days:

1. rest.module now is in a maintainable state

For the first time ever, the issue tracker for the rest.module Drupal core component fits on a single page: https://www.drupal.org/project/issues/drupal?component=rest.module. 48 46 open issues, of which several are close to RTBC, so that number will likely still go down. Breakdown:

  • 19 of those 48 are feature requests.
  • 6 are “plan” issues.
  • At least 10 issues are related to “REST views” — for which we are only fixing critical bugs.
  • And 12 are postponed — blocked on other subsystems usually.

(There is overlap among those breakdown bullets.)

Finally the REST module is getting to a place where it is maintainable and we’re not extinguishing whatever the current fire is! It’s been a long road, but we’re getting there!

2. Instilling API-First responsibility

Two days ago, #2910883: Move all entity type REST tests to the providing modules landed, which is a huge milestone in making Drupal 8 truly API-First: every module now owns its REST test coverage, which conveys the fact that every component/module is responsible for its own REST API-compatibility, rather than all responsibility lying in the rest.module component!

This is largely made possible by \Drupal\Tests\rest\Functional\EntityResource\EntityResourceTestBase, which is a base test class that each entity type should subclass to test how it behaves when exposed via REST. Every entity type in core has tests based on it, but contrib and custom entity types are encouraged to do the same!

API-First ecosystem

But the rest.module being stable is not the only thing that matters — it’s a key part, but not the only part of API-First Drupal. The remaining challenges lie elsewhere in Drupal: the issues tagged API-First Initiative are now mainly in the modules providing field types, entity types, Entity/Field API and Configuration API.

The good thing is that the fixes to any of those always help the entire API-First ecosystem:

If you want to follow along a bit more closely, or want the news right as it happens, follow REST: top priorities for Drupal 8.6.x!

Apr 28 2018
Apr 28

Wim, thanks for the presentation all the work you and others are doing on this initiative.

Also thanks for making the slides public! I used the slides at a recent Boston Drupal Meetup and they were a great way to catch people up on the initiative.

Apr 08 2018
Apr 08

This blog post summarizes the 572 comments spanning 5 years and 2 months to get REST file upload support in #1927648 committed. Many thanks to everyone who contributed!

From February 2013 until the end of March 2017, issue #1927648 mostly … lingered. On April 3 of 2017, damiankloip posted an initial patch for an approach he’d been working on for a while, thanks to Acquia (my employer) sponsoring his time. Exactly one year later his work is committed to Drupal core. Shaped by the input of dozens of people! Just *look at that commit message!*

Background: API-First Drupal: file uploads!.

  • Little happened between February 2013 (opening of issue) and November 2015 (shipping of Drupal 8).
  • Between February 2013 and April 2014, only half a dozen comments were posted, until moshe weitzman aptly said Still a gaping hole in our REST support. Come on Internets ….
  • The first proof-of-concept patch followed in August 2014 by juampynr, but was still very rough. A fair amount of iteration occurred that month, between juampynr and Arla. It used base64 encoding, which means it needed 33% more bytes on the wire to transfer a file than if it were transmitted in binary rather than base64.
  • Then again a period of silence. Remember that this was around the time when we were trying to get Drupal 8 to a shippable state: the #1 priority was to stabilize, fix critical bugs. Not to add missing features, no matter how important. To the best of my knowledge, the funding for those who originally worked on Drupal 8’s REST API had also dried up.
  • In May 2015, another flurry of activity occurred, this time fueled by marthinal. Comment #100 was posted. Note that all patches up until this point had zero validation logic! Which of course was a massive security risk. marthinal is the first to state that this is really necessary, and does a first iteration of that.
  • A few months of silence, and then again progress in September, around DrupalCon Barcelona 2015. dawehner remarked in a review on the lack of tests for the validation logic.
  • In February 2016 I pointed out that I’m missing integration tests that prove the patch actually works. To which Berdir responded that we’d first need to figure out how to deal with File entity type access control!
  • Meanwhile, marthinal works on the integration test coverage in 2016. And … we reached comment #200.
  • In May 2016, I did a deep review, and found many problems. Quick iterations fix those problems! But then damiankloip pointed out that despite the issue being about the general File (de)serialization problem, it actually only worked for the HAL normalization. We also ended up realizing that the issue so far was about stand-alone File entity creation, even though those entities cannot be viewed stand-alone nor can they be created stand-alone through the existing Drupal UI: they can only be created to be referenced from file fields. And consequently, we have no access control logic for this yet, nor is it clear how access control should work; nor is it how validation should work! Berdir explained this well in comment 232. This lead us to explore moving parts of https://www.drupal.org/project/file_entity into core (which would be a hard blocker). The issue then went quiet again.
  • In July 2016, garphy pointed out that large file uploads still were not yet supported. Some work around that happened. In September, kylebrowning stressed this again, and provided a more detailed rationale.
  • Then … silence. Until damiankloip posted comment #281 on April 3, 2017. Acquia was sponsoring him to work on this issue. Damian is the maintainer of the serialization.module component and therefore of course wanted to see this issue get fixed. My employer Acquia agreed with my proposal to sponsor Damian to work on REST file upload support. Because after 280 comments, some fundamental capabilities are still absent: this was such a complex issue, with so many concerns and needs to balance, that it was nigh impossible to finish it without dedicated time.
    To get this going, I asked Damian to look at the documentation for a bunch of well-known sites to observe how they handle file uploads. I also asked him to read the entire issue. Combined, this should give him a good mental map of how to approach this.
  • #281 was a PoC patch that only barely worked but did support binary (non-base64) uploads. damiankloip articulated the essential things yet to be figured out: validation and access checking. Berdir chimes in with his perspective on that in #291 … in which he basically outlines what ended up in core! Besides Berdir, dagmar, dawehner, garphy, dabito, ibustos all chimed in and influenced the patch. Berdir, damiankloip and I had a meeting about how to deal with validation, and I disagreed with with both of them. And turned out to be very wrong! More feedback is provided by the now familiar names, and the intense progress/activity continues for two months, until comment #376!
  • Damian got stuck on test coverage — and since I’d written most of the REST test coverage in the preceding months, it made sense for me to pick up the baton from Damian. So I did that in July 2017, just making trivial changes that were hard to figure out. Damian then continued again, expanding test coverage and finding a core bug in the process! And so comment #400 was reached!
  • At the beginning of August, the patch was looking pretty good, so I did an architectural review. For the first time, we realized that we first needed to fix the normalization of File entities before this could land. And many more edge cases need to be tested for us to be confident that there were no security vulnerabilities. blainelang did manual testing and posted super helpful feedback based on his experience. Blaine and Damian tag-teamed for a good while, then graphy chimed in again, and we entered September. Then dawehner chimed in once more, followed by tedbow.
  • On September 6 2017, in comment #452 I marked the issue postponed on two other issues, stating that it otherwise looked tantalizingly close to RTBC. aheimlich found a problem nobody else had spotted yet, which Damian fixed.
  • Silence while the other issues get fixed … and December 21 2017 (comment #476), it finally was unblocked! Lots of detailed reviews by tedbow, gabesullice, Berdir and myself followed, as well as rerolls to address them, until I finally RTBC‘d it … in comment #502 on February 1 2018.
  • Due to the pending Drupal 8.5 release, the issue mostly sat waiting in RTBC for about two months … and then got committed on April 3 2018!!!

Damian’s first comment (preceded by many hours of research) was on April 3, 2017. Exactly one year later his work is committed to Drupal core. Shaped by the input of dozens of people! Just look at that commit message!

Apr 08 2018
Apr 08

Drupal 8’s REST API has been maturing steadily since the Drupal 8.0.0 was released in November 2015. One of the big missing features has been file upload support. As of April 3 2018, Drupal 8.6 will support it, when it ships in September 2018! See the change record for the practical consequences: https://www.drupal.org/node/2941420.

It doesn’t make sense for me to repeat what is already written in that change record: that already has both a tl;dr and a practical example.

What I’m going to do instead, is give you a high-level overview of what it took to get to this point: why it took so long, which considerations went into it, why this particular approach was chosen. You could read the entire issue (#1927648), but … it’s one of the longest issues in Drupal history, at 572 comments. You would probably need at least an entire workday to read it all! It’s also one of the longest commit messages ever, thanks to the many, many people who shaped it over the years:

Issue #1927648 by damiankloip, Wim Leers, marthinal, tedbow, Arla, alexpott, juampynr, garphy, bc, ibustos, eiriksm, larowlan, dawehner, gcardinal, vivekvpandya, kylebrowning, Sam152, neclimdul, pnagornyak, drnikki, gaurav.goyal, queenvictoria, kim.pepper, Berdir, clemens.tolboom, blainelang, moshe weitzman, linclark, webchick, Dave Reid, dabito, skyredwang, klausi, dagmar, gabesullice, pwolanin, amateescu, slashrsm, andypost, catch, aheimlich: Allow creation of file entities from binary data via REST requests

Thanks to all of you in that commit message!

I hope it can serve as a reference not just for people interested in Drupal, but also for people outside the Drupal community: there is no One Best Practice Way to handle file uploads for RESTful APIs. There is a surprising spectrum of approaches. Some even avoid this problem space even entirely, by only allowing to “upload” files by sending a publicly accessible URL to the file. Read on if you’re interested. Otherwise, go and give it a try!

Design rationale

General:

  • Request with Content-Type: application/octet-stream aka “raw binary” as its body, because base64-encoded means 33% more bytes, implying both slower uploads and more memory consumption. Uploading videos (often hundreds of megabytes or even gigabytes) is not really feasible with base64 encoding.
  • Request header Content-Disposition: file; filename="cat.jpg" to name the uploaded file. See the Mozilla docs. This also implies you can only upload one file per request. But of course, a client can issue multiple file upload requests in parallel, to achieve concurrent/batch uploading.
  • The two points above mean we reuse as much as possible from existing HTTP infrastructure.
  • Of course it does not make sense to have a Content-Type: application/octet-stream as the response. Usually, the response is of the same MIME type as the request. File uploads are the sensible exception.
  • This is meant for the raw file upload only; any metadata (for example: source or licensing) cannot be associated in this request: all you can provide is the name and the data for the file. To associate metadata, a second request to “upgrade” the raw file into something richer would be necessary. The performance benefit mentioned above more than makes up for the RTT of a second request in almost all cases.

PHP-specific:

  • php://input because otherwise limited by the PHP memory limit.

Drupal-specific:

  • In the case of Drupal, we know that it always represents files as File entities. They don’t contain metadata (fields), at least not with just Drupal core; it’s the file fields (@FieldType=file or @FieldType=image) that contain the metadata (because the same image may need different captions depending on its use, for example).
  • When a file is uploaded for a field on a bundle on an entity type, a File entity is created with status=false. The response contains the serialized File entity.
  • You then need a second request to make the referencing entity “use” the File entity, which will cause the File entity to get status=true.
  • Validation: Drupal core only has the infrastructure in place to use files in the context of an entity type/bundle’s file field (or derivatives thereof, such as image fields). This is why files can only be uploaded by specifying an entity type ID, bundle ID and field name: that’s the level where we have settings and validation logic in place. While not ideal, it’s pragmatic: first allowing generic file uploads would be a big undertaking and somewhat of a security nightmare.
  • Access control is similar: you need create access for the referencing entity type and field edit access for the file field.

Result

If we combine all these choices, then we end up with a new file_upload @RestResource plugin, which enables clients to upload a file:

  1. by POSTing the file’s contents
  2. to the path /file/upload/{entity_type_id}/{bundle}/{field_name}, which means that we’re uploading a file to be used by the file field of the specified entity type+bundle, and the settings/constraints of that field will be respected.
  3. … don’t forget to include a ?_format URL query argument, this determines what format the response will be in
  4. sending file data as a application/octet-stream binary data stream, that means with a Content-Type: application/octet-stream request header. (This allows uploads of an arbitrary size, including uploads larger than the PHP memory limit.)
  5. and finally, naming the file using the Content-Disposition: file; filename="filename.jpg" header
  6. the five preceding steps result in a successfully uploaded file with status=false — all that remains is to perform a second request to actually start using the file in the referencing entity!

Four years in the making — summarizing 572 comments

From February 2013 until the end of March 2017, issue #1927648 mostly … lingered. On April 3 of 2017, damiankloip posted an initial patch for an approach he’d been working on for a while, thanks to Acquia (my employer) sponsoring his time. Exactly one year later his work is committed to Drupal core. Shaped by the input of dozens of people! Just look at that commit message!

Want to actually read a summary of those 572 comments? I got you covered!

Mar 02 2018
Mar 02

Now that Drupal 8’s REST API has reached the next level of maturity, I think a concise blog post summarizing the most important API-First Initiative improvements for every minor release is going to help a lot of developers. Drupal 8.5.0 will be released next week and the RC was tagged last week. So, let’s get right to it!

The REST API made a big step forward with the 5th minor release of Drupal 8 — I hope you’ll like these improvements :)

Thanks to everyone who contributed!

  1. text fields’ computed processed property exposed #2626924

    No more need to re-implement this in consumers nor work-arounds.

    "body":{
      "value":"<p>Hi!</p><script>alert('foo')</script>",
      "format":"basic_html"
    }
    

    "body":{
      "value":"<p>Hi!</p><script>alert('foo')</script>",
      "format":"basic_html",
      "processed":"<p>Hi!</p>"
    }
    
  2. uri field on File gained a computed url property #2825487

    "uri":{"value":"public://cat.png"} "uri":{"url":"/files/cat.png","value":"public://cat.png"}

  3. Term POSTing requires non-admin permission #1848686

    administer taxonomy permission create terms in %vocabulary% permission

    Analogously for PATCH and DELETE: you need edit terms in %vocabulary% and delete terms in %vocabulary%, respectively.

  4. Vocabulary GETting requires non-admin permission #2808217

    administer taxonomy permission access taxonomy overview permission

  5. GET → decode → modify field → encode → PATCH403 200 #2824851

    You can now GET a response, modifying the bits you want to change, and then sending exactly that, without needing to remove fields you’re not allowed to modify. Any fields that you’re not allowed to modify can still be sent without resulting in a 403 response, as long as you send exactly the same values. Drupal’s REST API now implements the robustness principle.

  6. 4xx GET responses cacheable: more scalable + faster #2765959

    Valuable for use cases where you have many (for example a million) anonymous consumers hitting the same URL. Because the response is not cacheable, it can also not be cached by a reverse proxy in front of it. Meaning that we’ll have hundreds of thousands of requests hitting origin, which can bring down the origin server.

  7. Comprehensive integration tests + test coverage test coverage

    This massively reduces the risk of REST API regressions/changes/BC breaks making it into a Drupal 8 release. It allows us to improve things faster, because we can be confident that most regressions will be detected. That even includes the support for XML serialization, for the handful of you who are using that! We take backwards compatibility serious.
    Even better: we have test coverage test coverage: tests that ensure we have integration tests for every entity type that Drupal core’s stable modules ship with!
    Details at API-First Drupal — really!. Getting to this point took more than a year and required fixing bugs in many other parts of Drupal!

Want more nuance and detail? See the REST: top priorities for Drupal 8.5.x issue on drupal.org.

Are you curious what we’re working on for Drupal 8.6? Want to follow along? Click the follow button at REST: top priorities for Drupal 8.6.x — whenever things on the list are completed (or when the list gets longer), a comment gets posted. It’s the best way to follow along closely!

The other thing that we’re working on for 8.6 besides the REST API is getting the JSON API module polished to core-worthiness. All of the above improvements help JSON API either directly or indirectly! More about that in a future blog post.

Was this helpful? Let me know in the comments!

Thanks to Ted for reviewing a draft of this blog post! And sorry for not changing the title to API First Drupal in 8.5.0: Amazing progress because of tedbow’s work on setInternal properties!!!!!!!!! even though that would’ve been totally accurate :D

For reference, historical data:

Dec 08 2017
Dec 08

This blog has been quiet for the last year and a half, because I don’t like to announce things until I feel comfortable recommending them. Until today!

Since July 2016, API-First Drupal became my primary focus, because Dries felt this was one of the most important areas for Drupal’s future. Together with the community, I triaged the issue queue, and helped determine the most important bugs to fix and improvements to add. That’s how we ended up with REST: top priorities for Drupal … plan issues for each Drupal 8 minor:

If you want to see what’s going on, start following that last issue. Whenever there’s news, I post a new comment there.

But enough background. This blog post is not an update on the entire API-First Initiative, it’s about a particular milestone.

100% integration test coverage!

The biggest problem we encountered while working on rest.module, serialization.module and hal.module was unknown BC breaks . Because in case of a REST API, the HTTP response is the API. What is a bug fix for person X is a BC break for person Y. The existing test coverage was rather thin, and was often only testing “the happy path”: the simplest possible case. That’s why we would often accidentally introduce BC breaks.

Hence the clear need for really thorough functional (integration) test coverage, which was completed almost exactly a year ago. We added EntityResourceTestBase, which tests dozens of scenarios in a generic way, and used that to test the 9 entity types, that already had some REST test coverage, more thoroughly than before.

But we had to bring this to all entity types in Drupal core … and covering all 41 entity types in Drupal core was completed exactly a week ago!

The test coverage revealed bugs for almost every entity type. (Most of them are fixed by now.)

Tip: Subclass that base test class for your custom entity types, and easily get full REST test coverage — 41 examples available!

Guaranteed to remain at 100%

We added EntityResourceRestTestCoverageTest, which verifies that we have test coverage for all permutations of:

  • entity type
  • format: json + xml + hal_json
  • authentication: cookie + basic_auth + anon

It is now impossible to add new entity types without also adding solid REST test coverage!

If you forget that test coverage, you’ll find an ASCII-art llama talking to you:

Good people of #Drupal, I present unto you the greatest method of all time. https://github.com/drupal/drupal/blob/8.5.x/core/modules/rest/tests/src/Functional/EntityResource/EntityResourceRestTestCoverageTest.php#L141 pic.twitter.com/TiWLPt7duH

— webcsillag (@webchick) December 8, 2017

That is why we can finally say that Drupal is really API-First!

This of course doesn’t help only core’s REST module, it also helps the contributed JSON API and GraphQL modules: they’ll encounter far fewer bugs!

Thanks

So many people have helped! In random order: rogierbom, alexpott, harings_rob, himanshu-dixit, webflo, tedbow, xjm, yoroy, timmillwood, gaurav.kapoor, Gábor Hojtsy, brentschuddinck, Sam152, seanB, Berdir, larowlan, Yogesh Pawar, jibran, catch, sumanthkumarc, amateescu, andypost, dawehner, naveenvalecha, tstoeckler — thank you all!

Special thanks to three people I omitted above, because they’re not well known in the Drupal community, and totally deserve the spotlight here, for their impressive contribution to making this happen:

That’s thirty contributors without whom this would not have happened!

And of course thanks to my employer, Acquia, for allowing me to work on this full-time!

Next

What is going to be the next big milestone we hit? That’s impossible to say, because it depends on the chains of blocking issues that we encounter. It could be support for modifying and creating config entities, it could be support for translations, it could be that all major serialization gaps are fixed, it could be file uploads, or it could be ensuring all normalizers work in both rest.module & jsonapi.module

The future will tell, follow along!

Nov 06 2017
Nov 06

The Drupal render pipeline and its caching capabilities have been the subject of quite a few talks of mine and of multiple writings. But all of those were very technical, very precise.

Over the past year and a half I’d heard multiple times there was a need for a more pragmatic talk, where only high-level principles are explained, and it is demonstrated how to step through the various layers with a debugger. So I set out to do just that.

I figured it made sense to spend 10–15 minutes explaining (using a hand-drawn diagram that I spent a lot of time tweaking) and spend the rest of the time stepping through things live. Yes, this was frightening. Yes, there were last-minute problems (my IDE suddenly didn’t allow font size scaling …), but it seems overall people were very satisfied :)

Have you seen and heard of Render API (with its render caching, lazy builders and render pipeline), Cache API (and its cache tags & contexts), Dynamic Page Cache, Page Cache and BigPipe? Have you cursed them, wondered about them, been confused by them?

I will show you three typical use cases:

  1. An uncacheable block
  2. A personalized block
  3. A cacheable block that you can see if you have a certain permission and that should update whenever some entity is updated

… and for each, will take you on the journey through the various layers: from rendering to render caching, on to Dynamic Page Cache and eventually Page Cache … or BigPipe.

Coming out of this session, you should have a concrete understanding of how these various layers cooperate, how you as a Drupal developer can use them to your advantage, and how you can test that it’s behaving correctly.

I’m a maintainer of Dynamic Page Cache and BigPipe, and an effective co-maintainer of Render API, Cache API and Page Cache.

Preview:

Conference: 

Drupalcon Vienna

Location: 

Vienna, Austria

Date: 

Sep 28 2017 - 14:15

Duration: 

60 minutes

Jul 09 2017
Jul 09

The first release of the CDN module for Drupal was 9.5 years ago yesterday: cdn 5.x-1.0-beta1 was released on January 8, 2008!

Excitement

On January 27, 2008, the first RC followed, with boatloads of new features. Over the years, it was ported to Drupal 6, 7 and 8 and gained more features (I effectively added every single feature that was requested — I loved empowering the site builder). I did the same with my Hierarchical Select module.

I was a Computer Science student for the first half of those 9.5 years, and it was super exciting to see people actually use my code on hundreds, thousands and even tens of thousands of sites! In stark contrast with the assignments at university, where the results were graded, then discarded.

Frustration

Unfortunately this approach resulted in feature-rich modules, with complex UIs to configure them, and many, many bug reports and support requests, because they were so brittle and confusing. Rather than making the 80% case simple, I supported 99% of needed features, and made things confusing and complex for 100% of the users.

CDN settings: 'Details' tab Main CDN module configuration UI in Drupal 7.

Learning

In my job in Acquia’s Office of the CTO, my job is effectively “make Drupal better & faster”.

In 2012–2013, it was improving the authoring experience by adding in-place editing and tightly integrating CKEditor. Then it shifted in 2014 and 2015 to “make Drupal 8 shippable”, first by working on the cache system, then on the render pipeline and finally on the intersection of both: Dynamic Page Cache and BigPipe. After Drupal 8 shipped at the end of 2015, the next thing became “improve Drupal 8’s REST APIs”, which grew into the API-First Initiative.

All this time (5 years already!), I’ve been helping to build Drupal itself (the system, the APIs, the infrastructure, the overarching architecture), and have seen the long-term consequences from both up close and afar: the concepts required to understand how it all works, the APIs to extend, override and plug in to. In that half decade, I’ve often cursed past commits, including my own!

That’s what led to:

  • my insistence that the dynamic_page_cache and big_pipe modules in Drupal 8 core do not have a UI, nor any configuration, and rely entirely on existing APIs and metadata to do their thing (with only a handful of bug reports in >18 months!)
  • my “Backwards Compatibility: Burden & Benefit” talk a few months ago
  • and of course this rewrite of the CDN module

CDN module in Drupal 8: radically simpler

I started porting the CDN module to Drupal 8 in March 2016 — a few months after the release of Drupal 8. It is much simpler to use (just look at the UI). It has less overhead (the UI is in a separate module, the altering of file URLs has far simpler logic). It has lower technical complexity (File Conveyor support was dropped, it no longer needs to detect HTTP vs HTTPS: it always uses protocol-relative URLs, less unnecessary configurability, the farfuture functionality no longer tries to generate file and no longer has extremely detailed configurability).

In other words: the CDN module in Drupal 8 is much simpler. And has much better test coverage too. (You can see this in the tarball size too: it’s about half of the Drupal 7 version of the module, despite significantly more test coverage!)

CDN UI module version 3.0-rc2 on Drupal 8 CDN UI module in Drupal 8. all the fundamentals
  • the ability to use simple CDN mappings, including conditional ones depending on file extensions, auto-balancing, and complex combinations of all of the above
  • preconnecting (and DNS prefetching for older browsers)
  • a simple UI to set it up — in fact, much simpler than before!
changed/improved
  1. the CDN module now always uses protocol-relative URLs, which means there’s no more need to distinguish between HTTP and HTTPS, which simplifies a lot
  2. the UI is now a separate module
  3. the UI is optional: for power users there is a sensible configuration structure with strict config schema validation
  4. complete unit test coverage of the heart of the CDN module, thanks to D8’s improved architecture
  5. preconnecting (and DNS prefetching) using headers rather than tags in , which allows a much simpler/cleaner Symfony response subscriber
  6. tours instead of advanced help, which very often was ignored
  7. there is nothing to configure for the SEO (duplicate content prevention) feature anymore
  8. nor is there anything to configure for the Forever cacheable files feature anymore (named Far Future expiration in Drupal 7), and it’s a lot more robust
removed
  1. File Conveyor support
  2. separate HTTPS mapping (also mentioned above)
  3. all the exceptions (blacklist, whitelist, based on Drupal path, file path…) — all of them are a maintenance/debugging/cacheability nightmare
  4. configurability of SEO feature
  5. configurability of unique file identifiers for the Forever cacheable files feature
  6. testing mode

For very complex mappings, you must manipulate cdn.settings.yml — there’s inline documentation with examples there. Those who need the complex setups don’t mind reading three commented examples in a YAML file. This used to be configurable through the UI, but it also was possible to configure it “incorrectly”, resulting in broken sites — that’s no longer possible.

There’s comprehensive test coverage for everything in the critical path, and basic integration test coverage. Together, they ensure peace of mind, and uncover bugs in the next minor Drupal 8 release: BC breaks are detected early and automatically.

The results after 8 months: contributed module maintainer bliss

The first stable release of the CDN module for Drupal 8 was published on December 2, 2016. Today, I released the first patch release: cdn 8.x-3.1. The change log is tiny: a PHP notice fixed, two minor automated testing infrastructure problems fixed, and two new minor features added.

We can now compare the Drupal 7 and 8 versions of the CDN module:

In other words: maintaining this contributed module now requires pretty much zero effort!

Conclusion

For your own Drupal 8 modules, no matter if they’re contributed or custom, I recommend a few key rules:

  • Selective feature set.
  • Comprehensive unit test coverage for critical code paths (UnitTestCase) + basic integration test coverage (BrowserTestBase) maximizes confidence while minimizing time spent.
  • Don’t provide/build APIs (that includes hooks) unless you see a strong use case for it. Prefer coarse over granular APIs unless you’re absolutely certain.
  • Avoid configurability if possible. Otherwise, use config schemas to your advantage, provide a simple UI for the 80% use case. Leave the rest to contrib/custom modules.

This is more empowering for the Drupal site builder persona, because they can’t shoot themselves in the foot anymore. It’s no longer necessary to learn the complex edge cases in each contributed module’s domain, because they’re no longer exposed in the UI. In other words: domain complexities no longer leak into the UI.

At the same time, it hugely decreases the risk of burnout in module maintainers!

And of course: use the CDN module, it’s rock solid! :)

Related reading

Finally, read Amitai Burstein’s OG8 Development Mindset”! He makes very similar observations, albeit about a much bigger contributed module (Organic Groups). Some of my favorite quotes:

  1. About edge cases & complexity:

    Edge cases are no longer my concern. I mean, I’m making sure that edge cases can be done and the API will cater to it, but I won’t go too far and implement them. […] we’ve somewhat reduced the flexibility in order to reduce the complexity; but while doing so, made sure edge cases can still hook into the process.

  2. About tests:

    I think there is another hidden merit in tests. By taking the time to carefully go over your own code - and using it - you give yourself some pause to think about the necessity of your recently added code. Do you really need it? If you are not afraid of writing code and then throwing it out the window, and you are true to yourself, you can create a better, less complex, and polished module.

  3. About feature set & UI:

    One of the mistakes that I feel made in OG7 was exposing a lot of the advanced functionality in the UI. […] But these are all advanced use cases. When thinking about how to port them to OG8, I think found the perfect solution: we did’t port it.

Apr 27 2017
Apr 27

As part of working in Acquia’s Office of the CTO, I’ve been working on the API-First Initiative for the past year. Where are we at? Find out :)

Preview:

Conference: 

DrupalCon Baltimore

Location: 

Baltimore, USA

Date: 

Apr 27 2017 - 11:45

Duration: 

60 minutes

Mar 22 2017
Mar 22

In my job at Acquia, I’ve been working almost exclusively on Drupal 8 core. In 2012–2013 I worked on authoring experience (in-place editing, CKEditor, and more). In 2014–2015, I worked on performance, cacheability, rendering and generally the stabilizing of Drupal 8. Drupal 8.0.0 shipped on November 19, 2015. And since then, I’ve spent most of my time on making Drupal 8 be API-first: improving the RESTful Web Services support that Drupal 8 ships with, and in the process also strengthening the JSON API & GraphQL contributed modules.

In the process, I’ve learned a lot about the impact of past decisions (by myself and others) on backwards compatibility. The benefit of backwards compatibility (BC). But the burden of ensuring BC can increase exponentially due to certain architectural decisions. I’ve been experiencing that first-hand, since I’m tasked with making Drupal 8’s REST support rock-solid, where I am seeing time and time again that “fixing bugs + improving DX” requires BC breaks. Tough decisions.

In this talk (which will be a core conversation at DrupalCon Baltimore), I analyzed the architectural patterns in Drupal 8, and provided suggestions on how to do better. I don’t have all the answers. But what matters most is not answers, but a critical mindset going forward that is consciously considering BC implications for every patch that goes into Drupal 8!

Not thoroughly considering new interfaces (API surface) risks making the software unmaintainable!

Preview:

Conference: 

Drupal Dev Days Seville 2017 + DrupalCon Baltimore

Location: 

Seville, Spain + Baltimore, USA

Date: 

Mar 22 2017 - 15:30

Duration: 

50 minutes

Feb 20 2017
Feb 20

This is an ode to Dirk Engling’s OpenTracker.

It’s a BitTorrent tracker.

It’s what powered The Pirate Bay in 2007–2009.

I’ve been using it to power the downloads on http://driverpacks.net since the end of November 2010. >6 years. It facilitated 9839566 downloads since December 1, 2010 until today. That’s almost 10 million downloads!

Stability

It’s one of the most stable pieces of software I ever encountered. I compiled it in 2010, it never once crashed. I’ve seen uptimes of hundreds of days.

[email protected]:~$ ls -al /data/opentracker
total 456
drwxr-xr-x  3 wim  wim   4096 Feb 11 01:02 .
drwxr-x--x 10 root wim   4096 Mar  8  2012 ..
-rwxr-xr-x  1 wim  wim  84824 Nov 29  2010 opentracker
-rw-r--r--  1 wim  wim   3538 Nov 29  2010 opentracker.conf
drwxr-xr-x  4 wim  wim   4096 Nov 19  2010 src
-rw-r--r--  1 wim  wim 243611 Nov 19  2010 src.tgz
-rwxrwxrwx  1 wim  wim  14022 Dec 24  2012 whitelist

Simplicity

The simplicity is fantastic. Getting up and running is incredibly simple: git clone git://erdgeist.org/opentracker .; make; ./opentracker and you’re up and running. Let me quote a bit from its homepage, to show that it goes the extra mile to make users successful:

opentracker can be run by just typing ./opentracker. This will make opentracker bind to 0.0.0.0:6969 and happily serve all torrents presented to it. If ran as root, opentracker will immediately chroot to . and drop all priviliges after binding to whatever tcp or udp ports it is requested.

Emphasis mine. And I can’t emphasize my emphasis enough.

Performance & efficiency

All the while handling dozens of requests per second, opentracker causes less load than background processes of the OS. Let me again quote a bit from its homepage:

opentracker can easily serve multiple thousands of requests on a standard plastic WLAN-router, limited only by your kernels capabilities ;)

That’s also what the homepage said in 2010. It’s one of the reasons why I dared to give it a try. I didn’t test it on a “plastic WLAN-router”, but everything I’ve seen confirms it.

Flexibility

Its defaults are sane, but what if you want to have a whitelist?

  1. Uncomment the #FEATURES+=-DWANT_ACCESSLIST_WHITE line in the Makefile.
  2. Recompile.
  3. Create a file called whitelist, with one torrent hash per line.

Have a need to update this whitelist, for example a new release of your software to distribute? Of course you don’t want to reboot your opentracker instance and lose all current state. It’s got you covered:

  1. Append a line to whitelist.
  2. Send the SIGHUP UNIX signal to make opentracker reload its whitelist.

Deployment

I’ve been in the process of moving off of my current (super reliable, but also expensive) hosting. There are plenty of specialized companies offering HTTP hosting and even rsync hosting. Thanks to their standardization and consequent scale, they can offer very low prices.

But I also needed to continue to run my own BitTorrent tracker. There are no companies that offer that. I don’t want to rely on another tracker, because I want there to be zero affiliation with illegal files. This is a BitTorrent tracker that does not allow anything to be shared: it only allows the software releases made by http://driverpacks.net to be downloaded.

So, I found the cheapest VPS I could find, with the least amount of resources. For USD $13.50, I got the lowest specced VPS from a reliable-looking provider: with 128 MB RAM. Then I set it up:

  1. ssh‘d onto it.
  2. rsync‘d over the files from my current server (alternatively: git clone and make)
  3. added @reboot /data/opentracker/opentracker -f /data/opentracker/opentracker.conf to my crontab.
  4. removed the CNAME record for tracker.driverpacks.net, and instead made it an A record pointing to my new VPS.
  5. watched http://tracker.driverpacks.net:6969/stats?mode=tpbs&format=txt on both the new and the old server, to verify traffic was moving over to my new cheap opentracker VPS as the DNS changes propagated

Drupal module

Since driverpacks.net runs on Drupal, there of course is an OpenTracker Drupal module to integrate the two (I wrote it). It provides an API to:

  • create .torrent files for certain files uploaded to Drupal
  • append to the OpenTracker whitelist file
  • parse the statistics provided by the OpenTracker instance

You can see the live stats at http://driverpacks.net/stats.

Conclusion

opentracker is the sort of simple, elegant software design that makes it a pleasure to use. And considering the low commit frequency over the past decade, with many of those commits being nitpick fixes, it also seems its simplicity also leads to excellent maintainability. It involves the HTTP and BitTorrent protocols, yet only relies on a single I/O library, and its source code is very readable. Not only that, but it’s also highly scalable.

It’s the sort of software many of us aspire to write.

Finally, its license. A glorious license indeed!

The beerware license is very open, close to public domain, but insists on honoring the original author by just not claiming that the code is yours. Instead assume that someone writing Open Source Software in the domain you’re obviously interested in would be a nice match for having a beer with.

So, just keep the name and contact details intact and if you ever meet the author in person, just have an appropriate brand of sparkling beverage choice together. The conversation will be worth the time for both of you.

Dirk, if you read this: I’d love to buy you sparkling beverages some time :)

Feb 14 2017
Feb 14

Hasselt University Professor Frank Neven asked me to come and talk a bit about my experience in open source, and how it helped me. It helped me during my studies, in my career and even in life :)

Preview:

Location: 

Hasselt University

Date: 

Feb 14 2017 - 15:30

Duration: 

50 minutes

Apr 20 2016
Apr 20

Today, Drupal 8.1 has been released and it includes BigPipe as an experimental module.

Six months ago, on the day of the release of Drupal 8, the BigPipe contrib module was released.

So BigPipe was first prototyped in contrib, then moved into core as an experimental module.

Experimental module?

Quoting d.o/core/experimental:

Experimental modules allow core contributors to iterate quickly on functionality that may be supported in an upcoming minor release and receive feedback, without needing to conform to the rigorous requirements for production versions of Drupal core.

Experimental modules allow site builders and contributed project authors to test out functionality that might eventually be included as a stable part of Drupal core.

With your help (in other words: by testing), we can help BigPipe “graduate” as a stable module in Drupal 8.2. This is the sort of module that needs wider testing because it changes how pages are delivered, so before it can be considered stable, it must be tested in as many circumstances as possible, including the most exotic ones.

(If your site offers personalization to end users, you are encouraged to enable BigPipe and report issues. There is zero risk of data loss. And when the environment — i.e. web server or (reverse) proxy — doesn’t support streaming, then BigPipe-delivered responses behave as if BigPipe was not installed. Nothing breaks, you just go back to the same perceived performance as before.)

About 500 sites are currently using the contrib module. With the release of Drupal 8.1, hopefully thousands of sites will test it.

Please report any issues you encounter! Hopefully there won’t be many. I’d be very grateful to hear about success stories too — feel free to share those as issues too!

Documentation

Of course, documentation is ready too:

What about the contrib module?

The BigPipe contrib module is still available for Drupal 8.0, and will remain available.

  • 1.0-beta1 was released on the same day as Drupal 8.0.0
  • 1.0-beta2 was released on the same day as Drupal 8.0.1, and made it feature-complete
  • 1.0-beta3 contained only improved documentation
  • 1.0-rc1 brought comprehensive test coverage, which was the last thing necessary for BigPipe to become a core-worthy module — the same day as the work continued on the core issue: https://www.drupal.org/node/2469431#comment-10899308
  • 1.0 was tagged today, on the same day as Drupal 8.1.0

Going forward, I’ll make sure to tag releases of the BigPipe contrib module matching Drupal 8.1 patch releases, if they contain BigPipe fixes/improvements. So, when Drupal 8.1.3 is released, BigPipe 1.3 for Drupal 8.0 will be released also. That makes it easy to keep things in sync.

Upgrading?

When you upgrade from Drupal 8.0 to Drupal 8.1, and you were using the BigPipe module on your 8.0 site, then follow the instructions in the 8.1.0 release notes:

If you previously installed the BigPipe contributed module, you must uninstall and remove it before upgrading from Drupal 8.0.x to 8.1.x.

Feb 16 2016
Feb 16

I spent about a week of my time at Acquia on improving Drupal 8’s REST support.

That time was spent fixing, reviewing, triaging and documenting.

Drupal 8’s REST works very well, we just have to make it more friendly & helpful, remove Drupalisms and support more REST capabilities.

Fixing, reviewing & triaging

I went through the entire issue queues of rest.module, serialization.module and hal.module. I was able to mark about a dozen bug reports as duplicates, fix a dozen or so support requests, have reviewed probably a few dozen patches, rerolled about a dozen patches, created at least another dozen patches and … triaged 100% of the open issues. I clarified titles of >30 issues.

Now the rest.module issue queue (the most important one) fits on a single page again! I collaborated a lot with neclimdul, klausi, damiankloip, dawehner and others.

dawehner and I decided to tag the issues that were especially relevant using the RX (REST Experience) issue tag.

I felt it was important to get a comprehensive picture of Drupal 8’s REST support state, so I insisted on going through all open issues (and was given the time to do so). This enabled me to document the current state of things (and upcoming improvements).

Documenting

So, I spent several days doing nothing else but writing and improving documentation, just like I did for the modules and subsystems I co-maintain. The following drupal.org handbook pages have either received minor updates, received complete overhauls or were written from scratch:

So, there you go, documentation covering fundamentals, like that for the Serialization API:

Serializing & deserializing Using the serializer service’s (\Symfony\Component\Serializer\SerializerInterface) serialize() and deserialize() methods:
$output = $serializer->serialize($entity, 'json');
$entity = $serializer->deserialize($output, \Drupal\node\Entity\Node::class, 'json');

Serialization format encoding/decoding (format → array → format The encoder (\Symfony\Component\Serializer\Encoder\EncoderInterface) and decoder (\Symfony\Component\Serializer\Encoder\DecoderInterface, to add support for encoding to new serialization formats (i.e. for reading data) and decoding from them (i.e. for writing data). Normalization (array → object → array) The normalizer (\Symfony\Component\Serializer\Normalizer\NormalizerInterface) and denormalizer (\Symfony\Component\Serializer\Normalizer\DenormalizerInterface), to add support for normalizing to a new normalization format. The default format is as close to a 1:1 mapping of the object data as possible, but other formats may want to omit e.g. local IDs (for example node IDs are local, UUIDs are global) or add additional metadata (such as URIs linking to related data). Entity resolvers In a Drupal context, usually it will be (content) entities that end up being serialized. When given an entity to normalize (object → array) and then encode (array → format), that entity may have references to other entities. Those references may use either UUIDs (\Drupal\serialization\EntityResolver\UuidResolver) or local IDs (\Drupal\serialization\EntityResolver\TargetIdResolver). For advanced use cases, additional mechanisms for referring to other entities may exist; in that case, you would add an additional entity resolver.

… to more practical information, such as the Getting started: REST configuration & REST request fundamentals handbook page for the rest module:

1. Configuration

First read RESTful Web Services API — Practical.

Now you know how to:

  1. expose data as REST resources
  2. grant the necessary permissions
  3. customize a REST resource’s formats (JSON, XML, HAL+JSON, CSV …)
  4. customize a REST resource’s authentication mechanisms (cookie, OAuth, OAuth 2.0 Token Bearer, HTTP Basic Authentication …)

Armed with that knowledge, you can configure a Drupal 8 site to expose data to precisely match your needs.

2. REST request fundamentals

2.1 Safe vs. unsafe methods

REST uses HTTP, and uses the HTTP verbs. The HTTP verbs (also called request methods) are: GET, HEAD, POST, PUT, DELETE, TRACE, OPTIONS, CONNECT and PATCH.
Some of these methods are safe: they are read-only. Hence they can never cause harm to the stored data, because they can’t manipulate them. The safe methods are HEAD, GET, OPTIONS and TRACE.
All other methods are unsafe, because they perform writes, and can hence manipulate stored data.

Note: PUT is not supported for good reasons.

2.2 Unsafe methods & CSRF protection: X-CSRF-Token request header

Drupal 8 protects its REST resources from CSRF attacks by requiring a X-CSRF-Token request header to be sent when using a non-safe method. So, when performing non-read-only requests, that token is required.
Such a token can be retrieved at /rest/session/token.

2.3 Format

When performing REST requests, you must inform Drupal about the serialization format you are using (even if only one is supported for a given REST resource). So:

  1. Always specify the ?_format query argument, e.g. http://example.com/node/1?_format=json.
  2. When sending a request body containing data in that format, specify the Content-Type request header. This is the case for POST and PATCH.

Note: Accept-header based content negotiation was removed from Drupal 8 because browsers and proxies had poor support for it.

3. Next

Now you’re ready to look at concrete examples, which start on the next page.

If that particular handbook page had already existed, it would have saved me so much time! The next page then contains examples for how to do GET requests, using various tools:

cURL

curl http://example.com/node/1?_format=hal_json

Guzzle

$response = \Drupal::httpClient()
  ->get('http://example.com/node/1?_format=hal_json', [
    'auth' => ['username', 'password'],
  ]);

$json_string = (string) $response->getBody();

jQuery

jQuery.ajax({
  url: 'http://example.com/node/1?_format=hal_json',
  method: 'GET',
  success: function (comment) {
    console.log(comment);
  }
});

… and the following pages then provide concrete examples (in those same tools) for POST, PATCH and DELETE requests.

Enjoy!

Why I did all of the above

It took me about three days to successfully PATCH a Comment entity.

Why days?

I first forgot to specify the Content-Type request header. Then it turned out I also forgot the X-CSRF-Token request header — which was not documented anywhere to be a thing. I eventually found out about that Drupal-specific request header by analyzing the REST PATCH test coverage. Why did I not find it sooner? Because Drupal 8 was giving utterly unhelpful, and actually downright nonsensical (and incorrect!) responses. It doesn’t end there though. Turns out that if you try to update an entity using JSON (and not HAL+JSON, which works fine), you MUST specify the bundle (otherwise it’s impossible to denormalize the entity you’re sending), but you also MUST NOT specify the entity type’s bundle if it’s a Comment (because you’re not allowed to modify this by CommentAccessControlHandler). So … it literally was impossible to update a comment!

I didn’t have any experience with/knowledge about Drupal 8’s REST API. But I’m deeply familiar with Drupal 8. And it still took me days. Of course I wanted to prevent anyone from ever having to go through that.

Today, anybody would start at d.o/documentation/modules/rest/start and then look at d.o/documentation/modules/rest/patch and would hence be able to avoid all these pitfalls. Soon, Drupal will provide more helpful responses and allow comments to be updated using JSON.

Jan 18 2016
Jan 18

This year, Performance Planet did an advent calendar again, just like the last few years. I also contributed an article — what follows is a verbatim copy of my “2013 is calling: are CMSes fast by default yet?” article for the 2015 Performance Calendar that was published exactly a month ago.

Two years ago, I wrote about how making the entire web fast is the real challenge. How only the big few companies are able to have very fast websites, because only they have the resources to optimize for performance. Many (if not most) organizations are happy just to have a functioning, decent looking site.

In that article, I said I thought that if we made the most popular CMSes faster by default. That would go a long way to make the majority of the web faster. Then less technical users do not need to know about the many arcane switches and knobs just to get the intended performance. No more “oh but you didn’t set it up correctly/optimally” — or at least far less of it.

Drupal 8: fast by default on many fronts

Since that article two years ago I’ve been working with >3300 other contributors to get the next version of Drupal done. Drupal 8 was released a month ago and is much faster by default.

All of the following changes improve perceived performance and Drupal 8 has all of them enabled by default:

… and for those last three fast defaults, each of which does caching: they don’t ever serve stale content.

Instantaneous, optimal invalidation thanks to cache tags. Lots of interesting possibilities with Drupal 8’s cacheability metadata.

Also, some CDNs already support cache tags. Which means instantaneous invalidation of content even when served from a CDN.

More fast defaults coming in minor Drupal 8 releases

We also paved the path for more optimizations to land in upcoming minor versions:

  • automatically use BigPipe for the dynamic/personalized/uncacheable parts of the page, have the initial part of the page sent immediately, thanks to the aforementioned partial page caching
  • JavaScript minification
  • responsive images: already supported but not enabled by default
  • HTTP/2 server push of assets: we already have a dependency graph of all assets — we’re mostly blocked on HTTP/2 servers actually supporting this.

There are of course many additional defaults that would help Drupal 8 be even faster by default, such as:

  • optimal image compression
  • load JavaScript asynchronously
  • automatic inlining of critical CSS

WordPress 4.4: responsive images by default

WordPress has made one significant change to improve performance: responsive images are enabled by default since the 4.4 release a week ago. Sadly, that’s the only significant WPO change in 2 years. (Six significant releases were made: versions 3.8 through 4.4.)

Joomla 3.x: status quo

Unfortunately, no significant steps forward have been made by Joomla to make it fast by default. 3.3.0 made the routing system faster and continued the conversion from MooTools to jQuery, which hopefully translates to less JavaScript being loaded. (Two significant releases were made: 3.3.0 and 3.4.0.)

Let’s…

So, Drupal has put a lot of effort into making version 8 fast by default. As Drupal 8 is rolled out, hundreds of thousands of sites will become faster.

Hopefully other CMSes (and frameworks) will follow.

Let’s make the entire web fast!

Nov 19 2015
Nov 19

I’m happy to announce that Fabian Franz and I managed to get a first release of BigPipe published today, coinciding with the Drupal 8.0.0 release!

Rather than explaining what it does, see for yourself:

[embedded content]

(That’s with 2 slow blocks that take 3 s to render. Only one is cacheable. Hence the page load takes ~6 s with cold caches, ~3 s with warm caches.)

Go download BigPipe 8.x-1.0-beta1!

Fastest Drupal yet!

After Drupal 8 already shipping with both the Page Cache and Dynamic Page Cache enabled by default earlier, this is the third and final step in our quest to make the entire web fast.

  • Fast anonymous user page loads: Page Cache — entire page is cached.
  • Fast authenticated user page loads: BigPipe — majority of page including main content is cached (thanks to Dynamic Page Cache) and sent first, the rest is rendered later and streamed.

Go and enjoy the fastest Drupal yet!

P.S.: none of this would have been possible without my employer Acquia, whom sponsored both my time and Fabian’s to make BigPipe a reality.

Nov 19 2015
Nov 19

Later today, Drupal 8 will be released! At this time, good docs are of course crucial.

As the maintainer and de facto co-maintainer of several Drupal 8 core modules and subsystems, I spent the last several days making sure that the documentation is up-to-date for:

  • the Text Editor module (editor)
  • the CKEditor module (ckeditor)
  • the Quick Edit module (quickedit)
  • the Filter module (filter)
  • the Cache system
  • the Render system (specifically the render caching part)
  • the Asset Library system

The following drupal.org handbook pages have either received minor updates, received complete overhauls or were written from scratch:

Enjoy!

P.S.: if you find anything unclear on those pages, ping me in #drupal-contribute — I want to make sure these docs are as clear and helpful as possible.

Oct 12 2015
Oct 12

Drupal 8 now has a Dynamic Page Cache. The Page Cache module only works for anonymous users, the Dynamic Page Cache module takes that a step further: it works for any user.

Since April 8, Drupal 8 had its Page Cache enabled by default. Exactly 5 months later, on September 8, the Dynamic Page Cache module was added to Drupal 8, and also enabled by default.

What?

The Page Cache module caches fully rendered HTML responses — it assumes only one variant of each response exists, which is only true for anonymous users. The innovation in 8 on top of 7’s Page Cache is the addition of cache tags, which allow one to use the Page Cache but still have instantaneous updates: no more stale content.

The Dynamic Page Cache module caches mostly rendered HTML responses — because it does not assume only a single variant exists: thanks to cache contexts, it knows the different variants that exist of each part of the page, and thus also of the final (fully assembled) page. During the rendering process, auto-placeholdering ensures that the parts of the page that are too dynamic to cache or are simply uncacheable are replaced with placeholders. These placeholders are only rendered at the very last moment. The Dynamic Page Cache module caches the page just before those placeholders are replaced.

So, conceptually speaking:

Page Cache Dynamic Page Cache
  • URL
  • cache contexts
↓ ↓
  • final response
  • cache tags
  • partial response
  • placeholders
  • cache tags

This table also illustrates quite clearly the strengths and weaknesses of both: Page Cache is faster because it contains the final response, unlike Dynamic Page Cache which only contains a partial response, but thanks to that response not yet being personalized it can be used across users.

In other words: Dynamic Page Cache saves us less work, but in doing so, allows us to apply it in many more cases.

Benchmark

Drupal 8’s Page Cache can respond in constant time. So, for anonymous users, Drupal 8 already responds in constant (amortized) time.

With Dynamic Page Cache, Drupal 8 is now able to respond in constant amortized time + the time it takes to render the placeholders. Which means that most parts of most pages can be cached in the constant part (and it doesn’t matter how much content is in there!). The parts of the page that end up as placeholders are the ones you want to optimize, and if possible, avoid. (Also see BigPipe further down.)

On my machine (ab -c1 -n 1000, PHP 5.5.11, Intel Core i7 2.8 GHz laptop, warm caches, 15 nodes with one comment each, 10 of which are listed on the frontpage):

  • Without Dynamic Page Cache
    • Front page: 61.3 ms/request (16 requests/s)
    • node/1: 92.3 ms/request (10.8 requests/s)
  • With Dynamic Page Cache
    • Front page: 38.3 ms/request (26 requests/s)
    • node/1: 73.8 ms/request (13.5 requests/s)

Histogram showing the before vs. after of the front page with Dynamic Page Cache.

Analysis

The front page only contains a single placeholder (for messages). This makes it is a strong indicator of the lowest response times to expect: roughly 38 ms on my machine.

  • ~38 ms goes to bootstrapping, routing, route access checking, and of course retrieving the cached response from the Dynamic Page Cache.
  • Compare that with the node/1 case: 35.5 more milliseconds are spent replacing placeholders there. Which means that we do some very expensive rendering there… and that is the comment form.

As you can see: Dynamic Page Cache actually becomes a helpful assistant while developing: it surfaces the impossible-to-cache-and-expensive-to-render parts of the page!

Note that we do have an issue to make Dynamic Page Cache run much earlier, see more about that below.

Easily tenfold that on actual servers

The above is tested with a concurrency of 1 client, to ignore web server performance (Apache in my case). See e.g. Rasmus Lerdorf’s profiling of the Page Cache.

Win-win

The real beauty is that it’s a win-win: enterprise (Acquia), medium, small, tiny (hobbyist) all win:

  • Enterprise sites get better tunability and reverse proxy/CDN-based hosting even for authenticated users
  • Tiny sites can handle more simultaneous authenticated users, and serve those faster

So my work was sponsored by Acquia, but benefits everyone!

People have been expressing concerns that Drupal 8 has become too complex, that it doesn’t care about site builders anymore, that it is only for enterprises, etc. I think this is a good counterexample — in fact an even stronger one than Page Cache being enabled by default: this was absolutely out of reach for the majority of Drupal 7 sites. Yet every Drupal 8 site will have this enabled by default.
Yes, we added the complexity of cacheability metadata, but that only affects developers — for whom we have good documentation. And most importantly: site builders reap the benefits: they don’t even have to think about this anymore. Manually clearing caches is a thing of the past starting with Drupal 8!

Drupal is very much like LEGO. Until Drupal 8, if you combined too many LEGO blocks, your Drupal site would become very slow. Setting up caching was very often prohibitively complex. Now the LEGO blocks must specify their cacheability metadata. Which allows Drupal to automate a huge portion of “making things fast”.

New possibilities for small sites (and shared hosting)

Smaller sites will be able to handle more authenticated users simultaneously. And have pages be much faster for those users.

Without any configuration.

Without any frustrating searching on the internet.

New possibilities for enterprise sites (and enterprise hosting)

On the other hand of the spectrum, enterprise hosting now gains the ability to tune: if they prefer to running caching infrastructure with enough storage capacity rather than rendering placeholders on the fly (i.e. more storage for less computation), then they can: just override the default configuration for auto-placeholdering conditions.

It also opens the door to — for the first time — caching responses for authenticated users in a CDN.

New possibilities for developers

The addition of cacheability metadata (cache tags, contexts and max-age), allow for greater insight and tooling when analyzing hosting, infrastructure, performance and caching problems. Previously, you had to analyze/debug a lot of code to figure out why something that was cached was not being invalidated when appropriate by said code.

Because it’s now all standardized, we can build better tools — we can even automatically detect likely problems: suspiciously many cache contexts, suspiciously low maximum ages, suspiciously frequent cache tag invalidations…

And finally, it makes it possible to do BigPipe-style rendering, making Drupal 8 the fastest Drupal yet:

[embedded content]

Dynamic Page Cache in 8 vs. Authcache in 7

Dynamic Page Cache’s closest comparison in Drupal 7 is the Authcache module. It’s possible to achieve great performance with Authcache in Drupal 7, but it requires a huge amount of work: all code (contrib & custom) must be analyzed to ensure all aspects that cause variations in the output (rendered pages) are taken into account. All of these factors must then be passed to Authcache so that it can calculate its “authcache key” correctly. The “key properties” that you specify there are conceptually similar to Drupal 8’s cache contexts. However, it is very easy to miss anything, because it’s you as a site owner that is responsible for identifying all the variations (and corresponding key properties) across all modules across all pages. By default, Authcache assumes the only thing that your site varies by, are a site’s base URL and a user’s roles. So unless your site is very simple (for example not even multilingual), you will need an enormous amount of knowledge/research in order to be able to use Authcache.

(For example, Drupal.org — which currently still runs on Drupal 7 — rolled out render caching of comments several months ago. It works very much like Authcache: you must specify the things that cause variations. One aspect (timezone) was forgotten, and it led to rather confusing broken output. If even Drupal.org’s very strong team can get it wrong, then what hope does a team with less expertise have?)

Dynamic Page Cache in 8 simply builds on the existing cacheability metadata concepts and infrastructure. There’s no need for custom hooks. There’s no possibility of something being forgotten as a site owner. It’s the individual modules’ responsibility now — and those modules know far better what they’re doing. The entire “analyze the entire codebase” phase is gone. And of course, Dynamic Page Cache also supports cache tags, so unlike Authcache in Drupal 7, responses cached in Dynamic Page Cache will actually be updated instantaneously.

In Drupal 8.0.0, Dynamic Page Cache runs after bootstrapping, routing and route access checking. Because Authcache makes certain simplifying assumptions, it can run before Drupal 7 has even fully bootstrapped, thus it is much faster most of the time. We do have an issue to make Dynamic Page Cache run much earlier, which would result in a great speed-up: #2541284 — we will aim to do that in Drupal 8.x.0.

Finally, because it is enabled by default in Drupal 8, Dynamic Page Cache has one enormous advantage: it brings this functionality to all Drupal sites and all Drupal developers. All Drupal developers are thus heavily encouraged to specify the appropriate cache contexts in their code. Which means that Drupal 8 contrib modules will be battle-hardened by many (tens of) thousands of websites, and missing cacheability metadata will be added. Instead of an enormous burden lying on the shoulders of the Authcache site owner, a very small burden lies on the shoulders of every contrib module maintainer (and user). Drupal 8 contrib modules should have integration test coverage with Dynamic Page Cache enabled.

Historical note

Somewhere in mid-February, Dries asked me what the state was of ESI support in Drupal 8, whether partial pre-rendering was possible, and so on. I replied in two ways: in short (No, ESI is not fully supported yet) and comprehensively, describing how full ESI support was now more feasible than ever, which other exciting things are possible with the render- and caching-related improvements, and so on. That e-mail I sent formed the foundation of Dries’ Making Drupal 8 fly blog post :)

But a side effect of me describing at a high level the exciting new possibilities got me thinking… I had spent a full year working on improving Drupal 8’s performance & cacheability. At that point, cache tags were ~90% done. Thanks to that, it had become clear that enabling page caching by default had finally become within reach (and it effectively happened two months later).

Cache contexts were also getting in better and better shape. What if… I would be able to leverage cache contexts in a similar way I was able to leverage cache tags? Complete cache tag support allowed us to enable page cache by default. What would complete cache context support allow? Cache tags capture data dependencies. Cache contexts capture (request) context dependencies. Cache tags allow for correct invalidations. Anonymous users do not get any variations, so cache tags are sufficient for them. Which means… that cache contexts would allow me to … do page caching, but for all users — i.e. also authenticated ones!

Or: how articulating an answer to a fairly specific question can make one think of other ideas :)

I started experimenting and was blown away by the results of my rough proof and concept patches. I opened the “SmartCache” ([#2429617]) issue and the “finalize cache contexts API & DX/usage, enable a leap forward in performance” ([#2429287]) issue that captured the many, many places where we’d still have to finish cache context support in order to be able to actually do what the experiment describes.

At the same time, I was talking to Fabian Franz. Together with him, I devised an algorithm (and a proof) to make it possible to bubble cache contexts, which is absolutely essential for Dynamic Page Cache.

Fast forward seven months and we’ve fixed many dozens of blockers for Dynamic Page Cache, and have landed Dynamic Page Cache itself, thanks to the hard work of many people!

Sep 23 2015
Sep 23

Together with Fabian Franz from Tag1 Consulting, I had a session about Big Pipe in Drupal 8, as well as related performance/cacheability improvements.

I’ll let the session description speak for itself:

With placeholders (https://www.drupal.org/node/2478483) having just gone into Drupal 8 Core, BigPipe being unblocked now and actively making its way in, Render Strategies around the corner, and out-of-the-box auth-caching in CDNs + Varnish a true possibility on the horizon, those are really exciting times for Drupal 8 Performance. But there is even more …

Come and join us for a wild ride into the depths of Render Caching and how it enables Drupal to be faster than ever.

The Masterplan of Drupal Performance (Next steps)

Here we will reveal the next steps of the TRUE MASTERPLAN of Drupal Performance. The plan we have secretly (not really!) been implementing for years and are now “sharing” finally with all of you! (Well you could look at the issue queue too or this public google doc, but this session will be more fun!)

Learn what we have in store for the future and what has changed since we last talked about this topic in Los Angeles and Amsterdam and why Drupal 8 will even be more awesome than what you have seen so far.

Also see a prototype of render_cache using the exact same Drupal 8 code within Drupal 7 and empowering you to do some of this in Drupal 7 as well.

Get the edge advantage of knowing more

Learn how to utilize cache contexts to vary the content of your site, cache tags to know perfectly when items are expired and cache keys to identify the objects - and what is the difference between them.

Learn how powerful ‘#lazy_builders’ will allow the perfect ESI caching you always wanted and how it will all be very transparent and how you can make your modules ready for the placeholder future.

See with your own eyes how you can utilize all of that functionality now on your Drupal 7 and 8 sites.

Get ready for a new area of performance

We will show you:

  • How to take advantage of #lazy_builders
  • How to tweak the auto-placeholdering strategies (depending on state of issue at time of session)
  • The biggest Do’s and Don’ts when creating render-cache enabled modules and sites
  • Common scenarios and how to solve them (mobile sites variation, cookie variation, etc.)
  • Drupal using an intelligent BigPipe approach (but a different one, one that is way more like Facebook does it …)

Get to know the presenters

This session will be presented by Wim Leers and Fabian Franz. Wim implemented a lot of what we show here in Drupal 8 and made the APIs easy and simple to use and made cache tags and #lazy_builders a very powerful concept. Fabian has prototyped a lot of this concepts in his render_cache module, introduced powerful Drupal 8 concepts into Drupal 7 and is always one step ahead in making the next big thing. Together they have set out on a crusade to rule the Drupal Performance world to bring you the fastest Drupal ever and with that trying to make the whole Web fast!

Frequently Asked Questions

  • I have already seen the session in Amsterdam and Los Angeles, will I learn something new?

Yes, absolutely. While the previous sessions focused more on the basics, this session will also cover how to use #lazy_builders and custom render strategies to empower your Drupal to be fast.

  • Will there again be a demo?

Yes, there will again be a nice demo :). You’ll love it!

  • Is it even possible to make it even faster than what we have seen?

Yes :)

Conference: 

DrupalCon Barcelona

Location: 

Barcelona

Date: 

Sep 23 2015 - 14:15

Duration: 

60 minutes

Sep 22 2015
Sep 22

Drupal 8 has comprehensive knowledge about the cacheability of the things it renders. This opens new doors. Did you know Drupal 8 will be able to cache everything at the edge?

For sites with many mobile users (high latency due to network), global audiences (high latency due to distance) and performance-sensitive sites (e-commerce), Drupal 8 will be a huge leap forward.

We’ll be showing how easy and powerful it is using the CloudFlare and Fastly CDNs.

Cache tags

Instantaneous purging of all (and only!) the affected pages when an article is updated. No more manual purging by content editors. No more fiddling with URLs to purge. It all just works. Optimally.

Cache anonymous pages without any effort. On your own reverse proxy, and on many CDNs — thanks to standardized configuration.

This sounds nice, but that’s just the anonymous user case. What about authenticated users?

Cache contexts

The typical example: a shopping site, users categorized in groups according to interests, and a shopping cart.

Automatic caching of the entire page, minus the shopping cart, on the edge. Reused across all users in the same group. And, if the CDN supports it, even the shopping cart can be cached on the edge (and be kept up-to-date thanks to cache tags). Otherwise only thatneeds to talk to the origin (via AJAX, for example).

Cache authenticated pages without any effort.  On your own reverse proxy, and on some CDNs — thanks to standardized configuration.

Goals

  • The caching concepts
  • Demos
  • BigPipe, ESI, hybrid rendering strategies
  • A peek at the future: ServiceWorker

Conference: 

DrupalCon Barcelona

Location: 

Barcelona

Date: 

Sep 22 2015 - 17:00

Duration: 

60 minutes

Jun 21 2015
Jun 21

While walking, I started listening to Jeff Eaton’s Insert Content Here podcast, episode 25: Noz Urbina Explains Adaptive Content. People must’ve looked strangely at me because I was smiling and nodding — still walking :) Thanks Jeff & Noz!

Jeff Eaton explained how the web world looks at and defines the term WYSIWYG. Turns out that in the semi-structured, non-web world that Noz comes from, WYSIWYG has a totally different interpretation. And they ended renaming it to what it really was: WYSIWOO.

Jeff also asked Noz what “adaptive content” is exactly. Adaptive content is a more specialized/advanced form of structured content, and in fact “structured content”, “intelligent content” and “adaptive content” form a hierarchy:

  • structured content
    • intelligent content
      • adaptive content

In other words, adaptive content is also intelligent and structured; intelligent content is also structured, but not all structured content is also intelligent or adaptive, nor is all intelligent content also adaptive.

Basically, intelligent content better captures the precise semantics (e.g. not a section, but a product description). Adaptive content is about using those semantics, plus additional metadata (“hints”) that content editors specify, to adapt the content to the context it is being viewed in. E.g. different messaging for authenticated versus anonymous users, or different nuances depending on how the visitor ended up on the current page (in other words: personalization).

Noz gave an excellent example of how adaptive content can be put to good use: he described how we he had arrived in Utrecht in the Netherlands after a long flight, “checked in” to Utrecht on Facebook, and then Facebook suggested to him 3 open restaurants, including cuisine type and walking distance relative to his current position. He felt like thanking Facebook for these ads — which obviously is a rare thing, to be grateful for ads!

Finally, a wonderful quote from Noz Urbina that captures the essence of content modeling:

How descriptive do we make it without making it restrictive?

If it isn’t clear by now — go listen to that podcast! It’s well worth the 38 minutes of listening. I only captured a few of the interesting points, to get more people interested and excited.

What about adaptive & intelligent content in Drupal 8?

First, see my closely related article Drupal 8: best authoring experience for structured content?.

Second, while listening, I thought of many ways that Drupal 8 is well-prepared for intelligent & adaptive content. (Drupal already does structured content by means of Field API and the HTML tag restrictions in the body field.) Implementing intelligent & adaptive will surely require experimentation, and different sites/use cases will prefer different solutions, but:

  • An intelligent_content module for Drupal 8: allow site builders/content strategists to define custom HTML tags (e.g. <product_description>) to capture site-specific semantics. A CKEditor Widget could hugely simplify the authoring experience for creating intelligent content, by showing a specific HTML representation while editing (WYSIWOO!), thanks to HTML (Twig) templates associated with those custom HTML tags.
  • An adaptive_content module for Drupal 8: a text filter that allows any tag to be wrapped in a <adaptive_content> tag, which specifies the context in which the wrapped content should be shown/hidden.
  • The latter leads to cacheability problems, because the same content may be rendered in a multitude of different ways, but thanks to cache contexts in Drupal 8 and the fact that text filters can specify cache contexts means adaptive content that is still cacheable is perfectly possible. (This is in fact exactly what it was intended for!) cache contexts

I think that those two modules would be very interesting, useful additions to the Drupal ecosystem. If you are working on this, please let me know — I would love to help!

May 13 2015
May 13

Update September 24, 2015: the fastest Drupal ever is no longer near, it is here!

Together with Fabian Franz from Tag1 Consulting, I had a session about Big Pipe in Drupal 8, as well as related performance/cacheability improvements. Fabian’s demo of BigPipe and other render strategies in the first ten minutes are especially worth watching :)

I’ll let Fabian’s session description speak for itself:

Come and join us for a wild ride into the depths of Render Caching and how it enables Drupal to be faster than ever.

The Masterplan of Drupal Performance

Here we will reveal the TRUE MASTERPLAN of Drupal Performance. The plan we have secretly (not really!) been implementing for years and are now “sharing” finally with all of you! (Well you could look at the issue queue too or this public google doc, but this session will be more fun!)

Learn what we have in store for the future and what has changed since we last talked about this topic in Amsterdam and why Drupal 8 will even be more awesome and why you don’t have to wait and can do it all in Drupal 7 right now with the help of the render_cache module (with some extra work).

Get the edge advantage of knowing more

Learn how to utilize cache contexts to vary the content of your site, cache tags to know perfectly when items are expired and cache keys to identify the objects - and what is the difference between them.

Learn how powerful ‘placeholders’ will allow the perfect ESI caching you always wanted and how it will all be very transparent and how you can make your modules ready for the placeholder future.

See with your own eyes how you can utilize all of that functionality now on your Drupal 7 and 8 sites.

Get ready for a new area of performance

We will show you:

  • The biggest Do’s and Don’ts when creating render-cache enabled modules and sites
  • Frontend performance pitfalls and why front-end performance is tied to backend performance more than you thought
  • Why libraries[] are so great and why single CSS/JS files make trouble.
  • Common scenarios and how to solve them (mobile sites variation, cookie variation, etc.)
  • Drupal using an intelligent BigPipe approach

Get to know the presenters

This session will be presented by Wim Leers and Fabian Franz. Wim implemented a lot of what we show here in Drupal 8 and made the APIs easy and simple to use and made cache tags and #post_render_cache a very powerful concept. Fabian has prototyped a lot of this concepts in his render_cache module, introduced powerful Drupal 8 concepts into Drupal 7 and is always one step ahead in making the next big thing. Together they have set out on a crusade to rule the Drupal Performance world to bring you the faster Drupal ever!

May 11 2015
May 11

Come and join us for a wild ride into the depths of Render Caching and how it enables Drupal to be faster than ever.

The Masterplan of Drupal Performance

Here we will reveal the TRUE MASTERPLAN of Drupal Performance. The plan we have secretly (not really!) been implementing for years and are now “sharing” finally with all of you! (Well you could look at the issue queue too or this public google doc, but this session will be more fun!)

Learn what we have in store for the future and what has changed since we last talked about this topic in Amsterdam and why Drupal 8 will even be more awesome and why you don’t have to wait and can do it all in Drupal 7 right now with the help of the render_cache module (with some extra work).

Get the edge advantage of knowing more

Learn how to utilize cache contexts to vary the content of your site, cache tags to know perfectly when items are expired and cache keys to identify the objects - and what is the difference between them.

Learn how powerful ‘placeholders’ will allow the perfect ESI caching you always wanted and how it will all be very transparent and how you can make your modules ready for the placeholder future.

See with your own eyes how you can utilize all of that functionality now on your Drupal 7 and 8 sites.

Get ready for a new area of performance

We will show you:

  • The biggest Do’s and Don’ts when creating render-cache enabled modules and sites
  • Frontend performance pitfalls and why front-end performance is tied to backend performance more than you thought
  • Why libraries[] are so great and why single CSS/JS files make trouble.
  • Common scenarios and how to solve them (mobile sites variation, cookie variation, etc.)
  • Drupal using an intelligent BigPipe approach

Get to know the presenters

This session will be presented by Wim Leers and Fabian Franz. Wim implemented a lot of what we show here in Drupal 8 and made the APIs easy and simple to use and made cache tags and #post_render_cache a very powerful concept. Fabian has prototyped a lot of this concepts in his render_cache module, introduced powerful Drupal 8 concepts into Drupal 7 and is always one step ahead in making the next big thing. Together they have set out on a crusade to rule the Drupal Performance world to bring you the faster Drupal ever!

May 08 2015
May 08

At least 20 people helped push one or more issues forward in Montpellier, at the Drupal Dev Days Performance Sprint!

Here’s an overview of what we set out to do, what we did, and what the next steps are.

The plan for DDD Montpellier

Drupal Dev Days Montpellier Performance sprint board — midweek The sprint board, midweek. Yellow stickies are performance stickies. Middle of the board is “in progress”. Bottom right is “fixed”.

1. Finding more performance issues

We already know that certain things are slow, and we know how to fix them. But significant portions of slowness do not yet have explanations for, let alone precise or even rough plans on how to reduce the slowness.
The parts of Drupal 8 that are slow that we do not have a strong grasp on yet are the bootstrap phase in general, but also routing, container services and route access checking.

Berdir, amateescu, dawehner, yched, znerol, pwolanin and I did a lot of profiling, testing hypotheses about why certain things took a given amount of time, comparing to Drupal 7 where possible, figuring out where the differences lied, and so on.
Bits and pieces of that profiling work are in https://www.drupal.org/node/2470679, including patches that help profile the routing system.

In the weeks since DDD Montpellier, effulgentsia and catch have continued discussions there, posted further analyses and filed many more issues about fixing individual issues.

2. Try to break Drupal 8’s internal page cache & fix it

So, Drupal 8 had the internal page cache enabled by default shortly before DDD Montpellier, with only a few known problems. Having many people to try to break it, using scenarios they use daily in their day jobs, that’s the ideal way to find any remaining cache invalidation problems.

Many tried, and most did not succeed in breaking it (yay! :)), but about half a dozen problems were discovered. See https://www.drupal.org/node/2467071.

What we got done

We fixed so incredibly many issues, and we had more than twenty people helping out! Notably, fgm was all over class loading-related issues, borisson_ and swentel got many of the page cache issues fixed, pwolanin pushed routing/REST/menu links issues forward significantly, and we overall simply made progress on many fronts simultaneously.

We made Drupal 8’s authenticated user page loads several percent faster in the course of that week!

Most of the page cache problems that were discovered (see above) were fixed right at the sprint! There are 4 known issues left, of which one is critical on its own, one is blocked, and the two others are very hard.

(If you want more details, we have day-by-day updates on what got done.)

Drupal Dev Days Montpellier Performance sprint board end result The sprint board, end result. Focused on the bottom right corner: look at all those yellow stickies!

Next steps

We currently have 11 remaining criticals with the “Performance” tag. Getting that to zero is our top priority. But many in that list are difficult.

If you specifically care about performance for authenticated users: less difficult issues can be found in the child issues of the Cache contexts meta issue. And for some least difficult issues, see the child issues of the SmartCache issue.

Generally speaking, all major issues tagged with either “Performance” or “D8 cacheability” can use a hand.

Hopefully see you in the queues! :)

Apr 09 2015
Apr 09

After more than a year and probably hundreds of patches, yesterday it finally happened! As of 13:11:56 CET, April 8, 2015, Drupal 8 officially has page caching enabled by default! And not the same page caching as in Drupal 7: this page cache is instantly updated when something is changed.

The hundreds of patches can be summarized very simply: cache tags, cache tags, cache tags. Slightly less simple: cacheability metadata is of vital importance in Drupal 8. Without it, we’d have to do the same as in Drupal 7: whenever content is created or a comment is posted, clear the entire page cache. Yes, that is as bad as it sounds! But without that metadata, it simply isn’t possible to do better.

I’ve been working on this near-full time since the end of 2013 thanks to Acquia, but obviously I didn’t do this alone — so enormous thanks to all of you who helped!

This is arguably the biggest step yet to make Drupal Fast By Default. I hate slow sites with a passion, so you can probably see why I personally see this as a big victory :)

(One could argue you could just enable Drupal 7’s page cache, but there are 3 reasons why this is inferior to Drupal 8’s page cache: a) no instantaneous updates; b) any node or comment posted causes the entire page cache to be cleared (!), c) it’s not enabled by default: many users don’t know they should enable this. Sure, you can get similar performance, but you’ll have to give up certain things, which makes it an apples vs. oranges comparison.)

Benchmark

By default, Drupal 8 is now between 2 and 200 times faster than Drupal 7 for anonymous users: Drupal 8 will respond in constant time, for Drupal 7 it depends on the complexity of the page.

On my machine (ab -c1 -n 1000, PHP 5.5.11, Intel Core i7 2.8 GHz laptop, warm caches):

Drupal 7

  • Front-page: 18.5 ms/request (55 requests/s)
  • node/1: 23.5 ms/request (43 requests/s)
  • More complex pages: easily hundreds of milliseconds, only few requests per second.

Drupal 8

Always 6.5 ms/request (154 requests/s).

Easily tenfold that on actual servers

The above is tested with a concurrency of 1 client, to ignore web server performance (Apache in my case).

Rasmus Lerdorf — creator of the PHP language — also did some benchmarking, and he found that Drupal 8 was able to serve >2500 requests/second on his server, with 20 concurrent clients.

Win-win

The real beauty is that it’s a win-win: enterprise (Acquia), medium, small, tiny (hobbyist) all win:

  • Enterprise sites get very nice reverse proxy/CDN-based hosting
  • Tiny sites can easily serve 100 requests/second (>8 million requests/day) on shared hosting.

So my work was sponsored by Acquia, but benefits everyone!

People have been expressing concerns that Drupal 8 has become too complex, that it doesn’t care about site builders anymore, that it is only for enterprises, etc. I think this is a good counterexample.
Yes, we added the complexity of cacheability metadata, but that only affects developers — for whom we have good documentation. And most importantly: site builders reap the benefits: they don’t even have to think about this anymore. Manually clearing caches is a thing of the past starting with Drupal 8!

Page cache is just a built-in reverse proxy

Drupal’s page cache is just a built-in reverse proxy. It’s basically “poormansvarnish”.

Drupal 8 bubbles all cacheability metadata up along the render tree, just like JavaScript events bubble up along the DOM tree. When it reaches the tree’s root, it also bubbles up to the response level, in the form of the X-Drupal-Cache-Tags header.

The page cache uses that header to know what cache tags it should be invalidated by. And because of that, other (“real”) reverse proxies can do exactly the same. The company behind Varnish even blogged about it. And CDNs are even starting to support this exact technique out of the box, for example Fastly.

Last but not least: all of Drupal 8’s integration tests use the page cache by default, which means all of our integration tests effectively verify that Drupal works correctly even if they’re behind a reverse proxy!

New possibilities for small sites (and shared hosting)

On one end of the spectrum, I see great shared hosting providers starting to offer Varnish even on their smallest plans. For example: Gandi offers Varnish on their €4/month plans. If users can configure Varnish — or even better, if they pre-configure Varnish to support Drupal 8’s cache tag-based invalidation — then almost all traffic will be handled by Varnish. (Update: see the official docs on how to use Varnish + cache tags.)

For 90% or more of all sites, this would quite simply be good enough: very cheap, very fast, very flexible.

I can’t wait until we see the first hosting provider offering such awesome integration out of the box!

New possibilities for enterprise sites (and enterprise hosting)

On the other hand of the spectrum, enterprise hosting now gains the ability to invalidate (purge) all and only the affected pages on a CDN. Without having to generate a list of URLs that a modified piece of content may appear on, and then purge those URLs. Without having to write lots of hooks to catch all the cases where said content is being modified.

At least equally important: it finally allows for caching content that previously was generated dynamically for every request, because it was a strong requirement that the information always be up-to-date. With cache tag support, and strong guarantees that cache tags indeed are invalidated when necessary, such use cases now can cache the content and still be confident that updates will immediately propagate.

New possibilities for developers

Finally, the addition of cache tags and by extension, all render cacheability metadata (cache tags, contexts and max-age), allow for greater insight and tooling when analyzing hosting, infrastructure, performance and caching problems. Previously, you had to analyze/debug a lot of code to figure out why something that was cached was not being invalidated when appropriate by said code.

Because it’s now all standardized, we can build better tools — we can even automatically detect likely problems: suspiciously frequent cache tag invalidations, suspiciously many cache tags … (but also cache contexts that cause too many variations, too low or too high maximum ages …).

Next steps

Warm cache performance is now excellent, but only for anonymous users.

Next week, at Drupal Dev Days Montpellier, we’ll be working on improving Drupal 8’s cold cache performance (including bootstrap and routing performance). That will also help improve performance for authenticated users.

But we already have been working several weeks on improving performance for authenticated users. Together with the above, we should be able to outperform Drupal 7. This is the plan that Fabian Franz and I have been working towards:

  1. smartly caching partial pages for all users (including authenticated users): d.o/node/2429617, which requires cache contexts to be correct
  2. sending the dynamic, uncacheable parts of the page via a BigPipe-like mechanism: d.o/node/2429287
Apr 07 2015
Apr 07

I’m working on making Drupal 8 faster as part of my job at Acquia. The focus has been on render caching, which implies that cacheability metadata is of vital importance in Drupal 8.

To be able to render cache all things that can possibly be render cached, Drupal 8 code must:

  • set the right cache max-age — to ensure only the cacheable parts of the page are cached
  • set the right cache contexts — to ensure content is varied as expected (per language, per role, per timezone, per user …)
  • set the right cache tags — to ensure rendered content is invalidated when the data it depends on is modified

Before Drupal 8, approximately zero attention was given to cacheability of the rendered content: everything seen on a Drupal 7 page is rendered dynamically, with only the occasional exception.

By flipping that around, we make developers more conscious about the output they’re generating, and how much time it takes to generate that output. This in turn allows Drupal 8 to automatically apply powerful performance optimizations, such as:

  1. enabling Drupal’s internal page cache (for anonymous users) by default: d.o/node/606840, which requires cache tags to be correct
  2. smartly caching partial pages for all users (including authenticated users): d.o/node/2429617, which requires cache contexts to be correct
  3. sending the dynamic, uncacheable parts of the page via a BigPipe-like mechanism: d.o/node/2429287

(The first of those three will likely happen this week. We’re working hard to make the last two a reality.)

Visualization

Caching means better performance, but it also means that without the correct cacheability metadata, the wrong content may be served to end users: without the right cache contexts, the wrong variation may be sent to a user; without the right cache tags, stale content may be sent to a user. Therefore we should make it as easy as possible to analyze the cacheability of a rendered block, entity (node/user/taxonomy term/…), view, region, menu, and so on.

It should work not only for cacheability metadata, but for all bubbleable metadata: it’d be very valuable to be able to see which part of the page caused an expensive cache context or tag , but it’d be at least equally valuable to see which part of the page attached a certain asset.

Since bubbling happens across a tree, it’s important to visualize the hierarchy. The best hierarchy visualization I know in the web developer world is the Firefox Developer Tools 3D view.

Firefox 3D view of wimleers.com

I think a tool for visualizing, analyzing and understanding the bubbleable metadata (cache contexts, cache tags, cache max-age, assets) should work in a similar way. The developer should be able to:

  • look at the document in 3D in different layers and/or queries (assets, cacheability as a whole, but also only cache contexts, only cache tags, or only max-age)
  • zoom in a specific element, and look at all of its bubbleable metadata
  • reposition the 3D view, to look from different angles — humans are very proficient at processing visual data

Prototype

So, over the past weekend, I worked on a prototype. I read the CSS Transforms spec and read a CSS 3D transforms introduction. As somebody with little CSS knowledge and not having touched CSS nor 3D programming in years, it was fun to play with this :) The result:

renderviz prototype, visualizing the 'timezone' cache context.

And finally, a short screencast demonstrating it in action:

Give it a try yourself by applying the attached patch to Drupal 8 at commit daf9e2c509149441d4d9a4d1964895179a84a12c and installing the renderviz module.

Want to help?

There are many rough edges — actually there are only rough edges. Everything needs work: CSS, JavaScript, UI (there isn’t any yet!), even the name!

But it’s a lot of fun to work on, and it’s very different from what most of us tend to work on every day. If you’d like to be able to build sites in Drupal 8 with a developer tool like this, please contact me, or leave a comment :)

Nov 06 2014
Nov 06

In Drupal 8, we’ve significantly improved the way pages are rendered. This talk explains the entire render pipeline, in some detail. But it also covers:

  • render caching — blocks and entities are now render cached automatically!
  • cache tags — finally we have the cache invalidation system we’ve always needed!
  • assets — only the necessary assets are loaded anymore, thanks to asset dependencies!
  • bubbling — rather than relying on global statics that broke caching, we now correctly bubble up all attached metadata — no more frustrations!

Besides that, I also cover a few of the most interesting new possibilities in Drupal 8:

  • anonymous page loads: invalidating Varnish/CDNs with perfect precision
  • authenticated page loads: not completely regenerated on every page load, but assembled from render cached parts
  • alternative render strategies, like Big Pipe

Update (November 14, 2014): since I gave this talk, https://www.drupal.org/node/2352155 was committed, so this talk is now indeed a comprehensive, correct talk about the finalized Drupal 8 render pipeline!

Update (March 20, 2015): there is now an official Drupal 8 documentation page: https://www.drupal.org/developing/api/8/render/pipeline.

Conference: 

DrupalCamp Ghent

Location: 

Ghent, Belgium

Date: 

Nov 7 2014 - 09:30

Duration: 

45 minutes

Pages

About Drupal Sun

Drupal Sun is an Evolving Web project. It allows you to:

  • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
  • Facet based on tags, author, or feed
  • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
  • View the entire article text inline, or in the context of the site where it was created

See the blog post at Evolving Web

Evolving Web