Mar 25 2020
Mar 25

Laravel is a PHP framework that has witnessed comparatively less attention than CMS heavyweights like Drupal and WordPress and frameworks like Symfony, but it nonetheless offers several compelling advantages for enterprise website implementations. Recently, a large organization worked with Tag1 to move legacy databases into a more modern approach that, coupled with Laravel and Vue.js, led to considerable improvements in not only developer experience but also user experience.

Moreover, Laravel is an excellent tool for architecting custom applications with complex logic, all without any complexity offloaded onto the developer experience. Thanks to the Laravel ecosystem, you can have a working PHP application with APIs and a functional single-page JavaScript application in Vue.js with minimal overhead. Instead of having to define low-level properties like in Symfony or deal with too many premade assumptions like in Drupal or WordPress, you can focus on user workflows in Laravel and benefit from a more reasonable learning curve as well.

In this Tag1 Team Talks episode, László Horváth (Senior Laravel Developer at Tag1) joins guests Fabian Franz (Senior Technical Architect and Performance Lead at Tag1), Michael Meyers (Managing Director at Tag1), and your host Preston So (Editor in Chief at Tag1 and Senior Director, Product Strategy at Oracle) for a deep dive into why Laravel should be your choice when building a mission-critical PHP architecture that encompasses a decoupled front end in JavaScript and multi-level access control and permissioning.

[embedded content]


Mar 18 2020
Mar 18

Table of Contents

Claro's bright future
--Rethinking the layout of administration pages
--Simplifying complex user interfaces
--A dream state for Claro?
--Why Drupal is a perfect place for UI development
Conclusion: How to get involved

Drupal has seen a range of administrative experiences over its history, starting with early Drupal mainstays like the Garland theme and Seven, perhaps the most recognizable iteration of Drupal's editorial experience in its history. Recently, another administration theme joined the pantheon of administration themes in Drupal core that have had an outsized influence on how users interact with Drupal not only for the first time but for countless times afterwards. The Claro administration theme is a new theme for Drupal 8, now available in Drupal core, which offers a more modern user experience thanks to its emphasis on usability and accessibility during the design and development process.

A short time ago, Cristina Chumillas (Claro maintainer and Front-End Developer at Lullabot), Fabian Franz (Senior Technical Architect and Performance Lead at Tag1) and Michael Meyers (Managing Director at Tag1) joined your correspondent (Preston So, Editor in Chief at Tag1 and author of Decoupled Drupal in Practice) for a wide-ranging conversation about why Claro is such a compelling new addition to Drupal's editorial experience and how Claro focused on enabling accessibility and user experience outcomes that would permit a narrowly scoped development cycle while also yielding the potential for a variety of other initiatives to extend Claro's design to encompass other use cases. In this multi-part blog series, we explore some of what we learned during this Tag1 Team Talks episode. In this fourth and final installment, we unlock some of the features ahead in Claro's roadmap and share how you too can contribute to the exciting future ahead of the Claro theme.

Claro's bright future

Before we proceed, if you have not yet taken a look at the first, second, and third installments of this blog series on Claro, I highly recommend you do so to better understand some of the context behind the history of Claro, the motivations behind a new administration theme, some of the data in the Drupal community that lent urgency to certain features, and how Claro's design took shape. In this upcoming section, we discuss some of the improvements and future roadmap items that Claro could see in the coming few years.

Rethinking the layout of administration pages

Claro: Drupal Node Add During our Tag1 Team Talks episode, we asked Cristina to share what the most significant improvements are that she would like to see in Claro's future. Cristina responded by citing the layouts of administration pages, namely the forms and other user interface components that constitute much of the editorial experience in Drupal. After all, she asks, "who hasn't wanted to create a node edit form that places something on the right sidebar?" This question is pressing not solely because of the possibilities such layout manipulation would unlock but also due to the fact that developer-friendly administration themes are typically the only way for Drupal users to heavily influence the layout of administration pages in Drupal.

Whereas in other content management systems this requirement might be considered superfluous, in Drupal it is a basic need, as this functionality has been available for quite some time to developers. Cristina also envisions an experience that would allow even greater flexibility by offering users the flexibility to determine a particular layout depending on the content on which an editor is currently working. Due to Drupal's high degree of customization when it comes to content types and their fields, node edit forms could feasibly have anywhere from two to forty fields that, especially in the latter case, test the limits of what is possible within an administration theme's design. In many scenarios, the person who decides how the content model should appear is the site builder, rather than the Drupal community enforcing a rigid approach to content modeling.

Simplifying complex user interfaces

Claro: Drupal Permissions Revamping the appearance of certain user interfaces in Drupal that have reputations for being overly complex is also a key roadmap item for the Claro theme. Views UI, for instance, is regularly cited as one of the most challenging and incomprehensible user interfaces in Drupal despite its powerful strengths. Other pages, such as the modules list page and the user roles and permissions page, are robust for a small number of modules or user roles but quickly become noticeably unwieldy as soon as the list reaches dozens or hundreds of items. Though there was an attempt to redesign these pages in the past, much of the work did not move forward due to the monumental amount of work involved.

Cristina also discusses the prospect of many of these complicated user interfaces becoming initiatives in their own right, and she recommends that motivated contributors add to the Ideas queue to make them a reality and find other interested parties. The first step, in Cristina's view, is to examine the user interface's requirements and determine from there the next best steps to realize ambitious experiences that include JavaScript-driven interactivity and modern design patterns.

A dream state for Claro?

Claro: Extend One of the questions we posed to Cristina during our Tag1 Team Talks episode had to do with what her ideal vision for Claro would look like — what is the dream state for Claro? Cristina responded by recommending a distinction be made between Claro and the concept of modernizing the administrative interface, as Claro has more of a focus on the aesthetic renewal of the administrative interface rather than enabling snazzy features like JavaScript-driven interactivity and animations. Now, concludes Cristina, "we can focus on the rest of the things we'll need."

To make Claro stable for the next release of Drupal, a substantial amount of work remains, but beyond that milestone, the Claro team is interested in pursuing an exploration of how Claro's components could work in complex environments like Views UI and other similar interfaces on Drupal's docket. Stress-testing Claro's components would also enable those components to target more of the foundational cases that user interface developers need, in addition to underpinning initial development of redesigned interfaces. Cristina's hope is that many in the Drupal community view Claro's components as a robust foundation on which diverse user interfaces can be implemented.

Why Drupal is a perfect place for UI development

Cristina argues during our Tag1 Team Talks episode that "Drupal is the perfect place to do lots of things from the UI perspective." And the vision she shared with us in the previous section is only one step on the way to an ambitious future state for Drupal. Cristina shares that she wants to see not only two- and three-column layouts but also JavaScript-driven interactions that are commonplace across the web these days.

WordPress's Gutenberg project has served as a valuable source of inspiration for Claro's architects, but Cristina acknowledges that many of Gutenberg's goals are not viable in the Drupal context due to Drupal's emphasis on modular and structured content that does not match Gutenberg's more full-page approach. For example, Cristina argues that having a drag-and-drop Gutenberg-like user interface on a content entity in Drupal with only five fields could be overkill for the node's requirements. Nonetheless, one of the most compelling next steps for Claro and other front-end modernization efforts in Drupal could be to facilitate several different ways of editing content based on its characteristics.

In the end, Drupal's focus on experimentation is part of what makes the community and ecosystem so rich. Experimenting with new user interfaces and new ways to interact with content is key to the success of any application, and Drupal's administrative interface is no exception. By shedding some of the anachronistic approaches implemented by the Seven theme, Claro represents an exciting next stage in Drupal's front end and user experience. Cristina concludes by restating the primary purpose of Drupal's administration layer: not to be a fancy front end but to provide the most robust solutions possible to manage content in an efficient way.

Conclusion: How to get involved

The Claro administration theme is an exciting new development for Drupal core, and I'm excited to see how users respond across a variety of use cases and personas to the biggest update to Drupal's administrative interface in quite some time. But best of all, you too can get involved in Claro's development and propose and contribute your own ideas. All conversations about Claro take place in Drupal's Slack organization. Design considerations are discussed in the #admin-ui-design channel, whereas the larger-scale work around revamping the administrative interface is centered around the #admin-ui channel. For discussions about JavaScript and its role in driving richer interactions, the #javascript channel has seen discussion around this topic. These channels are the most direct way available to communicate with other contributors online.

As for Drupal.org, Claro has its own component in Drupal core, searchable in issues on Drupal.org and assignable to issues that you create. In addition, the initiative's coordinators are not only active and prolific but also friendly and approachable. Lauri Eskola (lauriii) and Cristina attend Drupal events regularly and also are available at many times on Drupal's communication channels. As for me, I plan to apply the Claro administration theme to the editorial interfaces on my own personal site. Thanks to the Claro theme, Drupal enters a new decade with a modern look and feel and a carefully considered approach that evokes the best of open-source communities: deep collaboration and thoughtful innovation.

Special thanks to Cristina Chumillas, Fabian Franz, and Michael Meyers for their feedback during the writing process.

Photo by Rachel Ren on Unsplash

Mar 17 2020
Mar 17

Table of Contents

Shared editing in Drupal?
     How collaborative editing could become a reality in Drupal
     Yjs across ecosystems
Challenges of building collaborative editing
Other alternatives for shared editing
Conclusion

One of the seemingly intractable difficulties in content management systems is the notion of supporting collaborative editing in a peer-to-peer fashion. Indeed, from an infrastructural standpoint, enabling shared editing in a context where server-side CMSs rule the day can be particularly challenging. All of this may soon change, however, with the combination of Yjs, an open-source real-time collaboration framework, and Gutenberg, the new editor for WordPress. With the potential future outlined by Yjs and collaborative editing in WordPress, we can open the door to other thrilling potentialities such as shared layout building or even collaborative drawing in the CMS context.

On an episode of the Tag1 Team Talks show, your correspondent (Preston So, Editor in Chief at Tag1 and author of Decoupled Drupal in Practice) chatted with Kevin Jahns (creator of Yjs and Real-Time Collaboration Systems Lead at Tag1) and Michael Meyers (Managing Director at Tag1) about how WordPress is enabling shared editing in Gutenberg. In this multi-part blog series, we summarize some of the concepts we discussed. In this third installment of the series, we examine how shared editing could become a reality in Drupal and the unique challenges of implementing shared editing in the CMS context.

Shared editing in Drupal?

If you haven’t already, I recommend checking out the first and second installments of this blog series on shared editing with Gutenberg, because those blog posts outline some of the motivations for collaborative content creation in CMSs and some of the difficulties the intiative faced in supporting shared editing. In addition, over the course of those blog posts we inspected how key features like peer-to-peer collaboration and offline editing need to be part and parcel of any out-of-the-box implementation of collaborative editing in content management systems.

How collaborative editing could become a reality in Drupal

Drupal may have a similar background to WordPress’s, but its shared editing story is comparatively immature. Nevertheless, there are contributed working implementations of shared editing in Drupal, including ProseMirror, a shared editing tool that we covered in a previous Tag1 Team Talks episode. Tag1 Consulting recently evaluated ProseMirror along with a host of other solutions that would enable such use cases in Drupal, including a complete integration with Drupal’s roles and permissions system.

Tag1 recently produced an implementation of shared editing in Drupal for a top Fortune 50 client that leverages a client–server model without peer-to-peer capabilities. In addition, this Tag1 client needed to edit code collaboratively in addition to content, a use case enabled by CodeMirror, complete with code hints that provide next steps for editors. Michael cites this as yet another fascinating example of how Yjs can enable effective collaborative editing in a variety of settings. Fortunately, even Drupal sites are now implementing shared editing across the ecosystem, lending even more weight to the notion of collaborative content creation eventually becoming table stakes in content management.

Yjs across ecosystems

Many existing editors such as ProseMirror now integrate gracefully with Yjs. Fortunately, the y-webrtc plugin in the Yjs ecosystem is a relatively lightweight and straightforward add-on that could be part of the Drupal and WordPress ecosystems in the form of plugins as well. Once communication can be successfully established between peers with these plugins, we can enable collaboration on any shared editor that Yjs supports, including ProseMirror, Quill, and others. This idea will doubtlessly be interesting to people who use different content editors, because this feature can be easily enabled in the CMS.

Challenges of building collaborative editing

As Kevin attests from experience, implementing shared editing is particularly challenging. Someone who builds collaboration with JavaScript may require a different server algorithm in JavaScript to handle communication between client and server. Consider, for instance, the case of implementing text collaboration that leverages operational transformation (OT), a topic covered in a previous Tag1 Team Talks episode. OT is a particularly challenging algorithm to implement, and this same algorithm needs to be made available in the client-side context on rich text editors as well, which is a difficult proposition. In other words, it becomes double the work to support OT in any universal manner.

Most of the time, especially given the advent in recent years of universal JavaScript (also known as isomorphic JavaScript), architects would replicate the same client-side implementation of OT on a server environment. This sharing of code makes testing easier, because you can reuse the same libraries. You can ensure that the encoding of operations is precisely the same as what you would use in the server context. There are a host of issues with any universal approach, and WebSockets have not historically been well-supported in well-worn content management systems like Drupal and WordPress.

Other alternatives for shared editing

One of the important considerations in Gutenberg’s implementation of collaborative editing was determining whether there are any other compelling alternative solutions available. Kevin states during our Tag1 Team Talks episode that CKEditor 5 is an exceptionally well-built tool for editing and seamlessly communicates with CKEditor’s back-end solution, CKSource, to make any content become collaborative. CKEditor, in fact, leveraged the ideas behind collaborative editing much earlier, albeit with a centralized server approach. Unfortunately, however, one of the most significant issues inherent to CKEditor is that you can only use their proprietary CKSource server environment in order to enable shared editing. Nevertheless, CKSource washes away many of the concerns around shared editing such as scalability.

Worse still, CKEditor and CKSource are not viable solutions for content management systems like Drupal and WordPress, because every Drupal or WordPress implementation would need to set up CKSource and pay for a subscription. Kevin also highlights the issue of data ownership and asks whether collaborators can truly own their content if it is always sent to a separate central server. If the server goes offline and the content is lost, there is no way to restore the content, particularly from a local version.

As an example, many companies employ intranets to host content. Consider, for example, a company that routinely works with important government documents and secret information, all of which is kept secure through various means. Now, if you want this content to be collaboratively edited, but the content is comprised of highly privileged information, you need to host it either on a third-party server or pay a platform to host it on-premise. This makes maintainability, security, and software upgrades a potentially intractable challenge to solve.

Michael also cites the fact that CKSource was implemented first and foremost for CKEditor, and without CKEditor as a text editor available by default in a content management system, like it is in Drupal, there is no way to couple shared editing with CKSource for these open-source projects. Because Yjs is not only network-agnostic but also editor-agnostic, the same technology can enable the transformation of any aspect of your application into something collaborative. Whereas Yjs supports collaborative use cases beyond text such as layout building and drafting, CKSource makes this much more difficult.

Nonetheless, Kevin states during our show that CKSource and CKEditor, together, have led the market and offer the most robust end-to-end solution for collaborative text editing specific to those systems leveraging CKEditor. This exemplifies one of the key advantages of Yjs; not only can Yjs enable collaboration beyond text with use cases like drawing or even three-dimensional modeling, its agnosticism also means that integrations with a rich variety of editors is not only possible but easy to implement.

Conclusion

The concept of collaborative editing is nothing new to content editors around the world, but it is certainly a new idea for content management systems like Drupal and WordPress, which have never had such a feature available in the form of shared editing. Nonetheless, the motivations are clear, particularly in this modern age when offline editing and decentralized applications are in vogue. Thanks to the open-source real-time collaboration framework Yjs, however, Drupal and WordPress may soon have new approaches available to support shared editing out of the box. With the help of Yjs’ creator Kevin Jahns, Tag1 has enabled collaborative editing to become a reality in Gutenberg, one of the most important projects ever to emerge on the WordPress landscape.

In this third installment of this multi-part blog series on shared editing in Gutenberg, we examined some of the challenges that face a similar approach in Drupal. Nonetheless, Drupal and WordPress, along with Yjs, both benefit from rich ecosystems that could enable the sort of exciting innovation that comes with the prospect of shared editing. In the next and final installment of this series, we discuss the future of collaborative editing in Gutenberg, the reaction of the Gutenberg team, and how you yourself can get involved.

Special thanks to Kevin Jahns and Michael Meyers for their feedback during the writing process.

Photo by CoWomen on Unsplash

Mar 11 2020
Mar 11

Table of Contents

Accessibility in Claro
--Color
--Size and spacing
--Enabling better accessibility reviews
Scope
What makes Claro unique
--PostCSS
Conclusion

Throughout Drupal's existence, no other changes have made as much an impression on users as refreshes of the administrative interface in Drupal. Content editors and site builders have a long list of expectations when it comes to the editorial experience they wish to use, and surveys and tests had shown that Seven was showing its age. Now, thanks to Claro, the new administration theme for Drupal 8, user interfaces for all editors in Drupal are now optimized for accessibility and usability in addition to a realistic roadmap for the future.

Recently, Cristina Chumillas (Claro maintainer and Front-End Developer at Lullabot), Fabian Franz (Senior Technical Architect and Performance Lead at Tag1) and Michael Meyers (Managing Director at Tag1) sat down with me (Preston So, Editor in Chief at Tag1 and author of Decoupled Drupal in Practice) for a Tag1 Team Talks episode about the Claro administration theme and its bright future. In this multi-part blog series, we track the journey of Claro from beginnings to its current state. In this third installment, we uncover some of the ways in which Claro has improved outcomes for accessibility and stayed innovative in a fast-changing front-end landscape.

Accessibility in Claro

If you haven't yet read the first and second installments of this blog series, it's a good idea to go back to better understand where Claro came from and how it came to be. In this section, we examine some of the ways in which Claro prizes accessibility and ensures that Drupal users with disabilities can easily operate Drupal.

Color

Claro: Drupal Core Color Module During our conversation, Fabian asked Cristina: "Will there be a dark mode (in Claro)?" Though a dark mode toggle isn't part of the minimum viable product (MVP) for Claro, Cristina agrees that it could be a valuable addition to the theme, initially as part of the contributed theme ecosystem and then eventually part of core. The notion of a dark mode brings up one of the central considerations for accessibility: the use of color.

Currently, an issue on the Drupal Ideas queue proposes removing the Color module from Drupal core, a module that has been a mainstay of Drupal's available modules list for many years and allows the customization of Drupal themes through color variables. The discussion sparked a wide-ranging discussion about people wishing to customize the colors for administration interfaces for each customer. Though this is undeniably an interesting feature, the issue with offering color customization is that it grants far too much control to users who may configure color combinations that are inaccessible.

Size and spacing

Another piece of feedback that Claro received concerned what some perceived as too much additional space between form elements. Commenters wanted not only to see the elements displayed in a more compact manner but also to be able to customize the extent to which elements were spaced from one another, in addition to font sizes across the theme.

All of this has raised the important question of user customization of elements like size, spacing, and color and whether this sort of customization should be done at the theme level instead of in the user interface. Accessibility, after all, is hampered if a user sets a color scheme with insufficient contrast for certain users. One thing that Claro also implemented in Drupal's administrative interface was an increase in the font size across the board, as smaller font sizes are less accessible.

Enabling better accessibility reviews

A few months into development, the Claro team worked with Andrew McPherson, one of Drupal's accessibility maintainers, to review the designs for the administration theme and found important changes were necessary for elements like the aforementioned text field. An important discovery that the Claro team made was that providing a PNG or PDF file to a design reviewer and accessibility reviewer was less useful when the design isn't fully implemented in order to allow interaction testing as well.

Scope

Another key finding from Claro's development revolves around scope. In open source, of course, one of the fundamental constraints on project scope is always the number of contributor resources available to implement a given design. Having a tight release date also encourages a tighter scope, and Claro's team decided that having Claro ready for Drupal 8.8 was of paramount importance as otherwise Claro would possibly not have been eligible to become a stable theme in Drupal 8.8 or Drupal 8.9.

In the end, the team created a list of components that would be necessary to migrate from one theme to another, and though some of these continue to rely on Seven styles, they remain usable in Claro. By drawing a line in the sand and focusing on a revamp of the design first and foremost, without the risk incurred by engaging overly ambitious ideas like wildly new interfaces and layouts, Claro successfully entered Drupal core thanks in part to the constraints imposed by a narrow scope.

What makes Claro unique

Claro has been pushing the envelope when it comes to Drupal administration themes. Many common patterns now found on the web today are employed in Claro in lieu of the more "classic" components that Seven provided. Cristina says that Claro was "a chance to modernize the interface and get in a lot of patterns common around the web that make lives easier for users and I would say also for front-end developers."

PostCSS

Claro: Drupal Core PostCSS One of the key ways in which Claro demonstrates a high degree of innovation with regard to front-end development is the implementation of PostCSS for the administration theme. The Claro team decided explicitly not to utilize Sass as a Cascading Style Sheets (CSS) preprocessor, as is common in other Drupal themes. Instead, Claro uses PostCSS, a postprocessor for CSS, to provide features such as cross-browser support for CSS variables. Because Claro leverages PostCSS to process CSS written normally, it will remain compatible well into the future.

Although Sass offers many features that normal CSS did not offer several years ago, they provided many features that have now been integrated into CSS3 and modern browsers, which are receiving more regular updates and supporting specifications much more rapidly than before. This means that you can already access many CSS features provided by Sass in browsers including CSS variables — admittedly without Sass's unique features but also without the need for JavaScript to run the preprocessor.

Postprocessors, meanwhile, allow developers to write modern CSS with all of the vendor prefixes needed to support modern browsers. They change variables into functional code for CSS and converts the CSS you write into styles that will work in all browsers, including Internet Explorer. In Fabian's opinion, PostCSS can be considered a bridge from the old to the new much like a JavaScript shim in which one can use new language features available to the language but generate old JavaScript from the file.

Conclusion

Claro: Drupal Available Updates Claro is the new administration theme for Drupal 8, and it looks to be one of the most exciting developments for content editors and site builders in recent years, thanks to the design and development team's emphasis on accessibility, usability, and a narrow scope on implementing a moderate redesign rather than a fundamental rework. But Claro also demonstrates some of the key manners in which Drupal themes can be both accessible and innovative. Thanks to Claro's focus on providing accessible defaults, all users can successfully make use of the administration theme. In addition, with modern approaches like the use of PostCSS, a postprocessor that offers features like vendor prefixes for all modern browsers, Claro shows an innovative front end is possible for Drupal's administration layer.

In this multi-part blog series about Claro for Drupal 8, we have explored the history and origins of the Claro theme, some of the ways in which Claro addressed usability concerns, and how Claro has exemplified accessibility and front-end innovation. In this third installment, we spotlighted some of the ways in which Claro has allowed Drupal's administrative interfaces to become more accessible, including through font size, spacing between elements, and color contrast. We also inspected some of the additional features the Claro theme has added such as PostCSS, which obviates some of the need for CSS preprocessors thanks to the availability of features like CSS variables in modern browsers, and provides postprocessing to ensure vendor prefixes are present. In the following installment of this Claro blog series, we look forward to possible improvements to Claro and Cristina's dream vision for Drupal's new flagship administration theme.

Special thanks to Cristina Chumillas, Fabian Franz, and Michael Meyers for their feedback during the writing process.

Photo by Hulki Okan Tabak on Unsplash

Mar 04 2020
Mar 04

Table of Contents

Inspirations for Claro
--Finding inspiration in many places
--Honoring Seven's legacy
Claro's current state
--A new future for Views UI?
The origins of Claro's design
--Keeping Claro's redesign lean
Conclusion

Across Drupal's storied history, perhaps the most noticeable — and visible — changes in the CMS's evolution have come from the redesigns of the administrative interface that underpin every user's interaction with the Drupal editorial experience. Now, with the release of the Claro administration theme, which replaces the Seven theme and focuses on content author and site builder use cases, Drupal has reached another important inflection point in the history of its user experience. Claro is a theme that prizes accessibility, usability, and Drupal's legacy of extensibility.

This author (Preston So, Editor in Chief at Tag1 and author of Decoupled Drupal in Practice) had the privilege of moderating a wide-ranging conversation about Claro on a Tag1 Team Talks episode with Cristina Chumillas (Claro maintainer and Front-End Developer at Lullabot) and colleagues Fabian Franz (Senior Technical Architect and Performance Lead at Tag1) and Michael Meyers (Managing Director at Tag1). In this multi-part blog series, we traverse the trajectory of Claro from its origins to its current state and dig into its design and development processes to learn more about how this administration theme came to be. In this second installment, we focus on the inspiration behind and execution of Claro's design.

Inspirations for Claro

If you are unfamiliar with the Claro administration theme for Drupal 8, I highly recommend returning to the first installment in this blog series to learn more about Claro itself, its origins, and its goals to better understand the rest of this blog post. In this section, we explore some of the sources of inspiration for Claro and how modern interaction patterns underlined the urgency of a refresh for Drupal's vaunted administration interface.

Finding inspiration in many places

Roughly one year ago, during the beginning stages of the Claro development process, the team expressed an ambitious vision. At the time, Claro's architects were primarily focused on the content author persona. Cristina Chumillas recalls during our conversation that "ideally for a content author, you should have a drag-and-drop interface" or something snazzy that would not only match modern expectations but also convey a sense of futuristic panache. The team worked towards an interface that approximated some of the patterns found in WordPress Gutenberg, as they were finding considerable inspiration in that project.

However, the obstacle Drupal presents is that reaching a certain threshold of impressive transitions and other expectations of interactive user experiences requires considerable JavaScript. In addition, the notion of substituting Drupal's entire administration interface with a dynamic JavaScript application was not feasible. As a result, the team took a step back to reconsider the technical options available to them — at the same time Gutenberg's accessibility fiasco occurred — and determined that a better choice was to release something sooner so that users could experiment with it rather than wait for foundations that may never be stable.

Honoring Seven's legacy

Another important consideration that lent urgency to the endeavor was the need to refresh at least some of the color scheme utilized in Drupal's user interface, as this was the source of some comments about the aesthetic outdatedness of the Seven theme. In the end, Claro is comprised of what Cristina calls a "grown-up" version of Seven with color changes, all of which was made possible over the course of many months with the help of many volunteers.

All in all, the change in Claro's vision away from reinventing the wheel for Drupal's front end allowed for a more realistic and achievable end result to emerge in order to replace the Drupal user interface in a more sustainable fashion. Because Claro is ultimately based on the same stylesheet as Seven, the key advantage that Claro has over a solution such as a JavaScript application is that stability is not a concern.

Claro's current state

Claro is a beta theme in Drupal 8.8, and in several releases, it is slated to become a stable core theme. As Cristina states in our Tag1 Team Talks episode, Claro is perfectly usable at this very moment, and there are several Drupal sites in production already making use of it on their own administration layers. The good news about the widening adoption of and interest in the Claro administration theme is that there is growing momentum for larger-scale modifications to many more interfaces than were touched initially.

A new future for Views UI?

One of the chief examples of this momentum is strong interest in updating Views UI, which remains one of the most frequent user interfaces that Drupal users come in contact with. But as with all projects, Claro had the constraint of time and scope, and the team was loath to deem the stability of a redesigned Views UI a showstopper for a stable release of Claro. This situation would have led, Cristina argues, to a never-ending initiative without a finish line in sight.

Claro: Existing Views UI Cristina states during our webinar that eventually, the vision is to reach the place where all of Drupal's user interfaces have been redesigned from their previous Seven incarnations, from which point several other initiatives will kick off. But Cristina also rightfully adds that anointing an initiative with the responsibility of refreshing an administration interface cannot place every conceivable user interface under that umbrella if there is any hope of releasing a finished product within five years.

During our discussion, Fabian joined Cristina in recommending a strategy in which iterative initiatives that focus on regularly introducing small amounts of new features is the most sustainable form of initiative coordination. After all, at some point we must all decide that we are finished and there is no more to be done. In fact, the Claro team originally envisioned adding a new user role and even adding a second dashboard for content authors with draggable blocks, but questions quickly spiraled out of control, with suggestions for modifying the entire layout of Drupal's administration interface. Cristina admits that many portions of Drupal's administration layer will require initiatives in their own right in order to handle the inevitable demands that a redesign of a complex user interface will bring.

The origins of Claro's design

The initial ideas for the design of the Claro administration theme emerged almost from the beginning, as Cristina recalls during our Tag1 Team Talks episode. During those first discussions, it became readily apparent that any new administration theme had to look sufficiently different enough from Seven such that the vast majority of Drupal users would immediately spot the difference. But Claro's redesign had many limitations, including in Drupal's own brand: It was considered anathema to Drupal's brand identity to change the color scheme to one involving red and green that didn't reflect Drupal's color consistency.

Keeping Claro's redesign lean

Claro: Manage form display While the Claro team had sought lighter and more vivid colors for Claro and looked for inspiration in sources like Material Design and other design systems available online, they also understood that they could not reinvent the wheel with a revolutionary design, not only due to technical considerations but also aesthetic concerns. After all, administration interfaces are not intended to be the center of attention; Cristina efficiently defines the optimal administration interface as "something that works and that people understand."

Because of this sentiment across the team, Claro's architects decided to eschew fancy buttons and fieldsets, such as one particular interaction pattern in which a text field would change its outline color and display an underline to add dynamism to the page. During the first accessibility review, however, the consensus was that an overly "fancy" design was not appropriate for all users and that sticking to basic design principles was a better approach. Trying to keep things focused and tight, the Claro team sought a brighter look, initially employing a neon blue that was eventually darkened due to accessibility concerns. In the end, most of Claro's design elements came together based on inspiration from other designs already in the wild, and the team did their utmost to adhere to known patterns.

Conclusion

With the new Claro administration theme, Drupal 8 joins the growing list of content management systems that have revamped their look and feel for their most important users: the site builders and content authors that use the CMS on a daily basis. Claro demonstrates the sort of scrutiny on personas that also suggests substantial attention paid to the needs of all users, whether that entails accessibility, usability, or extensibility. And thanks to the give-and-take that characterizes accessibility reviews and design sprints, Claro emerged from its development with an efficient design that recalls many known patterns and is therefore a familiar and eminently learnable interface.

In this blog post, we discovered some of the inspiration that Claro built upon to produce its subtle and modern design. We also discussed some of the motivations that led to some of Claro's design decisions, including the choice to eschew a neon blue color and simplification of interactions with text fields in order to limit unnecessary distractions. In the third installment of this multi-part blog series about Claro, we spotlight some of the accessibility considerations that went into Claro's design and development, among other topics.

Special thanks to Cristina Chumillas, Fabian Franz, and Michael Meyers for their feedback during the writing process.

Photo by Aaron Greenwood on Unsplash

Feb 27 2020
Feb 27

Table of contents

Anxieties and aspirations for decoupled Drupal
--Overly different front-end worlds
--Web Components for better encapsulation
--The age-old debate: Decouple Drupal itself?
What’s next for decoupled Drupal?
--Security remains a competitive advantage
--An approach for dynamic components
--Further in the future: A more real-time Drupal
Conclusion

As another decade comes to a close in Drupal’s long and storied history, one important trend has the potential to present amazing — or unsettling, depending on how you look at it — opportunities and shifts in the Drupal ecosystem. That trend is decoupled Drupal. Nowadays, there is no limit to the content prolifically produced by the community about the topic, whether it’s in the form of a comprehensive book on the subject or a conference focused on the theme. Today, decoupled Drupal implementations seem, at best, to be a door presenting new approaches and ideas, and, at worst, here to stay for good without proper consideration for potential externalities.

On one of the latest Tag1 Team Talks episodes, we gathered four insider experts on decoupled Drupal, including yours truly (Preston So, Editor in Chief at Tag1 and author of Decoupled Drupal in Practice), Sebastian Siemssen (Senior Architect and Lead React Developer at Tag1 and maintainer of the GraphQL module), Fabian Franz (Senior Technical Architect and Performance Lead at Tag1), and Michael Meyers (Managing Director at Tag1). Over the course of the wide-ranging discussion, we revealed some of our reminiscences, misgivings, and hopes about decoupled Drupal in the years to come.

What trajectory will decoupled Drupal take over the coming decade? In the second part of this two-part blog series, we take a closer look at how decoupled Drupal has already revolutionized how we think about key features in Drupal with Fabian and Sebastian looking to reshape how Drupal renders markup and dynamic components and all of us highlighting security as one of decoupled Drupal’s competitive advantages.

Anxieties and aspirations for decoupled Drupal

One of the subjects we focused our attention on was our feelings on the current state of decoupled Drupal and how it is impacting the way we implement Drupal architectures today. Fabian, in particular, has mixed feelings about decoupled Drupal at present. While decoupled Drupal confers significant advantages, as we saw in the previous blog post, it also can lead to skepticism about the disadvantages it introduces. I wrote about the risks and rewards of decoupled Drupal back in 2016, and many of these pros and cons still resonate deeply today.

The loss of administrative features that are present in Drupal’s presentation layer but absent from decoupled front ends is an important rationale for many who believe monolithic architectures remain wholly viable. For instance, Fabian cited as evidence a recent Tag1 project that leveraged a decoupled Drupal architecture but also required some reinvention of the wheel to reimplement contextual links, a key front-end Drupal feature, in the decoupled presentation layer. Nonetheless, decoupled Drupal remains an excellent choice for mobile applications, digital signage, and even conversational interfaces.

Overly different front-end worlds

One of the issues with decoupled Drupal is the fact that developers are attempting to solve problems that, according to Fabian, should be solved within Drupal itself.

As Fabian argues in our recent webinar, “One of the mistakes we made in core is that we are mixing data and presentation within our render trees and entities.” The fact that the experience of React developers differs so substantially from Drupal theme developers further highlights this issue.

Web Components for better encapsulation

At DrupalCon Amsterdam, Fabian presented a session about Web Components that proposed a path forward to narrow the gap between front-end developers who are pursuing new paradigms such as React and Drupal developers who wish to implement components in Drupal front ends. Web Components are compelling because they provide for better-encapsulated code but still allow for data to come from any source, including Drupal.

In Fabian’s view, Drupal needs to do much more to bridge the gap between new front-end ecosystems and Drupal, together with a finely tuned developer experience that doesn’t require significant new learning but does allow for better closeness between the front end and the back end. For Fabian, it is essential that the Drupal community scrutinize some of the approaches to componentization found in JavaScript ecosystems.

The age-old debate: Decouple Drupal itself?

Sebastian argues that spitting the back end and front end of Drupal more at the kernel level is the right approach. There is significant weight in Drupal’s codebase that presents challenges not only for the administrative interface that many Drupal users work with on a daily basis but also the API layer within Drupal.

In fact, one of the original goals of the Web Services and Context Core Initiative (WSCCI), whose trajectory I cover extensively in Decoupled Drupal in Practice, was to recast core as a slim kernel that provides low-level functionality completely isolated from user interface components.

What’s next for decoupled Drupal?

So what’s next for decoupled Drupal? How will we as practitioners modify the approaches we take and the paradigms we utilize as decoupled Drupal continues to gain steam across the community? For this, I requested help from Fabian and Sebastian to offer their perspectives on some of the possible directions that decoupled Drupal might take as we traverse new territory.

Security remains a competitive advantage

One of the more enticing elements of decoupled Drupal is the fact that it can facilitate favorable security outcomes. Thanks to the rapidly evolving advantages offered by new JAMstack (JAM stands for JavaScript, APIs, and markup) and serverless approaches, security can become much less of a concern in decoupled architectures.

With a content server like that provided by Drupal, argues Fabian, we can feasibly implement JAMstack architectures that yield static HTML pages as the end result rather than dynamic applications with potential security vulnerabilities or, worse yet, a monolithic architecture that means an attacker could gain access to the entire Drupal system from end to end by exploiting a front-end weakness. After all, static HTML is completely secure, as there is no executable code. For Fabian, this is one of the most important and often overlooked advantages of the decoupled Drupal approach.

An approach for dynamic components

These innovations in the front-end development landscape comprise some of the rationales that Fabian cites for taking a closer look at the JavaScript community and how they handle encapsulated components. The crucial missing piece in any JavaScript implementation is sourcing data — and this is an area that Drupal can improve.

Sebastian agrees that decoupling data from internal Drupal objects like fields and entities could resolve many issues in Drupal, but he also argues that one of the key disadvantages of Drupal in its current conception is its push-based model. As an illustration of this, when Drupal fulfills a request, it executes a massive processing engine to pull data from sources and feed them into a rendering engine.

He goes on to argue that reversing Drupal’s render pipeline approach and pulling data from graphs (e.g. through GraphQL) as the template engine needs it, just in time, would allow Drupal to evolve to a more modern, pull-based approach. To Sebastian, this pull-based mechanism is precisely what an API-first architecture promises, and he cites recent innovations surrounding GraphQL in Twig as the initial steps toward this exciting vision.

Further in the future: A more real-time Drupal

One of the areas that has long fascinated me is the potential for Drupal to evolve into a more real-time system that confers significant advantages when it comes to dynamic components. Fabian argues that Web Components are a key step on the road to real-time Drupal, and a “components everywhere” approach could allow for this to occur. With a component-based approach, the only additional step necessary would be a Node.js server that updates components whenever data sees updates as well.

For all of Drupal’s benefits, real-time capabilities remain elusive. As Sebastian rightly states, real-time approaches are largely infeasible in PHP. After all, real-time data is something that Drupal is ill-equipped to handle itself; any real-time functionality needs to be handled on top of Drupal in a non-PHP layer. One option that Sebastian suggests is the combination of a Node.js server, Drupal’s PHP environment, and Redis, which would facilitate a push-based approach. Saving in Drupal could send a notification to a PubSub environment such as Redis or Kafka, and a Node.js microservice could update the client through a WebSocket.

Though this would mean forwarding Drupal’s APIs to a Node.js environment, fortunately, there are several ways to approach this in a way that keep Drupal’s benefits such as granular caching in Drupal 8, which is provided through the cache tag system. And the closer the data source is to an API like that built in GraphQL, more can be done to optimize the database queries and caching. Fortunately, GraphQL comes with WebSocket and subscription support, and it would be a compelling choice given the ongoing work in the GraphQL specification to send GraphQL queries and subscriptions in the same response.

Conclusion

Thanks to decoupled Drupal, the Drupal community is in the midst of a renaissance that is revolutionizing the way we think about the front end of Drupal while simultaneously challenging our long-held assumptions about how Drupal should look and operate. While this is a testament to Drupal’s considerable longevity, it also presents obstacles if we wish to guarantee the future of Drupal for the coming decade. Fortunately, owing to the vast amount of resources surrounding decoupled Drupal, including the book Decoupled Drupal in Practice and even a conference dedicated to the subject, more and more discussion around the topic will yield significant fruit for the promising future of Drupal.

While no one can prognosticate how decoupled Drupal will look in 2030, at the end of this current decade, we can certainly see that the story of decoupled Drupal is only just getting started. I strongly encourage you to check out our recent Tag1 Team Talks episode to learn more about what could influence Drupal’s trajectory substantially in the coming years. I, for one, believe that though we have come a long way, we still have much distance to cover and much thinking to do in order to prepare for a more decoupled Drupal. Nevertheless, the solutions recommended by Fabian and Sebastian excite me just as much as I hope they have inspired our readers too.

Special thanks to Fabian Franz, Michael Meyers, and Sebastian Siemssen for their feedback during the writing process.

Photo by Jesse Bowser on Unsplash

Feb 25 2020
Feb 25

Table of Contents

Defining decoupled Drupal
--The pros and cons remain largely unchanged
--A competitive advantage for commercial Drupal
Aside: Our stories in decoupled Drupal
GraphQL as a game-changer
--GraphQL queries at scale
--What about JSON:API vs. GraphQL?
--GraphQL v4 and custom schemas
Conclusion

Over the last five years, decoupled Drupal has grown from a fringe topic among front-end enthusiasts in the Drupal community to something of a phenomenon when it comes to coverage in blog posts, tutorials, conference sessions, and marketing collateral. There is now even a well-received book by this author and a yearly conference dedicated to the topic. For many Drupal developers working today, not a day goes by without some mention of decoupled architectures that pair Drupal with other technologies. While Drupal’s robust capabilities for integration are nothing new, there have been comparatively few retrospectives on how far we’ve come on the decoupled Drupal journey.

Recently, your correspondent (Preston So, Editor in Chief at Tag1 and author of Decoupled Drupal in Practice) sat down for a no-holds-barred discussion about decoupled Drupal on Tag1 Team Talks with three other early insiders in the decoupled Drupal mindspace: Sebastian Siemssen (Senior Architect and Lead React Developer at Tag1 and maintainer of the GraphQL module), Fabian Franz (Senior Technical Architect and Performance Lead at Tag1), and Michael Meyers (Managing Director at Tag1).

During the conversation, we spoke about the unexpected ways in which decoupled Drupal has evolved and where it could go in the near and far future. In this two-part blog series, we look back and peer forward into decoupled Drupal’s trailblazing and disruptive trajectory in the Drupal ecosystem, starting with Sebastian, who created Drupal’s GraphQL implementation, and Michael, who has witnessed firsthand the paradigm shift in Drupal’s business landscape thanks to the benefits decoupled Drupal confers.

Defining decoupled Drupal

As it turns out, defining decoupled Drupal can be tricky, given the diverse range of architectural approaches available (though these are covered comprehensively in Preston’s book). In its simplest form, decoupled Drupal is an idea centered around the notion of employing Drupal as a provider of data for consumption in other applications. In short, the “decoupling” of Drupal occurs when developers opt to utilize Drupal data by issuing requests to APIs that emit JSON (or XML) rather than through Drupal’s native presentation layer.

As Fabian notes in the webinar, there are multiple “flavors” of decoupled Drupal that have emerged. Fully decoupled Drupal implementations aim to replace the entire presentation layer of Drupal with a distinct application. Progressively decoupled Drupal, an idea first promulgated by Drupal project lead Dries Buytaert in a 2015 blog post, recommends leveraging API calls to provide for components in the existing Drupal front end rather than an entirely separate presentation layer.

The pros and cons remain largely unchanged

Decoupled Drupal presents both advantages and disadvantages that are worth considering for any implementation, and these are also covered in Decoupled Drupal in Practice. Fully decoupled Drupal, for instance, means jettisoning a great deal of what makes Drupal such an effective web framework by limiting its feature set substantially to solely the APIs for retrieving data and an administrative interface for content management.

Though decoupling Drupal inevitably means losing certain functionality, such as contextual links, in-place editing, and layout management, it does confer the opportunity to create something entirely custom atop the data model that Drupal provides, as Sebastian states in our Tag1 Team Talks episode. Nonetheless, while decoupled Drupal does introduce unprecedented flexibility, it also goes beyond a simple separation of the presentation layer from the data layer by rendering Drupal more replaceable. The notion of replaceability does belie an undercurrent of anxiety surrounding decoupled Drupal, and it presents risks for Drupal’s position in the CMS market.

A competitive advantage for commercial Drupal

Nonetheless, from the commercial standpoint and the customer perspective, there is no question that decoupled Drupal is a boon for ambitious digital experiences. For instance, during our conversation, Michael argues that decoupled Drupal actually represents a competitive advantage for Drupal in the commercial space rather than an erosion of its value. The narrative of integration in which Drupal has always trafficked has long been influential in swaying skeptical stakeholders. In short, in this view, Drupal’s core REST API, JSON:API, and GraphQL only strengthen this flexibility to integrate.

As a matter of fact, Tag1 Consulting’s yearslong work with Symantec supports this notion. Together with Tag1, Symantec embarked on a migration from version to version of Drupal in a nontraditional way that was only made possible thanks to decoupled Drupal technologies. By providing multiple front ends across Drupal versions, Symantec and Tag1 succeeded in both accelerating and effectively managing the migration to Drupal 8. Michael notes that from a client standpoint at Tag1, decoupled Drupal has been a valuable and sought-after asset; if anything, it has only increased evaluators’ interest in the platform.

Aside: Our stories in decoupled Drupal

Sebastian, Fabian, and I also reminisced about our respective stories in the decoupled Drupal landscape and how it has impacted the trajectory of our work even today. Arguably among the first in the Drupal community to leverage a decoupled architecture, Fabian admits building a decoupled site that consumed unstyled HTML snippets as far back as 2009, with other data coming from multiple sources. We can certainly forgive the fact that he opted to use Flash for much of the front end in that build.

Early in 2015, Sebastian found that the release of GraphQL by Facebook as a new specification made everything “click” for him as one of the crucial gaps in Drupal’s APIs. That same year, Sebastian began working on GraphQL for PHP, basing much of his ideas on the just recently open-sourced GraphQL JavaScript library. Thanks to Sebastian’s tireless work, the GraphQL module for Drupal first appeared as a codebase on Drupal.org in March 2015 and has since skyrocketed in interest and popularity.

My own journey is much less glamorous, as I assisted Drupal project lead Dries Buytaert with his initial thought leadership on the subject of decoupled Drupal back in 2015. At the time there was considerable hand-wringing about what disruptive advances in the front-end development world could mean for Drupal. Over the course of time, my perspectives have evolved considerably as well, and I believe there are many excellent use cases for monolithic implementations, something I stress in my book about decoupled Drupal. Today, I help run the biggest and oldest conference on decoupled and headless CMS architectures, Decoupled Days, which is now entering its fourth year.

GraphQL as a game-changer

One of the key questions for decoupled Drupal practitioners has been how to leverage the best that Drupal has to offer while also accessing some of the best ideas emerging from the API and front-end development spaces. For instance, among Drupal’s most notable selling points is the fact that not only does Drupal benefit from an excellent administrative interface; it also offers a robust customizable data model that can be crafted and fine-tuned. Entity API and Field API, for instance, have long been some of the most vaunted components of the back-end Drupal developer experience.

Tag1 recently found this with a project that employs Laravel, a more lightweight PHP framework for websites that is far less opinionated than Drupal. Light layers of entities and fields were required for the Laravel project, and Fabian remarks in our recent webinar that these are particularly straightforward to implement and validate in Drupal. GraphQL adds another fascinating dimension to this by allowing Drupal to handle the heavy lifting of important features like field validation while permitting client-tailored queries.

GraphQL queries at scale

Sebastian describes GraphQL as a reversal of the traditional relationship between the server and client. Rather than the client being forced to adhere to what the server provides, the server declares the data possibilities that it is capable of fulfilling, and based on these catalogued possibilities, the client defines exactly what it needs on a per-request basis. The client sends the query to the server in a particular format, which Sebastian characterizes as much like the “JSON you want returned from the API but only the keys.” Thereafter, the GraphQL API inserts the values and returns a response with precisely the same shape.

This brings us to one of the most fascinating aspects of GraphQL within the Drupal CMS, which often utilizes deeply nested relational data structures that are uniquely well-suited for modeling into a graph — which is the very core of what GraphQL does. One of the best aspects of GraphQL, moreover, is its capabilities for schema and query validation, with which we can prevent distributed denial-of-service (DDoS) attacks that attempt to overload queries with many nested layers by checking and analyzing the complexity of queries ahead of time, a task that is much easier thanks to the predictability of GraphQL schemas.

What about JSON:API vs. GraphQL?

While the story of GraphQL has been long-running in the Drupal ecosystem, it’s important to note the other big player in Drupal’s web services: JSON: API, which was just this year introduced into Drupal core. One of the biggest reasons why JSON:API has gained prominence is its relative stability and its high amount of documentation. Sebastian argues that React developers coming from their ecosystem are more likely to be familiar with GraphQL already, which also helps elevate its status within the decoupled Drupal ecosystem.

GraphQL v4 and custom schemas

One of the most anticipated releases of the GraphQL module is GraphQL v4 for Drupal, which means several significant changes for the ever-evolving GraphQL module. In the latest version of the GraphQL module, the GraphQL schema is fully in the control of Drupal developers, which is a substantial change from previous releases. After all, one of the best selling points for using GraphQL in the first place is schema customizability.

According to Sebastian, this means that you can decouple Drupal on the API access and contract level rather than foisting the data model and data internals of Drupal on the resulting GraphQL API, which may cause confusion among developers of consumer applications like React implementations.

Conclusion

Perhaps the most intriguing development in the long-running saga of decoupled Drupal is the diversification and proliferation of a wide range of approaches, like GraphQL, that improve the developer experience in Drupal significantly. Besides its obvious benefits of greater flexibility on the side of developers, moreover, Drupal users and agency customers are also discovering the advantages of decoupled Drupal for a variety of use cases.

As we recently discussed during our conversation, a mix of both nostalgia and forward thinking, decoupled Drupal is here to stay. Whereas in years past this statement may have caused considerable anxiety in the Drupal community, today it is emblematic of the ongoing explosion of options and capabilities that decoupled Drupal engenders for Drupal developers. But in the second installment in this two-part series, Fabian, Sebastian, and I discuss some of the anxieties and worries we share about decoupled Drupal, before attempting to predict what could come next for this fast-evolving paradigm.

Special thanks to Fabian Franz, Michael Meyers, and Sebastian Siemssen for their feedback during the writing process.

Photo by Jesse Bowser on Unsplash

Feb 19 2020
Feb 19

Content collaboration has long been table stakes for content management systems like WordPress and Drupal, but what about real-time peer-to-peer collaboration between editors who need direct interaction to work on their content? The WordPress Gutenberg team has been working with Tag1 Consulting and the community of Yjs, an open-source real-time collaboration framework, to enable collaborative editing on the Gutenberg editor. Currently an experimental feature that is available in a Gutenberg pull request, shared editing in Gutenberg portends an exciting future for editing use cases beyond just textual content.

Yjs is both network-agnostic and editor-agnostic, which means it can integrate with a variety of editors like ProseMirror, CodeMirror, Quill, and others. This represents substantial flexibility when it comes to the goals of WordPress to support collaborative editing and the potential for other CMSs like Drupal to begin exploring the prospect of shared editing out of the box. Though challenges remain to enable truly bonafide shared editing off the shelf in WordPress and Drupal installations, Gutenberg is brimming with possibility as the collaboration with Tag1 continues to bear significant fruit.

In this Tag1 Team Talks episode that undertakes a technical deep dive into how the WordPress community and Tag1 enabled collaborative editing in the Gutenberg editor, join Kevin Jahns (creator of Yjs and Real-Time Collaboration Systems Lead at Tag1), Michael Meyers (Managing Editor at Tag1), and your host Preston So (Editor in Chief at Tag1 and author of Decoupled Drupal in Practice) for an exploration of how CMSs around the landscape can learn from Gutenberg's work to empower editors to collaborate in real-time in one of the most exciting new editorial experiences in the CMS world.

[embedded content]

Feb 18 2020
Feb 18

In the first part of our two-part blog series on Drush 10, we covered the fascinating history of Drush and how it came to become one of the most successful projects in the Drupal ecosystem. After all, many of us know many of the most common Drush commands by heart, and it’s difficult to imagine a world without Drush when it comes to Drupal’s developer experience. Coming on the heels of Drupal 8.8, Drush 10 introduces a variety of new questions about the future of Drush, even as it extends Drush’s robustness many years into the future.

Your correspondent (Preston So, Editor in Chief at Tag1 and author of Decoupled Drupal in Practice) had the unique opportunity to discuss Drush’s past, present, and future with Drush maintainer Moshe Weitzman (Senior Technical Architect at Tag1), Fabian Franz (Senior Technical Architect and Performance Lead at Tag1), and Michael Meyers (Managing Director at Tag1), as part of the Tag1 Team Talks series at Tag1 Consulting, our biweekly webinar and podcast series. In the conclusion to this two-part blog series, we dig into what’s new in Drush 10, what you should consider if you’re making a choice between Drush and Drupal Console, and what the future for Drush might hold in store for Drupal’s first CLI.

What’s new in Drush 10

Drush 10 is the version of Drush optimized for use with Drupal 8.8. It embraces certain new configuration features available as part of the upcoming minor release of Drupal, including the Exclude and Transform APIs as well as config-split in core. Nevertheless, the maintainers emphasize that the focus of Drush 10 was never on new additive features; instead they endeavored to remove a decade’s worth of code from Drush and prepare it for many years to come.

To illustrate this fact, consider that Drush 9 was a combination of both old APIs from prior versions of Drush and all-new APIs that Drush’s maintainers implemented to modernize Drush’s commands. Therefore, while Drush 9 commands generally make use of the newly available APIs, if you call a site with Drush 9 installed from Drush 8, it will traverse all of the old APIs. This was a deliberate decision by Drush’s maintainers in order to allow users a year to upgrade their commands and to continue to interoperate with older versions. As a result of the removals of these older approaches, Drush 10 is extremely lean and extremely clean, and it interoperates with sites having Drush 9 but not those with earlier versions.

How should developers in the Drupal community adopt Drush 10? Moshe recommends that users upgrade at their earliest convenience through Composer, as Drush’s maintainers will be able to offer the best support to those on Drush 10.

Why Drush over Drupal Console?

One key question that surfaces frequently concerning Drupal’s command-line ecosystem is the distinction between Drush and a similar project, Drupal Console, and when to use one over the other. Though Drush and Drupal Console accomplish a similar set of tasks and share similar architectures because they both depend on Symfony Console, there are still quite a few salient differences that many developers will wish to take into account as they select a command-line interface to use with Drupal.

Commands, for instance, are one area where Drush and Drupal Console diverge. Command authors will find that commands are written quite differently. Drush leverages an annotated command layer on top of Symfony Console where developers employ annotations to write new commands. Drupal Console instead utilizes Symfony Console’s approach directly, with a few methods attached to each command. However, this is a minor consideration, as there is little to no difference in the CLI’s functionality, and it is merely a stylistic preference.

Drush and Drupal Console also differ significantly in their approaches to testing. Whereas Drupal Console performs unit testing, Drush prefers functional testing, with a full copy of both Drupal and Drush in their test suite. All Drush CLI commands are run on a real, fully functional Drupal site, whereas Drupal Console opts to leverage more mocking. There are admittedly many advantages to both approaches. But perhaps the most important distinction is of a less technical variety: Drupal Console has seen a bit less contribution activity as of late than Drush, which is an important factor to consider when choosing a CLI.

The future of Drush: Drush in core?

Though Moshe and Greg have committed themselves to maintaining and supporting Drush in the future, there are doubtlessly many questions about Drush’s roadmap that will influence decision-making around Drupal.

Drush’s inclusion in core has long been a key talking point with a variety of potential resolutions. Drupal already has two CLI commands in it unrelated to Drush, namely the site-install and quick-start commands, which are seldom used as they have limited coverage of key use cases. For instance, site-install only installs Drupal successfully on SQLite databases and lacks consideration for configuration. Drush’s maintainers are keen on considering a version of Drush in core, and an active discussion is ongoing.

Moreover, now that the starter template for Drupal projects is now deprecated in favor of core-recommended, there is an opportunity for Drush 10 to serve as a key dependency in those starter templates, initially as a suggested dependency and eventually as a required one. Some of the key commands that a hypothetical Drush in core would encompass include enabling and uninstalling modules as well as clearing caches and logging in as a user. In the not-too-distant future, a Drupal user could start a Drupal project and immediately have Drush and all its commands available from the very outset.

Conclusion

Drush 10 is an inflection point not only in the history of Drupal but in how Drupal developers interact with Drupal on a daily basis. Thanks to its leaner, faster state, Drush 10 marks a new era for remote interactions with Drupal. Because Drush 10 has tracked closely to the Drupal 8 development cycle, many of the core changes present in Drupal 8.8 are reflected in Drush 10, and the ongoing discussion surrounding the potential of Drush in core will doubtlessly continue apace.

For many of us in the Drupal community, Drush is more than a cherished tool; it is one of the primary entry points into Drupal development. With the help of your contributions, Drush can reach even greater heights. Moshe recommends that new contributors get started with improving Drush’s documentation and content concerning Drush, whether it comes in the form of blog posts or step-by-step tutorials that make learners’ experiences much better. The Drush maintainers are always happy to link to compelling content about Drush, to address bugs and issues in Drush’s issue queue, and to offer co-maintainership to prolific contributors.

While this was an exhaustive look at Drush 10, it by no means includes all of the insights we gathered from Moshe, and we at Tag1 Consulting encourage you to check out our recent Tag1 Team Talk about Drush 10 to learn even more about Drush’s past, present, and future.

Special thanks to Fabian Franz, Michael Meyers, and Moshe Weitzman for their feedback during the writing process.

Photo by Bill Oxford on Unsplash.com

Feb 13 2020
Feb 13

Table of contents

What is Tag1 Quo
How does Tag1 Quo work
What makes Tag1 Quo unique
--Immediate notice of vulnerabilities
--Backports of LTS patches
--Automated QA testing for Drupal 7 LTS
--Customer-driven product development
Conclusion

One of the challenges of securing any Drupal site is the often wide range of modules to track, security advisories to follow, and updates to implement. When it comes to Drupal security, particularly older versions of Drupal such as Drupal 6 and Drupal 7, even a slight delay in patching security vulnerabilities can jeopardize mission-critical sites. Now that Drupal 7 and Drupal 8 are fast approaching their end of life (EOL) in November 2021 (Drupal 6 reached end of life on February 24, 2016), the time is now to prepare your Drupal sites for a secure future, regardless of what version you are using.

Fortunately, Tag1 Consulting, the leading Drupal performance and security consultancy, is here for you. We’ve just redesigned Tag1 Quo, the enterprise security monitoring services trusted by large Drupal users around the world, from the ground up, with an all-new interface and capabilities for multiple Drupal versions from Drupal 6 to Drupal 8. Paired with the Tag1 Quo module, available for download on Drupal.org, and Tag1 Quo’s services, you can ensure the security of your site with full peace of mind. In this blog post, we’ll cover some of the core features of Tag1 Quo and discuss why it is essential for your sites’ security.

What is Tag1 Quo?

Tag1 Quo is a software-as-a-service (SaaS) security monitoring and alerting service for Drupal 6, Drupal 7, and Drupal 8. In addition, it includes long-term support (LTS) for Drupal 6 and is slated to commence backporting security patches for both Drupal 7 and Drupal 8 when both major versions no longer have community-supported backports. The centerpiece of Tag1 Quo integration with Drupal is the Tag1 Quo module, which is installed on your servers and communicates securely with our servers.

In addition, for a fee, we can help you with a self-hosted version of Tag1 Quo for sites hosted on-premise. This does require setup fees and entails higher per-site licensing fees, so we encourage you to reach out to us directly if you’re interested in pursuing this option.

How does Tag1 Quo work?

When a new module update is released on Drupal.org, or when a security advisory is announced that directly impacts your Drupal codebases, the Tag1 Quo system alerts you immediately and provides all of the necessary updates required to mitigate the vulnerability, with a direct link to the code you need to install to address the issue. Not only are these alerts sent over e-mail by default; they can also flow directly into your internal project workflows, including issue tracking and ticketing systems.

Tag1 Quo doesn’t stop there. As part of our long-term support (LTS) offering, when security releases and critical updates emerge, or when new security vulnerabilities are announced for community-supported Drupal versions, Tag1 audits these and determines whether the identified vulnerability also impacts end-of-life (EOL) versions of Drupal such as Drupal 6 and, in November 2021, Drupal 7. If those EOL versions are also susceptible to the vulnerabilities, we backport and test all patches to secure the EOL versions as well and distribute them to you through the Tag1 alert system.

Moreover, when a new security vulnerability is discovered in an EOL version of Drupal without an equivalent issue in a currently supported version, Tag1 creates a patch to rectify the problem and collaborates with the Drupal Security Team (several of whom are part of the Tag1 team) to determine if the EOL vulnerability applies vice-versa to all currently supported versions of Drupal so that they can be patched too. In short, no matter where the vulnerability occurs across all of Drupal’s versions, you can rest easy with Tag1 Quo’s guarantees.

What makes Tag1 Quo unique

Tag1 Quo features a centralized dashboard with an at-a-glance view of all of your Drupal sites and their current status, regardless of where each one is hosted. After all, most enterprise organizations juggle perhaps dozens of websites that need to remain secure. Such a perspective at an organizational level is essential to maintain the security of all of your websites. But the Tag1 Quo dashboard is only one among a range of capabilities unique to the service.

Immediate notice of vulnerabilities

Although several members of the Tag1 team are also part of the Drupal Security Team, and are aware of vulnerabilities as soon as they are reported, the Drupal Security Team’s first policy is to collaborate privately to address the issue before revealing its nature publicly. This is to facilitate progressive disclosure in the form of issuances of public advisories and releases of public patches before nefarious actors are able to attack Drupal sites with success. This is for your safety and for the viability of released patches.

Thanks to our deep knowledge of both projects used by our clients' websites and security advisories, Tag1 has the distinction of being among the very first to notify Tag1 Quo customers as soon as the official announcement is released. Immediately afterwards, Tag1 Quo will prepare you to apply the updates as quickly as possible to ensure your web properties’ continued safety.

Backports of LTS patches

If a fix for a vulnerability is reported for currently supported versions of Drupal but also applies to EOL versions, the patch must be backported for all Drupal sites to benefit from the patch. Unfortunately, this process can be complex and require considerable planning and analysis of the problem across multiple versions—and it can sometimes only occur after the patch targeting supported versions has been architected or completed. This means it may take more time to develop patches for LTS versions of Drupal.

Luckily, we have a head-start in developing LTS patches thanks to our advance notice of vulnerabilities in currently supported versions of Drupal. Despite the fact that we cannot guarantee that LTS updates will be consistently released simultaneously with those targeting supported versions, Tag1 has an admirable track record in releasing critical LTS updates at the same time as or within hours of the issuance of patches for supported Drupal versions.

Automated QA testing for Drupal 7 LTS

Throughout Drupal’s history, the community encouraged contributors to write tests alongside code as a best practice, but this was rarely the case until it became an official requirement for all core contributions beginning with the Drupal 7 development cycle in 2007. Tag1 team members were instrumental in tests becoming a core code requirement, and we created the first automated quality assurance (QA) testing systems distributed with Drupal. In fact, Tag1 maintains the current Drupal CI (continuous integration) systems that perform over a decade of concurrent years of testing within a single calendar year.

Because the Drupal Association has ended support for Drupal 7 tests and decommissioned those capabilities on Drupal.org, Tag1 is offering the Tag1 Quo Automated QA Testing platform as a part of Tag1 Quo for Drupal 7 LTS. The service will run all tests for Drupal 7 core and any contributed module tests that are available. Where feasible and appropriate, Tag1 will also create new tests for Drupal 7’s LTS releases. Therefore, when you are notified of LTS updates, you can rest assured that they have been tested robustly against core and focus your attention on integration testing with your custom code instead, all the while rolling out updates with the highest possible confidence.

Customer-driven product development

Last but certainly not least, Tag1 Quo is focused on your requirements. We encourage our customers to request development in order for us to make Tag1 Quo the optimal solution for your organization. By working closely with you to determine the scope of your feature requests, we can provide estimates for the work and an implementation timeline. While such custom development is outside the scope of Tag1 Quo’s licensing fees, we allot unused Tag1 Quo consulting and support hours to minor modifications on a monthly basis.

Examples of features we can provide for custom code in your codebases includes ensuring your internal repositories are relying on the latest versions of dependencies, and providing insights into your custom code through site status views on your Tag1 Quo dashboard. We can even do things like add custom alerts to notify specific teams and users responsible for these sites and customize the alerts to flow into support queues or other ticketing systems. Please get in touch with us for more information about these services.

Conclusion

The new and improved Tag1 Quo promises you peace of mind and renewed focus for your organization on building business value and adding new features. Gone are the days of worrying about security vulnerabilities and anxiety-inducing weekends spent applying updates. Thanks to Tag1 Quo, regardless of whether your site is on Drupal 6, Drupal 7, or Drupal 8, you can rest assured that your sites will remain secure and monitored for future potential vulnerabilities. With a redesigned interface and feature improvements, there is perhaps no other Drupal security monitoring service better tuned to your needs.

Special thanks to Jeremy Andrews and Michael Meyers for their feedback during the writing process.

Photo by Ian Schneider on Unsplash

Feb 12 2020
Feb 12

An effective administrative interface is table stakes for any content management system that wishes to make a mark with users. Claro is a new administration theme now available in Drupal 8 core thanks to the Admin UI Modernization initiative. Intended to serve as a logical next step for Drupal's administration interface and the Seven theme, Claro was developed with a keen eye for modern design patterns, accessibility best practices, and careful analysis of usability studies and surveys conducted in the Drupal community.

Claro demonstrates several ideas that not only illustrate the successes of open-source innovation but also the limitations of overly ambitious ideas. By descoping some of the more unrealistic proposals early on and narrowing the focus of the Claro initiative on incremental improvements and facilitating the work of later initiatives, Claro is an exemplar of sustainable open-source development.

In this closer look at how Claro was made possible and what its future holds for Drupal administration, join Cristina Chumillas (Claro maintainer and Front-End Developer at Lullabot), Fabian Franz (Senior Technical Architect and Performance Lead at Tag1), Michael Meyers (Managing Editor at Tag1), and Preston So (Editor in Chief at Tag1 and author of Decoupled Drupal in Practice) for a Tag1 Team Talks episode about the newest addition to Drupal's fast-evolving front end.

[embedded content]


Feb 11 2020
Feb 11

Table of Contents

If you’ve touched a Drupal site at any point in the last ten years, it’s very likely you came into contact with Drush (a portmanteau of “Drupal shell”), the command-line interface (CLI) used by countless developers to work with Drupal without touching the administrative interface. Drush has a long and storied trajectory in the Drupal community. Though many other Drupal-associated projects have since been forgotten and relegated to the annals of Drupal history, Drush remains well-loved and leveraged by thousands of Drupal professionals. In fact, the newest and most powerful version of Drush, Drush 10, is being released jointly with Drupal 8.8.0.

As part of our ongoing Tag1 Team Talks at Tag1 Consulting, a fortnightly webinar and podcast series, yours truly (Preston So, Editor in Chief at Tag1 and author of Decoupled Drupal in Practice) had the opportunity to sit down with Drush maintainer Moshe Weitzman (Senior Technical Architect at Tag1) as well as Tag1 Team Talks mainstays Fabian Franz (Senior Technical Architect and Performance Lead at Tag1) and Michael Meyers (Managing Director at Tag1) for a wide-ranging and insightful conversation about how far Drush has come and where it will go in the future. In this two-part blog post series, we delve into some of the highlights from that chat and discuss what you need to know and how best to prepare for the best version of Drush yet.

What is Drush?

The simplest way to describe Drush, beyond its technical definition as a command-line interface for Drupal, is as an accelerator for Drupal development. Drush speeds up many development functions that are required in order to take care of Drupal websites. For instance, with Drush, developers can enable and uninstall modules, install a Drupal website, block or delete a user, change passwords for existing users, and update Drupal’s site search index, among many others — all without having to enter Drupal’s administrative interface.

Because Drush employs Drupal’s APIs in order to execute actions like creating new users or disabling themes, Drush performs far more quickly than Drupal’s bootstrap itself, because there is no need to traverse Drupal’s render pipeline and theme layer. In fact, Drush is also among the most compelling real-world examples of headless Drupal (a topic on which this author has written a book), because the purest definition of headless software is an application that lacks a graphical user interface (GUI). Drush fits that bill.

The origins and history of Drush

Though many of us in the Drupal community have long used Drush since our earliest days in the Drupal ecosystem and building Drupal sites, it’s likely that few of us intimately know the history of Drush and how it came to be in the first place. For a piece of our development workflows that many of us can’t imagine living without, it is remarkable how little many of us truly understand about Drush’s humble origins.

Drush has been part of the Drupal fabric now for eleven years, and during our most recent installment of Tag1 Team Talks, we asked Moshe for a Drush history lesson.

Drush’s origins and initial years

Though Moshe has maintained Drush for over a decade to “scratch his own itch,” Drush was created by Arto Bendiken, a Drupal contributor from early versions of the CMS, and had its tenth anniversary roughly a year ago. Originally, Drush was a module available on Drupal.org, just like all of the modules we install and uninstall on a regular basis. Users of the inaugural version of Drush would install the module on their site to use Drush’s features at the time.

The Drupal community at the time responded with a hugely favorable reception and granted Drush the popularity that it still sees today. Nonetheless, as Drush expanded its user base, its maintainers began to realize that they were unable to fully realize the long list of additional actions that Drupal users might want, including starting a web server to quickstart a Drupal site and one of the most notable features of Drush today: installing a Drupal site on the command line. Because Drush was architected as a Drupal module, this remained an elusive objective.

Drush 2: Interacting with a remote Drupal site

Drush 2 was the first version of Drush to realize the idea of interacting with a remote Drupal website, thanks to the contributions of Adrian Rousseau, another early developer working on Drush. Today, one of the most visible features of Drush is the ability to define site aliases to target different Drupal sites as well as different environments.

Rousseau also implemented back-end functionality that allowed users to rsync the /files directory or sql-sync the database on one Drupal installation to another. With Drush 2, users could also run the drush uli command to log in as the root user (user 1 in Drupal) on a remote Drupal site. These new features engendered a significant boost in available functionality in Drush, with a substantial back-end API that was robust and worked gracefully over SSH. It wasn’t until Drush 9 that much of this code was rewritten.

Drush 3: From module to separate project

During the development of Drush 3, Drush’s maintainers made the decision to switch from Drush’s status as a module to a project external to Drupal in order to enable use cases where no Drupal site would be available. It was a fundamental shift in how Drush interacted with the Drupal ecosystem from there onwards, and key maintainers such as Greg Anderson, who still maintains Drush today seven versions later, were instrumental in implementing the new approach. By moving off of Drupal.org, Drush was able to offer site installation through the command line as well as a Drupal quickstart and a slew of other useful commands.

Drush 5: Output formatters

Another significant step in the history of Drush came with Drush 5, in which maintainer Greg Anderson implemented output formatters, which allow users to rewrite certain responses from Drush into other formats. For instance, the drush pm-list command returns a list of installed modules on a Drupal site, including the category in which they fit, formatted as a human-readable table.

Thanks to output formatters, however, the same command could be extended to generate the same table in JSON or YAML formats, which for the first time opened the door to executable scripts using Drush. During the DevOps revolution that overturned developer workflows soon afterwards, output formatters turned out to be a prescient decision, as they are particularly useful for continuous integration (CI) and wiring successive scripts together.

Drush 8: Early configuration support

Drush 8, the version of Drush released in preparation for use with Drupal 8 sites, was also a distinctly future-ready release due to its strong command-line support for the new configuration subsystem in Drupal 8. When Drupal 8 was released, core maintainer Alex Pott contributed key configuration commands such as config-export, config-import,config-get, and config-set (with Moshe’s config-pull coming later), all of which were key commands for interacting with Drupal’s configuration.

Due to Drush 8’s early support for configuration in Drupal 8, Drush has been invaluable in realizing the potential of the configuration subsystem and is commonly utilized by innumerable developers to ensure shared configuration across Drupal environments. If you have pushed a Drupal 8 site from a development environment to a production environment, it is quite likely that there are Drush commands in the mix handling configuration synchronicity.

Drush 9: A complete overhaul

About a year ago, Drush’s indefatigable maintainers opted to rewrite Drush from the ground up for the first time. Drush had not been substantially refactored since the early days in the Drush 3 era, when it was extracted out of the module ecosystem. In order to leverage the best of the Composer ecosystem, Drush’s maintainers rewrote it in a modular way with many Composer packages for users to leverage (under the consolidation organization on GitHub).

This also meant that Drush itself became smaller in size because it modularized site-to-site communication in a tighter way. Declaring commands in Drush also underwent a significant simplification from the perspective of developer experience. Whereas foregoing Drush commands were written in PHP as was the case in Drush 8, developers could now write Drush commands in a PHP method within a callback with the lines of Doxygen above the callback housing the name, parameters, and other details of the command. Also in the same release came YAML as the default format for configuration and site aliases in Drush as well as the beginning of Symfony Console as the runner of choice for commands.

Drush 9 introduced a diverse range of new commands, including config-split, which allows for different sets of modules to be installed and different sets of configuration to be in use on distinct Drupal environments (though as we will see shortly, it may no longer be necessary). Other conveniences that entered Drush included running commands from Drupal’s project root instead of the document root as well as the drush generate command, which allows developers to quickly scaffold plugins, services, modules, and other common directory structures required for modern Drupal sites. This latter scaffolding feature was borrowed from Drupal Console, which was the first to bring that feature to Drupal 8. Drush’s version leverages Drupal’s Code Generator to perform the scaffolding itself.

Conclusion

As you can see, Drush has had an extensive and winding history that portends an incredible future for the once-humble command-line interface. From a pet project and a personal itch scratcher to one of the most best-recognized and commonly leveraged projects in the Drupal ecosystem, Drush has a unique place in the pantheon of Drupal history. In this blog post, we covered Drush’s formative years and its origins, a story seldom told among open-source projects.

In the second part of this two-part blog post series, we’ll dive straight into Drush 10, inspecting what all the excitement is about when it comes to the most powerful and feature-rich version of Drush yet. In the process, we’ll identify some of the key differences between Drush and Drupal Console, the future of Drush and its roadmap, and whether Drush has a future in Drupal core (spoiler: maybe!). In the meantime, don’t forget to check out our Tag1 Team Talk on Drush 10 and the story behind Drupal’s very own CLI.

Special thanks to Fabian Franz, Michael Meyers, and Moshe Weitzman for their feedback during the writing process.

Photo by Jukan Tateisi on Unsplash

Feb 05 2020
Feb 05

What happens when you have a connection that isn't working, but you have a mission-critical document that you need to collaborate on with others around the world? The problem of peer-to-peer collaboration in an offline environment is becoming an increasingly pressing issue for editorial organizations and enterprises. As we continue to work on documents together on flights, trains, and buses, offline-first shared editing is now a base-level requirement rather than a pipe dream.

Yjs, an open-source framework for real-time collaboration, integrates gracefully with IndexedDB, the local offline-first database available in browsers, to help developers easily implement offline shared editing for their organization's needs. Paired in turn with other technologies like WebRTC, a peer-to-peer communication protocol, and Yjs connectors, a graceful architecture is possible that not only enables offline shared editing for a variety of use cases beyond textual content but also makes the developer experience as straightforward as possible.

In this technical and topical deep dive into how Yjs and IndexedDB make offline shared editing possible, join Kevin Jahns (creator of Yjs and Real-Time Collaboration Systems Lead at Tag1), Fabian Franz (Senior Technical Architect and Performance Lead at Tag1), Michael Meyers (Managing Editor at Tag1), and your host Preston So (Editor in Chief at Tag1 and author of Decoupled Drupal in Practice) for a Tag1 Team Talks episode you don't want to miss about how to enable offline shared editing for web applications and even CMSs like Drupal.

[embedded content]

Jan 29 2020
Jan 29

Preston is a product strategist, developer advocate, speaker, and author of Decoupled Drupal in Practice (Apress, 2018).

A globally recognized voice on decoupled Drupal and subject matter expert in the decentralized web and conversational design, Preston is Editor in Chief at Tag1 Consulting and Principal Product Manager at Gatsby, where he works on improving the Gatsby developer experience and driving product development.

Having spoken at over 50 conferences, Preston is a sought-after presenter with keynotes on five continents and in three languages.

Jan 29 2020
Jan 29

Preston is a product strategist, developer advocate, speaker, and author of Decoupled Drupal in Practice (Apress, 2018).

A globally recognized voice on decoupled Drupal and subject matter expert in the decentralized web and conversational design, Preston is Editor in Chief at Tag1 Consulting and Principal Product Manager at Gatsby, where he works on improving the Gatsby developer experience and driving product development.

Having spoken at over 50 conferences, Preston is a sought-after presenter with keynotes on five continents and in three languages.

Jan 28 2020
Jan 28

Testing is becoming an essential keyword and toolkit for developers and development teams who seek to architect and implement successful and performant websites. Thanks to the unprecedented growth in automated testing tools and continuous integration (CI) solutions for all manner of web projects, testing is now table stakes for any implementation. That said, many developers find automated testing to be an altogether intimidating area of exploration. Fortunately, when paired with a development culture that values quality assurance (QA), you can focus on adding business value instead of fixing issues day in and day out.

Three years ago, Yuriy Gerasimov (Senior Back-End Engineer at Tag1 Consulting) gave a talk at DrupalCon New Orleans about some of the key ideas that Drupal developers need to understand in order to implement robust testing infrastructures and to foster a testing-oriented development culture that yields unforeseen business dividends across a range of projects. In this four-part blog series, we summarize some of the most important conclusions from Yuriy’s talk. And in this third installment, we’ll take a closer look at two of the most essential parts of any testing toolkit: unit testing and functional testing.

Unit testing

Unit testing is a particularly fascinating topic, not only because many developers already know what user testing entails, but also because it enjoys a range of possible technologies that are readily available and that development teams can easily leverage. After all, unit testing is a commonly taught concept in universities. In its most reduced form, unit testing refers to the verification of a feature’s robustness by feeding it a variety of arguments, all of which test the limits of what is possible within the feature. Drupal developers, for instance, have access to a variety of unit tests for both core and contributed modules. And the best part of unit testing is that you can test functions and classes in isolation rather than evaluating large parts of the code.

The most optimal locations for unit tests are functions that are responsible for calculating some result. For instance, if you have a function that receives a set of arguments and that performs a series of calculations based on those arguments, you can feed unit tests arguments that evaluate whether the function works for a variety of possible inputs.

Unit tests as documentation

This also reveals one of the most interesting characteristics of unit tests. Because unit tests are focused on stretching the outer limits of what functions are capable of, they are also a convenient source of robust documentation. Yuriy admits that when he downloads external libraries for the first time in order to integrate them with his existing code, he often scrutinizes the unit tests, because they indicate what the code expects as input. When documentation is challenging to read or nonexistent, unit tests can be an ideal replacement for developers seeking to comprehend the purpose of certain code, because the unit tests are where developers ensure that logic operates correctly.

As an illustration of how unit tests can yield outsized benefits when it comes to well-documented code, consider the case of Drupal’s regular expressions. In Drupal 7, the regular expressions responsible for parsing .info files, which are used across both modules and themes, are extremely lengthy owing to the myriad demands on the files. Regular expressions, after all, are easy to write but difficult to understand. Though we are privileged as Drupal developers in that Drupal breaks regular expressions into separate, assiduously commented lines, many developers in the contributed ecosystem will avoid taking this additional step. For this reason, Yuriy recommends that all developers write unit tests for regular expressions that parse critical input.

Writing testable code is difficult but useful

Unit tests help developers think in a very different way about how to write software in a legible and understandable way. In Drupal 8, for instance, the new dependency injection container is particularly well-suited for unit tests, because we can swap services in and out of it. If you work with a database as an external service, for example, you can easily mock objects and ensure that your services work for those mock objects successfully.

To illustrate this, Yuriy cites the real-world example of the Services module in Drupal 7, which originally consisted of classical Drupal code but was subsequently coupled with unit tests. Yuriy witnessed through his addition of unit tests that he was able to parse different arguments entering certain Services functionality or inspect how routing functioned. With unit tests, ensuring the security of functions is much easier, because they can help you determine all of the arguments necessary to the code so that a call to globals or static variables is rendered unnecessary. Though unit testing requires considerable effort, Drupal 8 makes the process much easier for developers.

And now, with the introduction of PHPUnit into Drupal testing infrastructures, it is now even easier to test your code. The biggest part that was added is the capability to use mock objects that are presently the industry standard for unit testing.

Functional testing

As the name suggests, functional testing analyzes how actual users will interact with your software from a concrete rather than abstract standpoint. If you have an unlimited budget for your project, you can test every single last page on your website, but this is seldom—if ever—feasible from the perspective of project budgets. Whereas unit testing only requires perhaps ten to twenty percent of your development time, functional tests will require several days to write and noticeably more effort to implement.

Selling functional testing

In many of Yuriy’s previous projects, he was able to sell functional testing to customers by justifying the need to ensure that the software would be functional irrespective of the time of day or night. His teams had particular success selling functional testing in commerce projects, because stakeholders in commerce implementations are strongly invested in consumers successfully checking out at all times.

Often, the most challenging aspect of functional testing in commerce projects is the credit card details that customers must inevitably provide to the commerce platform. If you have a testing environment with all possible payment providers, however, customers no longer need to check to make sure that the checkout functions properly while they are at home or at the office. Your clients can simply click a single button in the continuous integration (CI) server and witness for themselves that the user flow is functional or configure notifications so that they are issued solely when that process breaks.

Maintenance costs of functional testing

Functional testing requires considerable maintenance, because it is primarily based on the document object model (DOM) of your website rather than abstract code. If you modify something on the website, you will need to revise your functional tests accordingly, including the selectors that you are targeting with functional tests. Yuriy warns that many may operate under the misconception that functional tests only require a mere twelve hours of implementation, but due to the unique attributes of each project, functional tests may require several days to implement.

As such, automating the process of functional testing is of paramount importance. Yuriy suggests that functional tests should not be made difficult for developers and especially project managers who are concerned about project quality. And as with other types of tests, if functional tests are not run on a regular basis, they are of limited usefulness.

Tools for functional testing

Luckily, there are many tools available for functional testing, the most notable among them being Behat. The Behat ecosystem makes a variety of extensions available, but it is by no means the only functional testing solution available in the landscape. Other solutions exist in the software-as-a-service (SaaS) space, including Selenium, which records clicks and provides a more visual representation of functional tests. For this reason, Yuriy recommends Selenium for junior developers with less exposure to functional testing.

Nevertheless, functional testing can fall victim to the vagaries of project budgets, and prospective expense remains among the most important considerations for customers interested in functional testing. To account for this, it is essential to consider which user flows are the most critical to the success of your implementation and to provide the appropriate level of coverage for those components. It is also a good idea to discuss at length with your customer what outcomes they would like to see and to ensure that the user flows they consider most crucial enjoy adequate coverage.

Conclusion

Thanks to the dual innovations engendered by unit testing and functional testing, you can have both an abstract and concrete view into how your code is performing in terms of quality with little to no overhead. With unit testing, which ensures that each designated function works properly with a variety of inputs that stretch the limits of what it can do, you can protect yourself from potential security vulnerabilities such as a lack of input sanitization. Functional testing, meanwhile, allows you to perform automated trials of your implementation to guarantee that your users are traversing the experience you intend.

In this blog post, we explored the boundaries of what unit testing and functional testing can offer you when it comes to a modern testing infrastructure and test-oriented development culture. In the fourth and final installment of this four-part blog series, we turn our attention to two of the areas of testing experiencing some of the most innovation as of late: visual regression testing and performance testing. As demands on our implementations shift, it is increasingly more important that we understand not only when changes impact user experiences but also how they perform under realistic loads.

Special thanks to Yuriy Gerasimov and Michael Meyers for their feedback during the writing process.

Please check out Part 1 and Part 2!

Jan 23 2020
Jan 23

Over the course of Drupal’s lengthy history, one of the most common feature requests has been automatic updates. A common complaint of Drupal site administrators, especially those who have smaller sites updated less frequently, is the frequently complex and drawn-out process required to update a Drupal site from one minor version to another. Updates can involve a difficult set of highly specific steps that challenge even the most patient among us. Indeed, many in the Drupal community simply choose to ignore the automatic e-mails generated by Drupal.org indicating the availability of a new version, and waiting can lead to compounding security vulnerabilities.

Fortunately, the era of frustration when it comes to automatic updates in Drupal is now over. As one of the roughly dozen Drupal Core Strategic Initiatives, Drupal automatic updates are a key feature that will offer Drupal users better peace of mind when minor releases occur. Over the last several years, Tag1 Consulting, well-known as leading performance and scalability experts in the Drupal community, has worked closely with the Drupal Association, MTech, and the Free and Open Source Software Auditing (FOSSA) program at the European Commission to make automatic updates in Drupal a reality.

Recently, I (Preston So, Editor in Chief at Tag1 and author of Decoupled Drupal in Practice) sat down with Lucas Hedding (Senior Architect and Data and Application Migration Expert at Tag1), Fabian Franz (Senior Technical Architect and Performance Lead at Tag1), Tim Lehnen (CTO at the Drupal Association), and Michael Meyers (Managing Director at Tag1) to host a Tag1 Team Talks episode about the story of Tag1’s involvement in the automatic updates strategic initiative. In this blog post, we dive into some of the fascinating background and compelling features in Drupal’s new automatic updates, as well as how this revolutionary feature will evolve in the future.

What are automatic updates in Drupal?

Listed as one of the Drupal Core Strategic Initiatives for Drupal 9, Drupal’s automatic updates are intended to resolve some of the most intractable usability issues in maintaining Drupal sites. Updating Drupal sites can be a challenging, tedious, and costly process. Building an automatic updater for Drupal is a similarly difficult problem, with a variety of potential security risks, but it’s a problem that other ecosystems have solved successfully. Following Dries’ announcement of automatic updates as a strategic priority, several early architectural discussions took place, especially at Midwest Drupal Summit 2018 in Ann Arbor.

Automatic updates in Drupal provide certain key benefits for users of all shapes and sizes who leverage Drupal today, including individual end users, small- to medium-sized agencies, and large enterprises. Advantages that apply to all users across the spectrum include a reduction in the total cost of ownership (TCO) for Drupal sites and especially a decrease in maintenance costs.

As for small- and medium-sized agencies and individual site owners, it can be difficult—and deeply disruptive and anxiety-inducing—to mobilize sufficient resources in a brisk timeframe to prepare for Drupal security releases that typically occur on Wednesdays. For many end users and small consultancies who lack experience with keeping their Drupal sites up to date, high-alert periods on Wednesday can be deeply stressful. And for enterprise users, how to incorporate security updates becomes a more complex discussion: Should we integrate manual updates into our security reviews or keep adhering to continuous integration and continuous deployment (CI/CD) processes already in place?

Where are Drupal’s automatic updates today?

The full roadmap for Drupal’s automatic updates is available on Drupal.org for anyone to weigh in, but in this blog post we focus on its current state and long-term future. Automatic updates in Drupal include updates on production sites as well as on development and staging environments, although some integration with existing CI/CD processes may be required. In addition, automatic updates support both Drupal 7 and Drupal 8 sites.

Because of the ambitious nature of the automatic updates initiative, as well as the desire by the module’s maintainers to undertake a progressive approach from an initial module in the contributed ecosystem to a full experimental module in Drupal core, the development process has been phased from initial architecture to present. Currently, a stable release is available that includes features like public safety alerts and readiness checks.

As for other developments within the scope of available funding from the European Commission, in-place automatic updates have also arrived. If a critical security release is launched, and your site has the automatic updates module installed, you’ll receive an e-mail notification stating that an update is forthcoming in the next several days. Once the update is available, the module will then automatically execute the in-place automatic update if all readiness checks show as green on the Drupal user interface, meaning that no additional action is required on the user’s part.

Key features of Drupal automatic updates

Together with MTech, the Drupal Association, and the European Commission, Tag1 has been heavily involved in architecting the best and most graceful approach, particularly in such a way that it can be generalized for and leveraged by other open-source software projects in the PHP landscape. This includes approaches seen in other ecosystems such as readiness checking, file downloading, and signature verification that generates “quasi-patches” as well as inspiration from the WordPress community. One of the team’s major concerns in particular is ensuring the continuous integrity of update packages such that users can be confident that such packages are installed from a trusted source.

There are three key features available as part of automatic updates in Drupal that will be part of the initial release of the module, and we discuss each of these in turn here.

Public safety messaging

After the noted security vulnerability in 2014 commonly known as “Drupalgeddon,” a notice was posted indicating that a critical release was forthcoming. When it comes to automatic updates, a similar process would occur: Several days before a critical security release for Drupal core or for a contributed project in Drupal, a notice would be posted several days prior and available on every Drupal site.

This sort of public safety messaging allows for an additional communication mechanism before a key update so that site owners can ensure they are ready for an update to land. In Drupal sites, the feed of alerts originate from the same security advisories (SAs) that the Drupal Security Team and Drupal’s release managers issue.

Readiness or “preflight” checks

Every Drupal site with automatic updates installed will also have readiness checks, also known as “preflight” checks, that run regularly every six hours through Drupal’s cron and will inform site owners if their site is prepared for an automatic update. Readiness checks are essential to ensure that sites are not functionally compromised after an automatic update.

For instance, if Drupal core has been hacked by a developer, if a site is running on a read-only filesystem, or if there are foregoing database updates that need to be run, readiness checks will indicate that these issues need resolutions before a site can automatically update. There are eight or nine readiness checks available currently, and some are simple warnings to aid the user (e.g. in situations where cron is running too infrequently to update the site automatically in a timely fashion), while others are errors (e.g. the filesystem is read-only and cannot be written to). Whereas warnings will not impede the commencement of an automatic update, errors will.

In-place updates

The final crucial component for automatic updates is in-place updates, the centerpiece of this new functionality. The in-place updates feature in Drupal’s automatic updates downloads a signed ZIP archive from Drupal.org. Using the libsodium library, the feature then compares the signature of the ZIP file to verify that the resulting archive matches Drupal.org’s official archive.

Thereafter, in-place updates will back up all files that are slated for update and update the indicated files. If the process concludes successfully, the site issues a notification to the user that the site has been upgraded. If something fails during the procedure, in-place updates will restore the available backup.

Common questions about automatic updates

On the recent Tag1 Team Talks episode about automatic updates in Drupal, contributors from Tag1 and the European Commission answered some of the most important questions on every Drupal user’s mind as the initiative continues to roll out automatic updates features.

What about using Composer versus tarballs?

One of the key differences between Drupal implementations today is the use of the Composer command-line interface to handle Drupal’s installed modules in lieu of managing module installations through tarballs. Due to the widening use of Composer in the Drupal community, if a site has updated to Drupal 8.8.0 or later, the site will be using Composer. And if the two key Composer-related files in Drupal codebases (namely composer.json and composer.lock) are not modified, automatic updates will continue to function properly. However, for sites leveraging Composer and subsequently modifying the /vendor directory in Drupal codebases, this question becomes more complicated.

At present, the automatic updates team will release early versions supporting all scenarios for Drupal sites, short of those sites that have modified composer.json and composer.lock directly. By observing users as they gradually adopt automatic updates, the team plans to learn much about how users juggle Drupal dependencies in order to release improved update methods that accommodate Composer much more gracefully.

Are automatic updates part of Drupal core?

As of now, automatic updates are not part of a vanilla Drupal installation, but all major components of the module will be incorporated into Drupal core in due course. The in-place updates feature presents the most overt difficulties.

Before in-place updates can land in core, the automatic update team plans to implement an A/B front-end controller that is capable of swapping between two full codebases and toggle back to the backed-up, out-of-date codebase if the update exposes certain issues mid-flight.

What is the future of automatic updates?

While the European Commission has funded the first twelve months of work over the course of 2019, there is much more work to do. The initial European Commission funding accounts for the three aforementioned key features, namely readiness checking, the delivery of update “quasi-patches,” and a robust package signing system, all focused on security updates, which are the most pressing. However, the current year of development excludes better support for Composer and contributed projects.

The longer-term roadmap for automatic updates includes the A/B front-end controller mentioned in the previous section, more robust support for Composer-powered sites, and other types of updates. These include updates for contributed modules and themes as well as batched successful updates for sites that have fallen particularly behind.

Conclusion

Automatic updates will reinvent how we maintain and upgrade Drupal sites, particularly in the realm of security. Because they allow novice and experienced Drupal users alike to save time without the need to worry about how they will implement updates, this strategic initiative improves the total cost of ownership for Drupal users of all sizes and backgrounds.

No account of the extraordinary initiative that is Drupal’s automatic updates would be complete without appreciation for the sponsors of the developers involved, especially from the Drupal Association, MTech, Tag1 Consulting, and the European Commission’s FOSSA program. Organizations and individuals alike have sponsored automatic updates in Drupal to widen awareness of their brands, to showcase their skills as developers, and to attract other Drupal contributors and resource Drupal teams.

To sponsor the continued success of Drupal’s automatic updates, please consider sponsoring development by contacting the Drupal Association. And for more insight into automatic updates directly from the module’s creators, check out our recent Tag1 Team Talks episode on the topic for information we were unable to fit in this blog post.

Special thanks to Fabian Franz, Lucas Hedding, Tim Lehnen, and Michael Meyers for their feedback during the writing process.

Click the following link for our Tag1 Team Talk on Drupal Automatic Updates!

Jan 22 2020
Jan 22

WebRTC, a protocol that facilitates peer-to-peer communication between two clients via the browser, is now supported by all modern browsers. Since its introduction it has mainly been used for web conferencing solutions, but WebRTC is ideal for a variety of other use cases as well. Because of its wide platform support, creating peer-to-peer applications for the web is now more straightforward than ever. But how do you manage many people working together at the same time on the same data? After all, conflict resolution for peer-to-peer applications remains a challenging problem. Fortunately, with Yjs, an open-source framework for real-time collaboration, developers can now combine WebRTC and Yjs to open the floodgates to a range of future-ready collaborative use cases.

Thanks to WebRTC and Yjs, anyone can build collaborative editing into their web application, and this includes more than just text Yjs enables collaborative drawing, drafting, and other innovative use cases. The advantage of such a peer-to-peer model (in lieu of a client–server model) in the CMS world is that collaborative editing can be added to any editorial interface without significant overhead or a central server handling conflict resolution. By integrating with y-webrtc, the Yjs connector for WebRTC, CMS communities can easily implement collaborative editing and make it natively available to all users, whether on shared hosting or in the enterprise. The future of Drupal, WordPress, and other CMSs is collaborative, and, together, WebRTC and Yjs enable collaborative editing out of the box.

In this deep dive into how Yjs enables peer-to-peer collaboration, join Kevin Jahns (Real-Time Collaboration Systems Lead at Tag1 and creator of Yjs), Fabian Franz (Senior Technical Architect and Performance Lead at Tag1), Michael Meyers (Managing Editor at Tag1), and Preston So (Editor in Chief at Tag1 and author of Decoupled Drupal in Practice) for a closer look at how you too can build peer-to-peer collaboration into your decentralized application.

[embedded content]
Jan 22 2020
Jan 22

WebRTC, a protocol that facilitates peer-to-peer communication between two clients via the browser, is now supported by all modern browsers. Since its introduction it has mainly been used for web conferencing solutions, but WebRTC is ideal for a variety of other use cases as well. Because of its wide platform support, creating peer-to-peer applications for the web is now more straightforward than ever. But how do you manage many people working together at the same time on the same data? After all, conflict resolution for peer-to-peer applications remains a challenging problem. Fortunately, with Yjs, an open-source framework for real-time collaboration, developers can now combine WebRTC and Yjs to open the floodgates to a range of future-ready collaborative use cases.

Thanks to WebRTC and Yjs, anyone can build collaborative editing into their web application, and this includes more than just text Yjs enables collaborative drawing, drafting, and other innovative use cases. The advantage of such a peer-to-peer model (in lieu of a client–server model) in the CMS world is that collaborative editing can be added to any editorial interface without significant overhead or a central server handling conflict resolution. By integrating with y-webrtc, the Yjs connector for WebRTC, CMS communities can easily implement collaborative editing and make it natively available to all users, whether on shared hosting or in the enterprise. The future of Drupal, WordPress, and other CMSs is collaborative, and, together, WebRTC and Yjs enable collaborative editing out of the box.

In this deep dive into how Yjs enables peer-to-peer collaboration, join Kevin Jahns (Real-Time Collaboration Systems Lead at Tag1 and creator of Yjs), Fabian Franz (Senior Technical Architect and Performance Lead at Tag1), Michael Meyers (Managing Editor at Tag1), and Preston So (Editor in Chief at Tag1 and author of Decoupled Drupal in Practice) for a closer look at how you too can build peer-to-peer collaboration into your decentralized application.

[embedded content]
Jan 22 2020
Jan 22

Automated tests are rapidly becoming a prerequisite for successful web projects, owing to the proliferation of automated testing tools and an explosion of continuous integration (CI) services that ensure the success of web implementations. Nonetheless, for many developers who are new to the space, automated testing can be an intimidating and altogether novel area that causes more than a few groans at weekly meetings. Luckily, with the right development culture and testing infrastructure in place, your team can focus on implementing new features rather than worrying about the quality of their code.

Yuriy Gerasimov (Senior Back-End Engineer at Tag1 Consulting) delivered a presentation at DrupalCon New Orleans about automated testing and its advantages for web projects of all shapes and sizes. In this four-part blog series, we explore some of the essentials that all developers should be aware of as they explore automated testing as well as the key fundamentals you need to know to incorporate automated testing into your daily workflows. In this second installment, we inspect how to implement a robust testing infrastructure and how to cultivate a development culture favorable to automated testing with the help of code checks.

Key elements of automated testing

Over the course of his career, Yuriy has interacted with a variety of consultancies to gather anecdotal evidence of how automated testing has worked across a range of scenarios. What he discovered is that most orgs tend to implement testing infrastructures and automated testing on a project-by-project basis rather than incorporating it as a part of their regular process for every project. This betrays a fundamental disconnect between the value that automated testing can provide and the inevitable inefficiencies that arise when automated testing is added to individual projects on an ad-hoc basis.

Implementing robust testing infrastructures

One of the classic problems that arise from such a situation is the notion of development teams only implementing a continuous integration (CI) server when customers are able to provide for it in the project budget. In other words, either you can build a server that extends a development culture centered around automated testing, or you risk a scenario in which shared characteristics and shared testing components are absent across projects, requiring you to bootstrap a new infrastructure every time.

Even upon improving quality in your projects thanks to a robust testing infrastructure, code reviews are essential and will allow your interactions between your team’s developers to be elevated to a higher caliber. Unfortunately, however, developers tend not to share substantial knowledge after they complete the features they are tasked to complete. If a developer sees a colleague not following best practices, code reviews can foster improvements in the code thanks to the knowledge that all parties gain in the process. Because of this, Yuriy suggests that development teams leverage a source control provider like GitHub, Bitbucket, or GitLab that incorporates built-in peer reviews functionality.

Fostering a development culture conducive to testing

Development culture is also essential to ensure the success of automated testing. This means that all developers should understand how the testing infrastructure functions in order to guard against regressions. When deployments are not tied to individual team members, for instance, this means that all members of the team understand how deployment occurs and are thus able to implement improvements themselves. For this reason, we discourage blocking deployments on a single function or individual contributor.

The optimal situation is one in which even a project manager who does not write code is capable of initializing deployments and kicking off a series of automated tests. When deployment is automated in this way to the point where even team members uninvolved in development can understand how quality is assessed across the project, this can level up the skill sets of the entire team.

For example, Yuriy recommends that every new developer on a team conduct a deployment themselves in isolation from the rest of the team. By doing so, the least experienced individual contributor may encounter inefficiencies theretofore unaccounted for by other team members and catalyze improvements in the quality of the code. When collaborators who are not yet on-boarded are able to foster advancement in the testing infrastructure across the team, the benefits can not only enrich the automated tests themselves but also cultivate a highly improved development culture across the board.

Considering maintenance costs

Nonetheless, maintenance costs are an important facet of automated testing to consider for any implementation, large or small, because they may be sufficiently exorbitant to encourage recalibration in the testing infrastructure. Some of the key questions to ask include: Do you have enough people to maintain the system from end to end? Do you have a dedicated systems administrator or DevOps specialist who can correct issues when discovered?

After all, testing infrastructures tend to be the components of projects that are scrutinized the least once they are complete—this is part of the blessing and curse of the notion of a “one and done” mindset. In the end, every project has different requirements, and other upcoming projects may demand different systems or other approaches to the same problem. When selecting automated testing systems, therefore, it is essential to consider their impact on the maintenance costs that your team will inevitably incur.

Code checks

Among the easiest and simplest to implement, code checks are static analyses of code that are not only educational about the code’s quality itself but also can perform very quickly unless your codebase includes hundreds of megabytes of code. As Yuriy notes, in that case, you have other problems to solve first. For many Drupal development teams, code checks for adherence to Drupal coding standards are the first line of defense against potential regressions.

By the same token, security checks, which evaluate the risk of potential vulnerabilities, are also critical. Security checks are capable of verifying that certain best practices are followed when it comes to possible attack vectors, such as the use of globals or session variables in key functions or the allowance of input into deeper recesses of Drupal without proper sanitization. These checks are also convenient in that in many cases, less experienced developers can run security checks and understand the implications of the results without consulting a more senior developer. Along these same lines, linters, which check for syntax errors and the like, can be hugely beneficial for front-end developers.

Complexity metrics and copy-paste detection

Another fascinating selling point for code quality is complexity metrics, which are comprised of assessments of how complex the code is. Among the most important of these is cyclomatic complexity. Consider a scenario in which you have written a function that contains a foreach cycle with multiple control structures (many if-statements) nested within. If your function has many levels of nesting, this can present problems not only in code readability and the likelihood of introducing bugs but also in maintenance. Code checks that analyze cyclomatic complexity can help you to uncover situations in which others would have a horrible experience reading your code by limiting the number of levels that code can be nested (e.g. no more than five levels). Such complexity metrics will aid you in isolating certain logic into other functions or exiting early from your cycles to help your code become more legible.

Finally, copy-paste detection is another hugely useful element of code checking that allows for you to encounter inefficiencies in code. Some developers, for better or worse, often copy and paste code from examples or Stack Overflow responses without necessarily considering how it can best be incorporated into the existing codebase. Copy-paste detection can thus inspect the code to detect code pasted in multiple places; if you use the same piece of code in multiple locations, it may be best to abstract it out by isolating it into another function instead.

Conclusion

All told, code checks are often so immediate that they can take mere fractions of a second. For this reason, they are a ubiquitous element of automated testing and can allow developers to become more productive in a short period of time. In this way, your team can create not only a robust underlying testing infrastructure well-suited for repeatability but also ensure the longevity of a testing-oriented development culture that values consistent code quality.

In this blog post, we covered some of the most crucial elements of any automated testing approach for projects both large and small, namely a robust testing infrastructure and a focused development culture for automated testing. In turn, we looked at the outsized benefits that code checks and security checks can have on your codebase. In the next installment of this four-part blog series, we will devote our attention to some of the other essentials in the testing toolbox: unit testing and functional testing.

Special thanks to Yuriy Gerasimov, Jeremy Andrews, and Michael Meyers for their feedback during the writing process.

Jan 22 2020
Jan 22

Automated tests are rapidly becoming a prerequisite for successful web projects, owing to the proliferation of automated testing tools and an explosion of continuous integration (CI) services that ensure the success of web implementations. Nonetheless, for many developers who are new to the space, automated testing can be an intimidating and altogether novel area that causes more than a few groans at weekly meetings. Luckily, with the right development culture and testing infrastructure in place, your team can focus on implementing new features rather than worrying about the quality of their code.

A few years back, Yuriy Gerasimov(Senior Back-End Engineer at Tag1 Consulting) delivered a presentation at at DrupalCon New Orleans about automated testing and its advantages for web projects of all shapes and sizes. In this four-part blog series, we explore some of the essentials that all developers should be aware of as they explore automated testing as well as the key fundamentals you need to know to incorporate automated testing into your daily workflows. In this second installment, we inspect how to implement a robust testing infrastructure and how to cultivate a development culture favorable to automated testing with the help of code checks.

Key elements of automated testing

Over the course of his career, Yuriy has interacted with a variety of consultancies to gather anecdotal evidence of how automated testing has worked across a range of scenarios. What he discovered is that most agencies tend to implement testing infrastructures and automated testing on a client-by-client basis rather than incorporating it as a part of their regular process for every project they come across. This betrays a fundamental disconnect between the value that automated testing can provide and the inevitable inefficiencies that arise when automated testing is added to individual projects on an ad-hoc basis.

Implementing robust testing infrastructures

One of the classic problems that arise from such a situation is the notion of development teams only implementing a continuous integration (CI) server when customers are able to provide for it in the project budget. In other words, either you can build a server that extends a development culture centered around automated testing, or you risk a scenario in which shared characteristics and shared testing components are absent across projects, requiring you to bootstrap a new infrastructure every time.

Even upon improving quality in your projects thanks to a robust testing infrastructure, code reviews are essential and will allow your interactions between your team’s developers to be elevated to a higher caliber. Unfortunately, however, developers tend not to share substantial knowledge after they complete the features they are tasked to complete. If a developer sees a colleague not following best practices, code reviews can foster improvements in the code thanks to the knowledge that all parties gain in the process. Because of this, Yuriy suggests that development teams leverage a source control provider like GitHub, Bitbucket, or GitLab that incorporates built-in peer reviews functionality.

Fostering a development culture conducive to testing

Development culture is also essential to ensure the success of automated testing. This means that all developers should understand how the testing infrastructure functions in order to guard against regressions. When deployments are not tied to individual team members, for instance, this means that all members of the team understand how deployment occurs and are thus able to implement improvements themselves. For this reason, we discourage blocking deployments on a single function or individual contributor.

The optimal situation is one in which even a project manager who does not write code is capable of initializing deployments and kicking off a series of automated tests. When deployment is automated in this way to the point where even team members uninvolved in development can understand how quality is assessed across the project, this can level up the skill sets of the entire team.

For example, Yuriy recommends that every new developer on a team conduct a deployment themselves in isolation from the rest of the team. By doing so, the least experienced individual contributor may encounter inefficiencies theretofore unaccounted for by other team members and catalyze improvements in the quality of the code. When collaborators who are not yet on-boarded are able to foster advancement in the testing infrastructure across the team, the benefits can not only enrich the automated tests themselves but also cultivate a highly improved development culture across the board.

Considering maintenance costs

Nonetheless, maintenance costs are an important facet of automated testing to consider for any implementation, large or small, because they may be sufficiently exorbitant to encourage recalibration in the testing infrastructure. Some of the key questions to ask include: Do you have enough people to maintain the system from end to end? Do you have a dedicated systems administrator or DevOps specialist who can correct issues when discovered?

After all, testing infrastructures tend to be the components of projects that are scrutinized the least once they are complete—this is part of the blessing and curse of the notion of a “one and done” mindset. In the end, every project has different requirements, and other upcoming projects may demand different systems or other approaches to the same problem. When selecting automated testing systems, therefore, it is essential to consider their impact on the maintenance costs that your team will inevitably incur.

Code checks

Among the easiest and simplest to implement, code checks are static analyses of code that are not only educational about the code’s quality itself but also can perform very quickly unless your codebase includes hundreds of megabytes of code. As Yuriy notes, in that case, you have other problems to solve first. For many Drupal development teams, code checks for adherence to Drupal coding standards are the first line of defense against potential regressions.

By the same token, security checks, which evaluate the risk of potential vulnerabilities, are also critical. Security checks are capable of verifying that certain best practices are followed when it comes to possible attack vectors, such as the use of globals or session variables in key functions or the allowance of input into deeper recesses of Drupal without proper sanitization. These checks are also convenient in that in many cases, less experienced developers can run security checks and understand the implications of the results without consulting a more senior developer. Along these same lines, linters, which check for syntax errors and the like, can be hugely beneficial for front-end developers.

Complexity metrics and copy-paste detection

Another fascinating selling point for code quality is complexity metrics, which are comprised of assessments of how complex the code is. Among the most important of these is cyclomatic complexity. Consider a scenario in which you have written a function that contains a foreach cycle with multiple control structures (many if-statements) nested within. If your function has many levels of nesting, this can present problems not only in code readability and the likelihood of introducing bugs but also in maintenance. Code checks that analyze cyclomatic complexity can help you to uncover situations in which others would have a horrible experience reading your code by limiting the number of levels that code can be nested (e.g. no more than five levels). Such complexity metrics will aid you in isolating certain logic into other functions or exiting early from your cycles to help your code become more legible.

Finally, copy-paste detection is another hugely useful element of code checking that allows for you to encounter inefficiencies in code. Some developers, for better or worse, often copy and paste code from examples or Stack Overflow responses without necessarily considering how it can best be incorporated into the existing codebase. Copy-paste detection can thus inspect the code to detect code pasted in multiple places; if you use the same piece of code in multiple locations, it may be best to abstract it out by isolating it into another function instead.

Conclusion

All told, code checks are often so immediate that they can take mere fractions of a second. For this reason, they are a ubiquitous element of automated testing and can allow developers to become more productive in a short period of time. In this way, your team can create not only a robust underlying testing infrastructure well-suited for repeatability but also ensure the longevity of a testing-oriented development culture that values consistent code quality.

In this blog post, we covered some of the most crucial elements of any automated testing approach for projects both large and small, namely a robust testing infrastructure and a focused development culture for automated testing. In turn, we looked at the outsized benefits that code checks and security checks can have on your codebase. In the next installment of this four-part blog series, we will devote our attention to some of the other essentials in the testing toolbox: unit testing and functional testing.

Special thanks to Yuriy Gerasimov, Jeremy Andrews, and Michael Meyers for their feedback during the writing process.

Jan 17 2020
Jan 17

With the release of Drupal 8.8, Drush is also due for an upgrade — to Drush 10. For this venerable command-line interface that many Drupal developers know intimately well, what does the present and future look like? What considerations should we keep in mind when selecting Drupal Console or Drush? What new features are available in Drush 10 that characterize the new CI/CD approaches we see expanding in the Drupal community?

In this Tag1 Team Talk, join the creator and maintainer of Drush Moshe Weitzman (Senior Technical Architect at Tag1), Fabian Franz (Senior Technical Architect and Performance Lead at Tag1), Preston So (Editor in Chief at Tag1), and Michael Meyers (Managing Director at Tag1) for a journey through Drush’s history and promising future. We take a deep look at what made Drush what it is today, the most compelling features in Drush 10, and how a hypothetical Drush in core could look.

[embedded content]
Jan 15 2020
Jan 15

Testing has become an important topic in recent years thanks to the explosion of testing technologies and continuous integration (CI) approaches but also due to the need for an ever-widening range of tests for a variety of use cases. For many developers, understanding how to incorporate testing into their development workflows can be daunting due to the many terms involved and, worse yet, the many tools available both in software-as-a-service (SaaS) companies and in open-source ecosystems like Drupal.

Yuriy Gerasimov (Senior Back-End Engineer at Tag1 Consulting) presented a session at DrupalCon New Orleans about modern testing approaches and how to decide on the correct suite of tests for your software development workflows. In this four-part blog series, we analyze the concepts in contemporary testing approaches that you need to know in your day-to-day and why they can not only protect but also accelerate your project progress. In this first installment, we take a look at how to sell testing as an important component of client (your stakeholders) projects, as well as why automated testing is an essential component of any web implementation.

Why testing?

Many people around the web development landscape have heard of testing, but when you ask about real-world examples on real-life projects, the same developers admit that their testing infrastructures are sorely lacking. The most important reason for this is that while testing is a compelling value-add for developers, it can be difficult to incorporate testing as a required line item in projects, especially when business stakeholders are looking to save as much money as possible. How to sell testing to clients continues to be a challenging problem, especially because requesting that a client pay for an additional 100 hours on top of the 500 already incurred can be anathema to their sense of frugality.

After all, many customers will respond by arguing that by choosing you as an agency or developer, they already trust you to foster a high-quality outcome. As such, many clients will ask, “Why do we need to spend extra money on testing?” Without the overt benefit that project components like architectural workshops and actual implementation provide, testing is often the forgotten and most easily abandoned stage of a build.

A real-world example of the need for testing

In his talk at DrupalCon New Orleans, Yuriy describes a large project (prior to his time at Tag1) on which he collaborated with many other people on a development team tasked with finishing the implementation in six months. The project was for a local municipality, with many integration points and features. Every feature needed to work perfectly, including critical features for civic life such as applying for permits, and tolerance for dysfunction was low.

By the end of the project, originally slated for six months, Yuriy’s development team ultimately spent six months developing and an additional six months fixing issues and testing functionality. Fortunately for his team, the municipality had already been through a project whose timeline ballooned out of control, and the team was able to deliver the project within a year as opposed to the previous partner, who spent two years on the same project.

One of the most alarming aspects of the project at the time was that all of the testing the team had done until that moment consisted of manual testing sessions. A meeting was convened, and every developer stood up, responsible for describing the rationale for each feature they had built and demonstrating the feature. Every team member would then test each constituent feature and fix issues on the spot, in the moment.

Learning from past mistakes

As one can imagine, this manual testing approach is highly unsustainable for projects that require tight timelines with a high degree of confidence. Yuriy learned many lessons from the project, and in a subsequent implementation six months later, in which he and his collaborators built an application for people with hearing and speech difficulties, he made considerable changes. The project was complex, with several servers communicating with the application through REST APIs and a high expectation for a user experience that would allow users to click icons representing elocutions that would speak in their place.

From the beginning, Yuriy and his team baked in automated testing up front to test communication with the REST APIs and ensure all requests were functioning properly. They built the project to be scalable because they knew that many users would be using the application simultaneously. In the end, the quality assurance (QA) overhead was minimal, because developers on the team could simply run automated tests and show the result to the client. Even though the size of the project was roughly the same, having built-in automated testing with acceptance criteria was a benefit difficult to overstate.

Defending quality: Selling testing to customers

When testing aficionados attempt to sell testing to customers, they must frame the investment in terms of quality and long term vs. short term costs (failing to deal with this in the short term will actually cost you more in the long term). However, it is admittedly difficult to sell something when its success cannot be measured. After all, from the client perspective, a buyer is selecting a vendor based on the quality with which they implement projects. But there are only anecdotal metrics that indicate whether an organization performs better than another with high-quality projects. For this reason, it is essential that developers interested in selling testing as part of their contracts offer metrics that are comprehensible to the customer.

In the end, the sole concern of customers is that software is delivered without bugs. While ease of maintenance is also important, this is generally considered table-stakes among stakeholders (or a problem for the future). In order to provide a high degree of confidence for issue-free builds, we need metrics for traits like performance and code quality (like adherence to Drupal’s coding standards). Thus, when a customer asks about the justification of a metric such as code quality, we can show the results of tools like code audits, which in Drupal consist of a single Drush command that generates a full report. By performing a code audit on a codebase written by a less experienced team, for example, clients can be sold immediately on the value of your team by seeing the results of a highly critical code audit—and will seldom be opposed to your team winning the contract.

Automated testing

For many developers who are new to the concept of automated testing, the term can unleash a torrent of anxiety and concern about the massive amount of work required. This is why Yuriy recommends, first and foremost, building a testing infrastructure and workflow that demands minimum effort while yielding maximum dividends. Nonetheless, successful automated testing requires a robust testing infrastructure and a healthy development culture that is supportive. Without these elements, success is seldom guaranteed.

Fortunately, the up-front cost of automated testing is low owing to the “one and done” nature of automated testing. Though it’s likely you’ll spend a few weeks building out the infrastructure, there is no need to repeat the same process over and over again. Nevertheless, Yuriy recommends exploring the approaches that other software companies and industries undertake to understand how they tackle similar requirements. For example, automated testing for the C language has been around for many years. Moreover, there is no need to write our own continuous integration (CI) server, thanks to the wide variety of services available on the market, including software-as-a-service (SaaS) solutions that charge as little as $50 per month.

Even if you have written a large number of tests for your project, one of the most important aspects of automated testing may seem particularly obvious. If you don’t run automated tests regularly, you won’t receive any benefits. For instance, it is certainly not adequate to inform your customer that you have implemented automated testing unless you are running said tests weekly or monthly based on the project requirements. Otherwise, the value of the time you have spent implementing automated tests becomes questionable.

Conclusion

As you can see, testing is frequently the most easily forgotten component of web projects due to the extent to which clients question its value. However, armed with the right approaches to selling tests, you too can cultivate a culture of quality assurance (QA) not only within your development team but also for your business’s customers. With the help of automated testing, you can reduce the headaches for your team down the road and justify additional time that means extra money in your pocket.

In this blog post, we covered some of the important aspects of modern testing approaches and why customers are beginning to take a second look at the importance of quality in their projects. In the second installment of this four-part blog series, we’ll turn our attention to implementing testing infrastructures and fostering a development culture favorable to testing within your existing projects. We’ll discuss some of the maintenance costs associated with implementing automated testing and begin to look at two of the most prominent areas of testing: code checks and unit testing.

Special thanks to Yuriy Gerasimov, Jeremy Andrews, and Michael Meyers for their feedback during the writing process.

Dec 06 2019
Dec 06

We are excited to share details about our recent integration with the Group module.

By default, Group module stores group memberships in the Drupal database. For a recent project, our Client required that group memberships are canonically stored in their existing LDAP directory. That way, memberships may be re-used to control access to 3rd party applications like real-time chat, file sharing, etc.

Group module provides a simple service, GroupMembershipLoader, that is responsible for loading all memberships for a group or all memberships for a user. We swapped that service for a custom one that queries LDAP instead of querying database. Further, we added a simple caching layer so that a user’s group memberships are cached in that user’s session.

// Cache a user’s own groups in their session for fast retrieval.
$duration = \Drupal::getContainer()->getParameter('ldap_directory.dgids_duration');
$this->getPrivateTempStore($account, $duration)->get('ldap_directory')->set('dgids', $dgids);

A group membership not only relates a user to a group but also assigns a role to that membership. In our implementation, a member can be a Reader, Contributor, or a Group Admin. We decided to model that by creating a DirectoryGroup content entity, and configuring Group module such that DirectoryGroups can become group content. So, when a new Group is created, an insert hook does the following:

  1. 3 LDAP groups are programmatically created
  2. 3 DirectoryGroup content entities are created+affiliated with the new Group.
  3. The group creator is added to the Group as an Admin member.

That last step hints at another part of the custom integration. The Group::addMember and Group::removeMember methods control what happens when memberships are created and removed. We overrode the default implementation in our Group bundle class and perform LDAP operations instead of DB operations. Then we clear that user’s Tempstore in order that they immediately see the new membership.

Judicious use of cache tags insures that this list is invalidated when the user is added or removed from Groups.

// Invalidate that user's group memberships
Cache::invalidateTags($account->getCacheTags());

Also, we are pulling user avatars and other profile info from LDAP. Avatars are cached locally at full resolution and we may then use Image Styles to scale them down as needed. Drupal’s re-use of LDAP data demonstrates that Drupal is a good enterprise citizen.

Kudos to kristiaanvandeneynde for the excellent architecture of group module that made this integration possible without any patches to Group.

Nov 13 2019
Nov 13
[embedded content]

Yjs, one of the most powerful and robust frameworks for real-time collaborative editing, enables developers to add shared editing capabilities to any application with relatively little effort. In order to make it so easy to use and extend Yjs, the framework abstracts all the complexities, many moving pieces, and deep technical concepts involved in empowering offline first, peer to peer, real time collaboration.

In this Tag1 Team Talk, we continue our deep dive into Yjs with the founder and project lead of this collaborative editing framework to learn more about how it enables not only collaborative text editing but also collaborative drawing, collaborative 3D modeling, and other compelling use cases. In particular, we focus on the three core features that make up any great collaborative editing application: awareness, offline editing, and versioning with change histories.

Join Kevin Jahns (Real-Time Collaboration Systems Lead at Tag1 Consulting and Founder and Project Lead of Yjs), Fabian Franz (Senior Technical Architect and Performance Lead at Tag1 Consulting), Michael Meyers (Managing Director at Tag1 Consulting), and moderator Preston So (Contributing Editor at Tag1 Consulting and Principal Product Manager at Gatsby) for the second part of our deep dive series on Yjs directly from its creator (and excited learners).

A Deep Dive into Yjs Part 1
Evaluating Real Time Collaborative Editing Solutions for a Top Fortune 50 Company
Modern Rich Text Editors: How to Evaluate the Evolving Landscape
Nov 12 2019
Nov 12
[embedded content]

Yjs is a very compelling choice when it comes to building real-time collaborative applications. A powerful open-source, offline first, peer to peer, shared editing framework that is modular and extensible, Yjs enables developers to easily add real time collaborative capabilities to any type of application. Rich text editing, drawing, 3d modeling... the list of potential use cases for Yjs is lengthy and remarkable. But how did it get started, what is the algorithm it’s based on, and what does the future hold for Yjs? In this Tag1 Team Talk, hear directly from Kevin Jahns, the creator of Yjs, as we dive deeply into the foundations of Yjs and where it’s headed.

Join moderator Preston So (Contributing Editor, Tag1 Consulting) and guests Kevin Jahns (Real Time Collaboration Systems Lead, Tag1; Creator of Yjs), Fabian Franz (Senior Technical Architect and Performance Lead, Tag1), and Michael Meyers (Managing Director, Tag1) for an insider’s perspective on the past, present, and future of Yjs.

A Deep Dive into Yjs Part 2
Evaluating Real Time Collaborative Editing Solutions for a Top Fortune 50 Company
Modern Rich Text Editors: How to Evaluate the Evolving Landscape
Nov 11 2019
Nov 11

Table of Contents

What makes a collaborative editing solution robust?
Decentralized vs. Centralized Architectures in Collaborative Editing
Operational Transformation and Commutative Replicated Data Types (CRDT)
Why Tag1 Selected Yjs
Conclusion

In today’s editorial landscape, content creators can expect not only to touch a document countless times to revise and update content, but also to work with other writers from around the world, often on distributed teams, to finalize a document collaboratively and in real time. For this reason, collaborative editing, or shared editing, has become among the most essential and commonly requested features for any content management solution straddling a large organization.

Collaborative editing has long existed as a concept outside the content management system (CMS). Consider, for example, Google Docs, a service that many content creators use to write content together before copy-and-pasting the text into form fields in a waiting CMS. But in today’s highly demanding CMS landscape, shouldn’t collaborative editing be a core feature of all CMSs out of the box? Tag1 Consulting agreed, and the team decided to continue its rich legacy in CMS innovation by making collaborative editing a reality.

Recently, the team at Tag1 Consulting worked with the technical leadership at a top Fortune 50 company to evaluate solutions and ultimately implement Yjs as the collaborative editing solution that would successfully govern content updates across not only tens of thousands of concurrent users but also countless modifications that need to be managed and merged so that content remains up to date in the content management system (CMS). This process was the subject of our inaugural Tag1 Team Talk, and in this blog post, we’ll dive into some of the common and unexpected requirements of collaborative editing solutions, especially for an organization operating at a large scale with equally large editorial teams with diverse needs.

Collaborative editing, simply put, is the ability for multiple users to edit a single document simultaneously without the possibility of conflicts arising due to concurrent actions—multiple people writing and editing at the same time can’t lead to a jumbled mess. At minimum, all robust collaborative editing solutions need to be able to merge actions together such that every user ends up with the same version of the document, with all changes merged appropriately.

Collaborative editing requires a balancing act between clients (content editors), communication (whether between client and server or peer-to-peer), and concurrency (resolving multiple people’s simultaneous actions). But there are other obstacles that have only emerged with the hyperconnectivity of today’s global economy: The ability to edit content offline or on slow connections, for instance, as well as the ability to resynchronize said content, is a baseline requirement for many distributed teams.

The provision of a robust edit history is also uniquely difficult in collaborative editing. Understanding what occurs when an “Undo” or “Redo” button is clicked in single editors without the need for real-time collaboration is a relatively trivial question. However, in collaborative editors where synchronization across multiple users’ changes and batch updates from offline editing sessions need to be reflected in all users’ content, the definition of undo and redo actions becomes all the more challenging.

Moreover, real-time collaborative editing solutions also need to emphasize the collaboration element itself and afford users the ability to understand where other users’ cursors are located in documents. Two of the most fundamental features of any collaborative editing solution in today’s landscape are indications of presence and remote cursors, both characteristics of free-to-use collaborative editing solutions such as Google Docs.

Presence indications allow for users in documents to see who else is currently actively working on the document, similar to the user thumbnails in the upper-right corner of a typical Google Docs document. Remote cursors, meanwhile, indicate the content a user currently has selected or the cursor location at which they last viewed or edited text.

During Tag1’s evaluation of the collaborative editing landscape, the team narrowed the field of potential solutions down to these four: Yjs, ShareDB, CKEditor, and Collab. See below for a comparison matrix of where these real-time collaborative editing solutions stand, with further explanation later in the post.

 

 

Yjs

ShareDB

CKEditor

Collab

License

MIT

MIT

Proprietary (On-Prem Hosted)

MIT

Offline editing

Decentralized

Network-agnostic

Shared cursors

Presence (list of active users)

Commenting

Sync after server data loss

✖ (sync error)

✖ (Unsaved changes are lost)

Can implement other collaborative elements (e.g., drawing)

Scalable


(Many servers can handle the same document)

✔ 

(Locking via underlying DB)


(Hosted)

(Needs central source of truth - a single host for each doc. Puts additional constraints on how doc updates are propagated to “the right server”).

Currently supported editors 

ProseMirror

Quill
Monaco
CodeMirror
Ace

Quill
CodeMirror
Ace

CKEditor

ProseMirror

Implementation

CRDT

OT

OT

Reconciliation

Demos

Editing, Drawing,

3D model shared state

Sync

Editing

Editing

Editing in Tip Tap

Whereas the features within a collaborative editor are of paramount importance to its users, the underlying architecture can also play a key role in determining a solution’s robustness. For instance, many long-standing solutions require that all document operations ultimately occur at a central server instance, particularly in the case of ShareDB and Collab.

While a centralized server does confer substantial advantages as a single source of truth for content state, it is also a central source of failure. If the server fails, the most up-to-date state of the content is no longer accessible, and all versions of the content will become stale. For mission-critical content needs where staleness is unacceptable, centralized servers are recipes for potential disaster.

Furthermore, centralized systems are generally much more difficult to scale, which is an understandably critical requirement for a large organization operating at considerable scale. Google Docs, for example, has an upper limit on users who can actively collaborate. With an increasing number of users, the centralized system will start to break down, and this can only be solved with progressively more complex resource allocation techniques.

For these reasons, Tag1 further narrowed the focus to decentralized approaches that allow for peer-to-peer interactions, namely Yjs, which ensures that documents will always remain in sync irrespective, as document copies live on each user’s own instance as opposed to on a failure-prone central server. This means users can always refer to someone else’s instance in lieu of a single authoritative source that may not be available. Resource allocation is also much easier with Yjs because many servers can store and update the same document. It is significantly easier to scale insofar as there is essentially no limit on the number of users that can work together.

The majority of real-time collaborative editors, such as Google Docs, EtherPad, and CKEditor, use a strategy known as operational transformation (OT) to realize concurrent editing and real-time collaboration. In short, OT facilitates consistency maintenance and concurrency control for plain text documents, including features such as undo/redo, conflict resolution, and tree-structured document editing. Today, it is used to power collaboration features in Google Docs and Apache Wave.

Nonetheless, OT comes with certain disadvantages, namely the fact that existing OT frameworks are very tailored to the specific requirements of a certain application (e.g. rich text editing) whereas Yjs does not assume anything about the communication protocol on which it is implemented and works with a diverse array of applications. Yjs leverages commutative replicated data types (CRDT), used by popular tools like Apple Notes, Facebook’s Apollo, and Redis, among others. As Joseph Gentle, a former engineer on the Google Wave product and creator of ShareDB, once wrote:

“Unfortunately, implementing OT sucks. There’s a million algorithms with different tradeoffs, mostly trapped in academic papers. The algorithms are really hard and time consuming to implement correctly. […] Wave took 2 years to write and if we rewrote it today, it would take almost as long to write a second time.”

The key distinction between OT and CRDT is as follows: Consider an edit operation in which a user inserts a word at character position 5 in the document. In operational transformation, if another user adds 5 characters to the start of the document, the insertion is moved to position 10. While this is highly effective for simple plain text documents, complex hierarchical trees such as the document object model (DOM) present significant challenges. CRDT, meanwhile, assigns a unique identifier to every character, and all state transformations are applied relatively to objects in the distributed system. Rather than identifying the place of insertion based on character count, the character at that place of insertion retains the same identifier regardless of where it is relocated to within the document. As one benefit, this process simplifies resynchronization after offline editing.

If you want to dive deeper, the Conclave real-time editor (which is no longer maintained and therefore was not considered in our analysis) has another great high-level writeup explaining OT and CRDT. Additionally, you can watch or listen to our deep dive on OT vs. CRDT as part of our recent Tag1 Team Talk, “A Deep Dive into Yjs - Part 1”.

While other solutions such as ShareDB, CKEditor, and ProseMirror Collab are well-supported and very capable solutions in their own right, these technologies didn’t satisfy the specific requirements of our client’s project. For instance, ShareDB relies on the same approach as Google Docs, operational transformation (OT), rather than relying on the comparatively more robust CRDT (at least for our requirements). CKEditor, one of the most capable solutions available today, relies on closed-source and proprietary dependencies. Leveraging an open-source solution was strongly preferred by our client for many reasons, foremost among them to meet any potential need by enhancing the software themselves, and they didn’t want to be tied to a single vendor for what they saw as a core technology to their application. Finally, ProseMirror’s Collab module does not guarantee conflict resolution, which can lead to merge conflicts in documents.

Ultimately, the Tag1 team opted to select Yjs, an implementation of commutative replicated data types (CRDT), due to its network agnosticism and conflict resolution guarantees. Not only can Yjs support offline and low-connectivity editing, it can also store documents in local databases on user devices (such as through IndexedDB) to ensure full availability without a stable internet connection. Because Yjs facilitates concurrent editing on tree structures, not just text, it integrates well with view libraries such as React. Also compelling is its support for use cases beyond simple text editing, including collaborative drawing and state-sharing for 3D models. Going beyond text editing to implement other collaborative features is an important future goal for the project.

Furthermore, Yjs performs transactions on objects across a distributed system rather than on a centralized server, the problem of a single point of failure can be avoided, and it’s extremely scalable with no limitations on the number of concurrent collaborators. Moreover, Yjs is one of the only stable and fully tested implementations of CRDT available, while many of its counterparts leverage OT instead.

Finally, because Yjs focuses on providing decentralized servers and connector technology rather than prescribing the front-end editor, there is no dependency on a particular rich text editor, and organizations can opt to swap out the editor in the future with minimal impact on other components in the architecture. It also makes it easy to use multiple editors. For instance, our project uses ProseMirror for collaborative rich text editing and CodeMirror for collaborative Markdown editing (and other text formats can be added easily).

Real-time collaborative editing surfaces unique difficulties for any organization seeking to implement content workflows at a large scale. Over the course of the past decade, many new solutions have emerged to challenge the prevailing approaches dependent on operational transformation. Today, for instance, offline editing and effective conflict resolution on slow connections are of paramount importance to content editors and stakeholders alike. These key requirements have led to an embrace of decentralized, peer-to-peer approaches to collaborative editing rather than a failure-prone central server.

Tag1 undertook a wide-ranging evaluation of available solutions for collaborative editing, including Yjs, ProseMirror’s Collab module, ShareDB, and CKEditor. In the end, Yjs emerged as the winner due to its implementation of CRDT, as well as its scalability and emphasis on network agnosticism and conflict resolution, both areas where the other solutions sometimes fell short. While any robust evaluation of these solutions takes ample time, it’s our hope at Tag1 that our own assessment guides your own thinking as you delve into real-time collaborative editing for your own organization.

Special thanks to Fabian Franz, Kevin Jahns, Michael Meyers, and Jeffrey Gilbert for their feedback during the writing process.

Nov 11 2019
Nov 11

Table of Contents

What is a Rich Text Editor?
The Modern Rich Text Editor and Emerging Challenges
How we Evaluated Rich Text Editors
Why Tag1 Selected ProseMirror
Conclusion

Among all of the components commonly found in content management systems (CMSs) and typical editorial workflows, the rich text editor is perhaps the one that occupies the least amount of space but presents the most headaches due to its unique place in content architectures. From humble beginnings in discussion forums and the early days of the web and word processing, the rich text editor has since evolved into a diverse range of technologies that support a lengthening list of features and increasingly rich integrations.

Recently, Tag1 embarked on an exploration of rich text editors to evaluate solutions for a Fortune 50 company with demanding requirements. In this blog post, we’ll take a look at what impact the choice of a rich text editor can have down the line, some characteristics of the modern rich text editor, and Tag1’s own evaluation process. In the end, we discuss some of the rationales behind Tag1’s choice of ProseMirror as a rich text editor and some of the requirements leading to a decision that can serve as inspiration for any organization.

At its core, a rich text editor enables content editors not only to insert and modify content but also to format text and insert assets that add to the content in question. They are the toolbars that line every body field in CMSs, allowing for a rich array of functionality also found in word processors and services like Google Docs. Most content editors are deeply familiar with basic formatting features like boldfacing, italicization, underlining, strikethrough, text color, font selection, and bulleted and numbered lists.

There are other features that are considered table-stakes for rich text editors, especially for large organizations with a high threshold for formatting needs. These can include indentation (and outdent availability), codeblocks with syntax highlighting (particularly for knowledge bases and documentation websites for developers), quotations, collapsible sections of text, embeddable images, and last but not least, tables.

While these features comprise the most visible upper layer of rich text editors, the underlying architecture and data handling can be some of the most challenging elements to implement. All rich text editors have varying degrees of customizability and extensibility, and all editors similarly have different demands and expectations when it comes to how they manage the underlying data that ultimately permits rich formatting. In the case of Tag1’s top Fortune 50 customer, for example, the ability to insert React-controlled views and embedded videos into content ended up becoming an essential requirement.

Whereas many of the foregoing rich text editors available in the late 1990s and early 2000s trafficked primarily in basic formatting, larger editorial organizations have much higher expectations for the modern rich text editor. For instance, while many rich text editors historically focused solely on manipulation of HTML, the majority of new rich text editors emerging today manipulate structured data in the form of JSON, presenting unique migration challenges for those still relying on older rich text editors.

Today, there are few to no robust rich text editors available that support swappable document formats between HTML, WYSIWYG, Markdown, and other common formats. Any conversion between HTML, WYSIWYG, and Markdown formats will result in some information loss due to differences in available formatting options. As an illustrative example, a WYSIWYG document can include formatting features that are unsupported in Markdown, such as additional style information or even visually encoded traits such as the width of a table column. While converting a document format to another preserves the majority of information, there will inevitably be data loss due to unsupported features.

Moreover, as rich text editors become commonplace and the expectations of content editors evolve, there is a growing desire for these technologies to be accessible for users of assistive technologies. This is especially true in large companies such as Tag1’s Fortune 50 client, which must provide for content editors living with disabilities. Rich text editors today frequently lack baseline accessibility features such as ARIA attributes for buttons in editorial interfaces, presenting formidable challenges for many users.

Tag1 evaluated a range of rich text editors, including ProseMirror, Draft.js, CKEditor 5, Quill, Slate, and TipTap. Our mission was to find a solution that would be not only robust for content editors accustomed to word processors and Google Docs but also customizable and robust in handling the underlying data. But there were other requirements as well that were particularly meaningful to the client for whom Tag1 performed this evaluation.

An important first requirement was the ability for the chosen rich text editor to integrate seamlessly with collaborative editing solutions like Yjs and Collab out of the box. In addition, because of the wide-ranging use of open-source projects at the organization, a favorable license was of great importance to allow teams to leverage the project in various ways. In addition, other characteristics such as plugin availability, an active contributor community, and some support for accessibility were considered important during the evaluation.

As mentioned previously, other requirements were more unique to the customer in question, including native mobile app support, which would allow for mobile editing of rich text, a common feature otherwise found in many responsive-enabled CMSs; embedding of React view components, which would provide for small but rich dynamic components within the body of an item of content; and the ability to annotate content with comments and other notes of interest to content editors.

The table below displays the final results of the rich text editor evaluation and illustrates why Tag1 ultimately selected ProseMirror as their editor of choice for this project.

* Doesn’t support feature yet, but could be implemented (additional effort & cost) ** Comments are part of the document model (requirements dictate they not be) *** Per CKEditor documentation -- needs to be verified (see review process below) ⑅ In Depth accessibility reviews must be completed before we can grade

Ultimately, Tag1 opted to choose ProseMirror as the rich text editor of choice for their upcoming work with a top Fortune 50 company. Developed by Marijn Haverbeke, the author of CodeMirror, one of the most popular code editors for the web, ProseMirror is a richly customizable editor that also counts on an exhaustive and well-documented API. In addition, Haverbeke is known for his commitment to his open-source projects and responsiveness in the active and growing ProseMirror community. As those experienced in open-source projects know well, a robust and passionate contributor community does wonders to lower implementation and support costs.

Out of the box, ProseMirror as a tool is not particularly opinionated about the aesthetics of its editor or especially feature-rich. But this is in fact a boon for extensibility, as each additive feature of ProseMirror is provided by a distinct module encapsulating that functionality. For instance, while core features that are considered table-stakes among rich text editors today such as basic support for tables and lists are part of the core ProseMirror project, others, like those that provide improved table support and codeblock formatting, are only available through community-contributed ProseMirror modules.

ProseMirror also counts among its many advocates large organizations and publishers that demand considerable heft from its rich text editing solutions. Newspapers like The New York Times and The Guardian, as well as well-known companies like Atlassian and Evernote, are leveraging ProseMirror and its modules. In fact, Atlassian published the entirety of their ProseMirror modules under the highly permissive Apache 2.0 license.

Beyond the fact that many editors are based on ProseMirror as a foundation available through open source, such as Tiptap, the Fiduswriter editor, and CZI-ProseMirror, it was a logical choice for Tag1 and part of Tag1’s commitment to enabling innovation in editorial workflows with a strong and stable foundation at their center. Through an integration between ProseMirror and Yjs, the subject of a previous Tag1 blog post on collaborative editing, all requirements requested by the top Fortune 50 company will be satisfied.

Choosing a rich text editor for your editorial workflows is a decision fraught with small differences with large implications. While basic features such as simple list and table formatting are now ubiquitous across rich text editors like ProseMirror, Draft.js, CKEditor, Quill, and Slate, the growing demands of our customers obligate us to consider ever more difficult requirements than before. At the request of a top Fortune 50 company, Tag1 embarked on a robust evaluation of rich text editors that satisfied some of their unique requirements such as React component embeds, accessibility, and codeblocks with syntax highlighting.

In the end, teams opted to leverage ProseMirror due to its highly active and passionate community, the availability of unique features such as content annotations, native mobile support, and accessibility support. Thanks to its large community and extensive plugin ecosystem, Tag1 and its client can work with a variety of available tools to craft a truly futuristic rich text editing experience for their content editors. As this evaluation indicates, it is always of utmost importance to focus not only on the use cases for required features, but also on the users themselves who will use the product—the content editors and engineers who need to write, format, and manipulate rich text, day in and day out.

Special thanks to Fabian Franz, Kevin Jahns, Jeffrey Gilbert, and Michael Meyers for their feedback during the writing process

Sep 30 2019
Sep 30

[embedded content]

Automatic updates are coming to Drupal at the end of October! Long one of the most commonly requested features in the Drupal community, Drupal 7 and D8 will soon have an automatic updater that will allow Drupal installations to stay up-to-date more easily. How does Drupal's new auto updater work, and what do you need to know about it? In this Tag1 Team Talk, we dive into not only Drupal's new automatic updates feature itself but also its architecture, components, and roadmap, as well as why it's such an important part of Drupal's Core Strategic Initiatives.

Join moderator Preston So (Contributing Editor, Tag1 Consulting) and guests Lucas Hedding (Senior Architect and Data and Application Migration Expert, Tag1), Tim Lehnen (CTO, Drupal Association), Fabian Franz (Senior Technical Architect and Performance Lead, Tag1), and Michael Meyers (Managing Director, Tag1) for a deep dive into the nuts and bolts of Drupal's groundbreaking automatic updates feature, directly from the module maintainer, and the strategic initiative sponsors including the Drupal Association, MTech, Tag1, and the European Commission.

------------------------------------
Further reading
------------------------------------

Automatic Update - Module
https://www.drupal.org/project/automatic_updates

Automatic Updates - D7 and D8 Documentation Pages
https://www.drupal.org/docs/7/update/automatic-updates
https://www.drupal.org/docs/8/update/automatic-updates

Automatic Updates - Issue Queue
https://drupal.org/project/issues/automatic_updates
To provide your feedback on this first generation of the Automatic Updates module, create an issue in the Automatic Updates issue queue

Drupal Core Strategic Initiative
https://www.drupal.org/about/strategic-initiatives/automatic-updates

More information about the PSA.json feed can be found in this issue: https://www.drupal.org/project/drupalorg/issues/3068539

Drupal.org uses a package hashing and signing architecture based on the BSD Signify project
https://www.openbsd.org/papers/bsdcan-signify.html

Drupal contributors created a php implementation of signify: https://github.com/drupal/php-signify

Sponsors & Supporting organizations:
Drupal Association, Funding & Direction http://association.drupal.org
European Commission, Funding https://ec.europa.eu/
Tag1 Consulting, Development http://tag1.com
MTech, Development https://www.mtech-llc.com/

Text Transcript

-[Preston So] - Hello, and welcome once again to the Tag Team Talks series. Today I'm joined by several guests from all around the world and talking about a subject that is very near and dear to my heart, I'm very excited to hear about this topic, Drupal's new automatic update feature. This is part of a Drupal core strategic initiative happening as part of the Drupal core roadmap. And today we're gonna talk with the maintainer of the project, as well as several special guests, about what it is, why it's important, and how you can get started with it right away and help us report any bugs or any issues you might encounter. But before we get started I just want to remind everyone who's joining us today, don't forget to check out our previous Tag Team Talks at tag1.com/tagteamtalks. We've done several amazing webinars and sessions with guests from all around the world about realtime collaborative editing, rich text editors, and of course, our work, working with a top 50 Fortune company in all of these issues. If you do like this session, if you enjoyed this conversation today, please remember to upvote, subscribe, and share it with all your friends and colleagues at your team. So, first thing I wanna do today is introduce myself. My name is Preston So. I'm a contributing editor at Tag1 Consulting, moderator here at Tag Team Talks. Joined by my friend Michael Meyers, managing director of Tag1, located here in New York City. Both of us are separate places here in New York. We're joined today by Lucas Hedding from Leon, Nicaragua, the senior architect and data and application migration expert at Tag1. Lucas is one of the top 20 most active contributors to Drupal 8. He's also the Drupal core Migrate subsystem maintainer, a core contribution mentor, a drupal.org project application reviewer, and the maintainer for the thing we're talking about today, Automatic Updates. We're also joined today by Fabian Franz in Switzerland, senior technical architect and performance lead at Tag1. Fabian is one of the five Drupal 7 core branch maintainers. Here's also one of the top 50 contributors to Drupal 8 and maintainer for several Drupal 8 core subsystems, including BigPipe, Dynamic Page Cache, and Theme API, all very important systems in Drupal. We're also joined today by Tim Lehnen, located in Portland, Oregon, the CTO at the Drupal Association. Tim has been leading the Drupal Association engineering team for a number of years now, it's a great success, overseeing amazing initiatives to support Drupal development in the community. This includes the contribution credit system, the recent move to GitLab, and he's taken point on managing the relationship with the European Commission as well as they sponsor this wonderful initiative here. So, just to kick things off I wanna turn over the mic to Mike Meyers to talk briefly about Tag1 and why it is that we're talking about Automatic Updates, which is, once again, one of the most incredible topics, I think, that we can talk about in this series.

-[Michael Meyers]- Awesome. Thanks, Preston. Happy to be here. Tag1 is the second all-time leading contributor to Drupal. We have the largest concentration of core maintainers, and our team maintains over a dozen core systems like User and Views. We do a lot of our business based on Drupal, and auto uploader is key to keeping your sites up to date. So I'm excited to hear how this great new feature is gonna help end users, agency partners, platform companies, better manage their Drupal sites.

-[Preston]- Absolutely. And automatic update, this whole notion of auto update is, it's been something that's in the back burner of Drupal for a long time. It's also part of a very interesting aspect of the Drupal community which I think is very important to call out, which are the 10 or so currently ongoing Drupal core strategic initiatives. Before we jump into the technology itself let's just step back for a bird's eye view for a moment. What is a strategic initiative in the Drupal community, and what are some of the past and current ones? How did this whole initiative come to be?

-[Tim Lehnen]- I think that's a great question, and I'd be happy to speak to it a little bit. This is Tim with the DA. The strategic initiatives for Drupal are laid out on maybe an annual basis, maybe a little bit longer than that in terms of horizon, frequently in Dries' keynotes at DrupalCon, where he talks about key priorities for the Drupal project that he's discussed together with the whole core maintainer team and prioritized and settled on as the major efforts that he wants to rally the community around, and certainly the core developers around, in terms of moving the Drupal project forward. In the past, these have been things like adding Composer support to the Drupal project along with the release of Drupal 8. Things like updating the Migrate system, things like providing the API-first features in Drupal, and a number of things like that. And so, it was, I think in 2017 or 2018 when Dries first called attention to the need for automatic updates as a sort of foundational feature for Drupal, pointing out that this is something that a lot of commercial and proprietary software already does, and does well, and even some other open source projects are doing it. And it really makes a big difference to the total cost of ownership of people who run Drupal sites. [Preston]- Let's talk a little bit more about that angle, Tim. I think there's a lot of folks on this session here today who are really interested in learning more about what you just mentioned, the TCO, total cost of ownership. What exactly is the benefit of auto update to these small and medium-sized businesses? You mentioned agencies as well. What sorts of use cases are we supporting with this?

-[Tim]- Yeah, it's a great question. Let me paint a picture first for what happens right now around updates, and particularly the time period that's, I think, the most time intensive and critical for most people who maintain sites, whether they're agencies maintaining on behalf of clients or the end users themselves. So, that's security updates. Security updates are, of course, something that are very important, something that people need to keep up with as quickly as they can. And so there's sort of this culture and community within the Drupal community of everybody getting together during a core security release window on Wednesdays, hanging out in Slack, talking to each other, and waiting for that security release to drop. Now, while we have these security release windows, sometimes they come out right on time, sometimes they take several hours, but that window is sort of U.S.-centric because that's where a lot of Security Team members are. And so what that means is teams all over the world are up on high alert whatever time it might be locally waiting for this critically important patch to drop so they can update sites and make sure they're secure. And so, this is something that takes a lot of time that people can't really plan for, so much. They have to be there in that window, and they don't really have much of a choice. And so this is a situation that can be expensive in terms of just keeping your developers on call, paying overtime if you're outside of the timezone that these windows happen, all those sorts of things. And there's a lot of anxiety around them. That was one of the first elements that we wanted to address is making the security update process in particular easier, something you don't have to be so worried about, and then expanding on that to other use cases.

-[Fabian]- One question I had because it's pretty fascinating for me is for years, the Drupal community has said, we need auto updates, we need auto updates, we need auto updates. And it was always like, hey, but it's not possible in all. So, how come this direction change that now we are thinking it's possible finally?

-[Tim]- That's a really good question. I think there's a few factors that all came into play. And I think Lucas can speak to this a little bit more than I can. But just from an overall view, I think just various things came together. The architecture of Drupal 8 had changed, starting to do things like support Composer workflows, but also incorporating other elements like those, Symfony as a core element, all those kinds of things. But also we began observing just in the open source world other people doing this. So, for example, WordPress has an auto update system. Some other projects have an auto update system. So I think people have been thinking about how we're gonna architect something that would work in a Drupal context, for quite a while. Lucas, do you wanna maybe speak to what came together to make it more possible to consider? I think we lost your audio, Lucas.

- [Fabian] Lucas, you're on mute. Can you hear us? [Lucas Hedding]- Hello, can you hear me?

- [Fabian] Yes, much better. [Lucas]- Ah great. I think it's a bit of a myth that we had bought into. We had said to ourselves, we can't do this, and we said it to ourselves long enough that we had convinced ourselves. But then as we started seeing the competition in WordPress and others doing in-place updates or auto updates of some form we said, well, let's see if we can solve this tough problem. And we're close. We've spent a lot of time architecting this. We're still not 100% there, but I think we can see the light at the end of the tunnel. I think it's gonna be a pretty good success as we get to the finish of this first part of the initiative.

-[Fabian]- Now I'm really curious. How did you split up this mega task of getting even started with this tough problem?

- [Lucas]- Well, I mean... Let's see, we did a lot of architectural thinking, a lot of discussions at various camps and DrupalCons.

-[Tim]- Yeah, I think one of the... It was a confluence of a few factors that came together and made it work well. We had a meeting of minds between some key contributors in the core team at Midwest Drupal Summit in 2018 and did a lot of architectural work, and at the same time, with these conversations with the European Commission had started, and they were talking about how Drupal was a critical part of their infrastructure and they wanted to find something to support. And so we put two and two together and said, hey, we can do this. But yeah, scope is always a problem. It's a huge project. And so, what we said was, okay, well, if we wanna do some sort of phased scenario, what's the most important thing to start with, what has the biggest impact on people, and what simplifies the task a little bit? And that's where we came to security updates in particular as being the focus of this first phase because it both lets us focus on just patch releases that shouldn't be destructive in terms of what's being changed in those patches, and also fixes a critical need that a lot of the community has.

[Fabian]- That's really great. [Michael Meyers]- Maybe before we jump into the underlying architecture and how all this comes together, can you give us a background on how this works? Is this gonna update my production website in the background? Is this something I can do on my development environment and put through a CICD system? How does this work at a high level?

-[Lucas]- I think the answer is yes to all of those questions. It's supposedly flexible. At least that's how we're designing it. I think it's gonna be a little bit like the configuration management system in that there was an original intent for how we do this. And I think the original intent, at this point, is that you'd be a small site owner and you'd wanna in-place update on a live site, so definitely we're trying to make that as stable as possible. But if you've got a continuous integration system setup, there's nothing stopping you as a site owner from wiring into that. And then we'll iterate. This is an initiative that has multiple phases planned out already.

-[Fabian]- So it could be possible for me as... I mean, if I had 100 Drupal sites or something like that it could be possible for me to just get, instead of waiting for many, many hours on this Wednesday I would just get the package pushed to me, just as potential scenario, and then once the patch arrives then I can distribute it, test it, if my basic tests are working, distribute it automatically, basically.

-[Lucas]- Yeah, I mean... Well, some of the initial conversations I had was at MidCamp with folks from Pantheon and from, at the time, it was Drupal Commerce. They've renamed themselves to... I'm gonna butcher their name.

-[Tim]- Centarro is the name.

-[Lucas]- Centarro, right. So, we've got different folks in the room, and we're all just chatting about this idea of auto update, and everyone, Pantheon has its upstreams. There's nothing stopping any of these, even infrastructure owners, from certifying things and updating the URLs where you pull down files so that it's from a vetted source of thing where they've already done QA and added their secret sauce to this. I'm not saying that that's what's gonna happen because I don't know the future. But we're definitely trying to build this as flexibly as possibly so that even the hosting providers and your service providers can add their own secret sauce. Some of this might or might not play real well with what we're doing around the hashing and cryptographic signing of some of the resources. But at the very least, there's flexibility in the base architecture.

- [Preston]- Sorry, go ahead, Fabian.

- [Fabian]- I'm really curious as a Drupal 7 maintainer. Is this only for Drupal 8, or would Drupal 7, where we still have probably a million sites out there, also be? [Lucas]- Because of the source of our funding here we're in an interesting place where the backport policy for Drupal is you have to do it in Drupal 8. Eventually it'll be Drupal 9 and then we backport from there. But the funding and the interest from the European Commission has really been Drupal 7. So we're playing to two camps here. We've gotta fulfill our contracts with the European Commission and so all of the goodness of Drupal 8 is getting backported as we go along. And at this point, essentially everything that's in there for 8 has already been built out for 7.

- [Preston]- Very interesting. And I wanna get back to what you just mentioned, Lucas, about flexibility and some of the really interesting elements there. But first, we've established that this is gonna be for both Drupal 7 and Drupal 8 which is, I'm sure, music to everyone's ears listening right now. I'm curious though, because this is such an ambitious project as both you, and Tim, and Fabian, have all mentioned, where are we right now in the life cycle? I know, by the way, you have a full roadmap already outlined. There's gonna be a link, by the way, in the description of this video if you wanna check out the full roadmap for auto updates. But what's currently part of the initial outflow as of right now?

- [Lucas]- So, we did an alpha one, I don't know, three or four weeks ago. And we did that because we now have something of value. There's this concept that we're building into the auto updates called PSAs, or public safety alerts, announcements. So when these Drupalgeddon type events happen, when these majorly critical things are coming down the pipe, we can now have another channel, yet another channel, to alert even small site owners that their site is now out of date. We're doing yet another channel because we've already got the existing, hey, you've got security updates, hey, we've got Twitter, hey, we've got Slack. That only alerts and notifies folks if they're following the Twitter feed. And we already send out the emails about your site has updates available for it for everything already, so it's a little bit like the boy calling wolf every other week. So this is just another channel, but it's going to be used with great care. It's gonna be used once or twice a year. And then we'll be able to send out these alerts to folks, and we have that now for both 7 and 8.

-[Fabian]- Just to clarify for me, so, I'm one of those, probably not, but just imagine I'm one of those that always clicks away this message, oh, there's updates available. Yes, I know, I haven't updated this module in a while, or there's security updates, oh, but this module I can't, and so I'm missing a crucial core update. But instead of that message, this generic message, and many also disable Update module because it makes things slower at some parts, the thing is, instead of that, I would be getting the message, on Wednesday, the date, there's a critical security update, please pay attention, or something so that it's more direct communication to...

[Lucas]- Direct communication, it'll happen in the days preceding to the Wednesday as well. So, whenever the Security Team decides that, hey, this is important, they send out the emails a day or so in advance. This would also be part of their process. So the other part of what we've got built in in this alpha is what's called readiness checks, or preflight checks if you wanna put it another way.

-[Preston]- Sorry, go ahead.

-[Lucas]- The readiness checks just basically go out and say, are you ready? If you got an update tomorrow, or tonight, in 10 minutes would you be ready to update using this technology? And then we spent a lot of time thinking, well, how do we go about this? We even renamed it from preflight checks because with our non-English speaking community we thought it doesn't have as much meaning as a readiness check. Readiness checking goes and out sees, hey, is your site hacked? And I'll get into that in a second. Do you have enough disk space? If we were to apply an update, do you have two megabytes of disk space and you're gonna fail? There's about eight of these things that we decided. And some of them are errors, some of them are warnings. As an example, we'll warn you if you're not running cron on your site frequently enough. If you're not running it frequently enough we know that you're not gonna automatically update your site in the next couple hours. But that's a warning. We're not gonna block you. But if your site is mounted on a read-only hard drive for reasons of security, you can't really go out and update a read-only hard drive. That's just gonna fail. And that would be an error. So we've got this whole concept around readiness checks. It's a plugin, part of the Drupal 8 plugin system. Drupal 7 we've done something similar to that, not plugins, but the same business logic is in there. Can we update you quickly? And both of these things at this point are now available in the 7 and 8 alphas.

- [Fabian]- We can download this?

- [Lucas]- Yeah, right now.

- [Preston]- That's amazing. We've already spent a great deal of time talking about readiness checks, and I wanna get more into the features of the Automatic Updates module. But first I wanna move into a very interesting subject for our audience, because this has been a very uniquely challenging problem for Drupal for a very long time. How did you all manage to architect this into the right solution? And also one that could potentially be expanded out to other PHP projects, like WordPress, for example.

- [Lucas]- I don't know that I was involved in the early, early conversations. But since the beginning of 2019 I've been involved. Tim, you wanna bring up some of the early conversations?

- [Tim]- I can get into some of the early architectural discussions. I think the concern has to do... As Lucas was saying earlier, we have this myth that Drupal use cases and Drupal sites were so varied and so complex that that meant there wasn't a good way to have a set of standards for doing an auto update process. And we just really started interrogating those things in the last two years or so and said, well, is that really true? And what it came down to be is that the concerns were all about confidence. They were all about that ability to understand, hey, can we really be confident that these updates will apply cleanly, that they're not gonna break folks' sites, what do we do if we miss something and it does break folks' sites. So we spent a lot of this time saying, okay, what kind of architecture can we create to give us the maximum confidence so that whether you're running this as an attended or even unattended auto update you'll be able to feel like it's not gonna break your site. And so, that meant that we needed to architect these readiness checks, we needed to talk about something that's coming in the second phase which is this notion of having an A/B version of your codebase so that you can run an update, but if something fails flip back to the known good version, almost like a bootloader concept is what we were inspired by. Then we had thought about, okay, well, what else is involved? Well, we need to make sure that we can generate something that is the equivalent of a patch. Not literally the patch that's just been released but a quasi-patch, as Lucas termed it, that will apply all these changes cleanly to sites. And then finally we need to give people confidence in what's being delivered. So that means having a good way to secure the package delivery system that comes from drupal.org, ensure that it's signed and verifiable so that people know that what they're installing is in fact coming from a trusted source. All of these concerns are about confidence and integrity of the system, and so that's kind of what we built into it. And then once we realized, hey, you know, there's ways to solve each of these problems, there's ways to solve confidence that the update will apply before you do it, there's ways to solve confidence in being able to roll back if you need it, there's ways to solve the confidence in the trusted source of the package, then we realized it's almost significantly simpler. If you assume that we've solved those problems we're just applying patches and a couple processes to make sure those things apply clearly. And from there we got into the real weeds of the architecture which I think Lucas could probably speak to more. [Lucas]- It's interesting that you say it's simple because the patch that we've built right now for this service to auto update your site, it's a service. Half of this patch is testing. Actually, probably more of the patch is testing. It's like 500 lines, 400 lines of code, to update your site. Seriously, it's a really small amount of code to grab an artifact off of drupal.org, download it, and overlay that, I'm calling it a quasi-patch because it's not a full patch, it's not a regular patch. It's all the files that would be touched in a version between 8.73 and 8.74, as an example. If there's 10 files, we grab all those 10 files, put them in this patch. That way we don't have to worry about git apply, or patchutils, or anything on the server. We just have to deal with what's already built into PHP, copy and paste. And we do the copy and paste, and we do all of this... I'm gonna credit Nate Lampton here. His vision here was do it all in the same HTTP request. If you do it all in the same HTTP request and you roll back in the same HTTP request, the site can't actually go down. Well, knock on wood. It still probably could in very rare edge cases. It's about, I don't know, 200 or 300, 400, 500 lines of code, depending on if you start counting some of our tests in there as well.

-[Preston]- Very interesting. I think the way that you've solved this problem with obviously these quasi-patches and just the thinking that's gone into this really reflects, I think, the amazing commitment that you all have shown to making this a really successful feature. I wanna dig into some of the features now. We've already delved a little bit into some of the public safety messaging and the readiness checks. But just for our readers, and our listeners, and all the folks in the audience today, I wanna define just very clearly what those three major components of auto updates are. The first is public safety messaging, which we talked about briefly already. The second is readiness checks, or preflight checks, which really indicate when a site is ready to go. And then finally, I think the most compelling of all of these, which are the actual in-place updates that occur. What can we learn... Lucas, you mentioned earlier about Drupalgeddon and some of the ways in which, for example, every time I get a security advisory tweet or see that pop up, sometimes just the urgency of it doesn't really register with me, or things of that nature. I know Fabian also mentioned that as well. So I'm curious what sorts of other aspects of public safety messaging did your team consider during this whole effort? And what other components are involved in that?

-[Lucas]- Well, I really wish there was more folks on this call because it's not just the folks here on this call that have really contributed. It's been a whole team in the auto updates channel in Slack, and folks at different camps and conferences. So, what have we... I mean, there is other thoughts around just doing figuring out what is on the site already, and calling home to drupal.org and sending information. But there's security and privacy concerns around that, and so for the scope of what we're doing right now we're trying to keep it simple for messaging.
- [Tim]- Yeah, I would add to that, basically, we coordinated very closely with the Security Team, naturally, because this first focus was on the security side of Automatic Updates. And so, the main thing that we did was say, look, they already have their security advisory content type architecture on drupal.org, which is the basis of how the regular, standard, canonical announcements roll out. So what we asked ourselves was, okay, what are the requirements to changing that to provide a feed that could be consumed, a JSON feed that could be consumed by this auto update system, and then how do we put policy restrictions in place to say, hey, look, this is gonna be a more prominent message appearing in people's admin interface, but we don't want to cry wolf, as people have said, we don't want to overuse it, so what controls do we put in place. And mostly that will be at the discretion of the Security Team to say when is it worthwhile to put out this message about those additional updates.

- [Preston]- I think the interesting thing here is that we're really trying to reduce the burden on the user through this JSON feed, through the way in which we're gonna be able to get this alert directly to the audience. Now, just to move into more of the way that the user experience, or the way that the site owner experience is really improving here. These readiness checks, these preflight checks, that run on every six hours, or via cron jobs, how do they really play into this whole feature and this whole notion of automatic updates?

- [Tim]- I was just gonna add before Lucas runs into the architecture. From my two cents, what Lucas and others have architected is really great because it's an extensible system. New kinds of readiness checks can be created as time goes on. And what's been developed so far has been the result of core contributors, and the Security Team, and just other general contributors, coming together and saying, okay, what are those scenarios and use cases that are gonna be most important to determining whether or not an end user can actually run this auto update process? What's gonna block them from taking advantage of this new system that will make things easier? And so building this in so these currently eight or so different categories of readiness checks run on a regular basis and then provide their results reports so the site owner knows, oh, hey, if I fix this read-only directory, or if I fix whatever these warnings are or errors are in advance of this next round of updates going on, it's gonna be much more of a breeze for me. I just think those elements are gonna be really valuable. They're already valuable. They're already some things that I think people should be taking advantage of if they wanna try out the alpha. But I think the underlying message is they're only gonna become more valuable. As people test the alpha and as we move into the first phase stable release and things like that, we're only gonna learn more from contact with the world about other things that we could be checking, other things that we could be creating warnings, or errors, or messages for in order to help people be ready to take full advantage of the system.

- [Fabian]- Quick question to the readiness checks, because really interesting. As I've understood, it's plugin based?

- [Lucas]- It's plugin based indeed.

- [Fabian]- So as the site owner or, for example, a hosting provider, I could add my own readiness checks, as a site owner I could add my own readiness checks so if I need something more, like for example running some very basic tests or something that my homepage is still reachable, or whatever I wanted, I could just do that with a plugin.

- [Lucas]- Yeah, you could. We tried to come up with the basic ones that are very reusable. Most of these readiness checks are somewhere in the range, of real code, of about five to 10 lines. Can we write to a file on your hard drive? You know, I mean, that doesn't take that long to figure out that you can't do that. It's really simple. You tag a plugin with the right tags, look at an example, and now as an agency or even as a large company, if you have some specific rules around, hey, we need to make sure that the moon is in the third phase right now, you can do that. There's nothing stopping you.

- [Tim]- I would add...

- [Fabian]- That's really cool. And I have a question. If I have hacked core, and as a core maintainer I sometimes have to do that, can I still use the auto updates in that? I mean, is it preventing me from that, especially if a file is updated that I never hacked?

- [Lucas]- We're not blocking you from updating core, or any contrib module, if you have a file that's modified. Remember we're using a quasi-patch approach here. So, if you are about to update something... For the readiness check, if you've got a modified, patched file in core, we will warn you, saying there's some uncertainty here. Because what happens if a patch, a quasi-patch between 8-dot-something and 8-dot-something else comes out tomorrow? Well, you can look at what's been modified. If it's in Book module and you don't even use the Book module on your site, or the Book module hasn't been modified in six years or something... If you can look at it and you say, well, I've patched this thing and it's rarely patched, or there's no security issues in it that have come out for many years then you have greater confidence that you'd probably be okay. That said, when we go to apply that update we're gonna take that specific one and we're gonna say, no, no, no, no, you can't override a file if it's been modified.

- [Tim]- So any of your unchanged files, as long as you check from the warnings and you think it's otherwise okay, any of your files that are unchanged from what the architecture has, when it does its comparison, then yeah, those can still apply.

- [Lucas]- I wanna go into a little bit about this how are we figuring out if files have changed. Because I think that's a really interesting component to all of this, one that I wish I had had more... I wish I could say that I had more thought into it, but it's really the community. There's this whole concept, though, that makes sense after it was explained to me the first time about doing a file hash. Just do an MD5 sum, well, not an MD5 sum. A SHA-256 or a SHA-512 sum. We do that of every single file in all of core, all of contrib. The Drupal Association has really helped us out a lot here. They're gonna provide an official feed for hashes for all of these things. This is the type of stuff that if you look at the Linux distributions out there in the world they've been doing this for a super long time. If you try to download Composer, even, there's a SHA sum there. And you grab that SHA sum and you're supposed to do SHA sum to compare and to make sure that what you just downloaded is actually what was provided on the website. That's the type of technology we're building into. We're doing it not only to see is your site been modified, we're also doing it to have greater certainty assurance that these quasi-patch archives that we're downloading haven't been hacked or modified by a man-in-the-middle attack, or some other attack. And we'll link this if you're really wanting to look into it. There's a whole project that the Drupal community has now provided to the world on GitHub called PHP Signify. And it's using chain signatures. We've been going through it, trying to figure out... But you know what it is? It's following a pattern that BSD is using. BSD Linux has used the same thing. We're just standing on their shoulders. The current concerns about, well, did you write this hashing and verification system yourself? No, we didn't. And we also have some super, super smart people in the community contribute to this at the Midwest Developer Summit this summer.

- [Preston]- Very interesting, and I think this just indicates the amount of thought that went into this. And the notion of actually introducing your own readiness checks is something very interesting. We mentioned what phase the moon is in. I can see this being useful if you have, for example, a decoupled Drupal implementation and you have to wait on other code that you're working with to be ready before starting an automatic update, such as your decoupled application or frontend. So, now with these, the third feature we just named was in-place updates. Which, as you mentioned, Lucas, just now, you wanna be very secure. You wanna make sure that potentially these ZIPs that are being downloaded are not in any way affected by man-in-the-middle attacks. How do you look at that from the standpoint of the in-place update feature? What's going on in there in the actual Drupal site itself now that you've got those updates ready to go?

- [Lucas]- So, the mechanics of this small patch that we've got right now is that we download a ZIP file. The ZIP file has six files in it. Bootstrap.include is one of them, as in my example here. But another file that we have in here is this signature file that's called CSIG. It's using the same format that BSD Linux is using. The project that we're using is called PHP Signify. It's standing on the shoulders of libsodium which is baked into PHP as of 7.0 or 7.1. Someone will comment in the feed here and correct us. Which also has backwards compatibility shims all the way back to PHP 5.3. So if you're still on a Drupal 7 site and you're on an old version of PHP, you could still have this stuff work for you. It's doing a hash and it's using things like SHA to hash these files, compare the hash to a publicly signed and privately signed key pair of things. We're even talking about, not talking, we're doing, the Drupal Association is in the process of acquiring and installing an HSM so that we can do this in a very, very secure manner. And the HSM will then be taken offline. You can't get more secure than this. Is Drupal really a target? Someone, a conspiracy theory might say that it is. But even if it isn't a target, or even if it is, we're doing this in the most secure manner that we can.

- [Tim]- Sorry, just to add to that a little bit, I think it's really interesting and robust the way it's architected. Drupal.org delivers these hashed and signed files with the quasi-patch plus all of the appropriate signature files with the relevant metadata to verify all that. And then, again, as Lucas said, anyone using the auto update system, as long as they're on even a very old version of PHP with these shims in place, the PHP process can do the verification of the signatures, and the hashing, and all of that, and then we know that it's not an interrupted package that's been modified in flight, there wasn't a man-in-the-middle attack, or anything like that. And then on our side using an HSM we're able to do secure key rotations on a regular basis, we're able to validate, hey, you know, when was the last time this was checked, is it within a valid window of being signed, all this kind of stuff. So it's a really, really robust system. And I have to say, we're sort of, we have the advantage... It took us a while to get here, but we have the advantage of being able to look at the example of other projects, whether that's a BSD system that a lot of this is based on, but also look at, say, WordPress or other projects that have started implementing their own systems and say, these are the impressive things they've done but they may be missing this piece in terms of the signatures and man-in-the-middle protection, or maybe they're missing this other piece, and how do we get a lot of that right from the start. And part of this is only possible because Drupal has maintained its status as kind of a centralized project. Drupal.org is the home of most, but not all, code related to the Drupal project and that helps us be able to sign and deliver all the relevant packages. And with our move to GitLab, I think some other folks who aren't here are probably coming back as well.

- [Fabian]- And I'm very glad that you're doing that, especially with an HSM, because the recent Webmin hack showed how bad is this if you deliver hacked software to your users.

- [Preston]- Oh, absolutely. And I think a lot of our audience is very concerned about that just given the amount of risk that's possible here. So, I have one question for you both, Tim and Lucas. One of the things that I know a lot of our audience is curious about is, well, you have these archives, it makes sense. But what happens if we're using Composer? I use Composer. I've got Drupal 8.8. What exactly is planned for Composer? Or does it already work for sites that use Composer?

- [Lucas]- So, when you go to 8.8.0, when that gets launched in October, you will have Composer whether you know it or not. The Composer initiative has been really rolling and rocking quite quickly here in the last few months and a lot of things have come together such that when you upgrade to 8.8.8, 8.8.0, you're gonna have Composer just by installing and upgrading to 8.8. So that does open up the question. Because when this whole initiative of auto update started we said, well, we don't have any official Composer support in core. Remember, the funding is coming from European Commission. That funding will dry up here shortly. Let's focus on the needs of tarball. So we have spent a lot of time on tarball because of pragmatic reasons. But none of that means... None of that, though, means that we've not thought about Composer. And we have thought about it. And there's two or three different scenarios that we've considered with Composer. And the first two of them, you're fine. You're not gonna have an issue. Let me talk about those. The first one is you're on a site that's been using tarball since day one. And again, let me pause there. This is really an issue in Drupal 8. When we go back to Drupal 7, the number of sites that are using Composer are very minimal. So you can do it, I guess. I've never done it. Drupal 8 though, it is an issue. If you've been doing tarballs and you're still running before 8.8, let's just make it simple. You're not gonna have any issues at all. 8.8 comes around, you've upgraded to that because you're being a good boy or girl. Now you've got Composer. You're still okay because we're just overlaying files. The problem that comes in is when you start using the capabilities of Composer and you do a Composer require, and your Composer JSON and your Composer lock file start to get modified. In the case of that, you're gonna have some additional considerations. Again, if it's just an overlay of files you still will probably work in nine out of 10 times. Nothing that we've done will have an issue for your site. If you're wanting to patch, say, Token module, that's one of the more popular modules out there, and security release comes out for Token and it's been managed through Composer, we'll do that, that's not a problem. It's when you're dealing with a thing like... I'm gonna pick on Commerce. Commerce has a lot of vendor dependencies as well. If you want to update Commerce with this in phase one, and I wanna be clear, phase one, there are lots more plans in the future, you might have it a little bit more tricky. If it's just one file in Commerce, again, you still probably won't have a problem. But if it's a file... I've got this all written down in an issue on drupal.org. We'll link to it because this gets tricky. If it's a file that isn't in a vendor folder then you're fine. But if you start dealing with modified Composer JSON, vendors... But again, if think about the audience here, it's for folks that probably wouldn't have been doing composer update, composer require, up til now because they were on tarball. And so I think at least for version 1.0 of this module, we're gonna learn so much. So all that to say, Composer is thought of. There's more to come with it. And we'll make it better with time. We're gonna beef up that support.

-[Preston]- Absolutely. And I wanna ask just the other elephant in the room here that I know a lot of people have on their minds. Is this gonna be a part of Drupal core? Is all of this going into core? At what point do we see this being in Drupal core?

- [Lucas]- We've been working with the core contributors, and we even have Fabian on the call today. We've been working with core contributors from day one. This is an initiative out of the Drupal Association and this is not done in a vacuum. The simple answer is all of this will go into Drupal core. We're gonna incubate it in contrib. That said, some of the actual implementations of how we're doing this are not gonna go into core. We're not gonna do this whole overlaying of files thing on the live site, or at least not unless something changes, because there's too much risk there. Even though it's still very low risk, the core maintainers and release managers are still feeling that that's too much. So we've got plans for phase two, unfunded plans. I'm gonna put a request out there for funds. We need more funds for this A/B controller where you've got a live site on A and over on B you've got the about to be updated site and we're gonna flip back and forth between A and B, and if B has a problem we switch it back to A. And then we just keep doing that. And as we do that we can also start doing things more with Composer perhaps, beefing up that support. The simple answer is all of this will go into core.

- [Fabian]- What's the roadmap for the public safety announcements? Because there doesn't seem that much to it besides that needing to be a secure channel that I could commit that to Drupal 7, kind of right now. So what are the plans from the 8 maintainers there when we could get this in?

- [Lucas]- We're still in discussions with you guys, Fabian. Because we wanna figure out when's the most ideal time to take these features over. We've really been focusing on the end of the year deadline. But if someone were to grab the code, roll a patch, and throw it up there, there's no stopping anyone from RTBCing that, reviewing, testing, and getting that into core now because it's pretty stable. I say that, but I also say we've gotta learn a lot. This whole thing, we need to learn more. More information needs to be gathered. I was hoping that we would have a little bit of a chance to have several hundred sites, thousand sites, with the contrib module installed with this on there, figure out the right wording for this messaging so that we don't annoy people to death, but we don't do it too infrequently either, and iterate and improve on it in contrib where the gates are a little bit lower, the stakes are a little bit lower. And then when it's ready, move it into core.

- [Fabian]- Yeah, I mean, you could always start with a simple experimental module. has worked very well. And just one real quick question to the Composer thing, just if I got this right. So, the problem with comments is, for example, that a security patch might itself depend on a vendor library update and then I've got problems because then I can't know what kind of inner library's in there. Is that where the problems go into?

- [Lucas]- Yeah. I really probably should link it, and I feel like I did a poor description. But yeah, like, some of our more recent Drupal 8 updates worked for with a far file and we've got a vendor file in here. If someone has run composer update at some point and they've already updated to a newer version, and in the midst of all of this they got a newer version of Symfony and some other components, it could get really messy. I think what we're hoping to do is just release this so that folks can give us that feedback, is this gonna blow up on me. There's gonna be a few folks that are gonna be like, I don't care. Just give me the updated site. And those are gonna be the risky folks. Those are the guys who go down the black diamond ski slopes without any protection and half kill themselves. We need those guys out there to do this because they're gonna report, and probably get upset, that they broke their site. But we need those people to give us feedback. But if they don't give us feedback, if we give the folks the flexibility to hang themselves and there's silence, well that also tells us a lot too. It says, well, maybe we can relax some of our... Relax some of what we're doing here, and even maybe promote some of these more risky adventures.

- [Fabian]- One quick question for Composer itself. Have you thought about, the idea came out originally two years ago or something like that, to just run Composer inside of that process with a virtual file system?

- [Lucas]- Composer right now takes too much memory. And on shared hosting where you're capped out at 512 megabytes, or 256 if you've got other processes, it becomes cumbersome. But a lot of the same folks that have been in these rooms have been thinking about this for a while. So, more to come on that. I think we've got some ideas to drop that in phase two as funding comes available for the community. We might have some more surprises in the wings.

- [Fabian] Sounds great.

- [Lucas]- Tim, you know more about the future. Do you wanna talk about where you see this going?

- [Tim]- Why don't we talk a little bit, yeah, just about phase two and what's going on, and just move into a discussion of the future. As we've said several times, the European Commission, because they rely on Drupal and they have a commitment to all the open source technologies that they're a part of as part of their FOSSA program, approached us before the beginning of the year and said that they wanted to set something up, and generously they've been supporting the work so far to do this phase one of the project. But as we said, that's gonna run out. Then in the next phase when we wanna have better Composer support and this A/B system so that it can be actually brought into core, those are really, I think, probably the two most major priorities, we're going to really need to figure out how we create a sustainable funding model for that. For folks who are out there who may be at large end user organizations who rely on Composer in your workflows but are also interested in the possibilities of Automatic Updates, it'd be great to hear from you. You can reach out to me at the Drupal Association. And that phase two we're gonna try and really plant a flag around that A/B controller and more robust Composer support and see who we can bring to the table to help us make that happen. It's really important to us that we move this forward. It's a huge priority for the project, of course, but it's also, it's not been an easy problem to solve. As people have mentioned, we've been talking about this in the Drupal community for probably five plus years and are only now beginning to make some real progress. But it's taken work. As we've alluded to before, there's a huge group of people involved. Staff on the Drupal Association side. People like Tag1 who've been working with us closely to actually do implementation, Lucas in particular. The Security Team, all the volunteers in the auto updates channel. Just a huge amount of people. The core maintainers have all been involved. And without funding, we might make a little bit of progress but I don't wanna go another five years without really delivering an even more robust system that we're capable of doing.

[Preston]- Yeah, I think this definitely shows the amount of support and backing from the community but also from these sponsors that have really contributed a lot to make this a success. Obviously we mentioned the Drupal Association, MTech, Tag1 Consulting, the European Commission and their Free and Open Source program. I know that you all are looking for more sponsors. Tim, just to be very clear, to reach out to you they can go to [email protected].

- [Tim]- That's right. You can reach me there directly.

- [Preston]- Absolutely. I just wanna mention briefly that this is a very important initiative for the future of Drupal and for the ability for site owners to really have people, or to have that governance and maintenance that they've been used to all along with so many of these projects. And so Tim, just to jump back into the notion of sponsorship and looking for contributions. Why should folks look at sponsoring this project? We've talked about obviously the benefits technically, but are there other benefits also to this?

- [Tim]- Yeah, I mean, in addition to the technical benefits and the benefits of just being a part of good citizens of an open source community and raising the tide for all ships, there's also, because this is a strategic initiative for core, there's a lot of visibility into this project. There's opportunity for marketing and awareness campaigns around your support of this initiative if you choose to get involved. Cross-promotional activities that can be done with the Drupal Association. There's also opportunities for recruiting some of the best talent in the Drupal community if your organization is looking to recruit some more key talent. A lot of the contributors who've been involved, whether on a volunteer basis or a contract basis with this initiative are some of the most talented folks in the whole community, and you could recruit those people to your team. And then also, this is something that gets talked about on stage during the keynote at DrupalCon. If, for example, we managed to get someone on board before DrupalCon Amsterdam, I'm sure we would be thrilled to announce that partnership with another organization. Moving forward into the next year during the keynote, during Dries' keynote, we're certainly gonna be talking about the progress on this initiative and all the organizations who've been involved in the first phase so far.

- [Preston]- Wonderful, and I'm looking forward to hearing a lot more about this at DrupalCon Amsterdam. Before we close things up here, though, I just wanna ask, are there any other things that we wanted to mention about where this project is headed, or any other things that the audience might be interested in?

- [Tim]- I would just reiterate something that we said before, which is, there's an alpha available now. It's been out for several weeks. So if you're in a position to try it out, to test it out and provide feedback, that would be wonderful. If you're interested in being involved in the initiative there's a lot of activity in the auto updates channel in Drupal Slack as well as in the drupal.org/projects/autoupdates so you can get involved there and help out. And yeah, certainly feel free to reach out to myself or others involved if you have questions.

- [Fabian]- Awesome work.

- [Preston]- Yes, this is fantastic.

-[Lucas]- I mentioned it before but this has been the work of many, many people, and we've only got five or six on this call which is a poor representation of the larger community. So my shout out to Drupal and the Drupal community.

-[Preston]- Absolutely. All righty, well, we have run out of time here at the Tag1 Team Talk. So, just to remind the audience here today, thanks again for joining. It was a really amazing conversation about Automatic Updates coming to Drupal. A really amazing thing. I'm looking forward to hearing about it in Amsterdam and Minneapolis. Just for your information, we post all of these, by the way, at tag1.com/tagteamtalks. All the links are gonna be available alongside this video and audio recording. If you like this talk, if you like what you heard today, please remember to upvote, subscribe, and share it with all your colleagues, your parents, your friends, your grandparents too. And as always, please, if you have any topics you wanna hear about, if Automatic Updates is interesting to you, you wanna bring Lucas and Tim back into the fray here, please write to us at [email protected]. I wanna say a big thank you to Lucas, a big thank you to Tim, and to Fabian, and to Michael as well, for joining us today for yet another Tag Team Talk. Thank you all, and goodbye.

Sep 26 2019
Sep 26

[embedded content]

Rich text editors are an integral part of content creation and content management workflows, but they can often present challenges for developers when it comes to robustness, extensibility, flexibility, and accessibility. What are some of the considerations you should keep in mind when evaluating rich text editors, especially for mission-critical systems like the application Tag1 is building for a top Fortune 50 company?

In this Tag1 Team Talk, we explore the new generation of rich text editors, which are based on a well defined data-structure rather than HTML, but still can export to Markdown or HTML. This allows us to tackle new requirements organizations have, including video embedding, cross-device support, and keyboard-navigable editors. After diving into some of the open-source solutions available in the market, such as Draft.js, CKEditor 5, Quill, Slate, and TapTap, join moderator Preston So (Contributing Editor) and guests Nik Graf (Senior Software Engineer), Kevin Jahns (Real-time Collaboration Systems Lead, Yjs creator), Fabian Franz (Senior Technical Architect and Performance Lead), and Michael Meyers (Managing Director) for an in-depth conversation about why ProseMirror is the best tool for our client’s project requirements.

Be sure to check out our related #TagTeamTalk, A Deep Dive Into Real Time Collaborative Editing solutions (e.g., Yjs, Collab, CKSource, etc.)


------------------------------------
Further reading
------------------------------------

ProseMirror Editor
------------------
https://prosemirror.net/
CZI ProseMirror: https://github.com/chanzuckerberg/czi-prosemirror/
Prosemirror Tables Demo: http://cdn.summitlearning.org/assets/czi_prosemirror_0_0_1_b_index.html
ProseMirror Atlaskit Yjs Demo: https://yjs-demos.now.sh/prosemirror-atlaskit/

Draft.js
------------------
https://draftjs.org/
https://github.com/nikgraf/awesome-draft-js#live-demos
https://quilljs.com/guides/designing-the-delta-format/

CKEditor 5
------------------
https://ckeditor.com/ckeditor-5/
CK5 Demo: https://ckeditor.com/ckeditor-5/demo/

Quill.js
------------------
https://quilljs.com/
Quill.js demo: https://quilljs.com/standalone/full/

Slate.js
------------------
https://github.com/ianstormtaylor/slate/blob/master/Readme.md
Slate Demo: https://www.slatejs.org/#/rich-text

TipTap
------------------
https://tiptap.scrumpy.io/

Fidus Writer
------------------
https://github.com/fiduswriter/fiduswriter

CodeMirror
------------------
one of the most popular code editors for the web
https://codemirror.net/

Text Transcript

Preston So: - Hello, and welcome to the second ever episode of the Tag Team Talks. Today we're gonna be talking about rich text editors and some of the solutions that are out there in this very exciting and growing space. First thing I want to do, though, is get a little bit of a look at our guests today. My name is Preston So. I am the moderator and contributing editor to Tag1 Consulting. And I'm joined today by several amazing folks from all around the world here to talk about rich text editing.

Michael Meyers: - Awesome. My name is Michael Meyers. I'm the managing director at Tag1. I handle business development, sales, partnerships, marketing, strategy, client relations, things of that nature.

Kevin Jahns: - Hi, I'm Kevin Jahns. I'm located in Berlin. And I'm an expert for shared editing and CRDTs. I currently work for Tag1 consulting on a realtime system.

Nik Graf: - Hey, I'm Nick. I've done a lot of frontend development over the last couple years, and also was digging into Draft.js, actually built a plugin system on top of Draft.js. And now doing a lot of work on the same project as Kevin, the realtime collaboration stuff with ProseMirror.

Fabian Franz: - Hi, my name is Fabian. At Tag1 I'm currently a senior technical architect and performance lead. But on this project I'm especially excited about bridging the gap for the editors. And I'm a Drupal enthusiastic, Drupal 7 core maintainer, but also a longtime Drupal Aid contributor where we're also kind of having this switch over from CKEditor 4 to maybe CKEditor 5, so going to the next generation. So this is really exciting to be working a project where we're exploring all of that.

Preston: - Thanks very much to all of our guests. It's a real pleasure to be here with all of you today. This is a very meaty topic. We're gonna be talking for quite some time about this, I'm sure. But first, I just want to say good morning, good afternoon, good evening, to wherever you are in the world. And if you haven't already checked it out there's actually a previous webinar that we've done related to this topic on collaborative editing. It's about how it relates to the ways in which people work today. And I want to make sure that we refer back to that webinar so please take a look at the link also available on this page. Alrighty, so let's go ahead and get a quick background on Tag1. Why are we interested in rich text editing, Mike?

Michael: - So, Tag1, we handle mission critical systems, emergency management. We've helped organizations like the American Civil Liberties Union go from raising $4 million a year in donations to over $120 million a year after President Trump in the U.S. came into power. So we do a lot of performance and scalability. We do high availability. We work with a lot of Fortune 500 companies like Symantec doing cybersecurity, infrastructure management. For this particular project that we're gonna be talking about today we're working with one of the top 10 Fortune 50 companies. They are rebuilding their intranet. It's a highly available, highly scalable, mission critical system used across 200 countries with over 20,000 active users in well over a dozen languages. Realtime collaboration is key to how the modern workforce operates. I spend a lot of my time in things like Google Docs collaborating with the team on all sorts of things. And while our goal with this intranet is to integrate a lot of different systems and not reinvent the wheel, so for example, you'll get a summary of what's going on in Slack on the intranet but all that information comes from Slack and the idea is just to link you off to Slack. These days, people use a lot of third-party tools for what they do best. The challenge with that is that they are disparate systems. And so if you have Box, and Slack, and Quip, and all these other things, it's hard to know what's where. So this system really organizes all of that with centralized authentication and user management so you can, say, create a space for a particular group and it will spin up all of the necessary artifacts we need, from say Slack to Quip, manage permissions. You can use any of these systems independently but everything is sort of synced, meta searched, and managed across this centralized system. And then a key component of this system itself is collaborative editing. And they have, as you can imagine with a global workforce of 150,000+ employees, they have a lot of people with different uses cases and needs. And so, some people, let's say technical people, love Markdown and want to work in one type of editor. People in other groups and departments might prefer WYSIWYG. Some people want to be able to edit HTML directly. And so, the reason that we're looking at editors on top of the ability to do realtime collaboration and work together on information in realtime is that we need to accommodate a lot of features, plugins, enhancements, and different users in different spaces. And so we took a wide range, an assessment of a wide range of editors in the marketplace, did an analysis based on our feature requirements, narrowed it down to a field that we're gonna talk about today, and ultimately selected an editor.

Preston: - I think this landscape is quite broad. There are so many options out there, and it's very difficult to choose which ones are appropriate, especially given that there are so many requirements that people have today. And being able to actually choose based on a variety of different features, which we'll talk about in just a little bit, is a huge prerogative. I mean, there's two areas that you just mentioned, Mike, that are very interesting. And the first is the realtime collaboration which has its own challenges and its own difficulties. Which was the subject, by the way, of our inaugural Tag Team Talk. And of course our second topic today, which is really what a rich text editor is. And combining those two really unleashes a lot of power for these content editors, but it is also very challenging from a technical standpoint. But let's go down to the very, very basics here, and the fundamentals. Sort of in its most basic sense, how would we, as a group, define a rich text editor?

Kevin: - I think that's really, really hard to make a general description of a rich text editor. I think most people think about Google Docs when they hear that. But I would say that a really basic rich text editor is something that supports boldness, italic, and headlines. That's it for me. Because often you really need that feature set. That's basically what you have in Markdown and that you want to have in all the other editors. Sometimes you write a blog post. You basically only need these features. For us developers it's really important to have code blocks too. I think that's a really important feature. But I don't think everyone agrees here. There's links and tables. Actually, a lot of people expect tables but not all editors support tables. So for me, a rich text editor is, yeah, something that supports this, contrary to pure text editors that only support working on text, maybe only paragraphs, and no rich text formatting.

Preston: - Was there a certain minimum, like a threshold that you wanted to reach in terms of the number of features? I know that you all have done a really strong comparison of all of the features available. Was there a certain point where you said, okay, well, we can put a dividing line here where we can say, all right, everything above here we can definitely look for, but everything below this line perhaps maybe we should strike out of our requirements?

Kevin: - I think a baseline for this project, yeah, we had a baseline, a feature set that we want to implement. And for our use case it was really important that our editor is adaptable. And this is not a requirement for all the projects that are out there. Sometimes you really just want to have a plug-in editor that just works and does the basic stuff right. But for us, we wanted to do some custom stuff, and some editors support that, and some not as well.

Nik:

- I could dive in here and give one example that Kevin mentioned. This is for example the tables. I worked a lot with Draft.js in the past, and I know you can do tables, and it's possible. But if you want to do more than just a simple text field and then have, rich content, again, in the table field, this is really, really hard to do with Draft.js. So what people came up with ideas like Draft.js per fields, like editors per field in the table. And then this gets really, really heavy because this has to run on the web browser. While others support this because the structure internally, how the data is managed is completely different. This is completely different. Basically, depending on what your needs are it completely rules out certain editors right away.

Fabian: - Yeah, that's also what I've found in my research of editors. Tables are really tricky with an image in a table, where every normal person is like, hey, that's so easy, it should just work in that. I've also seen for two other editors, either Slate or Quill, where the table plugin was basically instantiating another complete editor within that and then doing some magic to hide the one toolbar, show the other toolbar, so that it's still a seamless experience. Once you go away from those basic features like bold, italic, those they all can do, code play, quotation maybe a little bit more complicated. But basically, what's kind of used, what you are used to from all the old editors, most can do, that's not a problem. But once you get into the nitty gritty and really want some features like autocomplete, you type something and you get a table or something like that, we have that not yet, but it's so useful, and so practical, and so nice. But some editors, it's just way more harder to implement than others.

Preston: - I think we can all agree that as it gets more and more complex you kind of question the usefulness of some of these, especially the inline tables or some of those formatting options.

Preston: - Well, I think we've talked a lot about formatting, and clearly, formatting is a very, very strong interest to a lot of the content editors that we work with on a daily basis. But I think we also, Mike mentioned earlier something very interesting which was about document formats and the interchangeability between those. That's also a very important feature of rich text editors. Because you can do whatever editing you want to, but if you can't extract it and move it into a different format, or make it usable for other systems, it doesn't make any sense. And so I'm curious, when we talk about these document formats, do all of these editors support Markdown, HTML? Do they all support rich text? And I guess my even more pertinent question here is how easy is it to switch between them in these editors, or is it possible at all?

Kevin: - I think it's important to distinguish the underlying document model of the editor, that is often JSON-based, especially in rich text editors, and how you interact with the editor. Most editors somehow support some positional parameters. So, insert something at position X, something like that. Because that's how most humans comprehend text editors. So we somehow try to port that to rich text editors. There's some editors like ProseMirror that are more structured so you really need to say, okay, I want to insert something in this paragraph, inside this table, at this position. But this is also translated to index parameters. Because also ProseMirror, which is structured internally, accepts something like index parameters. So, insert something at position one, for example. And I really like that, especially... Like, in comparison, Quill.js has also an internal model that is purely based on position parameters. It accepts changes that are defined as the delta format. And I really love this data format because it's a really easy description how to insert something into the editor. It's index-based. And it is also perfectly suited for collaboration. But something that is really hard when you only work with index parameters is designing tables. So, when you work with tables, I think something like ProseMirror which is more structured, it's something really cool.

Fabian: - What Kevin said is very great and important but might have been a little bit too much already, too deep already into what our audience expect. So I would really like to step back and just show what is this kind of document model. And we are here at a very exciting point for me personally because we are at a transition point. All of the old generation editors, CKEditor 4, and whatever you all have, some nicer, some not so nice, they all have been built on something that's called a contenteditable. This contenteditable was basically supplied by the browser. It allowed basic formatting, and every of the trillion browsers that are out there, even browsers within the same name, implemented it differently. It was a huge headache. So all the editors said, no, no, no more contenteditable. We really don't want that anymore. The huge advantage of this old generation of editors is you threw them some HTML, it was outputted out of Word, and they could take it. It might have not looked nice, but they could take it. They could see it, they could display it. You could even edit it. So you just threw some HTML on them and then you get also HTML out. So for something like Google, that's perfectly suited. You load the HTML from the database. The user edits the HTML, now it's saved again to the database. The new generation of editors, CKEditor 5, ProseMirror, Quill, they are all having some internal document model. And we are seeing that a lot also in other cases of the web and everywhere, that we're kind of using these extra technologies, these languages that allow to express the same what was in the HTML, but differently. And because they are all having these internal document models what you can do is you can, for example, take... In theory, at least, you can take the same document model and ones display it as What You See Is What You Get, but ones who could also display it as Markdown, as long as you don't have something that Markdown doesn't support within it, and you can basically transfer it back and forth. Because the only thing that changes is the transformation from document model to what the user is editing, and how you are working on the document model. This was the other thing. And that makes for really cool demos. We'll put in a link to a ProseMirror demo where you can really, you have the technical person there, no Markdown, all comments out of the hat, they're just putting in Markdown. You have a non-technical person, they can collaborate on it in the same document because the other person can just click on bold and it's bold, and they see it as What You See Is What You Get. And that's so cool in the new generation of editors. And later we'll talk a little bit about the challenges, but I think that was a good introduction.

Preston: - And just to add a little bit even more context here, I think when you talk about, Fabian, the ways in which we've evolved over time, I mean, long gone are the days when we had those phpBB formatting toolbars which were limited to let's say three or four different buttons, and they never worked half the time. To nowadays, this very advanced and almost abstract way of working that is really kind of a layer above, where you're talking about working with JSON as opposed to working with direct HTML or direct text. We're actually talking about an object tree, which is really amazing, and I think is very compelling. So let's go ahead and move a little bit towards some of the more business requirements here. I do want to talk a little bit about this Fortune 50 client that you mentioned. We know that all of these editors and all of these powerful tools do have the functionality to do these formatting changes, have the abstraction layer as part of this new, this kind of new document model that we talked about. But I wanted to ask, there's kind of differences in how each of these editors like ProseMirror, like Draft.js, like Quill, how they manage all of the underlying data, and also how customizable they are. Can we talk about some of the key requirements here? What's maybe some of the major characteristics that you all wanted to see come out of this project?

Michael: - Before we jump into the technical stuff, I think one of the key things, well, first of all, it had to be collaboration ready because we're integrating this with a realtime collaboration system. But beyond the extensibility that Kevin talked about, which is critical because their needs are constantly changing, we need to integrate it with a lot of different third-party tools and systems. We want to add things like @mentions that tie into central authentication. I'll let these guys dig into that. There were a couple of business requirements. One of them was, you know, prove it. We looked at some really interesting editors that are still in the earlier stages of development, and we could swap them in in the future. That is another aspect of extensibility. We may choose to change editors in the future, or give different users different editors. But for launch we need something that's proven. Something that is really stable, that has a robust open source community behind it that is continuing to develop it with maintainers that are really responsive. We wanted to make sure that it was being used in enterprise production by large organizations. So, ProseMirror, for example, is used by The New York Times. And they've written some great posts about it. They were generous enough to get on the phone and talk to us a lot about their experience to sort of confirm some of our research and thinking in real world scenarios. That was really critical just from a, before we could even evaluate these editors and dig into the features, there was sort of a minimum bar.

Fabian:

- Yeah, and also what was important from the proving standpoint, ProseMirror for example, and we will come later to that, Confluence almost everyone knows, many work with it. It's built up on Atlaskit, and Atlaskit itself is built upon ProseMirror so that was another plus point. The CZI, Zuckerberg Initiative nonprofit, they are building some Google Docs-like clone based on ProseMirror. Also very interesting. So we had several things to just work with, and to see, and use in that. You use those demos, and they just work. Tables look great, things work in that, so that was a huge plus point here for ProseMirror in just being proven by being used by other larger organizations.

Nik: - Maybe I can add a word to Atlaskit. I mean, we'll kind of dig in later. But Atlaskit, as Fabian already mentioned, Confluence is built on Atlaskit, but not only Confluence. Basically every Atlaskit is this designed system from Atlassian, and everything we're building at the moment, everything we're rebuilding, redesigning, is built on top of Atlaskit. So the Atlas editor, core editor, built on top of ProseMirror in their design system. And this also gave us in terms of like, I don't know, kind of showing off at the client a good headstart in the beginning. I mean, they had a different design and they had different widgets, but you could take a lot of that stuff, put a different design on top of it, and get a lot these tools out there. So, while it was not really a requirement, it was a really, really good way to impress early on. And yeah, because Atlas, Atlassian has done a great job with Atlaskit it could give us a good headstart. And yeah, accessibility, multiplatform, all of that, is built in.

Preston: - Let's dig into some of these. I think, Nick, you just mentioned multiplatform. I mean, this is a really interesting kind of idea where you should be able to use a rich text editor on whatever device. On a phone, on a tablet, on an Electron app. Can you talk a little bit about how you thought about multiplatform and why was it so important to these folks?

Nik: - I think in general, the world is becoming way more mobile. And while desktop is still in the use case, probably, for this intranet, but people more often check... They want to edit something on mobile. And while we currently don't talk of it, we wanted to pick a platform that we later on can expand to it. I can tell from, like, some editors, they have their fair share of troubles with mobile because simply the environment, it is different browsers behaving differently, so the underlying document model sometimes already struggles. Mostly they're working fine and it's a matter of your IUX. But yeah, you basically want to pick something that definitely works on all platforms so you can expand in all directions.

Preston: - And one thing you just mentioned also as well, Nick, that I wanted to call out is the notion is extensibility. You know, third-party integrations, being able to work with other tools. One thing Mike had just mentioned was the notion of being able to tie in @mentions and integrate that with the central authentication system. I also know that there are other third-party tools you want to integrate with, and that you see as being important for these rich text editors. Can you give some examples of those? All the group feel free to jump in as well.

Nik: - Yeah, absolutely. Let's say you want to reference a Dropbox file, or a Box file, or you want to mention another user. These are then custom nodes. So you have an editor that only supports the standard HTML text and doesn't allow you to make your own nodes, then you can't do this. That's why this goes back to this document model. Basically, the document model of the editor has to be extensible so you can actually add your own, extend it, and then allow to build a user interface to add these custom nodes in there. And then however you want to implement them, you can just reference an ID to a GitHub issue, for example, and then you could load the data on demand or you could actually put the data in the document. This then, it ties together into, like, into this authentication system, and how you load the data, and so on. This is very dependent on the needs of security and customer requirements. But in the end, the gist of it, you want to be able to create something where you can add a toolbar item to add GitHub issues and connect them, and have them in the rich text document. This is something where you... I mean, you could have for example even still custom markdown syntax. This is where WYSIWYG usually outperforms Markdown or other systems, by far, because the experience is so much better.

Fabian: - For example, hover over them and then it would show you all the details of that GitHub issue, if there's a pull request or not, et cetera. Possibilities are endless, obviously. And I think that's so very cool about that. What's also really great about this kind of editor integration is that there's so many possibilities in extending that. For example, one thing we didn't talk yet about much, correct me if I'm wrong there, but we're building everything on React components. React, you just have your standard React component for like an autocomplete and then you can put it in the editor. And also another nice thing here about ProseMirror where you have the display in the document and how it's displayed in your document what's outside your document different, and that's also another important part for accessibility, which we probably also wanted to talk about.

Preston: - Absolutely, yeah. Accessibility is a topic that's very near and dear to my heart personally. I know that, Mike, you just mentioned earlier as well that when it comes to a large Fortune 50 company like this one, being able to work with this very large workforce that has a variety of different abilities and a variety of different needs is important. We alluded earlier to some of the challenges around accessibility with rich text editors. We talked about things like contenteditable, the contenteditable attribute on DOM elements, ARIA labels. I know that we've looked at some of the capabilities and we've talked about some of the nice things that these editors have. Are there any that I've missed besides the contenteditable and some of the ARIA features?

Nik: - [Nick] Kevin, you want to take this, or should I?

Kevin: - You do, please.

Nik: - In general, I mean, a lot of accessibility you get out of the box if you have the same content... Or, a document structure in HTML. So if you have like headlines well structured and so on, it makes it easier for people, for the screen readers and so on to parse the content and for people to jump around just by their voice input. But then if you... If you actually make popups, dialogs, if you have toggle buttons, that you get into the nitty gritty details, if you make your custom ones you really have to take care of accessibility by your own. If you look at all the standard toolbars and buttons that a lot of these editors provide, or come with, they have accessibility built in. And that's really good because that shows that this is already, like, a standard, that it's kind of expected. But as soon as you start to build your own, like, @mentions, or a GitHub plugin to reference pull requests, and you're doing your own popup and dialog, you really have to take care of yourself, take care of it by yourself. This is still a lot of work. We were fortunate that Atlaskit did a lot of good stuff out of the box. We already got the feedback. There are a couple of improvements we can do. But that's okay. The initial response was quite impressive. Maybe the gist of it is like, even with these new editors, although they're using contenteditable, you can make them very accessible but you have to, as soon as you do custom stuff you have to do it by yourself and you have to take care of it.

Preston: - Yeah, and I think that, you know, you just mentioned this notion of all of the custom work you have might potentially challenge some of the accessibility and be a problem. This is where having that notion of great flexibility, and extensibility, and customizability, comes with great responsibility as well. I know that one of these you mentioned was the popups. For example, having that autocomplete widget show up with the @mention, that's very challenging. As somebody who likes to think through how would I build that accessibly, I actually don't know where I would start. That's a very challenging one.

Nik: - We very recently had a call with an accessibility expert to talk through that one. And yeah, it's... There are things like... I've built a lot of React components in the past that were focusing on accessibility, but even I learned in this call a lot about your concepts like live regions. You can have a live region in your document and then you can announce, basically, state changes. So for example, one thing that we learned that we're currently not doing yet, but we definitely want to, is if you toggle, if you have some text and you toggle it to be bold, you should announce that. You should announce the state. Is it now bold, or is it not bold? Because by just hitting the toggle, if you listen to the voice, the screen reader will just tell you, you toggled the bold. You toggled the bold status. Like, uh, okay, but which one is it now? This is very, very interesting. You really... What I learned basically is turn on the screen reader and dim your screen so it's black and just try to... All the actions that you usually do in this text editor, try to do them just by navigating around with your keyboard or with voice. If you can get through it, then you're in a pretty good state already. By doing this test and this call, and learning about all these things, we noticed a bunch of things that we're missing. But we're working on it. It's an interesting journey.

Preston: - Well, I know what I'm gonna be doing this evening after I get off work. It sounds like actually a lot of fun. Like, playing the game Myst or something. I know that there are also some specific requirements that were more interesting. And I think that there are definitely some interesting implications that come about when you mix rich text editing with some of the other ways in which people like to work with components on the page. Like, maybe the most important component or most popular component right now, React components. How exactly have you thought about things like embedding React components or embedding videos? I know that you've thought about actually placing React views straight into these rich text editors. How has that worked out for you all?

Kevin: - I think that's definitely one of the more interesting things about ProseMirror. Because a lot of people seem to do that. They plug in their rendering engine, like there's UJS. I know a project, TipTop uses UJS instead of React. Other projects like Atlaskit just build on React to render custom node views. And you can basically render everything in the editor that you would render on a website. I saw a project where you render a PDF using a React view because there's this great PDF, React-PDF. I think it's called React-PDF. It's a really cool project. And you just plug it in, and you have a PDF inside your editor. That's really cool, right? There's a lot of other stuff that you can do just like that. And because ProseMirror is already built on this concept of mutable state it's a really nice adoption to just use React within ProseMirror. But because you can do everything without React, I would argue that in Quill.js it's really hard to use something like that, React inside the editor. But still you can do everything you want. You can build your custom tables and stuff like that. But React certainly makes a lot of stuff easier because they have a lot of knowledge in React. So, it really makes stuff easier.

Fabian: - Not only that. There's also the possibility to just reuse components and then combine the best out of the React world. That is also important from a perspective of how to get developers for this project, that really we focused on those collaboration developers as well as React developers to get the best of the best.

Kevin:

- I think we can all agree that you shouldn't manipulate the DOM manually anymore. For the editor itself, we have ProseMirror to handle the DOM. And for custom views, like for any kind of custom stuff, for example how a table is built, like there are a lot of divs around there, a lot of CSS, I wouldn't do that directly in the DOM and manipulate that information in the DOM directly. There are a lot of edge cases that you need to handle, and you can do a lot of stuff wrong. So, React really helps, I think.

Nik: - There was one more specific requirement that is probably worth mentioning, is our comments. We have these annotations or comments. And this is a very interesting aspect that we learned over time. There was for ones this requirement that for a different permission level it shouldn't be part of the document model, so we wanted to have that out of the document model. But it's also really interesting that if you start to do an annotation and you share the document model collaboratively on realtime, you don't really want to... If you're making a draft of a comment, you don't want to share that realtime. And this is the same for let's say @mentions. If you start typing, you don't want the other person on the other end see the autocomplete suggestions. This needs a little bit of rethinking because you basically have parts. The document model is really like the content that you want to share realtime. But there's the other parts like user interface elements or annotations that are seen in draft mode that you want to keep out of it. And then that's really, really useful to share the same component library so you can actually stay in the same system and not build the editor with one UI library and then build these other user interface elements with another library. It's really handy to use the same thing. It keeps us sane and easy to move forward.

- [Kevin] Well put.

Preston: - Let's jump into some of the actual tools that are out there. I think that we've heard a lot of names thrown around. There's been a lot of very interesting technologies mentioned, and a few that we haven't mentioned. We've talked about ProseMirror briefly. We talked about Draft.js. Very briefly talked about Quill and CKEditor 5. But there's also some others. There's Slate and TipTap. What are some of the top open source... When we look at these open source editors on the market, which ones were the ones that really were compelling for you all? And what were some of the strengths and weaknesses?

Kevin: - I think, for me, the biggest strength... Like, we can talk about each of the editors separately. Maybe we go from the most popular ones. Maybe Fabian can explain something about CK 5. He has the most experience with that.

Fabian: - Sure. CKEditor 5 is the popular successor of the very, very popular CKEditor 4. It also switched to a JSON-based model. However, they do up and downcast everything so what you basically in the end still get is HTML. So you have HTML, or you have structured HTML. So, for example you cannot have your hello tag. The document would just not know, what the heck is a hello tag, or a blink tag? It would just ignore it and everything that's in it. Because what it does, when it loads the HTML it loads it into its own document model which is also JSON-based, and then it puts it out. Basically, CKEditor 5 is pretty strong. It has good accessibility. And it also has nice collaboration. The collaboration just had one big flaw. It was not open source. This was unfortunately a deal breaker in terms of the extensibility in that, and also putting anything out for everyone. I mean, Drupal is open source. We work a lot with open source at Tag1. We love open source, and it's so cool that Kevin, as the developer of Yjs, is here. That's also kind of how we found Kevin and the other three. We directly talked with the people who are developing these editors and checked them for our project here when we were interested in some part, et cetera. That was kind of the team we then settled on. But CKEditor 5 is still a little bit young in its thing. It has just recently gotten out of beta. ProseMirror has a little bit longer history in being stable and being used for those things. It will not be a concern because some other big players at settling on CKEditor 5. But just saying experience in how you long you work with something is not worth playing around. And then there's a huge compatibility break with CKEditor 4. So what could have been a huge advantage with CKEditor 5, that all of our Drupal backends will directly work with it, et cetera, is no, because there's just a real break between. So, CKEditor 5 is a completely different product than CKEditor 4. Which has its advantages, but as there is no backwards compatibility, and the collaboration module was not open source, we looked more at the other editors. Slate, for example, we've not talked much about it. It's a great editor. It has, from what I've seen, the nicest backend system. It's really cool, very nice in that, but it's beta, and it's beta since a long time. And we want something proven, we want something stable. And something in beta where there could be hard BC breaks was just too much risk for us here in this project. Nick can maybe talk more about Slate because he knows. Draft.js was more like the original Facebook thing, monolith. It's a great editor. It has a nice backend system, it's React-based. It is harder to extend overall in that, and it's also aged a little bit in that. It's one of the more older editors in that. Also, the community is not as active as, for example for ProseMirror, where it's mainly Facebook committing some patches here and there and maintaining it at the stable state. But in the end, it didn't have the features we needed. So yeah, that was kind of the thing. TipTap with roo. If anyone needs an editor for rooJS, use TipTap, it's great. And it's also ProseMirror, basically, so yeah. It's kind of like a product for ProseMirror for roo. That's cool. And then we ended up with ProseMirror and Quill, and that was kind of the huge race between ProseMirror, Quill, going that. Now, Yjs supported both, so it was also no thing here. But in the end, ProseMirror won basically on the experience. Also the tables plugin looked much nicer in its experience, and how it looked and everything. Quill, the database format is great. It's also a collaboration ready editor. It directly works. But you are then... You need to use the data format it provides, and you then need to use ShareDB. And that again was putting our flexibility a little bit to the test. It's also OT-based, which we talk a little bit about in the other talk. If you're interested, check that out. And we want really something that, where in the end, maybe we'll never get there, maybe we will, but in the end we could at least think about a future of offline editing. And that's again something we talk about there. But Quill and ProseMirror was a really nice race in that ProseMirror is more giving you a framework where you have like nothing and you build your own editor, and Quill is like a ready made editor. You plug it into your application, it just works. It's great in that. But as you add Atlaskit, then Quill got out of the race.

Preston: - Yeah, I understand that... Oh, sorry, go ahead.

Kevin: - I think this was one of the bigger selling points. I think we had Quill and ProseMirror in the end listed down, and we compared it. Like, Quill.js has ShareDB. It's a proven concept, operational transformation. It also works with Yjs. There are a lot of companies that already use Yjs with Quill.js. And then there's ProseMirror. ProseMirror has all these teachers, a great community. I think it has a really interesting concept. And most modern editors nowadays, like all the new editors that pop up, for example Atlassian, they are all built on ProseMirror. We also have the Collab module which is kind of similar to OT. It's like a reconciliation approach. It doesn't really handle conflicts as well as operational transformation, but it clearly works. It's And also, Yjs works with ProseMirror, so I got you there because either way, we could choose any of the editors with Yjs. And this is what I really wanted to do. I really explained that and why we did that in the last webinar. But I think the biggest selling point I felt was one, the community behind ProseMirror, and when we saw Atlaskit and that we could just build on top of Atlaskit. Because we had this existing editor, a lot of features, nice tables, nice interactions. This is, I guess, a big selling point of ProseMirror. And a lot of open source components that you can just use, plug into the editor, and it just works. So yeah.

Preston: - Absolutely. One of the things I know, Nick, you mentioned about ProseMirror was the fact that Atlaskit helped so much more. Is there anything more that you wanted to mention about Atlaskit? I think it's a very compelling story.

Nik: - I think not much more to add. I quickly could repeat that because there's so much there that you really, you can simply start using Atlaskit then you have a good headstart. The biggest trouble you might have, and we went through this, is that this is a big mono-repository, so we had to take out the parts, the core editor, that we needed, and then basically continue to use the rest from Atlaskit and take the bits and pieces. Like, slowly replace the bits and pieces that we actually needed to be changed. But this was like, from a demo, or the experience in this project, this was very good because in a very short period of time, I think it was just a matter of like two weeks or so, we had something ready to show that the client could try, use, and actually feel. And then obviously if you then can test with re-users and so on, or potentially your re-users, you're making better decisions than like, just coming up with, like, hey, and we might to do this, and that, and have a button here. But if it takes you quite some time to slowly build it up than rather starting from something that is fully fleshed and then replacing bits and pieces, I think that was for, like, product thinking and product development, a really compelling story. It was possible for Atlaskit.

Fabian: - Definitely Atlaskit very great for us. And that was what Nick was saying was part of our strategy with this client, that we are showing progress every two weeks, and a huge demo. And with not only the client itself but there's a huge stakeholder team that can all watch the progress of that, how it's done. That was really great to make a great impression quick. But not only a great impression quick. What you shouldn't undersell, Nick, is how you definitely improved the build system. Because I think it took like three minutes, at the start, to just build everything, and now you've put it down to 30 seconds and rebuilds are like 10 seconds. No longer like 30 seconds wait time. Like, change some CSS, wait 30 seconds, drink a coffee.

Nik: - We should say there, props to Sebastian, to me. We was digging into the webpack configuration and getting the hot rebuilding working with good compile time. This was really helpful for faster development. One thing I could add there, though, is like about Atlaskit. Atlaskit is not built with realtime collaboration in mind. For example, certain features, they do things like they change the document model for just like showing different user interfaces. So for example, the @mentions that we are building, and the annotation or commenting section, we basically, we cannot use what's there in Atlaskit or we have to adapt it and change it. Otherwise it would be synced across to other users, and we don't want that. So, while Atlaskit was a good start, we now have to really, especially with this realtime collaboration in mind we have to change a lot of things. But that's fine. I think this was the strategy and the approach, and it was a good one. Highly recommended.

Kevin: - I think it was built with realtime but they use a custom version of the Collab module, which is like a different realtime approach. So we just plugged in the Yjs plugin so all the data that you have is shared immediately. And I'm sure that they have some, I don't know, filtering on the backend happening, filter that out, like all the data that you don't want to share. I'm not exactly sure how that works. But also, backend to the Atlaskit collaboration approach, it is, I think, proprietary. The source code is not available, I think. I'm not sure.

Fabian: - I haven't seen it. I've searched everything on collaboration that's out there on the internet. There's even some prototypes from The New York Times that still can be found. There's a five-year-old ProseMirror version if someone will dig into history.

Preston:

- Absolutely. Well, we are starting to run out of time, but I do want to give some opportunity to talk about some of the more interesting aspects here. By the way, what sort of additions have you made to ProseMirror? Just very, very quickly. I know, Fabian, you've done some work on this.

Fabian:

- One of the important things, and I've already talked a little bit about that with how I explained the document models is reintegrating ProseMirror with Drupal. Now someone says, well, but yeah, Drupal supports many editors, but yes, only those of the old generation. So what we are now talking about is kind of we have these JavaScript mega beasts that are usually run with Node. And they are coming to the old giants of PHP, like old not in terms of being very old, but Google has been around 20 years and it's traditional PHP-based. It's just base HTML that you store in a database. And you have this ProseMirror and it has this JSON model. And how you usually would do that is you would take this JSON, run it through Node. Node would launch a little instance of the editor then would display it, and then the webpage would be delivered. We cannot do that because we are basically bridging this old generation of editors with the new generation of editors. And that's very, very interesting because when I was starting up with that, the React developers were like, why do you want to output HTML? Why do we need that? And the Drupal developers were like, JSON? Why would we put JSON in the database? We are storing HTML, we are storing JSON. We're storing both in the database. We're storing the HTML only kind of like a cache for display purposes. That's what we will be displaying to the user. And the JSON is what we then feed again into ProseMirror, or Atlaskit, or our custom editor. In this case, for loading up the same state as it was before. So, that's very important in that so we don't need to store to HTML, load the HTML again, store it again, and convert it back and forth where we could be losing data where it's prone, but we are storing the document model of ProseMirror directly in the database. We are storing also the HTML so Drupal can display that. That was a little bit of a challenge. A challenge that the whole of Drupal agents somewhere or another will also face because now we're going, kind of like with Drupal itself, with Drupal core, into this direction of this new generation of editors. So, that's a lot of challenges, and I hope we can speak in more detail about that at some other point. But it's really interesting. And then also, just loading this whole React, and then you have a frontend which is still jQuery-based, dramatic person, and AJAX-based, and it's this traditional how Drupal used to work in that. But now this new framework comes and now you want your @mentions but also working with some traditional Drupal comments, and you have to combine those two worlds. And that's very, very interesting.

Preston: - So, it seems that... Oh, sorry, go ahead.

Fabian: - There's one part that we did that was really exciting for me besides all of those mentions, collapsers, sections, and the collaboration and shared editing.

Preston: - Well, unfortunately we are out of time. I did want to get to talking about the integrations but clearly we will have to save that for another time. I just wanted to say thank you so much to all of you in the audience for watching or listening to the live Team Talk. For all of the things that you heard in this call, in this webinar, things like ProseMirror, things like Yjs, things like Draft.js, all of these things, we're gonna have links with all of these technologies that you can take a look at. By the way, please don't forget to check out our previous webinar, the inaugural Tag Team Talk about shared editing, collaborative editing. And by the way, if you're interested in learning about a particular area or a certain topic, please feel free to reach out to us and the team at [email protected]. I want to give a big thank you to our guests today. First and foremost, Nick Graf, our senior software engineer, based in Austria. Fabian Franz, senior technical architect and performance lead. And Kevin Jahns, realtime collaboration systems lead and creator of Yjs. And of course, the managing director of Tag1, Michael Meyers. This is Preston So. Thank you all so much. And until next time, take care.

Sep 18 2019
Sep 18

[embedded content]

What is real-time collaborative editing, and what are some of the most compelling technologies available in the space? In the inaugural TAG Team Talk, hosted by Preston So (Contributing Editor, Tag1 Consulting), we conduct a wide-ranging discussion about both the business prerogatives and technical ins-and-outs of real-time collaborative editing and its landscape today, with our guests Kevin Jahns (creator of Yjs and collaborative editing expert at Tag1 Consulting), Fabian Franz (Senior Technical Architect and Performance Lead, Tag1 Consulting), and Michael Meyers (Managing Director, Tag1 Consulting). In this talk, we explore collaborative editing, diving into how it works and some of the challenges borne by shared editing. Through the lens of Yjs, a real-time collaboration framework that supports not just text but also collaborating on drawings and 3-D models, we take a look at Operational Transformation (OT) and how implementing Conflict-free Replicated Data Types (CRDT) drives decentralized server approaches in collaborative editing and supports more robust distributed applications with true real-time support.

Yjs: https://github.com/yjs/yjs

ProseMirror: https://prosemirror.net

Great Overview of CRDT
https://conclave-team.github.io/conclave-site/#conflict-free-replicated-data-type-crdt

Deep dive int CRDT by the author of Automerge: https://www.youtube.com/watch?v=yCcWpzY8dIA

Yjs was inspired by:
Sharedb https://github.com/share/sharedb
DerbyJS https://derbyjs.com/

Text Transcript

- Hello, good morning or good evening, wherever you are. And welcome to the first ever Tag1 Team Talk. This is Preston So speaking to you loud and clear from New York City. I'm the contributing editor to Tag1 Consulting. And it's my pleasure today to jump into a deep dive into realtime collaboration tools. I'm joined by three guests today. First and foremost, I want to introduce my partner in crime Michael Meyers, the managing director at Tag1 Consulting. I'm also joined by two folks from around the globe. We've got Fabian Franz, senior technical architect and performance lead, as well as Kevin Jahns, creator of Yjs, a key contributor to realtime collaboration projects, and a maintainer of open source projects. So, I wanted to start off by introducing ourselves one by one, and then we'll jump into what Tag1 Consulting's all about and get into some of these meaty topics we've got for today. My name is Preston. I'm the principal product manager at GatsbyJS as well as contributing editor to Tag1 Consulting. Mike, you wanna go next?

- Sure. Mike Meyers, managing director of Tag1 Consulting. I have 20 years of experience working in technology, managing and running teams. I've been working with Tag1 first as a client for over a decade, and joined the team about two years ago.

- Hi, so, I'm Kevin Jahns. I live in Berlin, Germany. I'm the creator of Yjs open source framework for realtime collaborative editing. Because of that, I got many interesting opportunities to work for really cool, interesting companies on realtime editing. They mainly use my framework and need feedback or help to deliver a great product. I'm currently really happy to work for Tag1 and one of the Fortune 150 companies to make that happen.

- Thanks Kevin, pleasure to have you with us. And Fabian?

- Hey, my name is Fabian Franz. And I'm at Tag1, as you already said. I'm a senior technical architect. I really enjoy architecting large websites and making them really fast. And I'm so excited about realtime collaboration because I think realtime kind of is the future of the web. But we also did a DrupalCon presentation slightly related to that topic, in Amsterdam. And besides that, I'm a Drupal core 7 maintainer right now, and working with Tag1 is just fun.

- And we've known each other for a long time, Fabian. Well, perfect. I think first thing we want to do, though, is set the stage a little bit. What exactly is Tag1, and why are we all here in this room? From what I understand, Mike, you all are the emergency support and rescue, and performance and security experts. Can you tell us a little bit about Tag1 Consulting and what you're all about?

- Definitely. We're the company you turn to when you have difficult and challenging problems to solve. We build mission critical, highly scalable, highly available and secure applications for companies like Symantec. We've worked for them for over a decade. We manage and built their infrastructure work and oversee their agency partners, setting architecture design and doing manual code reviews for security and quality. What we're gonna talk about today is a project that we're working on for a top 50 Fortune company, Fortune 50 company. They are rebuilding their intranet, and realtime collaboration is a key component of it. And frankly, I can't believe that realtime collaboration isn't a default feature in every online application, certainly every CMS. When I think about my daily workflow, I'm constantly working in something like Google Docs, collaborating with the team in realtime. And I know a lot of people, we're talking to another client about a project and they do all of their work in Google Docs and then just pump it into their Django CMS. And so, we really believe that this is the future of a lot of applications, and we're excited to be working on this particular project, so we wanted to talk to everyone about it because it's got some awesome tech behind it.

- Absolutely, I agree. I think that realtime collaboration is one of those intractable problems. It's not quite on the level of some of the key problems in computer science, but it definitely is a very, very, a large problem set that actually hasn't been solved completely across the board. Google Docs obviously has solved a lot of those issues. But I want to dig a little bit more into why it is that we're all on this call, why it is that we're so interested in realtime collaboration. Obviously it's something that is important to all of us. But can you tell us more about this Fortune 50 company and what some of their requirements are, and what exactly realtime collaboration means to them?

- Sure. It's a gargantuan project, multiyear project. It's used by 20,000 active users at any one time, across 200 countries, in well over a dozen different languages. It interfaces with a large number of third-party tools and systems, from centralized authentication access control through to Slack integration, as well as integration with other third-party tools and a number of internal proprietary tools. The idea is that we live in a world where you want to integrate the best technology for the job as opposed to reinvent the wheel. And so, what this... Their new generation of their intranet is to really pull in and integrate all of these different systems into one place. Because the downside of integrating all these different applications is that it can be hard to collaborate and find things if you have files in Box, if you have documents in Quip, if you have communications in Slack. All these disparate systems have different organizational structures, typically are run by different groups and departments. The idea is that the intranet will be the central hub that pulls together all these tools and information, helps you find what you need, but also provides that in a universal search and collaborative layer on top of it as sort of like the metasystem where you can talk about everything from, and track those tasks and schedules, to what your roadmaps and sprint reports are for your particular initiative.

- There's actually, and I recognized a Dilbert comic about it, where he's like, I need this document on my desk tomorrow, the boss says. And he's like, would you like it via Dropbox, via Box, via mail, via Slack, via whatever, via Skype?

- That's right. Sometimes I work with a friend and we don't have an organization to work under with. And I sometimes wonder how can I send a large file, like just 100 MB, to my friend. It's really hard right now.

- Yeah, it is really difficult. And I think this really calls out one of the things that you actually alluded to earlier, Mike, which was the fact that we have a lot of folks who are using CMS's as part of their daily workflows, but they need a realtime collaboration tool to bring it all together. I think the fact that we've heard from a lot of folks that they use Google Docs before copy and pasting over to a CMS or over to their desired tool shows that this is really the nexus of where all of these interactions occur. And so, I guess I'm curious just to dive into some of the business requirements just briefly before we get into the technology. I know that's where we definitely want to spend a lot of time. But why is it that something like Google Docs doesn't fit the bill for this use case?

- I can't talk too much about the company itself. But for business reasons, they are unable to use Google Docs. They're a competitor company and they can't have highly sensitive, mission critical information on third-party servers. This entire application is air gapped, highly secure. They have very different levels of projects, from public to ultra secure. There's even separation of data and communications. So, things that are for the highly secure projects are stored on different servers and the realtime communication goes through its own system. Everything from physical separation of data and information to air gap. And while they are able to use... Or sorry, not able to use Google Docs, the other reason is that they want this highly integrated into the system. A third-party collaboration tool, whether it be Slack, or Quip, or Google Docs, or something that facilitates that collaboration, has the same challenging problem that we talked about earlier. And so, by integrating this directly into their intranet it's a core feature. And I should also mention that they want to go well beyond text editing. We're working on collaborative drawing, for example, so that you could have whiteboards. In the future you could have your agile Kanban boards show up on this system even though they're coming from a third-party tool via APIs. You can see that information. Our goal isn't to replicate all of these systems in the intranet, but to present you with the most mission critical information that you need at a glance, and to then link you into these systems. In the case of collaborative editing, it's such a core feature to enable teams to work together that it's built in.

- Also, one of the challenges with using something like Google Docs always is access. Being a large company, obviously they have a central directory service where everything is managed and accessed, and you now need to enable this third-party system this access. That's pretty challenging because they're mostly scaled for a few users, in that you invite your team, and then you work with them, collaborate with them on that. But then you need to put it back into the CMS and there's this kind of gap between your workflow so you're always like, now you've published a version, but now some more changes need to be done. Who will synchronize that, who is dealing with that, et cetera. It's basically two systems you need to work with. Even for our workflows where Tag1 had a Drupal watchdog before, it was challenging, and kind of what is the source truth once you get to a certain point in that.

- Absolutely. I think the notion of something like realtime collaboration being built in, being an intrinsic part of an end-to-end system, is a kind of key feature. But I think what you just alluded to in terms of permissioning, and having to figure out, having to reconcile user roles and permissions across two very different systems can be very challenging. And having the ability to do that in a way that is really integrated with the overarching system I think makes a lot of sense. Let's go a little bit away from the business requirements and let's dig into a little bit of the main concepts behind realtime collaboration. I know that all of us on this webinar and podcast all know some of the basic components. But just so we can bring it down to the level of what some of the core fundamentals are, how does shared editing exactly work? How does this idea of realtime collaboration actually begin from the standpoint of clients, communication, concurrency? Where do we start with? What are the main elements that are involved?

- I do think we start with a very basic, just what is shared editing in general. And many companies also use already something like Etherpad where you can just have some text and collaborate it. I know we also used it for the Drupal community when we had challenging things to solve just to facilitate note keeping a little bit in that. The most basic thing about shared editing is really just text. It's not formatted text like Google Docs or something. It's really just text. And this is very simple. So, you have an example of like, for example, I have the text written, "This is some text." And now Michael wants to make it, "This is some nice text." But I want to write a little intro, so I wrote, "Hello Preston, this is some text." And now you need to know a little bit about how editors work, and an editor needs to know how this works. My, "Hello," would be inserted at position zero, so at the start. But Michael's, "Nice," would be inserted at position 12. But now, if I write before him and the editor still thinks he should put it in at position 12, then it's wrong, because we kinda... The positions changed because I've added some text, so editing has shifted a little bit. And the idea of CRDTs here is that instead of having this position-based system, just on a very, very basic level, they are very more complicated but just as a layman's explanation, is that for every character, you are giving it an identifier. And this identifier is also a little bit how a human would reduce this conflict, resolve this conflict. Because instead of saying, hey, insert this, "Nice," at position 12, you're just saying insert it after, "Some." And that's basically the idea, kind of. And because of that, however the text changes, Michael's, "Nice," will always be inserted after the, "Some." That's kind of how you can think about how you get shared editing to work on a very, very basic level.

- Let's talk a little bit about the communication. I think that that was a very, very great explanation of how concurrency would function in this kind of environment. I'm curious, though, there's a notion of a difference in communication between how some of the traditional collaborative tools have worked in the past and how, let's say Yjs or some of the more modern tools do it today. I know that, for example, in the past the way that people would do this kind of collaboration would be through a server and a client. A centralized server that houses all of the information about what's changing and manages all of that, as you said. But one of the key differences that I noticed about Yjs is that it's focusing on a more peer-to-peer, a more decentralized approach. Do you want to talk more about that, Fabian or Kevin? What does that mean to somebody who is evaluating some of these solutions?

- The classical approach is like, Google Docs in general, operational transformation, it mostly works in the client-server environment. So you have a central point where you do all the transformations between the document updates. For example, a user inserts something at position zero and another user inserts another character at position zero. The server decides how the document looks like at the end and does the transformation before sending it to the other peers. So there's like a limited concurrency because there's only concurrency between the server and the client. And, like, it's pretty unfair because their operational transformation approach is they do work peer-to-peer, but they have a lot of overhead, a lot of metadata. In CRDTs in general, you don't need a central point of transformation. Like, all CRDTs, they allow for commutative operations. So it doesn't matter in which order your changes come in. The document will look the same for every peer as long as all the peers exchange all the document updates. And that's a really cool feature. So, you don't rely on a central server anymore, and you can actually scale your server environment. You can do really cool things with that. Most people think, oh, peer-to-peer, you have peers communicating directly with each other, and I am a huge fan about that. But especially for businesses, it makes sense to have still a server environment. And here, the advantage of Yjs or CRDTs in general is that you can scale your servers indefinitely. If you know that Google Docs is limited to a limited number of users, I think the limit of users who can concurrently edit the document is set to about 15 people. And after that you will just see a copy of some snapshot, and maybe it gets updated. I haven't looked into it for quite some time, but I know there's a limit. Which is kind of strange. The problem here is that you have centrality and you need to make sure that you never lose document updates because the centrality is also your central point of failure. So if this point, like if this server burns down, your hard disk, your document is gone. You can never merge again, you can never restore that document. But in peer-to-peer or decentralized systems, well, it doesn't matter if one system burns down. You still have another one.

- I think that single point of failure is a really key and worthy of emphasizing aspect here. Which is that one of the things that is a very key concern for enterprises today, and just companies in general, is the notion of having the ability to keep things decentralized. If you have that central point of failure, everything falls over, you don't have access to anything. And the notion, I think, of peer-to-peer, decentralized, shared editing, allows for local copies of those documents to be on every single node, let's say, of every single person who's working on the document.

- Yeah.

- Yeah, go ahead, Fabian.

- Even if it's very different from how, for example, something like Git works, because it has very different properties, you can use the same examples as advantages. For example, one of the nice examples that is always quoted with Git is some developers can work on a plane together. They can collaborate on some code on the plane together because everyone has a whole repository of all changes. And that's kind of the same with Yjs, to some extent. You can optimize it to not have solo history, but you can have solo history, so two people that happen to be on the same flight could realtime collaborate on that document and later send it back to server. But they would not need a central server component. They could just do it. They could put work on the plane. And I think that's a really cool feature. And not just two, it could be a whole company, like 50 people finishing a last minute document collaboratively. Internet on planes has gotten better, yes, but it would be really bad if your Google Docs suddenly while you're on that plane. With Yjs, you just have the guarantee that your personal server works and you can finish your last minute document.

- It was pretty amazing, we actually saw this in action, unintentionally. We were doing a demo to key stakeholders. We had just integrated the realtime collaborative system which we kind of built independently at first, with Drupal, the core of the internet. And we were doing a demo of the system in action completely in development environments. We're pre-alpha. And during this demo, the server failed. Yet the collaborative demo just kept going and working perfectly because it was peer-to-peer in the backend. And the infrastructure engineer on the call was able to get the server back online, and it was just absolutely seamless. So it was really cool. All of us were like, my god, we didn't intend to demo that. That was amazing.

- And I think that, I think Fabian, you mentioned something that was really key there, which is that idea of a spotty connection. Obviously people working on flights. This case that you just described, Mike, as well is a very good example of this. This kind of approach that focuses on the notion that everyone has a local copy, just as you said, with Git-based approaches as well, this is the kind of thing that even if you're in, let's say the middle of nowhere and have a really horrible 3G connection you can still get the editing capabilities, and get that sync going, and make sure that things stay up to date. I think it's a very, very key value proposition. I want to jump back into Yjs itself because I think that a lot of people are really interested in operational transformation. A lot of people are interested in what exactly CRDT is, and all of those things. But first let's talk about a very key issue in collaborative editing that I think really uses a little bit of discussion. And that's the notion of an edit history. One of the biggest challenges that all content, all people working in content governance and content strategy have is the ability to distinguish between different undo histories. How does undo really work in the context of shared editing versus single editors? And how exactly does that work from a technical standpoint? Is there a global state, what's driving that state? How do you reconcile some of these things? We can talk about it from the perspective of Yjs or other tools as well.

- Man, undo/redo, it's a really, really hard problem, especially in CRDTs. In operational transformation, this problem is not that hard. You can just do undo/redo your own operations. You can just do the transformations in the document history. But you don't really have that history in Yjs, or in CRDTs because it just, it works differently. It's not really operations that you put in a linear log. It's more like operational set are in some kind of tree, and you can't really decide, okay, where do I need to start undoing. So the thing in Yjs is... Yeah, so, the way I implement it actually is in this tree I give a path of the undo/redo operations. So I basically say, this is the kind of state that you need to remove when you hit undo, and this is the kind of state that you need to add when you hit undo. And this is basically how I manage it. It's kind of complicated without deep diving into how Yjs works. But yeah, it's mostly a solved problem at this point. So, the undo/redo thing, it works on linear text really, really well. And there's great progress on structured documents, like on ProseMirror, which is, it's one of the editors that Yjs supports. So, there you basically have this structured document, a paragraph which has a table and the table has some lists maybe, and there you type and hit undo. You really need a good sense of what to undo and what the user expects when you hit undo/redo. Yeah, there's great progress. And the undo/redo implementation is configurable. I'm not saying that it's done yet. There's still users reporting, "Oh, that's weird," because it's pretty new, undo/redo on structured documents.

- Just to show how difficult undo/redo is, you can do some great experiments with other out-of-the-box solutions like Google Docs. You can have lots fun. Also with something like conflicting things, like Google Docs has some offline support, and I've taken the liberty to just test some scenarios. Like, you will have a paragraph, a long line, and do an enter break in the middle, and how does the system react? And it is really cool to just try out this, like, perfect product, built on knowledge of giants with Google Wave and all the technological and theoretical things, and then you try some undo/redo scenarios, and you get some really weird results even in the top notch product on the market right now. It's really interesting.

- Yeah, I think the notion of, like, what happens when you get a merge conflict or a sync conflict in Google Docs. It's always a fun experiment. It's rare that I get those messages these days, but it's always an interesting experiment. I do want to go back to something you just mentioned though, Kevin. And I do want to dwell a little bit on some of those really nitty gritty features of Yjs and really do the deep dive. But I'm curious, you just mentioned that the undo/redo behavior is actually configurable in Yjs. Can you talk a little bit about how that is, or what sorts of configuration can be expected?

- Oh yeah. The thing is, in Yjs you have... Yjs basically, well, my idea of Yjs is that you can see it as a shared type system. So you can create shared data types, something like an array, or a map with a key-value store. These data types, they work concurrently. So one user can manipulate them and the other one receives the updates, and then they sync. This is how I see Yjs. And then there are bindings to editors like ProseMirror, and CodeMirror, and many others. And often you have this big Yjs structure, and it may be a structure of several documents. So, when many users, or when you are working on several documents at the same time, and then you hit undo, what do you expect to happen? Does it also undo something from other documents? I don't think so. In my mind, I work on one document, I only undo the changes of that document. So this is what I mean of configure it, being able to configure it. You can specify what can be undone by your undo manager. There can be several undo managers, several document histories, for your local user. And you can specify what can happen and what cannot happen. And there are two mentions in Yjs. It's mainly you have a scope to a specific data type. For example, this undo manager can only undo stuff on this document. And then you have the scope also of... So, for example, what happens, when I created a paragraph, you created this big chunk of text in that paragraph but I didn't know about that, and then I hit undo? Can I undo the creation of that paragraph or not? Most users, as I found out, would say no, you should not delete content if you didn't create it. So in this case, you are not able to undo the changes of other users because sometimes these changes are intertwined. And the challenge here is really to figure out what is it what users expect when you hit undo.

- That's very interesting. I think that that really digs at, I think, the central issue of undo/redo, which is that people have different expectations of it. And having that as a configurable element I think is a very, very compelling idea. Well, okay, so let's dig out of undo history. Oh, go ahead. Sorry, Fabian.

- And there's also one of the key points why we think that CMS's itself should have the collaborative technology, and Drupal should have it out-of-the-box as a contributed module, Azure CMS should have it out-of-the-box. It should be really as simple as install a JavaScript library, like a plus module, and maybe a little server component if you don't want peer-to-peer. But then the centralized point here is that every client, every user, has their own expectations, they have their own challenges, they have their own things. And also, also all the editor space, even going to add a little bit of what we wanna go in more detail maybe in future episodes, they are no longer shipping out-of-the-box editors. They are shipping editor toolkits so every CMS can build their own editor to their needs. Every client, every project, can build their own editor. And I think that's a very, very key point to our current ecosystem of CMS's. And that someone else might need completely different undo managers than I need or this project needs. And that's also the beauty of open source. You can just take it, refine it, extend it, and do it. That's so cool with Yjs being open source that you just have this ability to do all these cool things.

- I'm not sure if we have the liberty to do this, but can we dig into a few of these potential scenarios where having this kind of differentiated configuration would be useful? Fabian, I think you mentioned a very interesting idea, which is that different users want to have different approaches to how they edit. How have you both seen this shake out in terms of the things you've tried, experiments you've tried? Or Mike, has this come up in terms of use cases? What sorts of things have you all seen in the real world that relate to this idea of having completely independent experiences?

- So basically, top 50 Fortune client it is customizability is key. It's very important that they have, for example in this case, their own editor which is exactly working to their needs. And the collaborative functionality is everything like you're expecting from Google Docs like just comments and later, even track changes, still working on that, pretty challenging. That's all very key in that. And it's important that, for example, if you use a technology like React upon which you can build your editor... Think of Confluence which is built upon Atlaskit. And Atlaskit is, by the way, also open source and one of the components we're using here as base. Then you can extend it, and you can build your own and you can customize it. That was really a key, central point to being able to not be stuck with some proprietary solution, or be stuck with some out-of-the-box solution where you cannot define anything, where you just get sent a black box and then you have to work with it. Because then if you need to change to the undo thing, well, yeah, you can pay Windows for it, obviously, but it's really out of your control. And here you can work directly with , but even later you can maintain it with a different team or whatever because it's accessible code. Another advantage of open source, basically. But yeah, the extendability, and being able to, for example, also define the server component. In this case, a server component is needed even though peer-to-peer would work and works well within the same browser or within even cross-browser with some tricks. Because you want, for example, authentication. If you store confidential data in a shared editing thing, you don't just want to be able to just access the server and see it, but really you want to... Which is usually a Node server, by the way. But really, you want to have authentication tokens like you're used from different APIs where you can secure your server. Then you can say, yeah, this operation is allowed or this operation is even not allowed. Even being able, as Yjs messages, even if it's not kind of like messages it's still some transformation sent over the wire. And then you could deep inspect it, and can say, hey, this is not allowed. This document was not touched for 300 days or whatever, and now someone is deleting all the content? Maybe deny that. Or keep a snapshot before that. And that's also so... So here, central server is nice and needed because you want to have that control but it gives us this flexibility in that, and you don't get that flexibility if you work just with a black box.

- Control is something that's a key theme of this project, this organization. Why we don't want to reinvent the wheel and rebuild certain awesome third-party technologies, this client of ours has repeatedly been let down by vendors that have made commitments to add certain features or do certain things over time. And one of the reasons that they're rebuilding this system and investing in these kinds of technologies is that the business cost of not being in control of sort of core technologies is huge. And so, relying on open source projects, being able to own and manage this code internally for their needs is critically important. This is gonna be a system that's gonna be around for 5+ years. The current system has been in place for a little over seven. And so, a key recurring theme is we need to be able to have long-term maintenance and control over key aspects of our technology.

- Yeah, and I think... I'm sorry, go ahead, Kevin.

- All right. I just wanted to add, one of the things that a lot of people appreciate when I talk to people who use Yjs is that you can easily use Yjs on top of your existing communication protocol. So there are many existing projects that want to integrate collaboration into their system, and they already have some kind of WebSocket connection or long polling setup, and they want to use that system. And for good reason. They don't want to change their whole infrastructure, add another communication layer like some proprietary WebSocket communication protocol, and build on that. They want to integrate Yjs into their existing protocol. That's something that you can easily do with Yjs. And this is also something that Fabian talked about just now. I completely agree with that. I didn't think about it when I implemented like that, but it just came up that it is really well appreciated. And because of that, I put everything into separate modules. Yjs is split up into like 20 modules right now. Once you build a system, you basically say, okay, how do I communicate? I use this existing module, or I can rewrite my own module on top of the Yjs code, or just on top of the documentation that you have. And if you want to support your own editor it's so easy because the editor support is not baked into Yjs. It's not just built for one thing, it's built for everything. And that makes it incredibly difficult to do, but I think it's a really nice concept.

- Yeah. And speaking to that, the editor support, if you have textareas somewhere and you want to make collaborative, you can do that right now. There's a translator for a rich text area, like the contenteditable, or just any textarea. You can just take it, put Yjs on it. And there's also a nice demo site. There will be some post information. We'll link to that with some links in that. There's a really cool, there's even two demo sites. One is really showing like ProseMirror, Quill, and even Atlaskit, the whole Confluence experience in that. But there's also shared editing for a nice thing. There's even a 3D modeling demo baked in together with 3D modeling. It's insane, it's really insane. Also drawing. It's, like, really cool. The possibilities are so endless. It's so nice that it's so framework agnostic. Really not just editors and really not just one editor. But really, if you have a textarea, you can use Yjs, it will be collaborative, it works. And even for older things, if someone wanted to invest in that CKEditor 4, it would be possible, in theory, to make that collaborative. People wouldn't even need to operate newer technologies. It would need work, investment, et cetera, of course, but it's possible.

- Yeah, I think this notion of flexibility and customizability, but also especially this notion of pluggability, where you can insert Yjs into any situation and it works. I think also the flexibility. Fabian, you mentioned earlier that in certain cases you do want to have a server component that acts as the safety kind of mechanism. But you might not want one, and Yjs leaves that option open as well. And I think just the notion of being able to insert it agnostically anywhere you want is a very, very key characteristic. I think one of the key characteristics that we identified a little bit earlier but we haven't really dwelled on, though, is one that's very interesting to me. Which is that you can use Yjs for things like text, you can use Yjs for textareas and contenteditable, but what about drawings and 3D modeling? I know, Kevin, you've sort of framed Yjs as being for more than just text. Can you talk a little bit about the drawing capabilities? What does collaborative drawing mean, and how does Yjs fit into that?

- Yeah, definitely. This all comes from the idea that Yjs is not just for specific applications, but it's just a framework for shared data types. You can build any application on data types because, well, that's what we usually do as developers. We have these key data structures, especially data. In JavaScript we have only two or three main data structures, which is arrays and maps. You can build your own data structures on top of that and they're abstractions, but basically you just have arrays and maps. And maybe set is also something that, well, I really like the idea of having a set too. And you have the same capabilities in Yjs but the difference is, these data structures or data types, they are collaborative, they are shared, so when you do an edit to this data type the change is transmitted to all the other peers automatically, and they sync, and then you get updates of what changed. And you can design your application like that. Just use the shared data types and build your application. And drawing was something that is really, really easy to implement on top of data types, or shared data types. A line is basically just a list, an array of vectors. So, this is what is happening in the drawing demo. When you start drawing, you have a map, you insert a line to that map, and then you start drawing. It's really easy. And then you can maybe configure it by adding some options to the map, or something to color, who created the line, all this information. You can just add it to the shared data structure. And this is also how the 3D modeling is created. It's really basic, but it's really cool because it shows that everything you do, like rotating the 3D object, the rotation is a part of the shared data type. You can add annotations and these annotations are just X, Y, Z values, and you add this to the shared data structure. And just like this you can build amazing applications, just on top of this idea.

- Yeah, I was thinking about the classic React example, the to-do list. And the to-do list basically, the results are often shown for Redux data stores like transformations, et cetera. And now put this final array that you put your data in, or this kind of map, probably an array in this case. Put that in the Yjs data structure and it's automatic. It's kind of almost automatically collaborative. That's just great.

- Is one of these in particular harder than the others? Is collaborative text editing actually harder than drawing?

- Collaborative text editing, like, I want to distinguish between text editing and rich text editing. Because text editing is pretty easy. It's harder than having a 3D model, in my opinion. From my perspective as a developer of Yjs, it is harder because text editing, you need to optimize that a lot to make it performant and don't block the thread. You want to be able to insert, like, a megabyte of text into your data structure. And that's not always easy. First of all, because of the time it takes to send to the other peers, but also the time it takes to parse that operation, that change. So, text data structure is heavily optimized in Yjs, and that's why it's performant. For you, it's as easy as the array, as using the array data structure. Rich text on the other hand, it's a bit weird because you can either have structured documents. This is what I think of in ProseMirror as like, you have a paragraph, inside that you have a table, and so on. And then you have also formatting attributes to your text. So, when you write the text, "Hello world," and you want to make, "world," bold, you don't remove the bold text and add some text, tags like in HTML, around this word, "world." You want to assign attributes to this text. And this is also possible in Yjs. It's like this rich text notion from the classical Windows. I think they developed the term rich text. You assign attributes to text. This is also one of the problems here. And like, as soon as you realize that this is going on, you can either have structured documents, or rich text documents, or both combined. In ProseMirror, they're kind of both combined. You have marks, so you can assign properties to it, to text. And as soon as you realize what's going on here, it's fairly easy to build applications on top of that. But building editor support, that's a whole different level for if you build a structured editor support, for example for ProseMirror. It was really, really hard to get right.

- Yeah. And speaking of performance for inserting, we did a test. And if you take a really, really long document, you paste it 10 times into Google Docs, it takes longer and longer. It takes around seven seconds. And with Yjs and CRDTs, it's instant. It's really, it's really just instant. I was astonished. I also tested some other things, which I don't name to not shame. But there I was even able to freeze, to completely freeze the collaborative editor, and it was not reacting anymore after a while, and undo . It's just not a pleasant experience. It was so nice to see this just working so well with Yjs. With one of the key points where I was able to present, with stakeholders that were evaluating the technology on the client side with me to say, hey, this is really working great in practice. Just try pasting a big document. Now try the same in Google Docs. It's like a difference of night and day.

- Absolutely. Yeah, and I think one of the things that's interesting is if you look at Yjs in relation to some of the other tools that are out there and available, I know that, for example one of the, some of the very common tools people use out there are things like CKEditor 5, ProseMirror collab, as well as Draft.js. But one of the things I know is that there are certain features that are lacking. I know that you don't want to name and shame, Fabian. But I'm curious, what makes Yjs superior? One of the things I know, for example, that Yjs does better is one thing you mentioned earlier, Kevin, around content annotations, being able to directly apply those annotations. What are some other examples of how Yjs is better than some of these other solutions and stacks like ShareDB, Automerge, the CKSource service? What are some of the things that you both found were either better in Yjs, or lacking in others, or things that you noticed?

- First of all, just to get it out of the picture, the CKSource solution unfortunately is proprietary. It's a server black box. You have to pay for it. Which is fine, that's fine. But as I've already outlined a lot on this call, open source is just better. Because, well, I mean, Drupal is open source, many technologies are open source, and those open source technologies, they thrive, because companies invest in them, they mature, and everyone can use them, and that's so important. That puts out the proprietary solutions. They might be useful for the one where each project of that, but they're not useful, in my mind, to the larger community in that. And for Automerge, ShareDB, I'll let Kevin speak. expert.

- Yeah. So, about these two projects, ShareDB, it was always my goal to be as good as ShareDB. It's a really cool project. And I was inspired a lot of the features they have, because they also have the notion that you can just manipulate your data and then everyone gets synced up. And I love that idea. At the time when I created Yjs, they didn't support, for example, objects as key values or so. And this was like, okay, it can't be so hard to implement that. I also want to make it peer-to-peer. So, this is why I created Yjs. And it only took me six years to do that, but that's fine. So, I think Yjs against ShareDB, they're both great projects, but ShareDB is based on operational transformation which is centralized, and I always loved the idea of having a decentralized system. And Automerge, also a really cool project. I really like the maintainer of that project. He is really active in the research community, and he's a lot more credible than I am in the papers he created. He is a really good writer. And if you get the chance, and are interested in CRDTs, you should definitely check out one of his talks about CRDTs because he really explains it well how it works. But now against Automerge, right now, Automerge is not as performant as Yjs. That's just what I want to say there. Yjs is really focused on shared text editing. And Automerge also supports text editing, but it's not as fast. I created some benchmarks. Maybe we can link that too. You can also find it on the GitHub repository of Yjs. But yeah, these are the main reasons against that. It still needs to be seen which algorithm is better. But the thing is, in Yjs I had more time to create Yjs and I implemented a lot of optimizations especially for shared text editing.

- Yeah, and it's also a very important point in if you want to do a collaborative system for application with a JSON structure, try out Automerge, try out Yjs. We are open source. There's always a good solution for your project that you need. But if you want to do text editing then Yjs gives you this undo manager. It gives you these functionalities for rich text. It gives you this control. It gives you this basic server component if you need. With Automerge, you build everything your own. You can do that, it's fine. People have done that with certain editors. But it's really Yjs gives you a headstart and a framework to work with shared editors here especially.

- Yeah, and this is really interesting. I think one of the things... By the way, just to help our viewers and our listeners, we keep on throwing this CRDT acronym around. It's actually one of the big reasons why I think Yjs is so compelling. CRDT, by the way, stands for commutative replicated data type. You can look at it on Wikipedia in the operational transformation article, very, very useful. But I think just to dig into CRDT a little bit briefly, I think one of the really key aspects of Yjs that we've obviously been touching on here and there is the fact that because of this focus on data types it's very powerful for rich text editing, more so than some of the, especially for those who need more than just the plain text editing feature. But I think this is actually just one example of some of the really interesting features in Yjs. One of the things that I found really interesting is because of that kind of agnosticism and because of that focus on that kind of lower level, we actually find that other things are possible with Yjs, like using multiple document formats. So you can use rich text, you can use markdown, you can use HTML, anything that's kind of a parsable AST. What I'm curious though is that there is a topic very near and dear to my heart which I know that Kevin, you focused on, which is actually the notion of accessibility. So I'm curious, how exactly does accessibility work in the context of realtime collaboration for Yjs? Especially given the fact that rich text editing, realtime collaboration, both very, very difficult things to do in an accessible way. Are you using things like ARIA labels? What are your thoughts on accessibility with realtime collaboration?

- Accessibility is firstly an issue for the editors. Yjs is not really concerned about accessibility because it's not really part of the framework. But if the editor supports accessibility then it's baked into Yjs. By the way... No, actually that's all I want to say about that. Most editors nowadays are really accessible so there's not a lot of concern there. I'm also not an expert in accessibility, but I'm also really concerned about what happens when you hit undo/redo, for example. Which is, by the way, not working in Google Docs or in most editors. Try hitting Edit and then Undo in Google Docs. It doesn't work. And I'll figure out why.

- Very interesting. Wow.

- But maybe this is part of the discussion when we talk about editor support, or which editor we choose, Tag1, and for the company that we contract for.

- We'll do a followup talk on that. I think that the whole editor component, we did a ton of research. So maybe we'll do a Tag talk next on the whole editor component and how we ended up with ProseMirror and the integration of all that.

- Yeah, because that's an area I'd love to dig into in terms of ProseMirror's capability. CKEditor also has really great accessibility features. But how that relationship comes together, how Yjs really integrates with some of these editors and how those features get exposed, that's a really interesting topic that I know we're gonna have to save for next time. In these last few minutes here... We've covered a lot of ground here. We've covered realtime collaboration. We've covered some of the problems with concurrency in these editing tools. We've also talked about CRDT from the very, very high level standpoint. I'm curious now, I think one of the things people are interested in is, well, what is the future of Yjs, Kevin? What exactly do the next couple years hold for Yjs? What are some of your goals in the future, moving forward?

- My goals. My goal is to create editor support for all major editors out there right now. There's currently editor support for code editors like Ace, and CodeMirror, and there's rich text editors, for example ProseMirror and Quill support. There are many interesting additions to Yjs, and I want to increase that number. I want a lot of people using that technology. Because if you use Yjs as the entry point for your data model you can have different text editors. For example, you can have CodeMirror and Ace, like one user uses that and the other uses a different editor, and that's a really interesting idea for me. The other thing is, what I'm really interested in is real decentralized systems. And there's mainly the dotProject and the IPFS project, and building on top of that system, like each centralized system, wow, it's so amazing. You can build an application without a central server only having, well, peers, just working stations meshed together and they somehow create an environment where you can store data, and that's very interesting to me.

- I think that is a very, very compelling idea that I'd love to talk more about with you, Kevin. Yeah, the idea of having a completely serverless implementation I think is very interesting. Well, I did want to say that we are out of time. However, I think one of the things I wanted to call out is something, Fabian, you said at the very beginning of this whole broadcast, which is, this should be something that's part of every CMS and online application. We should be building interfaces that are editable. We should be really enabling these content editor workflows. And I think clearly Yjs has the right idea, has the right journey in mind. And I can see, given that Tag1 is focused not just on Drupal but also all of these other technologies, and building things like central Node.js backends for these editors, all of that sort of work really highlights, I think, the notion that realtime collaboration is a very important concern for everybody. Not just developers, but also our content editors and our marketers who are working with us on a daily basis. Any last words about realtime collaboration, Yjs, something you want to leave our viewers and listeners with?

- I think it's just an awesome technology. And really, even repeating myself, it is such a difference if you start using the system, even if we are just in demo mode as we are pre-alpha, to just collaborate with someone within the Drupal scene. It feels so different to being in Google Docs or being isolated because it's where you create your content normally, it's where you work together, it's the user tools. You can even attach a file directly. You don't have to upload it to Google Docs and later upload it to Drupal. You really have the experience in that you can use a media browser, you can use a media browser, your whole media library that's on the CMS, it's not in Google Docs. You select your file, it's interactive, and it's collaborative. Your peers will see it as well. And I think that's just fantastic, and that's why I'm saying realtime systems for editors, but also realtime updates. Like, I made an update, you see that I made an update directly on your screen. That's kind of, in my mind, the future of CMS's and, in a way, also the web.

- And this is intended to be a CMS-independent solution. We look forward to adding this to Django and Wagtail, to WordPress. Every CMS should have this. I'd also say that we just scratched the surface of all of these technologies. I think this is a really interesting conversation so we'll definitely setup some future talks to dig more into the details, whether it's the editor or the underlying tech, to get into the nitty gritty.

- Absolutely. I think that today we've done a very good overview of the what, the how, and the Yjs. And I want to go ahead and wrap things up here. If you have any thoughts on how this show, what you're interested, in certain topics, please feel free to reach out to us. I want to thank our guests today. Fabian Franz, senior technical architect and performance lead at Tag1. Also Kevin Jahns joining us all the way from Berlin, right. The creator of Yjs as well as a key expert and contributor to realtime collaboration. And of course my partner in crime Michael Meyers, managing director at Tag1. Thank you all so much, and we'll see you next time for Tag1 Team Talk. Thank you.

Mar 26 2018
Mar 26
Michael Meyers

I’m excited to announce that Michael Meyers has joined the Tag1 team as Managing Director. Michael was one of our very first clients 10 years ago, we’ve worked together on many projects over the years, and we look forward to working even more closely with him now that he’s a part of the Tag1 team. Michael has extensive experience building market leading high-growth technology companies and is particularly well known in the Drupal Community for his role in driving innovation of the Drupal platform. Michael brings over 20 years of experience managing global technology teams building high traffic, high performance mobile and web applications. Tag1 recently celebrated our 10th anniversary, in that time we’ve established ourselves as the leading provider of highly available, scalable, secure, high performance systems and as the organization other agencies and the users of Drupal turn to for help with their most challenging problems. We will be working with Michael to expand on our success to date and to help lead Tag1 into the future.

Roots in Success

Michael joins Tag1 from Acquia, where he spent the last 5 years on the leadership team as VP of Developer Relations, Developer Marketing, and helped launch the Developer Products group. He also ran Acquia’s Large Scale Drupal (LSD) program, a strategic alliance that engaged Acquia’s largest clients and partners in open source collaborations to enhance Drupal. Tag1 participated in many aspects of Large Scale Drupal including speaking at events, and mentoring teams at LSD hackathons.

Prior to joining Acquia, Michael co-founded & was CTO of NowPublic.com, the first venture-backed Drupal based startup. With the help of Tag1, he grew NowPublic from an idea into a top 500 website. The open source performance optimized Pressflow distribution of Drupal that Tag1 co-maintains was based in large part on the work we did together scaling NowPublic. Later, as CTO of The Clarity Digital Group which acquired NowPublic, Michael and the Tag1 team worked closely together to rebuild Examiner.com — creating the first Drupal-based top 100 Internet site. The Examiner team (including a number of Tag1 employees) was the leading contributor to Drupal 7, and helped meaningfully shape the technology that today powers millions of websites.

Continuing to Build Upward

Michael now has a key role on our management team, helping us usher in the next chapter of growth for our company. I will continue in my role as CEO, ensuring Tag1 stays true to its roots and the strategy that has driven our success to date, with Michael leading our new business development & marketing efforts, focusing on growing our team and service offerings, and building on the success of our first SaaS product, Tag1 Quo.

The depth and breadth of Tag1’s Drupal expertise and contributions to Drupal is largely unparalleled, certainly among the best in the world. In the coming year, we plan to double down on our commitment to Drupal and the Drupal Community. Everyone at Tag1 is really excited about Drupal 8, and decoupled Drupal 8 in particular, which has been a huge success for our clients like Symantec where Drupal 8 has been leveraged to manage content between many disparate systems. For those of you coming to DrupalCon Nashville, we hope you will join Michael, BKJ Digital, and Symantec at our talk “Symantec - From Early Drupal Adoption to the Latest Drupal Innovations”. Michael will highlight how we enable a large Fortune 500 enterprise to drive success with agile practices at scale, foster collaboration across internal groups and external partners with true CI/CD, simplify infrastructure management by treating configuration as code, and enforce security and architecture through gatekeeping and code reviews.

Evolution of Tag1 Products and Services

Over the years Tag1 has greatly expanded its capabilities and offerings — in the face of the most difficult challenges, we’ve never let a client down, and as a result we are often asked by the organizations we work with to take on more and more full service development (from requirements gathering and UX/Design, to development & theming, QA, through to ongoing support and maintenance). Leveraging our broad and deep Drupal expertise, and development process best practices, we provide mentorship and gatekeeping for client teams, and even other services companies. Last year we introduced a SaaS subscription based product, Tag1 Quo, that enables organizations to monitor their Drupal sites (across all versions of the platform) for security issues and updates ensuring they patch systems as quickly as possible and avoid problems before they happen.

We’ve expanded beyond Drupal as well, and are increasingly working with Node.js, Django, Angular, and WordPress for projects where they are the best fit. We have also increased our involvement in stack independent projects like automating development, performance testing, and QA processes, managing infrastructure as code through advanced devops systems, and taking on a broader array of projects including big data, business intelligence & analytics systems.

The Next Episode

Together with Michael, we will lead Tag1 into its second decade, by increasing our commitment to Drupal while also continuing to expand beyond Drupal, working on a greater variety of projects, growing our team, and bringing our reputation of driving success to more and more clients. We’d love to work with you in 2018. Whether you need help with your existing Drupal systems, are exploring upgrade paths, or require help with projects outside of Drupal, please reach out to Michael to discuss how we can help.

On behalf of Tag1, please join me in welcoming Michael to the team!

Nov 29 2017
Nov 29

TL;DR

Introducing a new, easy to use, Drupal 8 module for background images: https://www.drupal.org/project/background_image

If your site is designed around utilizing background images, then this module is for you! Whether you need a surgical implementation that only administrators/developers can implement or provide the ability to allow users to attach their own background images to entities, this module has you covered.

If you're just looking on how to setup this module, you can skip ahead to the Installation section.

The Problem

Over the past month, I have attempted to solve a lot of problems in regards to displaying a user provided image (via Drupal's UI) on a rendered page in the form of background-image: url(...);) in Drupal 8.

In Drupal 7, there were several modules that attempted to do just that. However, much of time, it was simply easier to implement an entirely custom solution. While this did take some effort to do it just right (using custom image fields and then preprocessing via a sub-theme), it was relatively easy enough to accomplish if you knew what needed to be done.

Now, you may be thinking "Great another module that does the same as another module, but only slightly different." You would be kind of right, but it's actually quite a bit different than any previous implementation.

In Drupal 7, many of the "solutions" were primarily focused on using custom field types or field formatters. This required the same type of field or formatter settings to be applied to every single entity/bundle you needed this functionality on. It was still time consuming to implement in a consistent fashion across all the moving parts (which is why a custom implementation was often easier).

A solution in Drupal 8 can now be implemented in an entirely different way. One that eases the setup process and also brings an easier to use interface for your users.

Tag1's backend and frontend Engineers are among the top contributors worldwide to Drupal 8’s code-base.

Responsibility

I think one of the biggest challenges with this entire topic has always been "Whose responsibility is it to handle the UI and then output of said background image?" In the past, there has never really been a clear delineation of whether this responsibility specifically fell to a module or to a theme. Sure, in Drupal 7, some of the modules used fields, but often times you still had to customize the output (usually in a theme) given your specific needs.

In Drupal 8, themes now have very limited "power" in what hooks and plugins they are actually allowed to participate in. This inherently and dramatically forces the responsibility toward needing a dedicated module to handle all of the intricacies required for properly displaying a background image provided by a user.

Entities

Drupal 8 brought with it many great and wonderful things, the least of which is being able to create new entities (out-of-the-box) in a very easy to implement way. No need to depend on a contrib module or figure out complex database schemas. Core just handles the bulk of what you need if you just configure it right.

This module implements a truly dedicated entity. No more blanket custom "Background Image" node bundle entities. No new fields. No field formatters. No more wasting precious time.

This major difference is the primary reason behind creating an entirely new module and not attempting to "port" an existing one. Never mind the added inherit benefits of being able to do things like entity specific permissions.

Cacheability

As mentioned above, often times it was simply easier to implement custom solutions in Drupal 7. This usually meant going in the theme and preprocessing what you needed. However, gone are the days that simple "theme hacks" in the preprocess layer can accomplish what they used to.

Many of those "easier solutions" in Drupal 7 are simply and naturally hindered by the much needed and massive improvement to the cacheability layer in Drupal 8. Thus, themes can no longer inject a background image (on a page level) by simply retrieving the "image field" value for the current page's entity (node) object. Render cache will just "stick" to whatever image value was requested first.

This can be a big "gotcha" for Drupal 8 beginners. They aren't aware of the powerful caching layer that's used by default in Drupal. The exact reasons behind why this happens is extremely complex, but suffice it to say: there is no context for the specific entity's image at a page rendering level.

Sure, one could simply add the specific entity's cache context to the page, but then you would be rendering/caching every single entity (variation). That is not a good solution and actually defeats the entire purpose of (render) caching. This is where implementing a dedicated entity in Drupal 8 is extremely powerful: entities are also a separate cacheable object.

Optimization

Another beautiful feature of Drupal 8 are the two newly added modules: Breakpoint and Responsive Image. I wanted to take advantage of both of these (not required, but highly recommended) to optimize a background image even further; especially when dealing with 2x multipliers for retina displays.

However, doing so in regards to background-image: url(...); would mean that there would be a need to generate custom CSS for each and every image to account for the responsive media queries. CSS generation in Drupal has typically been string concatenations, but that's a lot of strings to ensure everything is "just right".

Cut to... Twig! Drupal 8 also already comes with Twig and it felt like a natural approach when needing to generate the multitude of rules without also adding a preprocessor requirement (which has always been a highly debated topic).

Note: if you enable the Advanced CSS/JS Aggregation module, this generated CSS will also be minified when the site's CSS is being preprocessed.

By using a .css.twig template, generating each media query rule was actually quite a delight to implement:

{# Media Queries #}
{% for media in media_queries %}
@media {{ media.query }} {
  .{{ background_image_class }}::after {
    background-image: url("{{ media.url }}");
  }
}
{% endfor %}

Obviously, generating this CSS each time would be a huge resource drain, especially when an image or its settings change. First, I attempted to use core's \Drupal\Core\Asset\AssetDumper. Storing it in this fashion seemed to be OK, but I soon realized that to get the actual CSS filename meant I had to generate the file right away. This caused issues with having to wait for the initial page response (think BigPipe).

Instead, I took a page from core's own image styles playbook and decided to generate this CSS only upon an actual CSS resource request from the browser. If it didn't actually exist at the specified location it would then, and only then, generate it on the fly.

Currently, this generated CSS can be regenerated upon a sitewide cache rebuild. There are plans to optimize this even further by providing a way in the administrative UI to clear only these generated files.

Preload Image/Color

While media queries certainly help reduce the unnecessary loading of a massive image on a small device, it doesn't help with network latency and what seems to be FOUC. This problem can be even more drastic when the page is using a dark image/style and you have a white background, waiting for the image to load.

Sites like Medium use various techniques to optimize the display of images. One of their solutions is easily doable in Drupal considering it comes standard with image styles: generate an extremely tiny version of the image, preload it and then use it as the initial background image. Since image styles' foundation utilizes \Drupal\Core\ImageToolkit\ImageToolkitInterface, we can also extract the average "color" of the image by resizing it all the way down to a 1x1 image.

This means that what an end user visually sees, depending on network latency, is done so in a very progressive manner:

  • Average background color that resembles the image
  • Preload image (100 x 67) low quality/blurred version of the image
  • Fallback (mobile) image, normal quality
  • Hi-resolution image

Inheritability

Depending on a site's design, there is often a need to "inherit" an image or its settings based on various conditions. For some sites (like this one), a "global" background image is needed. For others, individual entities need there own images/settings (see the various blog posts here). There are several "types" of background images supported by this module, with even more planned:

  • Global
  • Entity Bundle
  • Entity
  • View
  • Routes or Paths (or both, planned)

The background image types above are listed in order of how they are inherited. Each background image provides the ability to inherit or override an image or individual settings entirely. This allows for true customization without the "all or nothing" approach in previous Drupal 7 implementations.

Extensibility

Sometimes your specific "needs" aren't met by what you get "out-of-the-box" by a module. In the case of a background image, you may need to programmatically add additional information to the overlay text.

An example of this can be found at the top of this page where the "Author's information" is added to the background image overlay text (so it also displays when the image is using the full viewport setting). This can easily be accomplished using one of the many alter hooks this module provides:

<?php
/**
 * Implements hook_background_image_text_build_alter().
 *
 * {@inheritdoc}
 */
function MODULE_background_image_text_build_alter(array &$element, array &$context) {
  /** @var \Drupal\node\NodeInterface $entity */
  $entity = $context['entity'];

  /** @var \Drupal\Core\Datetime\DateFormatterInterface $date_formatter */
  $date_formatter = \Drupal::service('date.formatter');

  /** @var \Drupal\Core\Render\RendererInterface $renderer */
  $renderer = \Drupal::service('renderer');
  
  // Immediately return if this is not a blog post.
  $entity_type = $entity ? $entity->getEntityTypeId() : FALSE;
  $bundle = $entity ? $entity->bundle() : FALSE;
  if (!$entity || $entity_type !== 'node' || $bundle !== 'blog') {
    return;
  }

  // Retrieve the blog author and the time the blog post was created.
  $user = $entity->getOwner();
  $created = $entity->getCreatedTime();

  // Retrieve the user profile picture by using the "compact" view mode.
  $picture = \Drupal::entityTypeManager()->getViewBuilder('user')->view($user, 'compact');

  // Bubble the picture's cache metadata to the element.
  $meta = BubbleableMetadata::createFromRenderArray($element)->merge(BubbleableMetadata::createFromRenderArray($picture));
  $meta->applyTo($element);

  // Append the custom user "profile".
  // Note: it isn't recommended to use custom markup like this, however this is being processed
  // via a text formatter and not as a MarkupInterface object, so raw HTML is required.
  $element['#text'] .= '<div class="user-profile">';
  $element['#text'] .= $renderer->render($picture);
  $element['#text'] .= '<div>';
  $element['#text'] .= '<h3 class="name">' . $user->get('field_full_name')->value . '</h3>';
  $element['#text'] .= '<h4 class="title">' . $user->get('field_job_title')->value . '</h4>';
  $element['#text'] .= '<time class="date" datetime="' . $date_formatter->format($created, 'custom', 'c') . '">' . $date_formatter->format($created, 'custom', 'F d, Y') . '</time>';
  $element['#text'] .= '</div></div>';
}

Installation

You can download and install this module from https://www.drupal.org/project/background_image. Once installed, there are a few steps you will have to do before it will actually work.

Recommended Modules:

The following are a list of additional modules that you may find useful when working with this module:

  • Advanced CSS/JS Aggregation - Useful for minimizing the generated CSS for each background image.
  • ImageMagick - Useful in cases where PHP memory limits bug out due to large image processing.
  • Image Style Quality - Useful for further optimizing the preload background image style quality (recommend configuring somewhere around 30%)
  • Image Optimize - Useful for further optimizing all background images (progressive loading)
  • Inline Entity Form - Currently a soft dependency if you wish to embed the background image form on other entities.
  • Token - Provides a way to view which tokens are available to use when adding overlay text for a background image.

Note: the Background Image module merely provides the "muscle" behind handling background images for a site. While this module does generate CSS, it is merely for the display/positioning logic of the background images themselves. It does not actually stylistically theme the background image or other elements around it or within (like the overlay text).

Note: all subsequent administrative screenshots will be using the Seven admin theme (included with core) and all front-end facing screenshots will be a custom sub-theme of the Drupal Bootstrap base theme.

Step 1: Initial Configuration

  1. Navigate to /admin/config/media/background_image/settings on your site and choose which entity bundles you wish to enable support for background images on:
    Background Image: Administrative Settings
    • Enable - Indicates whether this specific entity bundle supports background images at all.
    • Embed - Embeds the background image add/edit form inside the specific entity bundle's add/edit form (currently requires the Inline Entity Form module, with plans to remove this soft dependency in the future). The dropdown next to it indicates which "Group" the form is added to. The "Default" group will place it at the bottom of the entire form. The "Advanced" group will place it in the "advanced" group that is present on some entities and some themes may move this to a sidebar (like in Seven).
    • Require - Explicitly requires an image of some kind.
  2. Click Save configuration.

Step 2: New Region

The Background Image module automatically and dynamically creates a dedicated background_image region in all active themes. This means you don't have to add it to your theme's .info.yml file.

Where you actually render this region is entirely up to you, however it should be noted that this region should be a top level region, positioned outside of all other regions or containers. It should also be positioned somewhere between your header (navigation) and your main content. This is so that when the "full viewport" setting is used, it can push the content (but not the navigation) down out of view.

Once you determine where you want to place this region, alter your theme's page.html.twig file to include the following code so it can actually render this region:

{# Background Image #}
{% block background_image %}
  {% if page.background_image %}
    {{ page.background_image }}
  {% endif %}
{% endblock %}

The following code is from a custom Drupal Bootstrap sub-theme and may give insight into where this region should be placed:

{#
/**
 * @file
 * page.html.twig
 * ...
 */
#}
{% set container = theme.settings.fluid_container ? 'container-fluid' : 'container' %}
{# Navbar #}
{% if page.navigation or page.navigation_collapsible %}
  {% block navbar %}
    {%
      set navbar_classes = [
        'navbar',
        theme.settings.navbar_inverse ? 'navbar-inverse' : 'navbar-default',
        theme.settings.navbar_position ? 'navbar-' ~ theme.settings.navbar_position|clean_class : container,
      ]
    %}
    <header{{ navbar_attributes.addClass(navbar_classes) }} id="navbar" role="banner">
      {% if not navbar_attributes.hasClass(container) %}
        <div class="{{ container }}">
      {% endif %}
      <div class="navbar-header">
        {{ page.navigation }}
        {# .btn-navbar is used as the toggle for collapsed navbar content #}
        {% if page.navigation_collapsible %}
          <button type="button" class="navbar-toggle" data-toggle="collapse" data-target="#navbar-collapse">
            <span class="sr-only">{{ 'Toggle navigation'|t }}</span>
            <span class="icon-bar"></span>
            <span class="icon-bar"></span>
            <span class="icon-bar"></span>
          </button>
        {% endif %}
      </div>

      {# Navigation (collapsible) #}
      {% if page.navigation_collapsible %}
        <div id="navbar-collapse" class="navbar-collapse collapse">
          {{ page.navigation_collapsible }}
        </div>
      {% endif %}
      {% if not navbar_attributes.hasClass(container) %}
        </div>
      {% endif %}
    </header>
  {% endblock %}
{% endif %}

{# Background Image #}
{% block background_image -%}
  {%- if page.background_image -%}
    {{- page.background_image -}}
  {%- endif -%}
{%- endblock %}

{# Main #}
{% block main %}
  <div role="main" class="main-container {{ container }} js-quickedit-main-content">
    <div class="row">

      {# Header #}
      {% if page.header %}
        {% block header %}
          <div class="col-sm-12" role="heading">
            {{ page.header }}
          </div>
        {% endblock %}
      {% endif %}

      {# Sidebar First #}
      {% if page.sidebar_first %}
        {% block sidebar_first %}
          <aside class="col-sm-3" role="complementary">
            {{ page.sidebar_first }}
          </aside>
        {% endblock %}
      {% endif %}

      {# Content #}
      {%
        set content_classes = [
          page.sidebar_first and page.sidebar_second ? 'col-sm-6',
          page.sidebar_first and page.sidebar_second is empty ? 'col-sm-9',
          page.sidebar_second and page.sidebar_first is empty ? 'col-sm-9',
          page.sidebar_first is empty and page.sidebar_second is empty ? 'col-sm-12'
        ]
      %}
      <section{{ content_attributes.addClass(content_classes) }}>

        {# Highlighted #}
        {% if page.highlighted %}
          {% block highlighted %}
            <div class="highlighted">{{ page.highlighted }}</div>
          {% endblock %}
        {% endif %}

        {# Help #}
        {% if page.help %}
          {% block help %}
            {{ page.help }}
          {% endblock %}
        {% endif %}

        {# Content #}
        {% block content %}
          <a id="main-content"></a>
          {{ page.content }}
        {% endblock %}
      </section>

      {# Sidebar Second #}
      {% if page.sidebar_second %}
        {% block sidebar_second %}
          <aside class="col-sm-3" role="complementary">
            {{ page.sidebar_second }}
          </aside>
        {% endblock %}
      {% endif %}
    </div>
  </div>
{% endblock %}

{% if page.footer %}
  {% block footer %}
    <footer class="footer {{ container }}" role="contentinfo">
      {{ page.footer }}
    </footer>
  {% endblock %}
{% endif %}

For customizability, the module does not assume that a background image will always be rendered in the provided background_image region or in which order these elements are rendered in.

Instead, it provides multiple background image based blocks that allow you to control what is displayed and where. The purpose of doing this is two-fold:

  1. It allows each element of the background image to be cached independently using the various cache contexts that this module provides. This is particularly useful if you "inherit", say, the background image, but only the overlay text has changed. It doesn't invalidate the entire background image and force a new CSS file with this image to be generated.
  2. Future proofing. Over time, new features will be added, like providing credit/attribution for an image (which is planned), and providing individual blocks for each of these rendering elements will allow features like this to be added in a backwards compatible way without "damaging" existing sites.
Once you have added the region to your page.html.twig template, you will need to clear the cache. You can use drush cr or navigate to /admin/config/development/performance and click Clear all caches:
Background Image: Clear cache

Step 3: Permissions

Navigate to /admin/people/permissions and configure the following permissions:
Background Image: Permissions

Step 4: Add Blocks

  1. Navigate to /admin/structure/block on your site and click the Place Block for the Background Image region
    Background Image: Place Block
  2. Add both the Background Image and Background Image - Text blocks:
    Background Image: Blocks

    Warning: do not set any of the "Visibility" settings unless you know what you are doing. The Background Image module automatically determines a block's #access based on whether there is a background image for the current request or not.


  3. Verify that the blocks were added to the background_image region (should look similar to the following):
    Background Image: Blocks Added

    Note: due to how core's config installation of blocks work, this module cannot provide any "default block placement"; you must place these blocks yourself for the module to work properly.

Step 5: Add Global Background Image

  1. Navigate to /admin/config/media/background_image on your site and click Add background image:
    Background Image: Add
  2. Enter a label (e.g. "Site wide") for the "Global" background image type and then click Continue:
    Background Image: Add Global
  3. Select an a background image (I'm going to use https://unsplash.com/photos/JgOeRuGD_Y4 for this example):
    Background Image: Add Image
  4. By default, the Background Image module determines if the image is "Dark" based on the preload background color which is calculated from the average color of a 1x1 resize of the image. You can choose to override these default settings here:
    Background Image: Dark & Preload Background Color settings
  5. Full Viewport setting:
    Background Image: Full Viewport setting
  6. Blur settings:
    Background Image: Blur settings

    The radius and speed blur settings are pretty self explanatory, but a couple of the blur "type" options need a little extra to help differentiate between the two:

    • Scrolling - The current scrolltop determines the radius of the blur (without going over the maximum blur radius). This means that when the scrolltop is 0, the blur is 0.
    • Scrolling (Full Viewport) - If the "Full Viewport" setting is enabled, then this will behave like the above. If it's not, then it will behave as if the "Persistent" option was selected.
  7. Overlay text:
    Background Image: Overlay text setting

    This is formatted text and allows various tokens to be used. You can configure this however you wish, but the following screenshot is how this site's global background image is configured to give an example of just how easy and power this is to use with tokens:

    <h2>[node:field_lead]</h2>
    <h1>[node:title][user:field_full_name][view:title]</h1>
    

    The first line shows a custom "lead" field that is on each node bundle on our site. This simply allows us to show larger verbiage that doesn't affect the node title. The second line is actually the page title for various entity types (so we can display outside the main content container). By default, if a token doesn't get replaced with a real value, it's removed.

    This allows us to use multiple tokens here depending on which entity type is being viewed. Thus, subsequent "child" background images do not actually need to override this value since it automatically inherits this value from the global background image and displays the proper title.

    Note: to allow users (or yourself) to view which tokens are available (link that displays a dialog like in the screenshot), you will need to install the Token contrib module.

  8. Once you've configured the global background image, go ahead and click Save.
  9. You will be redirected back to the background image listing page (/admin/config/media/background_image) and see the "Global: Site wide" background image you just created:
    Background Image: Administrative Listing

Step 6: Test

Navigate to your home page, you should see something like the following:

Without the "Full Viewport" setting enabled (blurred):

Background Image: Front without the Full Viewport setting enabled (blurred)

With the "Full Viewport" setting enabled:

Background Image: Front with the Full Viewport setting enabled

Reminder: this module does not generate stylistic CSS. Your sub-theme is responsible for styling different colors depending on whether an image is "dark" or not. The following screenshots are to prove that this is the "intended behavior" (without sub-theme styling). Please do not open an issue saying that it is "broken"; it's not.

Mark Carver, Drupal Bootstrap maintainer and Senior Frontend Performance Engineer, builds maintainable and responsive Drupal front-ends for Tag1's clients.

Future Roadmap

I really hope that if you've made it this far, you're as excited about this module as I am! I know it doesn't cover every use case and there are likely to be a few things that still need some fine tuning once it has survived its real world debut. Just know that this is just the beginning.

I've already mentioned a few planned features that will make their way into the module, but here are even more ideas that have been floating around in my head while I've been working on it:

  • Decouple the soft dependency on the Inline Entity Form module for embedding the Background Image form on other entities.
  • More UI based configuration - There's currently a lot of hardcoded config values, but it's all developed in a way that will allow for each config value to be easily configurable from the UI (or manually in a config file). Settings like determining the fallback and preload image styles, which responsive image configuration to use, retina media queries rules, the base CSS class to use, etc.
  • Routes and/or Path based background image types
  • The ability to credit/attribute the owner of the background image.
  • New image effects/filters (like the scrolling blur effect you see here).
  • Media API integration (still waiting for the initiative to land in core, in a stable manner).
  • And more... submit a feature request!
Apr 13 2017
Apr 13

Apache JMeter and I have a long and complicated relationship. It is definitely a trusted and valuable tool, but I am also quite confident that certain parts of it will make an appearance in my particular circle of hell. Due to this somewhat uncomfortable partnership, I am always interested in new tools for applying load to an infrastructure and monitoring the results. Locust.io is not exactly a new tool, but I have only recently begun to use it for testing.

What Is Locust?

Locust is a load-testing framework which allows you to write your load plan in regular Python. This is a welcome experience if you have ever had to manually edit a JMeter JMX file. Not only is it a more pleasant experience, but writing executors in Python makes it easy to create a very flexible load plan.

     Idea For A Circle Of Hell: Given a slightly corrupted JMX file that must be loadable and cannot easily be replaced, attempt to look through it to find the error preventing loading. Every time you save the file, some other tag corrupts slightly. Who needs eternal damnation, give me a large JMX file and some failing drives…

The other advantage of Locust is that it has a quite nice flask-based UI (that you can extend fairly easily) and it is quite easy to distribute load generation among multiple locust instances or servers.

Simple Load Plan

In the grand tradition of blog entries like this, let's build a completely impractical, simplistic example.

from locust import HttpLocust, TaskSet, task

class test_user(TaskSet):
    @task
    def front_page(self):
        self.client.get("/")
        
    @task
    def about_page(self):
        self.client.get("/about/")

class TestUserRunner(HttpLocust):
    task_set = test_user

The above code imports the required pieces to build a Locust test plan, these being the TaskSet, HttpLocust instance, and the task decorator. The class you create by inheriting the TaskSet class represents a type of user for the most part (Anonymous, Authenticated, Editorial, etc). In reality it is just a set of individual tasks, supporting elements and supporting methods. However, that matches the reality of a user rather well, so in general I define separate task sets for different user types.

The majority of this code is fairly self-explanatory. You can make requests via the client.get call and individual tasks are marked with the ‘@task’ decorator. These tasks will be what the main testing routine executes and you can weight each task differently, if you choose to. For example, in the above code we might want to weight the front_page higher than the about_page, since the front page will likely see more traffic. You do this by passing the task decorator a weight (where a higher weight equals increased likelihood of running), like so:

@task(10)
 
def front_page(self):

    self.client.get("/")

Running Our Simple Load Plan

Executing our load plan is not difficult. We save the code to plan.py (or any other name that is NOT locust.py) and run:

locust -f plan.py --host=<our test target>

We then open a browser and go to localhost:8089. You will be prompted for the number of users to spawn and how many users to spawn per second. Once you fill this out you can start the test. You will see something like the following:

Locust dashboard

This screen will allow you to monitor your load test, download csv files containing results, stop the load test, and see failures. A quick note on ramp-up: You may notice that you get results up until the point where all of your requested users are launched, then the results are cleared. This is so your final results only include numbers from when all users were launched, but it can take you by surprise if you are not expecting it.

Fetching Static Assets

While the above example can test Drupal's ability to build and deliver a page, it doesn't do much to actually test the site or infrastructure. This is partly because we aren't fetching any static assets. This is where things get a bit interesting. In JMeter, you would check a box at this point. Locust on the other hand trusts you to handle this yourself. It has a very high opinion of you.

Fortunately, it isn’t that hard. There are a few different tools you can use to parse a returned page and pull out static resources. I am going to use BeautifulSoup because I find the name amusing.

     NOTE: It is decisions like this that make me think I need to start planning for which circle of hell I end up in.

For my load tests I wrote a helper function called “fetch_static_assets”. The function is below:

def fetch_static_assets(session, response):
    resource_urls = set()
    soup = BeautifulSoup(response.text, "html.parser")
 
    for res in soup.find_all(src=True):
        url = res['src']
        if is_static_file(url):
            resource_urls.add(url)
        else:
            print "Skipping: " + url

The function is_static_file is quite important. The BeautifulSoup is going to return all URLs to you. Some of these may be broken, some may be off-site, etc. I recommend defining the is_static_file function and have it return false. Then look at what URLs are being skipped and slowly add patterns that match your static files and/or the URLs you want to fetch as sub-requests. In particular for a staging site, you don’t necessarily want to apply load to everything linked from your page. Here is an example of a very simplistic is_static_file function:

def is_static_file(f):
    if "/files" in f:
        return True
    else:
        return False

The rest of the fetch_static_assets function is below:

    for url in set(resource_urls):
        if "amazonaws.com" in url:
            session.client.get(url, name="S3 Static File")
            print "S3: " + url
        else:
            session.client.get(url, name="Static File")
            print "Regular: " + url

You can pass these static files into client.get for monitoring, but I would recommend setting the name to something consistent or else it is going to make your results quite messy. As you can see, I am tagging S3 URLs separately from regular static files in this example. Since you're defining all of this yourself, you have the flexibility to do basically whatever you want when you are parsing the page response and requesting sub-resources.

Below is an example of using this static asset function:

@task(10)
def frontpage(self):
    response = self.client.get("/")
    fetch_static_assets(self, response)

Logging Into Drupal

So, our load test can now fetch static assets. It can even fetch static assets of our choice and tag them as we would like. However, we are basically just testing the Drupal page cache at this point or perhaps Varnish or NGINX or even a CDN. Could be useful...probably isn’t though. To really be useful, we are going to have to login to the site. Fortunately this isn’t that difficult with Locust and we can use BeautifulSoup again. We are going to use the on_start method now. This is a special method of a Locust TaskSet that gets called on the start of the task set. It is not creatively named. Our example on_start is below:

def on_start(l):
    response = l.client.get("/user/login", name="Login")
    soup = BeautifulSoup(response.text, "html.parser")
    drupal_form_id = soup.select('input[name="form_build_id"]')[0]["value"]
    r = l.client.post("/user/login", {"name":"nnewton", "pass":"hunter2", "form_id":"user_login_form", "op":"Log+in", "form_build_id":drupal_form_id})

And there it is. Once this TaskSet logs in, Locust will keep the session cookie for the duration of that run. All requests from this TaskSet will be considered a logged in user. It is not uncommon for a test plan to have two TaskSets at the outside, one to cover the anonymous use-case and one for logged in users.

Locust is definitely a bit more difficult to approach than the JMeter GUI, but I have found it to be much easier to deal with when you are attempting to represent a somewhat complicated user pattern. In our next blog on this topic, we will be discussing how to use SaltStack and EC2 to automate Locust testing from multiple endpoints, i.e. spinning up a group of test VMs to run the Locust test plan and report back to a reporting node.

Download the demo load test described in this post.

Take the headache out of Drupal security with Tag1 Quo - Drupal Monitoring Made Easy.

Mar 15 2017
Mar 15

Though it came and went largely unnoticed, February 24th, 2017 marked an important anniversary to tens of thousands of Drupal website owners. February 24th 2017 was the 1-year anniversary of the End-of-Life (EOL) announcement for Drupal 6 as no longer supported by the Drupal community.

It is widely known that major Drupal version upgrades require non-trivial resources. Not only do they require significant planning, technical expertise, and budget, but the path is often determined by funding and availability of maintainers of popular contributed functionality (modules). Add the complexity of upgrading custom development, and the conditions create significant challenges for small to medium websites without large operating budgets. As evidence of this, our research indicates there are at least 150,000 publicly accessible sites still running Drupal 6.

Tag1 Quo is the only Drupal monitoring solution that supports Drupal 6 LTS, Drupal 7, and Drupal 8 under one dashboard.

For most D6 site managers, the most critical (and stressful) impact of EOL is the discontinuation of Drupal 6 security patches by the Drupal security team. When a major version reaches EOL, the Drupal security team ceases to release patches, or serve public Security Advisories for that version. Unless those sites are maintained by a skilled developer with the time to monitor upstream projects, review patches, and backport them by hand, Drupal 6 site managers find themselves in a vulnerable spot: ongoing, publicly announced vulnerabilities may be directly exposed on their site.

To its credit, the Drupal security team developed a plan so as not to abandon D6 site owners to the wilderness. Under the Long Term Support (LTS) initiative, they selected Tag1 and other qualified vendors to provide Drupal 6 patches as a paid service to site owners, under the condition that those patches also be made available to the public.

Tag1 Quo: A Year of LTS

With the EOL deadline rapidly approaching, Tag1—like many Drupal consulting firms—had clients still on Drupal 6. We were happy to sign on as an LTS provider to support our clients formally under the LTS initiative. It didn’t take us long to decide on automating patch delivery and empowering customers with some useful management tools. A few months into EOL, Tag1 Quo was launched with automated detection and notification of outstanding patches, and a unified view of security updates across all of their Drupal websites.

The vision was simple:Tag1 Quo Security Patch Notification

  • Provide D6 sites with a dashboard to quickly assess the status of their modules and themes, providing automated patches and pre-packaged releases delivered to their inbox, tested by our team of Drupal experts.
  • Make it platform and hosting-agnostic to provide maximum flexibility to the varied workflows and infrastructure of our customers.
  • Make it simple to setup and run from any site, in any environment, returning clear status reports and ongoing updates with the install of one module and a few clicks.
  • Price it aggressively: for less than the cost of 1 hour of senior developer time per month, a D6 customer could lease Quo as their security concierge, monitoring for patches around the clock.

Because of customers of Tag1 Quo and the LTS program, we’ve delivered on that vision. Paying customers of LTS have financed 25 Drupal 6 patches, written and contributed back to the community. While Drupal 8 continues to mature and add contributed functionality, the D6 LTS initiative is still going strong, giving site managers breathing room to fundraise, budget, and plan for their next big upgrade.

Enterprise Security with Tag1 Quo

Like many power users of Drupal, at Tag1 we maintain internal Drupal 6, 7, and 8 sites, as well as client sites on all of those versions. As we began designing and building Tag1 Quo, we quickly realized that the tools we wanted and needed for managing updates across sites were tools that would come in handy for other enterprise users:

  • agencies
  • universities
  • large corporations, and
  • infrastructure providers

In January 2017, we launched support for Drupal versions 7 and 8 on Tag1 Quo. With discounted rates for multiple sites, Tag1 Quo customers can now manage multiple sites via a centralized security dashboard with email notifications, across Drupal versions 6 through 8.

Tag1 Quo dashboard

Quo also provides powerful filtering tools across all sites. Filter by site, project, module version to see all instance of a particular module, across all sites. At-a-glance status color-coding tells you if your module has available updates, security-related or otherwise.

Filtering on modules in Tag1 Quo

Click in on a module to access a direct link to the latest release and access project metadata such as package, info, schema, and dependencies.Tag1 Quo module details

In managing our own sites, we’ve found that combining these tools in one central system has rapidly increased our turnaround on identifying and patching vulnerabilities, while lowering our management overhead. Eating our own dogfood has been satisfying: Tag1 Quo has freed up valuable developer time and budget to focus on feature development. If you are an agency maintaining client sites, or an IT department managing multiple corporate properties, you must have a security updates monitoring strategy and we’re confident that enterprise Tag1 Quo provides a solution.

Making Drupal maintenance easy forever

For years, the community has wrestled with the problem of expensive upgrades referenced in the beginning of this blog. How can Drupal continue to be a leader in innovation without becoming cost prohibitive to non-enterprise users? Last week, Dries published an important blog Making Drupal upgrades easy forever that announces an exciting, new approach for Drupal upgrades, based on a policy change initiated by Tag1’s Nat Catchpole.

Writes Dries:

we will continue to introduce new features and backwards-compatible changes in Drupal 8 releases. In the process, we sometimes have to deprecate old systems. Instead of removing old systems, we will keep them in place and encourage module maintainers to update to the new systems. This means that modules and custom code will continue to work. The more we innovate, the more deprecated code there will be in Drupal 8...Eventually, we will reach a point where we simply have too much deprecated code in Drupal 8. At that point, we will choose to remove the deprecated systems and release that as Drupal 9. This means that Drupal 9.0 should be almost identical to the last Drupal 8 release, minus the deprecated code.

For site owners and decision makers, this change is potentially earth-shattering. It replaces the monumental major version upgrade with incremental minor-version updates. Drupal 6 sites planning a Drupal 7 upgrade might want to revisit that plan. Drupal 7 sites waiting to upgrade directly to Drupal 9 may also want to reconsider. Site managers will need to invest more time on planning around minor releases: contributed code they rely on will be ported more frequently (though less dramatically). These changes are good for the Drupal ecosystem but issues of backward compatibility, legacy APIs, and deprecated code will likely require additional diligence.

We’ve built Tag1 Quo with an eye to this new future, with current and upcoming features to help site owners manage this complexity. If you are still on Drupal 6, Tag1 Quo has your back. If you are still on Drupal 7 when it goes EOL, Tag1 Quo will be there. And if you are somewhere in-between D8.7 and D8.11, Tag1 Quo will also be there for you, too.

In March 2017, get $50 credit towards your subscription with

Oct 25 2016
sam
Oct 25

When we left off last time, we’d assembled a definition of what versions are. Now, we’re going to dive into how we use them in Tag1 Quo: comparing them to one another!

The general goal is straightforward enough: we want to know if, say, 6.x-1.0 is less than 6.x-1.1. (Yup!) Or if 6.x-1.0-alpha1 is less than 6.x-1.0. (Also yup!) Let’s rewrite these two examples as tuple comparisons:

{6,1,0,4,0,0} < {6,1,1,4,0,0} = TRUE
{6,1,0,0,0,0} < {6,1,1,0,0,0} = TRUE

To determine if one tuple is less than the other, we proceed pairwise through the tuple’s values, comparing the integers at the same position from each, until we find different values. Whichever tuple’s value at that position is less is considered to be the lesser version. (Uniformity in this comparison operation is why the mapping for prerelease types assigns unstable to 0, rather than 4.)

However, this simple comparison operation doesn’t actually meet Quo’s requirements. Remember, Quo’s crucial question is not whether there are any newer versions, but whether there are newer security releases that are likely to apply to the version we’re investigating.

So, say we’re looking at 6.x-1.1 for a given extension, and there exists a 7.x-2.2 that’s a security release. While the latter is obviously less than the former:

{6,1,1,4,0,0} < {7,2,2,4,0,0} = TRUE

We don’t care, because these releases are on totally different lines of development.

...right? I mean, it’s probably true that whatever security hole existed in 7.x-2.1 doesn’t exist in 6.x-1.1. Maybe? Sort of. Certainly, you can't upgrade to 7.x-2.1 directly from 6.x-1.1, as that's changing major versions. But Quo came to be as part of the D6LTS promise - that IF there are security holes in later versions, we'll backport them to 6.x - so it's certainly possible that the problem might still exist. It all depends on what you take these version numbers to mean.

Yeah, we need to take a detour.

Versions are meaningless

As you become accustomed to a version numbering scheme - Drupal, semver, rpm, whatever - the meanings of the version components gradually work their way to the back of your mind. You don’t really “read” versions, so much as “scan and decode” them, according to these osmosed semantics. This peculiar infection of our subconscious makes it far too easy to forget a simple fact:

Version numbers have absolutely no intrinsic meaning. They have no necessary relationship to the code they describe.

Maybe this is obvious. Maybe it isn’t. If not, consider: what would prevent you from writing a module for Drupal 7 APIs, but then tagging and releasing it as 8.x-1.0? Or, for that matter, writing a module with no functions, but prints “spork” on inclusion of its .module file? (Answer: nothing.) Also, Donald Knuth uses a π-based numbering system for TeX’s versions, adding one more digit with each successive release. The version looks like a number, but the only property that matters is its length. Versions are weird.

This nebulous relationship is both the blessing and curse of versions. The curse is obvious: we can’t actually know anything with certainty about code just by looking at, or comparing, version numbers. But the blessing is more subtle: a well-designed version numbering system provides a framework for consistently encoding all of our intended semantics, together. Both of those words have specific meaning here:

  • “Together,” as in, it combines all the different aspects of changes to code that are important for Quo’s purposes: independent lines of development, Drupal core version compatibility, D6LTS’ own patch addenda, etc.

  • “Consistent,” as in, a numerical coördinate system - rather than an ad-hoc collection of flags, strings, and numbers - is a formal mathematical system without weird, nasty combinations of states.

The blessing outweighs the curse because, even if versions may lie to us about what the code actually is, they provide a formal structure in which it’s easy to understand what it should be. And, in the wild west of organic open source software growth, knowing with certainty about what things should be is a pretty good goal. It makes tasks concrete enough that you can actually build a business and product - like Tag1 Quo! Which takes us back to the main road after our detour - what’s the answer to this question?

{6,1,1,4,0,0} < {7,1,2,4,0,0}

The strictly mathematical answer is “yes.” But, for the question we’re actually interested in. we generally assume that security releases are only necessary when they’re on both the same core version, in the same line of development (major version). So, we say “no” here. And we’d also say “no” if the core version were the same:

{6,1,1,4,0,0} < {6,1,2,4,0,0}

This one is a little iffier, though. While porting from one Drupal core version to the next almost always involves a significant rewrite, that’s not necessarily the case for major versions. The security release may actually apply. It’s the kind of thing we need to investigate when deciding whether or not to release our D6LTS patch versions.

Today, Quo assumes security releases on different lines of development aren’t applicable to one another, but what’s important is that we know that’s an assumption. By representing the versions in a fully abstracted coördinate system as we have, rather than (for example) formally encoding assumptions about e.g. “lines of development” into the data itself, we allow ourselves the flexibility of changing those assumptions if they turn out to be wrong. Being that Quo’s entire business proposition turns on answering questions about versions correctly, it pays to build on a clear, flexible foundation.

This post rounded out the broader theory and big-picture considerations for versions in Quo. In the next post, I’ll get more into the nitty gritty - how Quo implements these ideas in a way that is practical, fast, and scalable.

 

Oct 25 2016
sam
Oct 25

When we left off last time, we’d assembled a definition of what versions are. Now, we’re going to dive into how we use them in Tag1 Quo: comparing them to one another!

The general goal is straightforward enough: we want to know if, say, 6.x-1.0 is less than 6.x-1.1. (Yup!) Or if 6.x-1.0-alpha1 is less than 6.x-1.0. (Also yup!) Let’s rewrite these two examples as tuple comparisons:

{6,1,0,4,0,0} < {6,1,1,4,0,0} = TRUE
{6,1,0,0,0,0} < {6,1,1,0,0,0} = TRUE

To determine if one tuple is less than the other, we proceed pairwise through the tuple’s values, comparing the integers at the same position from each, until we find different values. Whichever tuple’s value at that position is less is considered to be the lesser version. (Uniformity in this comparison operation is why the mapping for prerelease types assigns unstable to 0, rather than 4.)

However, this simple comparison operation doesn’t actually meet Quo’s requirements. Remember, Quo’s crucial question is not whether there are any newer versions, but whether there are newer security releases that are likely to apply to the version we’re investigating.

So, say we’re looking at 6.x-1.1 for a given extension, and there exists a 7.x-2.2 that’s a security release. While the latter is obviously less than the former:

{6,1,1,4,0,0} < {7,2,2,4,0,0} = TRUE

We don’t care, because these releases are on totally different lines of development.

...right? I mean, it’s probably true that whatever security hole existed in 7.x-2.1 doesn’t exist in 6.x-1.1. Maybe? Sort of. Certainly, you can't upgrade to 7.x-2.1 directly from 6.x-1.1, as that's changing major versions. But Quo came to be as part of the D6LTS promise - that IF there are security holes in later versions, we'll backport them to 6.x - so it's certainly possible that the problem might still exist. It all depends on what you take these version numbers to mean.

Yeah, we need to take a detour.

Versions are meaningless

As you become accustomed to a version numbering scheme - Drupal, semver, rpm, whatever - the meanings of the version components gradually work their way to the back of your mind. You don’t really “read” versions, so much as “scan and decode” them, according to these osmosed semantics. This peculiar infection of our subconscious makes it far too easy to forget a simple fact:

Version numbers have absolutely no intrinsic meaning. They have no necessary relationship to the code they describe.

Maybe this is obvious. Maybe it isn’t. If not, consider: what would prevent you from writing a module for Drupal 7 APIs, but then tagging and releasing it as 8.x-1.0? Or, for that matter, writing a module with no functions, but prints “spork” on inclusion of its .module file? (Answer: nothing.) Also, Donald Knuth uses a π-based numbering system for TeX’s versions, adding one more digit with each successive release. The version looks like a number, but the only property that matters is its length. Versions are weird.

This nebulous relationship is both the blessing and curse of versions. The curse is obvious: we can’t actually know anything with certainty about code just by looking at, or comparing, version numbers. But the blessing is more subtle: a well-designed version numbering system provides a framework for consistently encoding all of our intended semantics, together. Both of those words have specific meaning here:

  • “Together,” as in, it combines all the different aspects of changes to code that are important for Quo’s purposes: independent lines of development, Drupal core version compatibility, D6LTS’ own patch addenda, etc.

  • “Consistent,” as in, a numerical coördinate system - rather than an ad-hoc collection of flags, strings, and numbers - is a formal mathematical system without weird, nasty combinations of states.

The blessing outweighs the curse because, even if versions may lie to us about what the code actually is, they provide a formal structure in which it’s easy to understand what it should be. And, in the wild west of organic open source software growth, knowing with certainty about what things should be is a pretty good goal. It makes tasks concrete enough that you can actually build a business and product - like Tag1 Quo! Which takes us back to the main road after our detour - what’s the answer to this question?

{6,1,1,4,0,0} < {7,1,2,4,0,0}

The strictly mathematical answer is “yes.” But, for the question we’re actually interested in. we generally assume that security releases are only necessary when they’re on both the same core version, in the same line of development (major version). So, we say “no” here. And we’d also say “no” if the core version were the same:

{6,1,1,4,0,0} < {6,1,2,4,0,0}

This one is a little iffier, though. While porting from one Drupal core version to the next almost always involves a significant rewrite, that’s not necessarily the case for major versions. The security release may actually apply. It’s the kind of thing we need to investigate when deciding whether or not to release our D6LTS patch versions.

Today, Quo assumes security releases on different lines of development aren’t applicable to one another, but what’s important is that we know that’s an assumption. By representing the versions in a fully abstracted coördinate system as we have, rather than (for example) formally encoding assumptions about e.g. “lines of development” into the data itself, we allow ourselves the flexibility of changing those assumptions if they turn out to be wrong. Being that Quo’s entire business proposition turns on answering questions about versions correctly, it pays to build on a clear, flexible foundation.

This post rounded out the broader theory and big-picture considerations for versions in Quo. In the next post, I’ll get more into the nitty gritty - how Quo implements these ideas in a way that is practical, fast, and scalable.

 

Drupal 6 reached end of life, but Quo has security patches your site needs to remain in the game.

Oct 20 2016
sam
Oct 20

When Tag1 decided to build Tag1 Quo, we knew there was one question we’d have to answer over, and over, and over again: is there a security update available for this extension? Answering that question - at scale, for many websites, across many extensions, through all the possible versions they might have - is the heart of what Quo does.

The problem seems simple enough, but doing it at such scale, for “all” versions, and getting it right, has some deceptive difficulties. Given a site with an extension at a particular version, we need to know where it sits on the continuum of all versions that exist for that extension (we often refer to that as the “version universe,”), and whether any of the newer versions contain security fixes.

There are a few different approaches we could’ve taken to this problem. The one we ultimately settled on was a bit more abstracted than what might initially seem necessary. It was also not a “typical” Drupal solution. In this blog series, I’ll cover both the theoretical foundation of the problem, and the approach we took to implementation.

What’s a version?

Let's start at the beginning.

Quo works by having existing Drupal 6 (for now!) sites install an agent module, which periodically sends a JSON message back to the Quo servers indicating what extensions are present on that site, and at what versions. Those “versions” are derived from .info files, using functions fashioned after the ones used in Drupal core. Once that JSON arrives at Quo’s servers, we have to decide how to interpret the version information for each extension.

All versions arrive as strings, and the first decision Quo has to make is whether that string is “valid” or not. Validation, for the most part, means applying the same rules as applied by the Drupal.org release system. Let’s demonstrate that through some examples:

6.x-1.0

Drupal extension versions are a bit different than most software versions in the wild, because the first component, 6.x, explicitly carries information about compatibility, not with itself, but with its ecosystem: it tells us the version of Drupal core that the extension is supposedly compatible with.

The next part, 1.0, holds two bits of information: the major (1) and minor (0) versions. It’s wholly up to the author to decide what numbers to use there. The only additional meaning layered on is that releases with a 0 major version, like 6.x-0.1, are considered to be prerelease, and thus don’t receive coverage from the Drupal.org security team. Of course, Quo’s raison d’etre is that Drupal 6 is no longer supported by the security team, so that distinction doesn’t really matter anymore.

Now, 6.x-1.0 is an easy example, but it doesn’t cover the range of what’s possible. For example:

6.x-1.0-alpha1

This adds the prerelease field - alpha1. This field is optional, but if it does appear, it indicates the version to be some form of prerelease - and thus, as above, not covered by the security team. Drupal.org’s release system also places strong restrictions on the words that can appear there: alpha, beta, rc, and unstable. Additionally, the release system requires that the word must be accompanied by a number - 1, in this case.

There are a couple of notably different forms that valid Drupal versions can come in. There can be dev releases:

6.x-1.x-dev

Dev releases can only have the compatibility version and a major version, and cannot have a prerelease field. They’re supposed to represent a “line” of development, where that line is then dotted by any individual releases with the same major version.

And of course, Drupal core itself has versions:

6.43

Core versions have the same basic structure as the major version/minor version structure in an extension. Here, the 43 is a minor version - a dot along the development line of the 6 core version.

These examples illustrate what’s allowed by the Drupal.org release system, and thus, the shape that all individual extensions’ version universe will have to take. All together, we can say there are five discrete components of a version:

  • Core version
  • Major version
  • Minor version
  • Prerelease type
  • Prerelease number

Viewed in this way, it’s a small step to abstracting the notion of version away from a big stringy blob, and towards those discrete components. Specifically, we want to translate these versions into a 5-dimensional coördinate system, or 5-tuple: {core, major, minor, prerelease_type, prerelease_num}, where each of these dimensions has an integer value. Four of the components are already numbers, so that’s easy, but prerelease type is a string. However, because there’s a finite set of values that can appear for prerelease type, it’s easy to map those strings to integers:

  • Unstable = 0
  • Alpha = 1
  • Beta = 2
  • Rc = 3
  • (N/A - not a prerelease) = 4

With this mapping for prerelease types, we can now represent 6.x-1.0-alpha1 as {6,1,0,1,1}, or 6.x-2.3 as {6,2,3,4,0}.

However, that’s not quite the end of the story. The Quo service is all about delivering on the Drupal 6 Long Term Support (D6LTS) promise: providing and backporting security fixes for Drupal 6 extensions, now that they’re no longer supported by the security team.

Because such fixes are no longer official, we can’t necessarily expect there to be proper releases for them. At the same time, it’s still possible for maintainers to release new versions of 6.x modules, so we can’t just reuse the existing numbering scheme - the maintainer might later release a conflicting version.

For example, if D6LTS providers need to patch version 6.x-1.2 of some module, then we can’t release the patched version as 6.x-1.3, because we don’t have the authority to [get maintainers to] roll official releases (we’re not the security team, even though several members work for Tag1), and the maintainer could release 6.x-1.3 at any time, with or without our fix. Instead, we have to come up with some new notation that works alongside the existing version notation, without interfering with it.

Converting to the coördinate system gives us a nice tip in the right direction, though - we need a sixth dimension to represent the LTS patch version. And “dimension” isn’t metaphorical: for any given version, say {6, 1, 0, 1, 1} (that is, 6.x-1.0-alpha1), we may need to create an LTS patch to it, making it {6, 1, 0, 1, 1, 1}. And then later, maybe we have to create yet another: {6, 1, 0, 1, 1, 2}.

Now, we also have to extend the string syntax to support this sixth dimension - remember, strings are how the agent reports a site’s extension versions! It’s easy enough to say “let’s just add a bit to the end,” like we did with the coördinates, but we’re trying to design a reliable system here - we have to understand the implications of such changes.

Fortunately, this turns out to be quite easy: {6, 1, 0, 1, 1, 1} becomes 6.x-1.0-alpha1-p1; {6, 2, 3, 4, 0, 1} becomes 6.x-2.3-p1. This works well specifically because the strings in the prerelease type field are constrained to unstable, alpha, beta, and rc - unlike in semver, for example:

     A pre-release version MAY be denoted by appending a hyphen and a series
     of dot separated identifiers immediately following the patch version.
     Identifiers MUST comprise only ASCII alphanumerics and hyphen
     [0-9A-Za-z-].

     ...identifiers consisting of only digits are compared numerically and
     identifiers with letters or hyphens are compared lexically in ASCII
     sort order.

In semver, prerelease information can be any alphanumeric string, can be repeated, and are compared lexicographically (that is, alpha < beta < rc < unstable). If Drupal versions were unbounded in this way, then a -p1 suffix would be indistinguishable from prerelease information, creating ambiguity and making conflicts possible. But they’re not! So, this suffix works just fine.

Now, a coordinate system is fine and dandy, but at the end of the day, it’s just an abstracted system for representing the information in an individual version. That’s important, but the next step is figuring out where a particular version sits in the universe of versions for a given extension. Specifically, the question Quo needs to ask is if there’s a “newer” version of the component available (and if so, whether that version includes security fixes). Basically, we need to know if one version is “less” than another.

And that’s where we’ll pick up in the next post!

Oct 20 2016
sam
Oct 20

When Tag1 decided to build Tag1 Quo, we knew there was one question we’d have to answer over, and over, and over again: is there a security update available for this extension? Answering that question - at scale, for many websites, across many extensions, through all the possible versions they might have - is the heart of what Quo does.

The problem seems simple enough, but doing it at such scale, for “all” versions, and getting it right, has some deceptive difficulties. Given a site with an extension at a particular version, we need to know where it sits on the continuum of all versions that exist for that extension (we often refer to that as the “version universe,”), and whether any of the newer versions contain security fixes.

There are a few different approaches we could’ve taken to this problem. The one we ultimately settled on was a bit more abstracted than what might initially seem necessary. It was also not a “typical” Drupal solution. In this blog series, I’ll cover both the theoretical foundation of the problem, and the approach we took to implementation.

What’s a version?

Let's start at the beginning.

Quo works by having existing Drupal 6 (for now!) sites install an agent module, which periodically sends a JSON message back to the Quo servers indicating what extensions are present on that site, and at what versions. Those “versions” are derived from .info files, using functions fashioned after the ones used in Drupal core. Once that JSON arrives at Quo’s servers, we have to decide how to interpret the version information for each extension.

All versions arrive as strings, and the first decision Quo has to make is whether that string is “valid” or not. Validation, for the most part, means applying the same rules as applied by the Drupal.org release system. Let’s demonstrate that through some examples:

6.x-1.0

Drupal extension versions are a bit different than most software versions in the wild, because the first component, 6.x, explicitly carries information about compatibility, not with itself, but with its ecosystem: it tells us the version of Drupal core that the extension is supposedly compatible with.

The next part, 1.0, holds two bits of information: the major (1) and minor (0) versions. It’s wholly up to the author to decide what numbers to use there. The only additional meaning layered on is that releases with a 0 major version, like 6.x-0.1, are considered to be prerelease, and thus don’t receive coverage from the Drupal.org security team. Of course, Quo’s raison d’etre is that Drupal 6 is no longer supported by the security team, so that distinction doesn’t really matter anymore.

Now, 6.x-1.0 is an easy example, but it doesn’t cover the range of what’s possible. For example:

6.x-1.0-alpha1

This adds the prerelease field - alpha1. This field is optional, but if it does appear, it indicates the version to be some form of prerelease - and thus, as above, not covered by the security team. Drupal.org’s release system also places strong restrictions on the words that can appear there: alpha, beta, rc, and unstable. Additionally, the release system requires that the word must be accompanied by a number - 1, in this case.

There are a couple of notably different forms that valid Drupal versions can come in. There can be dev releases:

6.x-1.x-dev

Dev releases can only have the compatibility version and a major version, and cannot have a prerelease field. They’re supposed to represent a “line” of development, where that line is then dotted by any individual releases with the same major version.

And of course, Drupal core itself has versions:

6.43

Core versions have the same basic structure as the major version/minor version structure in an extension. Here, the 43 is a minor version - a dot along the development line of the 6 core version.

These examples illustrate what’s allowed by the Drupal.org release system, and thus, the shape that all individual extensions’ version universe will have to take. All together, we can say there are five discrete components of a version:

  • Core version

  • Major version

  • Minor version

  • Prerelease type

  • Prerelease number

Viewed in this way, it’s a small step to abstracting the notion of version away from a big stringy blob, and towards those discrete components. Specifically, we want to translate these versions into a 5-dimensional coördinate system, or 5-tuple: {core, major, minor, prerelease_type, prerelease_num}, where each of these dimensions has an integer value. Four of the components are already numbers, so that’s easy, but prerelease type is a string. However, because there’s a finite set of values that can appear for prerelease type, it’s easy to map those strings to integers:

With this mapping for prerelease types, we can now represent 6.x-1.0-alpha1 as {6,1,0,1,1}, or 6.x-2.3 as {6,2,3,4,0}.

However, that’s not quite the end of the story. The Quo service is all about delivering on the Drupal 6 Long Term Support (D6LTS) promise: providing and backporting security fixes for Drupal 6 extensions, now that they’re no longer supported by the security team.

Because such fixes are no longer official, we can’t necessarily expect there to be proper releases for them. At the same time, it’s still possible for maintainers to release new versions of 6.x modules, so we can’t just reuse the existing numbering scheme - the maintainer might later release a conflicting version.

For example, if D6LTS providers need to patch version 6.x-1.2 of some module, then we can’t release the patched version as 6.x-1.3, because we don’t have the authority to [get maintainers to] roll official releases (we’re not the security team, even though several members work for Tag1), and the maintainer could release 6.x-1.3 at any time, with or without our fix. Instead, we have to come up with some new notation that works alongside the existing version notation, without interfering with it.

Converting to the coördinate system gives us a nice tip in the right direction, though - we need a sixth dimension to represent the LTS patch version. And “dimension” isn’t metaphorical: for any given version, say {6, 1, 0, 1, 1} (that is, 6.x-1.0-alpha1), we may need to create an LTS patch to it, making it {6, 1, 0, 1, 1, 1}. And then later, maybe we have to create yet another: {6, 1, 0, 1, 1, 2}.

Now, we also have to extend the string syntax to support this sixth dimension - remember, strings are how the agent reports a site’s extension versions! It’s easy enough to say “let’s just add a bit to the end,” like we did with the coördinates, but we’re trying to design a reliable system here - we have to understand the implications of such changes.

Fortunately, this turns out to be quite easy: {6, 1, 0, 1, 1, 1} becomes 6.x-1.0-alpha1-p1; {6, 2, 3, 4, 0, 1} becomes 6.x-2.3-p1. This works well specifically because the strings in the prerelease type field are constrained to unstable, alpha, beta, and rc - unlike in semver, for example:

     A pre-release version MAY be denoted by appending a hyphen and a series
     of dot separated identifiers immediately following the patch version.
     Identifiers MUST comprise only ASCII alphanumerics and hyphen
     [0-9A-Za-z-].

     ...identifiers consisting of only digits are compared numerically and
     identifiers with letters or hyphens are compared lexically in ASCII
     sort order.

In semver, prerelease information can be any alphanumeric string, can be repeated, and are compared lexicographically (that is, alpha < beta < rc < unstable). If Drupal versions were unbounded in this way, then a -p1 suffix would be indistinguishable from prerelease information, creating ambiguity and making conflicts possible. But they’re not! So, this suffix works just fine.

Now, a coordinate system is fine and dandy, but at the end of the day, it’s just an abstracted system for representing the information in an individual version. That’s important, but the next step is figuring out where a particular version sits in the universe of versions for a given extension. Specifically, the question Quo needs to ask is if there’s a “newer” version of the component available (and if so, whether that version includes security fixes). Basically, we need to know if one version is “less” than another.

And that’s where we’ll pick up in the next post!

 

Tag1 Quo is the only Drupal monitoring solution that supports Drupal 6 LTS, Drupal 7, and Drupal 8 under one dashboard.

Aug 30 2016
Aug 30

Long Term Support for Drupal 6 might be my favorite new feature included in Drupal 8. (I know, that might be stretching things for the fundamentally awesome step forward that Drupal 8 is, but bear with me.)

Long Term Support for Drupal 6 Long Term Support for Drupal 6

If you're like me, you have loved the power of building websites for people that expose their ideas or services to the world. If you're like me, you've ended up "owning" a number of these websites that you somehow ended up supporting along the way too. And if you're like me, you've ended up with lots of Drupal 6 websites to support, even though D6 hit End-of-Life on February 24th, 2016.

There are a lot of D6 sites out there with no money for an upgrade, but which still have a niche to fill or useful information for the world. Those can be an albatross around our necks and a time sink. We don't have the resources to update (and their owners don't either) but we can't set the site owners adrift.

When previous versions of Drupal hit end-of-life, it was always a catastrophe for those of us with sites out there. Upgrade or else. Very costly in time and effort or money. But this time, with the release of Drupal 8, the Drupal Security Team and a number of vendors teamed up to actually offer real commercial support for Drupal 6. Yay! Yay!

One of those vendors is Tag1 Consulting with our slick new Tag1 Quo service, which monitors both Drupal.org security releases and your D6 site to make sure you know what the security issues are. We even work with the community and the Security Team to backport discovered D7 security fixes. (Full disclosure: I got to work on development of the very nice D8 Quo site.)

The Tag1 Quo Dashboard The Tag1 Quo Dashboard

I got to beta test Tag1 Quo with several of my old sites this year, and it was surprisingly easy. I just installed the D6 Tag1 Quo module on each D6 site, and immediately got a summary (on the Tag1 Quo site) of the security status of each site. Then, when Drupal 7 Security Advisories have gone out in recent months, I get an email with a patch or even a tarball for the affected module.

Drupal 6 Long Term Support is a huge step forward for our community, and great kudos are in order for the Security Team for arranging and supporting it and the vendors for providing it.

Tag1 has you covered with our Drupal security monitoring solution, Tag1 Quo.

Pages

About Drupal Sun

Drupal Sun is an Evolving Web project. It allows you to:

  • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
  • Facet based on tags, author, or feed
  • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
  • View the entire article text inline, or in the context of the site where it was created

See the blog post at Evolving Web

Evolving Web