Upgrade Your Drupal Skills

We trained 1,000+ Drupal Developers over the last decade.

See Advanced Courses NAH, I know Enough
Mar 12 2024
Mar 12

With Drupal now heavily used in the enterprise market by very large organisations, much of its direct competition is from well-funded proprietary products. From the perspective of my role on the Drupal Association board, I gave a talk at FOSDEM in February 2024 on the strategies and initiatives the Drupal community is starting to put in place to remain competitive in the enterprise market and how these approaches can be shared by other open source projects. 

[embedded content]

The original of this video recording was first published on the FOSDEM website

Drupal has historically had no centralised product management or marketing, let alone ANY coordinated budget! For comparison, Adobe spends around USD$2.7bn annually on product development, sales and marketing for its Experience Cloud product suite. 

In the talk, I discuss Drupal's recent recognition as a Digital Public Good and the way that the Drupal community is highly motivated by providing world-class software for free to anyone who wants to use it, promoting values of freedom, inclusion, participation and empowerment. The Drupal Association recently released a manifesto that defines the Drupal project's commitment to the Open Web, but in order to fulfil this mission, Drupal needs to be successful as a product in the open market.

Since Drupal 8 was released in 2015, it has been specifically targeted at building "ambitious digital experiences." While this has resulted in an overall drop in Drupal installs as smaller sites move to SAAS platforms, the Drupal economy is robust, with an estimated USD$3 billion spent on Drupal-related projects each year.

Unlike other open source projects, Drupal doesn’t have a single company doing the majority of the code contribution. The Drupal Association has run on a budget of around $3.5m or 1/1000th of the revenue being spent on Drupal projects each year. 

This was brought into focus for the Drupal Association during COVID when the primary source of income - running DrupalCon events - required an abrupt rethink. We had to refocus on how Drupal would be both successful and sustainable in the future. This has led to us recently embarking on a new strategy, where the Drupal Association play a more direct role in both Drupal product innovation and marketing.

Enterprise customers are key to maintaining a healthy ecosystem for a CMS. Their investment flows through to the agencies building, maintaining, supporting, and hosting large-scale projects, providing consistent, repeat income that ultimately benefits our open source community in the form of stable jobs, community funding, and sponsored code contribution. 

Looking more closely at the challenges of succeeding in the enterprise market, how do you get access and awareness with key decision makers in large organisations like the CIO, CTO and, increasingly, the CMO (Chief Marketing Officer)? They are the people likely to read analyst reports from Gartner and Forrester. While Acquia features as a leader in these reports and relies heavily on Drupal for its platform, Drupal's name recognition is largely absent from these reports. 

Acquia has also had great success with their Engage events that target key decision makers, but it's been a challenge to attract a similar audience to the more community and developer-focused DrupalCon events. 

While the Drupal Association itself has historically had limited relationships with Drupal's large end users, partner agencies who rely on Drupal's open source software for their clients absolutely do have these relationships.

The Drupal Association is in a strong position to provide our agency partners with as much assistance as possible to either retain or win new enterprise clients through any playbook-style information we can provide. For example, do we have a pitch deck on hand to help an agency argue why Drupal is superior to Adobe or Sitecore? Are there pre-packaged product demos that can be consistently updated to highlight new features?

This is an area where we currently fall short in the Drupal community, with most agencies replicating efforts for every new client engagement. It's something we're starting to address with the Drupal Certified Partner program, however, if we can harness the strength of hundreds of agency salespeople pitching Drupal to their clients every day. New agencies joining a partner program need to see a clear pathway to building their teams' expertise and being able to sell Drupal to their clients to grow their businesses. The largest global digital agencies have tended to struggle with engaging with open source software communities, so bridging that gap is critical.

The other group of people we need to convince in any large organisation are the people who’ll be using our product - the developers, content editors and systems engineers. C-level decision-makers lean heavily on this group to evaluate and make recommendations about what platform they should be considering. To influence this group, our product needs to look and function like a modern piece of software, fulfil contemporary requirements or be quickly downloadable for a working demo of the software.

In terms of where we already clearly win, rapid innovation is the thing that we do very well in the open source world. Maintaining the speed of innovation, though, is an area that has been harder for Drupal as both the software and community have matured. A big philosophical hurdle we’ve faced is the notion of the Drupal Association directing budget to innovation projects when people often have an expectation that contribution is “free”. But contribution has never been free! An individual or company has always borne the cost in personal time or wages. Other big open source projects have absolutely no stigma about funding projects with actual money, such as the Linux Foundation's $160m annual funding towards projects.

The Drupal community dipped our toe into this model last year with the Pitchburgh contest, which saw $98,000 worth of projects get completed in a relatively short amount of time because they had the budget. We’re also in the process of hiring people at the Drupal Association who can facilitate innovation and remove roadblocks to contribution.

Now, all we need is the funding to scale this model up. Imagine if just 1% of the $3bn spent on Drupal-related projects each year went towards funding strategic innovation - that would be a $30m budget to work with!

Similarly, the idea that Drupal would be “marketed” as a product by the Drupal Association has never been a core competency. This is the legacy of being structured as a 501c3 not-for-profit in the USA where funds are for the “advancement of a charitable cause”. Our charitable cause is ensuring Drupal remains a Digital Public Good that supports the United Nations’ Sustainable Development Goals. But if there isn't positive product awareness about Drupal in the broader market, then market share will slip and our ability to support the goals around being a Digital Public Good will suffer as a result. 

Whether we call it marketing or advocacy, we need to ensure Drupal as a product is commercially successful. We’ve had a Promote Drupal working group within the Drupal community for a number of years that has driven a range of broader marketing initiatives. The Drupal Association has now taken on an active role in this by commissioning a go-to-market strategy targeting the enterprise sector. This will be rolling out in 2024 as funding for specific marketing initiatives becomes available. 

At the cheaper end of the scale, this might include coordinating speakers at non-Drupal tech events or managing positive media coverage. At a higher budget scale, it might include Drupal-branded booths at major tech conferences, like the one we recently built for Web Summit in Lisbon, or running global campaigns to build Drupal product awareness. 

Our other huge advantage as an open source community is the strength and depth of our developer pool. We do encounter a perception issue when it comes to attracting younger developers to our platforms because there are so many shiny new things to play with. Building robust outreach, training, mentoring, certification and professional pathways is the key to maintaining a sustainable developer pool as those of us with 20+ years of experience head towards the other side of middle age.

So, where can you start to help with all of this? 

  1. If you're a professional services company that relies on Drupal for your business, get involved with the Drupal Certified Partner program. This is the fastest way to both contribute to Drupal's innovation as a product and play a direct role in driving adoption.

  2. If you rely on Drupal as your organization's CMS software, become a Supporting Partner and help fund Drupal's sustainability. 

  3. If you're passionate about maintaining the Open Web, the Drupal Association can accept your philanthropic donation

  4. Send your team members to DrupalCon or a regional DrupalCamp to connect with the community.

This level of engagement will help Drupal maintain its status as the platform of choice for large-scale projects.

Feb 26 2024
Feb 26

As experienced sponsors, we have well-tried tips for enjoying the Sprint and maximising your impact on the Drupal community. Find out how to get involved, why contribution is important and how on-the-day collaboration works.

We’re back! Once again, we're running and sponsoring the Code Sprint, this time at DrupalSouth in Sydney. 

Whether you join us in person or virtually, it's a day where we collaborate to work, learn and contribute to the Drupal open-source project and community.

Getting together this way creates new ideas and initiatives, pushing the boundaries of what is possible. Plus, we get to have fun doing it!

When is the Code Sprint?

For full information, check out the DrupalSouth registration form.

How do I get involved?

Registering for the Code Sprint is seamlessly integrated into the DrupalSouth registration process

Attendance is free, and participants of all experience levels are welcome.

Due to its popularity, we advise you to register early and secure your spot!

Participate remotely

Can't be there in person? You won't have to miss out!

Join us virtually through Slack and Zoom to contribute from anywhere worldwide.

Details for connecting remotely will be shared via Slack closer to the event date.

Tell me more about Sprints and why they matter

Drupal is an open-source project reliant on community contributions (from organisations and individuals) to keep it moving forward and improving. 

The Sprint is a focused day of progress, where we achieve tangible results, expertise is recognised and acknowledged, and developers can mentor one another.

Your participation in the Sprint helps to build your contribution levels for the Certified Partner Program

Whether you're a seasoned contributor or new to Drupal Sprints, these recorded sessions from previous events will help you prepare:

Sprints are about more than coding; they're opportunities to connect, discuss common interests, put forward ideas and collaborate on various topics.

Groups will mostly be arranged by topic. e.g. Bug Smash, Media, Drupal 10 porting. Look for a group working on something that interests you, and join in! 

There's no pressure to complete lines and lines of code on the day. We want this to be a positive experience for everyone.

Communication 

Communication during the Sprint will primarily occur through #drupalsouth-code-sprint in Drupal Slack, with dedicated threads for streamlined discussions.

Follow the instructions for how to join the Australian / New Zealand Drupal community in Slack. 

Can anyone contribute?

Everyone is welcome to contribute on Sprint Day, regardless of technical background. Contribution can take many forms, from coding to issue triaging.

Check out the contributor tasks to discover the ways you can get involved. 

Issue Queue

The Drupal.org Issue Queue has issues tagged with 'DrupalSouth' by other contributors. This is where you can also tag issues to add.

Development environment setup

When it comes to setting up a local development environment for working on Drupal, you have options.

For local development environment setup, our pick is Docker Compose. Follow the instructions for installing Docker Compose on OSX, Windows and Linux.

If you don't already have a local development environment for Drupal contribution, you can set up a starter project using this:

composer create-project mstrelan/drupal-contrib

See the README.md for more details. 

If you're more familiar with DDEV, we recommend you look at DDEV Drupal Contrib.

If you experience any issues, join us on Slack beforehand, and we'll happily answer your questions.

Code of conduct

The Sprint Day adheres to the DrupalSouth Code of Conduct, fostering a safe and inclusive environment for all participants. 

Summary

To make the most of the Code Sprint:

We can’t wait to see you there!

Feb 13 2024
Feb 13

Symfony Messenger and the SM integration permits time savings and improved DX for developers, making the pipeline and workers more flexible.

Real business value is added when messages can be processed in real-time, providing potential infrastructure efficiencies by permitting rescaling resources from web to CLI. End user performance is improved by offloading processing out of web threads.

Further integrations provide feedback to end users such as editors, informing them in real-time when relevant tasks have been processed.

Messenger messages are an opportunity to offload expensive business logic, particularly those which would be found in hooks. Instead of using the batch API, consider creating messages and deferring UI feedback to Toasty rather than blocking a user’s browser.

Given these points, integration with Drupal core could prove quite valuable.

The design of the SM project focuses on core integration from the beginning. The highest priority has been to maintain the original Symfony Messenger services and configuration, while minimising the introduction of new concepts.

In the context of core integration, niceties like SM’s queue interception feature may initially be left on the factory floor, but could prove useful when deprecating @QueueWorker plugins.

Concepts like the consume command may be a tricky requirement for some, but there are always workarounds that can be built, in the same vein as request-termination cron. Workarounds wouldn’t allow for the main advantage of Messenger, which is its real-time processing capabilities.

Next steps for the SM project and the Messenger ecosystem include

Read the other posts in this series:

  1. Introducing Symfony Messenger integrations with Drupal 
  2. Symfony Messenger’ message and message handlers, and comparison with @QueueWorker
  3. Real-time: Symfony Messenger’ Consume command and prioritised messages
  4. Automatic message scheduling and replacing hook_cron
  5. Adding real-time processing to QueueWorker plugins
  6. Handling emails asynchronously: integrating Symfony Mailer and Messenger
  7. Displaying notifications when Symfony Messenger messages are processed
  8. Future of Symfony Messenger in Drupal
Feb 12 2024
Feb 12

A trio of modules provide a unique experience for sites utilising SM and Symfony Messenger messages: Common Stamps, Pusher Mini, and Toasty. This combination ultimately shows UI toasts in the browser when Symfony messages relevant to the user are processed.

This post is part 7 in a series about Symfony Messenger.

  1. Introducing Symfony Messenger integrations with Drupal
  2. Symfony Messenger’ message and message handlers, and comparison with @QueueWorker
  3. Real-time: Symfony Messenger’ Consume command and prioritised messages
  4. Automatic message scheduling and replacing hook_cron
  5. Adding real-time processing to QueueWorker plugins
  6. Making Symfony Mailer asynchronous: integration with Symfony Messenger
  7. Displaying notifications when Symfony Messenger messages are processed
  8. Future of Symfony Messenger in Drupal
toast

The full picture

From beginning to end, when a message is dispatched additional metadata is applied to the message. Later, after the message has been successfully processed, the message is captured. Metadata is extracted from the message and transformed. Recipients for the message are determined, usually based on the user who was logged in when the message was dispatched. The processed metadata is dispatched to a Pusher-compatible websocket server, which pushes the processed metadata to web browser sessions of the message recipients.

The message

A message is constructed and dispatched as usual, though to utilise the UI the Description , Action , and Link stamps provided by the Common Stamps module are applied. These stamps provide additional user-friendly context describing what the message is doing, and subsequently display text, links, and buttons once the message has been successfully processed.

use \Drupal\stamps\Messenger\Stamp\DescriptionStamp;
use \Drupal\stamps\Messenger\Stamp\ActionsStamp;
use \Drupal\Core\Link;
use \Symfony\Component\Messenger\Envelope;

$envelope = new Envelope($testMessage, [
  DescriptionStamp::create(title: 'A custom message!', description: '...and a description.'),
  ActionsStamp::fromLinks(primary: [
    Link::fromTextAndUrl('Go home', Url::fromRoute(''))
  ]),
]);
$bus->dispatch($envelope);

This code produces toasts like:

ToastyToasty displaying notifications.

For example, a message may be scheduled for a future date (via \Symfony\Component\Messenger\Stamp\DelayStamp) to publish content. Metadata stamps could include information such as the content title, a description like The content was published, and a handy action link to the content itself.

Bus, middlewares, and processing

Once a message has been dispatched to the bus, middleware from both Common Stamps and Toasty will modify and send notifications respectively. The current user middleware from Common Stamps will apply a user stamp tracking which user, if any, was authenticated while the message was dispatched.

Later, after the message handler has successfully processed the message, Toasty’s’ middleware intercepts the message. Metadata like description, links, buttons, and recipients are compiled by reading the stamps applied to the message. This metadata is subsequently transmitted to the websocket server.

The websocket server

Toasty includes a Vue app to render the user interface and connects to the websocket server. It establishes a persistent connection and listens for notifications dispatched by Toasty to the websocket server.

The websocket server can be the official Pusher.com service, which provides 200,000 free messages per day. Alternatively, you can choose to self-host using Soketi, an open source app that is compatible with the Pusher API.

Pusher Mini combines Key, which is used for storing secrets, with the Pusher PHP library to connect to Pusher.com or a Soketi server.

After installation, a key is set up with app secrets. The Pusher Mini module configuration sets up routing, non-secrets, and connection details for both the client-websocket server, and another for the application-websocket server.

Key configPusher mini config

Once websockets are configured, messages will be pushed to the browser as soon as the middleware of Messenger and Toasty has processed the message.

Received messages are transformed into small notification toasts for the user to take action on or ignore.

toast

The next and final post in this Symfony Messenger series covers how we could bring the integration components introduced by SM into Drupal core.

Feb 07 2024
Feb 07

This post is part 6 in a series about Symfony Messenger.

  1. Introducing Symfony Messenger integrations with Drupal
  2. Symfony Messenger’ message and message handlers, and comparison with @QueueWorker
  3. Real-time: Symfony Messenger’ Consume command and prioritised messages
  4. Automatic message scheduling and replacing hook_cron
  5. Adding real-time processing to QueueWorker plugins
  6. Making Symfony Mailer asynchronous: integration with Symfony Messenger
  7. Displaying notifications when Symfony Messenger messages are processed
  8. Future of Symfony Messenger in Drupal

Since Swift Mailer and its Drupal contrib integration were recently deprecated, many projects have naturally switched to its replacement: Symfony Mailer, either via Drupal Symfony Mailer or Drupal Symfony Mailer Lite.

This post outlines how you can take advantage of Symfony Mailer’s first class integration with Symfony Messenger brought to Drupal via the SM project. This integration allows for dispatching emails off-thread, potentially improving performance of the dispatching (usually web-) thread by offloading email-related tasks to dedicated Symfony Messenger workers. This setup can be considered an alternative to using Queue Mail.

Setup

As of writing, of the two Symfony Mailer implementations in contrib, Drupal Symfony Mailer Lite has built in support for Symfony Messenger. Drupal Symfony Mailer does not yet support it, an issue and merge request exist to add it. Apply a patch until the changes are merged.

Symfony Messenger itself does not require any special configuration, other than installing SM.

To run asynchronously, the \Symfony\Component\Mailer\Messenger\SendEmailMessage message must have routing configuration to a transport. Or at least the fallback transport must be configured. Without transport configuration, Emails will still be dispatched through Messenger, however they will be executed synchronously in the same thread they were dispatched.

Opting out

If you happen to have both Symfony Mailer and Symfony Messenger installed but do not want emails to be sent asynchronously, you can configure routing for the \Symfony\Component\Mailer\Messenger\SendEmailMessage message to instead use the synchronous transport.

If you’re using the SM Config submodule:

screenshot of routing

Sending emails and dispatching emails

Emails may be dispatched using the usual Drupal mechanism, or you can dispatch using Symfony Mailer directly by constructing an email object:

$email = (new \Symfony\Component\Mime\Email())
  ->to('[email protected]')
  ->from('[email protected]')
  ->subject('Hello world!')
  ->text('Some sample text.')
  ->html('

some sample text.

'); /** @var \Symfony\Component\Mailer\MailerInterface $mailer */ $mailer = \Drupal::service(\Symfony\Component\Mailer\MailerInterface::class); $mailer->send($email);

After the send method is executed, Mailer checks Messenger is available, creates a new SendEmailMessage message to wrap the \Symfony\Component\Mime\Email object. Then dispatches SendEmailMessage to the messenger bus.

As is typical with Symfony Messenger, email messages must be serialisable. Avoid including any Drupal entities or service references in an email object, and render email contents before sending it.

Processing emails

To process email messages, run the worker with sm messenger:consume. This command will either listen or poll for messages and execute them in a dedicated thread, ensuring quick processing after they are dispatched. For more information on the worker, please refer to post 3 of this series.

In the next post, we’ll explore how to add a user interface to notify users when relevant tasks have been processed.

Feb 06 2024
Feb 06

This post is part 5 in a series about Symfony Messenger.

  1. Introducing Symfony Messenger integrations with Drupal
  2. Symfony Messenger’ message and message handlers, and comparison with @QueueWorker
  3. Real-time: Symfony Messenger’ Consume command and prioritised messages
  4. Automatic message scheduling and replacing hook_cron
  5. Adding real-time processing to QueueWorker plugins
  6. Making Symfony Mailer asynchronous: integration with Symfony Messenger
  7. Displaying notifications when Symfony Messenger messages are processed
  8. Future of Symfony Messenger in Drupal

QueueWorker plugins

@QueueWorker plugin implementations require no modifications, including the method of dispatch, data payload, or the processItem . The data payload must of course be serialisable. Fortunately, most QueueWorker plugins already comply since their data is serialised and stored to the queue table. As always, avoid adding complex objects like Drupal entities to payloads.

Runners

With queue interception, the sm command can be solely relied upon. Legacy runners such as Drupal web cron, request termination cron (automated_cron.module), and Drush queue:run will be rendered inoperable since they will no longer have anything to process. Consider decommissioning legacy runners when deploying queue interception.

Setup

Queue interception is a part of the primary SM module. Adding a single line in settings.php is the only action required to to enabling this feature:

$settings['queue_default'] = \Drupal\sm\QueueInterceptor\SmLegacyQueueFactory::class;

SM module will need to be fully installed before this line is added. Consider wrapping the line in a class_exists(SmLegacyQueueFactory::class) to enable in a single deployment.

Existing per-queue backends

Setup may be more complex if projects are utilising per-queue backends or anything other than the default database backend for queues, such as Redis. In that case, carefully evaluate whether to convert all or specific queues to use Symfony Messenger.

Whether per-queue backends are utilised can be determined by looking for queue_service_ or queue_reliable_service_ prefixed items in settings.php.

Routing

@QueueWorker jobs are converted to \Drupal\sm\QueueInterceptor\SmLegacyDrupalQueueItem messages in the backend. Knowing this class name allows you to configure transport routing. If routing for this message is not explicitly configured, it will naturally fall back to the default transport, or execute synchronously if there is no routing configuration.

Running the jobs

As usual, when a transport is configured, all you need to do is run sm messenger:consume to execute the tasks. The worker will either listen or poll for messages, and execute them in a very short amount of time after they are dispatched, in a dedicated thread. More information on the worker can be found in post 3 of this series.

The next post covers how Drupal emails can be dispatched to messages, so the web thread can execute faster.

Feb 05 2024
Feb 05

Symfony Scheduler provides a viable replacement to hook_cron wherein messages can be scheduled for dispatch at a predefined interval. Messages are dispatched the moment they are scheduled, and there is no message duplication, making tasks more reliable and efficient.

This post is part 4 in a series about Symfony Messenger.

  1. Introducing Symfony Messenger integrations with Drupal
  2. Symfony Messenger’ message and message handlers, and comparison with @QueueWorker
  3. Real-time: Symfony Messenger’ Consume command and prioritised messages
  4. Automatic message scheduling and replacing hook_cron
  5. Adding real-time processing to QueueWorker plugins
  6. Making Symfony Mailer asynchronous: integration with Symfony Messenger
  7. Displaying notifications when Symfony Messenger messages are processed
  8. Future of Symfony Messenger in Drupal

With this, the sm worker provided by the SM project, the Symfony Messenger integration with Drupal, can be solely relied on. Rather than legacy runners such as Drupal web cron, request termination cron (automated_cron.module), Drush cron, and Ultimate Cron.

Scheduler functionality is implemented by the Symfony Scheduler component. The Drupal integration is provided by the SM Scheduler module

Schedule provider

Create a message and message handler as usual, then create a Schedule Provider:

add(
      RecurringMessage::every('5 minutes', new MyMessage()),
    );
  }

}

A schedule provider is:

  • a class at the Messenger\ namespace
  • with a #[AsScheduler] class attribute
  • implementing \Symfony\Component\Scheduler\ScheduleProviderInterface
  • implements an getSchedule method. This method returns a message instance and the schedule frequency.

For dependency injection, schedule providers have autowiring enabled.

What would normally be the contents of a hook_cron hook would instead be added to the message handler. The message itself does not need to store any meaningful data.

Instead of intervals via RecurringMessage::every(...), crontab syntax can be used:

\Symfony\Component\Scheduler\RecurringMessage::cron('*/5 * * * *', new MyMessage());

Running the worker

Lastly, schedulers must be run via the consume command with a dedicated transport. The transport name is the schedule ID prefixed by scheduler_. For example, given the scheduler ID my_scheduler_name from above, the transport name will be scheduler_my_scheduler_name.

The command finally becomes: sm messenger:consume scheduler_my_scheduler_name .

Timing

Messages will be dispatched the moment their interval arrives. Normally intervals begin when the worker is initiated, however you can set a point in time to begin interval computation using the \Symfony\Component\Scheduler\RecurringMessage::every   $from parameter.

The worker must be running at the time when a message is scheduled to be sent. The transport won't retroactively catch-up with messages not dispatched during the time it wasn't running.

The next post outlines how to intercept legacy Drupal @QueueWorker items and insert them into the message bus.

Jan 10 2024
Jan 10

The greatest advantage of Symfony Messenger is arguably the ability to send and process messages in a different thread almost immediately. This post covers the worker that powers this functionality.

This post is part 3 in a series about Symfony Messenger.

  1. Introducing Symfony Messenger integrations with Drupal
  2. Symfony Messenger’ message and message handlers, and comparison with @QueueWorker
  3. Real-time: Symfony Messenger’ Consume command and prioritised messages
  4. Automatic message scheduling and replacing hook_cron
  5. Adding real-time processing to QueueWorker plugins
  6. Making Symfony Mailer asynchronous: integration with Symfony Messenger
  7. Displaying notifications when Symfony Messenger messages are processed
  8. Future of Symfony Messenger in Drupal

The Symfony Messenger integration, including the worker, is provided by the SM project. The worker is tasked with listening for messages ready to be dispatched from an asynchronous transport, such as the Doctrine database transport. The worker then re-dispatches the message onto the bus.

Some messages may be added to a bus with no particular execution time, in which case they are serialised by the original thread. Then unserialised almost immediately by the consume command in a different thread.

Since Messenger has the concept of delaying messages until a particular date, the DelayStamp can be utilised. The consume command respects this stamp and will not redispatch a message until the time is right.

The worker is found in the sm console application, rather than Drush. When SM is installed, Composer makes the application available in your bin directory. Typically at /vendor/bin/sm

The command takes one or more transports as the argument. For example if you’re using the Doctrine transport, the command would be:

sm messenger:consume doctrine

Multiple instances of the worker may be run simultaneously to improve throughput.

WorkerThe worker. Messages output to stdout for demonstration purposes.

Prioritised messages

The worker allows you to prioritise the processing of messages by which transport a message was dispatched to. Transport prioritisation is achieved by adding a space separated list of transports as the command argument.

For example, given transports defined in a site-level services.yml file:

parameters:
  sm.transports:
    doctrine:
      dsn: 'doctrine://default?table_name=messenger_messages'
    highpriority:
      dsn: 'doctrine://default?table_name=messenger_messages_high'
    lowpriority:
      dsn: 'doctrine://default?table_name=messenger_messages_low'

In this case, the command would be sm messenger:consume highpriority doctrine lowpriority

Routing from messages to transports must also be configured appropriately. For example, you may decide Email messages are the highest priority. \Symfony\Component\Mailer\Messenger\SendEmailMessage would be mapped to highpriority:

parameters:
  sm.routing:
    Symfony\Component\Mailer\Messenger\SendEmailMessage: highpriority
    Drupal\my_module\LessImportantMessage: lowpriority
    '*': doctrine

More information on routing can be found in the previous post.

The transport a message is sent to may also be overridden on an individual message basis by utilising the Symfony\Component\Messenger\Stamp\TransportNamesStamp stamp. Though for simplicity I’d recommend sticking to standard routing.

Running the CLI application

The sm worker listens and processes messages, and is designed to run forever. A variety of built in flags are included, with the ability to quit when a memory or time limit is reached, or when a certain number of messages are processed or fail. Flags can be combined to process available messages and quit, much like drush queue:run.

Further information on how to use the worker in production can be found in the Consuming Messages (Running the Worker) documentation.

The next post covers Cron and Scheduled messages, a viable replacement to hook_cron.

Jan 09 2024
Jan 09

This post covers Symfony Messenger’s message and message handlers, which are the day to day code developers using features of Symfony Messenger typically will be working on.

This post is part 2 in a series about Symfony Messenger.

  1. Introducing Symfony Messenger integrations with Drupal
  2. Symfony Messenger’ message and message handlers, and comparison with @QueueWorker
  3. Real-time: Symfony Messenger’ Consume command and prioritised messages
  4. Automatic message scheduling and replacing hook_cron
  5. Adding real-time processing to QueueWorker plugins
  6. Making Symfony Mailer asynchronous: integration with Symfony Messenger
  7. Displaying notifications when Symfony Messenger messages are processed
  8. Future of Symfony Messenger in Drupal

The Symfony Messenger integration with Drupal provided by the SM project is the only requirement for the following examples.

A message itself is very flexible, as it doesn't require annotations, attributes, or specific class namespace. It only needs to be a class serialisable by Symfony. For simplicity, don’t include any complex objects like Drupal entities. Opt to store entity UUIDs instead.

At its most simple implementation, a message handler is:

  • a class at the Messenger\ namespace
  • with a #[AsMessageHandler] class attribute
  • an __invoke method. Where its first argument is an argument typehinted with the message class.

Example message and message handler:

namespace Drupal\my_module;

final class MyMessage {

  public function __construct(public string $foo) {}

}
namespace Drupal\my_module\Messenger;

use Drupal\Core\State\StateInterface;
use Symfony\Component\Messenger\Attribute\AsMessageHandler;

#[AsMessageHandler]
final class MyMessageHandler {

  public function __construct(StateInterface $state) {}

  public function __invoke(\Drupal\my_module\MyMessage $message): void {
    // Do something with $message.
    $this->state->set('storage', $message->foo);
  }

}

And dispatch code:

$bus = \Drupal::service(\Symfony\Component\Messenger\MessageBusInterface::class);
$bus->dispatch(new MyMessage(foo: 'bar'));

Non-autowirable dependency injection

Message handlers use autowiring by default, so you don’t need ContainerFactoryPluginInterface and friends.

In the rare case that dependencies are not autowirable, you can opt to define a message handler as a tagged service instead of a class with #[AsMessageHandler] attribute and define dependencies explicitly. The same __invoke and argument typehinting semantics apply.

services:
  my_module.my_message_handler:
    class: Drupal\my_module\Messenger\MyMessageHandler
	arguments:
      - '@my_module.myservice'
    tags:
      - { name: messenger.message_handler }

Comparison with Legacy Drupal Queues

Typically, when setting up a Drupal queue, you’ll be putting together a rigid class with a verbose annotation. When compared to the functionality of the messenger and handler above, the equivalent @QueueWorker looks like:

namespace Drupal\my_module\Plugin\QueueWorker;

use Drupal\Core\Plugin\ContainerFactoryPluginInterface;
use Drupal\Core\Queue\QueueWorkerBase;
use Drupal\Core\State\StateInterface;
use Symfony\Component\DependencyInjection\ContainerInterface;

/**
 * @QueueWorker(
 *   id = "my_module_queue",
 *   title = @Translation("My Module Queue"),
 *   cron = {"time" = 60}
 * )
 */
final class MyModuleQueue extends QueueWorkerBase implements ContainerFactoryPluginInterface {

  private function __construct(
    array $configuration,
    $plugin_id,
    $plugin_definition,
    private StateInterface $state,
  ) {
    parent::__construct($configuration, $plugin_id, $plugin_definition);
  }

  public static function create(ContainerInterface $container, array $configuration, $plugin_id, $plugin_definition): static {
    return new static(
      $configuration,
      $plugin_id,
      $plugin_definition,
      $container->get('state'),
    );
  }

  public function processItem(mixed $data): void {
    // Do something with $data.
    $this->state->set('storage', $data['foo']);
  }

}

And dispatch code

\Drupal::service('queue')
  ->get('my_module_queue')
  ->createItem(['foo' => 'bar']);

Notice the hard-to-remember annotation, boilerplate dependency injection, and mixed-type processItem argument $data . In comparison, Symfony Messenger messages and message handlers are easier to use thanks to PHP attributes.

Routing messages to transports

All messages will be handled synchronously by default. To route messages to specific transports, routing needs to be configured.

Behind the scenes, routing is a simple map of class/namespaces to transports defined in a container parameter.

parameters:
  sm.routing:
    Drupal\my_module\MyMessage: doctrine
    Drupal\my_module\MyMessage2: synchronous
    'Drupal\my_module\*': doctrine
    '*': doctrine

Keys are either verbatim class names, partial class namespace followed by asterisk, or a standalone asterisk indicating the fallback. The values are the machine name of a transport. SM includes a synchronous transport out of the box, which indicates messages are handled in the same thread as it is dispatched. The doctrine database transport is available as a separate module. I’d recommend always using an asynchronous transport like Doctrine.

Routing configuration UI

SM includes a configuration UI submodule that allows site builders to build a routing map without needing to mess with YAML. The container parameter is set automatically as soon as the form is saved.

Routing configuration UI

Advanced usage of messages and handlers

Adding stamps to messages

A common use case for adding stamps to a message is to delay the message for an amount of time. A stamp is created and attached to the envelope containing the message to be processed:

$envelope = new Envelope(
  message: new MyMessage(foo: 'bar'),
  stamps: [\Symfony\Component\Messenger\Stamp\DelayStamp::delayUntil(new \DateTimeImmutable('tomorrow'))],
);
$bus = \Drupal::service(\Symfony\Component\Messenger\MessageBusInterface::class);
$bus->dispatch($envelope);

Multiple handlers per message

For more advanced use cases, multiple handlers can be configured for a message. Useful if you want to listen for messages that you do not own. For example, additional handling of the Symfony Mailer email message:

namespace Drupal\my_module\Messenger;

use Symfony\Component\Messenger\Attribute\AsMessageHandler;
use Drupal\Core\State\StateInterface;

#[AsMessageHandler]
final class MyMessageHandler {

  public function __construct(StateInterface $state) {}

  public function __invoke(\Symfony\Component\Mailer\Messenger\SendEmailMessage $message): void {
    $this->state->set(
      'sent_emails_counter', 
      $this->state->get('sent_emails_counter', 0) + 1,
    );
  }

}

Both this custom handler and the original \Symfony\Component\Mailer\Messenger\MessageHandler::__invoke handler will be invoked.

Multiple messages per handler

Handlers can be configured to handle multiple message types. Instead of using the #[AsMessageHandler] attribute on the class, use it with methods.

namespace Drupal\my_module\Messenger;

use Drupal\Core\State\StateInterface;
use Symfony\Component\Messenger\Attribute\AsMessageHandler;

final class MyMessageHandler {

  #[AsMessageHandler]
  public function myHandler1(\Drupal\my_module\MyMessage $message): void {
    // Do something with $message.
  }

  #[AsMessageHandler]
  public function myHandler2(\Drupal\my_module\MyMessage2 $message2): void {
    // Do something with $message2.
  }

}

The next post covers the worker, the heart of messenger’s real-time capabilities.

Jan 08 2024
Jan 08

Part one in a series of posts introducing Symfony Messenger, its ecosystem, and unique Drupal integrations to Drupal developers.

The SM project brings Symfony Messenger and its ecosystem together with Drupal.

Symfony Messenger is a powerful alternative to Drupal’s @QueueWorker, enabling real-time or precise execution of scheduled tasks. It’s also a viable replacement for hook_cron enabling you to schedule dispatch and processing of messages according to crontab rules, again in real-time, rather than waiting for server-invoked or request-termination cron.

Messenger can be used as a user-friendly alternative to Batch API, as the user's browser is not blocked, while integrations such as Toasty can be programmed to notify when the last of a “batch” of messages completes, communicating to the user via user-interface toasts.

The Drupal integration includes additional niceties, such as intercepting legacy QueueWorkers for processing data through the Symfony Messenger bus, and end-user UI notifying a user when tasks relevant to them have been processed.

During this and the following series of posts, we’ll be exploring the benefits of real-time processing and user-friendly features that improve the overall experience and outputs.

WorkerThe workerToastyToasty displaying notifications.

Messenger

First up, we’ll cover the main features of Symfony Messenger and how it works.

As a developer working with Messenger, the most frequent task is to construct message and associated message handlers. A message holds data, while a message handler processes the associated data.

A message is inserted into the bus. The bus executes a series of middleware in order, each of which can view and modify the message.

If a transport is configured, the message may be captured and stored for processing later.

Typically the bus, middleware, and transports are configured in advance and rarely changed. Message and message handlers are introduced often without needing other configuration.

  • Message — an arbitary PHP object, it must be serialisable.
  • Message handler — a class that takes action based on the message it is given. Typically a message handler is designed to consume one type of message.
  • Middleware — code that takes action on all message types, and has access to the containing envelope and stamps.
  • Bus — a series of middleware in a particular order. There is a default bus, and a default set of middleware.
  • Envelope — an envelope contains a single message, and it may have many stamps. A message always has an envelope.
  • Stamp — a piece of metadata associated with an envelope. The most common use case is to track whether a middleware has already operated on the envelope. Useful when a transport re-runs a message through the bus after unserialisation. Another useful stamp is one to set the date and time for when a message should be processed.
  • Transport — a transport comprises a receiver and sender. In the case of the doctrine database transport, its sender will serialise the message and store it in the database. The receiver will listen for messages ready to be sent, and then unserialise them.
  • Worker — a command line application responsible for unserialising messages immediately, or at a scheduled time in the future. Messages are inserted into the bus for processing.

The stars of the show are buses. One bus is ready out of the box, which comprises a series of ordered middleware. A message is dispatched into a bus, where each middleware has the opportunity to view and modify the message (and its envelope). It's unlikely you’ll need to think about middleware, as the default set may already be the perfect combination.

When a message is dispatched to a bus, you can choose to wrap it in an envelope and apply stamps like the DelayStamp . A message will always be wrapped in an envelope if you don’t do it explicitly.

Buses have a series of default middleware. The main middleware to note are the transport and message handler middlewares. When a transport is configured for messenger, the transport middleware will capture the message, serialise it, and store it somewhere. For example, a database in the case of the doctrine transport. Any middleware after the transport middleware are not executed, for now.

When running the worker, you are opting to choose which bus and transport to run. The command will listen for messages as they are stored, and if the time is right, messages will be unserialised and inserted into the bus. The message will begin its journey yet again, iterating through all the middlewares from the beginning. When the transport middleware is hit, it will detect the message has already been in the transport to prevent recursion. This is done by checking the ReceivedStamp stamp added to the message envelope.

Transports: synchronous, asynchronous

Out of the box, when a message is dispatched into the bus in a CLI or web request, it will be processed synchronously. All middleware will operate on the message in a set order, including the message handler middleware.

The greatest advantage of using Messenger is the ability to asynchronously handle messages outside of the thread they were originally dispatched. That is: asynchronously. This can be useful for improving the web request response times and reducing the memory usage and limit of web requests (allowing for more FPM threads on a machine). Bulky business operations that would typically, or should be, constrained by the limits of the web thread have more breathing room. A CLI runner/container may be set up with a little more memory and processing capability with the explicit direction to listen for messages and handle them in real-time, either as soon as possible or as scheduled.

Upcoming posts in this series will dive into aspects of Symfony Messenger and SM:

The next post covers the implementation of a message and message handler, and a comparison with Drupal core’s @QueueWorker plugins.

Dec 18 2023
Dec 18

Our client, ServiceNSW, is a committed open-source contributor, working closely with us to improve their customer experience while sharing these advances with the Drupal community.

How is client-backed contribution made possible? 

It helps when you work with a client that understands the value of contributing development time back to the Drupal community. ServiceNSW are members of Drupal and have co-presented with us at DrupalSouth, so they’re truly invested.

ServiceNSW Contributions

Solutions to client challenges, such as core patches or contributor modules, require upfront work. Doing this in a community setting is far more beneficial, allowing everyone to contribute and further improve it. That’s why SNSW recognises the future benefits of investing in the work done now. 

We also put a lot of focus on performance and security. This means SNSW receives the latest upgrades for both Drupal core and contributed modules, helping move issues along and ensuring they have the latest and greatest, including being one of our first clients to move to Drupal 10.1. In fact, during the lead-up to the release of Drupal 10.1, we committed over a dozen large core issues in collaboration with the SNSW development team.

The patches we worked on pre Drupal 10.1 upgrade

Over a period of three months, in the lead-up to Drupal 10.1, we targeted patches that were large and/or conflicted with other patches we were using. These were becoming increasingly hard to maintain. SNSW understood that these fixes would be a net gain in developer productivity and an improvement for the community.

  1. Issue #3198868: Add delay to queue suspend
  2. Issue #2867001: Don't treat suspending of a queue as erroneous
  3. Issue #2745179: Uncaught exception in link formatter if a link field has malformed data (a 7-year-old bug!)
  4. Issue #3059026: Catch and handle exceptions in PathFieldItemList
  5. Issue #3311595: Html::transformRootRelativeUrlsToAbsolute() replaces "\r\n" with " \n"
  6. Issue #2859042: Impossible to update an entity revision if the field value you are updating matches the default revision
  7. Issue #2791693: Remove sample date from date field error message and title attribute (another 7 year old one!)
  8. Issue #2831233: Field tokens for "historical data" fields (revisions) contain a hyphen, breaking twig templates and throwing an assertion error
  9. Issue #3007424: Multiple usages of FieldPluginBase::getEntity do not check for NULL, leading to WSOD

Revisions everywhere!

One of our largest pieces of work was Implementing a generic revision UI

Originally opened in 2014, this issue paved the way for one of the most sought-after features from our client - having Revisions for all entity types and a consistent user experience for them.

This was originally committed to the SNSW codebase in July of 2018 using this patch when we added a Block Content Type for a Notice feature on the website.

~3.5 years, ~250 comments, and a huge effort from PreviousNext and SNSW developers, along with many other community members and it was committed to 10.1.x.

This spawned several other core issues for other entity types:

  • Block Content - This was also committed to 10.1 alpha.
  • Media - which is committed and will be available in 10.2.0!
  • Taxonomy terms - which is currently RTBC and looking promising for 10.3!

Plus contributed projects to extend contributed module entity types with revisioning support, such as Micro-content Revision UI.

The patches committed to Drupal 10.1 that we were able to remove

With all this pre-work, we were well positioned when the 10.1 upgrade came around. As you may have noticed, we like to get the ball rolling early, and we had a Pull Request going for the 10.1 upgrade in late June (the day 10.1.0 was released, in fact). This allowed us to figure out which modules needed help, what patches needed re-rolling, and to catch any bugs early.

It wasn't until mid-August when that PR was finally merged, with multiple developers touching it every now and then, when there was some movement.

Here's a full list of Drupal core patches we were able to remove, thanks to the contributions from SNSW.

  1. Issue #2350939: Implement a generic revision UI
  2. Issue #2809291: Add "edit block $type" permissions
  3. Issue #1984588: Add Block Content revision UI
  4. Issue #3315042: Remaining tasks for "edit block $type" permissions
  5. Issue #2859042: Impossible to update an entity revision if the field value you are updating matches the default revision
  6. Issue #3311595: Html::transformRootRelativeUrlsToAbsolute() replaces "\r\n" with " \n"
  7. Issue #3007424: Multiple usages of FieldPluginBase::getEntity do not check for NULL, leading to WSOD
  8. Issue #2831233: Field tokens for "historical data" fields (revisions) contain a hyphen, breaking twig templates and throwing an assertion error
  9. Issue #3059955: It is possible to overflow the number of items allowed in Media Library
  10. Issue #3123666: Custom classes for pager links do not work with Claro theme
  11. Issue #2867001: Dont treat suspending of a queue as erroneous
  12. Issue #3198868: Add delay to queue suspend
  13. Issue #2984504: Access to 'Reset to alphabetical' denied for users without administer permission
  14. Issue #3309157: RevisionLogInterface is typehinted as always returning entity/ids, but cannot guarantee set/existing values
  15. Issue #2634022: ViewsPluginInterface::create() inherits from nothing, breaking PHPStan-strict
  16. Issue #3349507: DateTimePlus::createFromDateTime should accept DateTimeInterface
Drupal 10.1 Patch changes

Service NSW, a true Drupal partner

Service NSW has (at the time of writing this post) contributed to 19 Drupal core issues that were committed over the past three months.

We look forward to continuing this incredible partnership and contributing in the coming months!

Nov 27 2023
Nov 27

We're proud to announce the release of vite-plugin-twig-drupal, a plugin for Vite that we hope will improve your workflow for front-end development with Drupal.

The problem space

You're working with Twig in a styleguide-driven-development process. You're writing isolated components that consist of CSS, Twig and JavaScript. You want to be able to use Twig to render your components for Storybook. You want fast refresh with Vite. You want Twig embeds, includes and extends to work. You want to use Drupal-specific twig features like create_attributes etc. You want compilation of PostCSS and SASS to CSS. You want Hot Module Reloading (HMR) so that you can see how your components look without needing to endlessly refresh.

Enter vite-plugin-twig-drupal

The Vite plugin Twig Drupal is a Vite plugin based on Twig JS for compiling Twig-based components into a JavaScript function so that they can be used as components with Storybook. It allows you to import Twig files into your story as though they are JavaScript files.

Comparison to other solutions

  • Vite plugin twig loader doesn't handle nested includes/embeds/extends. These are a fairly crucial feature of Twig when building a component library as they allow re-use and DRY principles
  • Components library server requires you to have a running Drupal site. Whilst this ensures your Twig output is identical to that of Drupal (because Drupal is doing the rendering), it is a bit more involved to setup. If you're going to use single directory components or a similar Drupal module like UI patterns then this may be a better option for you.

Installation

This module is distributed via npm, which is bundled with node and should be installed as one of your project's devDependencies:

npm install --save-dev vite-plugin-twig-drupal

You then need to configure your vite.config.js.

import { defineConfig } from "vite"
import twig from 'vite-plugin-twig-drupal';
import { join } from "node:path"
export default defineConfig({
  plugins: [
    // Other vite plugins.
    twig({
      namespaces: {
        components: join(__dirname, "/path/to/your/components"),
        // Other namespaces as required.
      },
      // Optional if you are using React storybook renderer. The default is 'html' and works with storybook's html
      // renderer.
      // framework: 'react' 
    }),
    // Other vite plugins.
  ],
})

With this config in place, you should be able to import Twig files into your story files.

Examples

To make use of a Twig file as a Storybook component, just import it. The result is a component you can pass to Storybook or use as a function for more complex stories.

// stories/Button.stories.js
// Button will be a Javascript function that accepts variables for the twig template.
import Button from './button.twig';
// Import stylesheets, this could be a sass or postcss file too.
import './path/to/button.css';
// You may also have JavaScript for the component.
import './path/to/some/javascript/button.js';
export default {
  title: 'Components/Button',
  tags: ['autodocs'],
  argTypes: {
    title: {
      control: { type: 'text' },
    },
    modifier: {
      control: { type: 'select' },
      options: ['primary', 'secondary', 'tertiary'],
    },
  },
  // Just pass along the imported variable.
  component: Button,
};
// Set default variables in the story.
export const Default = {
  args: { title: 'Click me' },
};
export const Primary = {
  args: { title: 'Click me', modifier: 'primary' },
};
// Advanced example.
export const ButtonStrip = {
  name: 'Button group',
  render: () => `
    ${Button({title: 'Button 1', modifier: 'primary'})} 
    ${Button({title: 'Button 2', modifier: 'secondary'})}
  `
}

Here's how that might look in Storybook (example from the Admin UI Initiative storybook)

Screenshot of a button in storybook

Dealing with Drupal.behaviors

In cases where the JavaScript you import into your story file uses a Drupal behavior, you'll likely need some additional code in your Storybook configuration to handle firing the behaviors. Here at PreviousNext, we prefer to use a loadOnReady wrapper, which works with and without Drupal. However, if you're just using Drupal.behaviors something like this in your Storybook config in main.js (or main.ts) will handle firing the behaviors.

const config = {
  // ... existing config
  previewBody: (body) => `
    
  ${body}
  `
  // ... more config
}

Give it a try

We're looking forward to using this plugin in client projects and are excited about the other possibilities Storybook provides us with, such as interaction and accessibility testing.

Thanks to early testers in the community, such as Ivan Berdinsky and Sean Blommaert, who've already submitted some issues to the github queue. We're really happy to see it in use in the Admin Initiative's work on a new toolbar.

Give it a try, and let us know what you think.

Nov 06 2023
Nov 06

Since Drupalcon Pittsburgh, we've been working on a decoupled Layout Builder for Drupal. We've reached the end of the statement of work, so let's recap the requirements and discuss what’s next.

Requirement 1: Create an npm package for developing decoupled components

Issue: https://www.drupal.org/project/decoupled_lb/issues/3375410

Taking inspiration from @wordpress/scripts, develop an npm package that has tooling for building and packaging decoupled components.

The MVP of this will include:
 • a script for building JS and CSS

Out of scope but for future consideration:
• linting
• a script for checking license compatibility

Release this as a package on npm in the @drupal namespace.

Delivery

Not only does the developed package support building CSS and JS via drupal-scripts build, but we also added support for initialising a new project and code generation with drupal-scripts init and drupal-scripts generate.

Requirement 2: Determine how to deal with externals in Webpack

Issue: https://www.drupal.org/project/decoupled_lb/issues/3375412 

Wordpress has @wordpress/dependency-extraction-webpack-plugin, a webpack plugin that extracts dependencies that are provided by Wordpress.

We don't have enough budget to go down the full 'roll our own' approach and it may be overkill for now. Instead, derive configuration for Webpack externals that ensures components don't bundle dependencies Drupal will guarantee to be present.

This config will likely be part of the package at 1)

Out of scope but for future consideration:
Create our own plugin like Wordpress with some Drupal specific replacements

Delivery

As per our previous post, we used Vite externals and then wrote a module to add support for Import maps to Drupal. We wrote a new version of the React module that works with import maps.

Requirement 3: Provide a registry of React Layout Components

Issue: https://www.drupal.org/project/decoupled_lb/issues/3375413 

Create a simple layout registry and registration process for layout plugins. Create an MVP of some common layout plugins:
One column
Two column
Grid layout

Out of scope but for future consideration:
• Attempt to use the AST-generated Twig APIs to autogenerate React components that provide the markup for a layout plugin.
• Write a symfony/console command that can take the ID of a layout plugin and generate an equivalent React component.
• Bundle this as a part of a new experimental module.

Delivery

We wrote a context provider to create a layout registry and a way to pass this configuration from Drupal to React. We wrote React implementations of one and two column layouts.

Requirement 4: Persistence layer

Issue: https://www.drupal.org/project/decoupled_lb/issues/3375414 

Create an OpenAPI spec for the persistence layer. The MVP will include the following endpoints:
Choose section
Add section
Remove section
Choose block
Add block
Update block
Remove block

These will mirror existing HTML endpoints from layout builder\

Create JSON equivalent versions of these routes. Some of these routes will exist to serve information for the decoupled layout builder (e.g. choose block/section) whilst others will exist to perform mutations of the configured layout on the Drupal side.

Out of scope but for future consideration:
Configure section
Move block

Delivery

We wrote an API specification for the needed endpoints. We didn't need add section, remove section, add block, update block and remove block, as all these are handled client side. We did need additional endpoints for saving, discarding and reverting.

All of these endpoints are implemented, with test coverage in the Decoupled Layout Builder API module

Requirement 5: Create a block discovery mechanism to associate a block plugin with a React component

Issue: https://www.drupal.org/project/decoupled_lb/issues/3375416 

Create a registry component that can translate block plugin IDs to these components. Allow modules to register a component as a React version of a Drupal Block plugin so the decoupled layout builder knows how to render it.
This will take inspiration from registerBlockType in @wordpress/block

Create a React version of the InlineBlock block plugin as a proof of concept
Out of scope but for future consideration:
Convert other Block plugins to React

Delivery

We wrote a context provider to provide a block registry and a way to pass configuration of this from Drupal to React. We wrote a React implementation of the inline block, which uses formatter and widget components. This component is quite complex as it needs to load entity view and form display configuration from Drupal. We have a working version of the Field block plugin too.

Requirement 6: Create a Drupal-rendered block fallback React component

Issue: https://www.drupal.org/project/decoupled_lb/issues/3375416 (could have)

Create a fallback React block plugin component that can make a request to Drupal for a rendered previews. Extend the persistence layer to add another Route for this.

Out of scope but for future consideration:
• Support a legacy configuration form for these blocks as well

Delivery

We ship rendered 'fallback' versions of each block in the API. This saves the individual plugins from needing to make many single requests and ensures the best performance as loading happens once upfront.

Issue: https://www.drupal.org/project/decoupled_lb/issues/3375417 

Identify a list of initial widget plugins that make sense to convert to React as an initial proof of concept. Create a React version of these widget plugins making use of @wordpress/components.

Create a registry component that can translate widget plugin IDs to these components. Allow modules to register a component as a React version of a Drupal widget plugin so the decoupled layout builder knows how to use it to edit an inline block.

Out of scope but for future consideration:
A React powered media library widget

Delivery

We wrote a context provider to create a widget registry and a way to pass the configuration of this from Drupal to React. We wrote a React implementation of the text area widget, which uses an inline CKEditor for in-place editing with rich formatting. We didn't use WordPress components - more on this below.

Requirement 8: Create a discovery mechanism to associate a formatter plugin with a React component

Issue: https://www.drupal.org/project/decoupled_lb/issues/3375418 

Identify a list of initial formatter plugins that make sense to convert to React as an initial proof of concept. Create a React version of these formatter plugins.

Create a registry component that can translate formatter plugin IDs to these components. Allow modules to register a component as a React version of a Drupal formatter plugin so the decoupled layout builder knows how to render a preview of an inline block.

Out of scope but for future consideration:
• Devise a way for themes to modify the markup of these components, possibly by registering their own replacements in the registry.

Delivery

We wrote a context provider to create a formatter registry and a way to pass configuration of this from Drupal to React. We wrote a React implementation of the text default and string default plugins.

Requirement 9: React context provider/hooks

Issue: https://www.drupal.org/project/decoupled_lb/issues/3375420 

To allow blocks/layouts/widgets/formatters to access this data without passing down props from each component, create a series of context providers and hooks to allow components to access and update this data.

We can take inspiration from e.g. useBlockProps in @wordpress/block-editor

Delivery

We didn't end up using hooks and context providers for this. Instead, we used a Redux toolkit - a mature state management solution designed for large complex applications. There are various selector functions made available so components can select state as well as reducers for updating state. The use of Redux means we can get things like undo and redo without much effort (more on that later).

Requirement 10: Layout builder component

Issue: https://www.drupal.org/project/decoupled_lb/issues/3375421 

Add a component which manages the following UX behaviours:
Adding a section (layout plugin)
Adding an inline block
Editing an inline block

It is hoped that we can take inspiration from existing open-source components in this space including @wordpress/components and builder.io.

Out of scope but for future consideration:
• Editing (Configuring) a section
Moving a block

Delivery

Here we completed the following:

  • Adding a section
  • Adding a block
  • Editing a block
  • Moving a block
  • Moving a section
  • Configuring a section
  • Configuring a block
  • Viewing an outline of the layout, including support for moving components/sections
  • Saving a layout to Drupal, including autosave

You can try this for yourself in our interactive storybook.

Where to next?

The end of the Pitchburgh funding doesn't mean the end of the road. As we progressed through the project, we were building a backlog of out-of-scope items we discovered. A key aspect of this project was the evaluation of the feasibility of a decoupled layout builder. This work will also feed into the newly announced Page Building Experience initiative as one possible solution.

We tracked all the out-of-scope items in a backlog and have now added issues on Drupal.org for both the npm packages and the Drupal module.

We plan to keep working through these as part of our ongoing commitment to open-source. Do you have an interest in helping us? If so, reach out to me in the #layouts channel on Drupal Slack. There's a real mix of technology at use here - not just PHP but React, Typescript and CSS. 

We definitely need some help making the UI look nicer and more closely aligned with the Claro look and feel, as well as the work being done by the Admin UI initiative. If you've been looking for a way to get involved in Drupal contribution without a heavy understanding of Drupal's internals - this might be the ideal place to get started.

About Drupal Sun

Drupal Sun is an Evolving Web project. It allows you to:

  • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
  • Facet based on tags, author, or feed
  • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
  • View the entire article text inline, or in the context of the site where it was created

See the blog post at Evolving Web

Evolving Web