Upgrade Your Drupal Skills

We trained 1,000+ Drupal Developers over the last decade.

See Advanced Courses NAH, I know Enough
Jan 30 2024
Jan 30

As of writing this blog post in January 2024, LakeDrops is 13 years old, has 4 team members, maintains 60 Drupal modules on drupal.org and gets constantly credited for contributions. All this is happening purposefully, since LakeDrops allocates 20% of its resources towards Drupal contributions. That's one full day from each team member every single week.

A lot of that goes into code contribution, but that's not all:

  • writing documentation
  • organising local meetups
  • sharing knowledge as speakers at events
  • providing support, e.g. on Drupal Slack
  • writing blog posts
  • recording video tutorials

Those and more activities created a reputation for which the Drupal Certified Partner status in silver has now been assigned, and everybody in and around LakeDrops is celebrating this.

Why do we contribute?

Without Drupal, LakeDrops wouldn't be a thing. None of LakeDrops' projects would have been possible. Likewise, Drupal will be the foundation of every future project as well. For that selfish reason, it is the most logical business decision to do whatever the company can afford, to help the Drupal project to do well. At the same time, Drupal isn't much without its outstanding community, so that being an active part of it is not only rewarding but also fun, most of the time.

However, a project and community of that size can hardly sustain itself without governance. And that's why we're grateful for the Drupal Association (DA) as an organisation and every single person that's being involved in the DA to provide the framework for all of that. For many years, LakeDrops has advocated for the DA and supported them financially.

How can we afford 20% contribution?

We get this question a lot, and our response often seems to be a surprise. This is certainly not applicable to every organisation, but the maths for LakeDrops is simple: the cost for sales and marketing is actually zero, simply because it doesn't exist. Instead, we spend the equivalent "amount" in Drupal contribution. And that turns into a double win, just for ourselves: not only does this help the Drupal project, it also drives our own business; customers find us, we don't have to find our customers.

This might be an oversimplification, yes. Still, it has worked for so many years already, and there's no sign of getting worse. We hope this can work for others too, maybe this post is encouraging, and we'd be delighted to discuss this in more detail if you were interested. The more we all do together, the bigger the benefit for all of us.

Apr 17 2023
Apr 17

Drupal 7 is going End-of-Life eventually. While this had been originally scheduled for a date which has passed for a couple of years already, support got extended and may well be finished in November this year - although that may still get extended again for another year or two. That's why there are plenty of discussions in and around the Drupal community of what should be done to all those still existing and mostly deeply necessary Drupal 7 sites. The options are:

  • Upgrading to Drupal 10
  • Staying on Drupal 7
  • Switching to another platform, e.g. Backdrop, WordPress or others

Why is the first option certainly the best?

Let's face it, every Drupal site is hosted on that platform for good reasons. Different reasons for each of them. While the rich feature set is important to most, the performance, its scalability and security are must-haves not only for enterprise websites. A web presence of any size and purpose benefits from those aspects and many others, this blog post is not about to enumerate them all. But we know for sure, that hundreds of thousands of Drupal 7 sites haven't moved away from the platform even after that many years. Simply because Drupal is what suits them best. That's why moving away from Drupal is rarely a great idea.

So, why are so many still holding back on updating to the modern Drupal framework?

One out of these three reasons is always brought up - sometimes even all of them at once:

  • Missing resources, either budget or people power
  • Complexity of the modern technology
  • Missing functionality due to not yet updated modules

There is a lot to say about the resource constraints. While I can't speak for every individual case out there, what's very common these days is the unfortunate focus on short-term indicators. In the context of an upgrade to Drupal 10 that approach misses out on important gains, the composable architecture of the modern Drupal platform in particular. In other words, the initial jump might be more challenging. But from then on, the platform is maintainable and easily upgradable even across major updates for decades to come. Drupal has demonstrated this with already 2 major updates from 8 to 9 and then 9 to 10. Not only has the Drupal community kept up with an easy upgrade path, their experience helps them to make it even easier for each future generation of Drupal.

Having said that, the same infrastructure paradigm-shift is the reason why this composer-based dependency management is often perceived to be complicated. Where in reality the opposite is true. I'm not saying there isn't another learning curve ahead of us. There is one. And always will be. Not only in this post-Drupal-7 era, but everywhere in life. And that's a good thing because if we didn't progress, we would be falling behind continuously. Not only that, the technical debt of our old technology will cause hidden costs over and over again. In other words, not having a budget for a Drupal 7 to 10 update makes me wonder where the budget for the continued use of either outdated, or in the case of switching the platform, less capable technology should come from? While the ongoing effort for running the modern Drupal platform declines immediately, the cost of not upgrading increases exponentially as long as the decision is not being taken.

This debate is difficult, I know. And I feel sorry for alluding to this, as it may be challenging for many. Therefore, let's switch gears and have a look into the third reason why so many Drupal 7 sites haven't updated yet: missing functionality due to not yet updated modules.

Does Drupal 10 provide everything needed?

Drupal core has never been faster, it has never been more stable, it was never easier to get started while at the same time being the most user-friendly Drupal we've ever seen. And that describes just the status quo. So many initiatives are working hard, day and night, in moving forward on all technical levels of Drupal. So, yes, Drupal 10 is ready for prime-time.

However, there is probably not a single Drupal site that works without any additional modules that are not part of its core package. And from the perspective of a Drupal 7 site owner or maintainer, it is likely that a number of used modules on Drupal 7 don't seem to be available for Drupal 10. Even worse, it looks like as if some of those modules haven't even tried or declared officially that they won't ever upgrade to modern Drupal.

As a Drupal service provider, we at LakeDrops have been in that situation with many customer projects over the past years as well. What we've learned, though, there is always a solution when upgrading. Sometimes, it's not the same module in Drupal 10 that used to do the job back in the old days. Just like with spring-cleaning, moving to a modern technical platform comes with some re-structuring and re-thinking on how certain tasks should be done.

This is particularly true for the ECA module, which we've helped to architect, develop and continue to maintain, together with a growing team of Drupal community members. It's an event driven no-code solution, which allows configuring the execution of any Drupal provided action under configurable conditions. Hence, the name ECA, it stands for Events - Conditions - Actions. Very much like the really famous Rules module in Drupal 7, ECA allows the site builder to configure the behaviour of their Drupal site in literally all areas without having to hire programmers. And ECA does so much more that we even receive "love letters" from formerly frustrated Drupal 7 users who couldn't seem to find what they needed in Drupal 10, until they've found ECA. More to that in a minute.

How can ECA help to upgrade your Drupal 7 site?

The Rules module is one if not the most popular Drupal 7 module. As of writing in Spring of 2023, over 150.000 Drupal sites are using it, of which more than 90% are still on Drupal 7. Without judging, those users don't feel comfortable with the Drupal 9 or Drupal 10 version of that module and therefore got locked into their Drupal 7 environment. But that's not all, a significant number of mostly less known modules are no longer maintained and don't seem to have alternative solutions.

With ECA, all Drupal 7 users can shift their showstoppers aside and get onto Drupal 10. It's why we started the ECA project in the first place a couple of years ago. Most of our Drupal 7 customers couldn't afford to stay on Drupal 7, and we had to provide a solution to move forward. And ECA stood up not only for us and our customers, it does grow in popularity in seemingly all areas where Drupal is being utilized. It's in production for huge and extremely complex enterprise grade web applications as well as for large and medium commercial websites, online shops, intranets and portals. ECA also loves the small ones, believe it or not, as it is the solution for all those "tiny" requirements on personal blogs or other small websites as much as for the challenging tasks.

No doubt, while ECA provides the site owners with access to all the power of Drupal from within the admin interface, it has proven to be solid, maintainable, yet non-intrusive to the rest of the application or website. But is all that technical excellence convincing enough to get off the island and come to Drupal 10? No, there needs to be more.

Right from the beginning, the ECA team has been transparent, approachable and welcoming to other maintainers and users likewise. This lead to a flourish community of people who are interested in, working with, developing for, or otherwise get involved with ECA. As already mentioned, there are so many users who already managed to upgrade from Drupal 7 to 10 or are right in the middle of that process. When reading their comments, e.g. in the #ECA channel of Drupal slack, it becomes obvious how much burden ECA has taken off their shoulders.

Are you next?

Nov 22 2022
Nov 22

Summary

The complete power of Drupal is also accessible to all site builders with ECA and BPMN UI to automate processes and customize Drupal's behaviour (hooks) without programming.

Project description

ECA is a Drupal community project, started by LakeDrops, which rebooted the enormously popular Rules module from the pre-Symfony era and developed a technologically state-of-the-art, production-ready and scalable solution within 11 months. Shortly after the start of the project, OpenCampus GmbH from Munich joined the project, as they themselves had an almost congruent plan for their own products with the already existing prototype of ECA and wanted to start.

The videos can be found here: https://tube.tchncs.de/c/eca/videos

The Drupal experts from LakeDrops and OpenCampus, supplemented by other freelancers from the Drupal community, worked with high-pressure and clear goals (see next section) in mind almost day and night and were able to release the long-awaited version 1.0.0 in July 2022.

Countless talks and workshops at Drupal Meetups and Camps all over the world already showed during the development of ECA and its add-on modules how eagerly the whole Drupal community was waiting for exactly this solution. And barely 2 months after the official launch, hundreds of Drupal sites are already using this project. The feedback in the issue queue, on Slack and other channels is overwhelming and consistently positive.

Project goals and deliverables

The original objective arose from a customer project to migrate a very complex intranet from Drupal 7 to Drupal 9. All Drupal 7 components that were still needed were also available for Drupal 9 - only the Rules module, which was used extremely extensively in Drupal 7, was still in an alpha state that did not inspire much confidence.

After repeated analysis of the Rules module, we came to the conclusion together with the customer that Rules could not be completed for Drupal 9 with the present architecture. Instead, an entirely new approach based on the now existing Symfony framework was needed.

Another objective was to separate the processor from the user interface. Thus, the processor needs its own config format, from which it receives the instructions to be executed in the background during live operation of the Drupal site. The performance plays the decisive role. This should in no case be worse than that of a custom module to be written alternatively.

The user interface should be developed separately, if possible, and thus be interchangeable. The UI should implement a standard that is independent of Drupal, so that as many stakeholders as possible can be involved in the process of configuration, review and maintenance of the rules and regulations at the customer. One such standard for describing business processes is BPMN, which is globally recognized and for which there are extensive tools that can be integrated.

Long-term maintainability and easy extensibility were important concerns for both the customer and the development team. In this context, one of the most important architectural decisions was made: ECA must not have any dependencies other than Drupal Core and should use all existing APIs, i.e. not re-develop what already exists.

All of these targets were met in just 11 months of development:

  • Provide an upgrade path for all Drupal 7 sites that previously could not upgrade due to missing Rules module
  • State-of-the-art software architecture in the Symfony context
  • Best possible test coverage
  • Full feature set for all events, conditions and actions available in Drupal
  • The ECA module has no dependency other than Drupal Core
  • Around ECA there is a fast-growing ecosystem already a few weeks after the 1.0.0 release with further modules, which integrate ECA with important contrib modules and make them available in ECA
  • For the UI, a fully functional BPMN modeller could be integrated into Drupal with bpmn.io, which saves all its models in the ECA config format
  • Even a second modeller is already available, which works in a very simplified way using the Drupal Form API, thus supporting the claim that the ECA processor is not only completely separated from the UI, but that the UI is also interchangeable

In addition to this complete goal achievement, it also became clear to us during the project that we had achieved something else unintended with ECA and its ecosystem: ECA and bpmn.io together allow the complete customization, extension and automation of all Drupal capabilities without any programming. This means that Drupal is now also moving into the realm of "Low Code - No Code" - a requirement that is not only modern, but also fully in line with Drupal's strategic direction since the last Dries note: empowering site builders and giving them tools to make their jobs easier.

Challenges

The biggest hurdle before the project began was whether it would really be feasible not only to develop a replacement for the very extensive Rules module, but also to get it to a production-ready level and keep it there. However, when we succeeded in building a working prototype in just a few weeks thanks to existing components, the decision was easy.

The pandemic presented another challenge. The development team, some of whom did not even know each other beforehand, did not meet in person even once during the entire project. The coordination of the development ran completely via the issue queue, supplemented by video conferences every 2 weeks. This was not free of conflicts at times, which is more likely to happen in such a work environment than in face-to-face environments. But even these hurdles were overcome, and all participants reported very helpful experiences and learning effects in this regard afterwards.

Community contributions

The complete project is a community project. However, it is worth listing the individual components here:

  • Main ECA module (https://www.drupal.org/project/eca) with 13 sub-modules included:
    • ECA Access: events and actions for access control of entities and their fields.
    • ECA Base: Basic Events, Conditions and Actions
    • ECA Cache: Actions to write data into the cache, to read data from the cache or to invalidate the cache.
    • ECA Config: Events related to Drupal's Config management.
    • ECA Content: Events, Conditions and Actions around Content Entities
    • ECA Form: Events, Conditions and Actions around the Form API.
    • ECA Log: Events and Actions around the Drupal Log-System
    • ECA Migrate: Events of the Drupal Core Migrate Module
    • ECA Misc: Various events and conditions from Drupal Core and the kernel.
    • ECA Queue: Events, Conditions and Actions around the Queue-System
    • ECA User: Events, conditions and actions around user accounts
    • ECA Views: Actions with which views can be executed and, if necessary, exported.
    • ECA Workflow: Actions around Drupal Core's workflow and content moderation.
  • Other sub-modules:
    • ECA UI
    • ECA Development
  • Modeller
  • Other contrib modules
  • ECA Guide at https://ecaguide.org - here you can find the complete documentation, a library with examples, tutorials, videos and much more.

Why should this project win the Splash Awards?

This project is unique in many ways and serves as an encouraging example of the power of Drupal and the community. Not only was it possible to complete about 3,000 hours of development work in just 11 months to a finished solution without endless budget negotiations beforehand, but 2 companies - LakeDrops and OpenCampus - teamed up to make this project straightforward and completely open-source on drupal.org.

This added value of surely half a million Euros and more, is not only a monetary gain for Drupal, but also solves another and more and more urgent problem: As of today, well over 150,000 Drupal 7 sites are still using the Rules module and by far the largest part could not yet migrate to Drupal 9 due to the lack of Rules support. This blockade will be solved by this project and should be very helpful for the whole Drupal ecosystem.

Credit: This application was originally published in German on the Splash Award website and got re-published here with kind permission.

Nov 15 2022
Nov 15

Back in July, we've been so excited about the launch of ECA 1.0.0 after around 3,000 hours of development work in only 12 months, that we didn't even write a blog post at the time. The following weeks, the team was busy dealing with feedback and support, as well as starting to build even more features. That's how 4 months went by, certainly time for a recap of what has been happening and what ECA brings to the Drupal project and its community.

Winner at the German/Austrian SplashAwards 2022

Winning the "Solutions" category is a wonderful reward for all the hard work that went into the project. Being honoured with the special "Biggest Contribution Value" award is beyond our imagination. There is not enough space for celebrating this, but rest assured, we're just really happy!

Facts about the 1.0.0 Release

The analysis presented in June 2022 at the Drupal MountainCamp provides a great overview. Here are some key facts:

  • 358 files and 40k+ lines of code
  • hundreds of tests with thousands of assertions, giving us 90% test coverage
  • code style fully compliant
  • works with Drupal 9 and 10, PHP 7.4 and 8+

Installing and getting started is a matter of just a few minutes. As ECA comes without any dependencies other than Drupal core, there are no issues with contribution modules or packages.

From the roadmap in the same presentation, debugging of models had already been implemented before the 1.0 release. And three more have been implemented for the 1.1 release, made available as a 1.1.0-alpha1 release a few days ago. The release notes demonstrate the huge momentum behind this project.

The benefit of ECA

ECA got started to provide a “Rules” equivalent for Drupal 9 and beyond. While having delivered on that objective, ECA solves additional challenges and helps to trim-down the overhead of each Drupal application.

With ECA it's far less likely for most Drupal sites to require any custom module. This is achieved by providing access to almost any hook, form altering, validation and submit handlers as well as driving cron jobs and queues from within ECA models. In addition, ECA allows configuring many tasks without having to install individual contribution modules for each of them, examples being:

- Redirecting for countless circumstances like 403s, after login, per user role, etc.
- Form field validations
- Computed fields
- Automatic entity labels
- Stripping and trimming of field values

While being powerful and flexible with these smaller tasks, ECA gets to its strengths with the big challenges. We at LakeDrops and also at OpenCampus are using ECA in huge and demanding customer projects. We haven't seen any issues of either instability or performance whatsoever. And if a feature is missing, adding more plugins either in ECA itself or in separate integration modules on drupal.org is nice and easy, often done in minutes or a couple of hours at most.

More important, though, what the Drupal community thinks about and already accomplished with it.

Overwhelming feedback

We often ask ourselves, is it just the lovely people who are providing all this positive feedback? As we're from a country where a common statement is "Not complaining is praise enough", it is at least noticeable when you only receive positive feedback. Which is what happens to the ECA team.

Here are some quotes:

"Oh I'm very grateful for this welcome. I appreciate and believe in the power of this module. I was a hard rules user in D7 times."

"I applied some quite complex logic using ECA in a project and it turned to be a life-changer."

"This is just step one. I've got a lot more to do on this model (and others). But thanks for the tips, and especially thanks for the module. It's been a life saver."

"ECA is [...] going to be exactly what i have needed for years and years - i've been struggling with writing custom modules to replicate even a bit of what i was doing happily in D7 rules! I had some amazing things going on - I've always regretted coming up to D8 and D9 because so much of my site broke when i made the move"

"WOW - I can't believe my test worked. It was surprisingly intuitive."

"Just wanted to say hi. I am a recovering Ruleser (someone who used Rules a lot in D7) and am thrilled about ECA and its potential. With some tweaks, this tool could rival Views in the power it provides users at any level."

"I have lost weeks trying to find a reliable solution for some business rules and the lack of progress in the old rules module is problematic. A pity because it worked very well under D7. [...] Once you get a grip on ECA it is no doubt the best way to go. Although the documentation is still a weak spot."

"I think a good ECA system offers Drupal a complete different dimension than the classical CMS-functionality."

Of course, this is flattering. What's even more impressive to the development team: seeing what users actually accomplish with ECA. Bearing in mind this is a 1.0 release and, as quoted above, documentation is currently a "weak spot".

Stay tuned

The ECA ecosystem is moving fast. Not only is the first feature release 1.1.0 only a few weeks away, we're also working hard on improving the documentation and establish other ways to share knowledge and teach the Drupal community how to get the best out of ECA.

One of these ways will be a regular blog post with showcases of what has been done with ECA. Dozens of examples are in the issue queue almost weekly, on Slack and in other channels, that we will collect and share. Seeing real solutions is the best way to learn, what ECA can do for your projects.

Aug 03 2022
Aug 03

For an existing Drupal project, a customer requested a new feature for a marketing campaign they were about to launch: in their ERP system, they maintain a huge list of projects with many properties associated with them and a high frequency of data change. Both, existing projects get updated, and new projects get added to the list. Those projects should be promoted on their Drupal site, but of course, there is no point in maintaining that data manually or even twice.

To make this a bit more challenging, the data should not only be displayed in a view. That would have been possible with a secondary database configured for the Drupal site and telling the Views module to query that external database for the content. Instead, each of the projects should also come with a landing page displaying more details and a contact form so that visitors can easily submit their interest in it.

There's only one reasonable approach for this: each project in the remote database needs to become a content entity in Drupal.

Not enough of a challenge yet? Then let me tell you that the external database is MS-SQL hosted on Azure. As if all that Microsoft infrastructure and tooling was what we beloved Drupal engineers wanted to get close to. But that's a different story.

Defining the solution stack

As the main requirement is about content synchronization and creating entities in Drupal, there is only one answer to that: Drupal core's migrate module. This may come as a surprise to some, but that set of modules in not only capable for one-off migrations from e.g. Drupal 7 to 9. Migrate is an excellent tool to read data from an external source, process the data and store it in Drupal - not only once but as often as needed, and it contains all required mechanisms to recognize existing and maybe updated data records. To mention that it also includes a solid rollback feature is only to make the picture complete, not because it's required for this current project.

That left us with the requirement to talk to an MS-SQL database and, once resolved, to let migrate accept an SQL database as a data source for all its magic. Following the old saying "There is a module for that", we've been lucky to found one for each of the two requirements:

Equipped with those 3 components, implementation turned out to be extremely simple.

Implementation steps

Once we've downloaded and enabled the extra modules, we first defined the extra database in the existing settings.php file of the Drupal site:

$databases['azure']['default'] = [
  'database' => 'customer365projects',
  'username' => 'DrupalServiceUser',
  'password' => 'PASSWORD',
  'prefix' => '',
  'host' => 'sqlcustomer365.database.windows.net',
  'port' => '1433',
  'namespace' => 'Drupal\\sqlsrv\\Driver\\Database\\sqlsrv',
  'autoload' => 'modules/contrib/sqlsrv/src/Driver/Database/sqlsrv',
  'driver' => 'sqlsrv',
];

That's all you need to do, assuming that the database host is accessible from the Drupal host over TCP/IP. If that's not the case, which is very likely, then your IT has to establish a tunnel with either SSH or a VPN from the Drupal host to the database host. But that's beyond the scope of this blog post.

From now on, your Drupal site can "talk" to that MS-SQL database in Azure. Not that this was needed for our project, but to give an idea what you'd have to do if you wanted to write custom code to read and/or write from or to that database:

$database = \Drupal\Core\Database\Database::getConnection('default', 'azure');
$query = $database->select('...');

The final step for us was to configure the migrate module to import data from this database. We've created a migrate configuration like this:

id: projects
label: Projects
migration_dependencies: { }
source:
  plugin: custom_sql_query
  key: azure
  keys:
    - id
  sql_query: "SELECT *
        FROM projects p
                 INNER JOIN extras e
                            ON p.id = e.id
        WHERE p.status='for-drupal'"
destination:
  plugin: entity:node
  default_bundle: project
process:
  title: project
  updated: timestamp
  field_a: property_a
  field_b: property_b
  field_c: property_c

This is a simplified version of the configuration to show the important building blocks of this solution. Of course, you can facilitate all of migrates power to process data and store it into Drupal's default database.

The key settings to talk to our external database in this configuration are the source plugin, key and sql_query. The plugin custom_sql_query comes from the installed module, the key azure refers to the key we've used in the settings for that database and the SQL query is totally up to you and depends on the data model of your external database.

So, what's next?

drush migrate:import projects

And you're done. Really!

Conclusion

A pretty complex set of requirements gets resolved with just two contribution modules from drupal.org and two configurations in Drupal settings and for Drupal core's migration module. No custom code required. And yet, the customer now can run the migrate:import drush command with a system crontab every 5 minutes. All their project data from their internal ERP system shows up on the Drupal site, always accurate and always up-to-date. It doesn't get any better.

Jul 19 2022
Jul 19

For one of LakeDrops enterprise customers we've developed a fairly complex set of intranets with Drupal 9, where one of the sites is a content hub with tens of thousands of nodes and taxonomy terms that get maintained, reviewed and published on that hub. All other sites require those entities in read-only mode as well, and we've chosen the great module Entity Share to easily synchronize all entities from the hub to all other sites.

While this is working really great, we hit a big wall when rolling this out into production. The synchronization process kicked off successfully, only to fail badly after a couple of seconds.

Entity share uses Drupal's JSON-API to exchange all the information between the content hub and the clients, who want to receive the content entities. Therefore, it uses http requests to receive entity lists and afterwards the entities themselves, of course only those that haven't been synchronized yet or have been updated since the last synchronization. For such a large content hub, this results in a huge number of http requests from a single source to always the same destination. That said, after a few hundred 200 responses as expected, we suddenly only received 503 error codes - and no entities any longer.

The Drupal error log of the content hub contained no indication of what was going on. Those requests for which we received 503 responses didn't even get to that Drupal site at all. They must have been rejected somewhere upstream in the IT stack of that international enterprise, which is completely unknown to us, the Drupal development team.

Opening issues in Jira, a number of phone calls, plenty of emails back and forth, and several hours later we've had more questions unanswered than at the beginning of that process. Whereas, the explanation seems obvious: there is either a request limit or a WAF (web application firewall) involved, either of which got triggered by the request pattern that we submitted. Just finding out what exactly it is, who is in charge and which form we should fill out for the change request caused more headache than hope for a resolution. To make things worse, there have been several departments in various countries waiting for the result, as they booked their own time for the review of our setup. A quick fix was needed.

As it happened, all involved Drupal instances - the content hub and all clients reading from it - are located in the same LAN and can communicate to each other. So, why not requesting the content internally, i.e. from a client Drupal site to the content hub, without using the official domain and route through the full IT stack? Because that requires the content hub to run its own web server to receive http requests and respond to them. Asking IT to set up one for us, would have been equally difficult to explain, if not impossible.

That's where Drupal's CLI tool drush came to help. For whatever reason, drush has a built-in web server implemented in PHP, which of course was available in the existing infrastructure. So, we decided to give it a try, still being sceptical if it could handle the huge load from all the clients for all those entities:

$ > drush runserver 8080

[notice] HTTP server listening on 127.0.0.1, port 8080 (see http://127.0.0.1:8080/), serving site, sites/default
[Fri Jul 15 14:22:52 2022] PHP 7.4.30 Development Server (http://127.0.0.1:8080) started

That's all, no further configuration or setup required. This uses the existing Drupal site settings, accesses the database just like with a LAMP stack and best of all, it doesn't interfere with the existing infrastructure which is still served in parallel, while Drush's built-in web server responds to the internal requests from the entity share clients. They only needed to be re-configured to talk to the internal IP-address of the content hub instead of its public domain, and that was it.

But wait, this crashed after 4 hours! Why? Because the web server contained a hard-coded timeout of 14400 seconds. Well, no problem, creating a pull request in Drush's GitHub repository to remove that timeout, and only 5 hours later Moshe Weitzman accepted it and merged the change into Drush.

Problem solved, customer is happy, deadlines haven't been missed, and Drupal's strength has been demonstrated again. Thanks!

Jul 15 2022
Jul 15

A LakeDrops client runs a campaign where a single block containing a view should only be visible to invited, still anonymous users. This block is embedded in a landing page which remains accessible to all visitors. Due to high traffic, turning off Drupal's page cache isn't an option. So, how could that be done? The problem to solve: once the landing page gets visited the first time, that content will be cached and delivered as is to all subsequent anonymous visitors.

To solve this problem, we've chosen a 3-step-approach:

  1. set a cookie for invited users
  2. control access to the view by that cookie
  3. turn off page cache on the landing page only for those visitors with the cookie being present

This is not difficult at all, the most enduring challenge is behind us in finding out how this could be done. Let's go through the necessary steps.

Set a cookie for invited users

Let's say the landing page is available from /our-products, and we create another path for the campaign as /you-are-invited-for-special-offerings?token=SOMEHASH which will be sent out to all invited customers or prospects. Once they visit that URL, a cookie will be set and the visitor being redirected to the product list automatically.

To do this, we have to define a controller:

# File: mymodule.routing.yml
mymodule.campaign:
  path: '/you-are-invited-for-special-offerings'
  defaults:
    _title: 'Campaign Name'
    _controller: '\Drupal\mymodule\Controller\Campaign::build'
  requirements:
    _custom_access: '\Drupal\mymodule\Controller\Campaign::checkAccess'

The controller class then has to implement the access control and the build function:

public function checkAccess(): AccessResultInterface {
  if (($query = $this->request->getCurrentRequest()->query) && $query->get('token', FALSE)) {
    return AccessResult::allowed();
  }
  return AccessResult::forbidden();
}

public function build() {
  \Drupal::service('page_cache_kill_switch')->trigger();
  $token = $this->request->getCurrentRequest()->query->get('token');
  $cookie = new Cookie('mymodule_campaign_token', $token, '+24 hours');
  $response = new RedirectResponse(Url::fromUserInput('/our-products')->toString());
  $response->headers->setCookie($cookie);
  return $response;
}

The access control only allows access, if the request contains a query argument token, all others will receive an "access denied" response to that route.

The build method first kills the page cache for this route, as we need to process each request individually. We then create a cookie which contains the given token value and is valid for 24 hours only. The redirect gets the visitor to the product list and the cookie will be associated with that response. Step 1 of 3 completed.

Views access plugin for cookies

The view with the privileged content already exists, so we only need to control access to that view with the cookie. This can be done with a plugin:

t('Cookie');
  }

  /**
   * {@inheritdoc}
   */
  public function access(AccountInterface $account): bool {
    $cookieName = $this->options['cookie_name'];
    $cookieValue = $this->options['cookie_value'];
    $request = \Drupal::service('request_stack');
    if (($cookies = $request->getCurrentRequest()->cookies) && $value = $cookies->get($cookieName, FALSE)) {
      if (empty($cookieValue) || $cookieValue === $value) {
        return TRUE;
      }
    }
    return FALSE;
  }

  /**
   * {@inheritdoc}
   */
  public function alterRouteDefinition(Route $route): void {
    $route->setRequirement('_custom_access', '\Drupal\mymodule\Plugin\views\access\Cookie::access');
  }

  /**
   * {@inheritdoc}
   */
  protected function defineOptions(): array {
    $options = parent::defineOptions();
    $options['cookie_name'] = ['default' => ''];
    $options['cookie_value'] = ['default' => ''];

    return $options;
  }

  /**
   * {@inheritdoc}
   */
  public function buildOptionsForm(&$form, FormStateInterface $form_state): void {
    parent::buildOptionsForm($form, $form_state);
    $form['cookie_name'] = [
      '#type' => 'textfield',
      '#title' => $this->t('Name'),
      '#default_value' => $this->options['cookie_name'],
    ];
    $form['cookie_value'] = [
      '#type' => 'textfield',
      '#title' => $this->t('Value'),
      '#default_value' => $this->options['cookie_value'],
      '#description' => $this->t('Leave empty if you only want to check the existance of the cookie.'),
    ];
  }

  /**
   * {@inheritdoc}
   */
  public function getCacheMaxAge(): int {
    return 0;
  }

  /**
   * {@inheritdoc}
   */
  public function getCacheContexts(): array {
    return [];
  }

  /**
   * {@inheritdoc}
   */
  public function getCacheTags(): array {
    return [];
  }

}

This code might be self-explaining, so we call step 2 of 3 completed as well.

Disable page-cache only for requests containing the cookie

This one almost left us behind without any hair left, as we were pulling it out while trying a number of different attempts: pre-render hooks didn't work, nor did any event subscriber. They all get skipped for cached pages and are therefore useless in this context. Next stop, we've tried the implementation of our own StackMiddleware which can be triggered before the page cache middleware by Symfony. This approach then led us to the final approach which tailors the behaviour of the page cache: a page cache request policy.

Once we found out about such policies, the implementation turned out to be more than simple. First, let's define the policy service in our custom module:

# File: mymodule.services.yml
services:
  mymodule.page_cache_request_policy.disallow_campaign_requests:
    class: Drupal\mymodule\PageCache\DisallowCampaignRequests
    public: false
    tags:
      - { name: page_cache_request_policy }

The service class then actually feels minimalistic:

cookies->get('mymodule_campaign_token', FALSE) && $request->getPathInfo() === '/our-products') {
      return static::DENY;
    }
  }

}

Yes, that's all, final step completed as well.

Conclusion

This has not been the first time that Drupal's page cache impressed us to such an extent. It hooks into the http request so early, that almost everything Drupal provides is just being ignored completely. And therefore the response is so fast. Imagine, the cache content is stored in RAM, e.g. with Redis, the behaviour of such a Drupal site is close to the performance of a static website, isn't it?

Our biggest problem to provide the functionality for this customer requirement hasn't been technical, it's been lack of knowledge about the options available. Having solved it after a couple of hours leaves us with a notion of excitement - and that's one reason why we love Drupal that much!

Jun 22 2022
Jun 22

ECA stands for "Events - Conditions - Actions" and is a powerful, versatile, and user-friendly rules engine for Drupal 9+. The core module is a processor that validates and executes event-condition-action plugins. Integrated with graphical user interfaces like BPMN.iO, Camunda or other possible future modellers, ECA is a robust system for building conditionally triggered action sets.

Built from scratch in Drupal 9, ECA leverages all the Symfony and Drupal core APIs for events. It processes any number of ECA models and performs conditional actions on literally anything available inside a Drupal site. From simple field validations, value changes, tampering, sending emails, publishing entities, logging or displaying messages up to more complex models with loops, custom events, caching and stateful actions - ECA provides full access to all of Drupal's power from within a simple and easy-to-use UI. Welcome to the "Low code - no code" interface with no limits. Learn more about ECA on its project page on drupal.org.

Huge test coverage

Feature-wise, ECA was ready for a release candidate a couple of months ago. But the development team decided to hold on with the long-awaited RC1 until they felt confident that all the complexity is delivering on the promise to "just work" - and that they can continue working on it without breaking things down the road. Today, much more than 40.000 lines of code are covered with meaningful tests for 94% of all files and 83% of all lines.

Not only have all those tests passed on Drupal 9.3, 9.4 and 10.0.x with PHP 7.4 and PHP 8.1, all the code is also fully compliant with Drupal's coding standard and well commented. Maintenance will therefore be a pleasant task for decades to come, or as Max concluded when closing the planning task in the ECA issue queue: "I'm looking forward to continue on this, for having a sustainable community project."

How to get started

There is quick start guide, simply because there isn't even a long start ever needed. It only takes 2 steps: first, download the code with

composer require drupal/eca:^1.0@rc drupal/bpmn_io:^1.0@rc

and then go to Admin / Modules (/admin/modules) on your Drupal site to enable all the ECA submodules and the BPMN modeller to use. That's it, from now on, you access all the power from Admin / Config / Workflow / ECA (/admin/config/workflow/eca) in the admin menu of your site. No fiddling with configuration or dependencies, it really just works. Build your first model, save it, and it is in action right away.

Next steps

A period of around 1 month is scheduled for intense testing, feedback and support for and by the Drupal community - that's including you! The final 1.0 release is hopefully being published on July 22nd - exactly 1 year after the initial commit to this project. Of course, that won't be it. Although ECA is perceived feature complete for the 1.0 release, the roadmap for what's next looks as exciting as the existing feature list. Stay tuned, or better still: join the team and influence the direction of the ECA. For now, we're looking forward to your feedback and can't wait to learn, what the world thinks about it.

Jan 14 2022
Jan 14

Development in ECA has been intense over the last 4 months, since we first presented this new module to the Drupal community. Let's catch up with all the features and ideas, that already made it into it.

Who is working on it

In addition to the code maintainers Daniel, Jürgen, Max and Richard we also have supporting team members like Nico who contributed a nice logo, Ralf who helps us to get the word out and provides feedback - and a few others who already started using ECA and report back in the issue queue with issues and feature requests. This helped us a lot to stay focused and move towards a first stable release in such a short period of time.

How we moved forward

Although still in early stage, we optimized the core components by changing how ECA stores its config entities, as well as the caching mechanism to further reduce the performance impact when using ECA in production environments. The overhead ECA brings to your live site is negligible, it only consumes resources if something needs to be done as a result of configured events that your ECA models want to react upon.

The code structure got an overhaul too. There is now ECA Core which provides the config entity handling, the ECA processor, all required plugin managers and numerous interfaces, traits and base classes used by the now 10 submodules that comes with ECA:

  • ECA Base: provides access to context, cron, custom events, state system and tokens
  • ECA Config: integrates with Drupal's CMI
  • ECA Content: supports content entities, fields API and more tokens
  • ECA Form: integrates with Drupal's form API
  • ECA Migrate: integrates with events of the migrate module
  • ECA Misc: make available events from Symfony's kernel, Drupal core and the routing system
  • ECA Queue: allows to enqueue tasks for asynchronous or scheduled processing
  • ECA User: integrates with user account related events and actions like login, logout, cancel, account switching and roles
  • ECA Views: allows processing of views with further processing or exporting of the result set
  • ECA Workflow: integrates into Drupal's content moderation framework

Note that all of ECA depends solely on Drupal core and the new remarkable Context Stack module from Max, which we extracted from the ECA code base simply because its functionality is useful for other tools too. This is intentional so that you can easily onboard ECA without the requirement to have any other components being installed or enabled.

General features built into ECA core

With all the submodules enumerated above, you already get comprehensive lists of events, conditions and actions from either ECA or from Drupal core. Additionally, ECA has baked-in support for crucial APIs and functionality, that needs highlighting:

  • Context Stacks: each process triggered by any event as well as any action of such processes get executed within their own (stackable) context, so that ECA can restore the context after execution to the state it was right before processing. This not only makes each process predictable, but also allows to forward the current context into the queue, should a model want to process actions asynchronously.
  • Caching: all ECA models get cached in a special way such that ECA can determine, with almost no overhead, whether any of the triggered events in your Drupal site needed to be processed by ECA.
  • Loops: by default, actions in an ECA model operate on some data object, which often is something like a content entity, a route, an HTTP request or any other arbitrary value. If you provide a list of objects instead of a single instance, then ECA automatically loops over each of them with the subsequent actions in the process model.
  • Logging: everything in ECA is built to provide detailed log information, which is written to Drupal's logger service. It depends on your own setup, if that logging information is being written to watchdog, syslog or elsewhere. To avoid overwhelming logs just from ECA, you can configure the level of log information coming from ECA with a single setting, defaulting to error messages and exceptions only.
  • States: reading and writing from and to Drupal's state system helps to persist some data for later use, even if this is required over a longer timeframe.
  • Tokens: Drupal core's token environment is already pretty powerful. ECA builds on top of it and enhances it by introducing DTOs (Data Transfer Objects). This mechanism allows any event, condition or action in an ECA model to store any data into the token environment so that subsequent components of that model can access that data for further processing if needed.
  • TypedData: one of the most significant features and presumably the most used conditions and actions are around entity fields. Those fields can be more complex than just storing individual values, they can even store values with any cardinality. To allow ECA models generic access to any component of such fields, the TypedData API is the ideal tool provided by Drupal core, which has been utilized by ECA.
  • Recursion prevention: it may happen unintentionally, but ECA models, like any other rule definition, can quickly end up in a recursion. The simplest use case might be an entity update, which results in a process that makes additional modifications to that entity - being another entity update that already starts a recursion. As this would always lead towards an infinite loop, something you prefer not to experience on your production site, ECA inevitably prevents that from happening. Even at the cost of preventing you from building certain models. However, there should always be a way around that - and the issue queue on drupal.org or this Slack channel are great places to seek help in such situations.

Working on test coverage

We've already started building tests, from simple unit test to kernel and functional tests, but also a powerful framework to test any ECA model regardless how complex they are. This helps us to keep moving with ECA development while ensuring that existing functionality remains intact. Having said that, there's a lot that needs to be done to bring the test coverage to the necessary level. This is what we're focussing on in the weeks to come.

Integrating with other modules

Now that ECA core with all its integrated submodules stabilizes its own APIs, the time is right to start working with other maintainers of popular and useful modules, to make their functionality available to ECA and its models. The list of modules, we would like to integrate with, is being tracked in this issue and if anyone wanted to add more modules to the list, this is the right place to get the ball rolling.

Talking about the UI provided by modellers

As powerful and feature rich the ECA framework already is, it still requires the UI component which allows users to create and maintain their models. This is a separate task and completely decoupled from ECA as the backend processor. The component we're talking about here is called a modeller. And there can be more than just one. BPMN as one of the popular standards to describe business processes is supported by a variety of applications that allow to work with such models, two of which have already been integrated with ECA: Camunda and BPMN.iO. But others, even such that are not based on BPMN, can be integrated as well, and we hope to see initiatives who want to get started on such modeller development.

However, what matters to all the existing and future modeller UIs, is a great user experience (UX). This is what we haven't worked on yet for the existing integrations, and that's one of the areas where we have to spend more resources - but we also need help for that. The existing ECA team is busy with all the processing and background tasks. To move more quickly with the UI too, we're seeking for help.

Conclusion

A lot has been achieved! And I mean, A LOT. The team is really proud of it and feedback we receive is very positive too. We even consider ECA very close to being feature-complete for its 1.0 release soon. Yes, we admit, there is a way to go before it will be usable by "mainstream" users. The UI components mentioned above as well as comprehensive documentation are just the 2 main tasks that come to mind.

Having said that, it turns out that ECA not only accomplished its original objective to provide a "Rules" replacement for Drupal 9 and beyond, it also delivers benefits above and beyond that mission with 2 additional wins:

  1. Reduce the likelihood of writing a custom module for a client, only because you required a hook or a form alter function, that can now also been done with ECA.
  2. Eliminate the requirement for a number of helper modules that provide simple functionality like redirects after login or validating, stripping, trimming or otherwise manipulating certain fields.

In other words, the feature set provided by ECA is huge, yet it can reduce the complexity of any Drupal installation out there. And while doing it this way, ECA also improves transparency and maintainability by storing all the rules in one place, where it can be visualized, reviewed and even moderated so that all stakeholders get a better understanding of what's being implemented and configured at any given time. It also increases accessibility for users who understand the business logic implemented into a Drupal site but wouldn't ever be able or willing to dive deep into Drupal logic or even code base.

Let's move on with ECA, it's got that far that we're all very excited and motivated to move it over the finish line. Watch this space to stay up to date - or reach out on drupal.org or in Slack, if you either have ideas, feature requests or resources to help us get all this done.

About Drupal Sun

Drupal Sun is an Evolving Web project. It allows you to:

  • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
  • Facet based on tags, author, or feed
  • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
  • View the entire article text inline, or in the context of the site where it was created

See the blog post at Evolving Web

Evolving Web