Upgrade Your Drupal Skills

We trained 1,000+ Drupal Developers over the last decade.

See Advanced Courses NAH, I know Enough
Apr 23 2020
Apr 23

Everyone is looking for positive news, and we have some for you. Drupal web development in times of Covid-19 continues — severe crises are even opening new perspectives! One of the characteristics of a good web development company is that it only becomes stronger in hard times.

Read on to discover why the idea to hire remote software developers for your Drupal project is an especially good one during the Corona crisis. We also describe how we provide Drupal web development services now, under quarantine.

Remote work is our team’s natural environment

Due to the quarantine restrictions, we, like most IT companies, transferred our team to remote work mode. However, nothing has changed for anyone who wants to find software developers for hire. Some things have even become better.

In our case, remote work is not new — this is something we had mastered already because we had a chance to. Outsourcing web development overseas has always been a popular approach among businesses. Ukraine is always on the list of the best countries to outsource web development to, we have always seen Ukraine — indeed, there are particular good reasons to work with a Ukrainian outsourcing company. Multiplied by our skills and reputation, this has always provided the InternetDevels Drupal web development company with overseas customers.

That said, everyone who is looking to hire developers should know that the Covid-19 quarantine has had no impact on the quality of our web development services:

  • First, we established high-level remote communication and smooth workflows with the overseas customers long ago.
  • Second, we have always been flexible with allowing our own developers to do their work from home and this is not a problem.

As always, we invite you to hire remote developers from the InternetDevels team and promise to provide as efficient workflows as ever! We firmly believe that efficient Drupal development services should be available from any location.

Key components of our remote development services

The staff

Drupal developers, just like most other IT experts, value a quiet environment that is not always available in the office. They need inspiration and concentration in order to be productive. So the quality of their work may even improve from their home isolation. 

While working remotely, our team members are able to make themselves comfortable on their personal couch when they want to and deliver their genius ideas. In addition, they are never stuck in traffic jams and have more time for their families.

Of course, working from home takes self-discipline and responsibility. But we only keep people on the team who have proved responsible and reliable. They keep to the deadlines and regularly meet with the whole team and the customer via video calls according to the best Scrum practices.

Remote work requires establishing a good work-life balance, and our Drupal team tries to stick to the best recommendations here. These include having strict working hours, changing clothes from pyjamas to something more official, informing the family about their video conference time, and more. However, these are just recommendations, because it’s not the clothes that matter —- it’s the skills.

The equipment

Drupal development is in a very lucky industry that can continue through the lockdown. The only necessary equipment being computer hardware. It’s very easy — an office can be at home if it is properly equipped with the right PCs! For many IT specialists, the capacities of their PCs and the characteristics (and the number) of their monitors are indispensable for their productive work.

InternetDevels remote home office in times of quarantine

So we have provided our development team with all the hardware they need and have also helped with the deliveries to their homes. There was even a case when one member of our development team suddenly had their laptop fail, and we quickly delivered another one so this person did not lose a precious working day.

That said, if you hire remote developers from the InternetDevels web development agency, you can be sure they have everything they need to do the work as efficiently as ever.

The tools

As always, we use the best tools to provide for smooth remote communication and efficient workflows. They eliminate repetitive tasks, increase the development speed, take care of the shared work, and do much more. These include messengers, video conferencing tools, project management software, version control tools for developers, etc.

Among the tools are Zoom, Slack, Google Hangouts, Jira, Skype, Github, Bitbucket, Google Docs, Google Drive, and many more.They are crisis-proof and work at any distance. We have been using them with our overseas web development customers for a long time, so there is nothing new in them.

The new perspectives of your digital presence

Here is what we meant when we promised to tell you about your new perspectives. As you know, the world has moved online due to the lockdown. People browse the web instead of travelling, hiking, or doing other outdoor activities. They are online, and are you there for them?

Businesses that are represented online have a clear advantage. There is an increased need for online shopping, delivery services, online educational and entertainment activities, blogs and news, and much more.

The lockdown will end someday and Covid will be happily defeated — there is no doubt about it! People will travel again, walk around their beautiful cities, and see their loved ones.

However, the trends of online activities will remain ever after — the world has understood their convenience. For example, someone who once enjoyed shopping in your Drupal online store with a user-friendly shopping cart will continue to do it just because it’s quick and easy — so don’t miss a chance to hire the best e-commerce developers.

Loyalty to customers and affordability of services in times of crisis

We strive to help our customers in difficult times, so we are offering very reasonable pricing if you would like to hire remote developers. We want to take part in providing your business sustainability now.

We know that Drupal websites can absolutely give businesses a big boost by attracting customers, promoting their brand, and driving conversions.

We can always find more affordable Drupal development solutions — for example, the use of the core and contributed modules where it is relevant. The efficiency also matters — our devs are able to do much more work per hour in order to deliver the results to you quickly and without extra costs. We started a strategy to immediately divide the scope of work into the standard and custom parts and advising our customers on a faster and more affordable option. So drop us a line to discuss this for your site.

The benefits of outsourcing web development

In our big world full of Drupal talents and having a great level of remote workflows, there is no more need to look for “software developers near me.” Finding a software developer that meets your needs can be a much more successful task if you expand your search across the globe and hire overseas. Here are just a few reasons why outsource web development:

  • Considerable cost savings. Costs can be saved thanks to the big choice to hire developers at more affordable prices outside of your country. Another big advantage of outsourcing vs hiring in-house is the savings it gives you on the recruitment process, permanent monthly wages, work equipment, and so on. By outsourcing web design overseas, you only get your services when you need them.
  • Broader skills. Outsourcing web development is a chance to find top talents in the industry. If you hire an in-house developer, they will find it hard to compete with a team of experts who have worked on many projects, faced a variety of issues, and can combine their specific skills in ever-changing technologies on your project. In addition, UX design, QA, SEO, and other specialists are also part of Drupal web development teams and play their valuable role.
  • Focus on your business. When you hire an outsourcing team, you entrust it to experts without having to worry about or dedicate your precious time to it. You can focus on your own key business processes instead. Still, you are fully able to control the project — your input is welcome in the Agile practices and we are fully available through remote communication and transparent task tracking.
  • Stable workflows. What to do if a developer “falls out” of the process due to an illness or other unforeseeable circumstances, or a particular one cannot fully meet your needs? To cover these cases, outsourcing web agencies have other devs to substitute them. By signing a contract, they take the responsibility to provide stable workflows throughout your project. This is a significant advantage over hiring an in-house developer.
  • Flexibility on your project. Every stage of your project needs a specific amount of work from particular experts: developers, designers, QA engineers, etc. You don’t permanently need all this staff in-house. You are free to vary your requirements according to your business situation — for example, first create an MVP (minimum viable product) and then add features and scale your project. This is not a problem with Agile agencies.

Hire our remote web development team

If you are ready to join the customers who outsource development projects and enjoy the resulting efficiency, our Drupal team is here. Hire remote developers from InternetDevels to outsource web development at affordable prices today. Stand strong against the crisis and more forward thanks to your digital presentation! Contact us to discover more details.

Apr 23 2020
Apr 23

Greetings to all of you who are lucky enough to have a Drupal website! Hopefully, your business is doing well, and your site is helping it reach its goals. We know Drupal websites can do this perfectly — they just may sometimes need improvements, changes, updates, or fixes.

We strive to relieve all website owners of any troubles by taking care of all of the above. That’s why we offer Drupal support services at very affordable prices through our quick, easy-to-apply, and reliable Drupal helpdesk. Learn more about the tech support from WishDesk in this post.

12+ years of working with Drupal

Tech issues and ideas about your website can be different in their complexity, so they need a skilled approach. The WishDesk tech support team knows Drupal from the inside out due to 12+ years on the market of Drupal services. We know how to identify and resolve any tech issues on a case-by-case basis. Here is why many website owners entrust their Drupal enterprise support to us.

What services can you order in our Drupal support desk?

The tech support services include literally anything that keeps your website up-and-running and performing its mission. Among the key ones are:

All kinds of Drupal site audits

Speaking about tech support, we would like to mention website audit services first. Audits are comprehensive, in-depth checks of specific areas of a website’s work by experts to discover its weak points to be further improved. This can be a website performance audit, SEO audit, security audit, code review, and similar projects. In addition, this can be something not directly associated with your website — like an IT infrastructure audit performed by our DevOps team.

All kinds of Drupal site audits

Drupal updates and upgrades

One of our Drupal support benefits is that it can help you keep your website up-to-date. This is vital in terms of security, innovation, performance, usability, and more. Customers often ask to migrate from Drupal 6 to Drupal 8, upgrade from Drupal 7 to Drupal 8, prepare for Drupal 9 through a good cleanup of deprecated code, and more. Minor updates, like from D8.5 to 8.8, as well as Drupal security updates, are also among the tech support services in demand.

Website performance optimization

Website speed matters for business in terms of traffic, visibility in search engines, and conversions. We have devoted a special post to the ways in which a slow website can cost you money. Many website owners see that their website is slow, however, they don’t know the exact reasons. It may lie in the image optimization, caching issues, file aggregation, and much more. This is where the WishDesk tech experts come in — they know plenty of ways to increase website loading speed based on your particular case.

Bug fixes on Drupal websites

When a fresh website arrives from the development team, it is usually well-tested against bugs — at least if you have worked with a decent web development agency (if not, you can order website testing services and bug fixes from our Drupal website support team at any time). But even an initially clean website may become buggy with time due to factors like the absence of core and module updates, unprofessional changes, and many more. Our tech experts will identify and fix any bugs, which also includes bug fixing services for custom modules.

Adding new functionality to a Drupal site

Drupal websites are able to grow and develop together with their businesses, so it’s easy to equip them with new features at any time you ask. It may be something easy like adding a contact form or a menu, or something more challenging like creating a decoupled Drupal Commerce setup on your store or building a mobile app based on your site — the high tech skills of our Drupal help team allow it to do anything. So don’t hide your new ideas — share them with us!

Adding new functionality to a Drupal site

All kinds of Drupal site improvements

Anything you wish on your website can (and should!) be improved. Whenever you want to;

The WishDesk’s Drupal support and maintenance services will come in handy.

Urgent Drupal website help & recovery

Sometimes you need the assistance of our Drupal tech support team to resolve critical issues and recover your site. There are many reasons for this — for example, your website may have been hacked or you could have accidentally done something (like deleted your public_HTML folder), cannot restore your site from the backup, or any one of a number of possible issues. If your site is unavailable to users or your data has been compromised, there is no time to waste. Our Drupal support desk is ready to quickly recover, clean up, and increase your site's protection for the future.

Drupal website hosting support

Trouble-free hosting is vital to provide a good website performance. We offer web hosting support services that include:

  • troubleshooting server problems
  • informing you about the server or its software settings and updates
  • restoring your data from the backup
  • and more

Just rely on our DevOps team, which is equipped with the necessary tools and has extensive experience. By offering Drupal hosting and support, we cover the key tech needs of website owners in post-development time.

Making your Drupal website accessible

A very hot topic today is making websites accessible in line with the latest requirements. The reasons why accessibility is important for business include brand building, protection from lawsuits, audience expansion, SEO gains, etc. Your site’s ALT tags, form labels, menu links, HTML structure and much more needs to be accessible to assistive technologies. Our tech support will easily provide it in accordance with the latest practices and using the Drupal website accessibility features properly. We can also take care of your site’s mobile web accessibility.

Making your Drupal website accessible

Drupal tech support doesn’t have to be expensive!

If you worry about Drupal support costs, we have great news. We are convinced that Drupal tech support services need to be reasonably priced.

Tell us about any issues you may have or ideas for your website and our Drupal website support team will resolve it at the best price/quality ratio. This can be one-time tasks on demand or a long-term Drupal maintenance contract. So just contact us and let’s discuss what suits you best. The application process is very simple and the consultation is free!

The Drupal heart is beating strong

Apr 23 2020
Apr 23
Apr 23 2020
Apr 23

Yesterday was a good day for the Drupal community, because it very clearly showed how much power can be released when we stick together

The world has been upside down the last few weeks because of the COVID-19, affecting individuals and businesses around the world. To meet unforeseen financial troubles due to the cancellation of DrupalCon Minneapolis, the Drupal association reached out to the community for help. 

Supporting Drupal

Viral donations for the Drupal Association 

#DrupalCares went viral where individuals and companies were encouraged to help the Drupal Association with donations.

Last week Dries and Vanessa Buytaert announced that they would match individual donations up to $100k which then lead to a group of dedicated business leaders in the Drupal ecosystem coming together and deciding to match for another $100k

That means that every $1 donation will become $3. Now that's what I call a positive viral affect! 

1xINTERNET supports the Drupal project

1xINTERNET has always been a driving force when it comes to supporting the Drupal project and the Drupal Association, and therefore we didn’t think twice about taking part. Our whole team is deeply dedicated and I'm so proud of our employees at 1xINTERNET that also donated individually to the project.

Here we are in a powerful group of companies that today announced that together they will match individual #DrupalCares contributions for another $100,000, because we believe in the power of Drupal. We all have the same goal, to see Drupal thrive and grow and continue to be the force it is. 

#DrupalCares

It made my day to receive these positive news about the power of the Drupal community and what we are capable of. Last August I wrote a blogpost about Contributing to Open Source where I highlighted my opinion of the importance of being an active member in our community. I'm now more convinced of how important it is and I'm pretty sure a lot of you agree with me.

#DrupalCares Challenge

We see the purpose highlighted at this time when online presence has never been as important with the world physically shutting down around us. Drupal provides infrastructure for many public health organizations and health care systems that now are delivering information about the COVID-19 pandemic situation to millions of people as well as many Higher Educational websites providing important information to students all over the world.

I encourage you to take part and keep the Drupal heart pumping.

See all information about the status of the #DrupalCare project, its not too late to visit the  donation  page and take part.

Owen Lansbury, co-founder of PreviousNext: DrupalSouth, running code sprints in the sun & GovCMS

Apr 23 2020
Apr 23
Apr 22 2020
Apr 22

Drupal 8 to 9 Upgrade

As the release of Drupal 9 approaches, organizations are starting to think about when to upgrade to Drupal 9. Quick Drupal adoption isn't automatic. Historically, it's taken years for some significant Drupal versions to gain traction. With a relatively short window between the Drupal 9 release and Drupal 8's end-of-life, however, organizations must move more quickly to adopt Drupal 9 or make other arrangements.

No penalty for early Drupal 9 adopters

A common strategy for many technology release cycles is to avoid the initial version of a major software release. Some organizations wait until one or more point releases after a new version, while others prefer to wait months or even years after a major version release for things like bug fixes, additional features, and helpful resources created by early adopters. In the Drupal world, this delay is often exacerbated by waiting for contributed modules to be compatible with the new version.

The nice thing about Drupal 9 is that there is no penalty for early adopters, so there's no reason to wait for a later version. The initial Drupal 9 version release introduces zero new features. Drupal 9.0 core code matches Drupal 8.9 core. The only differences between Drupal 9.0 and Drupal 8.8 or 8.9 are the removal of deprecated code and required upgrades to third-party dependencies.

The primary consideration is whether or not your favorite contributed modules have declared that they are Drupal 9 compatible. With past upgrades, waiting for contributed modules to be ready for the new Drupal version caused months or even years of delays. But the Drupal 9 upgrade path for contributed modules is relatively easy, so they should be able to adapt quickly. Many modules are already compatible, and others will need minimal changes.

When your code is ready

One of the core components of the Drupal 9 upgrade is the removal of deprecated code in Drupal 8. However, this means that when you're planning your release window, you'll need to schedule some time for the pre-work of auditing and refactoring deprecated code. If you've already been doing this, you may not need to pad your schedule to compensate for this work. We'll dive deeper into how to get your code ready in a future article.

In addition to scheduling time to address code deprecations, you'll also need to give yourself time to address any third-party dependencies that require newer versions in Drupal 9. When you're looking at when to upgrade to Drupal 9, you should do it after you've had a chance to resolve any third-party dependency updates that conflict with other things in your stack. If you've got a contrib module or custom code that requires an older version of a third-party dependency, but Drupal 9 calls for a newer version of that dependency, you'll need to make a plan and address this conflict before you upgrade to Drupal 9.

Consider other website work

Many organizations have traditionally used major Drupal version migrations as a time to plan overall website redesign projects, information architecture work, and other web development projects. Because the upgrade to Drupal 9 is more like a minor release than a major one, there's no need to deep dive into information architecture - there's no migration! That means your organization needs to establish a new strategy for these projects; we're working on an upcoming article to cover web development strategy for Drupal 9 for more insights around this.

If business logic dictates that your organization plan other web development projects for this year, make sure you give yourself time to complete the Drupal 9 upgrade before Drupal 8 reaches end-of-life in November 2021. 

Take the availability of preferred partners and development teams into account

If you're planning to work with vendor partners, make sure you factor their availability into your project plan. With an upgrade window of slightly over a year between the release of Drupal 9 and the end-of-life of Drupal 8, some vendor partners may have limited availability, especially if yours is a larger project. Planning ahead helps to ensure you can work with your preferred partners; otherwise, you might add the stress of working with a new partner into the mix.

At the same time, don't forget about internal initiatives such as serving multiple stakeholders. For example, doing new feature development for content editors while simultaneously maintaining an up-to-date platform consistent with your organization's security policies can mean a dance to prioritize development resources to meet everyone's priorities and deadlines. While this complicates the release planning process, it's essential to consider these factors when determining the timing of upgrading to Drupal 9.

We dipped our toes into these considerations in Drupal 8 Release Planning in the Enterprise, and hope to release an updated version of this article soon for Drupal 9 release planning.

Missing the Drupal 9 upgrade window

To summarize, you should upgrade to Drupal 9 earlier rather than later. But what if your site can't upgrade to Drupal 9 before Drupal 8 reaches end-of-life? Unlike Drupal 7, Drupal 8 does not have an extended support program. The upgrade from Drupal 8 to Drupal 9 is such a minor replatforming effort compared to prior versions that the decision was made not to offer an extended support program for Drupal 8. 

Support will continue through November 2021 for sites upgraded to 8.9.x, but support for that version ends when that Drupal 8 end-of-life date arrives. Older Drupal 8.x versions will cease getting support before that date; 8.7.x stops getting security support as of June 3, 2020, and security support ends for Drupal 8.8.x on December 2, 2020.

Long-term, your organization needs a plan to upgrade to Drupal 9, or to consider other options. A future article in this series offers more information about what that plan might look like.

Thanks to the Lullabot team for contributing to this article and to Dachary Carey for drafting it.

Karen Stevenson

Thumbnail

Karen is one of Drupal's great pioneers, co-creating the Content Construction Kit (CCK) which has become Field UI, part of Drupal core.

Apr 22 2020
Apr 22

April 22, 2020

Kubernetes is a way of deploying resilient, scalable applications to the cloud.

  • Resilient because Kubernetes is designed to recover if something goes wrong.
  • Scalable because with Kubernetes, your application is not linked to a single virtual machine (VM), but rather to a cluster of VMs which you can scale up or down transparently.

What Kubernetes is not is a magic bullet. Before investing too much in Kubernetes, you are encouraged to read “Let’s use Kubernetes!” Now you have 8 problems, by Itamar Turner-Trauring, Python Speed, March 4th, 2020.

In this article, we will create a Kubernetes cluster and deploy a minimum viable Drupal installation to it, with the following features (this list will be our success criteria at the end of this article):

  • Minimal vendor lock-in: we will avoid vendor-specific resources such as database and volume storage where possible, prefering our own containers.
  • Deployment of Drupal alongside other applications: we will deploy applications other than Drupal to demonstrate how your Drupal app can coexist nicely on a Kubernetes cluster.
  • Secret management: Your Drupal application probably has secrets: environment-specific information such as API keys, or database passwords which should not be in the codebase. We will see how to manage these in Kubernetes.
  • LetsEncrypt: We will serve our different cluster applications via HTTPS using an Nginx reverse proxy, with set-it-and-forget-it automatic certificate renewals.
  • Volumes: Our Kubernetes applications will store their data in volumes which can be backed up. In the case of Drupal, the MySQL database and the /sites/default/files directory will be on volumes. All application code will be on containers, as we will see later.
  • Automation of incremental deployments: deployment should generally be as automated as possible; most modern applications see deployments to production several times daily. In the context of this tutorial we are not recommending Kubernetes on production just yet, but rather to serve development environments; the performance and security concerns related to Kubernetes on production are outside the scope of this article, and frankly at the time of this writing I haven’t yet used Kubernetes on production myself.
  • Easy local development: although having a local version of Kubernetes is possible, it can make your laptop really, really hot. We will use Docker and docker-compose rather than Kubernetes to develop our code locally.
  • Branch staging environments: we will spin up environments per GitHub branch and destroy the environments when the branch gets deleted.

Notice that I haven’t gotten into the jargon of Kubernetes: nodes, pods, deployments, services; for me this has taken a while to get my head around, so my approach in this article will be to introduce concepts only as we need them. You can always refer to the glossary at the end of this article if you’d like quick definitions.

This tutorial is presented in several sections for your convenience:

Please enable JavaScript to view the comments powered by Disqus.

Apr 22 2020
Apr 22

Some time ago, I announced that I was planning on writing a book on automated testing and test driven development with Drupal. I created a landing page and set up a mailing list, but I wasn't sure at that point what I was going to cover or create as part of the book.

I'm going to write a book on automated testing in Drupal. Join the mailing list for updates, and I'm happy to take suggestions on what to cover. https://t.co/YXNpe6f8Ft #drupal

— Oliver Davies (@opdavies) May 15, 2018

Being a meetup and DrupalCamp conference organiser, after some thought I decided to build a website for an example conference, and that some of this code would then be included in the book as example content. This seemed to cover most of what I originally wanted, through features like a call for papers for potential speakers to propose sessions, allowing organisers to administer and moderate those proposals, automatically sending notification emails to submitters and displaying the accepted sessions.

I've started building it with Drupal 8.8 and it is now available on GitStore to purchase access to, including all future updates as I continue building the application - adding new features and upgrading to Drupal 9 once it is released. There are some other interesting things there too, such as using feature flags to enable or disable functionality, and using GitHub Actions to run the tests automatically.

The book itself I've added a page for on Leanpub, and I'll be continuing to add content to it in parallel to building the example codebase. Once there is enough content, I will release the first draft for purchase.

Any purchases that are made via Gitstore or Leanpub, an amount will be donated to the Drupal Association and the #DrupalCares campaign to help sustain the Association during COVID-19.

Apr 21 2020
Apr 21

Covid-19 has upended daily life, and in many cases, revealed trends that were a long time in the making. 
 
Work-at-home requirements brought the essential need for online connections and services to the forefront. 
 
More so than ever before, county and municipal government websites serve as a virtual town square and the place for: 

  • Keeping informed about events and public health alerts,
  • Taking care of official business,
  • Showcasing major attractions,
  • Attracting future tourism,
  • Supporting local businesses, 
  • Staying connected to the community, and 
  • Drawing attention to praiseworthy people and points of pride. 

Multi-faceted Functions

Back in the day, city halls and county courthouses were architectural masterpieces, situated in the center of town with well-tended gardens, and surrounded by the businesses that fueled the town’s economic engine. 

The Wharton TX county courthouseThe Wharton, Texas County Courthouse. Completed in 1889.

Much of what had been accomplished in and around the limestone and granite structures that were built in the 19th and 20th Century is now taking place online.

As such, officials in places such as Martin County Florida are pouring a high level of focus and commitment into their citizens' experience of their websites -- just as their counterparts from previous eras understood the profound importance of creating a welcoming experience that was efficient to navigate and supported civic pride. 

home page of the Martin County Florida websiteHome page of the Martin County, Florida website

 

New World, Same Needs

Venues have evolved, but many essential needs and expectations of last century’s physical town squares have much in common with today’s virtual ones.

Here are the objectives that are driving excellence among public sector websites in the current era:

  • Support citizens’ pride of place with an engaging, visually appealing experience
  • Serve as the go-to information resource in times of crises and uncertainty
  • Drive collaboration and connection among commerce and community groups
  • Streamline access to a vast range of services that previously needed to be accomplished in person
  • Accommodate the different ways that the full range persona groups who visit the site interact with technology
  • Meet and exceed ADA Section 508 compliance standards and the web content accessibility guidelines (WCAG) that ensure accessibility for people with disabilities
  • Leverage the latest technologies and expertise concerning optimal user experience, navigation, and site flow
  • Align navigation and information flow with user needs vs. the internal organizational structure
  • Bring essential stats and facts to life with illuminating data visualization strategies
  • Build in flexibility to ensure that site administrators can easily and efficiently make changes whenever necessary

This is the starting point. Every state, county, municipality, and public sector website has distinct opportunities to connect with constituents, drive efficiencies, and serve as a trusted information resource. 

Looking to explore new possibilities for your public sector website? Contact us today


 

#DrupalCares: Thanks to Drupal Businesses, You Can Now Triple Your Impact

Apr 21 2020
Apr 21
Apr 21 2020
Apr 21

It is incredible to think how far the #DrupalCares campaign has come in such a short time. Last week, Dries and Vanessa Buytaert announced they would match individual donations alongside new and increased individual memberships up to $100k, and you truly answered the call.  The vigor and compassion of the community response is inspiring - and you inspired yet another amazing element of the #DrupalCares fundraising campaign!

When they saw the #DrupalCares matching gift challenge, a group of dedicated business leaders in the Drupal ecosystem came together and said, "We can make this even better."  Acting quickly and without prompting, these business leaders saw the impact the Drupal community was making, and in the truest sense of the open source spirit, decided to multiply that impact even further.

Today we announce that individual #DrupalCares contributions are now being matched for another $100,000 by these generous organizations who believe in the power of Drupal. Both matching gifts apply to all individual contributions, donations or memberships from the start of the campaign.

  • First, every $1 the community contributed created the core foundation that would ensure the Association's success.
  • Then, that $1 became $2 when Dries and Vanessa announced their generous matching gift challenge, and your impact was doubled.
  • As of today, that $1 has become $3, as these organizations who believe in the power of Drupal and Open Source have taken your contributions to a new height.

 Will you join the campaign, and triple your impact to secure the Drupal Association's future?

Give to #DrupalCares

Thank you to these organizations, who have funded this match

This incredible coalition of matching gift organizations and their generous donation further proves the power of the Drupal Project and the Drupal Community.

1xInternet

Acquia

Acro Media Inc

Appnovation

Axelerant

Chromatic

CivicActions

Dropsolid

Drupal Contractors

Drupalize.me

Digital Echidna

Elevated Third

Evolving Web

FFW

Forum One

Four Kitchens

imagex

Intracto

Kanopi Studios

Last Call Media

Lullabot

Oomph, Inc

Palantir.net

Phase2

PreviousNext

Skilld.

Tag 1 Consulting

Undpaul

Zivtech

Thank you to these #DrupalCares Champions

Several Drupal Community leaders in particular are responsible for bringing together this incredible expansion of the #DrupalCares campaign.  I would like to send my heartfelt thanks and sincere gratitude to: Anne Stefanyk of Kanopi, Baddy Breidert of 1xInternet, Chris Murray of Oomph, Inc, Jeff Walpole of Phase2, Michel van Velde of One Shoe, and Tiffany Farriss and George DeMet of Palantir.

Special thanks and extra #DrupalHugs

While there are so many people in the Drupal community deserving of our gratitude and praise right now - I want to give special recognition to Matt Westgate of Lullabot. Matt is the person who first reached out on Twitter to pledge Lullabot's sponsorship, regardless of what might be happening in the face of COVID-19, and the person who has quietly encouraged others in the Drupal ecosystem to step forward in this time of need.

With that pledge, Matt kicked the first pebble that created the momentum behind additional sponsor pledges, member renewals, and donations. In short, Matt and the Lullabot team made us believe that launching the #DrupalCares campaign really could work.

Thank you, Matt.

Apr 21 2020
Apr 21

We partnered with the City of Cambridge to redesign Find It Cambridge, an online opportunity locator serving city residents, in a unique way— designing and developing out in the open, and releasing the software under an open-source license so that other cities can spin up their own Find It platforms.

Cambridge, like many cities, has a wide array of programs and events happening to serve residents. However, it can be difficult for people to find and compare the many offerings out there. The city website has a calendar of events and list of departments. These are limited to government-run programs, though. There are myriad nonprofits and community groups that run programs and events that go unlisted. Social media platforms like Facebook have filled the gap in some areas, but promotion of these opportunities relies in many ways on people's social connections— leaving those most in need of services out of the loop. Find It Cambridge solves this problem by aggregating the many different opportunities happening into one website.

The upgrade and migration from Drupal 7 to Drupal 8 (ready for Drupal 9!) afforded Cambridge and Agaric the opportunity to make the directory of organizations, events, and programs better for families and their children and all involved, grounded in research and testing.

Intuitive, Structured Authoring Experience for Service Providers

The value of a directory are the listings within it and it's truly a community effort to assemble enough accurate and up-to-date resources in a single place, for it to be useful. For Cambridge, the service providers that work at government agencies and nonprofits are the lifeblood of the directory. Without them, there would be no Find It Cambridge.

The challenge then, is building a system that is easy enough for people (many already pressed for time) to take the time to enter their information into, while structuring the data to be easily searched and filtered on.

Through user research, we mapped the information architecture to the mental models that service providers hold for their events and programs.

Input Fields Grouped by Key Questions

Most service providers thought of their events in terms of what the event is about, how to contact the organizers, who it is for, when it is happening, where it is happening, how much it costs, and if there is any sort of registration required. So we organized fields into working tabs to match: About, Contact, For whom, When, Where, Cost, and Signup.

Autosave and Soft Required Fields

Even with fields grouped by tabs, the form can take some time to complete. That's why we introduced autosave and soft save features. When working on a form, the site automatically saves the current draft every few seconds. Service providers can also save a draft to return to later. Fields that are required for publishing, are optional for a draft.

Draft States to Save Work for Later

Service providers have many responsibilities to juggle. It's important that they can start creating an event or program, save it and return to it later before publishing it.

Drupal has powerful workflow states, which we've put to use to help service providers clearly know the status of their content.

A service provider can either save their content as a draft or publish it immediately. If saved as a draft, a banner appears on the page clearly indicating that the event or program is not yet published.

Screencast of save as a draft workflow. Authors can save their work as a draft, bypassing required fields until they're ready to publish.

Authors can also create a draft alongside a published version. This allows new versions to be worked on, while maintaining the current page for site visitors.

Screencast of workflow for having a draft while a previous version of a page stays publishe Authors can have a published version of a page and also have a working draft that eventually becomes the new version.

Help text and character counts for guidance

There are particular ways to write and format content on events and programs to make the most of Find It's features. We provide help text along the way to clue providers in on the best ways to write their content. We also include a character count so providers know if they're staying within the recommended limits for certain text fields.

Bulk Select for Quick Data Entry

Certain fields have many options. In some cases the majority of them apply. For example, many educational events are for all ages up to 18. In that scenario, having a "Select All" option speeds up the data entry process. The Selectize JavaScript library adds elegant toggle and check all options to multivalue fields. We created the CheckboxesJS Drupal project so that other Drupal sites can easily incorporate these features on fields of their choosing.

Conditional Fields to Show Only What is Relevant

Some fields on Event and Programs only need to show under certain conditions. For example, if an event doesn't require registration, then there's no need to worry service providers with a registration link field. Using conditional logic keeps forms simple and streamlined.

Multiple Dates

There was a lot of discussion on whether to support repeating rules or instead allow multiple dates. We decided on multiple dates as experience has shown that even repeating events oftentimes have exceptions (and because the events we import from Cambridge Public Libraries are a list of arbitrary dates rather than a recurring rule, and somehow no one in computer science has created a library to produce a best-effort recurring rule from a list of dates).

Multiple date fields. Multiple date fields allow for flexibility on events and programs that happen more than once.

Easy but Powerful Search for Residents

Find It search is powered by Apache Solr, a popular open-source enterprise search platform. We use its numerous features to make the search results as relevant as possible for site visitors.  It's an ongoing process of tweaks; here are some of the things we've done so far.

Weighted Fields for Relevance

On content with lots of data like the events and programs of Find It, certain fields carry more importance than others. The title of an event, for example, is one of the most important. The transporation notes, on the other hand, carries less significance in search queries. When someone types the keyword "music lesson", an event with music lesson in the title or summary shows up before a program for English lessons.

Synonym Matching

When someone searches "childcare" but a program uses "child care", the search engine should know these are equivalent. The same is true for "STEM" and "science education."

Find It supports synonyms. The site manager can define synonyms so that when site visitors search for a certain term, results with matching synonyms show up as well.

Key Information in Search Results

We used the results of our user research to show the critical information people need to pin point the right opportunities: title, neighborhood, and a short summary.

Filters for Sophisticated Queries

Filters help users narrow a search query down to specific criteria. In our testing, we found that age and neighborhood were most important, especially for low-income caregivers. For those of us that rely on public transportation, events and programs need to be nearby. We placed these filters accordingly towards the top of the page.

Naming conventions in Cambridge are unique, which is true for other cities too. Residents might not know the official name of their neighborhood or live at the border between two. We've included a labeled, clickable map to help users choose the right neighborhood. We built this so that other Find It platforms can upload their own SVG map to show their neighborhood.

Informative Opportunity Pages

Find It comes out of the box with four different types of opportunities: Events, Places, Organizations and Programs.

Organization

The organization serves as the foundation for opportunities posted on a Find It page. Every event and program posted to Find It, belongs to an organization. This helps an organization's page serve as a mini-website. When an event or program is published, it automatically shows up on its organization page.

Organizations can also have "child" organizations, which is helpful for larger groups that might have distinct sub-committees or departments that have sub-departments.

Related programs field. An organization can have a parent - child relationship.

Event

An event is an opportunity with a clear start and end date. When an event is published it shows up on the Homepage, Events page, Search page and on the organization's page.

Visitors can sort opportunities by start date to find upcoming events.

Find It event page. An event's multiple dates is converted into human friendly language.

Program

A program is similar to an event. In fact, most fields are shared between the two. A program though, implies more longevity and commitment than an event. Rather than requiring a specific date or dates, a program can simply be "ongoing." There is the option to include specific dates though.

Find It program page.

Place

In the first version of Find It Cambridge, a new opportunity surfaced that didn't quite fit into the event, program, or organization categories. Parks, neighborhood pools, and other destinations were a good fit for Find It's library of opportunities. They have open hours, but many of the event fields were irrelevant. The same went for Programs. In fact, sometimes these places have events or programs happening at them.

These are community-minded destinations people can go to. In other words, places.

Find It place page.

Bring Find It to Your City!

Find It is helping Cambridge residents connect with activities and services to improve their lives. We would love to help do the same for other cities, counties, and other communities. The platform is open-source and flexible so that communities can customize it to their needs.

Whether you are city IT staff, a developer that works with cities, or are a resident that could use a Find It in your community, we'd love to talk.

Supporting the Drupal Community Through #DrupalCares

Apr 21 2020
Apr 21

Guide for Project Managers working with development teams

Apr 21 2020
Apr 21

PreviousNext: Our ongoing support for the Drupal Association

Apr 20 2020
Apr 20
Apr 20 2020
Apr 20

You may have heard: Google+ is game over.

Officially, this is directly related to a leak of data that potentially impacted 500,000 Google+ accounts.

Businesswise, the attempts of Google to build a social media platform was not successful. If you had a personal account on Google+ and want to download your data before the shutdown, you can download it here.

The shutdown will happen in two phases:

Your website can be impacted by the shutdown of the Google+ API. This API was used by many providers and libraries to be able to login to a website using Google.

Hybridauth on Drupal 7

If your Drupal site is using Google to log n, it's possible that you are still using the Google+ API. A very popular module in D7 to log in using social media is called Hybridauth (to log in with LinkedIn, and Facebook, as well Google). You need to update that library and your Google key.

Here are the steps to keep your Hybridauth working in D7:

Apr 20 2020
Apr 20

Late last year I was lucky enough to attend DrupalCon. At the conference, I heard about Layout Builder, a new tool developed within Drupal 8 that allows editors to configure content more efficiently and easily. 

Having worked with Drupal for several years, I was excited to hear that they were building on an already great foundation and improving the user interface by making it more flexible and sophisticated for editors. 

As reported at DrupalCon, this change is part of the initiative to keep Drupal relevant and impactful, as well as making it easier for content editors and site builders. The Layout Builder will allow website administrators to customise page design more easily without needing a developer.

After recently using Layout Builder on a handful of projects, and with more clients becoming intrigued about this new module taking centre stage in the Drupal community, I thought I’d share my thoughts on its benefits. 

Easy user experience 

As mentioned above, I’ve worked as a Project Manager building Drupal websites for a while now. Within that time I’ve had lots of experience adding content and helping my clients understand their Content Management System. 

When I first started my career and was introduced to Drupal, I initially found the content editing experience somewhat over-complicated, especially for those with little technical knowledge. Drupal has really moved on since then, and the Layout Builder makes big steps forward in providing a better user experience for editors. 

Creating custom visual layouts using a simple drag and drop interface is one of my favourite things about Layout Builder. This easy to use interface is highly intuitive, enabling content editors to create multimedia-rich pages by interweaving text with photography, colour sections, videos and animations in no time. 

Not only is the out of the box authoring experience something to smile about, it works exactly how today’s content editors expect it to, meaning there is also less time spent training them.

Power to the editors 

Over the years I’ve noticed the huge amount of time and effort that editors put into making sure their website content stays fresh and relevant. Therefore, being able to change the look and feel of the pages with ease is important to not only upholding their reputation but bringing in new traffic.

Having a robust page builder that gives power to the editors means they are able to keep pages fresh and test layouts without having to rely on Developers or Project Managers.  

Review your changes whilst building 

While Paragraphs in Drupal are really useful and offer more flexibility to the content editor, building a page using them can be a slow process. You edit the page, add content and then save to view your changes. In comparison, using Layout Builder, users don’t have to save the page to view any changes they’ve made to the content. This saves valuable administration time and offers users a real-time preview of pages. 

Spin up one-off landing pages for campaigns or promotions 

The ability to act with agility and be able to spin up pages quickly on your website ensures for responsive and effective marketing.  

Importantly, the layout you build in Layout Builder then applies to that particular content type as default moving forward, which again saves valuable administration hours. If you then wanted to do something a bit different this could be seen as a limiting factor… but fear not the team at Drupal have listened to our editorial prayers and have given users the ability to override the default settings on content types without a developer’s input. This new feature is a big hit with editors who want to create a one-off landing page for campaigns using different layout settings. 

Accessibility 

It’s vitally important to make sure that websites are inclusive to everyone. For that reason, it's imperative that Layout Builder meets AA compliance with WCAG and ATAG requirements. Thankfully Accessibility is at the forefront of Drupal’s values and part of its DNA. 

Dries Buytaert, the founder of the Drupal CMS shows his commitment to making sure Layout Builder is accessible: 

A key part of bringing Layout Builder functionality to a “stable” state for production use will be ensuring that it passes our accessibility gate (Level AA conformance with WCAG and ATAG). This holds for both the authoring tool itself, as well as the markup that it generates. We take our commitment to accessibility seriously.

It’s positive to see that Drupal contributors are striving to make sure the Layout Builder tool is accessible so that it empowers not just front end users but also the website administrators and content editors. 

To reiterate, Layout Builder is a new module which is only available on Drupal 8. Layout Builder does not require a full rebuild, it can be applied to an existing site, and once it's set up and installed by a developer, it is then a tool that anyone can use in order to start creating your page layouts using blocks, fields and other entities. If you'd like to find out more about Layout Builder, get in touch

Apr 18 2020
Apr 18

What divides us pales in comparison to what unites us.

     - Ted Kennedy

This especially rings true in the time we're living in now.

Because of a much needed country-wide lockdown here in India to prevent the spread of COVID-19 disease, everyone has been locked-out in their homes and have socially distanced themselves form their communities. This, of course, includes the Drupal India community as well. And what better way to bring this community together than an online pan-India contribution weekend organized by Drupal India Association.

First-ever fully-remote pan-India contribution event

Today, on 18th April 2020, Drupal India Association (DIA) conducted it's first online 'Contribution Weekend'. It's the first event of its kind in that it was completely remote and saw participation from Drupalers across India.

We had 116 RSVPs, 22 Mentors, 96 Issues on Contribkanban board.

Thank you, organizers!

Before I talk more about the event, I would like to give a huge shout-out to everyone who was involved in the planning of this amazing event and a special shout-out to Rakhi Mandhania, Surabhi Gokte & Sharmila Kumaran (#WomenInDrupal FTW) for managing this so well. The planning team worked super hard to make this a success. They took time out of their personal and professional lives to have various meetings for issue triaging, communication and marketing for the event, arranging for speakers and much more. Thank you, everyone, who contributed to planning this.

The introductory hour for new/first-time contributors

DIA Contribution Weekend April 2020

We started the event at 8:45 in the morning,(Yes, That's 8:45 in the morning!) with a special session for new and first-time contributors. A big shout-out to all the newcomers and especially the mentors who took time out this early in the morning to volunteer for mentoring them. And boy did we have a lot of mentors! We had a mentor-to-mentee ratio of 5:1! Don't you feel special now mentees!

Main event

The main event started at 10:00 AM with more than 40 people joining in. And more people kept pouring in as the event went by. At a time we had around 70 people in the call!

Drupal India Association

Rakhi, our host, got us started and handed over the call to Mukesh Agarwal. Mukesh is CEO of Innoraft, one of the member organizations of DIA, and introduced us to DIA, its goals and purposes and the role it is going to play in representing the Indian Drupal community to the world. Major goals of DIA can be summarised as:

  1. Becoming an exemplary community leadership organization to the rest of the Drupal world
  2. Developing thought leaders in Drupal who will enhance India’s image in the Drupal world
  3. Becoming a major influencer and enabler in the adoption of Drupal in the Gulf & ASEAN region
  4. Making India an innovation hub for Drupal
  5. Making Impactful Contribution to community & the extended stakeholders of Drupal

Drupal Association needs our help!

Mukesh's introduction was followed by Tanisha Kalia, a representative from the Drupal Association, and she talked about DA and the financial effects of COVID-19 on DA and the larger Drupal community. DA is a very important entity for the whole Drupal community. Among the numerous things they do for the community, a few are:

  • Managing, hosting and conducting DrupalCons
  • Managing the infrastructure of drupal.org

Tanisha mentioned that the majority of DA's funding comes from the revenues of DrupalCon events and the almost inevitable possibility of cancellation/postponement of DrupalCon NA has put DA in a big financial hole. DA needs the community's support. 
Tanisha encouraged community members to either donate to DA directly via #DrupalCares initiative or support DA by purchasing/subscribing to be a member of DA, the cost of which starts from just 15 USD.

As of this moment, DA has reached 25% of its USD 500,000 goal:

DrupalCares Goal DIA Contribution Weekend 2020

As an individual member or an organization, you can help DA sustain the COVID-19 Impact and help reach its goal. Read more here: https://bit.ly/2VyKbmM

Tanisha also talked about the initiative taken by prominent members in the community to encourage donations and financial contributions. Some of these are:

  • From now until April 30, Vanessa and Dries are offering a dollar-for-dollar match for new, individual contributions to the Drupal Association - up to a total match of $100K! More details: https://bit.ly/2KefaPJ
  • Gábor Hojtsy, Acquia, will be donating 9EUR for every module ported to Drupal 9 for a total of up to 900EUR. More details: https://bit.ly/2wS5uaE
  • Jeff Geerling, @geerlingguy, will be donating 2$ for every like on his recent Youtube video for a total of up to 1000USD. Video link: https://youtu.be/fdk7zUwDQdM

A big thanks to all of you! 

Thanks to an amazing community, we had 6 new individual memberships registered before the introduction call finished.

Code-of-Conduct and Tips for the day

Next up Rakhi communicated the Code-of-Conduct to all the contributors and shared the various channels for communication that people could use throughout the day.
We were using channels on DIA slack. There were various channels set up for clear and effective communication:

  1. #new-contributors: For first-timers and newcomers.
  2. #regular-contributors: For experienced contributors.
  3. #issue-reviews-contribution day: For issues ready for review.

This setup was really efficient and was instrumental in keeping people engaged and active throughout the day.

Hussain kicking off the day 

Rakhi then handed over to Hussain Abbas from Axelerant who joined us as guest speaker for the sprint. He started by talking about the power of open source and how open source has in past sustained and in fact grown in face of financial crises. Truly, Open Source is here to stay!
Hussain then proceeded with talking about the path for Drupal core and contributes projects to Drupal 9 and how we can make sure that projects we frequently use and maintain can be made ready for Drupal 9. The Drupal community has created various tools to help us with these:

  1. drupal-check: A PHP CLI tool to get a report of any deprecated code used. This can be integrated into build processes and continuous integration systems.
  2. drupal-rector: A tool for automated deprecation fixes for some common cases.
  3. Upgrade status module: A wrapper module around drupal-check internals that generates a full site report of contributed and custom modules used in the site. This report can be used to assess the overall Drupal 9 compatibility of the modules.
  4. Upgrade rector: A user interface on top of drupal-rector, which also integrated with Upgrade Status.

Read more about these tools in the official Drupal documentation: https://bit.ly/3apZhAA

After Hussain finished his session and just before we were about to wrap up the call, we realized that Rachel Lawson, Community Liason - Drupal Association, has joined us in the call. The presence of the Community Liason from DA also inspired & motivated us to give our best to this contribution weekend.

Let's code!

After all the sessions, it was time for the code contribution now. From the start, one could tell that it was going to be a productive day. Just under an hour we made significant progress and moved a lot of issues to RTBC or Fixed.

Start of the day:

Start of the day

An hour into contribution:

An hour into contribution

Come for the code, Stay for the community!

Highlights of the day were the appreciation that the mentors received from the newcomers and first-time contributors for their help and support. We had a first-time contributor who had been contributing back to the WordPress community in the past as well as trying their hands-on Drupal contribution for the first time. They really loved the experience and appreciated the welcome and support they received from the community. In their own words: "You all (Drupal community) are awesome and doing great work!"

Words of appreciation from a wordpress community member.

With our spirits high we continued the rest of the day with the contribution. And to have the feeling of togetherness in this endeavour, we had the zoom bridge open throughout the day so that people could freely reach out to each other. We even had regular check-ins every couple of hours so that everyone feels connected.

We ended the even at 3:30 pm with a call where everyone shared their experience and appreciation for the community. It turned out to be a very fruitful day of contribution. By the end of it, we were able to move around 50 issues from needs work/active to RTBC/Fixed.Thanks to all amazing mentors for their support to make this happen: 

@azeets@JayKandari@piyuesh23

@yogeshmpawar@AshishVDalvi@joshua1234511

@ankushgautam76@meenakshig489@sonvir249

@abhisekmajumdar@forhemant@dipakmdhrm

@malavya88@nitesh624@durgesh29@subson

@iampratik_dk@vaibhavjain_in@heykarthikwithu

In the end, this would never have been possible without all you contributors who supported the event with writing patches, reviewing/testing them,  showing up to the event motivating all organizers. We hope for your continued participation over the coming Sprints.

My experience

Overall, it was a very productive day! I really enjoyed this new experience of contributing from the comfort of my home and it was also a much-welcomed distraction from the grim time we all have been having because of the current pandemic. I appreciate the opportunity to work with everyone from the community and would love to do it again soon.

Oh yes! That reminds me of something! Guess what? We already have another contribution weekend planned for next month! It's planned for May 16th and I would love to see and work with the community again. Read more about the event here: https://groups.drupal.org/node/535887.

See you all again on 16th next month.

Until then, stay safe!

What You Can Build With Drupal. Examples Based on Droptica Experience

Apr 17 2020
Apr 17

#DrupalCares: Frequently asked questions, answered!

Apr 16 2020
Apr 16
Apr 16 2020
Apr 16

Angie Sabin, the Drupal Association Director of Finance and Operations, answers your frequently asked questions about the #DrupalCares campaign.

Who are you?

Hi! I’m Angie Sabin, and I’ve been with the Drupal Association just over a year.  In that time we’ve made great strides around improving our operational and finance systems with a focus on efficiency and revenue diversification.  I had over 10 years in nonprofit operations, nonprofit management, and accounting under my belt prior to joining the team.  

How much money has been raised as part of #DrupalCares since the official launch?
For the latest progress on the #DrupalCares campaign you can view our landing page. Earlier today, Heather posted a short progress report here.  You can also find a separate landing page for the #DrupalCares match challenge, a $100,00 matching grant funded by Dries and Vanessa Buytaert, available through the end of April. 

What are the options to contribute? How are donations to #DrupalCares recognized?

  • DrupalCon Sponsors…
    … can commit to pledging your full sponsorship to the Association, regardless of what shape DrupalCon takes this year.  This will prevent the gap from getting wider.

  • Drupal Businesses…
    … Can join the supporting partner program, or increase your partner level. Organizations can also make tax deductible donations above and beyond their partnership tier.

  • Individuals…
    … Can join or renew the Drupal Association membership program, or make tax deductible individual donations.

  • Everyone…
    … can help us get the word out! The Drupal Association has deep, deep roots within the community, and tight relationships with those of you who build your livelihoods on Drupal. Unfortunately, there are 10 times as many end-users of Drupal out there who may not even know that the Association exists. Would you leverage your networks to help us reach them?

Your contributions to the #DrupalCares program will be recognized with the #DrupalCares badge on your individual or organizational profile. Organizational contributions will also be recognized with contribution credits in the Drupal.org marketplace.

How are donation dollars, and revenue in general, utilized by the Drupal Association?  Where does the money go?

The short answer is that all money goes toward meeting our mission and the programs we run to accelerate the Drupal project.  Net revenues from DrupalCon, Supporting Partner Program, memberships, digital advertising, technical programs, etc. are used to power 3 key program areas: Drupal.org, DrupalCon, and Drupal Community.  

As you know, drupal.org is the update server for Drupal, home of all localization tools, centralization of the contribution system, CI testing, security advisories, and the first touchpoint for many new evaluators and potential contributors. 

Though it is also our main source of revenue, DrupalCon itself is a mission-centric program bringing people together and invigorating project momentum.  It’s where we host Driesnote which celebrates recent success and details the vision for where Drupal is going. DrupalCon provides business development opportunities for your organization and professional development for your team.  It’s also the venue where Initiative Leads build momentum and recruit talent for key initiatives to move the Drupal product forward. 

The Association has a strong focus on the Drupal Community: connecting the Project to the talent pipeline, centralizing Drupal promotional materials that can be customized and reused by those selling Drupal services, creating spaces for our international Community to coordinate and thrive, and driving Diversity, Equity, and Inclusion initiatives. 

Note that while we talk about three key program areas for Drupal Association operations, they are actually very tightly entwined. Drupal promotion and marketing activities for example take place on Drupal.org and at DrupalCon. Professional development tools for Drupal community members are integrated across all program areas.  Staff are multi-disciplinary, supporting member communications in one moment, improving the evaluator experience on Drupal.org in another, and collectively fielding outreach to the community to support initiatives like mentorship, local DrupalCamps, and Diversity, Equity, and Inclusion efforts. As you can see all of these programs and the people behind them are interwoven. The big ticket items like Drupal.org and DrupalCon are really just the channels that allow us to execute this mission driven work. 

Drupal Association key program areas

Why does an altered DrupalCon affect the Drupal Association so dramatically?  Shouldn’t force majeure make everything ok?

As Dries mentioned in the Lullabot podcast this week, the Drupal Association originated as a need for an entity to organize DrupalCon so historically almost 100% of the Association’s revenue has come from events.  The role of the Drupal Association has evolved over time to be more integral to Drupal itself, and leadership has known for years that relying on event revenue as the main source of income is risky. Much work has been done over the last few years to mitigate that risk and further diversify funding streams, but it will take more time to get to an ideal ratio. Non-event sources of revenue include technical programs like Drupal Steward, digital advertising on drupal.org, individual memberships, and the Supporting Partner program. As of today, approximately 60% of revenue still comes from DrupalCon North America. Therefore, the inability to host the event as planned has a drastic effect on revenue without another immediately available stream of income to offset the approx. $1M gap in net revenue.   We plan to mitigate that gap by hosting an alternate DrupalCon event this summer, but that still leaves the $500k we are fundraising through the #DrupalCares campaign. 

As for force majeure, mid-May fell outside the window to trigger this clause for our event.  Through the skillful work by our small, but mighty, Events team and with the help of trusted advisors, we’ve been able to shed the additional $2million+ liability through venue and hotel contracts.  Some of these were confirmed as recently as yesterday! Without these cancellations confirmed, we weren’t able to move forward with any alternate plans. Force majeure helps us avoid much of the event costs, but it does not replace net revenue needed to meet our break-even budget. 

Why is the range of the potential budget gap that Heather mentioned last month so large ($400k - $1.1M)?  How can there be a $600k variance in the models? Are you and Heather just bad at math?

Well, I’m a seasoned professional with many years of financial analysis experience, and Heather is an industrial and systems engineer by education, so we’re actually pretty darn good at math! :)

We modeled 3 potential scenarios and their impact on the overall budget for 2020; a virtual conference, a postponed conference and cancellation with tickets deferring to Boston 2021.  Regardless of which path we take, we have sunk costs for conference planning to-date which we can’t recover such as staffing, marketing, site visits, and production support. These costs appear in each scenario.

The virtual conference model resulted in the low end of the gap with the assumptions being reduced ticket prices, refunding the difference in ticket pricing, selling a conservative number of new registrations and keeping some sponsorship intact. From the cost side of things: eliminating  in-person costs such as catering but adding in new costs for a virtual conference platform. On the high end of the spectrum we modeled a cancellation scenario in which ticket sales to-date are refunded or deferred to the following year, and we are able to keep sponsorship dollars intact.  From the cost side, all costs are eliminated except those as previously described for pre-event work. At the time we also assumed a cancellation might more drastically impact our ability to retain sponsorships in this scenario

The middle of the road scenario is a postponed conference with significantly fewer ticket sales, but the in-person costs remaining. 

We have also been working diligently to remove any liability for hotel attrition and the venue contract, both of which add to the potential gap. 

In addition to DrupalCon revenue, we have other budgeted revenue areas to meet which we modeled as conservative in all scenarios given the financial impact COVID-19 has had.

How is a postponed conference more costly than a virtual conference?

In a postponement version, we would be holding an in-person conference at the existing venue.  In any in-person scenario, costs are higher because of the infrastructure, people and materials required to deliver an in-person conference.  To be conservative, we assumed registrations at a postponed conference would be significantly lower than what we typically receive due to anxiety around travel as well as the unknown impact of COVID-19 moving forward.  

So essentially, we would burden the higher level of costs without matching revenue projections to get us to a net revenue number that works.

What steps has the Association taken to mitigate costs given the revenue shortfall, and what are you doing to be sure this doesn’t happen again?

Even before the launch of #DrupalCares, we looked at every line item that we could trim to mitigate the revenue shortfall.  We came into 2020 with a very lean, break-even budget, so we’ve had to make very tough decisions including a first round of staffing reductions that took place in March.  We have since revised our 2020 budget, any further costs reductions will have to come in the form of additional staff layoffs. Not only is it heartbreaking to have this affect the dedicated and hard-working Drupal Association team, but when we lose people we also lose the power behind making programs and initiatives happen for the Community.

Frankly, the impact of COVID-19 is the reason we are in this situation.  We are adapting as quickly as possible, as many of you are doing with your organizations as well.  Along with our board of directors, we are making realistic and conservative assumptions in our projections given the financial impact of health crisis, looking to continue diversification of revenues (as has been in our strategic plan before this event), and monitoring conditions that could affect plans for 2021 in an effort to remain as nimble as possible. 

What is the transparency for Drupal Association’s annual budget? How is financial oversight handled?
Every year there is a public board meeting where financial information is presented.  We release our 990 (nonprofit tax return) annually on our website (note that 2019's 990 will go up later this year).  Our financial records are audited biannually by an independent auditor, and reviewed (an “audit lite”) in the off years by an independent auditor.

We operate on a very lean budget.  For 2020, the board voted to approve a net-neutral budget.  This means that all of the revenue we budgeted is directly covering costs and not making a net profit.  In years when we can achieve a net profit, those funds will either be reserved for emergencies and/or invested back into the organization to further meet the mission. 

When will you know if the gap has been covered?
We are updating our financials and cash flow every week to monitor our overall budgeted revenue progress.  We measure the existing budgeted revenue buckets as well as the new contributions coming in related to our DrupalCares campaign.  In order to assess what “counts” toward the gap, we include any contributions, and upgrades to membership or items that weren’t planned as part of the 2020 revenue budget.  If we are able to go with a virtual conference model with lower pricing, for those that have registered and donate the difference of their ticket value, we’ll be able to count that as well.  We want to recognize everyone that contributes or has historically contributed, everything is valuable but in terms of tracking the gap specific to DrupalCon and COVID-19 we are analyzing by looking at revenue that we would not normally have received, ie, “new revenue”. 

What date do funds need to be raised by?
If we are unable to fill the gap by 5/31 we will need to consider a variety of cost cutting measures.  Given we’ve already had a round of layoffs, any additional cuts will critically reduce our ability to execute our existing programs.   As we progress into the campaign and understand the final gap, if any, we will start to make difficult decisions about what will be scaled back.

Are there any benefits you could take advantage of from the CARES act?

We have applied for the Payroll Protection Program through the CARES Stimulus Act for 2.5x our average monthly payroll (we may or may not be funded for the full amount).  If we are approved for the loan, the funds could be used to cover wages and related staff costs in the next 8 weeks. The loan is potentially forgivable. We do not want to rely on this loan to fill a gap since we do not know if we will receive a loan or if a loan would be forgiven.  However, it may be beneficial from a cash flow perspective.

What are the tax benefits for giving to Drupal
Association?

We are a 501c3 non-profit, so depending on how you contribute, your dollars may be eligible for a tax deduction.  The tax deduction for any nonprofit contribution is limited to the excess of the contribution over the fair market value of any items received in exchange for the donation.  Our individual members receive only intangible benefits therefore the entire membership is eligible for a tax deduction. Donations are fully eligible for a tax deduction.  Even your ticket to DrupalCon may be tax deductible.

Additionally, as part of the CARES Act, changes to tax benefits were made to incentivize charitable giving in the U.S. Beginning in 2020 and each year thereafter (this is not limited to only 2020), individuals can take a $300 above-the-line deduction on their tax returns for cash contributions to charities, regardless of whether or not the individual itemizes deductions.  Individuals will be eligible for a deduction up to 100% of adjusted gross income.

We aren’t tax attorneys or licensed tax preparers, nor can we legally advise you on preparing your taxes, so we have to use language like “may be eligible” when providing this information.

#DrupalCares Progress Report | Thank you

Apr 16 2020
Apr 16
Apr 16 2020
Apr 16

Since this blog was originally posted an additional matching grant from a coalition of Drupal businesses has been offered, now tripling the impact. For the latest campaign status visit the #DrupalCares Match Challenge page.

I could not have anticipated the strength of the Drupal Community's reaction when we launched the #DrupalCares campaign.  While we still have much ground to cover, the passion and commitment of this community has never been more clear. Even among open source projects where passion may be the norm, I am certain that Drupal is exceptional.

To date, the #DrupalCares campaign has made great strides: 

Supporting Partner organizations are vital to the support of the Drupal project, and many of them have expanded their partnership with the Drupal Association through additional pledges.  We've been highlighting these key partners throughout the #DrupalCares campaign, and this week I'd like to turn the spotlight to Acquia. Not only is Acquia responsible for the most contributions to Drupal's codebase, they are also a key supporter of Drupal Association programs and broader community initiatives.

Acquia’s CEO, Mike Sullivan, posted a blog today sharing the many ways that Acquia is renewing their commitment to the Drupal Association, specifically, and to the Drupal project in general - going above and beyond to provide additional support in this time of crisis. Examples of this community support include their offer of free training materials, including certification study guides, for those that want to advance their Drupal career or start a new career around Drupal.  Acquia will offer one free Drupal certification exam for six months to the first 150 participants who can use the “DrupalCares” coupon code.  Acquia will also be supporting the top contributors from among their partners with promotion, and has an open offer to support organizations working on the front lines during the COVID-19 crisis.

With organizations like Acquia and the many others who have contributed, as well as the tremendous wave of support from individual contributors, I am more confident than ever that the #DrupalCares campaign will be successful and Drupal will continue to thrive.

Thank you for coming together as one community with one goal - and thank you for your continued help in getting the word out to Drupal users around the world.

How Much Does It Cost To Build a Website With Drupal

Apr 16 2020
Apr 16

April Drupal for Nonprofits Chat

Apr 15 2020
Apr 15
Apr 15 2020
Apr 15

Our normally scheduled call to chat about all things Drupal and nonprofits will happen TOMORROW, Thursday, April 16, at 1pm ET / 10am PT. (Convert to your local time zone.)

No set agenda this month -- we can discuss whatever Drupal related thoughts are on your mind. If you would like to contribute to the conversation, please join us. 

All nonprofit Drupal devs and users, regardless of experience level, are always welcome on this call.

Feel free to share your thoughts and discussion points ahead of time in our collaborative Google doc: https://nten.org/drupal/notes

This free call is sponsored by NTEN.org but open to everyone.

REMINDER: New call-in information -- we're on Zoom now!

  • Join the call: zoom.us/j/7423234165
    • Meeting ID: 742 323 4165
    • One tap mobile
      • +16699006833,,7423234165# US (San Jose)
      • +13462487799,,7423234165# US (New York)
    • Dial by your location
      • +1 669 900 6833 US (San Jose)
      • +1 929 205 6099 US (New York)
  • Follow along on Google Docs: https://nten.org/drupal/notes
  • Follow along on Twitter: #npdrupal

View notes of previous months' calls.

Apr 15 2020
Apr 15

Drupal 8 to 9 Upgrade

This article is the first in a series discussing Who, What, Why, and How Drupal 8 sites can upgrade to the upcoming Drupal 9 release. In a future series will discuss upgrading Drupal 7 sites.

With Drupal 9 scheduled for release in summer 2020, and with both Drupal 7 and Drupal 8 scheduled for end-of-life (EOL) in November 2021, it’s time to think about whether to upgrade Drupal sites to the new version. Upgrading to newer versions in the past was a significant replatforming effort that required a substantial investment and a non-trivial release window. The Drupal 8 to Drupal 9 upgrade is different, though; this is the first major version upgrade that’s reputed to be as simple as a minor point release. Can it really be that simple? Who should upgrade to Drupal 9?

The Easy Path: Upgrading from Drupal 8

Organizations that are already on Drupal 8 are several steps ahead in upgrading to Drupal 9. One of the biggest benefits of upgrading to Drupal 8 is that the platform and core code of Drupal 8 form the basis for Drupal 9. 

Drupal 9.0 doesn’t introduce any new features or new code, so sites that are on the final Drupal 8 point release are essentially ready to upgrade to Drupal 9.0. No big lift; no major replatforming effort; no content migration; just a final audit to make sure the site doesn’t rely on any deprecated code or outdated Composer dependencies. 

Sites that have kept up-to-date with Drupal 8’s incremental updates (see Andrew Berry’s article Drupal 8 Release Planning in the Enterprise) should be ready to go when it comes to core code. Many sites are already using automated tools or workflows to keep them up-to-date on code deprecations for contributed and custom modules. Ideally, you have been continuously replacing older versions of contributed modules with versions that have removed deprecated code, removing deprecated code in your custom code, and dealing with any Composer dependency conflicts. If so, the upgrade effort for your site should be relatively simple. The same is true if you rely on widely-used and well-supported contributed modules and have little custom code.

If you have custom code and use less widely-used contributed modules, but you’ve been paying attention to code deprecations in your custom code and the readiness of your contributed modules, you’re probably in a good position to upgrade. If you have strong test coverage and aren’t relying on any deprecated third-party dependencies, you’re in even better shape. You shouldn’t see substantial changes from Drupal 8 to Drupal 9.0, so even custom code is likely to work without issue as long as it doesn’t rely on deprecated functions or methods that are removed. 

The caveat is that if your custom code or contributed modules rely on older versions of Composer dependencies that are deprecated in Drupal 9 in favor of newer versions, you may need to do some refactoring to make sure that code works with the new third-party dependencies.

Can you stay on Drupal 8 past its EOL?

There should be no reason for anyone on Drupal 8 not to upgrade to Drupal 9.  There will be a small window of time until November 2021, during which the last Drupal 8 release will be supported with security updates. That allows time to make the necessary changes to move to Drupal 9. But after that, you’ll need to make the switch.

When Drupal 6 reached its end of life, there was a Long Term Support (LTS) program, which made it possible to stay on Drupal 6 past its EOL. There are plans to provide an LTS program for Drupal 7; however, there will be no Long Term Support program for Drupal 8 because the upgrade path from Drupal 8 to Drupal 9 is much easier.

If you don’t make the move, you’ll be on your own to deal with security updates and other maintenance and bug fixes for your Drupal 8 code. And that would likely be more expensive and time-consuming than just doing the upgrade.

Prepare to upgrade or start considering alternatives.

With the Drupal 9 upgrade being relatively uncomplicated for sites that are already on Drupal 8, it's easy to recommend that those sites should upgrade. The main question is when, and what other options do you have? Later articles in this series will delve into more detail about how to prepare for the upgrade.

Thanks to the Lullabot team for contributing to this article and to Dachary Carey for drafting it.

Karen Stevenson

Thumbnail

Karen is one of Drupal's great pioneers, co-creating the Content Construction Kit (CCK) which has become Field UI, part of Drupal core.

Apr 15 2020
Apr 15

Your browser does not support the audio element. TEN7-Podcast-Ep-087-Privacy-Implications-of-Google-Analytics-and-What-You-Can-Do-About-Them.mp3

Summary

Do you know about the privacy implications of using Google Analytics, Google Tag Manager and reCAPTCHA? Have you ever thought about what they might be for yourself or for your clients? Our DevOps Engineer Tess Flynn schools us on the problems with these services and discusses some interesting options.

Guest

Tess Flynn, TEN7 DevOps Engineer

Highlights

  • Ye olde site counter
  • The dawn (and ubiquity) of Google Analytics
  • Alternative analytics apps have the same issues as Google Analytics
  • Behavioral aggregation, or how your TV knows what you were doing on your phone
  • If something’s free, you’re the product
  • Simple Analytics and ProtonMail
  • Keybase, another good idea spoiled by VC money
  • Matomo, an excellent candidate to replace Google Analytics
  • Talking to your client about their analytics needs
  • Problems with reCAPTCHA
  • What some free products are doing with your data
  • Moral of the story: educate your clients about privacy implications

Links

Transcript

IVAN STEGIC: Hey everyone! You’re listening to The TEN7 Podcast, where we get together every fortnight, and sometimes more often, to talk about technology, business and the humans in it. I’m your host Ivan Stegic. My guest today is Tess Flynn, who has been on the show a number of times before. She is DevOps Engineer here at TEN7, though today we’ll be talking a little bit about Google Analytics and reCAPTCHA and the vast amount of data that companies like Google have about us, and the implications that has. Welcome back to the show, Tess.

TESS FLYNN: Hello.

IVAN: Hello! So, this episode idea came from a tweet of yours.

TESS: Cause I like raising trouble whenever I get the chance.

IVAN: Yes, and not only do you raise trouble on Twitter, you post it in Slack and see what people say. [laughing] I’m going to read it for our listeners. So, the tweet went out and you said the following: “I really wish that we in the Drupal community didn’t just roll over and play dead when it comes to Google Analytics, Tag Manager and reCAPTCHA. Each of these have serious privacy implications and we often don’t even bring that up with clients.”

First of all, how’s my Tess imitation?

TESS: Well, what you have to do is you have to prevent the top part of your larynx from actually vibrating and then you’ll get closer.

IVAN: [laughing] I don’t think I can do that. [laughing] Okay, so what’s the genesis of those tweets. Why did you say this?

TESS: So, I was talking with some friends on a Mastodon, instance and one of them brought up that they really don’t want to work on any site that ever touches Google Analytics or any kind of privacy-violating technology. And that got me thinking about this, about the number of different sites that I have worked on over my career, all of which use Google Analytics, and it just occurred to me that I never really bothered to consider Is that a good thing? What do they do with this data? Where does it go?

For the most part we really have had two different eras in website tracking, or I suppose I could say three different eras. So the first era is the ye olde-fashioned counter application from way back in the stone age when we hewed GIFs and CGI scripts with bear skins and bone saws and all of those things. [laughing]

IVAN: One GIF per number of the counter so you only had 10 to use.

TESS: And then came a server-side analytics period where you had some built-in application that either was loosely or tightly coupled with either your application or your web server, and then you would log into a private portal and look at your stats and they were all rendered in hideous tables because it’s all about the data, nobody cares ever about the presentation, right?

IVAN: Right.

TESS: And then Google Analytics came along, and it kind of destroyed that entire market overnight. Here was a product which was ostensibly free, which had decent enough UI to not look terrible, it was relatively easy to set up, you didn’t have to pay for anything in order to use it, and once you had it it was very light on your own website, because you didn’t have to go and update a table or a database or a file somewhere. Instead just a bit of JavaScript sent off an API call to another server somewhere and it ended up in your analytics, and that seemed pretty great at the time.

But, now we’re in this era where technology is very pervasive. You can’t really operate in society without some level of technology. I get incensed when a lot of government officials for example, say that you don’t need the internet in order to live in present-day society. And then I go to their website and say, Okay, so how do I apply for unemployment benefits? Oh, it’s a website. I can’t call anybody. Hmm.

Yeah, it’s things like that that start getting on my nerves, and we’re in this period where we have so much data, so much tracking, and so much of a panopticon, that we don’t really realize how much we’ve contributed to it, either directly or indirectly, because we’re only concerned with our little corner of it, and Google Analytics is one of those things that contributes to the larger panopticon.

IVAN: It really is, but it’s so easy Tess, it’s so ubiquitous. Come on.

TESS: Well, so are some of the alternatives that you could use too.

IVAN: Really? Okay, let’s talk about what alternatives there are, because we’ve all heard of other measurement apps, right? There’s Kissmetrics, Heap, Optimizely and Mixpanel, but they all kind of have the same problem, right? They’re these big companies that gobble your data up and then sell them to other companies.

TESS: Exactly. So, they have all the tracking data, all the behavioral data, and then they resell that data, anonymized or not. And anonymizing data is kind of a misnomer. I have worked with private data, HIPAA compliant data before, and I know that there are ways of scraping information even out of that in aggregate, and that is the primary problem with technology today, is there’s a lot of aggregate data. It’s the old story of, Well, I’m not signed in on YouTube on my TV, and then about four weeks later it’s giving you the same exact recommendations that you have on your phone, which you’re logged in on. How does it do that? Behavioral aggregation.

IVAN: Yep. We don’t like that do we?

TESS: All of those solutions that we’ve already talked about are all data that you’re giving away. The reason why they don’t cost you anything or cost you so little is because you’re the product.

IVAN: That’s the old adage isn’t it? If something’s free and you’re not paying for the service, you are likely the product.

TESS: They’re using it as a con in order to get your information, then they turn that information over to advertisers and that’s where the real lucrative income is.

IVAN: Oh man, okay, so let’s talk about some privacy-respecting solutions then. I read about Simple Analytics a while ago, and that seems to be their business model, right? A privacy-respecting analytics platform. They’re based in the Netherlands, you pay them for the service, they promise never to sell your data. They still have your data.

TESS: It’s kind of a ProtonMail approach isn’t it?

IVAN: Exactly. Tell people what you mean by that.

TESS: Okay. So, ProtonMail is, I believe, a Swiss email provider. It provides webmail access, like Gmail, but it has the same kind of marketing pitch, that they exist in a “politically neutral country” insert laugh track here [laughing]. They ostensibly don’t sell your data again, it’s always private and always encrypted. Okay, yeah. How long until you switch CEOs and you break every one of those promises? Because I saw what happened to Keybase. [laughing]

IVAN: Tell me about Keybase before we go on here. What happened to Keybase?

TESS: Keybase was a very promising project where you could share public GPG keys and other security keys in order to facilitate secure communication with other individuals by verifying identity publicly via web servers, social media posts and so on. The problem is that it got all of that Silicon Valley tech-pro VC money into it, and it turned into a blockchain-riddled bitcoin wallet piece of garbage. [laughing]

IVAN: Yeah, I wondered what happened to that. I didn’t do any research into it, but I still have the Keybase account.

TESS: I still have it too, but I don’t even use it anymore. The primary promise that it had in addition to sharing keys was also the ability to have encrypted communication that was fully anonymous that you didn’t have to surrender some critical piece of information force such as your phone number which is what Signal has as a problem.

IVAN: Yeah, yeah, yeah. Well, it sounds like big VC money corrupts good ideas and privacy, is what I think we are saying here in the last few minutes, right? [laughing]

TESS: [laughing] Yeah, but let’s get back to analytics.

IVAN: Yes. I was talking about Simple Analytics and then I was going to tell our audience how you were telling me about something else called Matomo last week. Something that you can host yourself. I did a little bit of research myself on them, but you’ve played around with the product as well. So, Matomo actually goes back to 2007.

TESS: Yeah, it’s one of these second-wave analytics tracker projects inspired by that. But unlike a lot of those projects, this one actually managed to continue to be maintained, continued to get UI and UX improvements during that entire time. And now, it is a very modern application, which has a UI that’s if not comparable, I would almost say better than Analytics.

IVAN: Yeah, don’t say Analytics, say Google Analytics, because as soon as you just keep saying Analytics then it changes it into Xerox and that’s not what we need. [laughing] So, Matomo had another name, they were called “Piwik,” which I think is “kiwi” with the “p” added on backwards.

TESS: Because it’s a PHP application.

IVAN: That’s just brilliant isn’t it? [laughing]

TESS: [laughing] There’s a reason why they probably don’t have that name anymore. Matomo is actually a lot easier to say.

IVAN: And it’s also a word that means “decent” in Japanese.

TESS: That’s actually their third name by the way. They had even older names than that too.

IVAN: Oh, they did? What were they before Piwik?

TESS: Something like “My PHP Analytics” or something like that, going off of “My PHP Web Admin” and the like.

IVAN: Oh, I remember that.

TESS: It goes that far back. [laughing]

IVAN: Wow. Okay. So that’s good actually, right? Okay, so one of their values is transparency, and they say you own the data. So, what does Matomo give us?

TESS: So, there’s two different products with Matomo. There’s Matomo Cloud which is the one that is a hosted solution which is very much like Google Analytics, and Simple Analytics, and all of these other products. It has, again, the ProtonMail promise of Yes, we won’t ever sell your data, but how much do you really trust that. And then they have a fully self-hosted open source version that’s on GitHub. And that’s what got my attention, because now this is an open governance, open release, open code approach that allows me to actually host it myself, maintain the data myself, make sure that it’s subject compliant to my client’s host country laws.

For example, if I have a client in, say Canada, they don’t want to give their data to an American company which might hand it over to the FBI, or whomever. They want to make sure that it stays inside their country that’s subject to their privacy laws. By hosting it within their country, it is automatically subject to their privacy laws. So, that is a key advantage.

IVAN: That is a really important advantage. Okay, so they offer a cloud hosting solution so you could basically do the same thing as Google Analytics, except now it’s hosted by Matomo. Or you could download it and you can get it from GitHub and host it yourself. How hard is that typically, do you think?

TESS: If you’ve installed Drupal before, this is actually about the same, if not slightly easier. It’s not very difficult. You get an archive of—the techies will call that a Dist—that has the full files already set up, you don’t need to run Composer or any other applications in order to compile it. You put it on a web server, you make sure that you have SSL on it and you’re good to go. Give it a database and you’re ready to rock.

IVAN: Really? And it’s a PHP by SQL stack?

TESS: Yeah, and it can run behind FPM or nginx or Mod_php with Apache. All of those work.

IVAN: What about bandwidth usage and the amount of server resources you might need to run this? I would imagine not very much.

TESS: Well, that’s a very good question, because I actually haven’t had the opportunity yet to hard install it myself. I am still just trialing the cloud version, and I spent most of the weekend trying to build a container because, me, obviously, that will install Matomo itself. And their Kubernetes and container support isn’t really where I want it to be, so I’m trying to fix that and make my own version of it. I had some success making the container, but I haven’t gotten around to developing a Helm chart to put it on Kubernetes yet.

IVAN: So that’s the goal for you then?

TESS: That’s the next thing I’m going to try, and then once it’s on my server I’ll be able to see how much server draw that it actually takes. And then if it’s good enough, I’m going to pull Google Analytics off of my site. I’m not going to use it anymore.

IVAN: I think that’s a good aspirational place to be. I’m not sold that we need Google Analytics on our ten7.com site, and I’m also not sold that we have to be talking to our clients about keeping their Google Analytics either. If we can make that Matomo deployment into something we put into our Kubernetes hosting, I think there’s value to that.

TESS: Yeah, I think that’s actually very possible to set that up as a cluster-wide solution that can be multi-tenant. That’s actually one thing that’s really nice about Matomo is, it is a multi-tenant solution out of a box.

IVAN: Tell me about what that means, multi-tenant.

TESS: It means you can configure multiple user accounts with multiple settings.

IVAN: Multiple domain names, right? So, you don’t have to create a new account for every single Google Analytics client. They don’t all have to have their own accounts. You don’t have to worry about them having to create it all, you just create them for your client, they’re all on the same server, and if you wanted to share information between sites and clients, in theory, could you do that?

TESS: I believe so. I haven’t checked to see what the access restrictions are per user account. The indications have suggested that yes, that is possible, but I need to check on that.

IVAN: Okay.

TESS: And if not, it’s not a very difficult application to spin up. Once you have the necessary database behind it, you just spin up as many of those as you need.

IVAN: So, besides us owning our own data, and besides this information not feeding into the greater Google Analytics machine and the Google corporation or Alphabet, what else do we get from Matomo?

TESS: It also has another thing where it does real-time live tracking. So, one thing that is part of Matomo’s marketing pitch is that [Google] Analytics tends to be an aggregate solution, it doesn’t show you the exact data for every individual user that visits your site. Whereas Matomo does have the ability to do that every single time, and you can see what their behavior is, and what pages they go to. And it’s actually on your dashboard in Matomo. It has a nice little river of visits on your site which shows you the technology, the device type, how long they were on your site, their geographic locality if you have that enabled, and also which sites they visit in what order, which is really, really useful.

Another thing is that if you have something like the do not track headers enabled, it doesn’t track you, at all. It doesn’t have any weird, scummy, We’ll track you no matter what. If you decide to opt out of tracking it will not track you. Period.

IVAN: That’s nice. That’s really nice. So, I know that one of the things that Matomo’s using in their marketing to differentiate themselves from Google Analytics is this idea that Google Analytics isn’t actually giving you all the data when you look at your results. And Google refers to it as Google Analytics Data Sampling, and they basically say that “sampling is the practice of analyzing a subset of all data in order to uncover the meaningful information in the larger data set.” So, Google doesn’t actually give you all your data or analyze it.

TESS: Mm-hm. They keep it for themselves.

IVAN: They keep it for themselves. But Matomo lets you have it all right? And it basically crunches the numbers for you so that you don’t miss out on things, something like sampling is going to cause.

TESS: And It also does other things. It also allows you to run campaigns, just like Google Analytics, track particular subsets of your site, and it has its own tag manager built in.

IVAN: So, it actually competes with Google Tag Manager as well. That’s interesting.

TESS: And there’s a Drupal module for it.

IVAN: For Matomo?

TESS: It’s actually pretty easy. You just install it like any other Drupal module, give it a few pieces of information, and then you’re good to go.

IVAN: Version 7 and 8?

TESS: I only installed the 8 version. I’m pretty sure there’s a 7 version. It’s been around for a while.

IVAN: Yeah, I would imagine that there’s a 7 version as well. Wow. And I also saw that Matomo has GDPR and other relevant privacy-respecting banners and cookie options, right, that I don’t think you see with Google Analytics. At least I haven’t seen it.

TESS: It’s a lot more of a respectful solution compared to Google Analytics.

IVAN: So, what do we do as a community then, and not, like you say, roll over and play dead? I suppose talking about it is one thing, that’s why we’re doing this podcast. What else can we do, do you think, to promote privacy-respecting solutions like Matomo?

TESS: I usually would’ve liked to start a conversation with a client like that by asking, “What do you expect to get out of your Analytics? Are you looking for particular pages that are popular? Are you looking to get into particular market sets? Are you just looking for technology compliance information? What are you actually looking for?” Because you don’t want to turn into Google and just say, “I don’t know what we’re looking to do, we’ll take it all.” [laughing]

IVAN: Right.

TESS: Because taking it all is also not particularly great either, and that’s kind of similar to the whole reCAPTCHA discussion as well.

IVAN: And I suppose if the client’s requirements are We run a shop that is a commerce site on Drupal, and it’s integrated with something on Shopify, and we’re using Google Ads, and we’re using AdWords and Facebook ads, and we’re really trying to track and optimize our return on our investment, then switching out Google Analytics for something like Matomo might not be a good business decision for that particular client. On the other hand, if it’s a nonprofit, and they are simply trying to gauge visitors, and they don’t have all these deep integrations with commerce, then Matomo’s probably an easier solution, and probably something that your constituents of the nonprofit, if they’re interested in privacy, will respect.

TESS: Mostly I keep thinking that a lot of clients, like a lot of individuals, don’t realize the privacy implications of the technology that we decide that we’re going to use even though it’s everywhere. Like, one thing a lot of political activists will tell you is Don’t take your phone with you to an event, ever. Not even if it’s turned off, don’t even take it with you.

IVAN: Because?

TESS: Because it can be used to geolocate you either roughly or exactly without your knowledge.

IVAN: And you don’t want that.

TESS: No, because that can be used even if it’s a perfectly legal, perfectly peaceful event, that can be used to get you fired later, and that actually has been shown to happen.

IVAN: In the tweet you sent out you also mentioned reCAPTCHA. So, let’s talk about that. Remind everybody what reCAPTCHA is please.

TESS: So reCAPTCHA is a visual authentication mechanism. It usually presents you with an image or series of images, and you are to select the images that correspond to a list of text. Like, select all traffic cones, for example. And then you select the number of traffic cones, and then it validates you through. Sometimes it will also gather data silently, like your mouse wiggles, if you’ve changed windows, if that behavior is nonlinear or non-regular. As a result, it can actually determine from there that you’re probably a human with relatively high confidence, because you’re not acting like a script along a particular determined programmatic path to interact with the site. This is used as a stopgap in order to keep sites from having registrations from any number of bot accounts and bot systems and click farms in order to make sure that the people who sign up for your sight are “real people.”

There’s a number of different problems with this. One, it doesn’t always work. There are ways of defeating reCAPTCHA even though it’s there and it works most of the time, and that’s a practical concern. The next concern is, it is a nightmare for accessibility. This is nearly impossible to work if you’re using a screen reader. So, if you have visual issues or motor control issues and you can’t use a mouse, this is just going to make your day that much worse. And that is actually a legal problem for a lot of sites.

Then I usually want to ask a client, Why do you need it? What are you hoping to use it for? And then we have to think about that more strategically, Why do you have users registering for your site? Is that really necessary? How else can you track them? Are there other mechanisms that you can use to delay click farms and other bot scripts from actually accessing your site?

An old-fashioned pattern is you sign up for the site, and you don’t get to log in immediately, you actually have to wait for an email. Some sites like publishing firms that rely primarily on commenting in order to make sure that they have high engagement with their readership, that might not be ideal, but it is an effective method even today in order to prevent people from accessing it. And these were all the label-on-the-tin concerns; when you open the can [laughing] to find the worms inside, I have to drop a bombshell on you.

IVAN: Tell me what it is.

TESS: So, let’s say that you are using an iPhone. You pull out your iPhone and you’re using Face ID to unlock it. Perfectly normal, perfectly innocent.

IVAN: Right.

TESS: The same machine vision algorithm that is used to unlock your phone can also be used for target acquisition for a weaponized drone. Do we really want to contribute to that kind of technology, even as consumers?

IVAN: It doesn’t sound like we do. [laughing]

TESS: [laughing] I wouldn’t want to do that either. It is very uncomfortable and does not sit well with my conscience. And reCAPTCHA is a shadow method to do machine vision learning. That is exactly what it’s for. That’s why it gives you pictures. That’s also why Google Voice existed. Because it was a means to train vocal recognition so they could turn that technology around and sell it.

IVAN: Oh, that’s right. I’d forgotten about that.

TESS: Modern technology companies, if they are selling a product and it seems to be free, it’s not free. In fact, you’re two products at the same time. The one that they’re using to gain your behavioral data, and they’re using it to train something else that they’ll also turn around and sell later.

IVAN: There’s a trend here right? I mean, Google Analytics was an acquisition that Google made of Urchin, and we’ve talked about this on the podcast in the past with Dan Antonson. And so that’s why the Google Analytics UA numbers all start with UA, because it used to be called Urchin Analytics. reCAPTCHA is a Google product now. That was previously open source and was acquired by Google in 2009 I think, or 2008, something like that, so, more than 10 years ago. So, reCAPTCHA has really been this machine learning exercise that Google has been producing and using to the benefit of its shareholders and to the detriment of the community that is giving all this information to them for. So, that’s the trend. But, there are alternatives to reCAPTCHA. You mentioned honeypots and different ways of determining whether or not a user is actually human. But there’s this other CAPTCHA alternative that I read about the other day here, hCAPTCHA. Cloudflare is actually implementing hCAPTCHA across its end.

TESS: Cloudflare, that doesn’t make me suspicious already.

IVAN: Yes, so I have my own thoughts about Cloudflare as well.

TESS: I trust Cloudflare as far as I could throw them.

IVAN: At least, at least, it’s not going to Cloudflare and Google if you’re using Cloudflare endpoints now. I suppose that’s a silver lining.

TESS: One of the things that immediately stood out to me with hCAPTCHA when I was looking at it is, it primarily relies on you installing a browser extension, and that’s already nope city for me.

IVAN: Yeah. Is that really how it works? You have to have an extension?

TESS: That was right on the main info page, yeah.

IVAN: So, you can’t actually visit a site that has an hCAPTCHA on it without having the extension on it? That doesn’t seem like it’s going to work.

TESS: No. It’s probably not a very good option. A lot of these technologies are really trying to work around one fundamental thing, which is, we want to keep bots out of particular sites, keep humans in those sites, so that they don’t start posting ads all over the place. And, the thing is, there’s already an existing solution for that that works.

IVAN: What is it?

TESS: It’s called a content moderator. You pay someone.

IVAN: Yeah, okay.

TESS: [laughing] Because human beings are generally pretty good at figuring out if another person’s a human being or posting an ad when they shouldn’t be. [laughing]

IVAN: That doesn’t scale well.

TESS: And the problem is that it costs a lot of money and it’s very stressful and even then it has ethical implications for the moderator themselves. There are distinct studies that have shown that being a content moderator causes a lot of people to have PTSD issues for years and years and years after they leave that particular business, because it’s just that nasty. There are so many nasty people out there.

IVAN: Lots of trolls. I can’t imagine how it is to deal with that kind of PTSD after moderating the content.

TESS: This is why I keep going back to the same question of, Is this really something that you want to do? Is this something that’s really necessary for your site? Because if it’s not, then maybe you shouldn’t. Maybe it’s not that important for your business. Maybe there’s alternative ways of doing it.

IVAN: I like the idea of putting in your phone number or even your email address to log in to receive a special token, maybe it’s even a one-time use token, so that you can comment.

TESS: An email address would probably be my preferred one. Phone numbers have different implications that are also worrying.

IVAN: Yeah, phone numbers are also a dime a dozen too. You can very easily get them and very cheaply use them with APIs as well, so it’s not going to provide you a great deal of protection, right?

TESS: Mmm-mm.

IVAN: Okay. So, what’s the moral of the story here before we wrap up, do you think?

TESS: I think the moral of the story is that we need to talk to clients a lot more about the privacy implications of the technologies that we’re pulling off the shelf. If we’re going to be recommending a particular solution, we really should take a moment to consider, Is this the right solution? Do we actually want to implement that? Do we want to feed these other corporations who do who knows what with this data later, of which we will become subsequently culpable for supporting, even vicariously? So, that’s something that I think we do have to be aware of and we have to raise the knowledge and the understanding of ourselves and our clients in order to bring this forward. And only then can we hopefully make a small, tiny infinitesimal step towards a more ethical industry.

IVAN: I think you’re right. I think the moral of the story here is education. And as the vendors and the service providers in this situation, we should be educating our clients about options and alternatives to Google Analytics. Maybe Google Analytics is okay in some cases. Most cases, probably alternatives would be just fine, and that goes for reCAPTCHA as well.

Well, Tess, I hope this episode didn’t give you PTSD. [laughing] I really appreciate you spending your time with me today.

TESS: I just want people to think about things that they haven’t thought of before.

IVAN: Thank you very much. Tess Flynn is DevOps Engineer at TEN7, and you can find her online as @socketwench. That’s “wench” not “wrench.” And she’s on Twitter, Mastodon on Drupal.org, Patreon and more. You can check out her website as well at deninet.com.

You’ve been listening to The TEN7 Podcast. Find us online at ten7.com/podcast. And if you have a second, do send us a message. We love hearing from you. Our email address is [email protected]. Until next time, this is Ivan Stegic. Thank you for listening.

Keep Your Website Safe And Up-To-Date With Drupal Support

Apr 15 2020
Apr 15
Apr 15 2020
Apr 15

Suzanne is a co-founder of Evolving Web. She's in charge of  design, user experience, and development work. She also provides in-depth Drupal training to clients and thought leadership to the Drupal community.

Suzanne is an elected member of the Drupal Association Board and frequently makes presentations at professional events about development best practices and user experience. She loves traveling around helping teams learn Drupal, and making friends and exploring new cities in the process. 

Apr 14 2020
Apr 14

Read our roadmap to understand how this work falls into priorities set by the Drupal Association with direction and collaboration from the Board and community. You can also review the Drupal project roadmap.

...what a changed world we live in compared to just a month ago. There are so many people across the world who make Drupal what it is, and Drupal.org is where we all come together.

Tim Lehnen (hestenet) | CTO - Drupal Association

Project News

Drupal 9 release window confirmed

The Drupal Core release managers and the Drupal Association team successfully released Drupal 9.0.0-beta in March, which means that we are on track for the June release window of Drupal 9. The target release date for Drupal 9 is June 3, 2020. 

There are several things you can do to help: 

  • Test the latest Drupal 9 beta. At the time of writing, Drupal 9.0.0-beta2 is the latest available version.
  • Test the Drupal 8.9.0-beta. This will be the last minor release of Drupal 8, to come out simultaneously with Drupal 9. 
  • If your organization would like to formally join the beta test program, you are welcome to do that as well. 

It's also time to get your own sites, contributed modules, and custom modules ready for Drupal 9. The community has built a variety of tools to help: 

#DrupalCares Fundraising Campaign

#DrupalCaresAt the end of March we also kicked off the #DrupalCares fundraising campaign. This campaign is specifically targeted at closing the budget gap caused by the impact of COVID-19. 

Drupal is used across the globe - but did you know Drupal is also in use by organizations on the front-line of the COVID-19 fight? The National Institutes of Health, the CDC National Prevention Information Network, Oxfam, Unicef and many others are using Drupal to spread key messages about safety and prevention. The Drupal Association supports these organizations with the infrastructure that hosts new releases and security updates. 

As of today, project founder Dries and Vanessa Buytaert have also announced a #DrupalCares matching campaign. They will match all individual donations, membership upgrades, and membership renewals through the end of April - up to $100,000. We appreciate their leadership and support. 

Drupal.org Updates

Semantic Versioning Available for All Projects

Semantic versioning is now available for all contributed projects on Drupal.org. All versions of Drupal greater than Drupal 8.8.3 are compatible with semantic versioning, and the old version format will continue to be supported until Drupal 10. 

Example of Semver

Using semantic versioning allows contributed project maintainers to follow the same pattern that has been so successful for the core release cycle--enabling core to release significant features every six months. The semver version pattern is: MAJOR.MINOR.PATCH. To summarize how this is used in the Drupal project: PATCH versions should include bug fixes and security releases; MINOR versions can include new features, but should remain backwareds compatible; MAJOR versions can break backwards compatibility and remove deprecated code.

Project maintainers who want to continue to support versions of Drupal at or below 8.8.3 should continue using the old version schema.

Major overhaul to Packaging Pipeline

Drupal PackagingAs we mentioned in last month's update, we've significantly overhauled how Drupal.org's packaging pipeline works, to provide better support for scaffolding Composer installations, to improve introspection, and to make it easier to enhance and maintain as Drupal continues to evolve as a software platform. 

There were several components to this packaging update: 

  • The Packaging Process is now a multi-step Jenkins pipeline, making it easier to observe packaging progress at each phase. 
  • The Packaging Process uses composer create project to ensure that the generated .zip and .tar.gz archives are ready to use Composer. 
  • The subtree splitter step of the packaging process uses a local path repository, so the packaging infrastructure does not depend on GitHub/Packagist
  • Many steps of the pipeline have had performance improvements and reductions in redundant work, to speed up the release process as much as possible. 
  • The use of local path repositories will also help reduce the risk of early embargo break for any security releases. 

Coming soon: Community event listings on Drupal.org

Groups.Drupal.org is well past its prime, but there's still a tremendous need for the community to organize around local events. Especially in light of shelter-in-place orders during COVID-19, we want to support the community in making their virtual events accessible to all.

In collaboration with the Event Organizers Working Group - we've been building a new event listings feature for drupal.org that will allow users to submit their DrupalCamps and other events, include a feed that can be aggregated by tools like Drupical.com, and provide a baseline for future features that will allow us to sunset Groups.Drupal.org

Drupal Association Updates

Contribution Recognition Committee

We know that the community at large is very interested in the future of Drupal.org's Contribution Credit system. The Contribution Recognition Committee has been holding regular meetings since its formation at DrupalCon Amsterdam in 2019, and has been making good progress gathering community input and creating early models for the next generation of the credit system.

The next phase of the committee's work is interviews with key stakeholders in the community, before assembling a list of final recommendations. Please understand:The CRC is on a brief pause as many committee members have found their time constrained for extra child care and other concerns during shelter-in-place orders for COVID-19, but we hope to resume progress as soon as possible. 

You can reach out to the committee here

———

As always, we’d like to say thanks to all the volunteers who work with us, and to the Drupal Association Supporters, who make it possible for us to work on these projects. In particular, we want to thank: 

If you would like to support our work as an individual or an organization, consider becoming a member of the Drupal Association.

Follow us on Twitter for regular updates: @drupal_org, @drupal_infra

#DrupalCares - Announcing the Matching Gift Challenge

Apr 14 2020
Apr 14
Apr 13 2020
Apr 13

A crisis can put a website to the test, and never has this been more true than in the current COVID-19 outbreak. As social distancing and stay-at-home orders are fueling a heavier than usual reliance on online communications, people are looking to state and local websites for up-to-date and authoritative information that’s specific to their area.

Due to excellent planning, perspective, and great partnerships, the Martin County Florida website is serving as a trusted resource for residents on COVID-19 information. 

Solid Foundation for COVID-19 Response

When Martin County Florida partnered with Promet Source last year on the redesign of its website, the need for a flexible content management system (CMS) to communicate essential and up-to-the minute information about hurricanes and weather-related events was well understood. This is one reason why the team at Martin County chose Drupal for their web platform. 
 

The Martin County Florida website homepageMartin County Florida's Drupal website homepage

While the COVID-19 crisis was not anticipated, a high level of foresight and planning has served Martin County well. Since Martin County Florida is in a hurricane prone region, a communication strategy and website plan was already in place, along with yearly practice drills. Residents have relied heavily on the site in the past, and never more so than in the current environment.  

This emergency preparedness perspective has been a strong asset in developing a website that is providing residents with quick and comprehensive information on COVID-19. 

The recent outbreak of COVID-19 has driven a sharp increase in the number of visitors to the site. Between February 23rd and April 5th, 2020, the website had more than 375,000 sessions. A sharp increase than the previous month when COVID-19 was not yet considered by most people in the United States to be a significant threat, noted Jennifer Hagedorn, web content specialist. She added that the recent spike in web visits exceeds that of any other hurricane or crisis event by 44%. 

Surge in Mobile Usage

There has also been a shift in the number of visits to the site from a mobile device. Pre-pandemic, about 40% of visits were from a mobile device, that figure now stands at about 70%, and represents a 200% increase in mobile usage over the previous month. This recent surge in mobile usage has been factored into how information is being presented on the site. 

To facilitate the increased mobile usage, the website content editors at Martin County began to utilize shorter blocks of content and placing content within an accordion-style page layout for mobile users to scroll through multiple topics and quickly locate what they were looking for.
 

Accordion Design Interface for Mobile Friendly ContentAccordion style user interface design ideal for mobile users and to consolidate topics for easy browsing

While there are many metrics indicating that the website is proving itself to be successful in providing residents with the information that they need, Jennifer pointed out that there has been noteworthy feedback by residents who have reached out to express their appreciation for the quality of the content on the website and the frequency of the updates. 

Success Factors

Key among the factors that have driven the success of the Martin County Florida website:

  • The ability to build new pages as needed and create page alerts facilitated a high-functioning emergency response system. 
  • Martin County also set up a single, easy-to-navigate Coronavirus page for all updates and information, which kept track of only the most up-to-date information and deleted all out-dated information and announcements to prevent confusion from website visitors accessing outdated information and articles on their site.
  • The Drupal CMS and WYSIWYG editor provides for a high degree of flexibility, enabling non-technical members of the communications team to perform in-line editing, simply create new pages, add them to the menu, update alerts, and easily embed videos and insert images.
  • The communications team can easily collaborate and are given the appropriate level of permissions for their role so they can help edit content, and those with a higher level of permissions can publish approved content to the live site.
  • The Drupal site uses Paragraphs, a Drupal module, that allows regions of a page to be interchangeable by content editors and does not require complicated code by technical developers to quickly build in functionality and apply custom arrangements on site pages.
  • For purposes of ensuring messaging validity, all communications are consistently branded and include the Martin County seal both on the website and social media images.
  • Website messaging is closely aligned with shareable, social media graphics and posts that are shared to help reach more people with the critical messaging and updates.
  • Residents can sign up for regular text messaging, offered via a third-party provider. The county has branded this capability “Alert Martin,” and these mobile text messages provide critical updates with links back to the website for the full story. This functionality offers one explanation for the increased traffic from mobile devices.
  • The “ALERT” region on the home page provides emergency and informative alerts and is color coded by level of importance: blue for general information, yellow for health and safety updates, and red for critical issues and warnings that require immediate attention. 
Alert system for websiteMartin County’s color coded alert system notifying site users of crucial health, safety, and general county updates.

By including these features on their website, along with many more, their site has received positive feedback from local residents saying they were very happy with Martin County’s COVID-19 emergency response and found the site easy to use and access the information they were looking for. The site was also recognized with industry awards for visual aesthetic and superior digital experience, proving to be not only nice to look at, but also usable and useful in a time of need.

COVID-19 is a moving target that is being attacked on many fronts. Never before have local governments had a greater opportunity and responsibility to ensure that their websites provide a trusted and single source of truth while providing a flexible content management experience for the site’s managers.

Looking to ensure your website is ready to communicate in a crisis? Contact us today.

Introducing new "Update Manager Advanced" module

Apr 13 2020
Apr 13

Managing dependencies for a custom project is not hosted on Packagist or Drupal.org

Apr 13 2020
Apr 13
Apr 10 2020
Apr 10

In the 31 days of Drupal migrations series, we explained different aspects of the syntax used by the Migrate API. In today’s article, we are going to dive deeper to understand how the API interprets our migration definition files. We will explain how to configure process plugins and set subfields and deltas for multi-value field migrations. We will also talk about process plugin chains, source constants, pseudofields, and the process pipeline. After reading this article, you will better comprehend existing migration definition files and improve your own. Let’s get started.

Understanding the syntax of Drupal migrations.

Field mappings: process plugin configuration

The Migrate API provides syntactic sugar to make migration definition files more readable. The field mappings under the process section are a good example of this. To demonstrate the syntax consider a multi-value Link field to store links to online profiles. The field machine name is field_online_profiles and it is configured to accept the URL and the link text. For brevity, only the `process` section will be shown, but it is assumed that the source includes the following columns: `source_drupal_profile`, `source_gitlab_profile`, and `source_github_profile`.


process:
  field_online_profiles: source_drupal_profile

In this case, we are directly assigning the value from source_drupal_profile in the source to the field_online_profiles in the destination entity. For now, we are ignoring the fact that the field accepts multiple values. We are setting the link text either, just the URL. Even in this example, the Migrate API is making some assumptions for us. Every field mapping requires at least one process plugin to be configured. If none is set, the get plugin is assumed. It copies a value from the source to the destination without making any changes. The previous snippet is equivalent to the next one:


process:
  field_online_profiles:
    plugin: get
    source: source_drupal_profile

The process plugin configuration options should be placed as direct children of the field that is being mapped. In the previous snippet, plugin and source are indented one level to the right under field_online_profiles. There are many process plugins provided by Drupal core and contributed modules. Their configuration can be generalized as follows:


process:
  destination_field:
    plugin: plugin_name
    config_1: value_1
    config_2: value_2
    config_3: value_3

Check out the article on using process plugins for data transformation for a working example.

Field mappings: setting sub-fields

Let's expand the example by setting the a value for the Link text in addition to the URL. To accomplish this, we will migrate data into subfields. Fields can store complex data and in many cases they have multiple components. For example, a rich text field has a subfield to store the text value and another for the text format. Address fields have 13 subfields available. Our example uses Link fields which have three subfields:

  • uri: The URI of the link.
  • title: The link text.
  • options: Serialized array of options for the link.

For now, only the uri and title subfields will be set. This also demonstrates that, depending on the field, it is not necessary to provide values for all the subfields. One more thing we will implement is to include the name of the online profile in the Link text. For example: “Drupal.org profile”.


process:
  field_online_profiles/uri: source_drupal_profile
  field_online_profiles/title:
    plugin: default_value
    default_value: 'Drupal.org profile'

If you want to set a value for a subfield, you use the field_name/subfield syntax. Then, each subfield can define its own mapping. Note that when setting the uri we are taking advantage of the get plugin considered the default to simplify the value assignment. In the case of title, the default_value process plugin is used to set a fixed value to comply with our example requirement.

When setting subfields, it is very important to understand what format is expected. You need to make sure the process plugins return data in the expected format or the migration will fail. In particular, you need to know if they return a scalar value or an array. In the case of scalar values, you need to verify if numbers or strings are expected. In the previous example, the uri subfield of the Link field expects a string containing the URL. On the other hand, File fields have a target_id subfield that expects an integer representing the File ID that is being referenced. Some process plugins might return an array or let you set subfields directly as part of the plugin configuration. For an example of the latter, have a look at the article on migrating images using the image_import plugin. image_import lets you set the alt, title, width, and height subfields for images directly in the plugin configuration. The following snippets shows a generalization for setting subfields:


process:
  destination_field/subfield_1:
    plugin: plugin_name
    config_1: value_1
    config_2: value_2
  destination_field/subfield_2:
    plugin: plugin_name
    config_1: value_1
    config_2: value_2

If a field can have multiple subfields, how can I know which ones are available? For easy reference, our next blog post will include a list of subfields for different types of fields. To find out by yourself, check out this article that covers available subfields. In summary, you need to locate the class that provides the FieldType plugin and inspect its schema method. The latter defines the database columns used by the field to store its data. Because of object oriented practices, sometimes you need to look at the parent class to know all the subfields that are available. When migrating into subfields, you are actually migrating into those particular database columns. Any restriction set by the database schema needs to be respected. Link fields are provided by the LinkItem class whose schema method defines the three subfields we listed before.

If a field can have multiple subfields, how does the Migrate API know which one to set when no one is manually specified? Every Drupal field has at least one subfield. If they have more, the field type itself specifies which one is the default. For easy reference, our next blog post will indicate the default subfield for different types of fields. To find out by yourself, check out this article that covers default subfields. In summary, you need to locate the class that provides the FieldType plugin and inspect its mainPropertyName method. Its return value will be the default subfield used by the Migrate API. Because of object oriented practices, sometimes you need to look at the parent class to find the method that defines the default subfield. Link fields are provided by the LinkItem class whose mainPropertyName returns uri. That is why in the first example there was no need to specify a subfield to set the value for the link URL.

Field mappings: setting deltas for multi-value fields

Once more, let’s expand the example by setting the populating multiple values for the same field. To accomplish this, we will specify field deltas. A delta is a numeric index starting at 0 and incrementing by 1 for each subsequent element in the multi-value field. Remember that our example assumes that the source has the following columns: source_drupal_profile, source_gitlab_profile, and source_github_profile. One way to migrate all of them into the multi-value link field is:


process:
  field_online_profiles/0/uri: source_drupal_profile
  field_online_profiles/0/title:
    plugin: default_value
    default_value: 'Drupal.org profile'
  field_online_profiles/1/uri: source_gitlab_profile
  field_online_profiles/1/title:
    plugin: default_value
    default_value: 'GitLab profile'
  field_online_profiles/2/uri: source_github_profile
  field_online_profiles/2/title:
    plugin: default_value
    default_value: 'GitHub profile'

If you want to set a value for a subfield, you use the field_name/delta/subfield syntax. Then, every combination of delta and subfield can define its own mapping. Both delta and subfield are optional. If no delta is specified, 0 is assumed which corresponds to the first element of a (multi-value) field. If no subfield is specified, the default subfield is assumed as explained before. In the previous example, if there is no need to set the link text the configuration would become:


process:
  field_online_profiles/0: source_drupal_profile
  field_online_profiles/1: source_gitlab_profile
  field_online_profiles/2: source_github_profile

In this example, we wanted to highlight syntax variations that can be used with the Migrate API. Nevertheless, this way of migrating multi-value fields is not very flexible. You are required to know in advance how many deltas you want to migrate. Depending on your particular configurations, you can write complex process pipelines that take into account an unknown number of deltas. Sometimes, writing a custom migration process plugin is easier and/or the only option to accomplish a task. Even if you can write a migration with existing process plugins, that might not be the best solution. When writing migrations, strive for them to be easy to read, understand, and maintain. For reference, the generic configuration for mapping fields with deltas and subfields is:


process:
  destination_field/0/subfield_1:
    plugin: plugin_name
    config_1: value_1
    config_2: value_2
  destination_field/0/subfield_2:
    plugin: plugin_name
    config_1: value_1
    config_2: value_2
  destination_field/1/subfield_1:
    plugin: plugin_name
    config_1: value_1
    config_2: value_2
  destination_field/1/subfield_2:
    plugin: plugin_name
    config_1: value_1
    config_2: value_2

Process plugin chains

So far, for every field_name/delta/subfield combination we only have used one process plugin. The Migrate API does not impose any restrictions to the number of transformations that the source data can undergo before being assigned to a destination property or field. You can have as many as needed. Chaining of process plugins works similarly to Unix pipelines in that the output of one process plugin becomes the input of the next one in the chain. When the last plugin in the chain completes its transformation, the return value is assigned. We have covered this topic in greater detail in the article on using process plugins for data transformation. For now, let’s consider an example chain of two process plugins:


process:
  title:
    - plugin: concat
      source:
        - source_first_name
        - source_last_name
      delimiter: ' '
    - plugin: callback
      callable: strtoupper

In this example, we are using the concat plugin to glue together the source_first_name and source_last_name. A space is placed in between as specified by the delimiter configuration. The result of this is later passed to the callback plugin which executes the strtoupper PHP function on the concatenated value effectively making the string uppercase. Because there are no more process plugins in the chain, the string transformed to uppercase is assigned to the title destination property. If source_first_name is ‘Mauricio’ and source_last_name is ‘Dinarte’, then title would be set to ‘MAURICIO DINARTE’. Refer to the article mentioned before for other things to consider when manipulating strings. The configuration of process plugin chains can be generalized as follows:


process:
  destination_field:
    - plugin: plugin_name
      source: source_column_name
      config_1: value_1
      config_2: value_2
    - plugin: plugin_name
      config_1: value_1
      config_2: value_2
    - plugin: plugin_name
      config_1: value_1
      config_2: value_2

It is very important to note that only the first process plugin in the chain should set a source configuration. Remember that the output of the previous process plugin is the input for the next one. Setting the source configuration in subsequent process plugins is unnecessary and can actually make the chain produce unexpected results or fail altogether.

Source constants, pseudofields, and the process pipeline

We have covered source constants, pseudo-fields, and the process pipeline in the article on using data placeholders in the migration process. This time, we are only going to give an overview to explain their syntax. Constants are arbitrary values that can be used later in the process pipeline. They are set as direct children of  the source section. Let’s consider this example:


source:
  constant:
    DRUPAL_LINK_TITLE: 'Drupal.org profile'
    GITLAB_LINK_TITLE: 'GitLab profile'
    GITHUB_LINK_TITLE: 'GitHub profile'
process:
  field_online_profiles/0/uri: source_drupal_profile
  field_online_profiles/0/title: constant/DRUPAL_LINK_TITLE
  field_online_profiles/1/uri: source_gitlab_profile
  field_online_profiles/1/title: constant/GITLAB_LINK_TITLE
  field_online_profiles/2/uri: source_github_profile
  field_online_profiles/2/title: constant/GITHUB_LINK_TITLE

To define source constants, you write a constants key and set its value to an array of name-value pairs. When you need to refer to them in the process section, you use constant/NAME and they behave like any other column present in the source. Although not required, it is customary to name constants in uppercase. This makes it easier to distinguish them from regular source columns. Notice how their use makes assigning the link titles simpler. Instead of using the default_value plugin, we read the value directly from the source constants.

Pseudofields also store arbitrary values for use later, but they are defined in the process section. Their names can be arbitrary as long as they do not conflict with a property name or field name in the destination. The value can be set to a verbatim copy from the source (a column or a constant) or they can use process plugins for data transformations. For the next example, consider that there is no need for the link text to be different among online profiles. Additionally, there is another Link field that can only store one value. This new field is used to store the URL to the primary profile. The example can be rewritten as follows:


source:
  constant:
    LINK_TITLE: 'Online profile'
process:
  pseudo_link_text:
    - plugin: get
      source: constant/LINK_TITLE
    - plugin: callback
      callable: strtoupper
  field_online_profiles/0/uri: source_drupal_profile
  field_online_profiles/0/title: '@pseudo_link_text'
  field_online_profiles/1/uri: source_gitlab_profile
  field_online_profiles/1/title: '@pseudo_link_text'
  field_online_profiles/2/uri: source_github_profile
  field_online_profiles/2/title: '@pseudo_link_text'
  field_primary_profile: '@field_online_profiles/0'

A psedofield named pseudo_link_text has been created. It has its own process pipeline to provide the link text that will be used for all online profiles. When you want to use the pseudo, you have to enclose it in quotes (') and prepend an at sign (@) to the name. The pseudo_ prefix in the name is not required. In this case it is used to make it easier to distinguish among pseudofields and regular property or field names.

The previous snippets is also a good example of how the migrate process pipeline works. When setting field_primary_profile, we are reusing a value stored in another field: the first delta of field_online_profiles. There are many things to note here:

  • The migrate process pipeline lets you reuse anything that has been defined previously in the file. It can be source constants, pseudo fields, or regular destination properties and fields. The only requirement is that whatever you want to use needs to be previously defined in the migration definition file.
  • Source columns are accessed directly by name. Source constants are accessed using the constant/NAME syntax.
  • Any element defined in the process section can be reused later in the process pipeline by enclosing its name in quotes (') and prepending an at sign (@). This applies to pseudofields and regular destination properties and fields.

When reusing an element in the process pipeline, its whole structure becomes available. In the previous example, we set field_primary_profile to '@field_online_profiles/0'. This means that all subfields in the first delta of the field_online_profiles field will be assigned to field_primary_profile. Effectively this means both the uri and title properties will be set. Be mindful that when you reuse a field, all its delta and subfields are copied along unless specifically restricted. For example, if you only want to reuse the uri of the first delta you would use '@field_online_profiles/0/uri'. In none of these scenarios, indicating that you want to reuse something guarantees that it will be stored in the new element assignment. For example, the field_primary_profile field only accepts one value. Even if we used '@field_online_profiles' to reuse all the deltas of the multi-value field, only the first one will be stored per the field's (cardinality) definition.

The Migrate API is pretty flexible and you can write very complex process pipelines. The examples we have presented today have been exaggerated to demonstrate many syntax variations. Again, when writing migrations, strive for process pipelines that are easy to read, understand, and maintain.

What did you learn in today's article? Did you know that it is possible to specify deltas and subfields in field mappings? Were you aware that process plugins can be chained for multiple data transformations? How have you used source constants and psuedofield before? Please share your answers in the comments. Also, we would be grateful if you shared this article with your friends and colleagues.

Top Drupal blog posts from March 2020

Apr 10 2020
Apr 10
Apr 10 2020
Apr 10

Drupal 9 is scheduled to be released on June 03, 2020. This is when the final Drupal 8 minor release 8.9 will also be released. Considering the history of previous Drupal major upgrades, Drupal 9 will relatively be smooth. Thanks to the semantic versioning introduced in Drupal 8. The upgrade to Drupal 9 will just be another minor upgrade with deprecated code removed. Drupal 8 has brought a lot of standardization in the Drupal world, thus allowing Drupal as a project to grow incrementally. 

To put it in simple terms, Drupal 9 contains the same code as of 8.9 + deprecated code removed. Here’s a reference image from the Drupal 9 documentation.

Drupal 9

That is if you reduce Drupal 9 from the last Drupal 8 version (8.8.x) the rest is just the deprecated code and dependency upgrades. 

However, upgrading all the underlying dependencies and removing all deprecated API is a challenging task. Contributors around the world are working hard to get this done especially when the world is facing an epidemic. This is a challenging time for the entire world, yet Drupal contributors have shown their love towards Drupal and are helping it advance to the next release.

This article highlights some of the system requirements, the new dependencies and some of the most commonly used APIs which are going to be removed in 9.0.0.

Let us expedite Drupal’s journey to its 9th version!

| Third-party dependency updates

  • Upgraded Symfony from 3 to 4.4

  • Upgraded Twig from 1 to 2

  • Upgraded CKEditor from 4 and 5

  • Upgraded PHPUnit from 6 to 7

  • Upgraded Guzzle from 6.3 to 6.5.2

  • Popper.js updated to version 2.0.6

| System requirement updates

  • Apache, at least version 2.4.7
  • PHP - at least 7.3. (PHP 7.4 also supported)
  • Database: MySQL (5.7.8), MariaDB (10.2.7), SQLite (3.26), PostgreSQL (10)

| Modules being removed in D9

  • Block_place

  • Entityreference

  • Field_layout -> Replaced by Layout Builder.

  • SimpleTest module has been moved to contrib

  • Action renamed to action UI

| New features in Drupal 9

Drupal 9 will have the same new features as of Drupal 8.9. Thus, Drupal 9.0 will not include new features. But Drupal 9.1 will continue to receive new features as it did for the minor D8 releases.

| Breaking APIs

Following is a list of major APIs which were deprecated in Drupal 8 and will be removed in Drupal 9.0.0. That means if your code contains any of these codes, they need to be replaced before upgrading to Drupal 9. This list is not exhaustive and has been curated by scanning Drupal core codebase for deprecated warnings and sorted according to the usage of these APIs in contributed projects as given Drupal 9 Deprecation Status by Acquia.

  • drupal_set_message() removed

change record

drupal_set_message(), drupal_get_message() functions removed.

Recommendation: 

Use messenger service 

Example: 

\Drupal::service('messenger')->addMessage('Hello world');
  • Drupal::entityManager() removed

change record

EntityManager has been split into 11 classes.

Recommendation: 

Services like entity_type.manager, entity_type.repository, entity_display.repository etc to be used instead. See change records for more info.

Example: 

\Drupal::entityTypeManager()->getStorage('node')->load(1);
  • db_*() functions removed

change record

All of the db_* procedural functions part of the Database API layer have been deprecated.

Recommendation: 

Obtain the database connection object and execute operations on it. See change records for more info.

Example: 

$injected_database->query($query, $args, $options);

$injected_database->select($table, $alias, $options);
  • drupal_render() removed

change record

drupal_render() and drupal_render_root()  functions

Recommendation: 

Use renderer service

Example: 

\Drupal::service('renderer')->render($elements,$is_recursive_call);
  • Methods for generating URLs and links deprecated

change record

Drupal::l(), Drupal::url(), Drupal::linkinfo() etc and many methods for generating URL & links are removed

Recommendation: 

See example. 

Example: 

EntityInterface::toLink();

EntityInterface::toUrl();
  • File functions removed

change record

file_unmanaged_copy()
file_unmanaged_prepare()
file_unmanaged_move()
file_unmanaged_delete()
file_unmanaged_delete_recursive()
file_unmanaged_save_data()
file_prepare_directory()
file_destination()
file_create_filename()

Recommendation: 

Use file_system service 

Example: 

\Drupal::service('file_system')->copy($source, $destination, $replace);

\Drupal::service('file_system')->move($source, $destination, $replace);

\Drupal::service('file_system')->delete($path);


  • Loading of entities removed

change record

node_load(), entity_load(), file_load() etc removed.

Recommendation: 

Use entity_type.manager service to get specific entity storage and then use the respective load functions

Example: 

use \Drupal\node\Entity\Node;

$node = Node::load(1);

or

$file = \Drupal::entityTypeManager()->getStorage('file')->load(1);
  • Functions to view entities are deprecated

change record

entity_view(), entity_view_multiple(), comment_view() etc removed.

Recommendation: 

Use entity view builder handler

Example: 

$builder = \Drupal::entityTypeManager()->getViewBuilder('node'); 

$build = $builder->view($node, 'teaser');
  • Date formats API changes

Some of the functions and hooks removed (change record)

format_date()

hook_date_formats()

hook_date_formats_alter()

system_get_date_format()

system_get_date_format() etc.

Recommendation: 

Use date.formatter service 

Example: 

\Drupal::service('date.formatter')->format()
  • Unicode::* methods removed

Following functions removed (change record)

Unicode::strlen()

Unicode::strtoupper()

Unicode::strtolower()

Unicode::substr()

Unicode::strpos()

Recommendation: 

Use mb_* functions
 
Example: 

mb_strlen();
  • Functions retrieving extensions info removed

change record

_system_rebuild_module_data(), system_get_info() etc are removed

Recommendation: 

Use extension.list.module, extension.list.profile and extension.list.theme services

Example: 

\Drupal::service('extension.list.module')->getAllInstalledInfo();

\Drupal::service('extension.list.theme')->getAllInstalledInfo();


  • drupal_get_user_timezone() removed

change record

drupal_get_user_timezone() function removed.

Recommendation: 

Replaced with an event listener which updates the default timezone

Example: 

date_default_timezone_get();
  • SafeMarkup methods are removed

change record

SafeMarkup::checkPlain() SafeMarkup::format() etc removed.

Recommendation: 

See change record for replacements 

Example: 

Html::escape();

| More Changes/Updates

  • Drupal core themes no longer extend Classy. Read more

  • Drupal core themes, Bartik, Claro, Seven, and Umami no longer depend on Stable.

  • New Stable theme for D9, recommended new themes to be built on new D9 stable theme. Old D8 stable to be removed from core and be moved to a contributed project before D10.

  • A new administration theme, Claro (targeted for inclusion in Drupal 9.1)

  • Drupal 9 won't be able to run updates from 8.7.x or earlier databases anymore, it is necessary for all new updates added to the code base to be tested from a Drupal 8.8.x starting point. Read more

  • Changes to how HTML Elements are inserted via AJAX commands. Read more

  • ZendFramework/* packages have been updated to their Laminas equivalents. Read more

  • PhantonJS based testing removed. Read more

  • The jQuery UI asset libraries not in use by Drupal core have been marked deprecated and have been removed from core in Drupal 9.

  • Drupal 9 will continue to depend on CKEditor 4 and jQuery 3.4.

  • Modules to be compatible with Drupal 8 and 9 at the same time and to support semantic versioning for contributed projects

  • jquery.cookie has been replaced with js-cookie version 2. 

| Conclusion

At this point, 9.0.0-beta2 is released, which means the code for 9.0.0 is stable and is ready for testing by end-users. Now is a good time to test upgrade your existing D8 sites to the latest version of 9.  If you have contributed a project in drupal.org, it is also a good time to check your extensions for D9 readiness. There are several tools which can speed up this process of making your extensions compatible with D9.

Have questions about how Drupal 9 will impact your site? We are here to help. Check out our Drupal Services or send us an email at [email protected]

| Important References

If you see any discrepancies in the information provided above, please let us know.

Thanks!

Apr 10 2020
Apr 10

Drupal 9 is scheduled to be released on June 03, 2020. This is when the final Drupal 8 minor release 8.9 will also be released. Considering the history of previous Drupal major upgrades, Drupal 9 will relatively be smooth. Thanks to the semantic versioning introduced in Drupal 8. The upgrade to Drupal 9 will just be another minor upgrade with deprecated code removed. Drupal 8 has brought a lot of standardization in the Drupal world, thus allowing Drupal as a project to grow incrementally. 

To put it in simple terms, Drupal 9 contains the same code as of 8.9 + deprecated code removed. Here’s a reference image from the Drupal 9 documentation.

Drupal 9

That is if you reduce Drupal 9 from the last Drupal 8 version (8.8.x) the rest is just the deprecated code and dependency upgrades. 

However, upgrading all the underlying dependencies and removing all deprecated API is a challenging task. Contributors around the world are working hard to get this done especially when the world is facing an epidemic. This is a challenging time for the entire world, yet Drupal contributors have shown their love towards Drupal and are helping it advance to the next release.

This article highlights some of the system requirements, the new dependencies and some of the most commonly used APIs which are going to be removed in 9.0.0.

Let us expedite Drupal’s journey to its 9th version!

| Third-party dependency updates

  • Upgraded Symfony from 3 to 4.4

  • Upgraded Twig from 1 to 2

  • Upgraded CKEditor from 4 and 5

  • Upgraded PHPUnit from 6 to 7

  • Upgraded Guzzle from 6.3 to 6.5.2

  • Popper.js updated to version 2.0.6

| System requirement updates

  • Apache, at least version 2.4.7
  • PHP - at least 7.3. (PHP 7.4 also supported)
  • Database: MySQL (5.7.8), MariaDB (10.2.7), SQLite (3.26), PostgreSQL (10)

| Modules being removed in D9

  • Block_place

  • Entityreference

  • Field_layout -> Replaced by Layout Builder.

  • SimpleTest module has been moved to contrib

  • Action renamed to action UI

| New features in Drupal 9

Drupal 9 will have the same new features as of Drupal 8.9. Thus, Drupal 9.0 will not include new features. But Drupal 9.1 will continue to receive new features as it did for the minor D8 releases.

| Breaking APIs

Following is a list of major APIs which were deprecated in Drupal 8 and will be removed in Drupal 9.0.0. That means if your code contains any of these codes, they need to be replaced before upgrading to Drupal 9. This list is not exhaustive and has been curated by scanning Drupal core codebase for deprecated warnings and sorted according to the usage of these APIs in contributed projects as given Drupal 9 Deprecation Status by Acquia.

  • drupal_set_message() removed

change record

drupal_set_message(), drupal_get_message() functions removed.

Recommendation: 

Use messenger service 

Example: 

\Drupal::service('messenger')->addMessage('Hello world');
  • Drupal::entityManager() removed

change record

EntityManager has been split into 11 classes.

Recommendation: 

Services like entity_type.manager, entity_type.repository, entity_display.repository etc to be used instead. See change records for more info.

Example: 

\Drupal::entityTypeManager()->getStorage('node')->load(1);
  • db_*() functions removed

change record

All of the db_* procedural functions part of the Database API layer have been deprecated.

Recommendation: 

Obtain the database connection object and execute operations on it. See change records for more info.

Example: 

$injected_database->query($query, $args, $options);

$injected_database->select($table, $alias, $options);
  • drupal_render() removed

change record

drupal_render() and drupal_render_root()  functions

Recommendation: 

Use renderer service

Example: 

\Drupal::service('renderer')->render($elements,$is_recursive_call);
  • Methods for generating URLs and links deprecated

change record

Drupal::l(), Drupal::url(), Drupal::linkinfo() etc and many methods for generating URL & links are removed

Recommendation: 

See example. 

Example: 

EntityInterface::toLink();

EntityInterface::toUrl();
  • File functions removed

change record

file_unmanaged_copy()
file_unmanaged_prepare()
file_unmanaged_move()
file_unmanaged_delete()
file_unmanaged_delete_recursive()
file_unmanaged_save_data()
file_prepare_directory()
file_destination()
file_create_filename()

Recommendation: 

Use file_system service 

Example: 

\Drupal::service('file_system')->copy($source, $destination, $replace);

\Drupal::service('file_system')->move($source, $destination, $replace);

\Drupal::service('file_system')->delete($path);


  • Loading of entities removed

change record

node_load(), entity_load(), file_load() etc removed.

Recommendation: 

Use entity_type.manager service to get specific entity storage and then use the respective load functions

Example: 

use \Drupal\node\Entity\Node;

$node = Node::load(1);

or

$file = \Drupal::entityTypeManager()->getStorage('file')->load(1);
  • Functions to view entities are deprecated

change record

entity_view(), entity_view_multiple(), comment_view() etc removed.

Recommendation: 

Use entity view builder handler

Example: 

$builder = \Drupal::entityTypeManager()->getViewBuilder('node'); 

$build = $builder->view($node, 'teaser');
  • Date formats API changes

Some of the functions and hooks removed (change record)

format_date()

hook_date_formats()

hook_date_formats_alter()

system_get_date_format()

system_get_date_format() etc.

Recommendation: 

Use date.formatter service 

Example: 

\Drupal::service('date.formatter')->format()
  • Unicode::* methods removed

Following functions removed (change record)

Unicode::strlen()

Unicode::strtoupper()

Unicode::strtolower()

Unicode::substr()

Unicode::strpos()

Recommendation: 

Use mb_* functions
 
Example: 

mb_strlen();
  • Functions retrieving extensions info removed

change record

_system_rebuild_module_data(), system_get_info() etc are removed

Recommendation: 

Use extension.list.module, extension.list.profile and extension.list.theme services

Example: 

\Drupal::service('extension.list.module')->getAllInstalledInfo();

\Drupal::service('extension.list.theme')->getAllInstalledInfo();


  • drupal_get_user_timezone() removed

change record

drupal_get_user_timezone() function removed.

Recommendation: 

Replaced with an event listener which updates the default timezone

Example: 

date_default_timezone_get();
  • SafeMarkup methods are removed

change record

SafeMarkup::checkPlain() SafeMarkup::format() etc removed.

Recommendation: 

See change record for replacements 

Example: 

Html::escape();

| More Changes/Updates

  • Drupal core themes no longer extend Classy. Read more

  • Drupal core themes, Bartik, Claro, Seven, and Umami no longer depend on Stable.

  • New Stable theme for D9, recommended new themes to be built on new D9 stable theme. Old D8 stable to be removed from core and be moved to a contributed project before D10.

  • A new administration theme, Claro (targeted for inclusion in Drupal 9.1)

  • Drupal 9 won't be able to run updates from 8.7.x or earlier databases anymore, it is necessary for all new updates added to the code base to be tested from a Drupal 8.8.x starting point. Read more

  • Changes to how HTML Elements are inserted via AJAX commands. Read more

  • ZendFramework/* packages have been updated to their Laminas equivalents. Read more

  • PhantonJS based testing removed. Read more

  • The jQuery UI asset libraries not in use by Drupal core have been marked deprecated and have been removed from core in Drupal 9.

  • Drupal 9 will continue to depend on CKEditor 4 and jQuery 3.4.

  • Modules to be compatible with Drupal 8 and 9 at the same time and to support semantic versioning for contributed projects

  • jquery.cookie has been replaced with js-cookie version 2. 

| Conclusion

At this point, 9.0.0-beta2 is released, which means the code for 9.0.0 is stable and is ready for testing by end-users. Now is a good time to test upgrade your existing D8 sites to the latest version of 9.  If you have contributed a project in drupal.org, it is also a good time to check your extensions for D9 readiness. There are several tools which can speed up this process of making your extensions compatible with D9.

Have questions about how Drupal 9 will impact your site? We are here to help. Check out our Drupal Services or send us an email at [email protected]

| Important References

If you see any discrepancies in the information provided above, please let us know.

Thanks!

Apr 10 2020
Apr 10

Everything around Test-driven development (TDD) is a very interesting and very motivating world and besides, these are topics with a certain antiquity. So it is very easy to find related contents. But the relationship between this area and Drupal is even more interesting: there are multiple options to implement testing of different types and different orientation (Unit Test, Kernel, JavaScript, functional …) so it can be complex to introduce in this world. Today I want to take advantage of the experience of porting to Drupal 8 a small contributed module to share examples about browser-focused Functional Testing in Drupal 8 (or Drupal 9). We will learn together with simple use cases. It may be a small slice (Functional Testing of a very small features) of a big cake (Testing in Drupal), but it will be a nice entry point. Follow me.

Picture from Unsplash, user National Cancer Institute, @nci

Table of Contents

1- Introduction
2- Arrangements
3- The BrowserTestBase class
4- Basic Scaffolding
5- Your tests
6- Running the test
7- Read More
8- :wq!

1- Introduction

Well, as I said in the obligatory introductory paragraph, everything related to Testing is extensive, very broad and combines an ungraspable conjunction of philosophical-theoretical elements with practical-technical issues…so I guess that’s why I was so happy when in the context of the migration to Drupal 8|9 of the contributed module humans.txt, Pedro Cambra as its maintainer proposed in an issue to provide the module with a certain type of test.
It was a great opportunity to use it as a simple, didactic and intuitive approach.

About the testing we are going to see in this article, it is important to situate it and give it context: we will make some types of browser tests, which are part of the test types that come from PHPUnit classes and resources. How are they related? Let’s see this introduction made by James G. Robertson in the Atlantic BT website:

PHPUnit can handle different types of tests in Drupal core: unit tests, kernel tests, and functional tests. With these three tests, you can confirm the quality and reaction of code on edge cases in different layers. Unit tests test functions, methods, and classes without requiring a database connection to run. On the other hand, kernel tests test the integration of module APIs and require a database connection to be configured to run. Functional tests test the entire system and require more setup than the others. Within the functional tests, there are both Browser and JavaScript tests. In addition to these PHP-based tests, you may also run core JavaScript tests using the Nightwatch framework.

Right, so what we’re going to do in this case has to do with testing actions to be performed in the web interface of our Drupal installation but running through code using classes that already provide methods for replicating “manual actions”. Intuitively… What can we think that we will need? Let’s advance a possible outline of our possible actions:

  1. Maybe, loading a URL.
  2. Perhaps, pressing buttons on a form.
  3. We may need to load values into this former form.
  4. Or also check users, roles and permissions.

So we’ll need classes and resources that allow us to reproduce these actions through code (so that they can be automated). This is just a sketch of our possible needs, as we must first be clear about which features we want to test. So the first guideline shared in this introduction will be: You must know well your needs to test.

Ok, and what is the features we’ll have to test? the Humans.txt contrib module was created years ago to offers a way for building the humans.txt file from within Drupal. For several months we have been working on its portability to Drupal 8 and its features are summarized in two main tasks:

  1. Generating an object as a humans.txt file with values loaded from a configuration form.
  2. Offering to create a link to the object/file in the <head> section of pages.

Mainly, these will be the features that we will have to test.

This article is intended as a theoretical - practical guide to browser-based functional testing for Drupal, based on a certain testing issue history of the Humans.txt module. It includes bugs, more bugs, errors, misses, mismatched versions&mldr;uploads and more uploads, testing bots&mldr;and above all&mldr; a conversational sample of pair between a maintainer and a contributor when working in parallel. It may not be very agile, but it is very real. And I think that here is the most interesting part of the article: exposing the work cycle combined with all the successes and failures.

2- Arrangements

Well, in this section I’ve included the fun little story about how I discovered that I didn’t have the necessary resources installed in the test environment I chose. Pay attention.

It all started when I switched environments and realized that I didn’t have phpunit installed&mldr;I always use DDEV (- Get more info about DDEV - ) as a tool for creating local development environments and in this case I switched to one that didn’t have a Drupal installation with the –dev resources. So if this is your case, take advantage and review the following steps.

Environment

  • First, stop your local apache, if exists: sudo /etc/init.d/apache2 stop

  • Second, up with your DDEV project: ddev start. Make sure your DDEV project is up, running and functioning normally. You’re going to execute PHPUnit from within the DDEV main web container and you’ll use the db container too.

  • Third, go to the web container and install with composer the next resources:

    ddev ssh
    composer require phpunit/phpunit
    
    

Note: In my first iteration, I launched one request like this (opening the phpunit’s version to the latest available) and it caused me a lot of problems when trying to use PHPUnit:

Could not use "\Drupal\Tests\Listeners\HtmlOutputPrinter" as printer: class does not exist
PHPUnit 9.1.1 

What’s the problem? Drupal 8.x is not compatible with PHPUnit in that version (In fact there’are some issues proposing an update to PHPUnit 8 for Drupal 9, like this), so it’s better to uninstall it and request a slightly more limited version, which is not the last one, something around PHPUnit 7 for example.

For the rest of the dependencies, I recognize that I was playing to solve them just following the successive error messages trying to launch several tests again, like Class "Symfony\Bridge\PhpUnit\SymfonyTestsListener" does not exist and many others. In summary I have installed all the following resources in the approximate versions indicated:

composer remove phpunit/phpunit
composer require phpunit/phpunit:^7 # In my case, this installed phpunit 7.5.20
composer require symfony/phpunit-bridge:^3.4.3
composer require behat/behat:^3.4
composer require behat/mink:^1.8
composer require behat/mink-goutte-driver:^1.2

PHPUnit in Drupal Core

The next step was adjust the use of PHPUnit from the file provided by Drupal, present in /project/web/core/phpunit.xml.dist. It’s necessary to make a copy of the file and rename it as phpunit.xml simply, leaving this copy in the same location /core.

Now we need modify two environment variables within the copied file. Beware:

<!-- Example SIMPLETEST_BASE_URL value: http://localhost -->
<env name="SIMPLETEST_BASE_URL" value="http://localhost"/>
<!-- Example SIMPLETEST_DB value: mysql://username:[email protected]/databasename#table_prefix -->
<env name="SIMPLETEST_DB" value="mysql://db:[email protected]/db"/>

As you can see, now we need an internal URL and all the connection data to your database. Ok, in my case as I’m using DDEV I can extract the information from my prompt, just launching ddev describe. Remember that we’re looking for a string formatted as: mysql://username:[email protected]/databasename. And in the DDEV context, by default, all these data are the same: db. Let’s see:

Showing values from a DDEV installation for Drupal

Like an extra, I’ve configured a folder to save results from tests in a HTML format using the variables:

<env name="BROWSERTEST_OUTPUT_DIRECTORY" value="/var/www/html/web/sites/default/simpletest/browser_output/"/>
<env name="BROWSERTEST_OUTPUT_BASE_URL" value="http://migrations.ddev.site"/>

Where BROWSERTEST_OUTPUT_DIRECTORY is used as the directory where the output data will be saved by PHPUnit and needs to be an absolute local path, in my case the result HTML from test goes to /var/www/html/web/sites/default/simpletest/browser_output/ and while BROWSERTEST_OUTPUT_BASE_URL help to register the future links to the saved output, in my case is: migrations.ddev.site and is just the name of the DDEV project that I’m using. This will write all the output from the executed test in the directory using HTML format, and so you can see the pages that the emulated browser visited during the test. Theses HTML pages are saved as files and linked under the URL.

When you launch the test, Drupal 8 will use a simulated browser to execute actions and check assertions, is like an emulator offered by Mink, just a pure headless browser from the installed dependency behat/mink.
Now remember that you must ensure the write permissions in the destiny folder for the user that will launch the test. You can see my whole configuration for PHPUnit using DDEV here in the next gist:

So the second guideline shared in this section will be: tune up your environment well.

When you have available all the libraries and dependencies along with the configuration of the phpunit.xml file, you can try to run the tests that bring many modules through a console execution instruction.
The format of the instruction we will use will be related to the position for ourselves within the path /project/web/ and launch the call to phpunit in a format like this:

../vendor/bin/phpunit -c core modules/contrib/config_inspector 

Where the param -c points to the localization of the phpunit.xm file and it will run all the test located in the marked direction (in this example will be executed tests from the Config Inspector Contrib Module).

Executing test from the Config Inspector Contrib Module

UPDATE (15/04/2020) Well, After a change in the renaming of the /test folder to /tests, the Drupal.org Testing bot has started working and has detected&mldr;that two methods I used in the Test Class were not available, they didn’t exist. At that moment I thought about the version of phpunit I was using (and that I chose and installed myself) and as recommended for Drupal 8 it should be PHPUnit 6.5 and not 7.5 as I am using. So downgrading and adapt the methods to others available&mldr;Which are?

These methods:

$this->assertStringNotContainsString($this->fileLink, $tags, sprintf('Humans.txt link: [%s] is not shown in the -head- section.', $this->fileLink));
$this->assertStringContainsString($this->fileLink, $tags, sprintf('Humans.txt link: [%s] is shown in the -head- section.', $this->fileLink));

Should be replaced by these others, compatible with PHPUnit 6.5:

$this->assertNotContains($this->fileLink, $tags, sprintf('Humans.txt link: [%s] is not shown in the -head- section.', $this->fileLink));

$this->assertContains($this->fileLink, $tags, sprintf('Humans.txt link: [%s] is shown in the -head- section.', $this->fileLink));

From the Assert.php Class available in vendor/phpunit/phpunit/src/Franework/Assert.php

So note to my future me: Drupal 8 + PHPUnit 6.5, man.

3- The BrowserTestBase class

Now we are going to deal with a fundamental PHP class for this test case we are going to make, the BrowserTestBase.php class of the Drupal core test context, path: /core/tests/Drupal/Tests/BrowserTestBase.php (Don’t confuse it with an already deprecated version from the context of the simpletest core module called so BrowserTestBase).

This abstract class extends the TestCase class from PHPUnit (one of the main reasons why we need the use of PHPUnit here) and uses a lot of PHP traits providing many methods from different origins, finally grouped by this base class that we’ll have to extend for our goals. In general terms, we know that abstract classes are classes that are not instantiated and can only be inherited, thus transferring an obligatory functioning to the daughter classes. In this example, using BrowserTestBase we’ll have nearly forty methods available within the class (there are dozens of possible assertions) to perform specific checks on actions to be performed through the browser.

The info from the class is very explicit: You must use the class for functional Drupal test (ok), You have to put your new classes extending BrowserTestBase as a base class in path: /modules/custom/your_custom_module/test/src/Functional and avoid using t() function for translate unless you’re testing translation functionality.

/**
 * Provides a test case for functional Drupal tests.
 *
 * Tests extending BrowserTestBase must exist in the
 * Drupal\Tests\yourmodule\Functional namespace and live in the
 * modules/yourmodule/tests/src/Functional directory.
 *
 * Tests extending this base class should only translate text when testing
 * translation functionality. For example, avoid wrapping test text with t()
 * or TranslatableMarkup().
 *
 * @ingroup testing
 */  

Remember what we said before about knowing well what kind of actions you want to test? Think that this class will provide us with functions that will align possible actions with concrete methods to reproduce mechanics through code.
For example:

As you see, the possibilities are many and only require that we have some knowledge of the capabilities that the BrowserTestBase class offers to automate our tests.

4- Basic scaffolding

I will now share some actions that I need to do before I can start testing functions. As my goal is to prepare test for the contrib module “Humans.txt” the first thing I will do is download the module, install it and make sure I’m in the last commit of the branch I’m interested in testing (the one in version 8.x).

As I’m in a DDEV based context the first thing I’ll do is to access my web container and from there perform the rest of initial commands:

ddev ssh
composer require drupal/humanstxt
drupal moi humanstxt
cd modules/contrib/humanstxt
git checkout 8.x-1.x

Now I’m in the initial point for my job. Another aspect that I must evaluate is if it is convenient (or not) to create a submodule inside Humanstxt as a “test”, with its own installation and testing paths that respond to the same controllers as the main module, or if on the contrary it’s excessive (for now) for the testing of this module. At this moment I think I will create the tests directly, testing against the resources of the main module.

5- Your tests

In our current context, using a new class HumansTxtBasicText which extends BrowserTestBase, every method inside our class will be a unique test by itself, but there will a lot of assertions more to check, cause of in one of our used functions, we’re calling some internal assertions. For example, if we’re using the function drupalCreateUser(['permission name']) from the UserCreationTrait and originally named as createUser , with our direct assertions, we’re going to check other internals just like:

$valid_user = $account->id() !== NULL;
$this->assertTrue($valid_user, new FormattableMarkup('User created with name %name and pass %pass', ['%name' => $edit['name'], '%pass' => $edit['pass']]), 'User login');

Due to this, you’ll see more assertions than yours (or the explicit yours), like this feedback when I’m only using four explicit assertions:

Testing modules/contrib/humanstxt
..                                     2 / 2 (100%)

Time: 1.12 minutes, Memory: 4.00 MB

OK (2 tests, 16 assertions)

Ok, don’t fear. Let’s move on.

The next important point to know is that all your methods should start with the word ‘test’ in lowercase. So any method using this as prefix with public visibility will be automatically detected by PHPUnit and will be launched too.

And there’s one more important thing: any stuff you make within a method/test, won’t live outside it. So, for example if in a test you’re creating a new user, then in the next test you’ll have to create it again. Ok? That’s because every time you run a method/test, each test function will have a full new Drupal instance to execute tests.

Thinking about how to do it in an organized way, I thought about doing it in three functional blocks:

  1. First, check the access control to the configuration form and its fields, that is, check the behavior for different roles (administrators, users with basic and anonymous permissions).

  2. Then, check complementary questions with the information collected in the headers of the answers to requests or the cache tags stored.

  3. Finally, test if the file is created, kept accessible and its link is located in the right way. Everything for different user profiles: administrators, basic and anonymous permissions.

So, I’ll create an new folder within the module with path: humanstxt/test/src/Functional/, using a new class called HumansTxtBasicTest.php with namespace: Drupal\Tests\humanstxt\Functional.

Phase one: Checking access control

We’re going to test if three different users, one with admin permissions, other as a basic user without specific permissions and a last anonymous user can reach the configuration page for the Humans.txt module. Theoretically, only the admin user can access and the basic user or the anonymous cannot reach the config page. Let’s see.

First I’m gonna test the access for admins:

  /**
   * Checks if an admin user can access to the configuration page.
   */
  public function testHumansTxtAdminAccess() {
    // Build initial paths.
    $humanstxt_config = Url::fromRoute('humanstxt.admin_settings_form', [], ['absolute' => FALSE])->toString();

    // Create user for testing.
    $this->adminUser = $this->drupalCreateUser(['administer humans.txt']);

    // Login for the former admin user.
    $this->drupalLogin($this->adminUser);

    // Access to the path of humanstxt config page.
    $this->drupalGet($humanstxt_config);

    // Check the response returned by Drupal.
    $this->assertResponse(200);
  }

I’m using the Url class cause I don’t like work with explicit paths in testing. So, if a path changes in a routing.yml file, the test continues being valid and it will be run normally.

And then, I’ll check the access for the pair of users no-admin, using the same test:

  /**
   * Checks if a non-administrative user cannot access to the config page.
   */
  public function testHumansTxtUserNoAccess() {
    // Build initial path.
    $humanstxt_config = Url::fromRoute('humanstxt.admin_settings_form', [], ['absolute' => FALSE])->toString();

    // Create user for testing.
    $this->normalUser = $this->drupalCreateUser(['access content']);

    // Login for the former basic user.
    $this->drupalLogin($this->normalUser);

    // Try access to the path of humanstxt config page.
    $this->drupalGet($humanstxt_config);

    // Check the response returned by Drupal.
    $this->assertResponse(403);

    // Logout as normal user and repeat the former cycle as anonymous user.
    $this->drupalLogout();
    $this->drupalGet($humanstxt_config);
    $this->assertResponse(403);
  }

Now It’s time to check the access to the fields of the configuration form.

  /**
   * Checks if an administrator can see the fields.
   */
  public function testHumansTxtAdminFields() {
    // Build initial path.
    $humanstxt_config = Url::fromRoute('humanstxt.admin_settings_form', [], ['absolute' => FALSE])->toString();

    // Create user for testing.
    $this->adminUser = $this->drupalCreateUser(['administer humans.txt']);

    // Login for the the former admin user.
    $this->drupalLogin($this->adminUser);

    // Access to the path of humanstxt config page.
    $this->drupalGet($humanstxt_config);

    // The textarea for configuring humans.txt is shown.
    $this->assertSession()->fieldExists('humanstxt_content');

    // The checkbox for configuring the creation of the humanstxt link is shown.
    $this->assertSession()->fieldExists('humanstxt_display_link');
  }

  /**
   * Checks if a non-administrative user cannot use the configuration page.
   */
  public function testHumansTxtUserFields() {
    // Build initial path.
    $humanstxt_config = Url::fromRoute('humanstxt.admin_settings_form', [], ['absolute' => FALSE])->toString();

    // Create user for testing.
    $this->normalUser = $this->drupalCreateUser(['access content']);

    // Login for the former basic user.
    $this->drupalLogin($this->normalUser);

    // Access to the path of humanstxt config page.
    $this->drupalGet($humanstxt_config);

    // The textarea is not shown for basic users.
    $this->assertNoFieldById('edit-humanstxt-content', NULL);

    // The checkbox is not shown for basic users.
    $this->assertNoFieldById('edit-humanstxt-display-link', NULL);
  }

Phase Two: Checking Complementary Items

In this block I would like to check stuff like the headers of the returned object/file or the cache tags.

  /**
   * Checks if the header is right.
   */
  public function testHumansTxtHeader() {
    // Build initial path.
    $humanstxt_path = Url::fromRoute('humanstxt.content', [], ['absolute' => FALSE])->toString();

    // Access to the path of humans.txt object/file.
    $this->drupalGet($humanstxt_path);

    // Check the returned response.
    $this->assertResponse(200);

    // Check if the file was served as text/plain with charset=UTF-8.
    $this->assertHeader('Content-Type', 'text/plain; charset=UTF-8');
  }

  /**
   * Checks if cache tags exists.
   */
  public function testHumansTxtCacheTags() {
    // Build initial path.
    $humanstxt_path = Url::fromRoute('humanstxt.content', [], ['absolute' => FALSE])->toString();

    // Access to the path of humans.txt object/file.
    $this->drupalGet($humanstxt_path);

    // Check the returned response.
    $this->assertResponse(200);

    // Check the related cache tag.
    $this->assertCacheTag('humanstxt');
  }

Phase Three: Checking the delivered content

Ok, in order to check the content file or the link write in the section, I thought to make a central and unified test that would group everything for the different user profiles to be tested. What kind of things am I interested in checking? Well I would like to test if the access to the configuration form is possible for administrators (although this was already tested in a previous test) and then creating a new configuration of the Humans.txt in order to test if the saved content corresponds with the one visible inside the object/file (testing also the access to the file).
Finally I would like to test if the link associated to the meta tag in the section of the pages is being loaded, choosing one at random.

  /**
   * Checks if humans.txt file is delivered for Different Users as was configured.
   */
  public function testHumansTxtConfigureHumansTxtDifferentUsers() {
    // Build initial paths.
    $humanstxt_config = Url::fromRoute('humanstxt.admin_settings_form', [], ['absolute' => FALSE])->toString();
    $humanstxt_path = Url::fromRoute('humanstxt.content', [], ['absolute' => TRUE])->toString();
    $humanstxt_link = '<link rel="author" type="text/plain" hreflang="x-default" href="https://www.therussianlullaby.com/blog/functional-testing-for-browser-in-drupal-using-phpunit//' . $humanstxt_path . '">';

    // Create users for testing.
    $this->adminUser = $this->drupalCreateUser(['administer humans.txt']);
    $this->normalUser = $this->drupalCreateUser(['access content']);

    // Login as admin and get the config form page.
    $this->drupalLogin($this->adminUser);
    $this->drupalGet($humanstxt_config);

    // Load a new configuration for Humans.txt file and submit config Form.
    $test_string = "# Testing Humans.txt {$this->randomMachineName()}";
    $this->submitForm(['humanstxt_content' => $test_string, 'humanstxt_display_link' => TRUE], t('Save configuration'));

    // Check the object/file created.
    $this->drupalGet($humanstxt_path);
    $this->assertResponse(200);

    // Test header.
    $this->assertHeader('Content-Type', 'text/plain; charset=UTF-8');

    // Get page content.
    $content = $this->getSession()->getPage()->getContent();

    // Test the assert- if exists the test_string in the content.
    $this->assertTrue($content == $test_string, sprintf('Test string [%s] is shown in the configured humans.txt file [%s].', $test_string, $content));

    // Test if the link to the object/file is in HTML <head> section for Admins.
    $this->drupalGet($humanstxt_config);
    $this->assertResponse(200);
    $tags = $this->getSession()->getPage()->getHtml();
    $this->assertStringContainsString($humanstxt_link, $tags, sprintf('Test link [%s] is shown in the HTML -head- section from [%s].', $humanstxt_link, $tags));

    // Logout as admin and login as normal user.
    $this->drupalLogout();
    $this->drupalLogin($this->normalUser);

    // Repeat the previous cycle now as normal user.
    $this->drupalGet($humanstxt_path);
    $this->assertResponse(200);
    $this->assertHeader('Content-Type', 'text/plain; charset=UTF-8');
    $content = $this->getSession()->getPage()->getContent();
    $this->assertTrue($content == $test_string, sprintf('Test string [%s] is shown in the configured humans.txt file [%s].', $test_string, $content));
    $this->drupalGet($humanstxt_config);
    $this->assertResponse(403);
    $tags = $this->getSession()->getPage()->getHtml();
    $this->assertStringContainsString($humanstxt_link, $tags, sprintf('Test link [%s] is shown in the HTML -head- section from [%s].', $humanstxt_link, $tags));

    // Logout as normal user.
    $this->drupalLogout();

    // Now a third iteration as anonymous user.
    $this->drupalGet($humanstxt_path);
    $this->assertResponse(200);
    $this->assertHeader('Content-Type', 'text/plain; charset=UTF-8');
    $content = $this->getSession()->getPage()->getContent();
    $this->assertTrue($content == $test_string, sprintf('Test string [%s] is shown in the configured humans.txt file [%s].', $test_string, $content));
    $this->drupalGet($humanstxt_config);
    $this->assertResponse(403);
    $tags = $this->getSession()->getPage()->getHtml();
    $this->assertStringContainsString($humanstxt_link, $tags, sprintf('Test link [%s] is shown in the HTML -head- section from [%s].', $humanstxt_link, $tags));
  }

To check the insertion of the tag in I’ve used a double version of the former codeblock, playing with the values of the element from the Configuration Form: 'humanstxt_display_link' => FALSE and changing it for the next codeblock. This checkbox when false doesn’t insert the link to the humans.txt object/file in , and set to TRUE it will put the tag. The occurrence of the mentioned tag in is managed using a special kind of assertion method from PHPUnit called assertStringContainsString, available in my installed version of the testing framework (7.5) and its inverse for negative versions: assertStringNotContainsString().

So my idea is using the getSession() method from BrowserTestBase class which returns a Mink Session Object. This session object from Mink offers a method called getPage() that can return an object DocumentElement and use its method getHtml() that returns al the HTML code formatted as string. All of this is in the line:
$tags = $this->getSession()->getPage()->getHtml(); and in the variable $tags I’ll save all the HTML response ready to search the link, using the variable as haystack. In any case, all the HTML code from a page is too much for processing, and we don’t need to get all the HTML code, so we’ll get a substring cutting the output from the getHtml() method, extracting up to two thousand characters, enough to evaluate the whole section.

// All the HTML code is too much, we just need to inspect the <head> section.
$tags = substr($this->getSession()->getPage()->getHtml(), 0, 2020);
$this->assertStringContainsString($humanstxt_link, $tags, sprintf('Test link: [%s] is NOT shown in the head section from [%s] and this shouldn\'t happen.', $humanstxt_link, $tags));

Finally you can see this first version of the TestClass as Gist in Github. After uploading the patch to the issue there will surely be revisions and changes to the patch, so I promise to link the final version of the Test class that will be committed to the 8.x-1.x branch.

First version of the Testing Class in Gist.

UPDATE (14/04/2020) After the first feedback from the maintainer, I have taken all his indications and prepared a new refactored version of the Test class. I’ve reduced, adapted and simplified several things following the feedback from the maintainer. Now the class is using the setUp() method, the numbers of methods was reduced from eight to five, the assertions from hundred-six to sixty-four and the codelines from almost three hundred lines to hundred and forty lines of code.

I’m also using simple paths, except in the path to the humans.txt file to be inserted in the section as link, since originally the absolute path to the file is being saved: see this Issue and the last uploaded patch.

This second reduced version of the class seems to pass the tests well and gives positive results in all tests and assertions.

Executing the second version of the test for Humans.txt module

Many thanks to Pedro Cambra from Cambrico for the feedback and revisions, I am very grateful.

Here is the second version of the test class after refactoring:

Second version of the Testing Class in Gist.

UPDATE (15/04/2020) After starting the Drupal.org testing bot and checking that my version of PHPUnit was not the one needed for the Drupal core, and has been necessary to make changes in two methods. The semantics of the messages in those methods were set for testing by me and were confusing, were also wrong. They have been changed and now the tests are going well.

This will be the final version of the class of Tests that will be committed to the repository:

By the way, with this last commit, the migration to Drupal 8 of the contributed module Humans.txt has been completed. The module has been finally ported to Drupal 8 | Drupal 9. Humans.txt Releases. Mission accomplished.

6- Running the test

Well, and now with our test stored and the phpunit configuration initially resolved, it’s time to run our test and observe the results. To do this, in my case I’m located in /project/web/ and with phpunit.xml placed in /project/web/core/, I launch the instruction:

../vendor/bin/phpunit -c core modules/contrib/humanstxt

Which throws all the test located inside the humanstxt contrib module&mldr;

Executing test from the Humans.txt Contrib Module

As we can see, we’re executing eight tests with hundred and six assertions. Due to our phpunit.xml configuration, these test are writing over sixty four HTML files based in results using the browser emulator from Mink, thanks to which you will be able to reproduce certain actions by opening the files in your web browser.

In the next caption we can see the HTML output nº64 generated from one of last assertions in the code, just when an anonymous user try to get the Humanstxt configuration form and receive and error message with code HTTP 403 ‘Forbidden’:

Executing test from the Humans.txt Contrib Module

As you can see, you can move through the files using the Previous | Next Links in header, connecting all the secuence of HTML results from tests.

7- Read More

8- :wq!

[embedded content]

Pages

About Drupal Sun

Drupal Sun is an Evolving Web project. It allows you to:

  • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
  • Facet based on tags, author, or feed
  • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
  • View the entire article text inline, or in the context of the site where it was created

See the blog post at Evolving Web

Evolving Web