Sep 20 2019
Sep 20

Drupal is well known for its stability and security out-of-the-box. However, we all know how dangerous the internet can be with all the risks of having your site attacked. There are particular situations in which an extra security measures are needed. This tutorial will deal with specific ways to secure your Drupal site, and the modules involved in this process. 

Let’s get started!

The Out-of-the-Box Features

The Drupal theming system has been improved in Drupal 8 with the use of the templating language, Twig. Twig templating has many advantages over PHP templating. One of them is that it is much easier to visualize on code. Another advantage is the fact, that all variables passed to the template are auto-escaped automatically. That minimizes the risk of a variable including a result that could break the HTML, and therefore your site. The risk is also minimized because you do not have to write custom PHP code in your templates anymore.

The PHP input filter module was part of the Drupal core in Drupal 7, but in Drupal 8, it is a contributed module. This measure eliminates vulnerabilities.

Drupal 8 implemented the trusted hosts configuration. This allows you to associate the Drupal codebase with a specific domain, to prevent online spoofing attacks

Due to the new programmatic approach of Drupal, modules can be enhanced in their functionalities by adding plugins. These plugins add behaviors to the module. The code stays clean and easy to analyze.

The use of Composer as package manager opens the door to new development possibilities for Drupal Open Source software, and it also helps to maintain all modules and keep their dependencies up-to-date and working properly. This is a key factor for the stability of Drupal systems.

How to Enhance Security on Access - Contrib Modules

190919 drupal security

There are two alternatives:

  • Lock attack vectors within the system
  • Limit vulnerabilities by restricting or changing system functions and operations, taking the responsibility from the user

Here are some modules, which provide this kind of functionality:

Automated Logout

Link: https://www.drupal.org/project/autologout

This module allows you to configure the time period for users to be logged in on your site. You can configure different time periods for different user roles, assuming that a user with a higher role, i.e. with access to more valuable information or resources on the site, would need to log in more frequently, than a user with a lower role. 

Session Limit

Link: https://www.drupal.org/project/session_limit

With the Session limit module, you can determine the limit of simultaneous sessions per user on 2 different browsers. If you set the session limit to 1 and open a new session in another browser, the session of the first browser will automatically expire.

Login Security

Link: https://www.drupal.org/project/login_security

Login Security allows site administrators to authenticate user only from valid IP addresses, that way, it is possible to grant or restrict access to your site to whole countries if needed, just by configuring the right domain. Login security restricts also the number of login attempts, so brute force attacks can be avoided. An extra feature of this module is the fact that it can hide Drupal login error message. The attacker will have another problem trying out if the account he wants to access even exists. 

Security Kit    

Link: https://www.drupal.org/project/seckit

Security Kit allows developers to change response headers. This is useful for very specific security needs on sites with high probabilities of Cross-site scripting, Cross-site request forgery, and origin driven attacks.

Honeypot

Link: https://www.drupal.org/project/honeypot

Honeypot prevents robots from filling out the forms of your site, by determining if the user is human or not. It uses two identification methods. One of them is by using a hidden form field, which is not visible to humans if the field has been filled out, Honeypot detects this and blocks the submission. The other method is by taking the time in seconds, in which the form has been filled out. A low value here (2-3 seconds), would speak for a robot, so the module would block  this submission too.

CAPTCHA 

Link: https://www.drupal.org/project/captcha

A captcha is a challenge-response test, to determine whether the user is human or not. This way, the goal of blocking fake form submissions, is achieved since robots will not be able to decipher the captcha text or image.

Auditing - Checking Procedures

190919 drupal security 001

It is always important to permanently review the system logs. This is even more important if your site has already been compromised. The analysis of all this data will help you also to track transactions within your system and perform checks on the ongoing state of the system. A rule of thumb is to log as much data as possible - always!

Some of the modules, which provide this type of functionality are listed below:

Site Audit

Link: https://www.drupal.org/project/site_audit

Site audit performs a static analysis of the whole system, against a set of recommended configurations. The module also stores reports of every audit. By performing this check, you as a developer can be sure and confident that your site is meeting the required security standards.

Security Review

Link: https://www.drupal.org/project/security_review

Security review, like Site audit,  makes also an analysis of the system, but this time, against a set of potential security implications on your site, like file permissions, text formats, and potentially malicious PHP or JS code on the frontend. It also stores reports. 

Login History

Link: https://www.drupal.org/project/login_history 

Login history adds a report to the database with a log of every user login, including timestamp, IP address, and user agent information. As stated before, it is always good to log as much information as possible.

Authentication Measures

190919 drupal security 002

Often, the importance of the information, implicate a reduction of usability. Users take instead of that the extra hassle of these additional authentication procedures.

The modules that you can use for this purpose are listed below: 

Two-factor Authentication  

Link: https://www.drupal.org/project/tfa

Two-factor Authentication provides an additional verification step, which ensures the integrity of the user authenticating. It also provides an API to support different plugins (provided by modules), which integrate several authentication services, like the Google Authenticator module.

simpleSAMLphp Authentication 

Link: https://www.drupal.org/project/simplesamlphp_auth

This module will allow you to replace the default Drupal login with a single-sign-on implementation. The module communicates with identity providers for authenticating users. That way you can validate the identity of the user through a service like Twitter, Facebook or Google

Password Policy

Link: https://www.drupal.org/project/password_policy

The Password Policy module defines a set of rules to force users to have strong passwords. It also forces users to change their password from time to time, depending on the configured options. Password Policy provides an API to define your own set of rules. 

Password Strength

Link: https://www.drupal.org/project/password_strength

This module provides a star rating widget, for users to test the strength of their passwords. You can leverage the API of Password Policy to force users to enter a password with a high rating (minimum rating).

Encryption

190919 drupal security 003

The data at rest (data that is not being used) should be encrypted to prevent attacks of all types. The encryption provides solutions at all levels:

  • Hosting
  • Server
  • CDN
  • Drupal system

As a rule of thumb, and to guarantee the highest security, the best way is to perform encryption at a low level of this stack.

You should always use HTTPS and SSL for all your web traffic, and you should also ask your hosting or cloud provider about full disk encryption.

Some useful modules are:

Key

Link: https://www.drupal.org/project/key

The Key module manages the system and API keys. It provides an API with options to store and retrieve sensitive keys like API keys or encryption keys. The site admin can decide where to store these keys in the system. Examples of API keys are the public keys for services like AWS, PayPal, or MailChimp.

Encrypt

Link: https://www.drupal.org/project/encrypt

The Encrypt module is an API, which provides a common algorithms utility to encrypt and decrypt Drupal application data. Its API is leveraged by many modules, which make use of algorithms to encrypt/decrypt (Real AES, Diffuse), modules to encrypt data, like Field Encrypt and File Encrypt and other modules for more specific use cases, like Pubkey Encrypt.

File Encrypt

Link: https://www.drupal.org/project/file_encrypt

This module focuses on the file system, it performs a check of the encryption on all files.

Field Encryption

Link: https://www.drupal.org/project/field_encrypt

This module encrypts data at the field level, that is, it encrypts field values.

 DevOps (Developer Operations)

190919 drupal security 004

The development process is critical to proactively maintaining the security. You should always use code repositories and pull requests for all your files. Other measures imply performing regular code reviews, tag each one of your code releases, keep the site code always up-to-date (this includes JS libraries), and try always to avoid manual procedures because these entail mistakes. Always try to automate your scripts and use a tool like DRUSH.

Some of the relevant modules in this category are:

Coder

Link: https://www.drupal.org/project/coder

This module has PHP Code Sniffing extensions to test the code on your site and compare it against the coding standards of Drupal.org. Coder will not do anything on your UI. It is rather a command-line tool.   

Hacked

Link: https://www.drupal.org/project/hacked

The Hacked module scans the code of your core and contrib folders and compares it against the code hosted at Drupal.org. It shows the differences between both codebases, so you can take the proper measures regarding your code. 

Backup and migrate

Link: https://www.drupal.org/project/backup_migrate

This is a classic among Drupal modules. Backup and migrate performs regular backups of the codebase and of the database, so you can restore them, for example on a fresh installation. This is very useful if your system has been compromised, and you want to restore it. 

Environment

Securing the infrastructure, in which the system is hosted, is as important as securing Drupal itself. Try always to mitigate attacks before they happen. This module list is supposed to help you with that purpose. 

  1. Use a coding workflow - making sure that the best code ends at the production environment. 
  2. Make a very detailed analysis of the logs - there are very useful tools for this matter, like Splunk or the ELK stack.
  3. If it is possible, try to use cloud-based environments - these are more secure than hosted environments. 
  4. Try to use CDNs every time you can - this acts as a firewall preventing malicious attacks early in the process.
  5. Make sure you have an up-to-date failover environment and test what would happen in case of a failover. 

Please, leave us your comments and questions below.  Thanks for reading!


About the author

Jorge lived in Ecuador and Germany. Now he is back to his homeland Colombia. He spends his time translating from English and German to Spanish. He enjoys playing with Drupal and other Open Source Content Management Systems and technologies.
Sep 19 2019
Sep 19

You will notice that, along with thousands of websites around the world, Drupal.org posted a banner message this week declaring we are opting in to a global Digital Climate Strike on 20th September.

Will @drupal website add the #DigitalClimateStrike banner? @baluertl requested it and mocked up a visual... https://t.co/XcIj9Gf173 pic.twitter.com/Zl0ctyc7G6

— ClimateAction.tech (@climateActTech) September 18, 2019

Of course, because Drupal.org is an essential service to over a million websites around the world, we have to be sure that we still allow them all to continue to access resources here. As such, the full page banner that will appear on websites on the 20th September will be configured to allow visitors to cancel it, should they need to.

Fundamentally, the Drupal Association wants to be a good steward of the environment and recognizes the impact that technology has on environmental issues. We are committed to exploring ways for the Drupal project to reduce its carbon footprint and to become a more eco-friendly platform. Today, we stand with others in the technology industry to educate and inform the general public about some of the ways that the tech industry can support environmental causes.

If the environmental sustainability of Drupal websites is a subject as close to your hearts as it is to ours, you might like to know that recently a #sustainable Slack channel was created for discussion on the topic.

Sep 19 2019
Sep 19

The Drupal 8 View Unpublished module is a simple module that provides you a permission to allow specific roles to view unpublished content. It’s a useful module to help you build out your content editor workflows on your Drupal 8 website.

Download and install the View Unpublished module just like you would any other module.

composer require drupal/view_unpublished

After installing the module, go to the permissions page and search for the View Unpublished section. Here you can set the permission for who can view unpublished content on the site. After setting the permissions you will likely need to clear the cache and rebuild the access permissions before your users will be able to see the unpublished content.

That’s all there is to this module!

Sep 19 2019
Sep 19

Ubercart, once the go-to commerce option for Drupal and the precursor to Drupal Commerce, is slowly fading away. Its usage has been declining for years and a stable Drupal 8 release will never happen. Even one of the original creators has moved on to support a new Drupal ecommerce solution instead of continuing on with Ubercart. In you’re running an ecommerce site that uses Ubercart, this post is for you. Our goal is to show you why you should consider moving off of Ubercart now instead of waiting until it finally reaches end of life.

The decline of Ubercart today

As mentioned in the introduction. Ubercart usage has been declining for years. The Drupal 7 version of the module is where it saw most of its success with usage peaking in 2014/2015, but usage has been continuously dropping since then. The following graph is a snapshot of Ubercart’s usage history as recorded on Drupal.org.

Ubercart usage history
Ubercart usage history (source)

Ryan Szrama, one of the original creators of Ubercart, moved away from it and started the Commerce module for Drupal as a replacement. Since then, the majority of the ecommerce community around Drupal has also moved along with him making Drupal Commerce the new go-to option for ecommerce built on Drupal. Not only does Commerce now have more installs for both Drupal 7 and Drupal 8, but it is also a much more active development community.

Usage-statistics-for-Drupal-Commerce-1
Commerce usage history (source)

Ubercart and Drupal 8

The Ubercart module has never moved over to a proper Drupal 8 release. Development is stuck in alpha and without a new release in over 3 years, there is never going to be a stable Drupal 8 release.

What “alpha” means

In software development, alpha is a term given to a software release that is still very much in development and not ready for production. Here’s the description of alpha from Drupal.org.

alpha: Most reported errors are resolved, but there may still be serious outstanding known issues, including security issues. Project is not thoroughly tested, so there may also be many unknown bugs. There is a README.txt/README.md that documents the project and its API (if any). The API and DB schema may be unstable, but all changes to these are reported in the release notes, and hook_update_N is implemented to preserve data through schema changes, but no other upgrade/update path. Not suitable for production sites. Target audience is developers who wants to participate in testing, debugging and development of the project.

In contrast, the Drupal Commerce module has had many full production-ready releases for Drupal 8 and follows a release schedule for bug fixes and new features. The group behind Drupal Commerce is actively developing the core software and the wider community is also active in supporting the project.

Ubercart and Drupal 7

What Ubercart development still happens focuses on maintenance of the Drupal 7 version only. The catch here is that Drupal 7 reaches end of life November 2021, which will likely spell the effective end of Ubercart as well. If you’re using Ubercart and Drupal 7 together and you want new features and active development, that realistically ended years ago when the majority of the contributor community moved away from the project.

Here’s a couple snapshots of the commit history for both the core Ubercart module and the core Drupal Commerce module. A commit is a term given to code changes that have been added to the module. Commits are typically code improvements, new features, bug fixes and security updates that have been written, tested and approved for release.

ubercart-commit-historyUbercart commit history

drupal-commerce-commit-history
Commerce commit history

When looking at the graphs above, it’s important to know that it’s common to see number of commits trailing off over time. This is because the majority of the core software is built early on and so fewer commits are made over time as development of the core ramps down. What is important to see is that development of Drupal Commerce over Ubercart is still continuing, meaning new features and code improvements are being actively made to the core Commerce software but not to Ubercart.

Another point to note about these graphs is that when commits are ramping down to the core software, development efforts are likely being moved to community built extensions. This data isn’t reflected in the graphs above. The community built extensions is the ecosystem of new add-ons and features that aren’t found in the core software. In the case of Ubercart, this community development is very small and limited whereas for Drupal Commerce community is very active and engaged.

Where to go from Ubercart?

You’ve probably guessed this already, but the clear path moving away from Ubercart is to Drupal Commerce. Commerce is the Ubercart replacement and it’s capable of so much more. It’s also Drupal 8 ready and will provide a painless transition to Drupal 9, when that happens.

Commerce improvements over Ubercart

The following is a list of improvements Commerce for Drupal 8 has over Ubercart:

Drupal 8 improvements over Drupal 7 include:

  • Robust caching and performance for authenticate or unique users, very important for any ecommerce site
  • Drupal’s new rolling release schedule, no more large updates between versions makes updates easier
  • Modern object oriented design, which makes testing, extension and use of 3rd party libraries easier. Commerce follows all of the architectural improvements for Drupal 8 and has, in some cases, lead the way by innovating first.

Commerce improvements over Ubercart include:

  • More secure payment architecture. Commerce encourages the lowest level of PCI risk possible and enforces good practices with it’s payment API, compared to Ubercart’s primarily DIY payment model.
  • Proper variation based product model with unique SKUs for each variation
  • Robust and accurate promotions, discounts and pricing adjustments. If you’ve struggled with pricing accuracy in Ubercart you’ll understand.
  • Multi-store and multi-currency support is robust and built in.
  • And the list goes on…

Why move now instead of later?

While you could wait until Drupal 7 end of life to move your ecommerce site off of Ubercart and onto Drupal Commerce, this is not something we would ever recommend. The truth of the matter is that by waiting until the very end, you’re taking on a lot of unnecessary risk for both your business and your customers. You don’t want to be in a position where you’re scrambling to make-it-happen quickly when suddenly you’re not getting any more security updates to both Drupal 7 AND Ubercart. That is a worse-case scenario and you would be wise to avoid it.

Right now is an ideal time for you to consider making the switch. Both Drupal 8 and Commerce have been used in the wild now for years and the software is very stable. Most likely all of the features and functionality that you currently use has already been ported over to the new versions. The tools that help migrate Drupal 7 and Ubercart over to Drupal 8 and Commerce have been created to assist with the move. Really, from a technical standpoint there’s no reason to not make the move now.

Of course, it can’t be denied that completing a migration to the latest and greatest does take time and effort to do, and there will be a cost involved. All the more reason to start the process now. Right now you have the time to find the help you need and to properly budget and plan how your migration will be executed. Right now it’s not a hassle, it’s an opportunity to make your business better for both you and your customers while at the same time correcting any of the little things that bother you about your site now.

Acro Media has been helping ecommerce owners and operators with consultation and development for well over 10 years. We’re intimate with both Ubercart and Drupal Commerce, and we even staff some of the talented people who built Commerce and the migration tools everyone uses to make the move. If you want to learn more about how your migration would happen, we would love to talk. Click the link below to get started.

Read the full Gartner report

Sep 19 2019
Sep 19

We're back!  Our normally scheduled call to chat about all things Drupal and nonprofits will happen TODAY, September 19, at 1pm ET / 10am PT. (Convert to your local time zone.)

Feel free to share your thoughts and discussion points ahead of time in our collaborative Google doc: https://nten.org/drupal/notes

We have an hour to chat so bring your best Drupal topics and let's do this thing!

Some examples to get your mind firing: how do I recreate [feature] on my Drupal 7 site in Drupal 8? I need to explain [complicated thing] to a non-technical stakeholder -- any advice? How can I get Drupal and my CRM to play nicely?

This free call is sponsored by NTEN.org but open to everyone.

View notes of previous months' calls.

Sep 19 2019
Sep 19

Attending DrupalCon is an investment in your skills, professional development, and in building community connections. 

A lot of attendees don't buy their own tickets—most need to convince someone else (their boss) of the value.

Sep 19 2019
Sep 19

To scale and sustain Open Source ecosystems in a more efficient and fair manner, Open Source projects need to embrace new governance, coordination and incentive models.

A scale that is in balance

In many ways, Open Source has won. Most people know that Open Source provides better quality software, at a lower cost, without vendor lock-in. But despite Open Source being widely adopted and more than 30 years old, scaling and sustaining Open Source projects remains challenging.

Not a week goes by that I don't get asked a question about Open Source sustainability. How do you get others to contribute? How do you get funding for Open Source work? But also, how do you protect against others monetizing your Open Source work without contributing back? And what do you think of MongoDB, Cockroach Labs or Elastic changing their license away from Open Source?

This blog post talks about how we can make it easier to scale and sustain Open Source projects, Open Source companies and Open Source ecosystems. I will show that:

  • Small Open Source communities can rely on volunteers and self-governance, but as Open Source communities grow, their governance model most likely needs to be reformed so the project can be maintained more easily.
  • There are three models for scaling and sustaining Open Source projects: self-governance, privatization, and centralization. All three models aim to reduce coordination failures, but require Open Source communities to embrace forms of monitoring, rewards and sanctions. While this thinking is controversial, it is supported by decades of research in adjacent fields.
  • Open Source communities would benefit from experimenting with new governance models, coordination systems, license innovation, and incentive models.

Some personal background

Scaling and sustaining Open Source projects and Open Source businesses has been the focus of most of my professional career.

Drupal, the Open Source project I founded 18 years ago, is used by more than one million websites and reaches pretty much everyone on the internet.

With over 8,500 individuals and about 1,100 organizations contributing to Drupal annually, Drupal is one of the healthiest and contributor-rich Open Source communities in the world.

For the past 12 years, I've also helped build Acquia, an Open Source company that heavily depends on Drupal. With almost 1,000 employees, Acquia is the largest contributor to Drupal, yet responsible for less than 5% of all contributions.

This article is not about Drupal or Acquia; it's about scaling Open Source projects more broadly.

I'm interested in how to make Open Source production more sustainable, more fair, more egalitarian, and more cooperative. I'm interested in doing so by redefining the relationship between end users, producers and monetizers of Open Source software through a combination of technology, market principles and behavioral science.

Why it must be easier to scale and sustain Open Source

We need to make it easier to scale and sustain both Open Source projects and Open Source businesses:

  1. Making it easier to scale and sustain Open Source projects might be the only way to solve some of the world's most important problems. For example, I believe Open Source to be the only way to build a pro-privacy, anti-monopoly, open web. It requires Open Source communities to be long-term sustainable — possibly for hundreds of years.
  2. Making it easier to grow and sustain Open Source businesses is the last hurdle that prevents Open Source from taking over the world. I'd like to see every technology company become an Open Source company. Today, Open Source companies are still extremely rare.

The alternative is that we are stuck in the world we live in today, where proprietary software dominates most facets of our lives.

Disclaimers

This article is focused on Open Source governance models, but there is more to growing and sustaining Open Source projects. Top of mind is the need for Open Source projects to become more diverse and inclusive of underrepresented groups.

Second, I understand that the idea of systematizing Open Source contributions won't appeal to everyone. Some may argue that the suggestions I'm making go against the altruistic nature of Open Source. I agree. However, I'm also looking at Open Source sustainability challenges from the vantage point of running both an Open Source project (Drupal) and an Open Source business (Acquia). I'm not implying that every community needs to change their governance model, but simply offering suggestions for communities that operate with some level of commercial sponsorship, or communities that struggle with issues of long-term sustainability.

Lastly, this post is long and dense. I'm 700 words in, and I haven't started yet. Given that this is a complicated topic, there is an important role for more considered writing and deeper thinking.

Defining Open Source Makers and Takers

Makers

Some companies are born out of Open Source, and as a result believe deeply and invest significantly in their respective communities. With their help, Open Source has revolutionized software for the benefit of many. Let's call these types of companies Makers.

As the name implies, Makers help make Open Source projects; from investing in code, to helping with marketing, growing the community of contributors, and much more. There are usually one or more Makers behind the success of large Open Source projects. For example, MongoDB helps make MongoDB, Red Hat helps make Linux, and Acquia (along with many other companies) helps make Drupal.

Our definition of a Maker assumes intentional and meaningful contributions and excludes those whose only contributions are unintentional or sporadic. For example, a public cloud company like Amazon can provide a lot of credibility to an Open Source project by offering it as-a-service. The resulting value of this contribution can be substantial, however that doesn't make Amazon a Maker in our definition.

I use the term Makers to refer to anyone who purposely and meaningfully invests in the maintenance of Open Source software, i.e. by making engineering investments, writing documentation, fixing bugs, organizing events, and more.

Takers

Now that Open Source adoption is widespread, lots of companies, from technology startups to technology giants, monetize Open Source projects without contributing back to those projects. Let's call them Takers.

I understand and respect that some companies can give more than others, and that many might not be able to give back at all. Maybe one day, when they can, they'll contribute. We limit the label of Takers to companies that have the means to give back, but choose not to.

The difference between Makers and Takers is not always 100% clear, but as a rule of thumb, Makers directly invest in growing both their business and the Open Source project. Takers are solely focused on growing their business and let others take care of the Open Source project they rely on.

Organizations can be both Takers and Makers at the same time. For example, Acquia, my company, is a Maker of Drupal, but a Taker of Varnish Cache. We use Varnish Cache extensively but we don't contribute to its development.

A scale that is not in balance

Takers hurt Makers

To be financially successful, many Makers mix Open Source contributions with commercial offerings. Their commercial offerings usually take the form of proprietary or closed source IP, which may include a combination of premium features and hosted services that offer performance, scalability, availability, productivity, and security assurances. This is known as the Open Core business model. Some Makers offer professional services, including maintenance and support assurances.

When Makers start to grow and demonstrate financial success, the Open Source project that they are associated with begins to attract Takers. Takers will usually enter the ecosystem with a commercial offering comparable to the Makers', but without making a similar investment in Open Source contribution. Because Takers don't contribute back meaningfully to the Open Source project that they take from, they can focus disproportionately on their own commercial growth.

Let's look at a theoretical example.

When a Maker has $1 million to invest in R&D, they might choose to invest $500k in Open Source and $500k in the proprietary IP behind their commercial offering. The Maker intentionally balances growing the Open Source project they are connected to with making money. To be clear, the investment in Open Source is not charity; it helps make the Open Source project competitive in the market, and the Maker stands to benefit from that.

When a Taker has $1 million to invest in R&D, nearly all of their resources go to the development of proprietary IP behind their commercial offerings. They might invest $950k in their commercial offerings that compete with the Maker's, and $50k towards Open Source contribution. Furthermore, the $50k is usually focused on self-promotion rather than being directed at improving the Open Source project itself.

A visualization of the Maker and Taker math

Effectively, the Taker has put itself at a competitive advantage compared to the Maker:

  • The Taker takes advantage of the Maker's $500k investment in Open Source contribution while only investing $50k themselves. Important improvements happen "for free" without the Taker's involvement.
  • The Taker can out-innovate the Maker in building proprietary offerings. When a Taker invests $950k in closed-source products compared to the Maker's $500k, the Taker can innovate 90% faster. The Taker can also use the delta to disrupt the Maker on price.

In other words, Takers reap the benefits of the Makers' Open Source contribution while simultaneously having a more aggressive monetization strategy. The Taker is likely to disrupt the Maker. On an equal playing field, the only way the Maker can defend itself is by investing more in its proprietary offering and less in the Open Source project. To survive, it has to behave like the Taker to the detriment of the larger Open Source community.

Takers harm Open Source projects. An aggressive Taker can induce Makers to behave in a more selfish manner and reduce or stop their contributions to Open Source altogether. Takers can turn Makers into Takers.

Open Source contribution and the Prisoner's Dilemma

The example above can be described as a Prisoner's Dilemma. The Prisoner's Dilemma is a standard example of game theory, which allows the study of strategic interaction and decision-making using mathematical models. I won't go into detail here, but for the purpose of this article, it helps me simplify the above problem statement. I'll use this simplified example throughout the article.

Imagine an Open Source project with only two companies supporting it. The rules of the game are as follows:

  • If both companies contribute to the Open Source project (both are Makers), the total reward is $100. The reward is split evenly and each company makes $50.
  • If one company contributes while the other company doesn't (one Maker, one Taker), the Open Source project won't be as competitive in the market, and the total reward will only be $80. The Taker gets $60 as they have the more aggressive monetization strategy, while the Maker gets $20.
  • If both players choose not to contribute (both are Takers), the Open Source project will eventually become irrelevant. Both walk away with just $10.

This can be summarized in a pay-off matrix:

Company A contributes Company A doesn't contribute Company B contributes A makes $50
B makes $50 A makes $60
B makes $20 Company B doesn't contribute A makes $20
B makes $60 A makes $10
B makes $10

In the game, each company needs to decide whether to contribute or not, but Company A doesn't know what company B decides; and vice versa.

The Prisoner's Dilemma states that each company will optimize its own profit and not contribute. Because both companies are rational, both will make that same decision. In other words, when both companies use their "best individual strategy" (be a Taker, not a Maker), they produce an equilibrium that yields the worst possible result for the group: the Open Source project will suffer and as a result they only make $10 each.

A real-life example of the Prisoner's Dilemma that many people can relate to is washing the dishes in a shared house. By not washing dishes, an individual can save time (individually rational), but if that behavior is adopted by every person in the house, there will be no clean plates for anyone (collectively irrational). How many of us have tried to get away with not washing the dishes? I know I have.

Fortunately, the problem of individually rational actions leading to collectively adverse outcomes is not new or unique to Open Source. Before I look at potential models to better sustain Open Source projects, I will take a step back and look at how this problem has been solved elsewhere.

Open Source: a public good or a common good?

In economics, the concepts of public goods and common goods are decades old, and have similarities to Open Source.

Examples of common goods (fishing grounds, oceans, parks) and public goods (lighthouses, radio, street lightning)

Public goods and common goods are what economists call non-excludable meaning it's hard to exclude people from using them. For example, everyone can benefit from fishing grounds, whether they contribute to their maintenance or not. Simply put, public goods and common goods have open access.

Common goods are rivalrous; if one individual catches a fish and eats it, the other individual can't. In contrast, public goods are non-rivalrous; someone listening to the radio doesn't prevent others from listening to the radio.

I've long believed that Open Source projects are public goods: everyone can use Open Source software (non-excludable) and someone using an Open Source project doesn't prevent someone else from using it (non-rivalrous).

However, through the lens of Open Source companies, Open Source projects are also common goods; everyone can use Open Source software (non-excludable), but when an Open Source end user becomes a customer of Company A, that same end user is unlikely to become a customer of Company B (rivalrous).

For end users, Open Source projects are public goods; the shared resource is the software. But for Open Source companies, Open Source projects are common goods; the shared resource is the (potential) customer.

Next, I'd like to extend the distinction between "Open Source software being a public good" and "Open Source customers being a common good" to the the free-rider problem: we define software free-riders as those who use the software without ever contributing back, and customer free-riders (or Takers) as those who sign up customers without giving back.

All Open Source communities should encourage software free-riders. Because the software is a public good (non-rivalrous), a software free-rider doesn't exclude others from using the software. Hence, it's better to have a user for your Open Source project, than having that person use your competitor's software. Furthermore, a software free-rider makes it more likely that other people will use your Open Source project (by word of mouth or otherwise). When some portion of those other users contribute back, the Open Source project benefits. Software free-riders can have positive network effects on a project.

However, when the success of an Open Source project depends largely on one or more corporate sponsors, the Open Source community should not forget or ignore that customers are a common good. Because a customer can't be shared among companies, it matters a great deal for the Open Source project where that customer ends up. When the customer signs up with a Maker, we know that a certain percentage of the revenue associated with that customer will be invested back into the Open Source project. When a customer signs up with a customer free-rider or Taker, the project doesn't stand to benefit. In other words, Open Source communities should find ways to route customers to Makers.

Both volunteer-driven and sponsorship-driven Open Source communities should encourage software free-riders, but sponsorship-driven Open Source communities should discourage customer free-riders.

Lessons from decades of Common Goods management

Hundreds of research papers and books have been written on public good and common good governance. Over the years, I have read many of them to figure out what Open Source communities can learn from successfully managed public goods and common goods.

Some of the most instrumental research was Garrett Hardin's Tragedy of the Commons and Mancur Olson's work on Collective Action. Both Hardin and Olson concluded that groups don't self-organize to maintain the common goods they depend on.

As Olson writes in the beginning of his book, The Logic of Collective Action: Unless the number of individuals is quite small, or unless there is coercion or some other special device to make individuals act in their common interest, rational, self-interested individuals will not act to achieve their common or group interest..

Consistent with the Prisoner's Dilemma, Hardin and Olson show that groups don't act on their shared interests. Members are disincentivized from contributing when other members can't be excluded from the benefits. It is individually rational for a group's members to free-ride on the contributions of others.

Dozens of academics, Hardin and Olson included, argued that an external agent is required to solve the free-rider problem. The two most common approaches are (1) centralization and (2) privatization:

  1. When a common good is centralized, the government takes over the maintenance of the common good. The government or state is the external agent.
  2. When a public good is privatized, one or more members of the group receive selective benefits or exclusive rights to harvest from the common good in exchange for the ongoing maintenance of the common good. In this case, one or more corporations act as the external agent.

The wide-spread advice to centralize and privatize common goods has been followed extensively in most countries; today, the management of natural resources is typically managed by either the government or by commercial companies, but no longer directly by its users. Examples include public transport, water utilities, fishing grounds, parks, and much more.

Overall, the privatization and centralization of common goods has been very successful; in many countries, public transport, water utilities and parks are maintained better than volunteer contributors would have on their own. I certainly value that I don't have to help maintain the train tracks before my daily commute to work, or that I don't have to help mow the lawn in our public park before I can play soccer with my kids.

For years, it was a long-held belief that centralization and privatization were the only way to solve the free-rider problem. It was Elinor Ostrom who observed that a third solution existed.

Ostrom found hundreds of cases where common goods are successfully managed by their communities, without the oversight of an external agent. From the management of irrigation systems in Spain to the maintenance of mountain forests in Japan — all have been successfully self-managed and self-governed by their users. Many have been long-enduring as well; the youngest examples she studied were more than 100 years old, and the oldest exceed 1,000 years.

Ostrom studied why some efforts to self-govern commons have failed and why others have succeeded. She summarized the conditions for success in the form of core design principles. Her work led her to win the Nobel Prize in Economics in 2009.

Interestingly, all successfully managed commons studied by Ostrom switched at some point from open access to closed access. As Ostrom writes in her book, Governing the Commons: For any appropriator to have a minimal interest in coordinating patterns of appropriation and provision, some set of appropriators must be able to exclude others from access and appropriation rights.. Ostrom uses the term appropriator to refer to those who use or withdraw from a resource. Examples would be fishers, irrigators, herders, etc — or companies trying to turn Open Source users into paying customers. In other words, the shared resource must be made exclusive (to some degree) in order to incentivize members to manage it. Put differently, Takers will be Takers until they have an incentive to become Makers.

Once access is closed, explicit rules need to be established to determine how resources are shared, who is responsible for maintenance, and how self-serving behaviors are suppressed. In all successfully managed commons, the regulations specify (1) who has access to the resource, (2) how the resource is shared, (3) how maintenance responsibilities are shared, (4) who inspects that rules are followed, (5) what fines are levied against anyone who breaks the rules, (6) how conflicts are resolved and (7) a process for collectively evolving these rules.

Three patterns for long-term sustainable Open Source

Studying the work of Garrett Hardin (Tragedy of the Commons), the Prisoner's Dilemma, Mancur Olson (Collective Action) and Elinor Ostrom's core design principles for self-governance, a number shared patterns emerge. When applied to Open Source, I'd summarize them as follows:

  1. Common goods fail because of a failure to coordinate collective action. To scale and sustain an Open Source project, Open Source communities need to transition from individual, uncoordinated action to cooperative, coordinated action.
  2. Cooperative, coordinated action can be accomplished through privatization, centralization, or self-governance. All three work — and can even be mixed.
  3. Successful privatization, centralization, and self-governance all require clear rules around membership, appropriation rights, and contribution duties. In turn, this requires monitoring and enforcement, either by an external agent (centralization + privatization), a private agent (self-governance), or members of the group itself (self-governance).

Next, let's see how these three concepts — centralization, privatization and self-governance — could apply to Open Source.

Model 1: Self-governance in Open Source

For small Open Source communities, self-governance is very common; it's easy for its members to communicate, learn who they can trust, share norms, agree on how to collaborate, etc.

As an Open Source project grows, contribution becomes more complex and cooperation more difficult: it becomes harder to communicate, build trust, agree on how to cooperate, and suppress self-serving behaviors. The incentive to free-ride grows.

You can scale successful cooperation by having strong norms that encourage other members to do their fair share and by having face-to-face events, but eventually, that becomes hard to scale as well.

As Ostrom writes in Governing the Commons: Even in repeated settings where reputation is important and where individuals share the norm of keeping agreements, reputation and shared norms are insufficient by themselves to produce stable cooperative behavior over the long run. and In all of the long-enduring cases, active investments in monitoring and sanctioning activities are quite apparent..

To the best of my knowledge, no Open Source project currently implements Ostrom's design principles for successful self-governance. To understand how Open Source communities might, let's go back to our running example.

Our two companies would negotiate rules for how to share the rewards of the Open Source project, and what level of contribution would be required in exchange. They would set up a contract where they both agree on how much each company can earn and how much each company has to invest. During the negotiations, various strategies can be proposed for how to cooperate. However, both parties need to agree on a strategy before they can proceed. Because they are negotiating this contract among themselves, no external agent is required.

These negotiations are non-trivial. As you can imagine, any proposal that does not involve splitting the $100 fifty-fifty is likely rejected. The most likely equilibrium is for both companies to contribute equally and to split the reward equally. Furthermore, to arrive at this equilibrium, one of the two companies would likely have to go backwards in revenue, which might not be agreeable.

Needless to say, this gets even more difficult in a scenario where there are more than two companies involved. Today, it's hard to fathom how such a self-governance system can successfully be established in an Open Source project. In the future, Blockchain-based coordination systems might offer technical solutions for this problem.

Large groups are less able to act in their common interest than small ones because (1) the complexity increases and (2) the benefits diminish. Until we have better community coordination systems, it's easier for large groups to transition from self-governance to privatization or centralization than to scale self-governance.

The concept of major projects growing out of self-governed volunteer communities is not new to the world. The first trade routes were ancient trackways which citizens later developed on their own into roads suited for wheeled vehicles. Privatization of roads improved transportation for all citizens. Today, we certainly appreciate that our governments maintain the roads.

The roads system evolving from self-governance to privatization, and from privatization to centralization

Model 2: Privatization of Open Source governance

In this model, Makers are rewarded unique benefits not available to Takers. These exclusive rights provide Makers a commercial advantage over Takers, while simultaneously creating a positive social benefit for all the users of the Open Source project, Takers included.

For example, Mozilla has the exclusive right to use the Firefox trademark and to set up paid search deals with search engines like Google, Yandex and Baidu. In 2017 alone, Mozilla made $542 million from searches conducted using Firefox. As a result, Mozilla can make continued engineering investments in Firefox. Millions of people and organizations benefit from that every day.

Another example is Automattic, the company behind WordPress. Automattic is the only company that can use WordPress.com, and is in the unique position to make hundreds of millions of dollars from WordPress' official SaaS service. In exchange, Automattic invests millions of dollars in the Open Source WordPress each year.

Recently, there have been examples of Open Source companies like MongoDB, Redis, Cockroach Labs and others adopting stricter licenses because of perceived (and sometimes real) threats from public cloud companies that behave as Takers. The ability to change the license of an Open Source project is a form of privatization.

Model 3: Centralization of Open Source governance

Let's assume a government-like central authority can monitor Open Source companies A and B, with the goal to reward and penalize them for contribution or lack thereof. When a company follows a cooperative strategy (being a Maker), they are rewarded $25 and when they follow a defect strategy (being a Taker), they are charged a $25 penalty. We can update the pay-off matrix introduced above as follows:

Company A contributes Company A doesn't contribute Company B contributes A makes $75 ($50 + $25)
B makes $75 ($50 + $25) A makes $35 ($60 - $25)
B makes $45 ($20 + 25) Company B doesn't contribute A makes $45 ($20 + $25)
B makes $35 ($60 - $25) A makes $0 ($10 - $25)
B makes $0 ($10 - $25)

We took the values from the pay-off matrix above and applied the rewards and penalties. The result is that both companies are incentivized to contribute and the optimal equilibrium (both become Makers) can be achieved.

The money for rewards could come from various fundraising efforts, including membership programs or advertising (just as a few examples). However, more likely is the use of indirect monetary rewards.

One way to implement this is Drupal's credit system. Drupal's non-profit organization, the Drupal Association monitors who contributes what. Each contribution earns you credits and the credits are used to provide visibility to Makers. The more you contribute, the more visibility you get on Drupal.org (visited by 2 million people each month) or at Drupal conferences (called DrupalCons, visited by thousands of people each year).

Example issue credit on drupal orgA screenshot of an issue comment on Drupal.org. You can see that jamadar worked on this patch as a volunteer, but also as part of his day job working for TATA Consultancy Services on behalf of their customer, Pfizer.

While there is a lot more the Drupal Association could and should do to balance its Makers and Takers and achieve a more optimal equilibrium for the Drupal project, it's an emerging example of how an Open Source non-profit organization can act as a regulator that monitors and maintains the balance of Makers and Takers.

The big challenge with this approach is the accuracy of the monitoring and the reliability of the rewarding (and sanctioning). Because Open Source contribution comes in different forms, tracking and valuing Open Source contribution is a very difficult and expensive process, not to mention full of conflict. Running this centralized government-like organization also needs to be paid for, and that can be its own challenge.

Concrete suggestions for scaling and sustaining Open Source

Suggestion 1: Don't just appeal to organizations' self-interest, but also to their fairness principles

If, like most economic theorists, you believe that organizations act in their own self-interest, we should appeal to that self-interest and better explain the benefits of contributing to Open Source.

Despite the fact that hundreds of articles have been written about the benefits of contributing to Open Source — highlighting speed of innovation, recruiting advantages, market credibility, and more — many organizations still miss these larger points.

It's important to keep sharing Open Source success stories. One thing that we have not done enough is appeal to organizations' fairness principles.

While a lot of economic theories correctly assume that most organizations are self-interested, I believe some organizations are also driven by fairness considerations.

Despite the term "Takers" having a negative connotation, it does not assume malice. For many organizations, it is not apparent if an Open Source project needs help with maintenance, or how one's actions, or lack thereof, might negatively affect an Open Source project.

As mentioned, Acquia is a heavy user of Varnish Cache. But as Acquia's Chief Technology Officer, I don't know if Varnish needs maintenance help, or how our lack of contribution negatively affects Makers in the Varnish community.

It can be difficult to understand the consequences of our own actions within Open Source. Open Source communities should help others understand where contribution is needed, what the impact of not contributing is, and why certain behaviors are not fair. Some organizations will resist unfair outcomes and behave more cooperatively if they understand the impact of their behaviors and the fairness of certain outcomes.

Make no mistake though: most organizations won't care about fairness principles; they will only contribute when they have to. For example, most people would not voluntarily redistribute 25-50% of their income to those who need it. However, most of us agree to redistribute money by paying taxes, but only so long as all others have to do so as well.

Suggestion 2: Encourage end users to offer selective benefits to Makers

We talked about Open Source projects giving selective benefits to Makers (e.g. Automattic, Mozilla, etc), but end users can give selective benefits as well. For example, end users can mandate Open Source contributions from their partners. We have some successful examples of this in the Drupal community:

If more end users of Open Source took this stance, it could have a very big impact on Open Source sustainability. For governments, in particular, this seems like a very logical thing to do. Why would a government not want to put every dollar of IT spending back in the public domain? For Drupal alone, the impact would be measured in tens of millions of dollars each year.

Suggestion 3: Experiment with new licenses

I believe we can create licenses that support the creation of Open Source projects with sustainable communities and sustainable businesses to support it.

For a directional example, look at what MariaDB did with their Business Source License (BSL). The BSL gives users complete access to the source code so users can modify, distribute and enhance it. Only when you use more than x of the software do you have to pay for a license. Furthermore, the BSL guarantees that the software becomes Open Source over time; after y years, the license automatically converts from BSL to General Public License (GPL), for example.

A second example is the Community Compact, a license proposed by Adam Jacob. It mixes together a modern understanding of social contracts, copyright licensing, software licensing, and distribution licensing to create a sustainable and harmonious Open Source project.

We can create licenses that better support the creation, growth and sustainability of Open Source projects and that are designed so that both users and the commercial ecosystem can co-exist and cooperate in harmony.

I'd love to see new licenses that encourage software free-riding (sharing and giving), but discourage customer free-riding (unfair competition). I'd also love to see these licenses support many Makers, with built-in inequity and fairness principles for smaller Makers or those not able to give back.

If, like me, you believe there could be future licenses that are more "Open Source"-friendly, not less, it would be smart to implement a contributor license agreement for your Open Source project; it allows Open Source projects to relicense if/when better licenses arrive. At some point, current Open Source licenses will be at a disadvantage compared to future Open Source licenses.

Conclusions

As Open Source communities grow, volunteer-driven, self-organized communities become harder to scale. Large Open Source projects should find ways to balance Makers and Takers or the Open Source project risks not innovating enough under the weight of Takers.

Fortunately, we don't have to accept that future. However, this means that Open Source communities potentially have to get comfortable experimenting with how to monitor, reward and penalize members in their communities, particularly if they rely on a commercial ecosystem for a large portion of their contributions. Today, that goes against the values of most Open Source communities, but I believe we need to keep an open mind about how we can grow and scale Open Source.

Making it easier to scale Open Source projects in a sustainable and fair way is one of the most important things we can work on. If we succeed, Open Source can truly take over the world — it will pave the path for every technology company to become an Open Source business, and also solve some of the world's most important problems in an open, transparent and cooperative way.

September 19, 2019

22 min read time

Sep 19 2019
Sep 19

 

With the proliferation in the touchpoints that enterprises use to connect with customers and provide them with the valuable experience, it’s has become a tedious and challenging task to optimize the content far and wide.

Further, the number of devices that consumers use to access brand content- desktops, mobile phones, laptops, tablets, and smartwatches - with yet more looming on the horizon; have their own set of restrictions and specifications which again increases the complexities of content creators & marketers in the dissemination of the personalized content.

Also, this Gartner report suggested that marketers & decision-makers should now opt for a unified experience strategy to streamline their customer-facing content. This can be done through the implementation of the latest technology and channels to promote dynamic personalization and optimize content in an avant-garde manner. And all this can be executed by dint of Content-as-a-Service.

This blog provides further insights on CaaS, its use cases & features, and how enterprises and marketers can leverage Drupal as CaaS for managing their content efficiently.

What is Content as a Service?

Content-as-a-Service (CaaS) focuses on managing structured content into a unified repository or feed that other applications and properties consume.

The idea behind it is to provide a future-ready CMS that makes content readily available by employing API with or without developing the presentation tier. The presentation layer can be a website, a mobile app, or a feed into a device’s interface. 

The idea behind it is to provide a future-ready CMS that makes content readily available by employing API with or without developing the presentation tier

This separation between the content itself and its presentation implies that RESTful APIs, for instance, can provide the same content that serves both your website to an iOS or Android app.

Put simply, it draws a clear line between the people creating the content, the people delivering the content, and of course, the people consuming it.

A long box with different elements inside

Source: Bloomreach

Characteristics of Content-as-a-Service solutions include:

  • The content disseminated across all channels via a Rest-based API

  • A method of developing content as per prescribed content models

  • Structured formats for returning content via simple queries.

  • Distributed authoring and workflow content administration

  • A content repository hosted in the Cloud for universal access

  • Triggers that alert customer experience applications that consume content to content updates

  • Metadata definitions that can be defined and move along with the content via API

How Does CaaS work?

The actual implementation of CaaS can vary as in the case with any architectural pattern but here is a general overview of how CaaS platform may work-

 

Multiple boxes connected in flowchart

 

The content management UI is a web application to centralize all content authoring and content management of the platform. Content is placed inside centralized storage: it is to note that the format and technology used for the same does not matter at this point, what matters is the correct storage of data.

At last, the content is made available through a technology-agnostic API, like REST API. There are products available in the market which lets you author the content whilst working on the presentation layer to provide you with a wide array of applications you may need (for instance, web apps, mobile apps). 

You could, as an alternative, also provide access to the public APIs of these platforms, allowing others to take care of building their own presentation layers and saving you the trouble of working on that. 

Know how Srijan helps enterprises in modernizing their platforms to manage their content across various channels

Explore Our Services

Why CaaS?

Creating dedicated content for every specific medium becomes cumbersome to the point of being unworkable

 

Have you ever thought that how enterprises and marketers can tweak content for each one of the channels and yet ensure that the content is safe and sustainable for any modification in the future? Since it’s understood that creating dedicated content for every specific medium becomes cumbersome to the point of being unworkable.

So, how is it possible? The answer to this simple question is CaaS!

It can be efficient for enterprises those who want to upgrade their CMS either into one which can serve as CaaS or when there was nothing before.

However, the key deciding factor(s) at the end will be your current context. The reasons are mentioned below-

  1. Siloed Content

Enterprise deals with an enormous amount of content and the sources from where it comes in and having to manage them independently can prove labor-intensive. Either company can spend a lot of time from their schedule to simply manage the content or spend too many resources having a team manager & a set of independent tools with the added overhead of getting them to collaborate with each other. 

In either case, they are most likely dealing with one or maybe more of such situations:

  • They don’t own a uniform content format, which can be made use of for easy distribution. 

  • They don’t own a centralized method to make content available for consumers, be they internal or external ones.

  • Metadata is not given due importance in empowering their content and making it rich for consumers.

  • And centralized storage, so, companies have to put extra efforts to move from one source of data to the next.

The adoption of CaaS could be beneficial to anyone looking desperately to switch their content management strategies. A switch to content-centric approach, i.e., Content-as-a-Service, would significantly improve their performance.

2.   Limited formats for your content

 

Content has to be an abstract entity, and choosing the way how it should be consumed, should be your top priority

 

Your problem might not be about managing your content but inefficiency in reaching to the targeted consumers due to a restricted amount of formats you are compatible with. Content-as-a-Service is again the perfect solution for such kind of scenarios.

Many CMS, such as WordPress, take the responsibility for presentation ensuring that you don’t have to worry about it. However, you also get restricted to the number of devices with which representation of your content is compatible. There could be so many devices where your content can be rejected immediately or simply not pleasant to be consumed in. For instance, have you ever considered how will your online trading WordPress website will show stocks on your smartwatch? What about a VR headset? Or a holographic projection? Agreed that last one does not exist yet but you must ensure that the company is well-equipped and future-ready to be compatible with new technologies, especially when it is moving at breakneck speed and released to the public every day.

Even the new foldable phones are going to be accessible for the public now- what will happen then to the content?

Companies will limit their odds of success if they kept their content tied to their representation. Content has to be an abstract entity, and choosing the way how it should be consumed, should be your top priority.

3.  Native mobile app needing content

Content-as-a-Service provides you with the flexibility to use your content however you want, now or in the future

Since content display on mobile phones and apps demand extra attention, most of the traditional CMS fails to provide the necessary tools and facilities for the same. They only provide web-compatible formats (e.g., HTML) making it unfit for your app.

You can work around this by using a headless, decoupled CMS or Content-as-a-Service to simplify your work. 

In a nutshell, Content-as-a-Service provides you with the flexibility to use your content however you want, now or in the future.

What Drives the Adoption of CaaS?

There are two groups primarily that can leverage this type of content delivery the most: developers and business users/content creators.

  1. Developers

Developers do require CaaS no matter they are mobile app developers who need a backend to feed their apps with content or front-end developers who expect to interact with an API. 

Such technologies have been around since long and widely accepted as well, further fueling the demand for CaaS.

2.  Business
  • Those content creators who want to increase the reach of their content to as many platforms and channels as possible- web, mobile, social networks, smart devices, and so on. 

  • It becomes exorbitant to have a separate solution for every channel- development-wise and maintenance-wise. 

  • It is convenient to manage a single editorial team and a single software stack for all channels.

  • CaaS solutions can help developers in being more productive and efficient with the tools they like to use.

CaaS Use Cases

It’s often perceived that there is no single CMS that is equally good for maintaining both a personal blog and a huge online shop. Contrary to the assumptions, CaaS outperforms its harbingers in some use cases-

CaaS focuses on pushing wherever and whenever required, designers need not worry anymore

 

Pushing content on a mobile app via CaaS proves as the most effective way to have dynamic in-app content without having the need to resubmit the app to the app marketplace.

  • Multi-channel publishing

CaaS CMS is also beneficial when content needs to be transmitted across various platforms, for example, you want to push the same content to a website as well as to mobile apps.

  • Rich Web apps

Modern view controller, i.e., front-end frameworks, such as AngularJS, React, and Ember synchronizes well with structured content via APIs.

CMS can considerably reduce the complexities and simplify workflows in an existing project, for instance, eliminating hard-coded content from HTML pages, and maintaining them with a CMS. In contrast, the API by CaaS makes it highly integration-friendly and robust.

  • Tailored UX

The CMS of web age posed strong design restrictions. Though you could fully tweak the UI but building a Wordpress-powered web app from scratch was not very likely. 

On the other hand, as CaaS focuses on pushing wherever and whenever required, designers need not worry anymore!

It becomes a tedious task to manage already existing content and also the one arriving from multiple sources. Therefore, it is best-suited to upload content into one unified repository by creating content via APIs.

Content structured via API makes it easy for AI-powered agents and chatbots to move it around and churn it for ensuring relevance than screen scraping and using natural language for processing unstructured content.

How Drupal Can Prove to Be An Effective CaaS?

Drupal has unfolded the idea of Content-as-a-Service (CaaS) to solve the dilemmas posed by our newfangled digital ecosystem & its extremely high demand for new and different types of content. 

A square with multiple circles and squares connected to each other

 

Following features on how Drupal can be an effective CaaS-

  1. Reusable future-proof content

Drupal content can easily exist in the form of reusable chunks

Generally, CMSes manage content in a back-end repository and push it to the front-end templates for serving an experience.

However, Drupal decouples the back and front end whenever required. So, Drupal content can easily exist in the form of reusable chunks: free from the presentation and set for delivering content to sites and apps. Thus, content becomes future-ready.

  1. Set front-end developers free to create a better experience

With Drupal’s presentation-neutral content and a RESTful API, front-end developers can freely carry out their creative vision and build interactive sites & apps with the tools like Node, Angular, Backbone, Ember and others.

  1. Fill the content bucket more easily 

Content nowadays should not be restricted to one source only rather it should move in and out freely. And Drupal helps in that by ingesting third-party content (e.g. from aggregators and syndicators) to bring content into your Drupal ecosystem and making it further easy to push to any site, app or channel.

  1. Share content beyond your sites

Today, organizations want to share content on multi-channels where the audiences are inside of content aggregators disrupting the news business. Content teams need an optimal way to create content & then share it with minimal effort. And Drupal does that! The other sites and apps you choose can easily churn Drupal content.

  1. Alter the look

The separation of backend content from front-end presentation gives a leading edge to developers to refine an experience without worrying about the content in the CMS.

Additionally, Drupal’s 8.0 version comes with an inbuilt REST API which marked its beginning of API-first initiative.  

REST allows apps and websites to read and update information on the websites via the web. It also encourages developers to rely on HTTP methods to operate on resources managed by Drupal.

Furthermore, the Drupal community has been working on shipping Drupal modules with web service APIs instead of depending on a central API module in the upcoming releases of Drupal.

Contenta, one of the Drupal’s distributions, is an HTTP API provided for ready-to-use purpose with full auto-generated documentation. It offers modern API capabilities with JSON API, and also feeds content in the JS-driven websites, mobile applications, TV and even fridge applications.

Contenta supports Create Once, Publish Everywhere, be it single application development or multi-channel publishing.

Then there is another distribution, Reservoir, which helps in implementing the Decoupled Drupal architecture, is very flexible and simple-to-use for building content repositories of any application. It also helps in modeling content, governing content, and interacting with that content through HTTP APIs. 

In a nutshell, Drupal’s API-first approach offers the following benefits which further bolsters CaaS model-

  • The decoupled approach separates the presentation layer from the service layer thus allowing a detailed and dedicated focus on each of them.

  • A foolproof approach to help organizations connect to infinite digital signages for enhancing customer experience

  • Increased interaction with customers on their preferred devices will eventually scale up your marketing efforts

  • The decoupled approach is flexible and open for changes, addition, and modification of the structure.

  • Deploying a front-end framework like Angular or React will lead to sophisticated, enriched and dynamic web experience

 

Learn more about Drupal API-first initiative from here-

[embedded content]

Features to Lookout For in CaaS

CaaS comprises of three vital parts: the editing interface (typically a web app), the CMS infrastructure capabilities, and the development ecosystem.

Web editor

  • Enables content architects to design the structure of the content

  • Enables content editors to manage content from creating, updating to collaborating on it.

Technical infrastructure

  • Offers performance, uptime, and scalability to ensure that enterprises can rely on their vendor to deliver content in mission-critical applications.

  • SLAs with short incident response times and access to dedicated staff- so in case of a problem with a mission-critical app, companies can be provided back up again and fast.

  • Mobile delivery capabilities so that great user experience can be delivered even in network-challenged environments ( like subways, rural areas) and high bandwidth cost areas (such as emerging markets).

  • API-based importing, management, and delivery for controlling content programmatically in both ways

  • All-inclusive and up-to-date documentation to help the development team start using the tools quickly.

  • CDN ( content delivery network) to deliver the content rapidly

Development ecosystem

  • SDKs and libraries to increase the speed no matter what the tech stack is

  • Demo app source code so that developers don’t feel the need to reinvent the wheel all over.

  • Third-party integrations to obtain value from existing tools.

Other Characteristics of CaaS

The decoupled approach ensures that code and content are placed separately so that marketers and developers can do their respective work

  • Decoupled approach

The decoupled approach ensures that code and content are placed separately so that marketers and developers can do their respective work. Teams can also work parallelly on a creative copy, enticing visuals, and expert integrations in one unified platform.

This is the quintessence of the headless CMS approach - agnosticism towards how content is presented. This frees developers from creating highly custom front-ends and apps since they get to define the content display part.

A box with various elements listed inside and interconnected

                                                                       Source: Gartner 

  • Cloud setup

The complete separation of the content management and display part allows organizations to migrate infrastructure between Cloud and hybrid, even at the site level or project level. Some projects can be installed locally while some on Cloud depending on the business’ choices for optimization as per needs. 

Centralized Content-as-a-Service lets businesses evaluate the content consumption across the digital ecosystem. This ceases businesses from duplicating their efforts and content when posting to microsites, international sites, or apps. It can also measure the use of that content by looking at the API connections used to deliver that content, and keeping track of where the content is going. 

In the End

The digital revolution and breakthrough in technology have accelerated the efforts of content creators - be it creation, designing, or dissemination. The goal is clear- refined user experience.

With that said, the creation of content in abundance and its delivery as a service through thousands of APIs will generate more data thereby assisting content developers to create more precise business models.

The technology is already in place, and the architectural patterns will allow enterprise systems to scale up without hampering their performance.

Content-as-a-Service ensures that developers are rendered maximum freedom and flexibility to realize their digital innovation. Drupal as a CaaS has been delivering a wonderful experience to both content editors and developers alike.

It is definitely a convenient way to ensure that your strategy is future-proof and can handle any new media in the future.

Sep 18 2019
Sep 18

BADCamp is stoked to be offering two full days of trainings that can help advance your skills and career, led by some of the best teachers in the community.

Full-day trainings cost $50, and half-day trainings cost $25 to cover operating expenses. This year we are excited to be offering two free community workshops. 

Available Trainings:

 If you can't afford this or it is super complicated to get funding, please reach out to the BADCamp organizers via the contact form and we will help!

There are no training refunds after September 23. If you are not able to attend your training, do let us know so we can open up your spot to folks on the waiting list.

Community Workshops:

Do you think BADCamp is rad?

Willing to pay for your ticket?  If so, then you can give back to the camp by purchasing an individual sponsorship at the level most comfortable for you. As our thanks, we will be handing out some awesome BADCamp swag.

We need your help!

BADCamp is 100% volunteer driven and we need your hands! We need stout hearts to volunteer and help set up, tear down, give directions and so much more!  If you are local and can help us, sign up on our Volunteer Form.

Sponsors

A BIG thanks to our sponsors who have committed early. Without them, this magical event wouldn’t be possible. Interested in sponsoring BADCamp? Contact us via the contact form.

Sep 18 2019
Sep 18
[embedded content]

What is real-time collaborative editing, and what are some of the most compelling technologies available in the space? In the inaugural TAG Team Talk, hosted by Preston So (Contributing Editor, Tag1 Consulting), we conduct a wide-ranging discussion about both the business prerogatives and technical ins-and-outs of real-time collaborative editing and its landscape today, with our guests Kevin Jahns (creator of Yjs and collaborative editing expert at Tag1 Consulting), Fabian Franz (Senior Technical Architect and Performance Lead, Tag1 Consulting), and Michael Meyers (Managing Director, Tag1 Consulting). In this talk, we explore collaborative editing, diving into how it works and some of the challenges borne by shared editing. Through the lens of Yjs, a real-time collaboration framework that supports not just text but also collaborating on drawings and 3-D models, we take a look at Operational Transformation (OT) and how implementing Conflict-free Replicated Data Types (CRDT) drives decentralized server approaches in collaborative editing and supports more robust distributed applications with true real-time support.

Yjs: https://github.com/yjs/yjs

ProseMirror: https://prosemirror.net

Great Overview of CRDT
https://conclave-team.github.io/conclave-site/#conflict-free-replicated-data-type-crdt

Deep dive int CRDT by the author of Automerge: https://www.youtube.com/watch?v=yCcWpzY8dIA

Yjs was inspired by:
Sharedb https://github.com/share/sharedb
DerbyJS https://derbyjs.com/

Text Transcript

- Hello, good morning or good evening, wherever you are. And welcome to the first ever Tag1 Team Talk. This is Preston So speaking to you loud and clear from New York City. I'm the contributing editor to Tag1 Consulting. And it's my pleasure today to jump into a deep dive into realtime collaboration tools. I'm joined by three guests today. First and foremost, I want to introduce my partner in crime Michael Meyers, the managing director at Tag1 Consulting. I'm also joined by two folks from around the globe. We've got Fabian Franz, senior technical architect and performance lead, as well as Kevin Jahns, creator of Yjs, a key contributor to realtime collaboration projects, and a maintainer of open source projects. So, I wanted to start off by introducing ourselves one by one, and then we'll jump into what Tag1 Consulting's all about and get into some of these meaty topics we've got for today. My name is Preston. I'm the principal product manager at GatsbyJS as well as contributing editor to Tag1 Consulting. Mike, you wanna go next?

- Sure. Mike Meyers, managing director of Tag1 Consulting. I have 20 years of experience working in technology, managing and running teams. I've been working with Tag1 first as a client for over a decade, and joined the team about two years ago.

- Hi, so, I'm Kevin Jahns. I live in Berlin, Germany. I'm the creator of Yjs open source framework for realtime collaborative editing. Because of that, I got many interesting opportunities to work for really cool, interesting companies on realtime editing. They mainly use my framework and need feedback or help to deliver a great product. I'm currently really happy to work for Tag1 and one of the Fortune 150 companies to make that happen.

- Thanks Kevin, pleasure to have you with us. And Fabian?

- Hey, my name is Fabian Franz. And I'm at Tag1, as you already said. I'm a senior technical architect. I really enjoy architecting large websites and making them really fast. And I'm so excited about realtime collaboration because I think realtime kind of is the future of the web. But we also did a DrupalCon presentation slightly related to that topic, in Amsterdam. And besides that, I'm a Drupal core 7 maintainer right now, and working with Tag1 is just fun.

- And we've known each other for a long time, Fabian. Well, perfect. I think first thing we want to do, though, is set the stage a little bit. What exactly is Tag1, and why are we all here in this room? From what I understand, Mike, you all are the emergency support and rescue, and performance and security experts. Can you tell us a little bit about Tag1 Consulting and what you're all about?

- Definitely. We're the company you turn to when you have difficult and challenging problems to solve. We build mission critical, highly scalable, highly available and secure applications for companies like Symantec. We've worked for them for over a decade. We manage and built their infrastructure work and oversee their agency partners, setting architecture design and doing manual code reviews for security and quality. What we're gonna talk about today is a project that we're working on for a top 50 Fortune company, Fortune 50 company. They are rebuilding their intranet, and realtime collaboration is a key component of it. And frankly, I can't believe that realtime collaboration isn't a default feature in every online application, certainly every CMS. When I think about my daily workflow, I'm constantly working in something like Google Docs, collaborating with the team in realtime. And I know a lot of people, we're talking to another client about a project and they do all of their work in Google Docs and then just pump it into their Django CMS. And so, we really believe that this is the future of a lot of applications, and we're excited to be working on this particular project, so we wanted to talk to everyone about it because it's got some awesome tech behind it.

- Absolutely, I agree. I think that realtime collaboration is one of those intractable problems. It's not quite on the level of some of the key problems in computer science, but it definitely is a very, very, a large problem set that actually hasn't been solved completely across the board. Google Docs obviously has solved a lot of those issues. But I want to dig a little bit more into why it is that we're all on this call, why it is that we're so interested in realtime collaboration. Obviously it's something that is important to all of us. But can you tell us more about this Fortune 50 company and what some of their requirements are, and what exactly realtime collaboration means to them?

- Sure. It's a gargantuan project, multiyear project. It's used by 20,000 active users at any one time, across 200 countries, in well over a dozen different languages. It interfaces with a large number of third-party tools and systems, from centralized authentication access control through to Slack integration, as well as integration with other third-party tools and a number of internal proprietary tools. The idea is that we live in a world where you want to integrate the best technology for the job as opposed to reinvent the wheel. And so, what this... Their new generation of their intranet is to really pull in and integrate all of these different systems into one place. Because the downside of integrating all these different applications is that it can be hard to collaborate and find things if you have files in Box, if you have documents in Quip, if you have communications in Slack. All these disparate systems have different organizational structures, typically are run by different groups and departments. The idea is that the intranet will be the central hub that pulls together all these tools and information, helps you find what you need, but also provides that in a universal search and collaborative layer on top of it as sort of like the metasystem where you can talk about everything from, and track those tasks and schedules, to what your roadmaps and sprint reports are for your particular initiative.

- There's actually, and I recognized a Dilbert comic about it, where he's like, I need this document on my desk tomorrow, the boss says. And he's like, would you like it via Dropbox, via Box, via mail, via Slack, via whatever, via Skype?

- That's right. Sometimes I work with a friend and we don't have an organization to work under with. And I sometimes wonder how can I send a large file, like just 100 MB, to my friend. It's really hard right now.

- Yeah, it is really difficult. And I think this really calls out one of the things that you actually alluded to earlier, Mike, which was the fact that we have a lot of folks who are using CMS's as part of their daily workflows, but they need a realtime collaboration tool to bring it all together. I think the fact that we've heard from a lot of folks that they use Google Docs before copy and pasting over to a CMS or over to their desired tool shows that this is really the nexus of where all of these interactions occur. And so, I guess I'm curious just to dive into some of the business requirements just briefly before we get into the technology. I know that's where we definitely want to spend a lot of time. But why is it that something like Google Docs doesn't fit the bill for this use case?

- I can't talk too much about the company itself. But for business reasons, they are unable to use Google Docs. They're a competitor company and they can't have highly sensitive, mission critical information on third-party servers. This entire application is air gapped, highly secure. They have very different levels of projects, from public to ultra secure. There's even separation of data and communications. So, things that are for the highly secure projects are stored on different servers and the realtime communication goes through its own system. Everything from physical separation of data and information to air gap. And while they are able to use... Or sorry, not able to use Google Docs, the other reason is that they want this highly integrated into the system. A third-party collaboration tool, whether it be Slack, or Quip, or Google Docs, or something that facilitates that collaboration, has the same challenging problem that we talked about earlier. And so, by integrating this directly into their intranet it's a core feature. And I should also mention that they want to go well beyond text editing. We're working on collaborative drawing, for example, so that you could have whiteboards. In the future you could have your agile Kanban boards show up on this system even though they're coming from a third-party tool via APIs. You can see that information. Our goal isn't to replicate all of these systems in the intranet, but to present you with the most mission critical information that you need at a glance, and to then link you into these systems. In the case of collaborative editing, it's such a core feature to enable teams to work together that it's built in.

- Also, one of the challenges with using something like Google Docs always is access. Being a large company, obviously they have a central directory service where everything is managed and accessed, and you now need to enable this third-party system this access. That's pretty challenging because they're mostly scaled for a few users, in that you invite your team, and then you work with them, collaborate with them on that. But then you need to put it back into the CMS and there's this kind of gap between your workflow so you're always like, now you've published a version, but now some more changes need to be done. Who will synchronize that, who is dealing with that, et cetera. It's basically two systems you need to work with. Even for our workflows where Tag1 had a Drupal watchdog before, it was challenging, and kind of what is the source truth once you get to a certain point in that.

- Absolutely. I think the notion of something like realtime collaboration being built in, being an intrinsic part of an end-to-end system, is a kind of key feature. But I think what you just alluded to in terms of permissioning, and having to figure out, having to reconcile user roles and permissions across two very different systems can be very challenging. And having the ability to do that in a way that is really integrated with the overarching system I think makes a lot of sense. Let's go a little bit away from the business requirements and let's dig into a little bit of the main concepts behind realtime collaboration. I know that all of us on this webinar and podcast all know some of the basic components. But just so we can bring it down to the level of what some of the core fundamentals are, how does shared editing exactly work? How does this idea of realtime collaboration actually begin from the standpoint of clients, communication, concurrency? Where do we start with? What are the main elements that are involved?

- I do think we start with a very basic, just what is shared editing in general. And many companies also use already something like Etherpad where you can just have some text and collaborate it. I know we also used it for the Drupal community when we had challenging things to solve just to facilitate note keeping a little bit in that. The most basic thing about shared editing is really just text. It's not formatted text like Google Docs or something. It's really just text. And this is very simple. So, you have an example of like, for example, I have the text written, "This is some text." And now Michael wants to make it, "This is some nice text." But I want to write a little intro, so I wrote, "Hello Preston, this is some text." And now you need to know a little bit about how editors work, and an editor needs to know how this works. My, "Hello," would be inserted at position zero, so at the start. But Michael's, "Nice," would be inserted at position 12. But now, if I write before him and the editor still thinks he should put it in at position 12, then it's wrong, because we kinda... The positions changed because I've added some text, so editing has shifted a little bit. And the idea of CRDTs here is that instead of having this position-based system, just on a very, very basic level, they are very more complicated but just as a layman's explanation, is that for every character, you are giving it an identifier. And this identifier is also a little bit how a human would reduce this conflict, resolve this conflict. Because instead of saying, hey, insert this, "Nice," at position 12, you're just saying insert it after, "Some." And that's basically the idea, kind of. And because of that, however the text changes, Michael's, "Nice," will always be inserted after the, "Some." That's kind of how you can think about how you get shared editing to work on a very, very basic level.

- Let's talk a little bit about the communication. I think that that was a very, very great explanation of how concurrency would function in this kind of environment. I'm curious, though, there's a notion of a difference in communication between how some of the traditional collaborative tools have worked in the past and how, let's say Yjs or some of the more modern tools do it today. I know that, for example, in the past the way that people would do this kind of collaboration would be through a server and a client. A centralized server that houses all of the information about what's changing and manages all of that, as you said. But one of the key differences that I noticed about Yjs is that it's focusing on a more peer-to-peer, a more decentralized approach. Do you want to talk more about that, Fabian or Kevin? What does that mean to somebody who is evaluating some of these solutions?

- The classical approach is like, Google Docs in general, operational transformation, it mostly works in the client-server environment. So you have a central point where you do all the transformations between the document updates. For example, a user inserts something at position zero and another user inserts another character at position zero. The server decides how the document looks like at the end and does the transformation before sending it to the other peers. So there's like a limited concurrency because there's only concurrency between the server and the client. And, like, it's pretty unfair because their operational transformation approach is they do work peer-to-peer, but they have a lot of overhead, a lot of metadata. In CRDTs in general, you don't need a central point of transformation. Like, all CRDTs, they allow for commutative operations. So it doesn't matter in which order your changes come in. The document will look the same for every peer as long as all the peers exchange all the document updates. And that's a really cool feature. So, you don't rely on a central server anymore, and you can actually scale your server environment. You can do really cool things with that. Most people think, oh, peer-to-peer, you have peers communicating directly with each other, and I am a huge fan about that. But especially for businesses, it makes sense to have still a server environment. And here, the advantage of Yjs or CRDTs in general is that you can scale your servers indefinitely. If you know that Google Docs is limited to a limited number of users, I think the limit of users who can concurrently edit the document is set to about 15 people. And after that you will just see a copy of some snapshot, and maybe it gets updated. I haven't looked into it for quite some time, but I know there's a limit. Which is kind of strange. The problem here is that you have centrality and you need to make sure that you never lose document updates because the centrality is also your central point of failure. So if this point, like if this server burns down, your hard disk, your document is gone. You can never merge again, you can never restore that document. But in peer-to-peer or decentralized systems, well, it doesn't matter if one system burns down. You still have another one.

- I think that single point of failure is a really key and worthy of emphasizing aspect here. Which is that one of the things that is a very key concern for enterprises today, and just companies in general, is the notion of having the ability to keep things decentralized. If you have that central point of failure, everything falls over, you don't have access to anything. And the notion, I think, of peer-to-peer, decentralized, shared editing, allows for local copies of those documents to be on every single node, let's say, of every single person who's working on the document.

- Yeah.

- Yeah, go ahead, Fabian.

- Even if it's very different from how, for example, something like Git works, because it has very different properties, you can use the same examples as advantages. For example, one of the nice examples that is always quoted with Git is some developers can work on a plane together. They can collaborate on some code on the plane together because everyone has a whole repository of all changes. And that's kind of the same with Yjs, to some extent. You can optimize it to not have solo history, but you can have solo history, so two people that happen to be on the same flight could realtime collaborate on that document and later send it back to server. But they would not need a central server component. They could just do it. They could put work on the plane. And I think that's a really cool feature. And not just two, it could be a whole company, like 50 people finishing a last minute document collaboratively. Internet on planes has gotten better, yes, but it would be really bad if your Google Docs suddenly while you're on that plane. With Yjs, you just have the guarantee that your personal server works and you can finish your last minute document.

- It was pretty amazing, we actually saw this in action, unintentionally. We were doing a demo to key stakeholders. We had just integrated the realtime collaborative system which we kind of built independently at first, with Drupal, the core of the internet. And we were doing a demo of the system in action completely in development environments. We're pre-alpha. And during this demo, the server failed. Yet the collaborative demo just kept going and working perfectly because it was peer-to-peer in the backend. And the infrastructure engineer on the call was able to get the server back online, and it was just absolutely seamless. So it was really cool. All of us were like, my god, we didn't intend to demo that. That was amazing.

- And I think that, I think Fabian, you mentioned something that was really key there, which is that idea of a spotty connection. Obviously people working on flights. This case that you just described, Mike, as well is a very good example of this. This kind of approach that focuses on the notion that everyone has a local copy, just as you said, with Git-based approaches as well, this is the kind of thing that even if you're in, let's say the middle of nowhere and have a really horrible 3G connection you can still get the editing capabilities, and get that sync going, and make sure that things stay up to date. I think it's a very, very key value proposition. I want to jump back into Yjs itself because I think that a lot of people are really interested in operational transformation. A lot of people are interested in what exactly CRDT is, and all of those things. But first let's talk about a very key issue in collaborative editing that I think really uses a little bit of discussion. And that's the notion of an edit history. One of the biggest challenges that all content, all people working in content governance and content strategy have is the ability to distinguish between different undo histories. How does undo really work in the context of shared editing versus single editors? And how exactly does that work from a technical standpoint? Is there a global state, what's driving that state? How do you reconcile some of these things? We can talk about it from the perspective of Yjs or other tools as well.

- Man, undo/redo, it's a really, really hard problem, especially in CRDTs. In operational transformation, this problem is not that hard. You can just do undo/redo your own operations. You can just do the transformations in the document history. But you don't really have that history in Yjs, or in CRDTs because it just, it works differently. It's not really operations that you put in a linear log. It's more like operational set are in some kind of tree, and you can't really decide, okay, where do I need to start undoing. So the thing in Yjs is... Yeah, so, the way I implement it actually is in this tree I give a path of the undo/redo operations. So I basically say, this is the kind of state that you need to remove when you hit undo, and this is the kind of state that you need to add when you hit undo. And this is basically how I manage it. It's kind of complicated without deep diving into how Yjs works. But yeah, it's mostly a solved problem at this point. So, the undo/redo thing, it works on linear text really, really well. And there's great progress on structured documents, like on ProseMirror, which is, it's one of the editors that Yjs supports. So, there you basically have this structured document, a paragraph which has a table and the table has some lists maybe, and there you type and hit undo. You really need a good sense of what to undo and what the user expects when you hit undo/redo. Yeah, there's great progress. And the undo/redo implementation is configurable. I'm not saying that it's done yet. There's still users reporting, "Oh, that's weird," because it's pretty new, undo/redo on structured documents.

- Just to show how difficult undo/redo is, you can do some great experiments with other out-of-the-box solutions like Google Docs. You can have lots fun. Also with something like conflicting things, like Google Docs has some offline support, and I've taken the liberty to just test some scenarios. Like, you will have a paragraph, a long line, and do an enter break in the middle, and how does the system react? And it is really cool to just try out this, like, perfect product, built on knowledge of giants with Google Wave and all the technological and theoretical things, and then you try some undo/redo scenarios, and you get some really weird results even in the top notch product on the market right now. It's really interesting.

- Yeah, I think the notion of, like, what happens when you get a merge conflict or a sync conflict in Google Docs. It's always a fun experiment. It's rare that I get those messages these days, but it's always an interesting experiment. I do want to go back to something you just mentioned though, Kevin. And I do want to dwell a little bit on some of those really nitty gritty features of Yjs and really do the deep dive. But I'm curious, you just mentioned that the undo/redo behavior is actually configurable in Yjs. Can you talk a little bit about how that is, or what sorts of configuration can be expected?

- Oh yeah. The thing is, in Yjs you have... Yjs basically, well, my idea of Yjs is that you can see it as a shared type system. So you can create shared data types, something like an array, or a map with a key-value store. These data types, they work concurrently. So one user can manipulate them and the other one receives the updates, and then they sync. This is how I see Yjs. And then there are bindings to editors like ProseMirror, and CodeMirror, and many others. And often you have this big Yjs structure, and it may be a structure of several documents. So, when many users, or when you are working on several documents at the same time, and then you hit undo, what do you expect to happen? Does it also undo something from other documents? I don't think so. In my mind, I work on one document, I only undo the changes of that document. So this is what I mean of configure it, being able to configure it. You can specify what can be undone by your undo manager. There can be several undo managers, several document histories, for your local user. And you can specify what can happen and what cannot happen. And there are two mentions in Yjs. It's mainly you have a scope to a specific data type. For example, this undo manager can only undo stuff on this document. And then you have the scope also of... So, for example, what happens, when I created a paragraph, you created this big chunk of text in that paragraph but I didn't know about that, and then I hit undo? Can I undo the creation of that paragraph or not? Most users, as I found out, would say no, you should not delete content if you didn't create it. So in this case, you are not able to undo the changes of other users because sometimes these changes are intertwined. And the challenge here is really to figure out what is it what users expect when you hit undo.

- That's very interesting. I think that that really digs at, I think, the central issue of undo/redo, which is that people have different expectations of it. And having that as a configurable element I think is a very, very compelling idea. Well, okay, so let's dig out of undo history. Oh, go ahead. Sorry, Fabian.

- And there's also one of the key points why we think that CMS's itself should have the collaborative technology, and Drupal should have it out-of-the-box as a contributed module, Azure CMS should have it out-of-the-box. It should be really as simple as install a JavaScript library, like a plus module, and maybe a little server component if you don't want peer-to-peer. But then the centralized point here is that every client, every user, has their own expectations, they have their own challenges, they have their own things. And also, also all the editor space, even going to add a little bit of what we wanna go in more detail maybe in future episodes, they are no longer shipping out-of-the-box editors. They are shipping editor toolkits so every CMS can build their own editor to their needs. Every client, every project, can build their own editor. And I think that's a very, very key point to our current ecosystem of CMS's. And that someone else might need completely different undo managers than I need or this project needs. And that's also the beauty of open source. You can just take it, refine it, extend it, and do it. That's so cool with Yjs being open source that you just have this ability to do all these cool things.

- I'm not sure if we have the liberty to do this, but can we dig into a few of these potential scenarios where having this kind of differentiated configuration would be useful? Fabian, I think you mentioned a very interesting idea, which is that different users want to have different approaches to how they edit. How have you both seen this shake out in terms of the things you've tried, experiments you've tried? Or Mike, has this come up in terms of use cases? What sorts of things have you all seen in the real world that relate to this idea of having completely independent experiences?

- So basically, top 50 Fortune client it is customizability is key. It's very important that they have, for example in this case, their own editor which is exactly working to their needs. And the collaborative functionality is everything like you're expecting from Google Docs like just comments and later, even track changes, still working on that, pretty challenging. That's all very key in that. And it's important that, for example, if you use a technology like React upon which you can build your editor... Think of Confluence which is built upon Atlaskit. And Atlaskit is, by the way, also open source and one of the components we're using here as base. Then you can extend it, and you can build your own and you can customize it. That was really a key, central point to being able to not be stuck with some proprietary solution, or be stuck with some out-of-the-box solution where you cannot define anything, where you just get sent a black box and then you have to work with it. Because then if you need to change to the undo thing, well, yeah, you can pay Windows for it, obviously, but it's really out of your control. And here you can work directly with , but even later you can maintain it with a different team or whatever because it's accessible code. Another advantage of open source, basically. But yeah, the extendability, and being able to, for example, also define the server component. In this case, a server component is needed even though peer-to-peer would work and works well within the same browser or within even cross-browser with some tricks. Because you want, for example, authentication. If you store confidential data in a shared editing thing, you don't just want to be able to just access the server and see it, but really you want to... Which is usually a Node server, by the way. But really, you want to have authentication tokens like you're used from different APIs where you can secure your server. Then you can say, yeah, this operation is allowed or this operation is even not allowed. Even being able, as Yjs messages, even if it's not kind of like messages it's still some transformation sent over the wire. And then you could deep inspect it, and can say, hey, this is not allowed. This document was not touched for 300 days or whatever, and now someone is deleting all the content? Maybe deny that. Or keep a snapshot before that. And that's also so... So here, central server is nice and needed because you want to have that control but it gives us this flexibility in that, and you don't get that flexibility if you work just with a black box.

- Control is something that's a key theme of this project, this organization. Why we don't want to reinvent the wheel and rebuild certain awesome third-party technologies, this client of ours has repeatedly been let down by vendors that have made commitments to add certain features or do certain things over time. And one of the reasons that they're rebuilding this system and investing in these kinds of technologies is that the business cost of not being in control of sort of core technologies is huge. And so, relying on open source projects, being able to own and manage this code internally for their needs is critically important. This is gonna be a system that's gonna be around for 5+ years. The current system has been in place for a little over seven. And so, a key recurring theme is we need to be able to have long-term maintenance and control over key aspects of our technology.

- Yeah, and I think... I'm sorry, go ahead, Kevin.

- All right. I just wanted to add, one of the things that a lot of people appreciate when I talk to people who use Yjs is that you can easily use Yjs on top of your existing communication protocol. So there are many existing projects that want to integrate collaboration into their system, and they already have some kind of WebSocket connection or long polling setup, and they want to use that system. And for good reason. They don't want to change their whole infrastructure, add another communication layer like some proprietary WebSocket communication protocol, and build on that. They want to integrate Yjs into their existing protocol. That's something that you can easily do with Yjs. And this is also something that Fabian talked about just now. I completely agree with that. I didn't think about it when I implemented like that, but it just came up that it is really well appreciated. And because of that, I put everything into separate modules. Yjs is split up into like 20 modules right now. Once you build a system, you basically say, okay, how do I communicate? I use this existing module, or I can rewrite my own module on top of the Yjs code, or just on top of the documentation that you have. And if you want to support your own editor it's so easy because the editor support is not baked into Yjs. It's not just built for one thing, it's built for everything. And that makes it incredibly difficult to do, but I think it's a really nice concept.

- Yeah. And speaking to that, the editor support, if you have textareas somewhere and you want to make collaborative, you can do that right now. There's a translator for a rich text area, like the contenteditable, or just any textarea. You can just take it, put Yjs on it. And there's also a nice demo site. There will be some post information. We'll link to that with some links in that. There's a really cool, there's even two demo sites. One is really showing like ProseMirror, Quill, and even Atlaskit, the whole Confluence experience in that. But there's also shared editing for a nice thing. There's even a 3D modeling demo baked in together with 3D modeling. It's insane, it's really insane. Also drawing. It's, like, really cool. The possibilities are so endless. It's so nice that it's so framework agnostic. Really not just editors and really not just one editor. But really, if you have a textarea, you can use Yjs, it will be collaborative, it works. And even for older things, if someone wanted to invest in that CKEditor 4, it would be possible, in theory, to make that collaborative. People wouldn't even need to operate newer technologies. It would need work, investment, et cetera, of course, but it's possible.

- Yeah, I think this notion of flexibility and customizability, but also especially this notion of pluggability, where you can insert Yjs into any situation and it works. I think also the flexibility. Fabian, you mentioned earlier that in certain cases you do want to have a server component that acts as the safety kind of mechanism. But you might not want one, and Yjs leaves that option open as well. And I think just the notion of being able to insert it agnostically anywhere you want is a very, very key characteristic. I think one of the key characteristics that we identified a little bit earlier but we haven't really dwelled on, though, is one that's very interesting to me. Which is that you can use Yjs for things like text, you can use Yjs for textareas and contenteditable, but what about drawings and 3D modeling? I know, Kevin, you've sort of framed Yjs as being for more than just text. Can you talk a little bit about the drawing capabilities? What does collaborative drawing mean, and how does Yjs fit into that?

- Yeah, definitely. This all comes from the idea that Yjs is not just for specific applications, but it's just a framework for shared data types. You can build any application on data types because, well, that's what we usually do as developers. We have these key data structures, especially data. In JavaScript we have only two or three main data structures, which is arrays and maps. You can build your own data structures on top of that and they're abstractions, but basically you just have arrays and maps. And maybe set is also something that, well, I really like the idea of having a set too. And you have the same capabilities in Yjs but the difference is, these data structures or data types, they are collaborative, they are shared, so when you do an edit to this data type the change is transmitted to all the other peers automatically, and they sync, and then you get updates of what changed. And you can design your application like that. Just use the shared data types and build your application. And drawing was something that is really, really easy to implement on top of data types, or shared data types. A line is basically just a list, an array of vectors. So, this is what is happening in the drawing demo. When you start drawing, you have a map, you insert a line to that map, and then you start drawing. It's really easy. And then you can maybe configure it by adding some options to the map, or something to color, who created the line, all this information. You can just add it to the shared data structure. And this is also how the 3D modeling is created. It's really basic, but it's really cool because it shows that everything you do, like rotating the 3D object, the rotation is a part of the shared data type. You can add annotations and these annotations are just X, Y, Z values, and you add this to the shared data structure. And just like this you can build amazing applications, just on top of this idea.

- Yeah, I was thinking about the classic React example, the to-do list. And the to-do list basically, the results are often shown for Redux data stores like transformations, et cetera. And now put this final array that you put your data in, or this kind of map, probably an array in this case. Put that in the Yjs data structure and it's automatic. It's kind of almost automatically collaborative. That's just great.

- Is one of these in particular harder than the others? Is collaborative text editing actually harder than drawing?

- Collaborative text editing, like, I want to distinguish between text editing and rich text editing. Because text editing is pretty easy. It's harder than having a 3D model, in my opinion. From my perspective as a developer of Yjs, it is harder because text editing, you need to optimize that a lot to make it performant and don't block the thread. You want to be able to insert, like, a megabyte of text into your data structure. And that's not always easy. First of all, because of the time it takes to send to the other peers, but also the time it takes to parse that operation, that change. So, text data structure is heavily optimized in Yjs, and that's why it's performant. For you, it's as easy as the array, as using the array data structure. Rich text on the other hand, it's a bit weird because you can either have structured documents. This is what I think of in ProseMirror as like, you have a paragraph, inside that you have a table, and so on. And then you have also formatting attributes to your text. So, when you write the text, "Hello world," and you want to make, "world," bold, you don't remove the bold text and add some text, tags like in HTML, around this word, "world." You want to assign attributes to this text. And this is also possible in Yjs. It's like this rich text notion from the classical Windows. I think they developed the term rich text. You assign attributes to text. This is also one of the problems here. And like, as soon as you realize that this is going on, you can either have structured documents, or rich text documents, or both combined. In ProseMirror, they're kind of both combined. You have marks, so you can assign properties to it, to text. And as soon as you realize what's going on here, it's fairly easy to build applications on top of that. But building editor support, that's a whole different level for if you build a structured editor support, for example for ProseMirror. It was really, really hard to get right.

- Yeah. And speaking of performance for inserting, we did a test. And if you take a really, really long document, you paste it 10 times into Google Docs, it takes longer and longer. It takes around seven seconds. And with Yjs and CRDTs, it's instant. It's really, it's really just instant. I was astonished. I also tested some other things, which I don't name to not shame. But there I was even able to freeze, to completely freeze the collaborative editor, and it was not reacting anymore after a while, and undo . It's just not a pleasant experience. It was so nice to see this just working so well with Yjs. With one of the key points where I was able to present, with stakeholders that were evaluating the technology on the client side with me to say, hey, this is really working great in practice. Just try pasting a big document. Now try the same in Google Docs. It's like a difference of night and day.

- Absolutely. Yeah, and I think one of the things that's interesting is if you look at Yjs in relation to some of the other tools that are out there and available, I know that, for example one of the, some of the very common tools people use out there are things like CKEditor 5, ProseMirror collab, as well as Draft.js. But one of the things I know is that there are certain features that are lacking. I know that you don't want to name and shame, Fabian. But I'm curious, what makes Yjs superior? One of the things I know, for example, that Yjs does better is one thing you mentioned earlier, Kevin, around content annotations, being able to directly apply those annotations. What are some other examples of how Yjs is better than some of these other solutions and stacks like ShareDB, Automerge, the CKSource service? What are some of the things that you both found were either better in Yjs, or lacking in others, or things that you noticed?

- First of all, just to get it out of the picture, the CKSource solution unfortunately is proprietary. It's a server black box. You have to pay for it. Which is fine, that's fine. But as I've already outlined a lot on this call, open source is just better. Because, well, I mean, Drupal is open source, many technologies are open source, and those open source technologies, they thrive, because companies invest in them, they mature, and everyone can use them, and that's so important. That puts out the proprietary solutions. They might be useful for the one where each project of that, but they're not useful, in my mind, to the larger community in that. And for Automerge, ShareDB, I'll let Kevin speak. expert.

- Yeah. So, about these two projects, ShareDB, it was always my goal to be as good as ShareDB. It's a really cool project. And I was inspired a lot of the features they have, because they also have the notion that you can just manipulate your data and then everyone gets synced up. And I love that idea. At the time when I created Yjs, they didn't support, for example, objects as key values or so. And this was like, okay, it can't be so hard to implement that. I also want to make it peer-to-peer. So, this is why I created Yjs. And it only took me six years to do that, but that's fine. So, I think Yjs against ShareDB, they're both great projects, but ShareDB is based on operational transformation which is centralized, and I always loved the idea of having a decentralized system. And Automerge, also a really cool project. I really like the maintainer of that project. He is really active in the research community, and he's a lot more credible than I am in the papers he created. He is a really good writer. And if you get the chance, and are interested in CRDTs, you should definitely check out one of his talks about CRDTs because he really explains it well how it works. But now against Automerge, right now, Automerge is not as performant as Yjs. That's just what I want to say there. Yjs is really focused on shared text editing. And Automerge also supports text editing, but it's not as fast. I created some benchmarks. Maybe we can link that too. You can also find it on the GitHub repository of Yjs. But yeah, these are the main reasons against that. It still needs to be seen which algorithm is better. But the thing is, in Yjs I had more time to create Yjs and I implemented a lot of optimizations especially for shared text editing.

- Yeah, and it's also a very important point in if you want to do a collaborative system for application with a JSON structure, try out Automerge, try out Yjs. We are open source. There's always a good solution for your project that you need. But if you want to do text editing then Yjs gives you this undo manager. It gives you these functionalities for rich text. It gives you this control. It gives you this basic server component if you need. With Automerge, you build everything your own. You can do that, it's fine. People have done that with certain editors. But it's really Yjs gives you a headstart and a framework to work with shared editors here especially.

- Yeah, and this is really interesting. I think one of the things... By the way, just to help our viewers and our listeners, we keep on throwing this CRDT acronym around. It's actually one of the big reasons why I think Yjs is so compelling. CRDT, by the way, stands for commutative replicated data type. You can look at it on Wikipedia in the operational transformation article, very, very useful. But I think just to dig into CRDT a little bit briefly, I think one of the really key aspects of Yjs that we've obviously been touching on here and there is the fact that because of this focus on data types it's very powerful for rich text editing, more so than some of the, especially for those who need more than just the plain text editing feature. But I think this is actually just one example of some of the really interesting features in Yjs. One of the things that I found really interesting is because of that kind of agnosticism and because of that focus on that kind of lower level, we actually find that other things are possible with Yjs, like using multiple document formats. So you can use rich text, you can use markdown, you can use HTML, anything that's kind of a parsable AST. What I'm curious though is that there is a topic very near and dear to my heart which I know that Kevin, you focused on, which is actually the notion of accessibility. So I'm curious, how exactly does accessibility work in the context of realtime collaboration for Yjs? Especially given the fact that rich text editing, realtime collaboration, both very, very difficult things to do in an accessible way. Are you using things like ARIA labels? What are your thoughts on accessibility with realtime collaboration?

- Accessibility is firstly an issue for the editors. Yjs is not really concerned about accessibility because it's not really part of the framework. But if the editor supports accessibility then it's baked into Yjs. By the way... No, actually that's all I want to say about that. Most editors nowadays are really accessible so there's not a lot of concern there. I'm also not an expert in accessibility, but I'm also really concerned about what happens when you hit undo/redo, for example. Which is, by the way, not working in Google Docs or in most editors. Try hitting Edit and then Undo in Google Docs. It doesn't work. And I'll figure out why.

- Very interesting. Wow.

- But maybe this is part of the discussion when we talk about editor support, or which editor we choose, Tag1, and for the company that we contract for.

- We'll do a followup talk on that. I think that the whole editor component, we did a ton of research. So maybe we'll do a Tag talk next on the whole editor component and how we ended up with ProseMirror and the integration of all that.

- Yeah, because that's an area I'd love to dig into in terms of ProseMirror's capability. CKEditor also has really great accessibility features. But how that relationship comes together, how Yjs really integrates with some of these editors and how those features get exposed, that's a really interesting topic that I know we're gonna have to save for next time. In these last few minutes here... We've covered a lot of ground here. We've covered realtime collaboration. We've covered some of the problems with concurrency in these editing tools. We've also talked about CRDT from the very, very high level standpoint. I'm curious now, I think one of the things people are interested in is, well, what is the future of Yjs, Kevin? What exactly do the next couple years hold for Yjs? What are some of your goals in the future, moving forward?

- My goals. My goal is to create editor support for all major editors out there right now. There's currently editor support for code editors like Ace, and CodeMirror, and there's rich text editors, for example ProseMirror and Quill support. There are many interesting additions to Yjs, and I want to increase that number. I want a lot of people using that technology. Because if you use Yjs as the entry point for your data model you can have different text editors. For example, you can have CodeMirror and Ace, like one user uses that and the other uses a different editor, and that's a really interesting idea for me. The other thing is, what I'm really interested in is real decentralized systems. And there's mainly the dotProject and the IPFS project, and building on top of that system, like each centralized system, wow, it's so amazing. You can build an application without a central server only having, well, peers, just working stations meshed together and they somehow create an environment where you can store data, and that's very interesting to me.

- I think that is a very, very compelling idea that I'd love to talk more about with you, Kevin. Yeah, the idea of having a completely serverless implementation I think is very interesting. Well, I did want to say that we are out of time. However, I think one of the things I wanted to call out is something, Fabian, you said at the very beginning of this whole broadcast, which is, this should be something that's part of every CMS and online application. We should be building interfaces that are editable. We should be really enabling these content editor workflows. And I think clearly Yjs has the right idea, has the right journey in mind. And I can see, given that Tag1 is focused not just on Drupal but also all of these other technologies, and building things like central Node.js backends for these editors, all of that sort of work really highlights, I think, the notion that realtime collaboration is a very important concern for everybody. Not just developers, but also our content editors and our marketers who are working with us on a daily basis. Any last words about realtime collaboration, Yjs, something you want to leave our viewers and listeners with?

- I think it's just an awesome technology. And really, even repeating myself, it is such a difference if you start using the system, even if we are just in demo mode as we are pre-alpha, to just collaborate with someone within the Drupal scene. It feels so different to being in Google Docs or being isolated because it's where you create your content normally, it's where you work together, it's the user tools. You can even attach a file directly. You don't have to upload it to Google Docs and later upload it to Drupal. You really have the experience in that you can use a media browser, you can use a media browser, your whole media library that's on the CMS, it's not in Google Docs. You select your file, it's interactive, and it's collaborative. Your peers will see it as well. And I think that's just fantastic, and that's why I'm saying realtime systems for editors, but also realtime updates. Like, I made an update, you see that I made an update directly on your screen. That's kind of, in my mind, the future of CMS's and, in a way, also the web.

- And this is intended to be a CMS-independent solution. We look forward to adding this to Django and Wagtail, to WordPress. Every CMS should have this. I'd also say that we just scratched the surface of all of these technologies. I think this is a really interesting conversation so we'll definitely setup some future talks to dig more into the details, whether it's the editor or the underlying tech, to get into the nitty gritty.

- Absolutely. I think that today we've done a very good overview of the what, the how, and the Yjs. And I want to go ahead and wrap things up here. If you have any thoughts on how this show, what you're interested, in certain topics, please feel free to reach out to us. I want to thank our guests today. Fabian Franz, senior technical architect and performance lead at Tag1. Also Kevin Jahns joining us all the way from Berlin, right. The creator of Yjs as well as a key expert and contributor to realtime collaboration. And of course my partner in crime Michael Meyers, managing director at Tag1. Thank you all so much, and we'll see you next time for Tag1 Team Talk. Thank you.

Sep 18 2019
Sep 18

BADCamp is only a couple of days away!!

With the shift in venues, we thought we'd put together a list of local places that serve up good grub. Knowing that the attendees of BADCamp are diverse, so are the restaurants on our list. We have indicated if we have confirmed that they serve vegetarian and vegan cuisine.

Angeline's Louisiana Kitchen

Angeline's brings the flavor and atmosphere of a New Orleans neighborhood restaurant ​to downtown Berkeley, with great music, libations and the classic dishes invented in the Big Easy's greatest kitchens.

(Vegetarian, Vegan)

Cancun

Founded in 1994 by Jorge Saldana, Cancun opened its doors downtown providing a central location for easy, dine-in or takeout, healthy Mexican food in Berkeley. Beloved from the beginning, Cancun, with its local farm to table ingredients, fresh salsa bar, big open space and high ceilings, continues to offer homemade traditional Mexican dishes, made to order with love.

Eureka

Discover the Eureka! all-American culture through one-of-a-kind experiences, weekly events such as Steal The Glass, daily “Hoppy” Hour, and an inventive rotating beer and craft beverage program, small-batch whiskeys, and a mouthwatering menu featuring gourmet burgers.

(Vegetarian)

Jupiter

Housed in an old livery stable from the 1890's, with interior inspired by the oldest bar in Berlin, Jupiter exudes charm & rare atmosphere. Steps off BART, in the heart of Downtown Berkeley, this brewhouse features two stories of seating, a heated beer garden, live music, delicious food & incredible local beer.

The Butcher's Son

Yes, everything is vegan at The Butcher's Son.

(Vegetarian, Vegan)

Ippudo

Every steaming bowl is an “arigato” – a thank you, which Ippudo serves to their customers together with a smile.

The Veggie Grill

At Veggie Grill, vegetables are the rockstars! They see every season as an opportunity to create bold and delicious ways to bring people together.

(Vegetarian, Vegan)

top dog

top dog grew out of a boy's love of sausage, a staple in his German immigrants' New York home over the WWII years. Steaks? Tubesteaks! His paper route to a well-mixed neighborhood assured that Italian, Polish, even Hungarian sausages were soon no strangers to that developing appetite and palate. Nor had he far to go to a cart or stand offering kosher style "Franks", usually steeped but better off the griddle.

(Vegetarian)

Tuk Tuk Thai

Tuk Tuk Thai is a aid-back cafe serving popular Thai dishes like curries & noodle soups & offering delivery too.

(Vegetarian, Vegan)

Maison Bleue

Maison Bleue a little taste of France.

(Vegetarian)

Revival

Revival Bar + Kitchen is a ustainably Sourced California Brasserie and Cocktail Lounge.

(Vegetarian, Vegan)

The Flying Falafel

The Flying Falafel serves Mediterranean goodies with an aerial twist. Falafel cooked to order, hummus, and veggies galore. Serving catering and party platters.

(Vegetarian)

Venus

Custom California Cuisine made from local and sustainable ingredients.  Everything that comes from The Venus kitchen is handmade. No cans. No freezers.

(Vegetarian)

Sep 18 2019
Sep 18

Many thanks to Kaleem Clarkson (kclarkson) and his team for organizing a great DrupalCamp Atlanta. I had a time of learning, connecting and being inspired!

Doug Vann, Carole Bernard, and Rudy Dodier in a selfie

I started my day at DrupalCamp Atlanta by participating in the workshop “Introduction to Drupal,” led by longtime Drupal community member Doug Vann (dougvann). Joining me was Rudy Dodier, from Platform.sh. Doug covered everything from the history of Drupal, to setting up a basic website to how the word “system” in Content Management System can be an acronym for: Saves You Some Time Energy Money.

I took copious notes, as I continue to connect the dots to the power of the Drupal project - to how it is leading a digital transformation across industries. I absorbed it all, and was eager to learn more. I met other developers and individuals who contribute so much to the Drupal project and to the Drupal community. From my conversations with Ray Saltini (rgs) and Mike Anello (ultimike) to Suzanne Dergacheva (pixelite), I was struck by the level of commitment demonstrated by the community. You'll get a sense for this in Suzanne's slides for her Growing the Drupal Community talk.

Suzanne Dergacheva, Heather Rocker, and Carole Bernard standing together

Heather Rocker (hrocker) also attended and presented at the Career Fair. She spoke about the importance of the Association’s initiative on Diversity, Equity & Inclusion and the benefits that come from actively recruiting and welcoming new individuals (especially those from underrepresented communities) to lend their skills to the project.

I realize the extensive number of stories that are within this vast and passionate community, and I am excited to promote and talk about them. I am looking forward to being a communications and marketing advocate for the Drupal community, the Drupal project and the Drupal Association. From the specific needs of developers, to the importance of broadening our audience, to the necessity of career fairs to bring students on the Drupal train, and to the need for marketing to grow Drupal adoption, I heard and learned so much in a short visit to Atlanta. But, what impressed me as much as the day was the contagious enthusiasm for what the community is doing and for what it can accomplish!

Slide showing faces of the 8 Atlanta Camp Leaders - ATL Board Members - Kaleem Clarkson, Brandon Firfer, Sarah Golden, Adam Kirby, Trish Smith, Nikki Smith, Dominic Thomas, and Advisor Dave Terry
The DrupalCamp Atlanta Leadership Team, without whom the event wouldn't have been possible!

ATL team onsite together doing a power pose.

Thanks to everyone who came out for #DCATL this year! We loved meeting y'all. Let's continue to grow and support the @drupal community! pic.twitter.com/q9IgHjVRkA

— DrupalCamp Atlanta (@DrupalCamp_ATL) September 14, 2019

Sep 18 2019
Sep 18

The Drupal world is looking forward to the “ninth edition” of the great drop. It’s going to continue Drupal’s chosen path with its API-first orientation, the latest versions of third-party libraries, advanced editor-friendliness, and much more.

The Drupal 9 release is scheduled to come on June 3, 2020. One more big step has just brought it closer — the Drupal 8.7.7 version.

This is a very special update indeed. In this blog post, we will tell you what’s new in it and why you should update to Drupal 8.7.7 as part of your site’s easy and smooth journey to Drupal 9.

What’s new in Drupal 8.7.7: a big surprise inside

Drupal 8.7.7 came on September 4. Although it’s not even a minor update — it's been called a patch release — it brought a major new feature by introducing a new core versioning system that helps websites be more ready for Drupal 9.

New core version requirement key in Drupal 8.7.7

The new core version requirement key is meant for module and theme creators to mark their project’s compatibility with Drupal core versions. This compatibility can apply to multiple versions — for example, D8 and D9. This was not possible with the old “core: 8.x” key.

Drupal core contributor Gábor Hojtsy described this feature in his blog post. He emphasized that 8.7.7 is the first release to support modules compatible with multiple major core versions.

Gábor Hojtsy's quote about Drupal 8.7.7

The key is to be added to the info.yml file of the module or theme:

name: My Module Name

type: module

core_version_requirement: ^8 || ^9

Websites that update to Drupal 8.7.7 now will have another benefit. The new requirement key also allows marking the compatibility with minor and patch releases (e.g. ^8.7, ^8.8, ^8.7.7, ^8.8.0, ^8.7.23, and so on). “Such precision was never possible before!”, Gábor Hojtsy writes.

What happens to the old “core: 8.x” key?

  • The old key still exists for core versions prior to 8.7.7. They do not work with the new core_version_requirement key. So developers will need to list both types of keys in order to allow their code to work with the older and newer versions.
  • If modules or themes are not supposed to work with versions older than 8.7.7, it’s enough to just use the new key.

You can read more details about the key in the drupal.org announcement. But why is marking the Drupal 9 compatibility so important, and how should an update to D8.7.7 help your website’s future? The reason comes next.

Drupal 8.7.7 is a step towards your smooth upgrade to Drupal 9

The topic of D9 compatibility for modules and themes is so hot because many D8 websites have a chance to be fully ready for Drupal 9 the day it arrives.

D9 is being built on the basis of D8, but without deprecated code and with new versions of third-party libraries. Websites that are clean from deprecated code and are kept updated will be instantly welcomed to “cloud number 9”!

So if you update to Drupal 8.7.7 now, you will be closer to D9 than ever. Thanks to the new core versioning system, your modules and themes will be ready to “jump” to the ninth Drupal as soon as it comes.

Update to Drupal 8.7.7 and prepare for Drupal 9 with our team!

Our development team will help you prepare for Drupal 9 easily:

  • perform your website’s update to Drupal 8.7.7
  • update your modules and themes
  • find and replace the deprecated code
  • apply the new core version requirement key
  • follow all subsequent updates for your Drupal 9 readiness
  • make an upgrade to Drupal 8 (if you are still with the seventh version)

The future looks really bright for websites that follow the latest recommendations. Let yours be among them!

Sep 18 2019
Sep 18

It’s great to live at a time when a robust CMS can share its content with an ultrafast JavaScript front-end. One of the best examples of this is combining Drupal and GatsbyJS. We are happy to see a new tool for it that is fresh from the oven — Gatsby Live Preview module for Drupal 8. 

It provides Drupal editors with easy content creation workflows, making Drupal and Gatsby integration a more lucrative idea for developers and website owners.

GatsbyJS: a great companion for Drupal

The GatsbyJS modern site generator inspires the Drupal community more and more for many reasons. Here are at least a few:

  • It is based on the hottest technologies such as the ReactJS front-end framework, the GraphQL query language, and the Webpack JavaScript module bundler. 
  • It is amazingly fast and provides real-time content updates. Every static Gatsby site is, in fact, a full-fledged React app. 
  • It comes packed with 1250+ source plugins to retrieve data from particular data sources. This includes the Drupal source plugin that connects your Drupal site as a data source to your Gatsby site.
  • It has 250+ starter kits to quickly set up a Gatsby website that will display your Drupal 8 data.
GatsbyJS starter kit exampleGatsbyJS starter kit example 2

The Gatsby Live Preview module in Drupal 8

The contributed module called Gatsby Live Preview allows Drupal content editors to make content updates and immediately see how it will look on the Gatsby site before deployment. 

This easy content creation is provided by showing Drupal on the left and Gatsby on the right:

Gatsby Live Preview module in Drupal 8

The maintainer of the module, Shean Thomas, gave a talk and showed slides of the Gatsby Live Preview module at Decoupled Days in New York on July 18, 2019. 

Thomas explained the problem that the module solved. Previously, there was no easy way to see during content creation how changes would look like before you click “save.” Among the available options was to run the Gatsby development server before deploying the changes to live, which required the entire site regeneration. 

According to Shean Thomas, among the plans for the future is integrating the module with the Drupal 8’s Content Moderation module. The core Content Moderation and Workflows modules take content creation to a new level through handy editorial workflows in Drupal 8

The module is very new with its alpha release out on August 14, 2019. It is based on the tool introduced by the Gatsby team — the Gatsby Preview Beta

Steps to install and configure the module 

This part comes when the main setup is complete. So we assume you are done with:

  • Gatsby site creation
  • Gatsby Source Drupal plugin installation (version 3.2.3 or later)
  • configuring the gatsby-config.js file to list your Drupal website’s address
  • building up your Gatsby pages to display Drupal content

So the live preview setup steps are as follows:

  • install and enable the Gatsby Live Preview Drupal module the way you prefer
  • set up a Gatsby cloud preview account
  • set the “preview” flag to “true” in the “options” (the Gatsby Source Drupal plugin’s file)
  • Gatsby is now ready to follow the content changes at a particular URL
  • copy the preview URL from the Gatsby cloud to the “Gatsby Preview Server URL” (Configuration — System — Gatsby Live Preview Settings of your Drupal admin dashboard)

Examples of easy content creation & preview with the module 

The Decoupled Days' speech about the Gatsby Live Preview module greatly inspired the Drupal community. In order to make it easy for people to get started with Drupal and Gatsby integration. 

Drupal contributor Jesus Manuel Olivas decided to improve some features in the module.

The developer also added this setup to projects based on the Drupal Boina Distribution and shared his impressions about the module in the blog post with the video. Let’s have a look at this easy content creation process:

  • On the left side, we see the Drupal site where some content is added via the admin interface. 
Gatsby Live Preview module in Drupal 8 exampleGatsby Live Preview module in Drupal 8 example
  • On the right side, we see the Gatsby site update immediately after the “Save” button is clicked in Drupal.
Gatsby Live Preview module in Drupal 8 example

 

Get the most of Drupal and Gatsby integration!

Our developers will help you enjoy the incredible speed that GatsbyJS is able to give to your Drupal website! They can:

  • set up a Gatsby website and establish content retrieval from Drupal
  • build your Gatsby pages exactly according to your wishes thanks to GraphQL
  • install and configure the module for Drupal and Gatsby live preview for easy content creation

Our Drupal team are masters of modern JavaScript technologies. You can entrust this integration to us!

Sep 18 2019
Sep 18
many different people profile heads in different colors

Part 3 of the "Mastering Drupal 8 Multilingual" blog series provides site building and front-end tips and techniques to improve the multilingual experience for both editors and end-users.

Previous posts in this series covered planning and budgeting for multilingual, as well as the process for installing the modules needed and the basics of content, configuration and interface translation. If you missed posts one or two, you may want to read those posts before proceeding.

Sep 18 2019
Sep 18

Your browser does not support the audio element. TEN7-Podcast-Ep-070-Using-Kubernetes-for-Hosting.mp3

Summary

After months deep in the weeds of Kubernetes, our DevOps Tess Flynn emerged with the best practices for melding Docker, Flight Deck and Kubernetes to create a powerful open source infrastructure for hosting Drupal sites in production (powered by our partner, DigitalOcean). Ivan and Tess take a deep dive into why we chose this combination of tools, our journey to get here, and the nitty gritty of how everything works together.    

Guest

Tess Flynn, TEN7 DevOps

Highlights

  • Why offer hosting ourselves now?
  • Differences in hosting providers
  • The beauty of containerization, and the challenge of containerization
  • The best container orchestrator
  • What’s with hosting providers and their opaque pricing? (and why we like DigitalOcean)
  • Kubernetes’ highly dynamic environment: updated with just a code push
  • Flight Deck, the genesis of our journey to Kubernetes
  • Docker enables consistent environments
  • Flight Deck + Kubernetes + DigitalOcean
  • You can do this all yourself! (or we can help you with our training)
  • It all runs on Drupal OR other platforms
  • In order to enjoy Drupal + Kubernetes, you must let go of your local file system and SSH, and reevaluate your email system
  • Complex files vs. static files and S3
  • Kubectl! (it sounds cuter when you say it out loud)
  • Cron jobs run differently in Kubernetes
  • A Tess talk isn’t complete without a car analogy: Kubernetes is like a garage that comes pre-stocked with all the tools you’ll need to work on your car

Links

Transcript

IVAN STEGIC: Hey everyone! You’re listening to the TEN7 podcast, where we get together every fortnight, and sometimes more often, to talk about technology, business and the humans in it. I am your host Ivan Stegic. We’ve talked about DevOps at TEN7 on the show before. We’ve done an episode on why we decided to expand our hosting offering to Linode back at the end of 2017. We’ve talked about why we think it’s important to have a good relationship with your hosting company. And, we’ve written about automation and continuous integration over the years as well.

For the last year or so, we’ve been working on our next generation of hosting service, and our DevOps Engineer, Tess Flynn, has been deep in the weeds with Kubernetes. Today, we’re going to spend some time talking about what we’ve done—and how you could be doing it as well,—given that we’ve open sourced all of our work.

We’re also rolling out training at BadCamp this year, that’s in October of 2019, and we’ll be at DrupalCorn as well, in November. So, we’ll talk about that and what you might learn by attending. So, joining me again is our very own Tess Flynn. Hello, socketwench.

TESS FLYNN: Hello.

IVAN: Welcome, welcome. I’m so glad you’re on to talk shop with me. I wanted to start with why. Why are we hosting our own sites and those of our clients? There are so many good options out there for WordPress, for Drupal: you've got Acquia and Pantheon, Blue Host, and others. We typically use the provider that makes the most sense, based on our clients’ needs.

We’ve had a close relationship with ipHouse and their managed hosting services for a long time. But why start hosting now? For us, as an organization, it’s kind of been the perfect storm of circumstances, from the technology being mature, to the cost of it, and the availability of it, to where we are as an organization from a developmental point of view, to even being more conscious of vendor lock in and actively trying to avoid it.

So, I want to talk about technology a little bit more with you, Tess. What’s so different now than it was a few years ago? Why is it suddenly okay for us to be hosting ourselves?

TESS: There’s been kind of an explosion over the last few years of managed Kubernetes hosting providers. Now, we’ve had managed hosting providers forever. We’ve had things that are called Infrastructure as a service (IaaS) provider; that’s going to be things like AWS and Google Compute Cloud, as well as other providers, including DigitalOcean, but also say, Linode and other ones, which just provide raw hardware, virtual machine and root login. Lately, however, a lot of people would rather break up their workloads into containers, using something that’s similar to Docker. And I’ve talked about Docker before, but Docker is an alternative take on virtualization technologies, which works on taking applications and putting them in their own individual, virtual environment. I’m glossing over so many things when I say that, but it gets the general point across, with the two minutes before everybody else falls asleep.

IVAN: Right.

TESS: What’s really nifty about putting applications into a container is that now the container doesn’t really care where it is. You can run it on your system, you can run it somewhere else, you can run it on a hosting provider. And, the great thing about these containers is that you can download ones that other people have created. You can modify them, make your own, and you can string them together to build an entire application service out of them. And that’s really, really great. That’s like infrastructure Legos.

But the problem is, once you get the containers, how do you make sure that they’re on the systems, on the actual hardware where they are supposed to be, in the number of copies that there’s supposed to be, and that they can all talk to each other? And the one’s that aren’t supposed to talk to each other, can’t? That’s a lot trickier. For a long time the problem has been that you really only have two solutions: you do it yourself, or you use something like Docker Swarm. I don’t have the greatest opinion of Docker Swarm. I have worked with it before in a production environment, it’s not my favorite.

IVAN: It’s a little tough, isn’t it? We’ve had a client experience on that.

TESS: It’s a little tough, yeah. It’s not really set up for something like a Drupal workload. It’s set up more for a stateless application. A prototypical example is, you need to calculate the progression of matter within the known galaxy, factoring a certain cosmological constant. Take that variable, set it into a compute grid and go, “Hey, tell me what the results are in 15 years.” But you don’t really do that with Drupal. With Drupal, you’re not just going to send off one thing and always get the same thing back. There’s going to be state, which is preserved. That’s going to be in the databases somewhere, and there are going to be files that are uploaded somewhere. And then you have to get load balancing involved, and then it gets really complicated, and it’s like ugh. I really didn’t like how Swarm did any of this stuff. It was very prescriptive. It was, you do it their way, and nothing else.

IVAN: No flexibility.

TESS: No flexibility at all. It was really, really not fun, and it meant that we had to do a lot of modification of how Drupal works, and incur several single points of failure in our infrastructure, in order to make it work in its form. That whole experience just did not get me interested or excited to make a broader Swarm deployment anywhere else.

Then I ran across Kubernetes, and Kubernetes has a very different mentality around it. Kubernetes has more different options for configurations, and you can tailor how Kubernetes manages your workload, rather than tailoring your workload to work with Docker Swarm. That’s why I really liked it. What's really nifty is, once you have Kubernetes, now you have an open source project, which is platform agnostic, which doesn’t care about which individual hosting provider you’re on, as long as you have containers, and you can send configuration to it somehow, it’s fine, it doesn’t care.

A lot of managed hosting providers are going, “Hey, you know, VMs [virtual machines] were kind of nifty, but we really want to get in on all this container stuff now, too.” “Oh, hey, there’s a container orchestrator,” which is what Kubernetes is, and what Docker Swam is, as well, a container “orchestrator” which does all of the making sure the containers are on the right systems, are running, they can talk to the containers they're supposed to, and can’t talk to containers they're not supposed to.

That made a lot of infrastructure providers go, “This is not really a Platform as a service anymore. This is another form of Infrastructure as a service. As such, that is a segment that we can get into."

So, first it started with Google Kubernetes Engine, which is still considered today the defacto version. Amazon got into it, Azure got into it. And all of these are pretty good, but a lot of these huge cloud service providers, you can’t get clear pricing out of them to save your life.

IVAN: Yeah. That’s so frustrating, as a client, as a business owner. How do you do that? It’s insane.

TESS: I mean, the only way that it seems that is deterministic, in order to figure out what your bill is going to be at the end of the month, is to spend the money and hope that it doesn’t kill your credit card. [laughing]

IVAN: Yeah, right, and then try to figure out what you did, and ways of changing it, and then hell, you’re supposed to be just charged that every month from now on, I suppose.

TESS: It’s just a pain. It wasn’t any fun, whatsoever. So, an alternative approach is, you could actually install Kubernetes yourself on an Infrastructure as a service provider with regular VMs.

IVAN: And, we considered that, right?

TESS: Oh, I considered it, and I even spun that up on a weekend myself. It worked. But the problem is, I’m a colossal cheapskate and I didn’t want to spend $30.00 a month for it. [laughing]

IVAN: [laughing] If only there was a supporting ISP that had free Kubernetes support, and just charged you for the compute engines that you used.

TESS: I was really kind of sad that there wasn’t one, until six or eight months ago, when DigitalOcean announced that they have in beta (now it’s in production) a Kubernetes service, where the pricing was incredibly clear. You go to the cluster page, you select the servers that you want to see (the nodes as it were). I know, Drupal nodes, infrastructure nodes, it’s really confusing. Don’t even get physics people involved, it gets really complicated. [laughing]

IVAN: No, please. No, don’t. [laughing]

TESS: But you select which servers that you want to have in your Kubernetes cluster, the sizing, and the price is just listed, right there, in numbers that you can understand! [laughing]

IVAN: Per month, not per minute.

TESS: I know, per month, not per minute.

IVAN: It’s just the small things. Crazy.

TESS: And, it really targeted the kind of market that we are in for a hosting provider, and it made me really excited, and I really wanted to start putting workloads on it, and that’s what started the entire process.

IVAN: It really was, kind of a fortuitous series of events, and the timing kind of just really worked out. I think one of the biggest things for us, for me, is that with Kubernetes, we don’t have to worry about patching and security updates, and monitoring them, and these large hardware machines that we have to keep patched and updated. Essentially, it’s updated every time we do a code push, right? I mean, we’re still concerned with it, but it’s a much easier burden to bear.

TESS: Right. Now what’s going on is that, every time that we do a push, we’re literally rebuilding every system image necessary to run the underlying application. Which means that if we need to push a system update, it’s really just a matter of updating the underlying container's base image to the newest version. We’re already using Alpine Linux as our base containers, which already is a security-focused minimal container set.

IVAN: So, this is actually a good segue to what I wanted to talk about next. A few years back (as opposed to six to nine months back), which is how we kind of got down the road to get to Kubernetes was, I think the origin of all this really is, Flight Deck, and the desire for us to make it easy for developers who work at TEN7—and anyone else who uses Flight Deck, honestly—to have the same development environment locally. Basically, we wanted to avoid using MAMP and WAMP and different configurations so that we could eliminate that from any of the bug-squashing endeavors that we were going into. So, let’s talk about this started with Docker and led into Flight Deck, and what a benefit it is to have the same environment locally as we do in staging and production.

TESS: So, there’s a joking meme that’s been going around, and DevOp cycles, of a clip of a movie where, I think a father and son are sitting and having a very quiet talk on a bench somewhere in a park, where the kid is saying, “But it works on my machine.” And then the Dad hugs him and says, “Well, then we’ll ship your machine.” [laughing] And, that’s kind of what Docker does. But joking aside, I wanted to get that out of the way so I’m not taking myself too seriously. [laughing]

So, one of the problems with a lot of local development environments—and we still have this problem—is that traditionally we’ve used what I consider a hard-installed hosting product. So, we’re using MAMP or WAMP or Acquia Dev Desktop, or if you’re on Linux you’re just installing Apache directly. And all of those work fine, except when you start working on more than one site and more than one client. So, suddenly you have this one problem where, this one client has this really specific php.ini setting, but this other client can’t have that setting. And MAMP and WAMP work around this through a profile mechanism which, underneath the covers is a huge amount of hyperlinking and weird configurations, and spoofing, and like eww, it makes me shutter.

IVAN: Yeah, it makes me cringe just to talk about it, yeah.

TESS: And, the problem is that, every time you have to do this, every developer has to do this themselves, they can’t just standardize on it. So, if somebody has an individual problem on their system, that only happens on their system at 3:45 on a Thursday, after they’ve had chili for lunch or something or other, then you can’t really reproduce it. So, the solution really is, you need to have replicatable, shareable, consistent development environments across your entire team. And that’s what Docker does.

Docker provides that consistency, that shareability, and makes sure that everybody does, in fact, have the same environment across the board. That’s the entire point of that, and that’s where the whole joke about, “Well, then we’ll ship your machine,” [laughing] because that is in essence what containers are. They are system images that run particular bits of software. Now, once we moved everyone to Docker for development, we now had a consistent environment between all of our systems, so that now we didn’t have to work about a number of different problems.

Another good example is, this site uses PHP 5, this site uses PHP 7—a little out of date now, but it was very relevant two years ago—in which case, how do you make sure you’re on the right version? Well, with Docker, you change a text file, and then you boot the containers up, and that’s it.

IVAN: And that text file lives in a code repository, right? So, everybody else gets that change?

TESS: Mm hmm, because you are literally sharing the same environment; you are enforcing a consistent development environment across your entire team for each individual project. And, if you use that strategy, you have something that is flexible, yet at the same time incredibly consistent.

IVAN: And this is really important across all of our developers, and all of our local development that we do, but the challenge then becomes, how do you consistently replicate this in a staging or in a test environment, and even in production? So, that’s kind of the genesis of how we thought Kubernetes could help us here, right?

TESS: Right.

IVAN: So, the challenge to you from me was, how do we make this work in production?

TESS: So, the nice thing about Flight Deck is, it was always designed with the intention of being put into production, But the orchestration component just wasn’t there, and the hosting component wasn’t there. Kubernetes showed up, and that solved the orchestration component, and then, eventually DigitalOcean showed up and now we have the hosting component. So, now, we have all the pieces together to create a consistent environment that is literally the same containers, from the first time someone starts working on the project, to when it gets deployed to production. That is the height of continuous integration ideals, to make sure that you have consistency across all of your environments. That you don’t have different, weird shared environments along the way, that everything is exactly the same so that you know that it will work.

IVAN: I want to stop right there, just so our listeners can appreciate the power of what you just said. You basically said, “I’m going to be working on a website, or a web application locally, with some sort of stack of required server components, whose version numbers and installation profile is configured in a certain way. My teammate is able to replicate that environment exactly, to the version, simply by using the same repo, and by using Flight Deck.

Moreover, all of those version numbers and the stack that is being used, is actually also the same now in staging and, most amazingly to me, in production. So, we can guarantee that what container is functioning in production on the Kubernetes cluster, is actually on staging and on everyone else’s machine. We’ve totally eliminated any variability and any chance that the environment is going to be causing an issue that one person may be seeing that another isn’t.

TESS: That’s correct.

IVAN: That’s pretty amazing!

TESS: It’s a really difficult thing to do, but starting with the containers and building that from the base up actually makes it a lot easier, and I don’t think that any other local development environment, even container based local development environment such as DDEV and Lando are doing this quite yet. Last I heard, I think DDEV was working on a production version of their containers, but it’s not the same containers, whereas with Flight Deck, it literally is the same container.

IVAN: It’s the same configuration. Everything is the same. That’s pretty amazing. I’m still kind of really impressed with all of the stuff that we’ve done, that you’ve done. And, honestly, this is all open source too. This is not like TEN7’s proprietary product, right? We’ve open sourced this, this is all on the web, you can download it yourself, you can figure it out yourself, you can do this as well. You can start your own hosting company.

TESS: That’s correct. The key item which puts all this together is, the Ansible role called Flight Deck Cluster. What Flight Deck Cluster does is, it will create a Flight Deck-flavored Kubernetes cluster and it works perfectly well on DigitalOcean. There’s no reason why it can’t work on say, Google Kubernetes Engine or AWS or anyone else. The architecture that Flight Deck Cluster uses is meant to be simple, durable and transportable, which is something that a lot of other architectures that I’ve seen just don’t have.

IVAN: So, we’ve designed a lightweight set of Docker containers called Flight Deck that you can use locally. We’ve evolved them so that they work with Kubernetes, which you can deploy anywhere in staging and production. We’ve open sourced them. And, the fact that it runs Kubernetes, all you need is a service that supports Kubernetes and you should be able to run all of this in those other locations.

So, we’ve talked about how we started with Docker and how that evolved, and I talked about how we've open sourced it and it’s available to you. I want to spend a little bit of time getting into the details, into the nitty gritty of how you would actually do this for yourself. Is there an app I download? Is it all the YML, all the YML files that we’ve open sourced? What would someone who wants to try this themselves, what would they have to do?

TESS: The first thing that I would probably do is, start running Flight Deck locally. Because you don’t need to pay any extra money for it, you just need to use your local laptop, and it’s also a good experience for you to learn how to interact with Docker by itself. That looks good on a résumé and it’s a good skill to actually have.

I have a talk that I used to give about Docker, and I know that there’s a blog post series that I posted somewhere a long time ago, about how Docker actually works under the covers. Both of those are going to be invaluable to understand how to get Flight Deck working on your local environment, and once you have it working on your local environment, then the next problem is to figure out the build chain. Now the way that our build chain works is, that we have another server, which is a build server, and what the build server does, is it’s going to receive a job from Gitlab and that job is going to take all of the files that constitute the site, it will build them into a local file system, and then it will put those inside of a container which is based on Flight Deck. Then it will upload those to a container registry somewhere else. So that we already have a few additional pieces of technology involved. But the nice thing is, Gitlab is open source, Ansible is open source, and all of our build processes are run through Ansible, and the Docker registry is also open source. It's just a container that you can run somewhere. There’s also services that you can buy that will actually provide you a container registry on a fee basis. All of those are definitely options. Once you have the container in a registry somewhere, then you can run Flight Deck Cluster to build out the rest of the cluster itself.

IVAN: You make it sound so easy. [laughing]

TESS: I make it sound easy, but it’s a lot of code, but it is all open source and it is all there for you to use. Right now, our cluster is based on a development version of Flight Deck, which I’ve been calling Flight Deck 4, and this version is intentionally natively designed for a Kubernetes environment. But it still works perfectly fine under Docker Compose locally, and it is literally the containers that we are using in production right now, at this minute. All of those containers have been thoroughly documented. They have nice readmes which describe exactly how you configure each individual container. And the Flight Deck Cluster role on GitHub also has an extensive readme document which describes how every individual piece is supposed to work.

IVAN: So, the easiest way to get to all that documentation into the repo is to simply go to flight-deck.me. That will redirect you to a blog post about Flight Deck on the ten7.com website, and at the bottom of that post you’ll see links to the GitHub repos and all of the other information that you’ll need to get to that.

So, I wanted to talk about the fact that the hosting itself, the Kubernetes hosting that we have, is optimized for Drupal right now—I kind of struggle to say "optimized for Drupal." It’s just configured for Drupal. There’s no reason that Kubernetes is, and what we’ve released, is locked into Drupal. We are hosting our own React app on there. We have a CodeIgniter app that’s running, we even have a Grav CMS site on it. There’s no reason why you couldn’t host WordPress on it, or ExpressionEngine or any other php, MySQL, Apache, Varnish, Stack on it. Right? There’s nothing innately that forces you to be Drupal on this, right?

TESS: Nope.

IVAN: And that’s also from a design perspective. That was always the intention.

TESS: It’s intended to be run for Drupal sites. However, it always keeps an eye towards being as flexible as possible.

IVAN: So, I think that’s an important thing to mention. Let’s talk about some of the challenges of running Kubernetes in a cluster in production. It’s not like running a server with a local file system, is it?

TESS: [laughing] No, it isn’t.

IVAN: [laughing] Okay. Let’s talk about the opportunities of things to learn.

TESS: The biggest, scariest thing about Kubernetes and Drupal is, you have to let go of your local file system. That is the most scary thing that I have to tell people about Kubernetes.

IVAN: So, no file system, huh?

TESS: No file system.

IVAN: Does that make it slow?

TESS: Well, not really. Let me describe why. The problem is, that— and I’ve had this in my Return of the Clustering talk—is that we’re used to something which is called “block storage.” Now, block storage is pretty great. It is a literal attached disk to the server. So, it is mounted on the server, you have direct access to it, and you can store all kinds of things to it. And it’s fast, and it’s right there. It has no failover, it can’t be shared across the systems, but ehhh, whatever, we have one big server, who cares about that.

Then, if you do try building a traditional server cluster, well, you can’t quite do that. So then you get network file system involved, NFS. And then now, all of the file reads and writes occur over the network to some other centralized server. Okay, it still looks like a local block storage, it still works like block storage, so, okay, sure. But the problem with that is that network file systems, by their base nature, introduce a single point of failure.

Now, that’s not good by itself. If the NFS server goes down, your entire site no longer looks or functions correctly. But the problem is, that it also doesn’t scale either. There’s a natural limitation between the number of different replications for frontend server, servers that intercept the actual requests from people, and then send them to the Drupal backend for processing, and then push back their responses. There’s a natural limitation between those systems and those that can access NFS. And as soon as you have too many accesses, suddenly NFS is not going to be keeping up with you and your performance drops to the floor.

Also, NFS is kind of persnickety. You have to tune it. You have to make sure that it has enough RAM, enough bandwidth. You have to make sure it’s physically proximate to the rest of the servers. And, all of this is because it’s trying to replicate block storage. Now, block storage is great for a whole bunch of data, but in a cloud architect's perspective, there are really two different kinds of data. There’s complex data and static data.

And when I tell people about this, they go, “Well, what’s a complex file?” A lot of people will say, “Well, we have a whole bunch of files which are all linked together, that’s complex, right?” Nope. “Well, we have some Excel documents that’s on an NFS file, that’s complex, right?” Not really. So, what is a complex file? 

I spent hours, tried to squeeze an answer [laughing] out of the internet for this, and eventually arrived at the answer from a cloud architect's perspective: “complex files, such as the files which constitute the actual underlying disk storage for say, a MySQL database.” Data, which is written sparsely and seemingly randomly in multiple locations at multiple times with strict concurrency requirements. Now when I say that, does that sound like anything that we actually upload in a Drupal site?

IVAN: Nope.

TESS: Nope. None of it does. Block storage is required for complex data. But for static data, which is virtually everything that a Drupal site hosts, we don’t need it, it’s too much. It’s way too complicated. And, it doesn’t scale. So, what’s the solution? The solution really is, we need to treat the file system like an API. We need to treat the file system like a database. We don’t care where the database is, as long as you have an IP, a login and the correct credentials to actually get to the database, and then we have multiple readers, multiple writers. That’s what we want for a file system, right? Well, it turns out, there’s a thing that does that already, it’s called S3.

IVAN: Yes, AWS, hello. [laughing]

TESS: And the nice thing about S3 is, it’s perfect for static data. It’s API accessible and it can be made internally redundant. So, it has its own high availability built in that we don’t need to worry about. The nice thing that’s even more than that, is when we say S3, most people go, “Oh, Amazon.” No. S3 is, in fact, a standard. It is not just Amazon’s implementation of S3. There are multiple implementations of S3. So, I usually like saying an S3-compatible hosting provider. And that’s going to include anybody who runs any kind of S3-compatible service. And there’s actually an open source product called Ceph that actually provides an S3 frontend for file storage. And that is actually a service that DigitalOcean also provides. They have DigitalOcean spaces, which provide an S3-compatible static file interface, that’s actually powered by a Ceph cluster underneath the covers. So, open source all the way down to the core.

IVAN: Well, I didn’t know that spaces was Ceph underneath the covers. That’s cool.

TESS: It’s just buried in there. You could find it though.

IVAN: Cool. So, file storage is a challenge, but we fix that by using S3.

TESS: Yep, because Drupal 7 and 8 actually have very good S3 support. There’s S3 FS, that particular module which is excellent for doing Drupal 7 sites. We’ve been using Fly System for Drupal 8 for a few different reasons, but there are reasons that are a little bit easier for us. But your mileage may vary.

IVAN: And, if you’re going to host something that’s not Drupal related, you would need to find some other S3-compatible layer module, right?

TESS: Like for the CodeIgniter application, we are currently looking at implementing that as well.

IVAN: And, there’s a React app as well that we’ve deployed. That uses the underlying Drupal site, though, doesn’t it?

TESS: Yes, that doesn’t actually need a local file system.

IVAN: There’s no SSH access to a cluster of Kubernetes, is there?

TESS: Yes, that’s the other thing. It’s like after I already brutalized you with saying, “No, you can’t have a local file system,” now I take your SSH away as well. [laughing]

IVAN: [laughing] But there is something to use to replace it, right?

TESS: There is. The problem is that, you really, really, really, really, really, really, really shouldn’t use SSH in Kubernetes. SSH is a very dangerous thing to have running anywhere, because it is a potential security access point that can be used and abused, both internally and externally. You really don’t want to have to run it, because if you want to run SSH in Kubernetes, you have to run it in a container. And if you run it in a container, you’re running it as root. And if you’re running it as root, you’re running it as root on the underlying hardware that’s powering the cluster, and that’s bad. [laughing] You don’t want to do that.

So, instead you want to access what is typically called “the backplane.” The backplane is going to be access to the workload via the orchestration system. So, for Kubernetes, the backplane access comes in the form of a command line application called Kubectl or “Kube control” or “Kubey control” or “Kubectl” or like 15 other different names. [laughing] I always thought of Kubectl, that’s my favorite.

IVAN: Let's spell it out. [laughing] I like that one too. k-u-b-e-c-t-l

TESS: And this application not only lets you interact with the orchestrator, but also allows you to directly access individual containers as well. Although getting to an individual container is a little bit more difficult, once you’ve done it a few times, it’s not that hard. Because Kubernetes is so popular, there’s a lot of other command line environments, which will have auto completion assistance for Kubectl as well. So, for me, if I enter in a parameter to Kubectl, say for name space, I can hit tab and it will give me a list of the name spaces that I have. So I don’t actually have to type it out.

IVAN: Pretty slick.

TESS: I use Z Shell (ZSH) but that’s me, I’m weird. Some people like using Fish or some other shell. And I’m sure there’s auto completion mechanisms for your favorite shell somewhere.

IVAN: There’s not a whole lot of challenges then, with Kubernetes. You’ve kind of mentioned a few that are surmountable. Is there anything else, a budding developer, a budding DevOps person should know about, that are looking to start to explore hosting for themselves?

TESS: Well, they should also keep in mind that email is a problem.

IVAN: Yes! We discovered that in the last few weeks, didn’t we?

TESS: Yes, we did.

IVAN: So, we decided that we were going to use an external, transactional email provider. We ended up on SendGrid. But you don’t think of these things once when you’re working on a cluster that’s managed because, hey, these machines all have SendMail on them.

TESS: Yup, and that’s one thing that you really can’t rely on when you start working with a container-based workload. It exposes a lot of these things. But, we’re not where we were two or three years ago where this would’ve been a huge, scary, problem. These things have existing solutions, which are not that difficult to implement, even today.

IVAN: And there are some free tiers as well that you can use, especially if you don’t have a high volume of emails that you’re sending out.

TESS: If you’re only sending 500 emails a day, you can configure your G Suite email as the SMTP provider.

IVAN: Exactly. What about cron? Isn’t that a problem too?

TESS: Cron is a little bit different in Kubernetes. So, the thing with cron is that, in Kubernetes, cron isn’t just something that runs a command. In a traditional server workload, cron is some background process that exists in the system, and when a certain time shows up, it runs a certain command that you tell it to. And, it assumes that you’re running it on literally the same exact system that is running everything else, your web workload. Right?

IVAN: Right.

TESS: That’s not quite the case in Kubernetes. In Kubernetes, a cron job actually runs a container. So, when you actually have your web workload, you’re going to have one container, say, for Apache, somewhere, which is running your site. Then you have a cron job in Kubernetes, and that cron job will literally spin up a completely separate container in order to actually run that process.
So, that’s a bit different.

Now, the only real part of that which gets really confusing is, if you don’t have a nice separation of all of the different infrastructure we just finished talking about, if you don’t have any local disks that you need to worry about, if you don’t have SendMail you have to worry about, if you don’t have any of this stuff and you can scale out your web container to 10 or 20 or more, and not have a problem because they all rely on external API-based providers, then it doesn’t really matter what you do with cron. You just literally run the same container that you run for your web workload, with the same configuration and everything else, but you only tell it run a particular command, instead of "Run Apache." And that’s it. That’s what we do. And, it’s actually not very hard.

IVAN: What’s your favorite thing about Kubernetes? I’m only going to give you five minutes at the most. [laughing]

TESS: [laughing] I think the thing that I like the most about it, is probably the ability to easily scale things. Once you actually have solved all the underlying infrastructure problems, you basically have just a container-based workload that you can say, “I need to run three of these.” Then you can tell it and it will run three of them, and it will just run it, that’s it, you don’t need to worry about it. It already load balances it for you. How can I describe this? Well, let’s go back to the infamous car analogies again.

IVAN: They work.

TESS: They work, but you know they work within a US cultural context of a certain decade period, of a certain geographic location, but let’s put that aside for a second.

So, a car analogy. Let’s say you have a car, and you want to do some work on it. And you go to your garage and what do you see? The car and an empty garage. That’s often what a lot of other systems look like. When you have to do traditional clustering with regular virtual machines, or even self-hosted physical machines, you have to go over to your local hardware store, buy all the tools, buy the car jack, buy an engine lift, buy an air compressor and a whole bunch of other stuff, in order to do your car stuff, and it’s a lot of work and a lot of investment.

With Kubernetes, it’s more like, Okay, I go to my garage and I have Kubernetes. So I have all the tools already. All the tools are just there on the walls, right now. I can just start working. That’s what I really like about Kubernetes. It provides me a room with all the tools for me to actually make this workload do what I want it to do, rather than having to go and grab yet another thing, then another thing, then another thing. Then try to make compromises to make two things, which aren’t the thing that I can’t get right now, but they’re the two I have, to work together.

IVAN: I love the analogy. [laughing] I think that works, Tess. So, what about training? Wouldn’t it be great if, instead of trying to figure this all out yourself (like we did), you could just have us show you how to do it?

TESS: Gee, wouldn’t it? [laughing]

IVAN: Wouldn’t it be great? Well, guess what? That actually exists. We’re going to be doing some free trainings at BadCamp and then at DrupalCorn as well. We’ll be at BadCamp next month, the beginning of October. Now, they’re free trainings, but there is a cost of use to attending the training itself, so I think you have to register and it’s $20, or $10 at DrupalCorn. They’re free as far as we’re concerned.

Can you talk through, just a little bit about the format of the training that we have set up? What are you going to learn and who is it for?

TESS: So, we’ll briefly touch upon different kinds of Kubernetes hosting providers, as well as what Kubernetes actually is and what it does, and what it gives you. Then afterwards, we’re going to start containerizing your particular application. So, we’ll start working with containers, putting them onto Kubernetes, getting used to how to use Kubectl, how to work with individual definitions within Kubernetes, and making all of these pieces work together.

IVAN: And, it’s a four-hour workshop, it’s half a day, you get to spend time with Tess, and I think I’ll be there too. It’s going to be great. So, if you want to contribute to Flight Deck, or to Kubernetes, the Kubernetes Flight Deck Cluster that we have, we’d love it. It’s all online. You can visit ten7.com, and you’ll find it there on the what we give back page and you can also visit us on github.com/ten7, and you’ll see all the repos there. We’d love your help. Thank you, Tess, so much for spending your time with me today. This has been truly great.

TESS: Not a problem.

IVAN: So, if you need help with your own hosting, or figuring out what makes most sense to you, we’d love to be there to help you, whether you’re a developer or a large university, or a small business, it doesn’t matter. We’re happy to provide consulting, whether that means deploying your own Kubernetes or having us do it for you, or even selecting another vendor that makes the most sense to you.

Just send us an email and get in touch. You can reach us at [email protected]. You’ve been listening to the TEN7 Podcast. Find us online at ten7.com/podcast. And if you have a second, do send us a message. We love hearing from you. Our email address is [email protected]. And don’t forget, we’re also doing a survey of our listeners. So, if you’re able to, tell us about what you are and who you are, please take our survey as well at ten7.com/survey. Until next time, this is Ivan Stegic. Thank you for listening.

Sep 18 2019
Sep 18

Your browser does not support the audio element. TEN7-Podcast-Ep-070-Using-Kubernetes-for-Hosting.mp3

Summary

After months deep in the weeds of Kubernetes, our DevOps Engineer Tess Flynn emerged with the best practices for melding Docker, Flight Deck and Kubernetes to create a powerful open source infrastructure for hosting Drupal sites in production (powered by our partner, DigitalOcean). Ivan and Tess take a deep dive into why we chose this combination of tools, our journey to get here, and the nitty gritty of how everything works together.    

Guest

Tess Flynn, TEN7 DevOps Engineer

Highlights

  • Why offer hosting ourselves now?
  • Differences in hosting providers
  • The beauty of containerization, and the challenge of containerization
  • The best container orchestrator
  • What’s with hosting providers and their opaque pricing? (and why we like DigitalOcean)
  • Kubernetes’ highly dynamic environment: updated with just a code push
  • Flight Deck, the genesis of our journey to Kubernetes
  • Docker enables consistent environments
  • Flight Deck + Kubernetes + DigitalOcean
  • You can do this all yourself! (or we can help you with our training)
  • It all runs on Drupal OR other platforms
  • In order to enjoy Drupal + Kubernetes, you must let go of your local file system and SSH, and reevaluate your email system
  • Complex files vs. static files and S3
  • Kubectl! (it sounds cuter when you say it out loud)
  • Cron jobs run differently in Kubernetes
  • A Tess talk isn’t complete without a car analogy: Kubernetes is like a garage that comes pre-stocked with all the tools you’ll need to work on your car

Links

Transcript

IVAN STEGIC: Hey everyone! You’re listening to the TEN7 podcast, where we get together every fortnight, and sometimes more often, to talk about technology, business and the humans in it. I am your host Ivan Stegic. We’ve talked about DevOps at TEN7 on the show before. We’ve done an episode on why we decided to expand our hosting offering to Linode back at the end of 2017. We’ve talked about why we think it’s important to have a good relationship with your hosting company. And, we’ve written about automation and continuous integration over the years as well.

For the last year or so, we’ve been working on our next generation of hosting service, and our DevOps Engineer, Tess Flynn, has been deep in the weeds with Kubernetes. Today, we’re going to spend some time talking about what we’ve done—and how you could be doing it as well—given that we’ve open sourced all of our work.

We’re also rolling out training at BadCamp this year, that’s in October of 2019, and we’ll be at DrupalCorn as well, in November. So, we’ll talk about that and what you might learn by attending. So, joining me again is our very own Tess Flynn. Hello, socketwench.

TESS FLYNN: Hello.

IVAN: Welcome, welcome. I’m so glad you’re on to talk shop with me. I wanted to start with why. Why are we hosting our own sites and those of our clients? There are so many good options out there for WordPress, for Drupal: you've got Acquia and Pantheon, Blue Host, and others. We typically use the provider that makes the most sense, based on our clients’ needs.

We’ve had a close relationship with ipHouse and their managed hosting services for a long time. But why start hosting now? For us, as an organization, it’s kind of been the perfect storm of circumstances, from the technology being mature, to the cost of it, and the availability of it, to where we are as an organization from a developmental point of view, to even being more conscious of vendor lock in and actively trying to avoid it.

So, I want to talk about technology a little bit more with you, Tess. What’s so different now than it was a few years ago? Why is it suddenly okay for us to be hosting ourselves?

TESS: There’s been kind of an explosion over the last few years of managed Kubernetes hosting providers. Now, we’ve had managed hosting providers forever. We’ve had things that are called Infrastructure as a service (IaaS) provider; that’s going to be things like AWS and Google Compute Cloud, as well as other providers, including DigitalOcean, but also say, Linode and other ones, which just provide raw hardware, virtual machine and root login. Lately, however, a lot of people would rather break up their workloads into containers, using something that’s similar to Docker. And I’ve talked about Docker before, but Docker is an alternative take on virtualization technologies, which works on taking applications and putting them in their own individual, virtual environment. I’m glossing over so many things when I say that, but it gets the general point across, with the two minutes before everybody else falls asleep.

IVAN: Right.

TESS: What’s really nifty about putting applications into a container is that now the container doesn’t really care where it is. You can run it on your system, you can run it somewhere else, you can run it on a hosting provider. And, the great thing about these containers is that you can download ones that other people have created. You can modify them, make your own, and you can string them together to build an entire application service out of them. And that’s really, really great. That’s like infrastructure Legos.

But the problem is, once you get the containers, how do you make sure that they’re on the systems, on the actual hardware where they are supposed to be, in the number of copies that there’s supposed to be, and that they can all talk to each other? And the one’s that aren’t supposed to talk to each other, can’t? That’s a lot trickier. For a long time the problem has been that you really only have two solutions: you do it yourself, or you use something like Docker Swarm. I don’t have the greatest opinion of Docker Swarm. I have worked with it before in a production environment, it’s not my favorite.

IVAN: It’s a little tough, isn’t it? We’ve had a client experience on that.

TESS: It’s a little tough, yeah. It’s not really set up for something like a Drupal workload. It’s set up more for a stateless application. A prototypical example is, you need to calculate the progression of matter within the known galaxy, factoring a certain cosmological constant. Take that variable, set it into a compute grid and go, “Hey, tell me what the results are in 15 years.” But you don’t really do that with Drupal. With Drupal, you’re not just going to send off one thing and always get the same thing back. There’s going to be state, which is preserved. That’s going to be in the databases somewhere, and there are going to be files that are uploaded somewhere. And then you have to get load balancing involved, and then it gets really complicated, and it’s like ugh. I really didn’t like how Swarm did any of this stuff. It was very prescriptive. It was, you do it their way, and nothing else.

IVAN: No flexibility.

TESS: No flexibility at all. It was really, really not fun, and it meant that we had to do a lot of modification of how Drupal works, and incur several single points of failure in our infrastructure, in order to make it work in its form. That whole experience just did not get me interested or excited to make a broader Swarm deployment anywhere else.

Then I ran across Kubernetes, and Kubernetes has a very different mentality around it. Kubernetes has more different options for configurations, and you can tailor how Kubernetes manages your workload, rather than tailoring your workload to work with Docker Swarm. That’s why I really liked it. What's really nifty is, once you have Kubernetes, now you have an open source project, which is platform agnostic, which doesn’t care about which individual hosting provider you’re on, as long as you have containers, and you can send configuration to it somehow, it’s fine, it doesn’t care.

A lot of managed hosting providers are going, “Hey, you know, VMs [virtual machines] were kind of nifty, but we really want to get in on all this container stuff now, too.” “Oh, hey, there’s a container orchestrator,” which is what Kubernetes is, and what Docker Swam is, as well, a container “orchestrator” which does all of the making sure the containers are on the right systems, are running, they can talk to the containers they're supposed to, and can’t talk to containers they're not supposed to.

That made a lot of infrastructure providers go, “This is not really a Platform as a service anymore. This is another form of Infrastructure as a service. As such, that is a segment that we can get into."

So, first it started with Google Kubernetes Engine, which is still considered today the defacto version. Amazon got into it, Azure got into it. And all of these are pretty good, but a lot of these huge cloud service providers, you can’t get clear pricing out of them to save your life.

IVAN: Yeah. That’s so frustrating, as a client, as a business owner. How do you do that? It’s insane.

TESS: I mean, the only way that it seems that is deterministic, in order to figure out what your bill is going to be at the end of the month, is to spend the money and hope that it doesn’t kill your credit card. [laughing]

IVAN: Yeah, right, and then try to figure out what you did, and ways of changing it, and then hell, you’re supposed to be just charged that every month from now on, I suppose.

TESS: It’s just a pain. It wasn’t any fun, whatsoever. So, an alternative approach is, you could actually install Kubernetes yourself on an Infrastructure as a service provider with regular VMs.

IVAN: And, we considered that, right?

TESS: Oh, I considered it, and I even spun that up on a weekend myself. It worked. But the problem is, I’m a colossal cheapskate and I didn’t want to spend $30.00 a month for it. [laughing]

IVAN: [laughing] If only there was a supporting ISP that had free Kubernetes support, and just charged you for the compute engines that you used.

TESS: I was really kind of sad that there wasn’t one, until six or eight months ago, when DigitalOcean announced that they have in beta (now it’s in production) a Kubernetes service, where the pricing was incredibly clear. You go to the cluster page, you select the servers that you want to see (the nodes as it were). I know, Drupal nodes, infrastructure nodes, it’s really confusing. Don’t even get physics people involved, it gets really complicated. [laughing]

IVAN: No, please. No, don’t. [laughing]

TESS: But you select which servers that you want to have in your Kubernetes cluster, the sizing, and the price is just listed, right there, in numbers that you can understand! [laughing]

IVAN: Per month, not per minute.

TESS: I know, per month, not per minute.

IVAN: It’s just the small things. Crazy.

TESS: And, it really targeted the kind of market that we are in for a hosting provider, and it made me really excited, and I really wanted to start putting workloads on it, and that’s what started the entire process.

IVAN: It really was, kind of a fortuitous series of events, and the timing kind of just really worked out. I think one of the biggest things for us, for me, is that with Kubernetes, we don’t have to worry about patching and security updates, and monitoring them, and these large hardware machines that we have to keep patched and updated. Essentially, it’s updated every time we do a code push, right? I mean, we’re still concerned with it, but it’s a much easier burden to bear.

TESS: Right. Now what’s going on is that, every time that we do a push, we’re literally rebuilding every system image necessary to run the underlying application. Which means that if we need to push a system update, it’s really just a matter of updating the underlying container's base image to the newest version. We’re already using Alpine Linux as our base containers, which already is a security-focused minimal container set.

IVAN: So, this is actually a good segue to what I wanted to talk about next. A few years back (as opposed to six to nine months back), which is how we kind of got down the road to get to Kubernetes was, I think the origin of all this really is, Flight Deck, and the desire for us to make it easy for developers who work at TEN7—and anyone else who uses Flight Deck, honestly—to have the same development environment locally. Basically, we wanted to avoid using MAMP and WAMP and different configurations so that we could eliminate that from any of the bug-squashing endeavors that we were going into. So, let’s talk about this started with Docker and led into Flight Deck, and what a benefit it is to have the same environment locally as we do in staging and production.

TESS: So, there’s a joking meme that’s been going around, and DevOp cycles, of a clip of a movie where, I think a father and son are sitting and having a very quiet talk on a bench somewhere in a park, where the kid is saying, “But it works on my machine.” And then the Dad hugs him and says, “Well, then we’ll ship your machine.” [laughing] And, that’s kind of what Docker does. But joking aside, I wanted to get that out of the way so I’m not taking myself too seriously. [laughing]

So, one of the problems with a lot of local development environments—and we still have this problem—is that traditionally we’ve used what I consider a hard-installed hosting product. So, we’re using MAMP or WAMP or Acquia Dev Desktop, or if you’re on Linux you’re just installing Apache directly. And all of those work fine, except when you start working on more than one site and more than one client. So, suddenly you have this one problem where, this one client has this really specific php.ini setting, but this other client can’t have that setting. And MAMP and WAMP work around this through a profile mechanism which, underneath the covers is a huge amount of hyperlinking and weird configurations, and spoofing, and like eww, it makes me shutter.

IVAN: Yeah, it makes me cringe just to talk about it, yeah.

TESS: And, the problem is that, every time you have to do this, every developer has to do this themselves, they can’t just standardize on it. So, if somebody has an individual problem on their system, that only happens on their system at 3:45 on a Thursday, after they’ve had chili for lunch or something or other, then you can’t really reproduce it. So, the solution really is, you need to have replicatable, shareable, consistent development environments across your entire team. And that’s what Docker does.

Docker provides that consistency, that shareability, and makes sure that everybody does, in fact, have the same environment across the board. That’s the entire point of that, and that’s where the whole joke about, “Well, then we’ll ship your machine,” [laughing] because that is in essence what containers are. They are system images that run particular bits of software. Now, once we moved everyone to Docker for development, we now had a consistent environment between all of our systems, so that now we didn’t have to work about a number of different problems.

Another good example is, this site uses PHP 5, this site uses PHP 7—a little out of date now, but it was very relevant two years ago—in which case, how do you make sure you’re on the right version? Well, with Docker, you change a text file, and then you boot the containers up, and that’s it.

IVAN: And that text file lives in a code repository, right? So, everybody else gets that change?

TESS: Mm hmm, because you are literally sharing the same environment; you are enforcing a consistent development environment across your entire team for each individual project. And, if you use that strategy, you have something that is flexible, yet at the same time incredibly consistent.

IVAN: And this is really important across all of our developers, and all of our local development that we do, but the challenge then becomes, how do you consistently replicate this in a staging or in a test environment, and even in production? So, that’s kind of the genesis of how we thought Kubernetes could help us here, right?

TESS: Right.

IVAN: So, the challenge to you from me was, how do we make this work in production?

TESS: So, the nice thing about Flight Deck is, it was always designed with the intention of being put into production, But the orchestration component just wasn’t there, and the hosting component wasn’t there. Kubernetes showed up, and that solved the orchestration component, and then, eventually DigitalOcean showed up and now we have the hosting component. So, now, we have all the pieces together to create a consistent environment that is literally the same containers, from the first time someone starts working on the project, to when it gets deployed to production. That is the height of continuous integration ideals, to make sure that you have consistency across all of your environments. That you don’t have different, weird shared environments along the way, that everything is exactly the same so that you know that it will work.

IVAN: I want to stop right there, just so our listeners can appreciate the power of what you just said. You basically said, “I’m going to be working on a website, or a web application locally, with some sort of stack of required server components, whose version numbers and installation profile is configured in a certain way. My teammate is able to replicate that environment exactly, to the version, simply by using the same repo, and by using Flight Deck.

Moreover, all of those version numbers and the stack that is being used, is actually also the same now in staging and, most amazingly to me, in production. So, we can guarantee that what container is functioning in production on the Kubernetes cluster, is actually on staging and on everyone else’s machine. We’ve totally eliminated any variability and any chance that the environment is going to be causing an issue that one person may be seeing that another isn’t.

TESS: That’s correct.

IVAN: That’s pretty amazing!

TESS: It’s a really difficult thing to do, but starting with the containers and building that from the base up actually makes it a lot easier, and I don’t think that any other local development environment, even container based local development environment such as DDEV and Lando are doing this quite yet. Last I heard, I think DDEV was working on a production version of their containers, but it’s not the same containers, whereas with Flight Deck, it literally is the same container.

IVAN: It’s the same configuration. Everything is the same. That’s pretty amazing. I’m still kind of really impressed with all of the stuff that we’ve done, that you’ve done. And, honestly, this is all open source too. This is not like TEN7’s proprietary product, right? We’ve open sourced this, this is all on the web, you can download it yourself, you can figure it out yourself, you can do this as well. You can start your own hosting company.

TESS: That’s correct. The key item which puts all this together is, the Ansible role called Flight Deck Cluster. What Flight Deck Cluster does is, it will create a Flight Deck-flavored Kubernetes cluster and it works perfectly well on DigitalOcean. There’s no reason why it can’t work on say, Google Kubernetes Engine or AWS or anyone else. The architecture that Flight Deck Cluster uses is meant to be simple, durable and transportable, which is something that a lot of other architectures that I’ve seen just don’t have.

IVAN: So, we’ve designed a lightweight set of Docker containers called Flight Deck that you can use locally. We’ve evolved them so that they work with Kubernetes, which you can deploy anywhere in staging and production. We’ve open sourced them. And, the fact that it runs Kubernetes, all you need is a service that supports Kubernetes and you should be able to run all of this in those other locations.

So, we’ve talked about how we started with Docker and how that evolved, and I talked about how we've open sourced it and it’s available to you. I want to spend a little bit of time getting into the details, into the nitty gritty of how you would actually do this for yourself. Is there an app I download? Is it all the YML, all the YML files that we’ve open sourced? What would someone who wants to try this themselves, what would they have to do?

TESS: The first thing that I would probably do is, start running Flight Deck locally. Because you don’t need to pay any extra money for it, you just need to use your local laptop, and it’s also a good experience for you to learn how to interact with Docker by itself. That looks good on a résumé and it’s a good skill to actually have.

I have a talk that I used to give about Docker, and I know that there’s a blog post series that I posted somewhere a long time ago, about how Docker actually works under the covers. Both of those are going to be invaluable to understand how to get Flight Deck working on your local environment, and once you have it working on your local environment, then the next problem is to figure out the build chain. Now the way that our build chain works is, that we have another server, which is a build server, and what the build server does, is it’s going to receive a job from Gitlab and that job is going to take all of the files that constitute the site, it will build them into a local file system, and then it will put those inside of a container which is based on Flight Deck. Then it will upload those to a container registry somewhere else. So that we already have a few additional pieces of technology involved. But the nice thing is, Gitlab is open source, Ansible is open source, and all of our build processes are run through Ansible, and the Docker registry is also open source. It's just a container that you can run somewhere. There’s also services that you can buy that will actually provide you a container registry on a fee basis. All of those are definitely options. Once you have the container in a registry somewhere, then you can run Flight Deck Cluster to build out the rest of the cluster itself.

IVAN: You make it sound so easy. [laughing]

TESS: I make it sound easy, but it’s a lot of code, but it is all open source and it is all there for you to use. Right now, our cluster is based on a development version of Flight Deck, which I’ve been calling Flight Deck 4, and this version is intentionally natively designed for a Kubernetes environment. But it still works perfectly fine under Docker Compose locally, and it is literally the containers that we are using in production right now, at this minute. All of those containers have been thoroughly documented. They have nice readmes which describe exactly how you configure each individual container. And the Flight Deck Cluster role on GitHub also has an extensive readme document which describes how every individual piece is supposed to work.

IVAN: So, the easiest way to get to all that documentation into the repo is to simply go to flight-deck.me. That will redirect you to a blog post about Flight Deck on the ten7.com website, and at the bottom of that post you’ll see links to the GitHub repos and all of the other information that you’ll need to get to that.

So, I wanted to talk about the fact that the hosting itself, the Kubernetes hosting that we have, is optimized for Drupal right now—I kind of struggle to say "optimized for Drupal." It’s just configured for Drupal. There’s no reason that Kubernetes is, and what we’ve released, is locked into Drupal. We are hosting our own React app on there. We have a CodeIgniter app that’s running, we even have a Grav CMS site on it. There’s no reason why you couldn’t host WordPress on it, or ExpressionEngine or any other php, MySQL, Apache, Varnish, Stack on it. Right? There’s nothing innately that forces you to be Drupal on this, right?

TESS: Nope.

IVAN: And that’s also from a design perspective. That was always the intention.

TESS: It’s intended to be run for Drupal sites. However, it always keeps an eye towards being as flexible as possible.

IVAN: So, I think that’s an important thing to mention. Let’s talk about some of the challenges of running Kubernetes in a cluster in production. It’s not like running a server with a local file system, is it?

TESS: [laughing] No, it isn’t.

IVAN: [laughing] Okay. Let’s talk about the opportunities of things to learn.

TESS: The biggest, scariest thing about Kubernetes and Drupal is, you have to let go of your local file system. That is the most scary thing that I have to tell people about Kubernetes.

IVAN: So, no file system, huh?

TESS: No file system.

IVAN: Does that make it slow?

TESS: Well, not really. Let me describe why. The problem is, that— and I’ve had this in my Return of the Clustering talk—is that we’re used to something which is called “block storage.” Now, block storage is pretty great. It is a literal attached disk to the server. So, it is mounted on the server, you have direct access to it, and you can store all kinds of things to it. And it’s fast, and it’s right there. It has no failover, it can’t be shared across the systems, but ehhh, whatever, we have one big server, who cares about that.

Then, if you do try building a traditional server cluster, well, you can’t quite do that. So then you get network file system involved, NFS. And then now, all of the file reads and writes occur over the network to some other centralized server. Okay, it still looks like a local block storage, it still works like block storage, so, okay, sure. But the problem with that is that network file systems, by their base nature, introduce a single point of failure.

Now, that’s not good by itself. If the NFS server goes down, your entire site no longer looks or functions correctly. But the problem is, that it also doesn’t scale either. There’s a natural limitation between the number of different replications for frontend server, servers that intercept the actual requests from people, and then send them to the Drupal backend for processing, and then push back their responses. There’s a natural limitation between those systems and those that can access NFS. And as soon as you have too many accesses, suddenly NFS is not going to be keeping up with you and your performance drops to the floor.

Also, NFS is kind of persnickety. You have to tune it. You have to make sure that it has enough RAM, enough bandwidth. You have to make sure it’s physically proximate to the rest of the servers. And, all of this is because it’s trying to replicate block storage. Now, block storage is great for a whole bunch of data, but in a cloud architect's perspective, there are really two different kinds of data. There’s complex data and static data.

And when I tell people about this, they go, “Well, what’s a complex file?” A lot of people will say, “Well, we have a whole bunch of files which are all linked together, that’s complex, right?” Nope. “Well, we have some Excel documents that’s on an NFS file, that’s complex, right?” Not really. So, what is a complex file? 

I spent hours, tried to squeeze an answer [laughing] out of the internet for this, and eventually arrived at the answer from a cloud architect's perspective: “complex files, such as the files which constitute the actual underlying disk storage for say, a MySQL database.” Data, which is written sparsely and seemingly randomly in multiple locations at multiple times with strict concurrency requirements. Now when I say that, does that sound like anything that we actually upload in a Drupal site?

IVAN: Nope.

TESS: Nope. None of it does. Block storage is required for complex data. But for static data, which is virtually everything that a Drupal site hosts, we don’t need it, it’s too much. It’s way too complicated. And, it doesn’t scale. So, what’s the solution? The solution really is, we need to treat the file system like an API. We need to treat the file system like a database. We don’t care where the database is, as long as you have an IP, a login and the correct credentials to actually get to the database, and then we have multiple readers, multiple writers. That’s what we want for a file system, right? Well, it turns out, there’s a thing that does that already, it’s called S3.

IVAN: Yes, AWS, hello. [laughing]

TESS: And the nice thing about S3 is, it’s perfect for static data. It’s API accessible and it can be made internally redundant. So, it has its own high availability built in that we don’t need to worry about. The nice thing that’s even more than that, is when we say S3, most people go, “Oh, Amazon.” No. S3 is, in fact, a standard. It is not just Amazon’s implementation of S3. There are multiple implementations of S3. So, I usually like saying an S3-compatible hosting provider. And that’s going to include anybody who runs any kind of S3-compatible service. And there’s actually an open source product called Ceph that actually provides an S3 frontend for file storage. And that is actually a service that DigitalOcean also provides. They have DigitalOcean spaces, which provide an S3-compatible static file interface, that’s actually powered by a Ceph cluster underneath the covers. So, open source all the way down to the core.

IVAN: Well, I didn’t know that spaces was Ceph underneath the covers. That’s cool.

TESS: It’s just buried in there. You could find it though.

IVAN: Cool. So, file storage is a challenge, but we fix that by using S3.

TESS: Yep, because Drupal 7 and 8 actually have very good S3 support. There’s S3 FS, that particular module which is excellent for doing Drupal 7 sites. We’ve been using Fly System for Drupal 8 for a few different reasons, but there are reasons that are a little bit easier for us. But your mileage may vary.

IVAN: And, if you’re going to host something that’s not Drupal related, you would need to find some other S3-compatible layer module, right?

TESS: Like for the CodeIgniter application, we are currently looking at implementing that as well.

IVAN: And, there’s a React app as well that we’ve deployed. That uses the underlying Drupal site, though, doesn’t it?

TESS: Yes, that doesn’t actually need a local file system.

IVAN: There’s no SSH access to a cluster of Kubernetes, is there?

TESS: Yes, that’s the other thing. It’s like after I already brutalized you with saying, “No, you can’t have a local file system,” now I take your SSH away as well. [laughing]

IVAN: [laughing] But there is something to use to replace it, right?

TESS: There is. The problem is that, you really, really, really, really, really, really, really shouldn’t use SSH in Kubernetes. SSH is a very dangerous thing to have running anywhere, because it is a potential security access point that can be used and abused, both internally and externally. You really don’t want to have to run it, because if you want to run SSH in Kubernetes, you have to run it in a container. And if you run it in a container, you’re running it as root. And if you’re running it as root, you’re running it as root on the underlying hardware that’s powering the cluster, and that’s bad. [laughing] You don’t want to do that.

So, instead you want to access what is typically called “the backplane.” The backplane is going to be access to the workload via the orchestration system. So, for Kubernetes, the backplane access comes in the form of a command line application called Kubectl or “Kube control” or “Kubey control” or “Kubectl” or like 15 other different names. [laughing] I always thought of Kubectl, that’s my favorite.

IVAN: Let's spell it out. [laughing] I like that one too. k-u-b-e-c-t-l

TESS: And this application not only lets you interact with the orchestrator, but also allows you to directly access individual containers as well. Although getting to an individual container is a little bit more difficult, once you’ve done it a few times, it’s not that hard. Because Kubernetes is so popular, there’s a lot of other command line environments, which will have auto completion assistance for Kubectl as well. So, for me, if I enter in a parameter to Kubectl, say for name space, I can hit tab and it will give me a list of the name spaces that I have. So I don’t actually have to type it out.

IVAN: Pretty slick.

TESS: I use Z Shell (ZSH) but that’s me, I’m weird. Some people like using Fish or some other shell. And I’m sure there’s auto completion mechanisms for your favorite shell somewhere.

IVAN: There’s not a whole lot of challenges then, with Kubernetes. You’ve kind of mentioned a few that are surmountable. Is there anything else, a budding developer, a budding DevOps person should know about, that are looking to start to explore hosting for themselves?

TESS: Well, they should also keep in mind that email is a problem.

IVAN: Yes! We discovered that in the last few weeks, didn’t we?

TESS: Yes, we did.

IVAN: So, we decided that we were going to use an external, transactional email provider. We ended up on SendGrid. But you don’t think of these things once when you’re working on a cluster that’s managed because, hey, these machines all have SendMail on them.

TESS: Yup, and that’s one thing that you really can’t rely on when you start working with a container-based workload. It exposes a lot of these things. But, we’re not where we were two or three years ago where this would’ve been a huge, scary, problem. These things have existing solutions, which are not that difficult to implement, even today.

IVAN: And there are some free tiers as well that you can use, especially if you don’t have a high volume of emails that you’re sending out.

TESS: If you’re only sending 500 emails a day, you can configure your G Suite email as the SMTP provider.

IVAN: Exactly. What about cron? Isn’t that a problem too?

TESS: Cron is a little bit different in Kubernetes. So, the thing with cron is that, in Kubernetes, cron isn’t just something that runs a command. In a traditional server workload, cron is some background process that exists in the system, and when a certain time shows up, it runs a certain command that you tell it to. And, it assumes that you’re running it on literally the same exact system that is running everything else, your web workload. Right?

IVAN: Right.

TESS: That’s not quite the case in Kubernetes. In Kubernetes, a cron job actually runs a container. So, when you actually have your web workload, you’re going to have one container, say, for Apache, somewhere, which is running your site. Then you have a cron job in Kubernetes, and that cron job will literally spin up a completely separate container in order to actually run that process.
So, that’s a bit different.

Now, the only real part of that which gets really confusing is, if you don’t have a nice separation of all of the different infrastructure we just finished talking about, if you don’t have any local disks that you need to worry about, if you don’t have SendMail you have to worry about, if you don’t have any of this stuff and you can scale out your web container to 10 or 20 or more, and not have a problem because they all rely on external API-based providers, then it doesn’t really matter what you do with cron. You just literally run the same container that you run for your web workload, with the same configuration and everything else, but you only tell it run a particular command, instead of "Run Apache." And that’s it. That’s what we do. And, it’s actually not very hard.

IVAN: What’s your favorite thing about Kubernetes? I’m only going to give you five minutes at the most. [laughing]

TESS: [laughing] I think the thing that I like the most about it, is probably the ability to easily scale things. Once you actually have solved all the underlying infrastructure problems, you basically have just a container-based workload that you can say, “I need to run three of these.” Then you can tell it and it will run three of them, and it will just run it, that’s it, you don’t need to worry about it. It already load balances it for you. How can I describe this? Well, let’s go back to the infamous car analogies again.

IVAN: They work.

TESS: They work, but you know they work within a US cultural context of a certain decade period, of a certain geographic location, but let’s put that aside for a second.

So, a car analogy. Let’s say you have a car, and you want to do some work on it. And you go to your garage and what do you see? The car and an empty garage. That’s often what a lot of other systems look like. When you have to do traditional clustering with regular virtual machines, or even self-hosted physical machines, you have to go over to your local hardware store, buy all the tools, buy the car jack, buy an engine lift, buy an air compressor and a whole bunch of other stuff, in order to do your car stuff, and it’s a lot of work and a lot of investment.

With Kubernetes, it’s more like, Okay, I go to my garage and I have Kubernetes. So I have all the tools already. All the tools are just there on the walls, right now. I can just start working. That’s what I really like about Kubernetes. It provides me a room with all the tools for me to actually make this workload do what I want it to do, rather than having to go and grab yet another thing, then another thing, then another thing. Then try to make compromises to make two things, which aren’t the thing that I can’t get right now, but they’re the two I have, to work together.

IVAN: I love the analogy. [laughing] I think that works, Tess. So, what about training? Wouldn’t it be great if, instead of trying to figure this all out yourself (like we did), you could just have us show you how to do it?

TESS: Gee, wouldn’t it? [laughing]

IVAN: Wouldn’t it be great? Well, guess what? That actually exists. We’re going to be doing some free trainings at BadCamp and then at DrupalCorn as well. We’ll be at BadCamp next month, the beginning of October. Now, they’re free trainings, but there is a cost of use to attending the training itself, so I think you have to register and it’s $20, or $10 at DrupalCorn. They’re free as far as we’re concerned.

Can you talk through, just a little bit about the format of the training that we have set up? What are you going to learn and who is it for?

TESS: So, we’ll briefly touch upon different kinds of Kubernetes hosting providers, as well as what Kubernetes actually is and what it does, and what it gives you. Then afterwards, we’re going to start containerizing your particular application. So, we’ll start working with containers, putting them onto Kubernetes, getting used to how to use Kubectl, how to work with individual definitions within Kubernetes, and making all of these pieces work together.

IVAN: And, it’s a four-hour workshop, it’s half a day, you get to spend time with Tess, and I think I’ll be there too. It’s going to be great. So, if you want to contribute to Flight Deck, or to Kubernetes, the Kubernetes Flight Deck Cluster that we have, we’d love it. It’s all online. You can visit ten7.com, and you’ll find it there on the what we give back page and you can also visit us on github.com/ten7, and you’ll see all the repos there. We’d love your help. Thank you, Tess, so much for spending your time with me today. This has been truly great.

TESS: Not a problem.

IVAN: So, if you need help with your own hosting, or figuring out what makes most sense to you, we’d love to be there to help you, whether you’re a developer or a large university, or a small business, it doesn’t matter. We’re happy to provide consulting, whether that means deploying your own Kubernetes or having us do it for you, or even selecting another vendor that makes the most sense to you.

Just send us an email and get in touch. You can reach us at [email protected]. You’ve been listening to the TEN7 Podcast. Find us online at ten7.com/podcast. And if you have a second, do send us a message. We love hearing from you. Our email address is [email protected]. And don’t forget, we’re also doing a survey of our listeners. So, if you’re able to, tell us about what you are and who you are, please take our survey as well at ten7.com/survey. Until next time, this is Ivan Stegic. Thank you for listening.

Sep 18 2019
Sep 18

So, you’re thinking about starting an international web portal? Or maybe you have a website that is targeted for more than one language? You might have a B2B application and want to expand your market to other parts of the world? Then you should read on ...

Most of you probably think that in order to launch your product, site or service world-wide, all you need is to translate it. Guess again. That’s not enough. 

The world is represented by many different countries and by extension by many different cultures. Each culture has their own “niche” habits, behaviors and even perspectives on things. The same sentence might appeal to someone while offending someone else.

Even the structure of the content can lead to bad conversion rates if it’s not tailored to the target audience. This is where localization comes into play. 

As the name implies, localization means to make something feel local. Something that connects with the audience you are targeting. This means that you need to get your hands dirty and do the research. 

For example, if you want to expand your product to China, make sure to study its culture and their habits. How do most  Chinese sites structure their content? What are the best practices for user experience? How does the navigation look? How big are the images? How do they read the text? Those are just a few questions that you need to answer. 

After you have most (if not all) of the answers, you need to start implementing the solutions. This means that you often need to drastically change the layout and the content of the site. Even changing an image on a blog post can have a positive effect on its performance. 

A great example of good localization is the MSN website. The screenshots below demonstrate the English and the Chinese website. Notice the difference?

English version of the MSN website

 

Chinese version of the MSN website

If you take the time and visit both msn.com and msn.cn you will see quite a difference in both the layout and the content itself. In comparison, we can deduce that the regular website favors imagery over text, and the opposite applies to the Chinese website. And this is only the homepage we’re talking about!

Another good example is Starbucks' website. Below you can see the comparison of Starbucks.com and the Japanese version. 

English version of the Starbucks website

 

Japanese version of the Starbucks website

As you can see again, the pages are vastly different. The Japanese website is packed with a lot more information compared to the regular website. Again the trend of large images over text is clearly visible. 

Localization by itself is a huge topic and we won’t cover all of its aspects in this post, but I want to briefly talk about one website feature that doesn't need localization, as it is seen as a best practice in any culture - namely, good website performance. 

Many of you might live in a part of the world where you get quite a decent internet connection. I like to think of internet speed like water. There are places in the world where there are large bodies of water with fast streams, but there are also places where water is scarce. The same applies to internet speed. 

This means that we need to make sure that our websites run as fast and are as optimized as they possibly can be. Not everyone can afford the luxury of fast internet access and if the page loads slowly you’re likely to lose a potential new client or user. Humans are not patient beings that are willing to wait for your page to load. 

One of the things that impact the performance of a website are images. There are a lot of handy ways to optimize images in order to achieve faster-loading websites. If your site is built on the Drupal CMS, however, you don’t even need to do any extra coding - all the image optimization features are available right in the core

If you want to learn about more ways of improving the performance of your Drupal website, Ana has you covered with her tips to speed up your Drupal site

This brings this post to a close, but just to recap: 

  • Translations are not enough.
  • Make sure to study your target audience and their habits.
  • Customize the structure and the content of the website.
  • Make sure to optimize your website for slow internet connections. 
  • Don’t be afraid to drastically customize the layout of the website.
  • Small changes can go a long way.
Sep 18 2019
Sep 18

These are interesting times for securing tech talent.

With the current unemployment rate at 3.7 percent, the job market is highly competitive, and that’s particularly true for the technology sector. 

Reaching out to agencies that can offer the right talent when and where it is needed is proving to be the solution among savvy organizations. 

Key among the advantages that the right agency relationship offers: the opportunity to leverage specific expertise on an ongoing, ad-hoc, or one-off basis.

In our rapidly changing market, here are six reasons why joining forces with an agency might be a better idea than hiring or relying on in-house talent:

  1. Hiring processes are costlier and more complicated than ever before. 
  2. Employees are an investment and represent overhead in benefits, office space, and training.
  3. Ensuring that an in-house staff consists of employees who have the depth and breadth of skills required to meet current and future needs may not be feasible for many organizations. 
  4. Top talent knows their worth. Along with their increasingly high salary demands, they tend to continue looking for new and better opportunities.
  5. If the hiring decision doesn’t work out, there are costs associated with parting ways along with the risk of legal liabilities.
  6. The market is moving forward at a rapid pace with constantly emerging needs for new types of specialization. The expectation that a few great hires or one exceptional, in-house team, can anticipate and proactively take on every new challenge impedes both agility and opportunity. 

 

Agility Rules

How to respond to these challenges in an environment where not keeping pace with what’s new and next is not an option? Strategic agency relationships. 

Reaching out to firms with targeted expertise that specialize in focused training, rapid-fire development, exceptional design, astute marketing, or incisive consultation on a depth and breadth of topics is proving to be the optimal path forward.  

While there is sometimes a tendency to view relationships with contractors as “all business” and lacking the connections that easily develop among in-house teams, I’ve often experienced a high degree of commitment and connectedness within agency and client relationships that are built upon a personal stake in clients’ success. Bridging this divide calls for an intentional focus, and can be facilitated by a workshop that’s designed to provide a deep understanding of a client's business and knowledge transfer to the agency partner.

There is no question that relationships optimize outcomes. Trust, genuine commitment, and true connections serve to drive game-changing synergies. In many respects, I’ve found that the quality of the relationships can make the difference between a transactional, task-focused approach, and a strategic, long-term vision.

And who can argue that work is considerably more fulfilling and fun when we’re connected with each other both professionally and personally

Looking to join forces with an agency that offers industry-leading expertise, and launch a relationship that can ignite new possibilities in web development, human-centered design, strategic planning, and accessibility remediation? Contact us today

Sep 18 2019
Sep 18

With every new release, Drupal is emerging a valuable asset in the enterprise marketing stack. The additions to Drupal core, especially with Drupal 8 and after, have made it a digital platform that comes equipped for all the standard marketing best practices right out of the gate. In addition to that, the larger Acquia ecosystem is also delivering new solutions that empower Drupal be more than just a CMS. These bring in some powerful martech capabilities that have made Drupal into a platform that’s ready to deliver the results that enterprise marketing teams want.

This post delves into the key modules and solutions that enable smart content management in Drupal, both in terms of creating and publishing content, as well as leveraging that content in diverse ways to drive results.

Smart Content

Smart Content is a Drupal toolset that can help deliver anonymous website personalization in real time, for Drupal 8 sites. Essentially, site admins get the ability to display different content to site visitors, based on whether they are authenticated or anonymous users.

Some examples of how you can leverage it include:

  • Displaying a smart block showcasing your latest offer or most popular blog to a first time visitor to the site
  • Displaying a smart block that showcases different industry specific case studies for different users in your database
  • Displaying a smart block only for mobile viewers of your site, maybe asking them to view it on your mobile app

Now this module in itself has limited functionality, but becomes very useful when used in combination two other Drupal modules:

Smart Content Blocks

Included within the Smart Content module, these allow you to insert a Smart Block on any page and set up conditions that govern the content being displayed within the block. These conditions can be used to hide or show a specific content in certain cases, and form the basic personalization rules for your Drupal site. The blocks have an easy interface within the Drupal 8 backend, giving you the flexibility to add any number of blocks, anywhere on a page. 

It's important to note that all of your content, irrespective of format, is available to show and promote through Smart Content Blocks. Ebooks, videos, images, blogs, service pages—anything that’s already in the Drupal CMS can be delivered to a block. 

Smart Content Segments

A complete set of conditions grouped together to achieve a reusable composite condition. For example, a set of the following three conditions:

  • showcase only on desktop
  • showcase if location is France
  • showcase for anonymous users

               will create a smart content segment that can be applied to any smart content block to ensure                 that it's displayed to anonymous users from France, viewing the site on a desktop. This                             feature saves you time as you don' have to set up the same set of individual conditions every                   time.

At the heart of Smart Content are the conditions, editable rules that allow you to govern the display of content. The interface is easy to manage, and familiar to marketers working on a Drupal site. 

sc-1

You have your choice of the basic conditions for personalization like the browser, language, device type etc. You also have the more advanced options like targeting different industries based on third party IP lookups, or tapping into existing segmentations or campaigns from a marketing automation system. Essentially, anything that has an API with available data can be used as conditions to help drive your personalization strategy with Smart Content.

Layout Builder

The Layout Builder module, experimental in Drupal 8.5 and 8.6, had a stable release with Drupal 8.7. This module allows content authors to easily build and change page layouts and configure the presentation of individual content, content types, media and nodes. It also allows you to add user data, views, fields and menus. 

This is a huge asset for enterprise marketing and digital experience teams because:

  • The module gives a drag-and-drop interface to create custom layouts for specific websites sections and pages, with the ability to override templates for individual landing pages when required
  • Content authors can seamlessly embed video across the site to create a more interactive user experience, and increase engagement and conversions
  • Marketers can now build and preview new pages at their own pace, without the fear of negatively affecting the existing user experience.

All of this means that marketing teams now have more control over the site, and can make changes and additions independently. This also reduces the turn-around-time for new campaigns by reducing, or even eliminating, dependencies on development teams. Think high-impact landing pages designed exactly as you want, but without the waiting around or constant back-and-forth with developers.

Media Library

With the release of Drupal 8.7, the CMS now has a stable media library module.

It provides a visually appealing interface for browsing through all the media items in your site. With the new version, multimedia properties can be added to content either by selecting from existing media or by uploading new media through bulk upload support. Once uploaded, users can remove or reorder any images ready for import. 

It provides an easy way to upload several media assets in your Drupal website quickly. Let’s you add alt-text, check the images before uploading.

Powered by Views, it allows site builders to customize the display, sorting, and filtering options.

Acquia Lightning

As enterprise marketing teams launch large scale campaigns, they often need to put together new microsites that work flawlessly. And they usually have to do it at a short notice, to leverage critical marketing opportunities in time. 

Having to depend upon the development teams to create one from scratch, and the constant coordination required to make that happen, can lead to the marketing team losing precious time. 

Acquia Lightning, an open source Drupal 8 distribution, is the perfect solution for this challenge. Lightning give you a basic ready-to-launch site with pre-selected modules and configurations that can cut development time by 30%. This allows:

  • Development teams to publish optimized Drupal 8 sites in short time frames
  • Editorial teams can easily work with layout. Media and content on these sites, and have them campaign-ready in no time

Some of the key features in Lightning that are particular great for marketers are:

Moderation Dashboard

This dashborad gives you  complete visibility into your Drupal content status, with a structured overview of where every pieces of content is in the editorial process. Besides tracking content status, you can also manage access controls determinig who can access which pieces of content at the backend.

Screenshot 2019-10-01 at 6.50.09 AM

The key pieces of information you can view of the dashboard are:

  • Current drafts in progress
  • Content you created
  • Content needing review
  • Recent site activity
  • Individual editor activity in the last 30 days

Moderation Sidebar

Screenshot 2019-10-01 at 7.03.38 AM

The moderation sidebar allows you to stay on the website frontend as much as possible while making edits and managing the editorial process for any piece of content. Actions like editing text and layout, publishing a piece, creating new draft and more can be easily achieved with the sidebar. And it's quickly accessible by clicking "New Tasks" on any piece of content. For marketers no really keen on getting into the backend, this sidebar is a simple way to make the edits they need, with minimal chances of error. 

Scheduled Publishing

As the name suggests, this feature in Acquia Lightning allows you to set a piece to publish at a future date. This functionality give you a better view of when content is set to launch, and also ensure that it launches at optimal times, according to reader preferences. And this happens without you having to be on the job at odd hours, just waiting around to publish content.

Screenshot 2019-10-01 at 7.17.14 AM

You can schedule publish times from on individual pieces by editing the 'Current Status' to select “Schedule a Status Change” . Then choose “Published” and select your preferred publishing date and time.

Acquia Lift

We cannot talk of smart content management with Drupal without talking about Acquia Lift. For enterprise sites built on Drupal, there’s nothing more suitable for the personalization than Acquia Lift.

Acquia Lift is a solution designed to bring in-context, personalized experiences to life. It’s a powerful blend of data collection, content distribution, and personalization that enables enterprise marketing teams to closely tailor the user experience on the site. And all this without excessive dependence on development or IT teams.

Acquia Lift gives enterprises three key elements to drive their personalization and reflect it with their website content:

Profile Manager

This helps build a holistic 360 degree profile of your users, right from when they are anonymous visitors on the site, up until the stage where they are repeat customers. It collects user demographic data, historical behaviour data, and real-time interactions so you can get a complete understanding of who your users are, what they want, and then work on how best to deliver that.

Content Hub

The Content Hub is a cloud-based, secure content syndication, discovery and distribution tool. Any piece of content created within the enterprise can be aggregated and stored here, ready to be pushed out to any channel, in any format. 

Faceted search and automatic updates give visibility into the entire gamut of content being created within the enterprise - in different departments, across websites, and on different platforms.

Experience Builder

This is the heart of Acquia Lift - the element that allows you to actually build out a personalized experience from scratch. The Experience Builder is a completely drag-and-drop tool that lets you customize segments of your website to showcase different content to different target segments, based on data pulled from the Profile Manager.

Enterprise marketing teams can 

  • set up rules that define what content should be shown to which segment of site visitors
  • perform A/B tests to accurately determine what type of content drives more conversions for which user segments. 

All this can be done with simple overlays atop the existing website segments, without impacting the base site, and without depending on IT teams for implementation.

With a commitment to creating ambitious digital experiences, every new Drupal release has brought in new features to add to the marketing ecosystem. While the overarching focus is on being flexible and scalable, these solutions are creating real impact on customer experience, conversions, online sales and brand proliferation.

And for enterprise teams contemplating shifting to Drupal for diverse proprietary CMSes, the payoff from empowered marketing team alone makes it worth the effort.

While most of the features mentioned here can be accessed by your teams easily if they are already using Drupal, some require guidance. Especially Acquia Lightining and Acquia Lift will need skilled teams to set it up for you, before marketers can start reaping the benefits. 

If you are looking to deploy Lift or Lightning, just drop us a line and our Drupal experts will be in touch.

Sep 17 2019
Sep 17

DrupalCon Amsterdam is approaching — next month! Are you still deciding about going? With the robust program, each day will be full of exciting sessions, social activities and contribution sprints galore! Check out the breadth and scope of our offerings, with insight you won’t want to miss.

The event kicks off with Trainings on the morning of Monday, October 28. Choose from these seven options to further your learning:

Sep 17 2019
Sep 17

If your website is on the right CMS, it becomes easy to create marketing campaigns, drive leads, and tell your brand’s story to the world. However, making content available on every new device in the market accessible to a user becomes a challenge for marketers.

Headless Drupal may sound exactly what a marketer needs - a platform that helps content reach any device a user uses. Yet, there are some significant problems that it poses to the marketer. Let’s understand them in detail.

Revisiting Headless Drupal

A traditional Drupal has a back-end (stores the content) and front-end (which decides the delivery of that content). Now as there is no limit to devices accessible to users, brands need to go beyond just delivering content on websites and web apps.

With a pure headless CMS, tightly coupled front-end is removed, and it delivers content through an API anywhere and on any device (commonly referred to as API-first).

Headless Drupal offers faster functioning than traditional Drupal and offers highly responsive and fast websites ensuring rich user experience.

When the user interface is decoupled from the CMS, the logic for displaying content on each device is on the front-end and its native tools are responsible for controlling the user experience.

How Headless Benefits Marketers?

It is important for marketers to be where their customers are and send the right communication, on the right channel, at the right time. Here are the 3 benefits of headless Drupal to marketers:

1. Platform Independent Communication

Headless Drupal CMS offers great flexibility to marketers as they can deliver one piece of content in multiple formats – to a desktop, smartphone, app, VR devices, smart speakers, and smart appliances. It saves marketers a lot of time previously spent creating and optimizing content for different devices.

2. Freedom on Content Display

Marketers prefer to use headless as it offers choice over how your content appears on the frontend, with extra security over traditional Drupal. JavaScript frameworks has gained more traction due to the demand for more flexibility in the front end. Its emphasis on client-side rendering offers a more engaging and dynamic user experience.

3. The Faster, The Better

Decoupled Drupal is also faster as the logic for displaying the content is decided by the front-end interface. As marketers are in a constant urge to impress the existing customers and at the same time attract new ones, a faster site helps them in engaging with customers as fast as possible.

Why it is Not Marketers’ First Choice?

Though headless Drupal has been beneficial for developers, but is it valuable to marketers as well? Below are the reasons why marketers, despite its advantages, don’t prefer to go for headless Drupal.

1. No Preview Available

With no presentation layer in a headless Drupal, marketers are not able to create and edit content with a WYSIWYG editor as they would with the traditional Drupal. The most challenging part is they can’t preview their content before publishing to their audience.

2. Dependency on Developers

With headless Drupal, development teams can create a custom-built front-end to customize the layout and entire design of individual pages.

The marketers will have to be fully dependent on developers to carry out tasks for conversion optimization purposes, which proves to be an inefficient solution for them.

3. Marketers Have to Manage Fragmented Environment

Today’s marketers have to engage with their audience in real-time, publish content in line with the latest trends, launch landing pages, deploy microsites, track progress, monitor data, collaborate with advertising campaigns, and much more.

A headless Drupal makes the marketers manage content workflows, form building, and microsite deployments. Managing everything at such a huge scale, soon creates an expensive and hard to manage ecosystem. Not only it complicates the life of a marketer, it also gets in the way of creating a seamless and connected customer experience.

4. Impacts the SEO

Marketers lose standard SEO functionality on adopting headless Drupal for their content strategy and will eventually have to invest additional time and cost for Drupal SEO development.

What It Means For Marketers?

Marketers can consider going for decoupling Drupal when they want to publish the content on more than one platform such as multiple websites, various front-end devices or when they need real-time updates of a site where performance would be killed by using traditional Drupal.

However, if their requirement is to manage a responsive website, headless Drupal won’t be beneficial and will slow down time to market. And, also the costs involved are too high.

Solution For Marketers - Progressive Decoupling

Decoupled Drupal loosely separates the back-end from the front-end, creating an architecture which serves perfectly to both developers and marketers simultaneously.

As a marketer, you can benefit by its user-friendliness and the API-driven omnichannel delivery capabilities. The content layer separated from the presentation layer allows marketers to have an authoring experience that feels familiar. The presentation layer above the API layer allows for seamless integration and blending of different tools and technologies.

So to conclude, headless Drupal isn’t for everyone, and in many cases sticking with a traditional CMS or choosing decoupled Drupal is the best option.

If considering decoupled Drupal strategy seems intimidating, Srijan can help you connect with the experts to help drive your marketing strategy with it. Contact us to get the best out of Drupal.

Sep 17 2019
Sep 17
local_library

September 17, 2019

Jacob

This is the last in a series of blog posts about creating an app with React Native. To get started with React Native, read the Basics of React Native: Part 1. If you want to include content from a Drupal website in the app, read Integrating Content with React Native: Part 2.

It’s almost impossible to build a professional app without including third party or custom libraries. Using libraries where and when they’re needed upholds the tenant of reusability that React Native sets forth. It saves time and money by reusing code that already exists. There are only three steps to include a library in a React Native project:

  1. Install the library with npm (node package manager)
  2. Use ES6 (ECMAScript 6)  import statements to import whatever components are needed from the library.
  3. Implement the components within your component or screen.

Here is an example of how to include a library like GooglePlacesAutocomplete in a React Native app. Install the library using:

$ npm install react-native-google-places-autocomplete

Import components from the library wherever they are needed using:

$ import {GooglePlacesAutocomplete} react-native-google-places-autocomplete

Finally, implement the component within the file by calling on it like any other component:

<GooglePlacesAutocomplete placeholder='Enter Location' minLength={2} autoFocus={false} returnKeyType={'default'} fetchDetails={true} currentLocation={false} />

Leveraging your own components is even easier than leveraging components from libraries. In order to include a custom component, use ES6 import statements to import your components, and implement your components within your component or screen. A common place to store these custom-built components is within a project’s components directory. To use a custom component like CustomComponent.js inside a screen include: import CustomComponent from '../components/CustomComponent' at the top of the file, and <CustomComponent /> when you want to use the component.

Libraries enforce code reusability standards because they encourage writing a piece of code only once and implementing it anywhere else it is needed. Your project will be more organized and easier to expand upon later because this method enforces proper library use. It also saves you from wasted time and headaches down the line because your libraries won’t look like spaghetti when you come back to them months later.

We hope you enjoyed this series about creating an app with React Native. Comment below to share what you found useful or more tips that we missed. For updates on new blog posts, our work, and more, like and follow us on Twitter, Facebook, LinkedIn, and Instagram.

Sep 17 2019
Sep 17

“Should I upgrade or should I just wait?” This question has been constantly bothering business decision-makers when it comes to migrating their website from Drupal 7 to Drupal 8. Change can be hard and terrifying, especially at its inception. Yet, a change is what allows you to grow, evolve and progress. It can get painful to take a decision as big as a migration of your Drupal 7 (or 6) website – the one that you knew and have loved. But soon you will know you have made the most brilliant business decision, ever! 

Drupal 8 Migration - A Long-term vision

There has always been a perception that Drupal is a difficult CMS to get a hang of. Starting from end-users to developers, Drupal was considered as having a huge learning curve.  Yes, with the previous major versions (before Drupal 8), the process of upgrading and adjusting to the change was harder. It was also more expensive (needed more resource time), the release of contributed modules (and necessary features) were slower and the release cycles got longer.

But with Drupal 8, everything changed. 

Tom Wentworth, (SVP Product Marketing from Acquia), summed up accurately in his article that unlike a few other CMS’s, “Drupal 8 was a teardown all the way to the foundation”. Creating an upgrade based on the same old foundation would have been a much easier task for the Drupal community. But starting from Drupal 8, the Drupal community has focused on long-term sustainability and on getting people to adopt Drupal effortlessly. This called for a complete re-architecture of Drupal 8 with the adoption of Symphony for high performance, Twig for a more modern templating engine, object oriented programming for easier maintenance, modern user experience design creators and editors for rich content editing, and much more. 

Drupal 8’s continuous innovation approach propels an evolution with regular (and shorter) minor releases, semantic versioning (in a ‘major.minor.patch’ format) that helps in backwards-compatibility enhancements and faster stability in modules by releasing experimental modules in core.

Your Drupal 8 Questions, Answered. (To some extent)

Although it has been a while since Drupal 8 has been around and stable, we still get asked a ton of questions by our customers before a migration. 

1. Why should I upgrade to Drupal 8 (from Drupal 7) when Drupal 9 is just around the corner? (We get this nearly every time)

I have a whole blog dedicated to this question, but if you insist, here are your benefits of upgrading to Drupal 8 now -

  • Time crunch – So Drupal 9 does not release until June 2020 and Drupal 7 reaches its end-of-life by December 2021. Which means you have only a year and a half to upgrade to Drupal 9. If your website is considerably simple and needs less customizations, this is a viable option. Else, you’d better off start an upgrade to Drupal 8 now and migrating from Drupal 9 from Drupal 8 is as easy as upgrading to a next minor release.
  • Living with a FOMO – That’s a term I recently learnt about – Fear Of Missing Out. Why do you want to miss out on some powerful and modern enhancements when you can migrate to Drupal 8 now and boost your Drupal website’s performance and experience? Upgrading from Drupal 8 to Drupal 9 is a cakewalk anyway!
  • Just a better version – Drupal 9 is just Drupal 8 minus the deprecated code and modules. Migrate to Drupal 8 now, enjoy a better performing website and an easy upgrade to Drupal 9 (and any future versions of Drupal)

2. We’re still stuck on Drupal 6. Help!

If you’re still stuck on to Drupal 6, you must know that everyone else has moved on. Today, the web has changed and so has Drupal. The Drupal community no longer supports Drupal 6 since February 2016. Which means, there will be no new Drupal modules or features to look forward to, no more bug-fixes, security updates and patches. Thus putting your website’s security at high-risk and of course depriving it of some TLC from the community. If you still want the best for your website, migrate to Drupal 8 now! Yes, you can completely skip Drupal 7. Drupal Migrate module is now included in Drupal 8 core and makes the switch easy and fast.

3. What performance upgrades does Drupal 8 offer?

Drupal 8 comes packed with performance-enhancing features and modules that can turn your website into a speedy and high performing one. Here are a few to name -

  • The Symfony Framework – Drupal 8’s adoption of Symfony framework isn’t just a great move for developers but for website owners as well. Symfony offers a robust, flexible and high-performance framework that allows for easy scalability of a website.
  • BigPipe Caching - It lets you segregate your page into different sections (called Pagelets) which can be rendered as they get available (Cached first). This lets you drastically improve your page’s perceived performance and speed.

migration-d8

  • Caching for Authenticated Users – Drupal 8 uses Varnish to cache page content and serve them to authenticated users for better and faster performance.
  • PHP7 support – Did you know PHP 7 is now two times faster than PHP 5.6 because of its new Zend engine? With PHP 7 support in Drupal 8, your websites can see a performance rise up to about 110% and reduced memory usage.

4. What challenges will we encounter during a Drupal 8 migration? What can be done to alleviate those issues?

Challenges encountered during a website migration from say Drupal 7 to Drupal 8, completely depends upon the complexity of a website, if it includes a redesign, the amount of content needed to be migrated and many more other factors. The first and the most crucial step towards a Drupal 8 migration is to audit your existing website. Auditing and analysing your website could be the biggest challenge if not handled well and could lead to a successful (and quick) migration when done right. If not planned well, you might run into problems where you are unprepared to handle -

  • Module compatibility issues
  • Might migrate old and unused modules that will increase migration time
  • Unavailability of existing modules/features/themes/views/entities (in core or contrib) 
  • The need to rebuild and rewrite custom modules in Drupal 8. (Coz as discussed earlier, D8 has restructured itself to be able to be more future-ready)
  • A rebuild/repackage of features and views
  • A redevelopment of the theme – because of Drupal 8’s new and powerful templating engine Twig

How do we fix this? – Easy. Audit your website well. Get a Drupal technology partner to do a complete analysis and audit of your existing website and list out features, modules and other elements that need to be migrated. They will need to provide you with details on what needs a rebuild and what can just be easily ported. You can also use evaluation modules like the Upgrade checker which will give you a comprehensive list of migration components and an estimate of how long it might take.

5. Can we migrate to Drupal 8 and yet preserve our existing data whilst remaining GDPR compliant?

Absolutely! The reason why Drupal is so successful is because of its proactive and battle-ready Drupal community. The Drupal GDPR Compliance team project aims at providing websites with modules and features that can assist in making them GGDPR compliant. There are over 15 new modules in Drupal 8 for GDPR compliance to choose from with some modules that can be ported to Drupal 8 and some that may need a rewrite. Check here for a list of Drupal modules that help you build GDPR compliant websites.

6. What happens to my content? 

Drupal understands how important content is to every organisation. With efforts from more than 500 contributors, the release of Drupal 8.5.0 brought together a stable and robust Drupal Migrate architecture. Modules like Migrate API, Drupal Migrate module and Migrate Drupal UI allows for a flexible and easy content migration from the database or sources like JSON, CSV or XML.

7. If we migrate to Drupal 8, will it break any of our existing features/modules?

The answer to this question depends on your website structure, complexity and the way Drupal 7 (or Drupal 6) was implemented on your website. Many times, there is no direct path for a Drupal 8 upgrade. Custom modules will need a rebuild and will break if simply ported because Drupal 8 is now built on Symfony framework (and OOPS principles). Themes will need to be redeveloped as with the new template engine Twig, migrating your existing Drupal theme will not work.

8. Will our integrations with third-party software break when we migrate to Drupal 8?

Integrations with third-party software have just gotten better with Drupal 8. With web services in core in Drupal 8, creating RESTful APIs are easy and fast. This is invaluable in connecting with many third-party applications. Additionally, Drupal 8 has added many more integration modules to its list.

9. Will our core Drupal 7 modules still work?

Yes. Drupal 7 Core modules have made their way to Drupal 8 and some of them are even better in Drupal 8! While most of them are automatically upgraded, a few modules will need manual work if they don’t have an automatic upgrade path. Some Drupal 7 (or 6) modules are not mapped to the same Drupal 8 module. For example, the Block module in Drupal 7 is now divided into a Block and Custom Block module in Drupal 8. Nonetheless, many contributed modules in Drupal 7 are now in Drupal 8 core (like the Views module). 

10. What happens to our custom and contributed modules?

After Drupal 8’s adoption of the Symfony framework and Object Oriented Programming principles, Drupal has opened its doors to wider set of developers and programmers. This also helps in building code that is more robust and reusable. But this time-saving, future-ready concept brings along some bad news too. The bad news is that most of the existing custom modules and some contributed modules will need to be rebuilt from scratch to be able to support Drupal 8’s futuristic mission. But the great part about this is from Drupal 8 onwards, any major/minor upgrade will be easy as pie.

11. Will our Drupal theme break on migrating to Drupal 8?

Unfortunately, yes it will. Since Drupal 4.7 up until Drupal 7, PHPTemplate has been the default Drupal Theme engine. But with the adoption of the Twig (part of Symfony2) for a more powerful, secure and modern templating engine, themes will need to be redeveloped. However, parts of code can be replaced as is.

12. How can Drupal 8’s API-first approach benefit us?

By the year 2020, there are going to be more than 50 billion internet connected devices. Content is now consumed via a plethora of mediums – computers, mobiles, IoTs, wearables, conversational interfaces, smart TVs… and the list keeps growing. Which means, your brand needs to interact with a lot more devices and in many more formats than just a website. Content delivery has gotten a lot more challenging.

Just so we are on the same page, an API (Application Programing Interface) is a set of rules or routines (functions or programs) that specifies how applications can interact with each other. For example, if you want to display the current weather on your website, you can invoke an API with websites that offer this service.
To be able to handle the content delivery challenge efficiently, content needs to be treated like well-structured data. Drupal’s API-first approach, lets you create an API before you build your website or mobile app. This futuristic approach allows you to turn content into services which can then interact with diverse devices irrespective of the formats. While Drupal 7, also supports the API-first approach with the help of additional modules, Drupal 8 comes built-in with the content-as-a-service model.
This is what our in-house expert Drupal Practice Head, Malabya Tewari has to say about Drupal 8’s API first approach – “Drupal 8 has taken this approach to another level and here’s why- REST module is now in core, where you can create own custom web-services using Views (which is also added in core in D8). It is easier to create custom REST APIs using the core REST module. Adding basic authentication is in core as well. You can get APIs, including JSON API and GraphQL, for all entities - out of the box! 

13.What are the benefits of upgrading to Drupal 8?

One of the most stunning features of Drupal 8 is that you have (almost) everything you need, out-of-the-box. 

  • Responsive websites are not a luxury anymore, they are a necessity. All of Drupal 8’s themes are responsive off-the-rack – which not only works great with all devices also makes configuration and set up of your Drupal website a lot easier.
  • A built-in, well configured WYSIWYG editor CKEditor lets you preview and edit your content in a breeze. You also have an in-place editor that lets you edit blocks, content, menus, etc. right in the same page.
  • SEO gets you noticed and out-there. With some of Drupal’s built-in powerful SEO modules, you can take your website places! Modules like SEO Checklist, PathAuto, Redirect, MetaTag, etc. are killing it!
  • The newest and most powerful version of HTML, which is HTML5 is now built-into Drupal 8. It lets you embed complex input elements like audio, video, date, email, etc with ease and better functionality across all devices.
  • Take your business global with Drupal 8’s out-of-the-box multilingual support. You can not only create pages enabled with language-based views, even the admin interface allows you to select our preferred language. The built-in content translate modules enables you to translate any content entity into different languages.
Sep 17 2019
Sep 17

The Drupal 8 Publish Content module is a simple module that provides you additional permissions to allow users to publish or unpublish content without having to give the user the ability to administer all the content on your site. This module is a lightweight solution to help you build out your content management workflow on your Drupal 8 site.

The module also adds some additional features such as:

  • A publish/unpublish menu tab or action link for your content
  • A publish/unpublish link that can be used in views to create a customized publishing workflow

Download the Drupal 8 Publish Content module like you would any other Drupal module. You can download it with Composer using the following command:

composer require drupal/publishcontent

Install the Publish Content module.

Click on the Permissions link under the Publish Content module listing on the Extend page or Navigate to People > Permissions and find the Publish Content section. You will find a bunch of granular publishing permissions for deciding who can publish or unpublish content on your site.

Publish Content Permissions

If you navigate to a piece of content on your site, you will notice there is now an Unpublish/Publish link.

Publish Content Menu Tab

This link is also available to add in views for content listing views.

Publish Content View Link

This module provides you a ton of flexibility for building out your content management workflow on your Drupal 8 website.

Sep 16 2019
Sep 16

The Problem with Upgrades

One of the more challenging tasks when upgrading Drupal from version 6 to 7 to Drupal 8 is the upgrade/migration process. We already know that themes and modules have to be rebuilt or otherwise ported when upgrading, but even migrating content can be a time-consuming chore.

The great news is that Drupal 9 might be the last big Drupal upgrade you ever have to deal with thanks to its backward-compatibility. That said, let’s zoom in on the problem at hand and address how we can make the last big upgrade you’ll ever be tasked with go more smoothly.

Errors halt migration

What can be frustrating about upgrading from Drupal 7 to 8 is that during the process there is a good chance your new Drupal 8 build will run into a field, entity or plugin that it doesn’t understand. This can lead to uncaught exceptions which then leads to an error and an incomplete migration. 

In some cases, you find the right contributed module or patch that will add a migration path for the configuration in question and the error is resolved on the next pass. More often than not, however, you will find that no such path exists and you have to deal with the error yourself. This could entail either updating the migration yml config to skip or reroute the configuration, perhaps by writing a custom migrate plugin. It’s possible you have to repeat this step multiple times until you have addressed each error that prevents a successful migration.

One problem with Drupal upgrades is that by default it assumes you want to upgrade everything from your Drupal 6 or 7 site. The Drupal upgrade will try to migrate some configuration that is honestly rarely needed, such as theme and system configuration. It’s more typical that developers will want a subset of the content and configuration from the previous version of Drupal.

Introducing Migrate Pack

migrate pack logo

We decided to take a different approach with migrations, particularly the early stage where you are trying to get through that first pass successfully. It’s this step where you want to grab as much content and configuration “as-is” into Drupal 8 so that you can start engineering the new site in earnest.

Migrate Pack’s Approach

The Migrate Pack module, hosted on Drupal.org, is an opinionated solution with the goal to capture 80% of the content and configuration you need without errors and without manual intervention.

Migrate Pack, first of all, will skip about half of the migration configuration that typically gets imported and then discarded as a way to cut down on the “cruft” that might weigh down a migration. This configuration can be overridden to further dictate which migrations will be included or skipped.

Next, this feature will attempt to avoid some common errors with things like invalid links or entity bundles that don’t exist. Rather than halt the migration with a fatal error, a notice will get logged which can be reviewed after the process is completed. 

Finally, Migrate Pack uses configuration to skip entities and bundles that do not have a known migration path. This again allows for the migration to complete with warnings being logged along the way. All of these settings can be customized per project. We think this results in a smoother process overall.

Using Migrate Pack

By leveraging Composer we hope to bundle the various modules you’ll need to get started quickly, even if you do not have a lot of experience with migrations.

The usage for this project is fairly straightforward. The first step is requiring the feature using Composer to download the module and all of its dependencies:

$ composer require drupal/migrate_pack

Once that is done you will enable the module to enable those dependencies:

$ drush en -y migrate_pack

While Drupal can be migrated through the UI, it’s usually easier to use drush commands provided by the migrate_upgrade module (which we enable by default). The drush “migrate:upgrade” command adds the configuration you will need. Below is a common example of the full command with parameters. You will want to examine your options with the “--help” parameter the first time you run this command.

$ drush --yes migrate:upgrade --legacy-db-key drupal7 --legacy-root sites/default/files --configure-only

After Drupal 8 generates the migrate configuration you can test drive a full migration or limit the migration in any number of ways. The “--all” command will attempt to execute all migrations, example below:

$ drush migrate:import --all

With Migrate Pack’s overrides you should have fewer errors (hopefully none) on the path to a successful first pass at migration.

Finally, it’s time to export your configuration. This can be done with the drush “config-export” command (see below):

$ drush config-export -y

The idea is that now the work begins to remap configuration, integrate custom plugins and any other development required to complete the desired full migration. As mentioned, Migrate Pack has its own settings.yml that will be exported to your configuration which can be adjusted as needed. If you run into any errors or problems along the way we recommend you create issues in the project issue queue.

Next Steps

We think that this feature can be a great tool to bust through errors that typically hamper a Drupal upgrade.  We would love for you to give this module a try and help make it more robust so that Drupal developers can complete migrations faster with fewer headaches along the way.

For issues and roadmap be sure to check out the project page and issue queue for the latest updates.

Got general feedback or questions? Use the comments below or hit me up on Twitter @drupalninja 

Sep 16 2019
Sep 16

Our lead community developer, Alona Oneill, has been sitting in on the latest Drupal Core Initiative meetings and putting together meeting recaps outlining key talking points from each discussion. This article breaks down highlights from meetings this past week.

You'll find that the meetings, while also providing updates of completed tasks, are also conversations looking for community member involvement. There are many moving pieces as things are getting ramped up for Drupal 9, so if you see something you think you can provide assistance on, we encourage you to get involved.

Getting Involved Guide Refresh Meeting

September 10, 2019

This meeting:

  • Usually happens on alternate Tuesdays

  • is text only!

  • happens in threads, which you can follow to be notified of new replies even if you don’t comment in the thread. You may also join the meeting later and participate asynchronously!

  • has a public agenda issue anyone can add to.

  • Meeting transcripts are still a work in progress, but please comment on the meeting agenda issue so we can grant you credit for attending/contributing!

Issues & Progress

Personas

Out of the Box Initiative Meeting

September 10, 2019

Keith Jay and Ofer Shaal talked about Use Media images in Umami demo issue, since Media is very close to becoming stable. Umami team is working hard to prepare Umami to support it.

New recipe is coming up!

The Umami team is discussing adding new recipe content with the help of Anna Christoforou, who has very kindly allowed us to use some of her recipes. Take a look at Anna's Instagram page where you'll see some of her amazing food and images. Keith Jay is working with Anna to add her recipe for homemade hummus, currently being created as a patch for which there will be plenty of opportunity for anyone who wants to help with general reviews and translation into Spanish. Since this recipe is very simple and doesn't require any cooking time, it has been suggested that it could be the basis of a video that we'll produce to demonstrate media.

Sep 16 2019
Sep 16

The things that make me most excited about the pending release of Drupal 9 in June 2020 is that as a community we have the opportunity to look back at what we have accomplished, let go of some old deprecated code, and can now start thinking about what is next for Drupal 10.

Accomplishing

To my mind, the biggest accomplishments with Drupal 8’s release was shifting it’s entire code base and community to be more object-oriented. The adoption of Symfony into Drupal core allowed us to build a more robust and flexible Content Management Framework. The Webform module for Drupal 8 is a testament to how healthy Drupal has become as a framework. Its robustness is a combination of Drupal's Plugin API and Form API with object-oriented concepts, including dependency injection. A straightforward and personal statement is.

Drupal 8 made me a better programmer and software architect.

Looking at the code and concepts behind core and contributed modules, it is immediately apparent that everyone stepped back and reapproached a lot of technical challenges - our collective hard work and contributions resulted in great solutions and better code. Core improvements are documented as initiatives on Drupal.org. Contributed projects generally don't have initiatives. When you look at the flexibility behind a "simple" module like Metatag, maintained by Damien McKenna (DamienMcKenna) or something potentially as "complex" as Domain Access, maintained by Ken Rickard (agentrickard), it’s impressive to witness what is being built and maintained by the Drupal community.

I think it’s safe to say that everyone is ready to keep improving our software and let go of some of the technical baggage, aka deprecated code.

Deprecating

While Drupal 8 has evolved, some code has become deprecated and replaced by better code and design patterns, allowing us to improve the software. In Dries Buytaert's post, Plan for Drupal 9, he states…

"Instead of working on Drupal 9 in a separate codebase, we are building Drupal 9 in Drupal 8. This means that we are adding new functionality as backward-compatible code and experimental features. Once the code becomes stable, we deprecate any old functionality."

Regarding deprecation, what isn’t said but is worth mentioning is that some code is deprecated to improve the underlying software architecture and fix mistakes like bad naming conventions or unneeded complexity. When the Drupal community adopted Symfony, we entered uncharted waters and mistakes are going to be made.

Recently, I started sketching with my ten-year-old daughter. An important lesson I’m trying to teach her is that it is okay to make mistakes, especially because you can always fix them later. Making mistakes is part of the process in wanting to improve something. Within art, the term 'mistake' is open to interpretation. In software development however, we know when code smells bad and have a process in place to refactor bad code.

Smelling

"Bad smelling code" is an extreme statement which frankly I would only use to describe my code.

Fixing bad smelling code is going to cause a bump in the Webform module's road to Drupal 9.

Before I explain the specific issue in the Webform module for Drupal 8, I want to emphasize that the Drupal community is in good shape with a plan to move on to Drupal 9.

Since Dries' published the plan for Drupal 9, people have been busy coordinating the Drupal 9 initiative. Matt Glaman (mglaman) created Drupal Check, a tool which helps everyone find and remove deprecated code. Now, everyone needs to start removing deprecated code from their module.

Several contributors have been helping me remove deprecated code from the Webform module. Removing a deprecated service from a class constructor that which relies on dependency injection seemed impossible, but then Shawn Duncan (FatherShawn) demonstrated to the NYC Drupal Meetup an implementation pattern, which properly extends a class and safely injects dependencies.

The simplest explanation I can give for non-technical users is the Webform module failed to make it easy to extend the Webform module without having hardcoded tightly coupled dependencies. In other words, some things are going to break when the Webform is installed via Drupal 9. On top of making it easier to extend, applying this new implementation pattern reduced code complexity. This change record on Drupal.org provides a more technical explanation with a code example of the improvements.

It’s also important to acknowledge that Shawn learned this pattern while attending Alex Pott's presentation, Drupal 9 is coming: Getting your code ready. Alex credits Lee Rowlands (larowlan) and Thomas Seidl (drunken monkey) for documenting how to safely extending Drupal 8 plugin classes without fear of constructor changes.

In Alex's presentation, he recommends.

"Be very very careful with sub-classes and overriding constructors"

Sadly, I wasn't aware of how to properly sub-classes and override constructors while building the Webform module, and now we have some technical debt that needs fixing.

We must fix the sub-classing and overridden constructor problem because it paves the Webform module's road to Drupal 9. Fixing this issue is a massive patch that is going to break any Webform add-on or custom code that implements a Webform element or handler and overrides the constructor. Breaking backward compatibility in Drupal core or contributed module requires tagging a new version. Therefore, we are going to have bump the Webform module's version from 5.x to 6.x to get ready for Drupal 9.

I think everyone is starting to understand the bad news, which is that Webform 6.x is going to break some code. This change will result in fatal show-stopping errors, which are easy to fix. Developers need to apply the implementation pattern documented in the change record. Because most patches and changes to the Webform module do not alter a class' constructor, there is some good news: We should be able to maintain a Webform 8.x-5.x and Webform 8/9.x-6.x branch at the same time.

Following the tradition established over the last three years, any significant bump to the Webform module's release status has been tied to the New Year.

Along with preparing for Drupal 9, the 8/9.x-6.x branch of the Webform module will include any significant API changes or improvements. For example, I would like to fix how conditional logic handles a disabled element's value.

Another significant and thankless task that needs to be accomplished is migrating all the Webform module's SimpleTests to PHPUnit. Fortunately, many people have thanked me by backing the Webform module's Open Collective. Now there are funds available to pay for the monotonous task of migrating dozens of tests.

Understanding

Last year's stable release of Webform 8.x-5.x was a Christmas present, and I hope everyone understands that this year's upcoming tagging of Webform 9.x-6.x is not a lump of coal but a big step in the Webform module's road to Drupal 9 and beyond.

I rarely get to commit a patch that makes my code easier to maintain. I am thankful for Shawn Duncan (FatherShawn), recognizing how cool and important this implementation pattern was during Alex's presentation and then sharing it with the NYC Drupal community. This is how the hive mind that is Open Source works to build Drupal. The Drupal community should thank everyone involved who helped transitioned our code and community to Drupal 8. In a few months, we can thank them once again for continuously improving and refactoring Drupal core and contributed projects and paving our road to Drupal 9 - collectively and collaboratively, we’ll get there...it’s what we do.

Almost done…

We just sent you an email. Please click the link in the email to confirm your subscription!

OK

Sep 16 2019
Sep 16

The countdown for Drupal 9 has begun. Whether you’ve recently moved to Drupal 8 or not, soon it will be time for another upgrade.

In the past, migrating from one version to another has been similar to moving from Drupal to just another CMS, bringing in more time and fatigue.

However, the upgrade can be made much easier and painless this time. Let’s dive into more details and understand as to why moving on to Drupal 9 would be a better choice.

Why Should You Upgrade?

With end of life approaching for Drupal 7 and 8 in November 2021, operating the website on them won’t be an option.

However, at the same time, it might be overwhelming for Drupal 7/8 site owners to know their website will need the upgrade. And when the site is running absolutely fine, then it might be difficult to reach a conclusion of whether an upgrade is needed or not.

Here are 3 reasons why you should consider upgrading your site:

  1. With security patches discontinued and older versions not maintained, you need to secure your site from potential problems
  2. Upgrade to the latest features and usability enhancements, eliminating the need of a third-party modules and extensions
  3. Simplify and clean up content and ponder over ways to improve user experience and design

The good news for Drupal 7/8 site owners is that even when it goes out of official support in November 2021, remaining Drupal 7/8 sites won't stop working at that point.

The existing customers who wish to remain on Drupal 7/8 need to assume hosting responsibility post 2021. However, staying with older version might make you vulnerable to several security challenges, which need to be dealt with.

timeline-for-drupal9

Should An Existing Drupal 7 Site Be Upgraded to Drupal 8 or 9?

One of the major reasons that 7 lac Drupal 7 sites still haven’t migrated to Drupal 8, is due to the known challenges in migration process. And with the majority of people on Drupal 7, it is quite likely that most of them do not want to upgrade their CMS twice in the span of one year.

A safe bet seems to be migrating from Drupal 7 to Drupal 9. But will the site be secure? Let’s get to know a few facts.

Moving from Drupal 8 to Drupal 9 is a minor upgrade in terms of functionalities, whereas it is a completely new whole thing to move from Drupal 7 to 9. Hence, it will cost you the same whether you migrate from Drupal 7 to 9 or take up the option of going from Drupal 7 to 8 to 9.

Also, with the recent upgrade, it might take some time for the marketplace to create Drupal 9 compatible modules. With Drupal 7’s end of life approaching, you will have to upgrade to Drupal 9 and go live within a span of 18 months, to stay secure with your site.

However, this won’t happen if you stay updated with the latest version of Drupal 8. So, it is recommended to follow the upgrade cycle from Drupal 7 to 8 to 9.

Drupal 9 Brings No New Features. Then Why Upgrade?

“Drupal 9.0 should be almost identical to the last Drupal 8 release ... Updating from Drupal 8's latest version to Drupal 9.0.0 should be as easy as updating between minor versions of Drupal 8.” Dries Buytaert

Drupal 8.9 will be the last minor release which will have new features, and thereafter no new features will be added. As we will run out of single digit numbers to the right of the decimal point at 8.9, the next update gets called Drupal 9.0.

So if you have Drupal 8.9, and your site works well on it and there’s no dependency on any deprecated code, the upgrade to Drupal 9 should be just like any other release.

Post 2021, no security patches will be further rolled out by the Drupal community, and moreover Drupal 7 will not be compatible with upcoming PHP versions.

A large number of changes made to Drupal 8 at the time of its release has made it relatively scalable and future-proof. It was designed specifically to make it simple to transition to the latest version, simplifying the entire migration process.

Moving to Drupal 9 would be a necessity as Drupal 8’s major dependency - Symfony 3 will no longer be supported, post November 2021. Currently Drupal community is ensuring to optionally support Symfony 4 in Drupal 8, to allow sites to evaluate it before it is required in Drupal 9.

Drupal 9 will be similar to Drupal 8 and will offer backward-compatibility (refers to allowing interoperability with the previous system). This means the only way to keep the migration process easy would be to keep its modules and themes up-to-date with the latest Drupal 8 APIs, and get rid of deprecated codes as new features mark their success.

We have already mentioned how to find and fix the deprecated code in our blog - Site Owner’s Guide to a Smooth Drupal 9 Upgrade Experience.

The deprecated code is being continually removed from Drupal 8 as per the data collected (as shown below).

drupal8-deprecated-code

Source

Why Remove Deprecated Code in Drupal 9?

Drupal 9 is built on the code base of Drupal 8, ensuring Drupal 8 to 9 is not a big leap. This has the following 3 benefits:

  1. The all new Drupal 9 ready code gets deployed on Drupal 8 sites and issues can be tested.
  2. We can work out the issues in the new code before Drupal 9 release.
  3. Feedback is provided based on this new code and can be improved.

With time, effort is being made to make Drupal better. There are functions which have been around for a long time but will not be a good fit in the latest release. Most were deprecated in Drupal 8.7.0, which will be removed in Drupal 9.

To sum it all, the key to achieving this smooth transition to Drupal 9 is keeping your Drupal 8 site fully up-to-date.

We at Srijan are here to help with any Drupal-related questions that you might have and can help you plan out your Drupal roadmap. Contact us for a smooth upgrade to the latest release to make your site future-ready.

 

Sep 16 2019
Sep 16

Your local websites are always developed within the same operating system, that is the operating system of your machine (Windows, OSX, Linux). However, the online version of your site is probably hosted on some type of Linux server. Vagrant allows developers to have the conditions to replicate these systems within a Vagrant box. There are multiple kinds of boxes since Vagrant is a very popular alternative amongst developers. One of these boxes is called Scotch Box. Scotch Box is a preconfigured Vagrant Box with a LAMP (Linux, Apache, MySql, PHP) or a LEMP (Linux, Apache/Ngnix, MySql/MongoDB, PHP) stack in it.

This tutorial will explain the basic installation of these tools. Keep reading to learn how!

Required Tools

  1. Vagrant
  2. VirtualBox
  3. Git
  4. Scotch Box  

Step #1. - Install Vagrant

  • Download the Vagrant installer for your operating system
  • Double click or whatever you do to install applications on your system

190917 drupal vagrant

After installation, open the terminal application of your system and type:

vagrant -v

190917 drupal vagrant 001

Step #2. - Install Virtualbox

  • Download the Virtualbox installer for your operating system
  • Install this application on your system too

190917 drupal vagrant 003

If you type virtualbox -v on your terminal, it will open Virtualbox. That means it has been installed correctly. You can now close Virtualbox.

Step #3. - Install Git

Git has pretty much the same installation procedure. So here is a link to the downloads page.

Step #4. - Clone Scotch Box

Once you have downloaded and installed everything required, go to the terminal application of your system and head over to your Projects folder and type:

git clone https://github.com/scotch-io/scotch-box mydrupalbox


mydrupalbox is the name of the project (or the box), so you can replace this name with your own project name.

190917 drupal vagrant 004

  • cd mydrupalbox
  • ls

190917 drupal vagrant 005

Take a look at the contents of the project folder, and you will see the same files as in the Github repository (from where you actually have cloned them).

  • Type vagrant up

190917 drupal vagrant 006

This will take some time depending on the speed of your internet connection. Take into consideration that your box is going to contain a whole operating system.

Once the installation process has finished, open your browser and type in the address bar:

192.168.33.10

You can scroll down and take a look at the configuration defaults of Scotch Box.

190917 drupal vagrant 007

Step #5. - Access Scotch Box via Terminal

  • Type in your terminal: 

vagrant ssh

190917 drupal vagrant 008 

You are now inside a Ubuntu 16.04 LTS system.

 Step #6 - Create a Database

  • Type:   mysql -u root -p

Enter root as password (you won’t see any characters rendered on the screen). Since this is a local development environment, this is no problem at all.

190917 drupal vagrant 009

To create the database and assign permissions to a user on it, 

  • type these 3 commands in order:

create database my_drupal_8;

grant all privileges on my_drupal_8.* to admin identified by '1234';

flush privileges 

You can assign your own user names and passwords for your databases. I like those for my local machine.

190917 drupal vagrant 010

Type exit to leave mysql.

190917 drupal vagrant 011

Step #7 - Install Drupal

  • Type cd /var/www/public

This is the public folder of your local server, just like the public_html folder at your hosting provider.

Type the following commands one after the other: 

wget https://www.drupal.org/download-latest/tar.gz

This will download the latest Drupal version in a file called tar.gz

tar xzvf tar.gz

This command decompresses the downloaded file 

rm index.php

Deletes the default index.php file (the configuration defaults screen you saw on your browser on Step #4)

cp -r drupal-8.7.7/* .

To copy all the files and folders inside the decompressed drupal-8.7.7 folder one level up to the public folder.

rm -rf drupal-8.7.7/

To delete this folder

rm tar.gz

To delete the compressed file

composer install

Adds Composer to your Drupal installation

190917 drupal vagrant 013

cd sites/default

To access the default folder

cp default.settings.php settings.php

Copies the default settings and creates a new settings file

Now go to your browser and type again in the address bar:  192.168.33.10 

You will see the Drupal installation screen. 

  • Choose your language and click Save and continue
  • Select an Installation profile and click Save and continue
  • Verify the requirements and click Continue
  • Enter the required fields to connect the database to Drupal (you created these in step #6)
  • Click Save and continue

190917 drupal vagrant 015

The Drupal installation will begin.

  • Provide the required information for your site 
  • Click Save and continue

190917 drupal vagrant 016

You will see the home page of your site.

Congratulations, you have just installed Drupal on top of a Vagrant box called Scotch box.

The virtualization occurs through Virtualbox. All this happens in the background through an API which communicates Vagrant with Virtualbox. The virtual boxes are stored in the home directory of your system (OSX, Linux) inside a folder called VirtualBox VMs.

To shut down the virtual machine, type:

exit to leave the ssh mode 

vagrant halt

More about Vagrant here.

Please, leave us your comments below. Thanks for reading!


About the author

Jorge lived in Ecuador and Germany. Now he is back to his homeland Colombia. He spends his time translating from English and German to Spanish. He enjoys playing with Drupal and other Open Source Content Management Systems and technologies.
Sep 13 2019
Sep 13

Wanna know my rules when I am writing templates in PatternLab for Drupal view modes such as teasers, cards, search results, etc? Read on, my friend...

BEM Classes for Streamlined Classes

Classes used in the card view mode should be prefixed with card__ not node__

  • We need to make sure that CSS for cards does not leak out into other components
  • We need to make sure we have as little selector specificity as possible - so we have .card__content instead of .card .node__content in our CSS.

Card Variations

If you want to have slight variations for cards, please add that as an option in the classes array. Examples of this might be for a card for a specific content type, or a card when there is no image present, or a card that has the image on the left/right.


  1. {% set classes =
  2. [

  3. "card",

  4. content_type ? "card--" ~ content_type,

  5. image ? "card--has-image" : "card--no-image",

  6. alignment ? "card--" ~ alignment,

  7. ]

  8. %}

This will print:


  1. <article class="card card--event card--no-image card--image-left"></article>

We can then apply overrides using these variations. For example, if the date field on a event is bold but but on a blog is not, we can have code like so:


  1. .card__date {

  2. color: $c-grey--darkest;

  3. }

  4. .card--event .card__date {

  5. font-weight: $fw--bold;

  6. }

Avoid Twig Blocks

Try not to use Twig blocks in view modes - {% block image %} If we use Twig blocks, then in the Drupal template anything can be put in the {% block %} to override it. If we change the order of things in the {% block %} in PL, this will have no effect in Drupal, since the Drupal version will be overriding it.

For example, the Drupal template could have something like this:


  1. {{ content.field_date }}

Avoid Drupal Templates

Try to avoid having templates based on specific content types for view modes. This is usually necessary for Full view mode, but for Teaser, Card, etc let's try to keep them more generic.

If you need to use a content-type template, that is fine; but it should be the exception, not the rule.

In general, since each content type Card should be similar, then each content type should be able to use the same node--card.html.twig template file.

Avoid {% if not page %}

{% if not page %} Should not be needed in view modes. A Card or Teaser will never be a full page. For the same reason, we can usualy leave this out in the content type full view mode templates, since the full view mode will always be a page.

Do Not Hardcode Heading Levels

Unless you know that the Card view mode is never going to have a <h2> item above it, do not set <h2> or <h3> etc as the heading element.

We do not want to have a HTML structure like this:


  1. <h1>Page Title</h1>

  2. <h2>Views Card Block Title</h2>

  3. <h2>Card Title</h2>

  4. <h2>Card Title</h2>

  5. <h2>Card Title</h2>

Instead, we would like this:


  1. <h1>Page Title</h1>

  2. <h2>Views Card Block Title</h2>

  3. <h3>Card Title</h3>

  4. <h3>Card Title</h3>

  5. <h3>Card Title</h3>

In general, view modes will take a <h2>, so let's place that as our default.


  1. {% set card_title_element = card_title_element|default('h2) %}
  2. <article>

  3. <{{ card_title_element }}>{{ card_tite }}</{{ card_title_element }}>

  4. </article>

Then, in our Drupal template, we can set the element depending on whether it has a parent <h2> or not; for example, if it is in a 'Related Content' building block and that building block has a title such as: <h2 class="related-content__title">Read More Like This</h2>.

We can do so in the Drupal template with a variation of this:


  1. {% if node._referringItem.parent.parent.entity.field_p_rc_title.value %}
  2. {% set teaser_title_element = 'h3' %}

If the above returns as false, our default h2 from the PL pattern will kick in.

Use Prefixes for Variables

Let's use prefixes for variables, using {{ card_content }} instead of {{ content }}. This will help avoid global variables being used by accident in our components (unless we want them to, such as the image_alt variable from data.json). The {{ content }} if used, will not necessarily print the content of your component when that component is used outside of its own context: for example, if you print a card in a views list in a sample page.

Variables for Specific Content Types

If there's a variable for a specific content type, such as the location field for an event card, we can just wrap that in an {% if %} statement, like so:


  1. {% if content_type == 'event' %}
  2. {% if event_location %}
  3. <div class="card__location">

  4. {{ card_location}}

  5. </div>

Then that variable will not print in any other content type's template.

Sep 13 2019
Sep 13

Agiledrop is highlighting active Drupal community members through a series of interviews. Now you get a chance to learn more about the people behind Drupal projects.

We had a great talk with Suzanne Dergacheva, co-founder of the Canadian web agency Evolving Web and member of the Drupal Association Board of Directors. She's also involved in the Admin UX study and in the Promote Drupal initiative, and is an active member of the Canadian Drupal community. Find out more about Suzanne in our interview.

1. Please tell us a little about yourself. How do you participate in the Drupal community and what do you do professionally?

I started a web agency in Montreal about 12 years ago, Evolving Web, and after about a year we decided to specialize in Drupal; so, I started going to Drupal meetups and got really involved in the local community in Montreal, even organizing DrupalCamp Montreal. 

Then over the years I’ve built up my Drupal team at Evolving Web, kept going to events and got more and more involved, organizing things like codesprints and then getting involved in contributing in small ways - with documentation, etc.

A couple years ago I started getting involved in the Admin UX study for Drupal and I’ve been really passionate about that. It’s an initiative to improve the content editor experience for Drupal. One of the things I’m most excited about right now is actually the Promote Drupal initiative, which I think has a lot of potential to build the market for Drupal.

Last year, I was then elected to the board of the Drupal Association; it’s been a lot of fun just getting right in there and seeing the potential of the communities and all these ideas around growing Drupal. I’m really excited about that too. 

2. When did you first come across Drupal? What convinced you to stay, the software or the community, and why?

I think the first time I downloaded Drupal was about 6 months before we decided to start using it for projects. My business partner and husband Alex said “Oh, maybe we should try using Drupal!” and I think I went to drupal.org and tried to install it, and I didn’t get very far. That was probably my first encounter.

But the second encounter was when we had a project for a political party in Quebec. Every website here in Quebec has to be in English and French, so they were pretty keen to use Drupal. So we said to ourselves, “Okay, we know Ruby on Rails, we know WordPress, I think we can figure out Drupal. No problem, we’ll figure it out!” 

This was when Drupal 6 had just come out, and there were some bugs in the multilingual system that we found. So, the first encounter with Drupal was a very positive one, but also a challenge, and we got right in there and started fixing things. 

To the question of what convinced me to stay, the software or the community, I would say both. In the last 6 years, I built a training program around Drupal, so I think what keeps me really excited about Drupal is that when I teach people, they feel very empowered. And there is a learning curve around Drupal, but I think when I get to actually help people learn and get over that learning curve it’s very inspiring, and so that’s a big part of what motivates me. 

And what keeps me involved in the community is just how much I can learn from everyone and how passionate everyone is. There are so many examples of people in the community who have really inspired me. That’s what keeps me not just using Drupal but giving back and being involved.

3. What impact has Drupal made on you? Is there a particular moment you remember?

One of the great experiences I’ve had was I think about 6 or 7 years ago now when we organized a DrupalCamp in Montreal and we decided that instead of just a DrupalCamp we also wanted to organize a codesprint. 

We figured that since we’re in Montreal and we all build multilingual websites, we should organize a multilingual codesprint. This was when work on Drupal 8 was just starting, it was still a long way off and work was just beginning on the Multilingual initiative

Because we decided to do this far enough in advance, we were able to get a community grant from the Drupal Association to help us pay for different people to come from Europe to participate. That meant that we had Gábor come, as well as Francesco Placella (plach__) who created the Entity Translation module for Drupal. 

We had these people coming from Europe to participate and that inspired a lot of excitement in the local community. The developers on our team got really into it; we had a lot of momentum behind this codesprint.

It was just so exciting to see how this local group could create an international codesprint and really get some good work done. Drupal 8 was a long way off, but still we were able to make some good progress that weekend.

4. How do you explain what Drupal is to other, non-Drupal people?

What I would normally say is that when you look at websites, they are really very similar, they have a lot of the same features. You have a menu across the top, you have a logo in the top left corner, so instead of creating a website from scratch you want to use a platform to do it.

What Drupal lets you do is get a lot of these features that everybody uses out of the box, while also giving you the flexibility to customize your website however you want. You can add features, you can add things like event search and ecommerce - the sky’s the limit. Drupal strikes that balance of providing key features out of the box, and also letting you customize everything.

5. How did you see Drupal evolving over the years? What do you think the future will bring?

I see Drupal evolving to become more mature and used by bigger and bigger organizations, which is really exciting. I also hope that Drupal, being such a flexible tool, will still be useful to the organizations that use it today. I think it’s versatile enough that it can be used by many different communities; for example, the higher education and non-profit communities have really embraced Drupal and I see that being a long-term thing.

I see Drupal kind of growing to new places, new applications, especially with the maturing of decoupled Drupal solutions. I think it’s really going to evolve a lot in the near future. We’re going to see more standardization on how to do decoupled Drupal and that’s really going to change the landscape.

I also think that the Drupal community is evolving. At the last DrupalCon, there was a content editors / digital marketers track, and it was really exciting to see people coming from the community who aren’t necessarily developers, but more people on the content and marketing side, people we think of as users.

I think Drupal needs more of that, our community needs to embrace people with a larger set of skills and backgrounds in order to keep growing. We need to have marketers, we need to have UX designers and project managers involved in the project in order for it to be successful. 

6. What are some of the contributions to open source code or to the community made by you or other Drupalists that you are most proud of?

There are a couple of things. I’m really excited about things like the Media initiative, which seems like it’s really driven by creating a great user experience. It’s always really positive to start to see development that isn’t just driven by a functional set, but more in terms of “here’s what the users wanted and here’s a picture of what we want to build”, with the focus on a good user experience.

Another initiative that I’m really impressed by is the Layout Initiative. It’s such a mature initiative. The focus on accessibility really shows that Drupal is such a leader in the open source community. The team is creating a tool that’s so flexible and innovative, and gives so much power to the content editor, but at the same time really focused on creating a tool that’s accessible and can be used by anyone.

7. Is there an initiative or a project in the Drupal space that you would like to promote or highlight?

Beyond code there are some of these initiatives that I think are really worth highlighting. I’m really excited about where the Promote Drupal initiative is going and how to get more marketers involved in that. There’s also an event organizers working group that’s being formed to help Drupal Camps come together and share resources. I think both of these have the potential to grow our community.

8. Is there anything else that excites you beyond Drupal? Either a new technology or a personal endeavor. 

This summer I joined a bike club and that’s been really fun, doing something not in front of a screen and instead just getting outside. And, just like with Drupal, people are really passionate about cycling and welcoming novices like me into the community. So I’m excited about going to DrupalCon Amsterdam in October and cycling around!
 

Sep 13 2019
Sep 13

One of the increasingly popular architectural paradigms that Drupal has been seen as a leader in, is the concept of a single Drupal software package that can be spun up to power networks of websites with varying degrees of commonality. This is usually driven by the ambitious goal of being able to code and configure Drupal once and then leverage that effort as either an entire platform or foundation for many "networked" sites.

Beginning down the path of starting a project like this is complex and unfortunately isn't helped by some of Drupal's (confusingly named) features which describe aspects of reusability but aren't in themselves a full fledged approach to architecting such a network. In addition to that, there are many misconceptions about Drupal's capabilities and affordances when it comes to building such networks.

In order to try and expose some of the discovery and decision making process behind starting an ambitious Drupal network project, the following is a non-exhaustive list of popular architectural paradigms that exist, evaluated on the following axis:

  • Up-front investment: the up-front cost of starting a network of sites.
  • Per-unit investment: the cost of introducing a new site to the network
  • Flexibility: the ability to customise and create bespoke experiences within each network site
  • Platform maintainability: the ability to iterate and evolve the network as a whole

As with all complex projects, there are a large number of requirements and constraints which factor into technical decision making, so these approaches are a broad view of the landscape of Drupal's capabilities.

Models of networked sites

Starter-kits

A starter-kit consists of creating a Drupal distribution or install profile, catering to as much common functionality as possible across the network in an initial development phase and then allowing each new website to make any additional required customisations as needed. These customisations may consist of writing additional code, enabling new dependencies and modifying the configuration shipped with the initial distribution.

For each individual site, this model affords the most flexibility. By allowing each site to evolve independently any new requirements or features perceived as bespoke can be implemented and deployed without making consideration to the starter-kit itself or other websites within the network.

The major drawback of this approach is being able to maintain and evolve the network of sites as a whole. Each new site in the network creates a new deployment with it's own configuration, dependencies and code, meaning new features and bug fixes can't be deployed across the whole network without specific individual effort and conflict resolution for each site. In practice once a site is live under this model, it can effectively be considered a siloed project without a significant relationship to other sites in the network.

As far as how feature rich an initial starter kit is largely depends on the project. For example, early versions aGov 8, the starter-kit distribution PreviousNext built and maintained for Australian government organisations was intentionally fairly rudimentary in the amount of content types it shipped with. The goal was a foundation to launch you into best practices, without being overly prescriptive. When future iterations of aGov were released that baked in Drupal 8's new media capabilities, it was not possible to deploy this iteration to all existing installations.

In a similar vein, I would classify govCMS8, the Australian governments own Drupal distribution under this same model. By default, the distribution ships with a lot more features and a lot more ready to go configuration than most starter-kits, however both SaaS and PaaS deployments of govCMS allow a wide scope of deep structural configuration changes to Drupal, which essentially sets up each project to function as a standalone unit after the initial build.

Products

Another less widespread approach is the product model. Under this model, a Drupal distribution is leveraged as the full feature set for all sites in the network and all sites make use of the initial and ongoing development roadmap of the product. This approach is arguably less flexible, since each individual site doesn't have unfettered access to extend and manipulate the platform.

The major advantage of this approach is, a single team can scale their ongoing enhancement and maintenance efforts to the entire network of sites regardless of the number of sites in the network.

Under the product model, since the feature set running on each site is a known and strictly defined set of configuration and dependencies, a team could feasibly migrate hundreds of sites to using new features of Drupal by working on a single software product. All sites would be the target of all new evolutions of the platform and benefit from it's ongoing maintenance. Evolutions of the platform would not strictly be limited to features, but also updating dependencies or moving to new major versions of Drupal. 

An example of a project PreviousNext delivered under this model was a video content platform for group of media organisations. Each organisation could leverage the product to spin up a website to share their videos and engage with their audience. New features were added regularly and at its height, 26 different websites serving vastly different audiences of users would evolve together. One of the challenges of maintaining a product distribution is the governance around flexibility and change. While tempting to allow each site to develop its own product roadmap, when each site required it's own bespoke features, the process for such requests would be followed:

  • The site owner raises a request a change to their website, "please replace the hero image on the homepage with a slideshow of images".
  • The team evaluates the request and places it on the product roadmap.
  • Instead of replacing all hero images with slideshows, the feature is developed an optional choice for content editors: you may either upload slides or a hero image.
  • The feature would be built, tested and deployed to all sites in the network.
  • The site owner is then able to upload slides and all other site owners in the network have the same capability.

This approach certainly takes a level of control and focused organisation effort to accomplish, however leveraged properly can have significant payoffs for the maintainability of a network of sites as a whole. Examples of product based distributions in the Open Source Drupal ecosystem are Open Social or Drupal Commons.

Federated back-ends

Another approach to building out a network of sites is the notion of a federated back-end. This dovetails with terms like "service oriented architecture", "multi-tenancy" or "content hub", where a single deployment and instance of Drupal is leveraged as a large repository of content for multiple web front-ends.

Under this model, instead of the boundary between users and content being defined by different databases and deployments of Drupal, they must instead be implemented in the application layer. That is, Drupal itself is customised and tailored to meet the access needs of the organisations sharing the same Drupal site. While this is certainly additional work and complexity, if a single group of content authors is responsible for content across the whole network, it can be advantageous to lower these barriers. Maintenance for the content hub is also fairly light touch under this model, since only one installation needs to be updated and maintained.

This pattern also intersects with the "product" style of application. Since all web properties are powered by the same underlying application, they closely share a feature set and product roadmap. While this model is also often deployed in conjunction with a decoupled front-end, Drupal sites are capable of delivering integrated front-ends to network sites from a single federated back-end. In some cases, the federated model has an elevated risk of being a single point of failure, given a single deployment and instance is responsible for sites in the network.

An example of the federated back-end model can be illustrated in the "Tide" component of vic.gov.au's "Single Digital Presence" project. Drupal 8 is deployed as a single instance serving multiple decoupled front-ends. The features of the single content repository are documented and available for evaluation by prospective users. 

Independent sites

One option that isn't often considered in of a lot of organisations when evaluating reusability of features and effort across a network of Drupal sites is simply building multiple completely unrelated sites and using smaller units of functionality as a mechanism for reuse. Drupal has a mature concept for sharing functionality across Drupal sites: the humble module.

While this approach in general doesn't strictly fit the theme of this blog post, in some cases writing and publishing a general case module which doesn't boil the ocean on dictating the tools, approach and features used to build a new Drupal site, is the best solution for some projects.

These kinds of projects would be driven by websites that are characterized as mostly unique with various related and unrelated feature sets. This is also an approach consistent with day to day of Drupal development outside the scope of network projects. With Drupal's open ecosystem, opportunities for collaboration and reuse often drive building functionality in reusable modules and publishing those modules on drupal.org.

Common misconceptions

Install profiles

Drupal "profiles" or "distributions" are not a silver bullet for developing and organising an architecture for networks of Drupal sites. They are incredibly flexible, so how they are deployed and leveraged still contain the same governance and architectural decisions discussed.

Configuration management

Configuration management is not a silver bullet. While configuration management has simplified a range of complex deployment problems that were present in Drupal 7 sites, it doesn't drive a particular architecture for networks of sites. The tools in Drupal core are continually getting sharper and while innovations in contrib have experimented with new mechanisms for deploying and updating configuration, it's not an end-to-end architectural solution for network sites.

Multisite

The multisite concept in Drupal is frequently misunderstood as an approach for building a network of sites. In reality, multisites are a tool for deploying any configuration or setup of Drupal sites to a shared document root. It doesn't produce any tangible outcome as far as project architecture is concerned beyond forcing multiple sites to be hosted on the same server.

Headless Drupal & JavaScript frameworks

While some of these approaches, like the "federated back-end" are significantly benefited by a decoupled front-end, headless Drupal is compatible with all models of network sites. You could build a product or starter-kit that was either fully integrated, progressively decoupled or completely decoupled and the same back-end architectural decisions would apply.

Drupal has strong API based functionality, which can be enabled and configured as required. The approach of evaluating and selecting frameworks or front-ends to consume Drupal, have their own set of considerations that need to be carefully evaluated for fitness in any given project.

Styling and appearance

Styleguide-driven development has largely matured to solve issues with reusability, inheritance and extensibility of visual components. This approach has been the foundation for all new sites built by PreviousNext for last few years, see our blog posts. By building components and style guides, duplication of effort when building front-ends has been minimised across both traditional and network based projects. For that reason the visual style and consistency of sites within a network is not necessarily a factor when considering Drupal architectural paradigms.

Summing up

Given the size of Drupal's ecosystem and the increasingly rapid pace of evolution, describing all of the factors and challenges that play into a large network site project is difficult. As always the process of rigorous discovery and deeply understanding a project's goals and requirements should always be the first step in beginning a technical project.

Photo of Sam Becker

Posted by Sam Becker
Senior Developer

Dated 13 September 2019

Comments

It should be mentioned that if Aegir (https://www.aegirproject.org/) is used for the hosting, management, provisioning, etc. of sites in any of the above scenarios, it'll save a lot of trouble. For example, all sites running on the same platform (Aegir-speak for a Drupal codebase) can be upgraded with a single button click, with rollback on any failures.

Pagination

Add new comment

Sep 13 2019
Sep 13

Throughout the series, we explored many migration topics. We started with an overview of the ETL process and workflows for managing migrations. Then, we presented example migrations for different entities: nodes, files, images, taxonomy terms, users, and paragraphs. Next, we shifted focus to migrations from different sources: CSV, JSON, XML, Google Sheet, Microsoft Excel, and LibreOffice Calc files. Later, we explored how to manage migrations as configuration, use groups to share configuration, and execute migrations from the user interface. Finally, we gave recommendations and provided tools for debugging migrations from the command line and the user interface. Although we covered a lot of ground, we only scratched the surface. The Migrate API is so flexible that its use cases are virtually endless. To wrap up the series, we present an introduction to a very popular topic: Drupal upgrades. Let’s get started.

Note: In this article, when we talk about Drupal 7, the same applies to Drupal 6.

What is a Drupal upgrade?

The information we presented in the series is generic enough that it applies to many types of Drupal migrations. There is one particular use case that stands out from the rest: Drupal upgrades. An upgrade is the process of taking your existing Drupal site and copy its configuration and content over to a new major version of Drupal. For example, going from Drupal 6 or 7 to Drupal 8. The following is an oversimplification of the workflow to perform the upgrade process:

  • Install a fresh Drupal 8 site.
  • Add credentials so that the new site can connect to Drupal 7’s database.
  • Use the Migrate API to generate migration definition files. They will copy over Drupal 7’s configuration and content. This step is only about generating the YAML files.
  • Execute those migrations to bring the configuration and content over to Drupal 8.

Preparing your migration

Any migration project requires a good plan of action, but this is particularly important for Drupal upgrades. You need to have a general sense of how the upgrade process works, what assumptions are made by the system, and what limitations exist. Read this article for more details on how to prepare a site for upgrading it to Drupal 8. Some highlights include:

  • Both sites need to be in the latest stable version of their corresponding branch. That means the latest release of Drupal 7 and 8 at the time of performing the upgrade process. This also applies to any contributed module.
  • Do not do any configuration of the Drupal 8 site until the upgrade process is completed. Any configuration you make will be overridden, and there is no need for it anyways. Part of the process includes recreating the old site’s configuration: content types, fields, taxonomy vocabularies, etc.
  • Do not create content on the Drupal 8 site until the upgrade process is completed. The upgrade process will keep the unique identifiers from the source site: `nid`, `uid`, `tid`, `fid`, etc. If you were to create content, the references among entities could be broken when the upgrade process overrides the unique identifiers. To prevent data loss, wait until the old site's content has been migrated to start adding content to the new site.
  • For the system to detect a module’s configuration to be upgraded automatically, it has to be enabled on both sites. This applies to contributed modules in Drupal 7 (e.g., link) that were moved to core in Drupal 8. Also to Drupal 7 modules (e.g. address field) that were superseded by a different one in Drupal 8 (e.g. address). In any of those cases, as long as the modules are enabled on both ends, their configuration and content will be migrated. This assumes that the Drupal 8 counterpart offers an automatic upgrade path.
  • Some modules do not offer automatic upgrade paths. The primary example is the Views module. This means that any view created in Drupal 7 needs to be manually recreated in Drupal 8.
  • The upgrade procedure is all about moving data, not logic in custom code. If you have custom modules, the custom code needs to be ported separately. If those modules store data in Drupal’s database, you can use the Migrate API to move it over to the new site.
  • Similarly, you will have to recreate the theme from scratch. Drupal 8 introduced Twig which is significantly different to the PHPTemplate engine used by Drupal 7.

Customizing your migration

Note that the creation and execution of the migration files are separate steps. Upgrading to a major version of Drupal is often a good opportunity to introduce changes to the website. For example, you might want to change the content modeling, navigation, user permissions, etc. To accomplish that, you can modify the generated migration files to account for any scenario where the new site’s configuration diverts from the old one. And only when you are done with the customizations, you execute the migrations. Examples of things that could change include:

  • Combining or breaking apart content types.
  • Moving data about people from node entities to user entities, or vice versa.
  • Renaming content types, fields, taxonomy vocabularies and terms, etc.
  • Changing field types. For example, going from Address Field module in Drupal 7 to Address module in Drupal 8.
  • Merging multiple taxonomy vocabularies into one.
  • Changing how your content is structured. For example, going from a monolithic body field to paragraph entities.
  • Changing how your multimedia files are stored. For example, going from image fields to media entities.

Performing the upgrade

There are two options to perform the upgrade. In both cases, the process is initiated from the Drupal 8 site. One way is using the Migrate Drupal UI core module to perform the upgrade from the browser’s user interface. When the module is enabled, go to `/upgrade` and provide the database credentials of the Drupal 7 site. Based on the installed modules on both sites, the system will give you a report of what can be automatically upgraded. Consider the limitations explained above. While the upgrade process is running, you will see a stream of messages about the operation. These messages are logged to the database so you can read them after the upgrade is completed. If your dataset is big or there are many expensive operations like password encryption, the process can take too long to complete or fail altogether.

The other way to perform the upgrade procedure is from the command line using Drush. This requires the Migrate Upgrade contributed module. When enabled, it adds Drush commands to import and rollback a full upgrade operation. You can provide database connection details of the old site via command line options. One benefit of using this approach is that you can create the migration files without running them. This lets you do customizations as explained above. When you are done, you can run the migrations following the same workflow of manually created ones.

Known issues and limitations

Depending on whether you are upgrading from Drupal 6 or 7, there is a list of known issues you need to be aware of. Read this article for more information. One area that can be tricky is multilingual support. As of this writing, the upgrade path for multilingual sites is not complete. Limited support is available via the Migrate Drupal Multilingual core module. There are many things to consider when working with multilingual migrations. For example, are you using node or field translations? Do entities have revisions? Read this article for more information.

Upgrade paths for contributed modules

The automatic upgrade procedure only supports Drupal core modules. This includes modules that were added to core in Drupal 8. For any other contributed module, it is the maintainers’ decision to include an automatic upgrade path or not. For example, the Geofield module provides an upgrade path. It is also possible that a module in Drupal 8 offers an upgrade path from a different module in Drupal 7. For example, the Address module provides an upgrade path from the Address Field module. Drupal Commerce also provides some support via the Commerce Migrate module.

Not every module offers an automated upgrade path. In such cases, you can write custom plugins which ideally are contributed back to Drupal.org ;-) Or you can use the techniques learned in the series to transform your source data into the structures expected by Drupal 8. In both cases, having a broad understanding of the Migrate API will be very useful.

Upgrade strategies

There are multiple migration strategies. You might even consider manually recreating the content if there is only a handful of data to move. Or you might decide to use the Migrate API to upgrade part of the site automatically and do a manual copy of a different portion of it. You might want to execute a fully automated upgrade procedure and manually clean up edge cases afterward. Or you might want to customize the migrations to account for those edge cases already. Michael Anello created an insightful presentation on different migration strategies. Our tips for writing migrations apply as well.

Drupal upgrades tend to be fun, challenging projects. The more you know about the Migrate API the easier it will be to complete the project. We enjoyed writing this overview of the Drupal Migrate API. We would love to work on a follow up series focused on Drupal upgrades. If you or your organization could sponsor such endeavor, please reach out to us via the site’s contact form.

What about upgrading to Drupal 9?

In March 2017, project lead Dries Buytaert announced a plan to make Drupal upgrades easier forever. This was reinforced during his keynote at DrupalCon Seattle 2019. You can watch the video recording in this link. In short, Drupal 9.0 will be the latest point release of Drupal 8 minus deprecated APIs. This has very important implications:

  • When Drupal 9 is released, the Migrate API should be mostly the same as Drupal 8. Therefore, anything that you learn today will be useful for Drupal 9 as well.
  • As long as your code does not use deprecated APIs, upgrading from Drupal 8 to Drupal 9 will be as easy as updating from Drupal 8.7 to 8.8.
  • Because of this, there is no need to wait for Drupal 9 to upgrade your Drupal 6 or 7 site. You can upgrade to Drupal 8 today.

Thank you!

And that concludes the #31DaysOfMigration series. For joining us in this learning experience, thank you very much! ¡Muchas gracias! Merci beaucoup! :-D We are also very grateful to Agaric.coop, Drupalize.Me, and Centarro.io for sponsoring this series.

What did you learn in today’s blog post? Did you know the upgrade process is able to copy content and configuration? Did you know that you can execute the upgrade procedure either from the user interface or the command line? Share your answers in the comments. Also, we would be grateful if you shared this blog post with others.

This blog post series, cross-posted at UnderstandDrupal.com as well as here on Agaric.coop, is made possible thanks to these generous sponsors: Drupalize.me by Osio Labs has online tutorials about migrations, among other topics, and Agaric provides migration trainings, among other services.  Contact Understand Drupal if your organization would like to support this documentation project, whether it is the migration series or other topics.

Sep 13 2019
Sep 13

When one starts working with migrations, it is easy to be overwhelmed by so many modules providing migration functionality. Throughout the series, we presented many of them trying to cover one module at a time. We did this to help the reader understand when a particular module is truly needed and why. But we only scratched the surface. Today’s article presents a list of migration related Drupal modules, all good for Drupal 8, for quick reference. Let’s get started.

Core modules

At the time of this writing, Drupal core ships with four migration modules:

  • Migrate: provides the base API for migrating data.
  • Migrate Drupal: offers functionality to migrate from other Drupal installations. It serves as the foundation for upgrades from Drupal 6 and 7. It also supports reading configuration entities from Drupal 8 sites.
  • Drupal Migrate UI: provides a user interface for upgrading a Drupal 6 or 7 site to Drupal 8.
  • Migrate Drupal Multilingual: is an experimental module required by multilingual translations. When they become stable, the module will be removed from Drupal core. See this article for more information.

Migration runners

Once the migration definition files have been created, there are many options to execute them:

Source plugins

The Migrate API offers many options to fetch data from:

Destination plugins

The Migrate API is mostly used to move data into Drupal, but it is possible to write to other destinations:

  • Migrate (core): provides classes for creating content and configuration entities. It also offers the `null` plugin which in itself does not write to anything. It is used in multilingual migrations for entity references.
  • Migrate Plus: provides the `table` plugin for migrating into tables not registered with Drupal Schema API.
  • CSV file: example destination plugin implementation to write CSV files. The module was created by J Franks for a DrupalCamp presentation. Check out the repository and video recording.

Development related

These modules can help with writing Drupal migrations:

Field and module related

Modules created by Tess Flynn (socketwench)

While doing the research for this article, we found many useful modules created by Tess Flynn (socketwench). She is a fantastic presenter who also has written about Drupal migrations, testing, and much more. Here are some of her modules:

Miscellaneous

  • Feeds Migrate: it aims to provide a user interface similar to the one from Drupal 7’s Feeds module, but working on top of Drupal 8’s Migrate API.
  • Migrate Override: allows flagging fields in a content entity so they can be manually changed by side editors without being overridden in a subsequent migration.
  • Migrate Status: checks if migrations are currently running.
  • Migrate QA: provides tools for validating content migrations. See this presentation for more details.

And many, many more modules!

What did you learn in today’s blog post? Did you find a new module that could be useful in current or future projects? Did we miss a module that has been very useful to you? Share your answers in the comments. Also, I would be grateful if you shared this blog post with others.

Next: Introduction to Drupal 8 upgrades

This blog post series, cross-posted at UnderstandDrupal.com as well as here on Agaric.coop, is made possible thanks to these generous sponsors: Drupalize.me by Osio Labs has online tutorials about migrations, among other topics, and Agaric provides migration trainings, among other services.  Contact Understand Drupal if your organization would like to support this documentation project, whether it is the migration series or other topics.

Sign up to be notified when Agaric gives a migration training:

Sep 12 2019
Sep 12

In recent articles, we have presented some recommendations and tools to debug Drupal migrations. Using a proper debugger is definitely the best way to debug Drupal be it migrations or other substems. In today’s article, we are going to learn how to configure XDebug inside DrupalVM to connect to PHPStorm. First, via the command line using Drush commands. And then, via the user interface using a browser. Let’s get started.

Important: User interfaces tend to change. Screenshots and referenced on-screen text might differ in new versions of the different tools. They can also vary per operating system. This article uses menu items from Linux. Refer the the official DrupalVM documentation for detailed installation and configuration instructions. For this article, it is assumed that VirtualBox, Vagrant, and Ansible are already installed. If you need help with those, refer to the DrupalVM’s installation guide.

Getting DrupalVM

First, get a copy of DrupalVM by cloning the repository or downloading a ZIP or TAR.GZ file from the available releases. If you downloaded a compressed file, expand it to have access to the configuration files. Before creating the virtual machine, make a copy of default.config.yml into a new file named config.yml. The latter will be used by DrupalVM to configure the virtual machine (VM). In this file, make the following changes:

# config.yml file
# Based off default.config.yml
vagrant_hostname: migratedebug.test
vagrant_machine_name: migratedebug
# For dynamic IP assignment the 'vagrant-auto_network' plugin is required.
# Otherwise, use an IP address that has not been used by any other virtual machine.
vagrant_ip: 0.0.0.0

# All the other extra packages can remain enabled.
# Make sure the following three get installed by uncommenting them.
installed_extras:
  - drupalconsole
  - drush
  - xdebug

php_xdebug_default_enable: 1
php_xdebug_cli_disable: no

The vagrant_hostname is the URL you will enter in your browser’s address bar to access the Drupal installation. Set vagrant_ip to an IP that has not been taken by another virtual machine. If you are unsure, you can set the value to 0.0.0.0 and install the vagrant-auto_network plugin. The plugin will make sure that an available IP is assigned to the VM. In the installed_extras section, uncomment xdebug and drupalconsole. Drupal Console is not necessary for getting XDebug to work, but it offers many code introspection tools that are very useful for Drupal debugging in general. Finally, set php_xdebug_default_enable to 1 and php_xdebug_cli_disable to no. These last two settings are very important for being able to debug Drush commands.

Then, open a terminal and change directory to where the DrupalVM files are located. Keep the terminal open are going to execute various commands from there. Start the virtual machine by executing vagrant up. If you had already created the VM, you can still make changes to the config.yml file and then reprovision. If the virtual machine is running, execute the command: vagrant provision. Otherwise, you can start and reprovision the VM in a single command: vagrant up --provision. Finally, SSH into the VM executing vagrant ssh.

By default, DrupalVM will use the Drupal composer template project to get a copy of Drupal. That means that you will be managing your module and theme dependencies using composer. When you SSH into the virtual machine, you will be in the /var/www/drupalvm/drupal/web directory. That is Drupal’s docroot. The composer file that manages the installation is actually one directory up. Normally, if you run a composer command from a directory that does not have a composer.json file, composer will try to find one up in the directory hierarchy. Feel free to manually go one directory up or rely on composer’s default behaviour to locate the file.

For good measure, let’s install some contributed modules. Inside the virtual machine, in Drupal’s docroot, execute the following command: composer require drupal/migrate_plus drupal/migrate_tools. You can also create directory in /var/www/drupalvm/drupal/web/modules/custom and place the custom module we have been working on throughout the series. You can get it at https://github.com/dinarcon/ud_migrations.

To make sure things are working, let’s enable one example modules by executing: drush pm-enable ud_migrations_config_entity_lookup_entity_generate. This module comes with one migration: udm_config_entity_lookup_entity_generate_node. If you execute drush migrate:status the example migration should be listed.

Configuring PHPStorm

With Drupal already installed and the virtual machine running, let’s configure PHPStorm. Start a new project pointing to the DrupalVM files. Feel free to follow your preferred approach to project creation. For reference, one way to do it is by going to "Files > Create New Project from Existing Files". In the dialog, select "Source files are in a local directory, no web server is configured yet." and click next. Look for the DrupalVM directory, click on it, click on “Project Root”, and then “Finish”. PHPStorm will begin indexing the files and detect that it is a Drupal project. It will prompt you to enable the Drupal coding standards, indicate which directory contains the installation path, and if you want to set PHP include paths. All of that is optional but recommended, especially if you want to use this VM for long term development.

Now the important part. Go to “Files > Settings > Language and Frameworks > PHP”. In the panel, there is a text box labeled “CLI Interpreter”. In the right end, there is a button with three dots like an ellipsis (...). The next step requires that the virtual machine is running because PHPStorm will try to connect to it. After verifying that it is the case, click the plus (+) button at the top left corner to add a CLI Interpreter. From the list that appears, select “From Docker, Vagrant, VM, Remote...”. In the “Configure Remote PHP Interpreter” dialog select “Vagrant”. PHPStorm will detect the SSH connection to connect to the virtual machine. Click “OK” to close the multiple dialog boxes. When you go back to the “Languages & Frameworks” dialog, you can set the “PHP language level” to match the same version from the Remote CLI Interpreter.

Remote PHP interpreter.

CLI interpreters.

You are almost ready to start debugging. There are a few things pending to do. First, let’s create a breakpoint in the import method of the MigrateExecutable class. You can go to “Navigate > Class” to the project by class name. Or click around in the Project structure until you find the class. It is located at ./drupal/web/core/modules/migrate/src/MigrateExecutable.php in the VM directory. You can add a breakpoint by clicking on the bar to the left of the code area. A red circle will appear, indicating that the breakpoint has been added.

Then, you need to instruct PHPStorm to listen for debugging connections. For this, click on “Run > Start Listening for PHP Debugging Connections”. Finally, you have to set some server mappings. For this you will need the IP address of the virtual machine. If you configured the VM to assign the IP dynamically, you can skip this step momentarily. PHPStorm will detect the incoming connection, create a server with the proper IP, and then you can set the path mappings.

Triggering the breakpoint

Let’s switch back to the terminal. If you are not inside the virtual machine, you can SSH into the VM executing vagrant ssh. Then, execute the following command (everything in one line):

XDEBUG_CONFIG="idekey=PHPSTORM" /var/www/drupalvm/drupal/vendor/bin/drush migrate:import udm_config_entity_lookup_entity_generate_node

For the breakpoint to be triggered, the following needs to happen:

  • You must execute Drush from the vendor directory. DrupalVM has a globally available Drush binary located at /usr/local/bin/drush. That is not the one to use. For debugging purposes, always execute Drush from the vendor directory.
  • The command needs to have XDEBUG_CONFIG environment variable set to “idekey=PHPSTORM”. There are many ways to accomplish this, but prepending the variable as shown in the example is a valid way to do it.
  • Because the breakpoint was set in the import method, we need to execute an import command to stop at the breakpoint. The migration in the example Drush command is part of the example module that was enabled earlier.

When the command is executed, a dialog will appear in PHPStorm. In it, you will be asked to select a project or a file to debug. Accept what is selected by default for now.  By accepting the prompt, a new server will be configured using the proper IP of the virtual machine.  After doing so, go to “Files > Settings > Language and Frameworks > PHP > Servers”. You should see one already created. Make sure the “Use path mappings” option is selected. Then, look for the direct child of “Project files”. It should be the directory in your host computer where the VM files are located. In that row, set the “Absolute path on the server” column to  /var/www/drupalvm. You can delete any other path mapping. There should only be one from the previous prompt. Now, click “OK” in the dialog to accept the changes.

Incoming Drush connection.

Path mappings

Finally, run the Drush command from inside the virtual machine once more. This time the program execution should stop at the breakpoint. You can use the Debug panel to step over each line of code and see how the variables change over time. Feel free to add more breakpoints as needed. In the previous article, there are some suggestions about that. When you are done, let PHPStorm know that it should no longer listen for connections. For that, click on “Run > Stop Listening for PHP Debugging Connections”. And that is how you can debug Drush commands for Drupal migrations.

Path mappings.

Debugging from the user interface

If you also want to be able to debug from the user interface, go to this URL and generate the bookmarklets for XDebug: https://www.jetbrains.com/phpstorm/marklets/ The IDE Key should be PHPSTORM. When the bookmarklets are created, you can drag and drop them into your browser’s bookmarks toolbar. Then, you can click on them to start and stop a debugging session. The IDE needs to be listening for incoming debugging connections as it was the case for Drush commands.

PHPStorm bookmarlets generator

Note: There are browser extensions that let you start and stop debugging sessions. Check the extensions repository of your browser to see which options are available.

Finally, set breakpoints as needed and go to a page that would trigger them. If you are following along with the example, you can go to http://migratedebug.test/admin/structure/migrate/manage/default/migrations/udm_config_entity_lookup_entity_generate_node/execute Once there, select the “Import” operation and click the “Execute” button. This should open a prompt in PHPStorm to select a project or a file to debug. Select the index.php located in Drupal’s docroot. After accepting the connection, a new server should be configured with the proper path mappings. At this point, you should hit the breakpoint again.

Incoming web connections.

Happy debugging! :-)

What did you learn in today’s blog post? Did you know how to debug Drush commands? Did you know how to trigger a debugging session from the browser? Share your answers in the comments. Also, I would be grateful if you shared this blog post with others.

This blog post series, cross-posted at UnderstandDrupal.com as well as here on Agaric.coop, is made possible thanks to these generous sponsors: Drupalize.me by Osio Labs has online tutorials about migrations, among other topics, and Agaric provides migration trainings, among other services.  Contact Understand Drupal if your organization would like to support this documentation project, whether it is the migration series or other topics.

Sep 12 2019
Sep 12

This blog has been re-posted and edited with permission from Dries Buytaert's blog.

An in-depth analysis of how Drupal's development was sponsored between July 1, 2018 and June 30, 2019.

The past years, I've examined Drupal.org's contribution data to understand who develops Drupal, how diverse the Drupal community is, how much of Drupal's maintenance and innovation is sponsored, and where that sponsorship comes from.

You can look at the 2016 report, the 2017 report, and the 2018 report. Each report looks at data collected in the 12-month period between July 1st and June 30th.

This year's report shows that:

  • Both the recorded number of contributors and contributions have increased.
  • Most contributions are sponsored, but volunteer contributions remains very important to Drupal's success.
  • Drupal's maintenance and innovation depends mostly on smaller Drupal agencies and Acquia. Hosting companies, multi-platform digital marketing agencies, large system integrators and end users make fewer contributions to Drupal.
  • Drupal's contributors have become more diverse, but are still not diverse enough.

Methodology

What are Drupal.org issues?

"Issues" are pages on Drupal.org. Each issue tracks an idea, feature request, bug report, task, or more. See https://www.drupal.org/project/issues for the list of all issues.

For this report, we looked at all Drupal.org issues marked "closed" or "fixed" in the 12-month period from July 1, 2018 to June 30, 2019. The issues analyzed in this report span Drupal core and thousands of contributed projects, across all major versions of Drupal.

What are Drupal.org credits?

In the spring of 2015, after proposing initial ideas for giving credit, Drupal.org added the ability for people to attribute their work in the Drupal.org issues to an organization or customer, or mark it the result of volunteer efforts.

Example issue credit on drupal org

A screenshot of an issue comment on Drupal.org. You can see that jamadar worked on this patch as a volunteer, but also as part of his day job working for TATA Consultancy Services on behalf of their customer, Pfizer.

Drupal.org's credit system is truly unique and groundbreaking in Open Source and provides unprecedented insights into the inner workings of a large Open Source project. There are a few limitations with this approach, which we'll address at the end of this report.

What is the Drupal community working on?

In the 12-month period between July 1, 2018 and June 30, 2019, 27,522 issues were marked "closed" or "fixed", a 13% increase from the 24,447 issues in the 2017-2018 period.

In total, the Drupal community worked on 3,474 different Drupal.org projects this year compared to 3,229 projects in the 2017-2018 period — an 8% year over year increase.

The majority of the credits are the result of work on contributed modules:

A pie chart showing contributions by project type: most contributions are to contributed modules.

Compared to the previous period, contribution credits increased across all project types:

A graph showing the year over year growth of contributions per project type.

The most notable change is the large jump in "non-product credits": more and more members in the community started tracking credits for non-product activities such as organizing Drupal events (e.g. DrupalCamp Delhi projectDrupal Developer DaysDrupal Europe and DrupalCon Europe), promoting Drupal (e.g. Drupal pitch deck or community working groups (e.g. Drupal Diversity and Inclusion Working GroupGovernance Working Group).

While some of these increases reflect new contributions, others are existing contributions that are newly reported. All contributions are valuable, whether they're code contributions, or non-product and community-oriented contributions such as organizing events, giving talks, leading sprints, etc. The fact that the credit system is becoming more accurate in recognizing more types of Open Source contribution is both important and positive.

Who is working on Drupal?

For this report's time period, Drupal.org's credit system received contributions from 8,513 different individuals and 1,137 different organizations — a meaningful increase from last year's report.

A graph showing that the number of individual and organizational contributors increased year over year.

Consistent with previous years, approximately 51% of the individual contributors received just one credit. Meanwhile, the top 30 contributors (the top 0.4%) account for 19% of the total credits. In other words, a relatively small number of individuals do the majority of the work. These individuals put an incredible amount of time and effort into developing Drupal and its contributed projects:

Rank Username Issues 1 kiamlaluno 1610 2 jrockowitz 756 3 alexpott 642 4 RajabNatshah 616 5 volkswagenchick 519 6 bojanz 504 7 alonaoneill 489 8 thalles 488 9 Wim Leers 437 10 DamienMcKenna 431 11 Berdir 424 12 chipway 356 13 larowlan 324 14 pifagor 320 15 catch 313 16 mglaman 277 17 adci_contributor 274 18 quietone 266 19 tim.plunkett 265 20 gaurav.kapoor 253 21 RenatoG 246 22 heddn 243 23 chr.fritsch 241 24 xjm 238 25 phenaproxima 238 26 mkalkbrenner 235 27 gvso 232 28 dawehner 219 29 e0ipso 218 30 drumm 205

Out of the top 30 contributors featured this year, 28 were active contributors in the 2017-2018 period as well. These Drupalists' dedication and continued contribution to the project has been crucial to Drupal's development.

It's also important to recognize that most of the top 30 contributors are sponsored by an organization. Their sponsorship details are provided later in this article. We value the organizations that sponsor these remarkable individuals, because without their support, it could be more challenging for these individuals to be in the top 30.

It's also nice to see two new contributors make the top 30 this year — Alona O'neill with sponsorship from Hook 42 and Thalles Ferreira with sponsorship from CI&T. Most of their credits were the result of smaller patches (e.g. removing deprecated code, fixing coding style issues, etc) or in some cases non-product credits rather than new feature development or fixing complex bugs. These types of contributions are valuable and often a stepping stone towards towards more in-depth contribution.

How much of the work is sponsored?

Issue credits can be marked as "volunteer" and "sponsored" simultaneously (shown in jamadar's screenshot near the top of this post). This could be the case when a contributor does the necessary work to satisfy the customer's need, in addition to using their spare time to add extra functionality.

For those credits with attribution details, 18% were "purely volunteer" credits (8,433 credits), in stark contrast to the 65% that were "purely sponsored" (29,802 credits). While there are almost four times as many "purely sponsored" credits as "purely volunteer" credits, volunteer contribution remains very important to Drupal.

Contributions by volunteer vs sponsored

Both "purely volunteer" and "purely sponsored" credits grew — "purely sponsored" credits grew faster in absolute numbers, but for the first time in four years "purely volunteer" credits grew faster in relative numbers.

The large jump in volunteer credits can be explained by the community capturing more non-product contributions. As can be seen on the graph below, these non-product contributions are more volunteer-centric.

A graph showing how much of the contributions are volunteered vs sponsored.

Who is sponsoring the work?

Now that we've established that the majority of contributions to Drupal are sponsored, let's study which organizations contribute to Drupal. While 1,137 different organizations contributed to Drupal, approximately 50% of them received four credits or less. The top 30 organizations (roughly the top 3%) account for approximately 25% of the total credits, which implies that the top 30 companies play a crucial role in the health of the Drupal project.

Top conytinuying organizations

Top contributing organizations based on the number of issue credits.

While not immediately obvious from the graph above, a variety of different types of companies are active in Drupal's ecosystem:

Category Description Traditional Drupal businesses Small-to-medium-sized professional services companies that primarily make money using Drupal. They typically employ fewer than 100 employees, and because they specialize in Drupal, many of these professional services companies contribute frequently and are a huge part of our community. Examples are Hook42, Centarro, The Big Blue House, Vardot, etc. Digital marketing agencies Larger full-service agencies that have marketing-led practices using a variety of tools, typically including Drupal, Adobe Experience Manager, Sitecore, WordPress, etc. They tend to be larger, with many of the larger agencies employing thousands of people. Examples are Wunderman, Possible and Mirum. System integrators Larger companies that specialize in bringing together different technologies into one solution. Example system agencies are Accenture, TATA Consultancy Services, Capgemini and CI&T. Hosting companies Examples are Acquia, Rackspace, Pantheon and Platform.sh. End users Examples are Pfizer or bio.logis Genetic Information Management GmbH.

A few observations:

  • Almost all of the sponsors in the top 30 are traditional Drupal businesses with fewer than 50 employees. Only five companies in the top 30 — Pfizer, Google, CI&T, bio.logis and Acquia — are not traditional Drupal businesses. The traditional Drupal businesses are responsible for almost 80% of all the credits in the top 30. This percentage goes up if you extend beyond the top 30. It's fair to say that Drupal's maintenance and innovation largely depends on these traditional Drupal businesses.
  • The larger, multi-platform digital marketing agencies are barely contributing to Drupal. While more and more large digital agencies are building out Drupal practices, no digital marketing agencies show up in the top 30, and hardly any appear in the entire list of contributing organizations. While they are not required to contribute, I'm frustrated that we have not yet found the right way to communicate the value of contribution to these companies. We need to incentivize each of these firms to contribute back with the same commitment that we see from traditional Drupal businesses
  • The only system integrator in the top 30 is CI&T, which ranked 4th with 795 credits. As far as system integrators are concerned, CI&T is a smaller player with approximately 2,500 employees. However, we do see various system integrators outside of the top 30, including Globant, Capgemini, Sapient and TATA Consultancy Services. In the past year, Capgemini almost quadrupled their credits from 46 to 196, TATA doubled its credits from 85 to 194, Sapient doubled its credits from 28 to 65, and Globant kept more or less steady with 41 credits. Accenture and Wipro do not appear to contribute despite doing a fair amount of Drupal work in the field.
  • Hosting companies also play an important role in our community, yet only Acquia appears in the top 30. Rackspace has 68 credits, Pantheon has 43, and Platform.sh has 23. I looked for other hosting companies in the data, but couldn't find any. In general, there is a persistent problem with hosting companies that make a lot of money with Drupal not contributing back. The contribution gap between Acquia and other hosting companies has increased, not decreased.
  • We also saw three end users in the top 30 as corporate sponsors: Pfizer (453 credits), Thunder (659 credits, up from 432 credits the year before), and the German company, bio.logis (330 credits). A notable end user is Johnson & Johnson, who was just outside of the top 30, with 221 credits, up from 29 credits the year before. Other end users outside of the top 30, include the European Commission (189 credits), Workday (112 credits), Paypal (80 credits), NBCUniversal (48 credits), Wolters Kluwer (20 credits), and Burda Media (24 credits). We also saw contributions from many universities, including the University of British Columbia (148 credits), University of Waterloo (129 credits), Princeton University (73 credits), University of Austin Texas at Austin (57 credits), Charles Darwin University (24 credits), University of Edinburgh (23 credits), University of Minnesota (19 credits) and many more.

A graph showing that Acquia is by far the number one contributing hosting company.

Contributions by system integrators

It would be interesting to see what would happen if more end users mandated contributions from their partners. Pfizer, for example, only works with agencies that contribute back to Drupal, and uses Drupal's credit system to verify their vendors' claims. The State of Georgia started doing the same; they also made Open Source contribution a vendor selection criteria. If more end users took this stance, it could have a big impact on the number of digital agencies, hosting companies and system integrators that contribute to Drupal.

While we should encourage more organizations to sponsor Drupal contributions, we should also understand and respect that some organizations can give more than others and that some might not be able to give back at all. Our goal is not to foster an environment that demands what and how others should give back. Instead, we need to help foster an environment worthy of contribution. This is clearly laid out in Drupal's Values and Principles.

How diverse is Drupal?

Supporting diversity and inclusion within Drupal is essential to the health and success of the project. The people who work on Drupal should reflect the diversity of people who use and work with the web.

I looked at both the gender and geographic diversity of Drupal.org contributors. While these are only two examples of diversity, these are the only diversity characteristics we currently have sufficient data for. Drupal.org recently rolled out support for Big 8/Big 10, so next year we should have more demographics information

Gender diversity

The data shows that only 8% of the recorded contributions were made by contributors who do not identify as male, which continues to indicate a wide gender gap. This is a one percent increase compared to last year. The gender imbalance in Drupal is profound and underscores the need to continue fostering diversity and inclusion in our community.

A graph showing contributions by gender: 75% of the contributions come from people who identify as male.

Last year I wrote a post called about the privilege of free time in Open Source. It made the case that Open Source is not a meritocracy, because not everyone has equal amounts of free time to contribute. For example, research shows that women still spend more than double the time as men doing unpaid domestic work, such as housework or childcare. This makes it more difficult for women to contribute to Open Source on an unpaid, volunteer basis. It's one of the reasons why Open Source projects suffer from a lack of diversity, among others including hostile environments and unconscious biases. Drupal.org's credit data unfortunately still shows a big gender disparity in contributions:

A graph that shows that compared to males, female contributors do more sponsored work, and less volunteer work.

Ideally, over time, we can collect more data on non-binary gender designations, as well as segment some of the trends behind contributions by gender. We can also do better at collecting data on other systemic issues beyond gender alone. Knowing more about these trends can help us close existing gaps. In the meantime, organizations capable of giving back should consider financially sponsoring individuals from underrepresented groups to contribute to Open Source. Each of us needs to decide if and how we can help give time and opportunities to underrepresented groups and how we can create equity for everyone in Drupal.

Geographic diversity

When measuring geographic diversity, we saw individual contributors from six continents and 114 countries:

A graph that shows most contributions come from Europe and North America.

Contributions per capita

Contribution credits per capita calculated as the amount of contributions per continent divided by the population of each continent. 0.001% means that one in 100,000 people contribute to Drupal. In North America, 5 in 100,000 people contributed to Drupal the last year.

Contributions from Europe and North America are both on the rise. In absolute terms, Europe contributes more than North America, but North America contributes more per capita.

Asia, South America and Africa remain big opportunities for Drupal, as their combined population accounts for 6.3 billion out of 7.5 billion people in the world. Unfortunately, the reported contributions from Asia are declining year over year. For example, compared to last year's report, there was a 17% drop in contribution from India. Despite that drop, India remains the second largest contributor behind the United States:

A graph showing the top 20 contributing countries.

The top 20 countries from which contributions originate. The data is compiled by aggregating the countries of all individual contributors behind each issue. Note that the geographical location of contributors doesn't always correspond with the origin of their sponsorship. Wim Leers, for example, works from Belgium, but his funding comes from Acquia, which has the majority of its customers in North America.

Top contributor details

To create more awareness of which organizations are sponsoring the top individual contributors, I included a more detailed overview of the top 50 contributors and their sponsors. If you are a Drupal developer looking for work, these are some of the companies I'd apply to first. If you are an end user looking for a company to work with, these are some of the companies I'd consider working with first. Not only do they know Drupal well, they also help improve your investment in Drupal.

Rank Username Issues Volunteer Sponsored Not specified Sponsors 1 kiamlaluno 1610 99% 0% 1% 2 jrockowitz 756 98% 99% 0% The Big Blue House (750), Memorial Sloan Kettering Cancer Center (5), Rosewood Marketing (1) 3 alexpott 642 6% 80% 19% Thunder (336), Acro Media Inc (100), Chapter Three (77) 4 RajabNatshah 616 1% 100% 0% Vardot (730), Webship (2) 5 volkswagenchick 519 2% 99% 0% Hook 42 (341), Kanopi Studios (171) 6 bojanz 504 0% 98% 2% Centarro (492), Ny Media AS (28), Torchbox (5), Liip (2), Adapt (2) 7 alonaoneill 489 9% 99% 0% Hook 42 (484) 8 thalles 488 0% 100% 0% CI&T (488), Janrain (3), Johnson & Johnson (2) 9 Wim Leers 437 8% 97% 0% Acquia (421), Government of Flanders (3) 10 DamienMcKenna 431 0% 97% 3% Mediacurrent (420) 11 Berdir 424 0% 92% 8% MD Systems (390) 12 chipway 356 0% 100% 0% Chipway (356) 13 larowlan 324 16% 94% 2% PreviousNext (304), Charles Darwin University (22), University of Technology, Sydney (3), Service NSW (2), Department of Justice & Regulation, Victoria (1) 14 pifagor 320 52% 100% 0% GOLEMS GABB (618), EPAM Systems (16), Drupal Ukraine Community (6) 15 catch 313 1% 95% 4% Third & Grove (286), Tag1 Consulting (11), Drupal Association (6), Acquia (4) 16 mglaman 277 2% 98% 1% Centarro (271), Oomph, Inc. (16), E.C. Barton & Co (3), Gaggle.net, Inc. (1), Bluespark (1), Thinkbean (1), LivePerson, Inc (1), Impactiv, Inc. (1), Rosewood Marketing (1), Acro Media Inc (1) 17 adci_contributor 274 0% 100% 0% ADCI Solutions (273) 18 quietone 266 41% 75% 1% Acro Media Inc (200) 19 tim.plunkett 265 3% 89% 9% Acquia (235) 20 gaurav.kapoor 253 0% 51% 49% OpenSense Labs (129), DrupalFit (111) 21 RenatoG 246 0% 100% 0% CI&T (246), Johnson & Johnson (85) 22 heddn 243 2% 98% 2% MTech, LLC (202), Tag1 Consulting (32), European Commission (22), North Studio (3), Acro Media Inc (2) 23 chr.fritsch 241 0% 99% 1% Thunder (239) 24 xjm 238 0% 85% 15% Acquia (202) 25 phenaproxima 238 0% 100% 0% Acquia (238) 26 mkalkbrenner 235 0% 100% 0% bio.logis Genetic Information Management GmbH (234), OSCE: Organization for Security and Co-operation in Europe (41), Welsh Government (4) 27 gvso 232 0% 100% 0% Google Summer of Code (214), Google Code-In (16), Zivtech (1) 28 dawehner 219 39% 84% 8% Chapter Three (176), Drupal Association (5), Tag1 Consulting (3), TES Global (1) 29 e0ipso 218 99% 100% 0% Lullabot (217), IBM (23) 30 drumm 205 0% 98% 1% Drupal Association (201) 31 gabesullice 199 0% 100% 0% Acquia (198), Aten Design Group (1) 32 amateescu 194 0% 97% 3% Pfizer, Inc. (186), Drupal Association (1), Chapter Three (1) 33 klausi 193 2% 59% 40% jobiqo - job board technology (113) 34 samuel.mortenson 187 42% 42% 17% Acquia (79) 35 joelpittet 187 28% 78% 14% The University of British Columbia (146) 36 borisson_ 185 83% 50% 3% Calibrate (79), Dazzle (13), Intracto digital agency (1) 37 Gábor Hojtsy 184 0% 97% 3% Acquia (178) 38 adriancid 182 91% 22% 2% Drupiter (40) 39 eiriksm 182 0% 100% 0% Violinist (178), Ny Media AS (4) 40 yas 179 12% 80% 10% DOCOMO Innovations, Inc. (143) 41 TR 177 0% 0% 100% 42 hass 173 1% 0% 99% 43 Joachim Namyslo 172 69% 0% 31% 44 alex_optim 171 0% 99% 1% GOLEMS GABB (338) 45 flocondetoile 168 0% 99% 1% Flocon de toile (167) 46 Lendude 168 52% 99% 0% Dx Experts (91), ezCompany (67), Noctilaris (9) 47 paulvandenburg 167 11% 72% 21% ezCompany (120) 48 voleger 165 98% 98% 2% GOLEMS GABB (286), Lemberg Solutions Limited (36), Drupal Ukraine Community (1) 49 lauriii 164 3% 98% 1% Acquia (153), Druid (8), Lääkärikeskus Aava Oy (2) 50 idebr 162 0% 99% 1% ezCompany (156), One Shoe (5)

Limitations of the credit system

It is important to note a few of the current limitations of Drupal.org's credit system:

  • The credit system doesn't capture all code contributions. Parts of Drupal are developed on GitHub rather than Drupal.org, and often aren't fully credited on Drupal.org. For example, Drush is maintained on GitHub instead of Drupal.org, and companies like Pantheon don't get credit for that work. The Drupal Association is working to integrate GitLab with Drupal.org. GitLab will provide support for "merge requests", which means contributing to Drupal will feel more familiar to the broader audience of Open Source contributors who learned their skills in the post-patch era. Some of GitLab's tools, such as in-line editing and web-based code review will also lower the barrier to contribution, and should help us grow both the number of contributions and contributors on Drupal.org.
  • The credit system is not used by everyone. There are many ways to contribute to Drupal that are still not captured in the credit system, including things like event organizing or providing support. Technically, that work could be captured as demonstrated by the various non-product initiatives highlighted in this post. Because using the credit system is optional, many contributors don't. As a result, contributions often have incomplete or no contribution credits. We need to encourage all Drupal contributors to use the credit system, and raise awareness of its benefits to both individuals and organizations. Where possible, we should automatically capture credits. For example, translation efforts on https://localize.drupal.org are not currently captured in the credit system but could be automatically.
  • The credit system disincentives work on complex issues. We currently don't have a way to account for the complexity and quality of contributions; one person might have worked several weeks for just one credit, while another person might receive a credit for 10 minutes of work. We certainly see a few individuals and organizations trying to game the credit system. In the future, we should consider issuing credit data in conjunction with issue priority, patch size, number of reviews, etc. This could help incentivize people to work on larger and more important problems and save smaller issues such as coding standards improvements for new contributor sprints. Implementing a scoring system that ranks the complexity of an issue would also allow us to develop more accurate reports of contributed work.

All of this means that the actual number of contributions and contributors could be significantly higher than what we report.

Like Drupal itself, the Drupal.org credit system needs to continue to evolve. Ultimately, the credit system will only be useful when the community uses it, understands its shortcomings, and suggests constructive improvements.

A first experiment with weighing credits

As a simple experiment, I decided to weigh each credit based on the adoption of the project the credit is attributed to. For example, each contribution credit to Drupal core is given a weight of 11 because Drupal core has about 1,1 million active installations. Credits to the Webform module, which has over 400,000 installations, get a weight of 4. And credits to Drupal's Commerce project gets just 1 point as it is installed on fewer than 100,000 sites.

The idea is that these weights capture the end user impact of each contribution, but also act as a proxy for the effort required to get a change committed. Getting a change accepted in Drupal core is both more difficult and more impactful than getting a change accepted to Commerce project.

This weighting is far from perfect as it undervalues non-product contributions, and it still doesn't recognize all types of product contributions (e.g. product strategy work, product management work, release management work, etc). That said, for code contributions, it may be more accurate than a purely unweighted approach.

Top contributing individuals based on weighted credits.

The top 30 contributing individuals based on weighted Drupal.org issue credits.

Top contributing organizations based on weighted credits.

The top 30 contributing organizations based on weighted Drupal.org issue credits.

Conclusions

Our data confirms that Drupal is a vibrant community full of contributors who are constantly evolving and improving the software. It's amazing to see that just in the last year, Drupal welcomed more than 8,000 individuals contributors and over 1,100 corporate contributors. It's especially nice to see the number of reported contributions, individual contributors and organizational contributors increase year over year.

To grow and sustain Drupal, we should support those that contribute to Drupal and find ways to get those that are not contributing involved in our community. Improving diversity within Drupal is critical, and we should welcome any suggestions that encourage participation from a broader range of individuals and organizations.

Sep 12 2019
Sep 12

The Hub - No Question It Can't Answer

The Bloomington Public Schools District in Bloomington, MN is a leader in harnessing technology to support student learning. We’ve been working with them since 2012, when we helped them move their K-12 site from a proprietary content management system that was expensive, inflexible and not mobile-friendly to a Drupal-powered site that provided more control over the site structure and content. We also added multi-site capabilities for the district’s 17 schools and departments, so they could manage their own content while retaining the broader district branding.

BPS has always been an early adopter of technology for school systems. However, back in 2012, there weren’t any student information systems like we have today; schools were still using paper planners to track homework. So they challenged us: build an online portal for teachers, students and their parents to access school data that was only accessible internally. The goal was to surface the data and make it easily consumable by everyone.

Nothing like this existed at the time, even in the Drupal world. The TEN7 development team wrote custom code (effectively creating original Drupal modules) to accomplish the task. The end result was a separate password-protected web application called “The Hub.” The Hub compiled data from third-party sources such as TIES, hundreds of Google calendars of students and teachers, data from school and district websites, as well as the district’s internal database.

The first release of The Hub was simple: a dashboard showing the students' class schedules and news feeds. However, the secure portal soon became more than a homework planning resource—The Hub has grown into a student, school and district information portal.

Bloomington Public Schools

New Features

  • Pathways to Graduation graph, which allow students (and their parents) to track their progress toward their post-secondary goals
    Bloomington Public Schools Pathways to Graduation
  • Data Warehouse, which holds lots of student info, such as student testing results, and even health records
    Bloomington Public Schools HUB Grades
    Bloomington Public Schools Hub Health Records
  • The Hub Digest emails, which give students and parents an overview of the upcoming week in their classes
  • Notes, which lets students, parents and teachers notes to any event or activity in the Hub
  • Students can add their own calendar events

“We have weekly meetings with TEN7, and they’re very innovative in coming up with creative solutions for the crazy ideas and problems we bring them. We serve 14,000 parents, almost 11,000 students and 1500 staff members—we have a huge backlog of crazy ideas. There’s always more we can give our stakeholders. With off-the-shelf software, you get what you get. But because we can do whatever we want with The Hub, it’s become the solution for lots of gnarly problems in our system. There are always more gnarly problems to solve and The Hub is almost always the answer.”
—Katrina Mezera, Digital Learning & Data Manager

The Hub is considered an indispensable tool by parents, students and staff. On any given day, thousands of students (up to 50% of the student body) use The Hub. Other school districts have reviewed and admired the functionality of the portal. Bloomington Public Schools presented The Hub at the 2015 Google in Education Conference in Mankato, MN, and other school districts inquired about having the same system implemented for their district.

TEN7 continues to collaborate with Bloomington Public Schools to add features and refine the user experience.

Sep 12 2019
Sep 12

The profession of building websites has seen many changes in the last few years. SEO, website performance, multi-screen responsiveness, and accessibility are no longer luxuries. On top of that, tools have emerged that have improved the development experience and simplified scalability. Finding a modern CMS or framework that can incorporate ALL of these considerations is difficult. Especially when the flexibility to be able to create unique websites is also important. This is where Drupal 8 outshines other frameworks.

Here is a list of major benefits that you are missing out on if your website is built in Drupal 7 (or below), WordPress, or most other common content management systems.

1. Website Accessibility, Security, and Scalability  

Accessibility - One of the major changes in web development these days is the standard of building sites that are easily accessible by all visitors no matter their abilities or disabilities. For some clients, this is no longer a luxury but a regulation. For others, there is no law, but they can be opening themselves up to discrimination lawsuits by not making an effort to build a site that is accessible. A well-implemented Drupal 8 site can automate many aspects of accessibility and facilitate best practices in maintaining the accessibility of the site. Compliance will also help your website gain points with major search engines.

Security - One reason that major companies and government agencies move to Drupal is because of its fast security updates, and the large community of experienced developers who stand behind it. Drupal 8’s adoption of standard technologies, such as Composer, makes it a lot easier to keep your site up to date and secure. This also means less time fixing the code and more time improving and adding new features.

Scalability - In the past whenever there was a new major release of Drupal, version 6 to 7, it wasn’t just an update. It really meant a full install of the new version of the Drupal core, then maliciously and painfully rebuilding and migrating custom code, configurations, users, and all of the content from the old site to the new site. In other words, hundreds of hours of rebuilding and months before the new site was ready. After Drupal 8, that will no longer be a problem. Drupal 8 was built to allow major updates without having to rebuild the site. Three to five years down the road, all you may need is a fresh facelift to have your website up to date with new trends. So taking the time to architect a well built Drupal 8 site will pay off in the long run.

2. Drupal 8 as API first

Drupal 8 was created as an API (Application Programming Interface), making it more powerful than ever. It now has the ability to skip the generation, all the HTML markup and simply serving the data required to be implemented on the front end of the website, allowing other technologies like javascript to serve dynamically generated pages specifically targeted to the user.

Best of all, it can be used to connect back and forth with other APIs to get any information you want to serve and share as a service. All of this is already built into Drupal 8 to serve any data without having to write one single line of code.

If you haven’t wrapped your mind around what this means yet. Let me give you a couple of examples of this: “Web Apps” & “Mobile apps”. These can be developed to use the same data living in Drupal 8. Your Drupal site becomes the backbone content management system without having to update multiple sources of data to keep things in sync. Think about it… your website, your mobile apps, your own intranet, and content from other places all managed in one place, making one perfect ecosystem.

3. Development  

Drupal 8 was completely recoded from scratch with all new improved code.  Many things that were a constant hassle in the older versions are now a breeze to build.

Migration of old to new: If you have a website with large amounts of content, for example, news, products, or blogs, setting up an automated migrating of all the data into Drupal 8 is possible from almost any source. Yes, that even means if your old website was built in WordPress.

One of my favorite development improvements for sure is Configuration Management. Now you are able to export and import configuration changes from a live site to your local development, or vice versa, without affecting content. That feature alone has cut hundreds of hours from my development workflow. This not only helps reduce the cost of overall development but it gives developers time to concentrate on getting the features built and not wasting time deploying already built features manually.

The implementation of Composer as the dependency manager to install and maintain modules, custom code, and patches, makes it super easy to create copies of your website in a development environment and automatically turn on all the development helper features by just having a few files on hand that are environment-aware. With a few simple commands to sync code and configurations, you are ready to develop and deploy in no time.

After a couple of years of developing in Drupal 8, I wonder how I was able to do without all the new development features for so long.

Drupal 8 is a dream-machine for large or small websites. All of these features make it quick to build a simple website and powerful enough to build the largest, most complex websites.  Drupal’s flexible and innovative system is only limited by your imagination.

Happy coding!

Pages

About Drupal Sun

Drupal Sun is an Evolving Web project. It allows you to:

  • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
  • Facet based on tags, author, or feed
  • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
  • View the entire article text inline, or in the context of the site where it was created

See the blog post at Evolving Web

Evolving Web