Continuous Integration for Automating Drupal Workflow

The Research and Development team at BBC (British Broadcasting Corporation) have been working on IP production for a number of years building a model for end-to-end broadcasting that will allow a live studio to run entirely on IP networks. During this period, several software applications and libraries have been built in order to prototype techniques, develop their understanding further and implement emerging standards. To do all these, they leverage Continuous Integration along with a number of tools to aid with Continuous Delivery of their software. Why is Continuous Integration a preferred option for large organisations like BBC?

A software development methodology like Continuous Integration (CI) can be of paramount importance for an efficient software delivery. Drupal-based projects can gain a lot with the implementation of CI leading to better teamwork and effective software development processes.

Predates many of the Agile’s ancestors

CI is older than many of the ancestors of agile development methodology. Grady Booch, an American software engineer, gave birth to the term ‘Continuous Integration’ through the Booch Method (a method used in object-oriented analysis and design) in 1991.
 
Booch, in his book called Object-Oriented Analysis and Design with Applications, states that:
 
The needs of the micro process dictate that many more internal releases to the development team will be accomplished, with only a few executable releases turned over to external parties. These internal releases represent a sort of continuous integration of the system, and exist to force closure of the micro process.

Principles of Continuous Integration

A software development methodology like Continuous Integration allows members of the team integrate their work frequently. It involves each of the team members integrating at least daily thereby leading to multiple integrations every day. Each of the integrations is checked by an automated build (that includes the test) for detecting integration errors faster.

“Continuous Integration doesn’t get rid of bugs, but it does make them dramatically easier to find and remove.” — Martin Fowler, Chief Scientist, ThoughtWorks

Developers can frequently commit to a shared repository with the help of a version control system such as Git. They can choose to run local unit tests on their code before each commit as a mark of added verification layer before integrating. A CI service automatically builds and runs unit tests on the new code alterations for immediate identification of any errors.

With Continuous Delivery, code alterations are automatically built, tested and pushed to production. Continuous Delivery expands upon Continuous Integration by deploying all code alterations to a test environment and/or production environment after the build stage.

Key practices that form an effective Continuous Integration

• Single source repository: A decent source code management system should be in place.
• Build automation: Automated environments for builds are a common feature of systems that ensure that you can build and launch your system using a single command.
• Self-testing: Automated tests in the build process can help in catching bugs swiftly and efficaciously.
• Daily commits: By committing to the mainline every day, developers can correctly build their code including passing the build tests.
• Integration machine: Regular builds should be happening on an integration machine and the commit should be considered done only if this integration build succeeds.
• Immediate fix of broken builds: The integral part of doing a continuous build is that if the mainline build fails, it should be immediately fixed.
• Rapid feedback: It is of consummate importance to keep the build fast and provide rapid feedbacks.
• Test environment: Always test in a clone of the production environment.
• Finding the latest executable: Anyone involved in a software project should get the latest executable easily and be able to run it for demos or exploratory testing.
• See the state of the system: Everyone should be able to see what’s happening with the system and the alterations that have been made to it.
• Deployment automation: You should have scripts that will let you deploy the application into any environment easily.

Continuous Integration Workflow

• At first, developers check out code into their private workspaces
• When performed, commit the alterations to the repository.
• The CI server monitors the repository and checks out alterations when they occur.
• The CI server builds the system, runs unit and integration tests, releases deployable artefacts for testing
• The CI server assigns a build label to the version of the code which was just built by it and informs the team of the successful build
• The CI server alerts the team members if the build or tests fail
• The team members immediately fix the issue
• The team continues to integrate and test throughout the project

Responsibilities of the team members

• Check in frequently
• Do not check in broken code, untested code or when the build is broken
• Do no go home after checking in until the system builds

Continuous Integration tools

A development team uses CI software tools for automating parts of the application build and constructing a document trail. The following are examples of CI pipeline automation tools:

• CircleCI is a continuous integration platform. When connected to a Drupal site, alterations made in version control in code repository such as GitHub alert CircleCI to start the build of the application and execute predefined testing suite.
• Travis CI is similar to CircleCI which integrates with GitHub, Bitbucket, and other applications. It creates application builds and runs testing suites when code alterations are pushed.
• Jenkins is an open source automation server installed and handled by the user unlike platform services like CircleCI and Travis CI. It is extendable with plugins and works well with Git. It lets you do a wide range of configurations and customisation.
• The open source GitLab repository and platform can run unit and integration tests on several machines and can split builds to work over numerous machines for decreasing execution times.
• JetBrains TeamCity is an integration and management server for enabling developers to test code before they commit alterations to a codebase. It features Build Grids which allow developers to run several tests and builds for different platforms and environments.

Merits of Continuous integration

Eliminates the blind spot

Deferred integration is troublesome as it is an arduous task to predict how long it will take to do a project or even worse how far are you through the process. So much so that you are putting yourself in a blind spot at one of the critical parts of a project. One of the most significant merits of Continuous Integration is minimised risk. With CI, there is no long integration and it completely eliminates the blind spot. You will know where you are, what is working and what is not, and the outstanding bugs that you have in your system.

Easy bug detection

CI does not entirely remove bugs but makes it easier to detect it faster. Projects with CI tend to have dramatically fewer bugs both in production and in process. The degree of this merit is directly proportional to how good your test suite is.

Frequent deployment

CI promotes frequent deployment which lets your users get new features more rapidly, to provide more rapid feedback on those features and collaborate more in the development cycle.

Continuous Integration for Drupal

DrupalCon Dublin 2016 had a presentation which showed how you can leverage the Jenkins 2 pipelines for implementing Continuous Integration/Deployment/Delivery for the Drupal site while taking care of the principles like Infrastructure as Code, Configuration as Code, DRY (Don’t Repeat Yourself) and Open/Closed principle (from SOLID principles).

The process that was followed in the presentation required pushing a commit into self-hosted (Gitlab) or private Github repo. It involved building the doc root from the various sources, deployment procedures, auto-tests on servers and everything was done in the same pipeline configured as separate stages.
 
It showed the auto-generation of deploy pipelines for each branch/state like development, staging or production configured. It utilised different approaches for governing code structure such as Composer-based workflow and ‘all-code-in-repo’ solution in the same doc root. Auto-checking that code was delivered to the server prior to the start of drush deploy procedures and testing.

The project team can have their own deploy scripts

Universal deploy script was used that will be useful for any project. Additional project-specific deploy scripts - DRY (or override) basic deploy script - was leveraged which could be useful when you delegate this part of the responsibility to the different team. You can control what drush commands or command options can be used for certain projects, that is, the project team can have their own deploy scripts.
 
Configuration as a code on both the Jenkins side and the Drupal side was performed. On the Jenkins side, all Jenkins jobs (pipelines) was stored as code in Git and regenerated in case of code alterations with the assistance of Job DSL. On the Drupal side, every configuration alteration was performed in code and then processed on Production servers during code deploys.
 
It showed auto-creation of URLs and databases on hosting platform based on multisite setup. It displayed automated backups before code deploys, copied whole sites inside doc root or between different doc roots with one click and added custom actions for offering additional functionality specific to the projects.

Market trends

The Forrester Wave™: Continuous Integration Tools, Q3 2017 delineated the 10 providers that matter most and how they stack up.

A report on Markets and Markets stated that the CI tools market size is expected to grow from USD 483.7 million in 2018 to USD 1139.3 million by 2023 at a Compound Annual Growth Rate (CAGR) of 18.7% during the forecast period.

A study by Data Bridge Market Research states that major players operating in the global continuous integration (CI) tools market are Atlassian (Australia), IBM (US), Microsoft (US), Micro Focus (UK), CA Technologies (US), Cloudbees (US), AWS (US), Puppet (Oregon), Red Hat(US), CA Technologies (US), Oracle (US), Micro Focus (UK), SmartBear (US), Jetbrains (Czech Republic), CircleCI (US), Shippable (US), Electric Cloud (US), V-Soft Technologies (South Africa), BuildKite (Australia), TravisCI (Germany), AutoRABIT (US), AppVeyor (Canada), Drone.io (US), Rendered Text (Serbia), Bitrise (Hungary), Nevercode (UK), and PHPCI (Belgium) are among others.

Conclusion

Software development can be drastically improved with the incorporation of Continuous Integration resulting in better team collaboration, reduced risk, and faster delivery. Drupal-based projects can reap the merits of CI tools. With Drupal being a huge perpetrator of digital innovation, implementing Continuous Integration in Drupal projects is viable.

OpenSense Labs has a pool of Drupal experts who have been a force to reckon with when it comes to enabling digital transformation dreams of enterprises with its suite of services.

Contact us at [email protected] to know how can you leverage continuous integration for Drupal projects.