Developer Tools Initiative - Part 5: Announcing our Migration

This is the fifth post in our series about integrating Drupal.org with a 3rd party developer tooling provider:

• Where we stand now, and our evaluation criteria.
• The options we considered.
• An illustration of what Drupal.org integration with a 3rd party tooling provider could look like.
• A possible implementation plan for this initiative.
• You are here: Announcing our migration

In this post we are announcing our migration to a new tooling provider, and outlining the phases of that migration process to take place over the next several months.

Announcing our partnership with GitLab

Wait, what?

Yes, in our four part series from December of last year it certainly looked like we were going in a different direction for the future of Drupal's developer stack.

So what changed?

Last year we laid out a model for integrating Drupal.org with a third party tooling provider, which we described as "Drupal Flow". This model was deliberately laid out to be agnostic to the provider we chose, so long as certain requirements were met. We worked with representatives from three potential providers at the time: GitHub, GitLab, and BitBucket, and each one had pros and cons. Once we had completed our evaluation, BitBucket was the only provider without hard blockers to the integration we wanted to build.

However, following our blog series, the GitLab team reached out directly to the Drupal Association team, and asked us to give them the chance to resolve the blockers and close the gaps in our integration.

At the same time, we saw an outpouring of feedback from our community asking us to see if we could find a way to make GitLab work.

And so we did.

The Agreement

For the past six months we've been working closely with Eliran Mesika, the Director of Partnerships at GitLab, in addition to CEO Sid Sijbrandij and members of GitLab's engineering team. They've escalated the internal priority of issues that blocked our adoption of GitLab, offered technical and financial support for the migration, and made a commitment to ongoing support for the Drupal project.

And so we're happy to announce that Drupal.org is going to be moving our code collaboration tools for our forty-five thousand projects to GitLab over the course of the coming months.

Three Phases to the Migration

Phase 1: Replacing Drupal.org's Git backend

The first phase of the Drupal.org migration

• Transparently replace Drupal’s current underlying Git infrastructure (for repository hosting, maintainer permissions, code viewing) with GitLab repositories, GitLab roles and permissions for maintainers, and the GitLab code viewing UI.
• Enable inline code editing (only for maintainers for this phase).
• During this phase, Drupal.org will remain the primary source of information.  SSH keys, new projects, etc. will be created on Drupal.org.

This first phase, while modest, will bring some concrete benefits to the project:

• Maintainers will be able to begin familiarizing themselves with GitLab's code collaboration tools.
• Code viewing will receive a significant upgrade from CGIT to GitLab's built-in code viewer.
• And Drupal.org's old Git server will be phased out.

Phase 2: Enabling Merge Requests, Inline Code Editing, and Web-based Code Review

• The timeline for Phase 2 is dependent on GitLab’s resolution of a diskspace deduplication issue, which they have committed to on our behalf: https://gitlab.com/gitlab-org/gitlab-ce/issues/23029
• Enable GitLab Merge Requests, GitLab inline code editing in the web UI, and GitLab web-based code review.
• During this phase, Drupal.org will handle any 'create branch/merge request' integrations from the Drupal.org Issue queues, and related call-backs from GitLab into the Drupal.org issue comment stream.

Phase 2 is where we realize some tremendous benefits to developer velocity and collaboration:

• By adding merge requests, contributing to Drupal will become much more familiar to the broad audience of open source contributors who learned their skills in the post-patch era.
• By adding inline editing and web-based code review, it will be much easier to make quick contributions. This not only lowers the barrier to contribution for people new to our community, it also saves significant effort for our existing community members, as they'll no longer need to clone work locally and generate patches.
• Finally, by creating a tight integration between the Drupal.org issue queues and GitLab's development tools, we'll be able to transition to this new toolset without disrupting the community's existing way of collaborating.

Phase 3: Evaluating Additional Features

Phase 3 has no strict timeline, but will be dependent on feedback from the community as they get up to speed on using the new GitLab-based contribution workflow for Drupal.

• Evaluate additional features such as:

  • Integrating or replacing DrupalCI with GitLab CI
  • Enabling GitLab issues for a sub-set of projects
  • Enabling GitLab confidential issues for specific use-cases (security releases)
  • Possible MatterMost integration, etc.

These additional features may allow us to further improve the velocity of the Drupal project, or realize additional cost savings for the association. For example, we may be able to use GitLab's test runner integration to orchestrate tests across a wider variety of cloud platforms, helping us find the best pricing. We may be able to entirely replace security.drupal.org with a private issue tracker, eliminating an entire sub-site for the Drupal.org team to maintain. We may even be able to enhance existing community services like SimplyTest.me by integrating features like GitLab's AutoDevops tools to automatically create review environments for issues or branches.

We won't really know what's possible within the scope of our resources until the first two phases are completed, but this helps to show that by hitching our toolset to a partner that specializes in collaboration, we may be able to realize even more benefits for our community.

Changes to Git Remotes

• Password authentication for Git is deprecated. We recommend that all users switch to ssh key authentication.

• Git remote urls for pushes to full projects have changed:

  • If you have an established Git remote in the format
    <username>@git.drupal.org:project/<yourproject>.git
    the format should be changed to:
    [email protected]:project/<yourproject>.git

• HTTPS clone urls for full projects are unchanged.
• HTTPS clone urls and Git remote urls for sandbox projects have changed:
  • For remotes of the format:
    <username>@git.drupal.org:sandbox/<username>/<node-id>.git
    the format should be changed to:
    [email protected]:sandbox/<username>-<nodeid>.git
  • Clone urls will be changing from:
    https://git.drupal.org/sandbox/<username>/<nodeid>.git
    to the format:
    https://git.drupal.org/sandbox/<username>-<nodeid>.git

Important: If you have any automated systems which authenticate to Git, such as CI pipelines or repo mirroring, ensure they are updated as well.

For more detailed information about these changes, as well as instructions for changing your Git remotes or setting ssh keys, please consult these instructions: https://drupal.org/gitauth

How to follow our progress

Issues for the Drupal.org migration to GitLab will be opened in the Drupal.org Infrastructure queue and tagged 'GitLab'.

For questions or concerns, please create an issue at https://www.drupal.org/node/add/project-issue/infrastructure