Jun 11 2018
Jun 11

Read our Roadmap to understand how this work falls into priorities set by the Drupal Association with direction and collaboration from the Board and community.

AnnouncementsDrupal Association logo

Change your git remote configuration

We will be deprecating the git remote format @git.drupal.org/project/;.git in favor of [email protected]/project/<yourproject>.git in preparation for changes to our developer tooling stack. If you used the <username>@ format for your git remotes, you should change your remote to the [email protected] format. You can use $ git remote set-url to make this change for existing repositories you have cloned.

We have updated the version control instructions on Drupal.org to reflect this change, and will be updating the git daemon to warn developers who are using the deprecated remote format.

Proposal: Improving Core's Relationship with Composer

In May, Mixologic from the Drupal Association engineering team worked with community members Mile23, Bojanz, Webflo, and others in the community to develop a proposal for improving Drupal Core's relationship with Composer.

In its simplest form, the proposal is to: Conceptually separate Drupal, the product, from Drupal's git repository, and provide a mechanism that creates a composer ready Drupal installation.

Going into early June, we've been circulating this proposal to the Core Committers, the Auto-Updates Initiative team, Contrib maintainers, Distribution maintainers etc.

Credit for non-code projects on Drupal.org

We're excited to announce that we've created a 'Community Projects' section in the Drupal.org issue queues. This section exists to record all the tremendous community labor exercised to promote the Drupal project in ways other than code. This format was pioneered by the Drupal Diversity and Inclusion group, who started recording their meeting minutes in the issue queues so they could provide contribution credit for attendees. This same model can be used by initiative coordinators, camp organizers, or any other Drupal community group that would like a place to recognize their work with the official contribution credit system.

The Contribution Credit system is one of the Drupal projects most successful innovations in the way that open source projects are managed, and it will continue to evolve and grow as time goes on.

Updates for GDPR

AEU GDPRre the words "We've updated our privacy policy" burned into your laptop screen yet? Well in May we did the same.  In particular, we've updated our Terms of Service, Privacy Policy, Digital Advertising Policy, and Git Contributor Agreement to clarify our compliance with the EU's GDPR. We also initiated a re-consent campaign for our marketing lists. If you have not re-consented to communications we strongly encourage you to do so

Launched the Customer Supporter program

Have you built great relationships with your Drupal customers? Help them contribute back to the project by becoming part of our Customer Supporter program.

Drupal.org Updates

Self-nominations for the Drupal Association board are live

Each year one of the two community-held seats on the Drupal Association board comes up for election. We opened the self-nomination process for this year, and some passionate and dedicated members of the community have already stepped forward with their candidacy.

To learn more, you can view our portal for the 2018 Elections Process. Key dates to remember:

  • Self nominations: 1-11 June, 2018
  • Meet the candidates: 12-29 June 2018
  • Voting: 2-13 July, 2018
  • Votes ratified, Winner announced: 25 July, 2018

Better landing pages for Drupal's strategic initiatives

Did you know there are 12 active Drupal strategic initiatives right now?

To help the initiative coordinators promote this work, and recruit more open source contributors to the cause, we've given initiative coordinators new landing page tools. Check out the first initiative to use this landing page: the Admin UI and Javascript Modernization Initiative.

These tools are the first step in improving the project management tools available to initiative coordinators to help move the Drupal project forward.

Historical user and organization contribution data is now available.

Drupal.org user profiles show the last year's worth of contributions by users. We chose the one year window deliberately, to promote the importance of a user's more recent activity. However, seeing a user's complete contribution history can be valuable as well. We've recently added a link to the bottom of this view to display that history.

Similarly, organization profiles have shown the last 90 days of contributions by organization. Again, we chose this very deliberately to emphasize the  importance of recent and ongoing contribution. However, as with user accounts, these profiles now also include a link to the organization's complete contribution history. You can see an example of where to find this link below:

Organizations - View all credit history

Expanded spam protections

After the sunsetting of Mollom in March of this year, we've been implementing a new set of tools to mitigate spam on Drupal.org. We expanded these protections in May, using a combination of bot detection, content analysis, rate limiting, and more to try and reduce the impact of spam attacks on Drupal.org. The less time the community spends wading through spam, the greater the velocity of the Drupal project.

And a Thank You

We'd also like to give a special shout out to contributor Wim Leers, for his incredibly kind 'Ode to the Drupal Association' about our work on the testing infrastructure. The nature of software engineering has a tendency to draw our attention to things that are broken, buggy, or unoptimized, and so when things are working well that success can sometimes feel invisible.

Fortunately, the Drupal community puts people first, and celebrates our collective success, and Wim's words are a tremendous example of that ethos.

Thank you, Wim - and thank you to everyone who takes the time to recognize the hard work and dedication of your fellow contributors.

———

As always, we’d like to say thanks to all the volunteers who work with us, and to the Drupal Association Supporters, who make it possible for us to work on these projects. In particular we want to thank:

If you would like to support our work as an individual or an organization, consider becoming a member of the Drupal Association.

Follow us on Twitter for regular updates: @drupal_org, @drupal_infra

May 23 2018
May 23

Our global community includes many EU citizens and residents of the EEA, and we have taken steps to comply with the GDPR which takes effect on May 25, 2018.

Your rights under this law and how Drupal.org complies with GDPR

We've updated our Terms of Service, Privacy Policy, Git Contributor Agreement, and Digital Advertising Policy based on the requirements of the EU General Data Protection Regulation. We've also begun a campaign to reconfirm your consent to our marketing messages.

For easy and clear access to the changes: 

Human Readable Summary

Disclaimer: This summary is not itself a part of the Terms of Service, Privacy Policy, Git Contributor Agreement, or Digital Advertising Policy, and is not a legal document. It is simply a handy reference for understanding privacy rights and regulations. Think of it as the user-friendly interface to the legal language.

In plain language, regulations such as GDPR define the following roles, rights, and responsibilities:

  • Data Subject - this is you, the end user.
  • Data Controller - this is us, the Drupal Association as the owners and operators of Drupal.org and its sub-sites.
  • Data Processor - any other organization that processes personal data on behalf of the Data Controller.

Rights of the Data Subject

  • Right to be Informed - A data subject has the right to know whether personal information is being processed; where; and for what purpose.
     
  • Right to Access - A data subject has a right to access the information about them that is stored by the Data Controller.
     
  • Right to Rectification - A data subject has the right to correct any errors in the data about them. This can be done by editing your user account, or contacting the Drupal Association directly.
     
  • Right to Restrict Processing - A data subject has the right to request that data not be processed, and yet also not be deleted by the Data Controller.
     
  • Right to Object - A data subject has the right to opt out of marketing, processing based on legitimate interest, or processing for research or statistical purposes.
     
  • Right to be Forgotten - Also known as the right to revoke consent, the right to be forgotten states that a data subject has the right to request erasure of data, the cessation of processing by the controller, and halting processing of the data by third party processors.

    The conditions for this, as outlined in article 17, include the data no longer being relevant to original purposes for processing, or a data subjects withdrawing consent.

    It should also be noted that this right requires controllers to compare the subjects' rights to "the public interest in the availability of the data" when considering such requests.

  • Data Portability - A data subject has the right to receive a copy of their data in a 'commonly used and machine readable format.'

    This information is outlined in the sections below titled "Your Choices About Use and Disclosure of Your Information" and "Accessing and Correcting Your Information".

Responsibilities of the Data Controller and Data Processors

  • Privacy by Design - 'The controller shall..implement appropriate technical and organisational measures..in an effective way.. in order to meet the requirements of this Regulation and protect the rights of data subjects'. Article 23 of the GDPR calls for controllers to hold and process only the data absolutely necessary for the completion of its duties, as well as limit the access to personal data to those who need it to carry out these duties.
     
  • Breach Notification - The Data Controller must notify the appropriate data processing authority and any affected end user of any breach that might result in 'risk to the rights and freedoms of individuals' within 72 hours of becoming aware of the breach.

    A Data Processor must notify the Data Controller of any breach 'without undue delay.'

  • Data protection officer - A Data Controller or Processor must appoint a Data Protection Officer when: a Data Controller represents a public authority; or the core operations of the Controller require regular and systematic monitoring of Subjects on a large scale; or when the Controller's core operations depend on processing a large scale of special categories of data (including but not limited to health data, criminal conviction information, etc).
     

    The Drupal Association's core operations do not require the Association to establish a Data Protection Officer.

We take privacy and security very seriously, as all Drupal professionals do! We will continue analyzing the legal landscape and collecting feedback for future revisions.

If you have any questions or concerns about our GDPR compliance, or if you want to point out a mistake or provide a suggestion for the Terms of Service, Privacy Policy, Git Contributor Agreement, or Digital Advertising policy, you can send an email to [email protected].

Apr 30 2018
Apr 30

Read our Roadmap to understand how this work falls into priorities set by the Drupal Association with direction and collaboration from the Board and community.

Drupal.org UpdatesDrupal Association Logo

Drupal.org's new front page and persona pages launched

As you've probably seen by now, just before DrupalCon Nashville we launched a makeover of the Drupal.org front page. This was a research-based redesign focused on addressing the three key personas that come to Drupal.org: Developers, Marketers/Content Editors, and Agencies.

The new redesign simplifies the number of calls to action on the front page, and directs each of these personas into a more focused funnel, to ensure they are more likely to find the information they really need. To learn more about this redesign and the Promote Drupal initiative, read our recent blog post. We want to thank SixEleven for their help with this new design initiative.

Promote Drupal Initiative

Redesigning the front page was just the start, we kicked off DrupalCon by announcing a new 'Promote Drupal' initiative, asking the community to come together to help bring Drupal to new audiences, and to convince people who've used older versions in the past to give Drupal 8 another look.

We need your support to make the Promote Drupal initiative happen!

Updated top navigation and IA

Along with the front page changes, we've updated Drupal.org's top level IA, providing a more logical structure for navigating to the major areas of the site depending on a user's persona.

Drupal.org Top Nav Redesign

Promoting Nonprofit solutions built with Drupal

And last, but not least, in our efforts to #PromoteDrupal we've launched a new Nonprofit solution page, promoting the power of Drupal for Nonprofits and NGO's around the globe. Drupal has long been the choice for well-recognized, global nonprofit organizations to extend their reach and maximize their impact.

Drupal nonprofit

Simplify Drupal Initiatives

In project founder Dries Buytaert's keynote at DrupalCon Nashville he proposed a series of initiatives to simplify Drupal - lowering the barriers to adoption and improving the user experience of site administrators and content editors. Some of these initiatives are to improve features of Drupal core itself, whereas others are focused on the evaluator experience and will be managed in collaboration with the Drupal Association.

In particular, the Drupal Association will collaborate with the core initiatives teams on:

These initiatives are not going to be quick or easy. They rely on collaboration between the Drupal Association, Drupal's core committer team, and a variety of volunteers throughout the community. We'll need your help.

Drupal.org and GDPR

GDPR, the General Data Protection Regulation passed by the EU last year, begins enforcement on May 25th, 2018. We've been preparing for this new regulation for some time, and will be implementing a few changes in the coming weeks:

  • Updates to our:
  • Updates to our mailing lists
  • Publishing our data retention policy
  • Publishing our data portability policy
  • Updating our breach notification policy

Drupal securitySecurity Release

SA-CORE-2018-003

Drupal Core coordinated a security release with the CKEditor team to ensure that the security fix for CKEditor was immediately available in Drupal 8. As Drupal becomes further integrated into a world of third party dependencies, this kind of coordination between open source projects becomes increasingly important. We want to thank the CKEditor team and the volunteer Drupal Security team for their hard work and careful collaboration.

SA-CORE-2018-004

After the release of SA-CORE-2018-002 in March, a related vulnerability was discovered and an additional security advisory for Drupal 7 and 8 released in April. If you have not yet updated your Drupal sites to address these vulnerabilities they may already be compromised. If that is the case, we encourage you to read this PSA, which provides some steps you can take.

Security releases tend to spark quite a bit of conversation in the community about the nature of software security, proprietary vs open source, and related issues. Community member @rickmanelius provided some much-needed context to keep these security focused efforts in perspective:

The recent SA-CORE-2018-004 and SA-CORE-2018-002 security advisories have sparked a lot of conversations in the Drupal community regarding all things security. IMHO, it's important to highlight several talking points to keep things in perspective.

— Rick Manelius, PhD (@rickmanelius) April 26, 2018

DrupalCI: Support for DrupalCI.yml

DrupalCI now supports the use of Drupalci.yml files in projects to customize and override elements of testing. This makes the testing capability of DrupalCI much more powerful and flexible for project maintainers. We're still working on documenting these new features, but you can read about the new features here.

———

As always, we’d like to say thanks to all the volunteers who work with us, and to the Drupal Association Supporters, who make it possible for us to work on these projects. In particular we want to thank:

If you would like to support our work as an individual or an organization, consider becoming a member of the Drupal Association.

Follow us on Twitter for regular updates: @drupal_org, @drupal_infra

Apr 27 2018
Apr 27

Read our Roadmap to understand how this work falls into priorities set by the Drupal Association with direction and collaboration from the Board and community.

This month's update comes a bit later than usual, as we return from DrupalCon Nashville. Expect our April update to follow after shortly.

Representing Drupal at Google's CMS Leadership Summit

In mid-March, we attended the Google CMS Leadership Summit, as representatives of the Drupal project. The Summit was a one-day event hosted by Google to unite the 20 or so projects in the CMS space responsible for more than 50% of the content on the web.

The goal was to understand how to preserve an open web, by empowering better authoring experiences, content consumption, and performance in our CMS platforms.

This level of dialogue and engagement with an organization like Google is new and exciting for us, and we're looking forward to ongoing conversations, both with Google and with the other CMS project leaders they assembled at the event.

Drupal.org UpdatesDrupal Association Logo

Researching the anonymous traffic to Drupal.org

One of the focuses of the Drupal Association in 2018 has been to better understand our audience. When it comes to users who register on Drupal.org, and our DrupalCon attendees, we have quite a bit of information about who our users are.

However, when it comes to the wider ecosystem of Drupal users (evaluators and end-users who do not have Drupal.org accounts) we've been largely in the dark for most of the project's history. One way we want to improve this is by working with Drupal Core to add telemetry to Drupal, but that is an effort that will take some time.

In the meantime, we've implemented several Audience Insight tools to help us learn more about our anonymous users. Privacy is always a paramount concern, so we chose only insight tools which provide aggregate, anonymized data, and we wrapped those tools in our own implementation of Do-Not-Track so that we could ensure that user privacy preferences are respected.

The table below demonstrates the job functions held by the anonymous visitors to Drupal.org. (Please note: these job functions might be held within any kind of industry, this data is about the user's role, not their target market).

Job Function

D.O Front Page Visitors

All D.O Visitors

Diff

Engineering

26.20%

44.90%

-18.70%

Information Technology

14%

16.40%

-2%

Business Development

11.60%

9.60%

2.00%

Entrepreneurship

10.30%

10.70%

-0.40%

Arts and Design

7.70%

5.30%

2.40%

Media and Communication

6.60%

6%

0.60%

Marketing

6.30%

3.70%

2.60%

Education

6.10%

4.60%

1.50%

Operations

5.20%

3.50%

1.70%

Program and Project Management

4.30%

4.20%

0.10%

Sales

4.20%

2.60%

1.60%

Consulting

2.90%

3.30%

-0.40%

Research

2.40%

1.70%

0.70%

Community and Social Services

2.40%

1.80%

0.60%

Administrative

2.30%

1.30%

1.00%

We've used this data to inform the redesign of Drupal.org, as well as our new persona pages. Learn more about that process in our April update. The redesign work was carried out in collaboration with SixEleven, who also produced the DrupalCon brand and design for DrupalCon Nashville.

Nashville guitar

Preparing for DrupalCon Nashville

In the lead up to DrupalCon Nashville in April, the team was in high-gear preparing for the event. We participated in a panel about the future of pull requests on Drupal.org, the public board meeting, and handled the keynote livestream process.

DrupalCon is always an incredible opportunity for the team to connect with the community about upcoming initiatives, drupal.org support requests, and to plan for the future.

We were happy to see so many of you there, and we'll talk more about this in our April update.

Documentation enhancements

Our efforts to improve the quality of Drupal's documentation continued in March and April, as we added features:

  • New D7/D8 guides are now automatically approved, so new project contributors aren't blocked on documenting their projects.
  • Added Drupal version to page title for better searchability.
  • Follow/Unfollow links are available directly on the discuss page of any documentation.

In-context links to newer and older releases

To ensure that users are aware when there are newer releases than the one they may be looking at, we now provide newer and older release history on release nodes. In the sidebar of any release page you will see links and dates to related releases. Among other things, we hope this will prevent users from accidentally installing an older release when another new one has just come out!  Other releases

Email notifications for new maintainers

Encouraging succession planning in module maintership is an incredible challenge for any open source project. We want to encourage maintainers to invite contributors to their projects to help maintain those projects, but we recognize that we also have to make sure the appropriate tools are in place to make this a smooth process.

To make sure that new maintainers of projects are welcomed into the fold, we've added email notifications to let a user know when they've been added as a project co-maintainer. If you are invited by an existing maintainer of a project to help maintain it, you should now receive a warm welcome.

Drupal Security Team

Security Release

SA-CORE-2018-002

The Drupal Association Engineering Team collaborated with the Security Working Group and Security Team to coordinate 3 significant security releases in March and April.

The primary release was SA-CORE-2018-002, a highly critical security release for Drupal 7 and 8. For more information about all Drupal security releases and PSAs, please visit our security portal.

The volunteer Security Team has always been a tremendous asset to our community, and the Drupal Association is proud to support their work.

Infrastructure Updates

DrupalCI: Support for testing themes

DrupalCI has enabled support for testing Themes, so now Theme projects on Drupal.org can include tests. This has become more and more necessary as javascript becomes critical to modern web design, and we hope this will help accelerate the build out of themes for Drupal 8 and increase their quality.

———

As always, we’d like to say thanks to all the volunteers who work with us, and to the Drupal Association Supporters, who make it possible for us to work on these projects. In particular we want to thank:

If you would like to support our work as an individual or an organization, consider becoming a member of the Drupal Association.

Follow us on Twitter for regular updates: @drupal_org, @drupal_infra

Mar 07 2018
Mar 07

Read our Roadmap to understand how this work falls into priorities set by the Drupal Association with direction and collaboration from the Board and community.

Drupal.org UpdatesDrupal Association

Reimagining Drupal.org's front pages to serve distinct personas

Drupal serves a wide audience of users, from developers to marketers to content editors and beyond. Historically, Drupal.org has been focused on our community of contributors, whether those contributions are in code, documentation, volunteer support, camp organizing, etc. However, only 1 in 15 visitors to Drupal.org are an authenticated user, and the rest are primarily visiting Drupal.org as representatives of an end-user organization that is evaluating Drupal. We want to serve these visitors better.

In February we held an off-site in Portland to consolidate our research about the personas within end-user organizations who make the decision to adopt Drupal. We identified three key roles:

  • Technical evaluators - who are often developer evangelists within their organization

  • Marketing and business users -who are evaluating Drupal as a platform. They are interested in the editorial experience and time-to-market for building a solution that integrates with tools they already use

  • Agencies - who are already using Drupal for their clients, or are considering making it central to their business.

From there, we developed some initial concepts for a reimagining of the front page of Drupal.org to better serve these first three personas.

This work will carry us through DrupalCon Nashville and beyond, so expect additional updates over the coming months.

Contribution credit update

The Drupal project has an innovative system of crediting users and sponsoring organizations for the work they contribute. However, as a system that we've pioneered, there is always room for additional improvements. One area that needed improvement was the date used for the assignment of credit. In the past, the credit for a user or organization would be tracked to the timestamp of the latest activity on the issue. This was a good approximation, but additional comments after issue resolution would bump the date of the credit.

We've updated the way that contribution credits are calculated - so that it is now based on the date that the issue was closed(status last changed) instead of the date of the last change to the issue. This change affects both individual contribution credits and the marketplace ranking.

Documentation improvements

As our new team member Dhanya has come on board, she has helped make some great improvements to the documentation system, including: fixing the display of sidebar lists of guide contents, increasing the visibility of the current page indicators, and swapping the grid treatment for a more readable guide contents layout

Doc Guide Preview

Accessibility and readability

We've made two additional small fixes.

One for accessibility - improving the keyboard 'skip to…' links in the Drupal.org top navigation.

Skip to main

Skip to search

… And one for navigating issues, fixing a bug that prevented links to comments on multi-page threads. Now, any Drupal.org user who receives an email notification about a multi-page issue should be properly linked to the correct comment.

Preparing our live-streaming capability for DrupalCon Nashville

For DrupalCon Nashville, rather than relying on a vendor, we are going to be managing the live stream of the keynotes and closing session ourselves, together with the AV staff of the venue. In February we spent some time putting together our equipment and running some streaming tests.

Continued work to reduce our PCI scope

In February we finished migrating our donation process for both USD and Euro donations to new payment processors to reduce our PCI scope and thus maintenance costs. We've also launched the beta of the membership system, and will hopefully complete the migration of existing memberships soon.

If you are not yet a member of the Drupal Association, and would like to support us both by joining and helping us test the new membership system, you can sign up here. (If you are an existing member, please continue to process your renewals on the original system for now).

Infrastructure Updates

Git servers updated

We migrated our existing git infrastructure from bare metal servers to virtual machines, which will help to make our infrastructure more flexible and portable in the future. This has been an ongoing effort, and the git servers are among the last of the servers to be migrated.

Continued tuning of Perimeter XPerimeterX

PerimeterX helps to identify bad actor behavior and DDOS attempts and mitigate them at the edge of our network. We established our relationship in January, and throughout February have been monitoring and tuning our configuration to better protect Drupal.org. We've already managed to mitigate a persistent DDOS attack which has recurred every couple of months, and hopefully we can make more improvements to protect Drupal.org, and reduce the pager burden on our team.

———

As always, we’d like to say thanks to all the volunteers who work with us, and to the Drupal Association Supporters, who make it possible for us to work on these projects. In particular we want to thank:

If you would like to support our work as an individual or an organization, consider becoming a member of the Drupal Association.

Follow us on Twitter for regular updates: @drupal_org, @drupal_infra

Feb 26 2018
Feb 26

JetBrainsWe are happy to announce today that JetBrains is continuing their support of the Drupal Association and the Drupal project in a new way:

It has been part of JetBrains mission to give back to key contributors in the open source world, and we're grateful that they want to offer this opportunity as a benefit for Drupal.org contribution credits.

To receive the free PhpStorm Open Source license, please fill out this open source license request form. Be sure to include:

  •  Project Name: Drupal
  •  # of Developers: 1 - since you are requesting only a single license for yourself
  • Project role: Contributor
  • Your role in project: <-- This should include the link to your Drupal.org profile.
  • Fill in the rest of the fields as you are best able - the ones above are the key fields for this offer.

The Jetbrains team will be checking the profile to ensure you meet the 35 contributions threshold, they will be looking for the line which reads Credited on XXX issues fixed in the past 1 year. You can read more about PhpStorm and the Open Source support program by JetBrains.

Many thanks to JetBrains, we’re excited for their continued partnership.

Feb 20 2018
Feb 20

Read our Roadmap to understand how this work falls into priorities set by the Drupal Association with direction and collaboration from the Board and community.

In January the Drupal Association kicked off the new year with our Winter Staff Retreat, where the whole organization came together to review the past 6 months and lay out our strategy for the new year. In addition to taking a big picture look at our upcoming priorities, we also made some great improvements this past month.

Before we dive into those, we'd like to welcome Dhanya Girish to the Drupal Association team. Her time has been generously sponsored by Zyxware and she joins us as a skilled engineer. Please welcome Dhanya!

Drupal.org Updates

Celebrating Drupal's 17th Birthday

To celebrate the 17th birthday of Drupal, we've embedded a wonderful video on the home page, celebrating some of the incredible things that have been built with Drupal. Drupal has grown from being a dorm-room experiment to being the driver for some of the most powerful digital experiences on the web. We can't wait to see what the next 17 years bring.

[embedded content]

Proposed new initiatives for collaboration with Core

The Drupal Association engineering team meets monthly with the Drupal Core committers to align our goals and ensure that we're on the same page about what the project needs moving forward. At the beginning of this year we outlined some new proposed initiatives that, in collaboration with Core, we believe could be a tremendous value to the project.

None of these initiatives can be accomplished quickly, nor can they be moved forward without the collaboration of key contributors and the core committers, but we strongly believe they will be important next steps for the future of Drupal.

Refresh of the Hosting Listings program

A major focus in January was the refresh of our hosting listings program. The new hosting listings now display all providers on the primary view, with a series of filters to help users find exactly the right hosting partner. This update also brings a much needed visual refresh to the listings. New filters and search facets will help end-users find the hosting partner that is right for their needs, industry, and budget, and will enable users to identify and support those platforms that support the project.

Promoting industry solutions in the marketplace

Over the last year, we've been focused on pushing a message about crafting the perfect solution for particular industries with Drupal. We link to this information off of the front page, but we've also added a contextual block in the sidebar that will appear whenever users filter the marketplace by the relevant industry.

These efforts are an ongoing part of better serving the various personas who come to Drupal.org. Look forward to hearing about even more changes on this front as we approach DrupalCon Nashville in April.

In-context issue tag explanations

One of the cornerstones of Drupal.org contribution and issue management is our issue tagging system. However, it's been difficult in the past to understand what each tag is used for, especially for tags that are carefully monitored and curated by project maintainers or the core committer team. We've enhanced the tagging system by providing in-context hover-states that describe what tags are used for.

Infrastructure Updates

Implemented PerimeterX for increased protection from DDOS, crawlers, badbots etc.

Would you believe that Drupal.org is the target of a bad crawler or outright DDOS attack at least once every two months? The team does an incredible job reducing the impact on end users of Drupal.org, but to make that job easier, we've partnered with PerimeterX for bad bot protection integrated with our CDN. PerimeterX integrates closely with Fastly, our CDN, and so can provide intelligent logic and protection at the edge.

We're starting off using this system simply for greater introspection into the nature of traffic patterns and the attacks of bad actors. We're gradually enabling the protection as we tune the system for Drupal.org's unique needs. We want to thank PerimeterX for supporting the project, and for helping to make Drupal.org a better place for our community.

DrupalCI: Support for yarn-based core tests

We've added a yarn build-step plugin to DrupalCI, allowing us to support new testing types, including nightwatch javascript tests in core and contrib. We want to provide a special thanks to community contributors who helped: justafish, dawhener, and mile23.

———

As always, we’d like to say thanks to all the volunteers who work with us, and to the Drupal Association Supporters, who make it possible for us to work on these projects. In particular we want to thank:

  • PerimeterX - *NEW* Signature Technology Supporting Partner
  • publicplan - *NEW* Premium Supporting Partner
  • WebEnertia - *NEW* Premium Supporting Partner
  • Electric Citizen - *NEW* Classic Supporting Partner
  • Factorial - *NEW* Classic Supporting Partner
  • One Shoe - Renewing Classic Supporting Partner
  • DRUD - Renewing Classic Hosting Supporter

If you would like to support our work as an individual or an organization, consider becoming a member of the Drupal Association.

Follow us on Twitter for regular updates: @drupal_org, @drupal_infra

Jan 05 2018
Jan 05

Read our Roadmap to understand how this work falls into priorities set by the Drupal Association with direction and collaboration from the Board and community.

Announcements

DrupalCon License program launched

DrupalCon In 2018, the Drupal Association made the difficult decision to pause DrupalCon Europe for the year, so that we could re-envision the program for greater scalability and sustainability. We're pleased to announce that we have launched a new program for licensing DrupalCon, so that local entities can bring this great event to their area. Read the announcement by Executive Director Megan Sanicki for more information.

Analysis of Developer Tooling options published

For more than a year, the Drupal Association has been evaluating options for improving the tooling used by developers on Drupal.org. We have recently concluded our study, and published a detailed analysis of the options, as well as our next steps. As expected, this blog series has sparked ongoing conversations about the future of our tools both among the community and with our potential partners - so look out for more updates.

Drupal.org Updates

Easier management of Drupal Association Membership

Drupalcon Individual Member Badge In December we worked on updates to membership management, so that Drupal Association members will be able to manage their membership information.

New DA Membership Directories launched

Last month we mentioned the launch of the new individual member directory on Drupal.org. In December we expanded on this work to update our directory of organization members as well. Drupalcon Organization Member Badge You can explore the directory of Drupal Association members here, as well as the new directory of organization members. If you have feedback on either of these directories, please let us know!

Akismet for spam protection

The content analysis tool Mollom is rapidly approaching its end of life. And so to continue to protect the Drupal community from spam, we have implemented Akismet on Drupal.org. We are currently running it in silent mode, side-by-side with Mollom and our other protection methods, to ensure a smooth transition.

DrupalCI: Chrome Webdriver available for JS testing

DrupalCI One of the major services provided by the Drupal Association is continuous integration testing for the Drupal project. An increasingly important component of this is our javascript testing stack. Previously we tested javascript for the project using PhantomJS. However, that library is now deprecated. We have since created a new testing environment running Chrome Webdriver, and are working with core and contrib developers to ensure it meets their needs.

Drupal.org Updates

Mitigating the risks of Spectre and Meltdown

By now everyone in the technology industry is likely aware of Meltdown and Spectre, the two major security vulnerabilities recently disclosed in major CPU architectures. Drupal Association staff are in close coordination with our infrastructure partners at Tag1Consulting, to ensure that any vulnerable machines in our infrastructure are protected as soon as possible, and our community's data is kept safe. ——— As always, we’d like to say thanks to all the volunteers who work with us, and to the Drupal Association Supporters, who make it possible for us to work on these projects. In particular we want to thank:

If you would like to support our work as an individual or an organization, consider becoming a member of the Drupal Association. Follow us on Twitter for regular updates: @drupal_org, @drupal_infra

Dec 20 2017
Dec 20

This is the fourth post in our series about integrating Drupal.org with a 3rd party developer tooling provider:

With our plan to create modular integration points for our tooling options, we have a few clear steps for moving forward:

Phase 1: Prep work

  • Deprecation of password authentication for Git, since many external tooling services no longer support it.
  • Working with core to provide compatibility for semver versioning for contrib, both because this is needed for Composer, and because all of the third party developer toolsets we are considering have begun to standardize on semver.

Phase 2: Initial implementation, replacing current features

  • Replacement of custom, bespoke Twisted Git daemon with standard Bitbucket repositories.
  • Replacement of unmaintained CGit with supported Bitbucket code viewing.

Phase 3: New features

  • Integration of merge request 'hook' into issue queues, to allow contributors to use a pull request workflow instead of patches.
    • Modular - to be used with Bitbucket for now, but potentially another solution when more mature.
  • Integration of code review 'hook' into issue queues, to give us powerful inline code commenting tools.
    • Modular - to be used with Bitbucket for now, but potentially another solution when more mature.

Phase 4: Implement Hybrid Integrations for other toolsets

  • Updating project page integrations such that those projects which are already hosted on third party tools such as GitHub or GitLab (for example, Drush) can easily login with SSO, synchronize their repositories, and choose the canonical home of their issues.

On-going: Evaluation

  • Re-evaluate other tooling solutions as blocking issues are resolved and their feature-sets evolve.

So that's the update!

In short: after more than a year's evaluation of the current leaders in open source tooling solutions, including direct collaboration with several of those teams, we are going to focus on making Drupal.org modular to integrate with the best tooling solution as it develops. For now, we will be implementing key improvements for Drupal developers using Bitbucket for our repository hosting, code viewing/review, inline editing, and merge requests - integrated with the existing project pages and issue queues.

We'd like to thank the following people for their involvement in this initiative at various times through the process:

Drupal Association Staff

The Technical Advisory Committee (TAC)

Members of the community

The teams of our potential partner organizations

Each of these teams is committed to serving open source and we thank them for their collaboration during our evaluation process:

  • GitHub
  • GitLab
  • Atlassian/BitBucket
Dec 20 2017
Dec 20

This is the third post in our series about integrating Drupal.org with a 3rd party developer tooling provider:

Below are some rough mockups of what a modular integration between Drupal.org's issue queues and a third party toolset could look like. For the sake of this example, I've used Bitbucket as the tooling provider, as that is likely to be our interim toolset for the reasons outlined above. However, one can easily imagine these functions being substituted for their equivalents in other toolsets.

The following is only an early concept. These are by no means final designs for each of these integration points, but they will help to visualize how these tools will be integrated with Drupal.org.

Mockup: Workspaces in the issue summary

The biggest change for contributors would be the addition of a new place in the issue summary to contain these workspaces. For most issues we anticipate a single workspace, as most issues are resolved via collaboration on a single code path, however we would be able to generate multiple workspaces per issue for multiple solutions being proposed in parallel:

Code workspace in issue summary

The key functions of the Proposed code workspace are: the ability to make a new workspace, the ability to clone an existing workspace, the ability to create/view/edit a workspace merge request. The latest test result and ad-hoc testing options for each workspace would be provided here as well.

Mockup: Propose new Code

The "Propose new code" button is very simply a way to automatically generate a new workspace. This would only be used when a contributor wants to propose an alternate solution to whatever is currently under development. When pressed, we would generate a new workspace with the back-end tooling provider (whether as a fork, branch, or Git namespace) and add it to the table to be cloned, viewed for comments/inline editing, or ultimately to create a new merge request.

A second workspace in the issue summary

The new workspace will be generated with a name based on the issue id, and will present its own test result, clone, and merge request features.

If a user wants to collaborate on an existing solution instead, they can simply clone that workspace locally, or view it to make inline edits. Because some issues can languish for months, or even years, we also want to automate the process of rebasing these workspaces from their parents.

Mockup: Automatic comments as workspaces are updated

Whenever a change is made within the third party toolset, we would use the API to call back to Drupal.org and leave a system message describing the change, as well as linking to the relevant part of the third party toolset UI.

This is what an automated issue comment for code review might look like:

hestenet commented on 2893061-0 - "This is just a test comment" Read more...

This is what a comment for recently pushed changes would look like:

hestenet pushed code to: 2893061-0: View the diff This is a commit messag. Read more...

And this is what a comment for inline edits could look like:

hestenet used the inline editor to update 2893061-1: drupal/core/modules/update/update.api.php - View the diff

Mockup: Clone

The clone button would simply provide a basic modal pop-up with instructions for collaborators to clone the workspace locally with Git. This is not the final url pattern (one of our goals is to avoid changes in canonical git urls where possible) but it could look something like this:

$ git clone ssh://[email protected]/is/drupal.git

Mockup: Create Merge Request (in Bitbucket)

Merge requests allow contributors to propose changes to projects they don't maintain, and provide maintainers an easy way to view proposed changes.

The create merge request button would allow the user to view the current workspace and request that it be merged into the canonical parent. Only a project maintainer would have the permissions to complete the merge.

Create pull request

Code in a workspace with an open pull request can continue to be iterated on by the contributors. Code comments, diffs, and reviews are summarized on the request.

Mockup: View Merge Request (in Bitbucket)

View existing merge request

View merge request, like 'create merge request' would allow any collaborator to view the current workspace. This could also be used to initiate inline code edits, or leave code comments.

Mockup: Inline editing

Inline editing allows a quick and easy way for people to propose "standalone" changes to e.g. documentation, markup, etc. without having to have an entire development stack.

We would rely on the third party tooling provider's inline editing functionality. In the example below, we can see how an inline edit is made using Bitbucket:

Inline editor

Mockup: Code review

As with inline editing, we would rely on the third-party tooling provider's tools for code review. In the example below you can see the code review options provided by Bitbucket:

Code review tools and comments

As indicated above, these mockups are simply an illustration of what this integration could look like, not final designs. However, this has hopefully shown how we can introduce a hybrid model to integrate the best features of a third party tooling provider (pull requests, code review, inline editing), while retaining the essential nature of the drupal-flow.

Dec 20 2017
Dec 20

This is the second post in our series about integrating Drupal.org with a 3rd party developer tooling provider:

In this update we'll talk about the specific options we've been evaluating to improve the tools for developers on Drupal.org. For each of these options we've built prototype workflows, stood up test integrations, and opened a dialogue with the creators of these toolsets about any gaps we identified. You'll see a summary of how each tooling option does or does not live up to the criteria we outlined in our last post.

Issue Workspaces

At DrupalCon Los Angeles in 2015, Mixologic from the DA Engineering team proposed the concept of Issue Workspaces. While we've historically talked about this idea as a single concept, it actually breaks down into two key components:

  1. The definition of a modern, idealized workflow for the Drupal project.
  2. A technical implementation using a bespoke tooling solution built on Git Namespaces to implement this workflow.

Criteria (for a bespoke implementation):

  • Familiar workflow.
    • ❌/ While an implementation of workspaces based on a bespoke git-namespaces implementation has several technical advantages, it will not appear nearly as familiar to outside developers as a more standard 'pull request' implementation. (Though it would be functionally very similar).
  • Preserve the most valuable parts of Drupal's workflow.
    • By implementing a bespoke solution we could exactly tailor the solution to the needs of the Drupal community.
    • ❌ However, the needs of that community are great, and our capacity to build a 100% bespoke solution that meets all the community's needs is not clear.
  • Leverage a partner who will keep evolving their code collaboration featureset.
    • ❌ Building issue workspaces using a bespoke git namespaces implementation would require us to continuously compete on features with major third party tooling providers that have much greater resources.

A purely bespoke implementation misses the mark on two of our three primary criteria. Fortunately, however, two years after the original proposal, it is no longer necessary to build a bespoke solution in order to implement an idealized workflow for the Drupal project. It is something that can be done by integrating the existing Drupal.org project pages and issue queues with third-party tooling solutions. Leveraging third party solutions provides several advantages including: familiarity to a wider audience of developers, a continually evolving featureset, and mitigation of the risks of maintaining a bespoke solution with a small team of engineers.

Here is the idealized 'Drupal-flow' visualized:

Drupal Flow - the interaction between issues and workspaces

  • Every issue should be associated with one or more canonical workspaces (git repos+merge requests) which can be collaborated on by any contributor, and merged into the canonical parent by a project maintainer.
  • Contributors should be able to modify the code in these workspaces by:
    • Checking out the code and committing/pushing changes to the workspace.
    • Inline editing files within a workspace.
    • Legacy: uploading a patch.
    • Any contribution method should trigger the appropriate tests.
  • Workspaces should be rebased (manually or automatically) on any changes to the canonical parent.

This workflow is not dissimilar to the default GitHub flow, GitLab flow, or even the suggested Bitbucket flow. These are all variations of the generic concept of 'Git flow' created by Vincent Driessen. However, the important difference comes in how each of these solutions tries to integrate a gitflow model into their issue/pull-request model.

The dominant model of these large third party tooling providers is not particularly collaborative. It encourages forks-of-forks, separates conversation about potential solutions onto multiple branch comments or merge requests threads, and generally caters to single developers working on individual proposed solutions, with one ultimately selected by the maintainer and the rest thrown away.

This is a key area in which the 'Drupal-flow' differs from the standard workflow that is implicitly enforced by these toolsets. Our ethos encourages collaboration of many developers on a single solution, and a single threaded conversation.

The gitflow behind the scenes may be identical, but the user experience wrapped around that flow makes very different assumptions about how people will collaborate.

GitHubGitHub

GitHub is of course the dominant player in the open-source tooling space. For many years though, their toolset was very much targeted at smaller open-source projects, and their default workflow highly encourages a many-to-many, forks-of-forks workflow, rather than the many-to-one single-threaded collaboration that is part of the Drupal community's ethos. More recently, GitHub has been improving their featureset, providing better issue management tools, more control over organization-level namespaces, and a new app marketplace that allows developers to extend GitHub's core featureset.

We reached out to GitHub and spoke with members of the technical and partnership teams, and while they were excited by the idea of bringing a project like Drupal to GitHub, we did run into some unresolvable blockers - most critically, by policy GitHub does not allow users of GitHub Enterprise to run their own public instances. This means we would have to migrate Drupal projects to GitHub.com, rather than self-hosting our instance.

Criteria:

  • Familiar workflow
    • GitHub is unquestionably the dominant platform for code collaboration on the web today.
  • Preserve the most valuable parts of Drupal's unique workflow
    • ❌ No way to enforce Drupal.org as the home of projects. While we can still keep project pages on Drupal.org there is nothing to encourage visitors who find the project on GitHub to treat the Drupal.org project page as canonical.
    • ❌ ;No way to enforce our workflow - forks of forks of forks of Drupal core into personal namespaces would be one click away.
    • ❌ Issue management tools are not setup for large, crowd-sourced teams.
      • Even for large community projects, it's more typical for only one or two developers to work on a single github issue at a time. Collaboration tends to be divided across issues, rather than within them.
    • ❌ Hard blocker Changing issue metadata requires maintainer permissions. (i.e: No way for non-maintainer collaborators to set an issue to RTBC)
    • ❌ Limited ability to manage project maintainers, especially in the case of abandoned or insecure projects.
  • Leverage a partner who will keep evolving their code collaboration featureset.

Other issues:

GitHub is a no-go.

GitLabGitLab

GitLab is the up-and-comer, and is becoming exceptionally popular in the open-source space - even beginning to be adopted by larger open source projects such as GNOME and Debian.

We've worked closely with the GitLab team as well. In contrast to GitHub, GitLab's terms would allow us to run our own instance of the service. GitLab's fundamental workflow is very much like GitHub's - a one-to-many model of forks of forks. After discussing our workflow with their engineers GitLab has begun implementing their new 'fork network' feature, allowing merge requests across forks. However, this doesn't quite achieve the goal of allowing our many-to-one collaboration model just yet.

GitLab is very close to being a good solution for an Open Source project of Drupal's scale, and the rate at which they are improving the service is impressive. GitLab is also already a favorite of many in our community for personal or professional projects. Given some more time, it may ultimately be able to tick all the boxes.

Criteria:

  • Familiar workflow
    • GitLab's workflow is very similar to GitHub's and it is the second most widely used code collaboration toolset.
  • Preserve the most valuable parts of Drupal's unique workflow
    • ❌ Hard Blocker - Maintainers cannot yet collaborate on downstream merge requests
    • ❌ Monolithic toolset - GitLab's featureset is designed to be used in an all or nothing way. There is no provision for using specific elements while disabling others.
      • For example, there is no way to disable GitLab project pages in order to keep the canonical project pages on Drupal.org.
    • No plugin architecture - changes must be accepted as merge requests upstream in GitLab, or re-rolled as patches with each release (aka, "hacking core").
      • No easy way to turn on or off elements of the GitLab UI.
      • No way to extend the GitLab featureset.
      • Very limited issue management states, and no ability to customize them.
  • Leverage a partner who will keep evolving their code collaboration featureset.
    • GitLab's roll-out of new features is the fastest paced of the tooling providers that exist today.

Other issues:

  • License/Source availability
    • / GitLab Community Edition is open source, but GitLab Enterprise Edition (which we would need for a project of our scale and with our workflow needs) is not open source, only source-available.
  • API availability
    • GitLab has a fairly robust api, but in our evaluation we found that many api features were undocumented, and others were deprecated without notification between minor versions.
  • Hybrid integration
    • A hybrid integration between Drupal.org and GitLab is possible with SSO, automatic repository syncing, and a choice of using our issue queue or theirs.
    • ❌ However, there would be no portability of issues across a project that uses GitLab to one that does not.
    • ❌ Projects choosing to use the GitLab issues would not have DrupalCI integration, or issue credits.
  • Data portability/vendor lock-in
    • GitLab's API and documented data model should make it possible to migrate our data to another system should it become necessary.
  • Developer support
    • As an open-source project, GitLab has a growing community of contributors, as well as the support of a venture funded mothership organization.
  • Maintainability for an engineering team of limited resources
    • ❌ Hard Blocker - GitLab's current file-handling makes full copies of a repository for every fork. Taking the number of open core issues as a representative example, Drupal.org would need a new 200 TB+ redundant storage apparatus, just to accommodate Core. GitLab needs to transition to a shared git-object model, or an alternative like git namespaces.
  • Cost
    • The GitLab team is willing to provide the Enterprise Edition to the Drupal community for free.
  • Unintended consequences

GitLab: not-yet.

BitbucketBitbucket

Bitbucket was the dark horse candidate. Atlassian products are well known in software development circles, but they are often more tolerated than beloved. However, when we sat down at Midwest Drupal Summit in August and looked at the options we explored so far and found them to be not quite baked - Bitbucket began to show some advantages as a solution.

For one thing, Bitbucket has recently put a heavy focus on user experience and collaboration features. It also has the most flexible permissions model for control over branch permissions, forking, and automatic rebasing of any of the toolsets we evaluated. Finally, it can be implemented modularly - allowing us to use the components that serve our workflow and disable the ones that don't.

Criteria:

  • / Familiar workflow
    • Bitbucket implements a very standard pull request workflow. It is not as widely popular as GitHub or GitLab, but should be easily learned by developers familiar with either of those systems.
  • Preserve the most valuable parts of Drupal's unique workflow
    • Of all the options we prototyped, Bitbucket is the only one which provides the flexibility needed to preserve key elements of our workflow - in particular the ability to cleanly use only the parts of Bitbucket that we need (specifically, merge requests, inline editing, and code commenting) without having to use issues or project pages that are less sophisticated than our existing solutions.
  • Leverage a partner who will keep evolving their code collaboration featureset.
    • Though it's flown under the radar, Atlassian has been making some significant improvements to Bitbucket from a feature and user experience point of view over the course of the past year.

Other issues:

  • License/Source availability
    • /❌ Bitbucket is not open-source, only source-available, though they have an open source licensing program.
  • API availability
  • Hybrid integration
    • Bitbucket is much more flexible in terms of choosing which features to use, and which to disable- allowing a fairly tight hybrid integration.
  • Data portability/vendor lock-in
    • / Bitbucket stores repositories as standard git objects on the file system, making it relatively straightforward in case of a future migration to another system.
  • Developer support
    • Bitbucket has the professional support of Atlassian, but does not have a robust open source community driving feature development.
  • Maintainability for an engineering team of limited resources
    • Bitbucket is implemented with an api-first methodology, and is well documented, making it straightforward for a relatively small team to support a selective integration such as what we propose here.
  • Cost
  • Unintended consequences

Bitbucket: the best tool in the current landscape given our community's needs.

To sum up, each of the toolsets we evaluated have advantages and disadvantages. However, if we want to move forward with making improvements for our developers TODAY, we have determined that the best way to do that is to create modular integration points, and use the toolset that currently meets our criteria: Bitbucket. As these toolsets continue their rapid development, we will continue to periodically re-evaluate them.

Dec 20 2017
Dec 20

The initiative to improve Drupal.org's developer tools is part of a broader effort to broaden the reach of Drupal, not just to end-users and evaluators, but to a wider audience of developers as well. Improvements to our tools are an opportunity to remove friction to changes, increase the quality of changes, improve velocity of changes, and make contributing to Drupal delightful.

The initiative began in coordination with Drupal Association Staff, the Technical Advisory Committee, and a small group of volunteers from the community. We first announced the initiative in October of 2016, and provided our last update at the end of April of this year. Since that time a great deal of work has been underway to evaluate and prototype our options, collaborate with the vendors who supply these toolsets, and make some decisions about the direction we want to move in, with more data in hand.

Buckle up! Because this is such a large topic we've broken it into multiple posts:

Want a TLDR?

Where do we stand now?

Drupal is one of the longest-running open source projects on the web, and for more than 15 years (and many more to come), Drupal.org has been the home of the project. Over the years, we've built a developer toolset to serve the unique needs of the project. These tools have evolved as the open source environment changed. Right now, we use a combination of best-of-breed third party technologies, such as Git, Jenkins, Fastly, OpsGenie, integrated with our own bespoke tools, including issue queues, project pages, etc.

Right now, some of the third party and bespoke tools that we are using are market leading, whereas others have fallen behind. For example, CGIT (third party) and the patch workflow (bespoke) have both fallen behind compared to toolsets that can be found on other tooling providers. On the other hand, our issue crediting system (bespoke) is market leading, and a model for other projects to follow.

In the end, we will always be negotiating a balance between what we can do uniquely well with bespoke solutions, and integrating the latest and greatest of third party solutions as they coalesce around best practices that we want to adopt as well.

At DrupalCon Vienna it was decided that the Technical Advisory Committee had fulfilled their threefold mandate:

  • Helping the technical leadership transition on the DA engineering team after the downsizing in summer of 2016.
  • Helping manage the prioritization and scope of work.
  • Making recommendations to the board and DA staff on key initiatives, such as the project application process changes, and the direction to take with our developer tools.

We want to thank Angie Byron, Steve Francia, and Moshe Weitzman for serving on the committee. From here, DA engineering staff will carry the torch, though we will continue to rely on the individuals who were part of the TAC and other volunteers for feedback and help from time to time as we move forward.

Our evaluation

The most recent phase of this initiative was an evaluation of the leading contenders for open source tooling providers that Drupal.org could integrate with. These options were GitHub and GitLab, and after setting out to develop prototypes for these options, we added BitBucket to the list as well. For each of these options we built MVP prototypes, either as integrations with Drupal.org development environments, or with private organizations/repositories. Finally, we wanted to compare these options to the bespoke Issue Workspaces solution that we had proposed several years ago.

The Criteria

For the Developer Tools initiative to be successful we have to understand the criteria for improvement. From a user requirements point of view, contrib and core developers are unique stakeholders with unique requirements. Ideally we want a solution that increases velocity for both types of users, without fundamentally sacrificing the needs of one or the other.

  • Adopt a developer workflow that will be familiar to the millions of developers outside our community.
  • Preserve those unique elements of how we collaborate that have made the Drupal project so successful.
    • Many-to-one collaboration: that is to say, many developers collaborating on a single solution to a problem.
    • Issue workflow.
    • Picking up on long standing issues where other collaborators left off.
    • Contribution credit.
  • If possible, leverage an expert partner who will help keeping our tooling up to date as open source collaboration tools continue to evolve.

There are also some technical requirements that came out of our evaluation process.

  • Retention of our data/ability to migrate.
    • Where possible, retain existing Git remote urls for projects.
  • Maintainability for a small staff.
  • Project maintainer management: including abandoned project reassignment, fork control, security release management, etc.

In terms of features, we were looking for:

  • Merge/pull requests.
  • Code review.
  • Inline editing.
  • Branch permissioning to allow collaboration on merge/pull requests.
  • Administrative tools for managing project maintainership.
  • Project management tools that equal or exceed what we have with the issue queues.*
  • Extensibility, so that we can preserve areas where the Drupal project is a market leader, such as with our contribution credit system.

*Our tools for issues are very sophisticated on an individual issue level, however we are sorely lacking in tools for grouping and prioritizing sets of issues, ie: issue boards.

Finally, cost:

  • Given the care with which we must use our funding from the community, any option we consider must be cost neutral with the current cost of maintaining our tools.

It's difficult to condense the scope of our evaluation into a short blog series, but in the following posts we'll talk about how the options we considered measured up to these criteria, what implementing one of these options could look like for Drupal.org, and our suggested implementation roadmap.

Dec 12 2017
Dec 12

Read our Roadmap to understand how this work falls into priorities set by the Drupal Association with direction and collaboration from the Board and community.

Announcements

Clarifications to Drupal licensing policy

There have been some long-standing questions about Drupal project licensing policy. In collaboration with the Licensing Working Group and Dries, we have updated the official licensing policy with the following clarifications and changes:

  • We explicitly affirm that code with a GPL-compatible license can be included in a Drupal.org hosted project, but will be redistributed under our standard "GPL2 or later" policy.
  • We clarify that GPL-incompatible non-code assets may be packaged and/or distributed "in aggregate" with GPL code in Drupal.org projects, per the final stipulation of Sec 3-2 of the GPL, so long as the maintainer has the rights to do so.
  • We clarify that Drupal.org hosted projects can depend on and/or link to GPL-incompatible code (via composer, for example), but that Drupal.org cannot host or distribute those GPL-incompatible dependencies.
  • We explicitly affirm our interpretation of the GPL that a Drupal service provider's act of assembling a codebase while under contract to a client does not fall under the more restrictive terms of 'distribution' per the GPL.

You can find more detail about these changes in the Repository Usage Policy and Licensing FAQ.

Welcoming new staff members

Brooke CandelariaWe want to welcome two new members to the Drupal Association team. Brooke Candelaria who will be taking over as Conference Director, joins us from Houston, TX. Brooke has a background in PR and event management, and has worked with other open source communities in the past, for example working on LinuxWorld and, most recently, with the Python community. Brooke's focus will be on evolving DrupalCon to meet the needs of every persona within our community, and helping to make the Con more sustainable and scalable.

Rachel Lawson in MoldovaRachel Lawson is taking on the role of Community Liaison. Based in the UK, Rachel has been a tremendous member of the community in her own right, participating in Camps, helping to organize Mentored Sprints, and serving on the Community Working Group. Rachel will be helping the community to understand the role and the functioning of the Drupal Association, while also keeping the Drupal Association more closely tied to all parts of our diverse community. We hope you'll welcome them!

DrupalCon Updates

DrupalCon | Be Human, Think DigitalLaunched new DrupalCon brand

We've just launched a new unified look and feel for DrupalCon, that will carry into all of our events moving forward. This will help give DrupalCon a stronger identity among OSS events, and also give us a place to put centralized content that applies to DrupalCon as a whole, rather than just a singular event.

Launched DrupalCon Nashville site

Alongside our new brand, we've launched the DrupalCon Nashville website, the first of the events to use the new unified look and feel. Registration is open now, as well as the call for papers. Take a look at some of the new tracks! \

Nashville guitar

We'll see you there!

Sponsorship makes DrupalCon possible - learn why you should be a part of it.

Drupal.org Updates

Reminder: New issue shortcuts and friendly url structure

Last month we announced that we would be implementing some changes to the issue url structure, as well as some shortcuts to help users navigate to issues more easily on Drupal.org. These changes went live in November. Here's a primer:

URL Pattern for issues:

https://drupal.org/project/drupal/issues/2922626 When an issue is moved between projects the alias will be updated.

Shortcuts

The search bar will now automatically redirect you to a node if you enter its id directly: Search Bar Node ID shortcut A new menu callback will help you get to issues with a shorter url string: https://drupal.org/i/

New individual member directory

We've been overhauling the Drupal Association membership experience, and as part of that process we now have a new directory of Drupal Association members. This new directory also includes new filters and sorting options so that you can see the latest community members to join the Association and filter by country, username, or first and last name.

In development: new organization member directory

We're working on a new directory of organization members as well. The current directory allows you to filter by organization type, and we will be adding a filter for Association membership as well. If you have feedback on this directory, please let us know!

———

As always, we’d like to say thanks to all the volunteers who work with us, and to the Drupal Association Supporters, who make it possible for us to work on these projects. In particular we want to thank:

If you would like to support our work as an individual or an organization, consider becoming a member of the Drupal Association. Follow us on Twitter for regular updates: @drupal_org, @drupal_infra

Nov 17 2017
Nov 17

Read our Roadmap to understand how this work falls into priorities set by the Drupal Association with direction and collaboration from the Board and community.

Announcement

New issue shortcuts and friendly url structure

Drupal contributors have been managing bug fixes, feature requests, and code reviews on Drupal.org for around 15 years now. Passing an issue node id around a sprint table is a stable of DrupalCon and camps around the world. In October we announced that we would be implementing some changes to the issue url structure, as well as some shortcuts to help users navigate to issues more easily.

URL Pattern for issues:

https://drupal.org/project/drupal/issues/2922626
When an issue is moved between projects the alias will be updated.

Shortcuts

The search bar will now automatically redirect you to a node if you enter its id directly: Search Bar Node ID shortcut

A new menu callback will help you get to issues with a shorter url string:
https://drupal.org/i/<nid>

And of course you can still use the old https://drupal.org/node/<nid> urls if you still have them bookmarked.

Spoiler alert: These shortcuts and new url patterns were deployed in November and you can use them right now!

Drupal.org updates

Composer instructions on release pages

To make it easier for site builders to figure out how to use a release with composer we've added the composer command line instructions to release pages.

Composer instructions on release nodes

This command installs the package with the current release number specified as a minimum version parameter. We also provide a link to the documentation on using Composer to manage Drupal site dependencies, to help users who may be unfamiliar with Composer learn how to use it.

The Community Section

The community is the heart of the Drupal project, but until now community news has not had it's own place to live. We've now made the community page a proper section with its own blog, so that community posts and CWG information has a dedicated place to live.

When the first posts in this new section go live, we'll add this blog feed to Drupal Planet as well. Over time, we hope to further refine the community section and improve the tools we provide for the community to connect with each other.

WYSIWYG for Forums (CKEditor)

We're always looking for ways to make improvements to the site that have a high impact to effort ratio. One such change was enabling the CKEditor for editing in the forums. CKEditor has been in the wild as a WYSIWYG editor on Drupal.org for other content types for quite a while now, and we felt confident it was ready for use on forums as well.

CKEditor WYSIWYG for the Forums

Bug-fix: Dev releases on project pages

In the runup to DrupalCon Vienna we made a number of improvements to project pages - however a bug or two crept in as well. A race condition was causing dev releases not to display in some cases, and we resolved this issue in October. If you're a project maintainer on Drupal.org and see anything else go missing, please let us know!

Dev Releases

Infrastructure

DrupalCI: Faster, more affordable testing

DrupalCI uses spot requests on Amazon Web Services to spin up testbots on-demand for the Drupal project. In the past, instances were provisioned in minimum increments of one hour, meaning to make the most of testing we had to queue up tests to reuse the remainder of any paid-for instance-hours.

Because AWS has enabled per-second billing, we no longer have to try to fill instance-hours, and so we have reconfigured our spot instance requests to provision testbots faster, while still saving money overall compared to the previous configuration.

DrupalCI: More efficient RTBC testing

We also discovered that a bug in our automated RTBC retesting system was triggering more tests than necessary. We've fixed the bug, and now only the most appropriate recent test/environment will be retested for RTBC issues.

Server Maintenance Windows

Finally, we scheduled several maintenance windows in cooperation with our infrastructure services partner to schedule updates/restarts of our servers.

If you want to keep up-to-date with Drupal.org-related changes and maintenance windows you can subscribe to our Change Notices, or follow us on Twitter.

https://t.co/57fE1VZHTn db maintenance window is complete. We may schedule a follow up maint window, and will notify here if needed.

— Drupal infra (@drupal_infra) October 31, 2017

———

As always, we’d like to say thanks to all the volunteers who work with us, and to the Drupal Association Supporters, who made it possible for us to work on these projects. In particular we want to thank:

If you would like to support our work as an individual or an organization, consider becoming a member of the Drupal Association.

Follow us on Twitter for regular updates: @drupal_org, @drupal_infra

Nov 10 2017
Nov 10

Drupal developers have been shouting node id numbers across tables at sprints for almost 15 years now. The Drupal Association has added some shortcuts and friendly urls for issues to make this a little easier.

Shortcuts

The first shortcut we added is a searchbar feature. If you enter any node id in the search bar you'll be automatically redirected to that node. This works for more than just issues!

Searchbar Node Shortcut

The second shortcut we added is a quick url pattern for finding issues. If you know your node id, just type:

https://drupal.org/i/<node-id>

 and you're there!

Friendly-Urls - Updated 2017-11-12

We've implemented a new friendly url pattern on Drupal.org to make the canonical urls more sensible and search engine friendly.

Project pages:  Unchanged https://drupal.org/project/drupal

Issue Queue:  Unchanged (for now*) https://drupal.org/project/issues/drupal

UPDATED - Specific Issue: https://drupal.org/project/drupal/issues/2922626

This new pattern should be backfilled to all issues by the beginning of next week.

And of course, if you have old bookmarks linked, you can still get there by going to: https://drupal.org/node/<node-id>

These are small, but powerful improvements that should make the lives of everyone who contributes to the Drupal project easier.

_________________
UPDATED - *After this change to specific issue urls will we create a follow up to change issue queue urls to match the pattern: https://drupal.org/project/drupal/issues

Oct 17 2017
Oct 17

Read our Roadmap to understand how this work falls into priorities set by the Drupal Association with direction and collaboration from the Board and community.

DrupalCon Vienna

We're back from DrupalCon Vienna, with updates on what's new from the month of our European event.

Announcement

TLS 1.0 and 1.1 deprecated

Drupal.org uses the Fastly CDN service for content delivery, and Fastly has depreciated support for TLS 1.1, 1.0, and 3DES on the cert we use for Drupal.org, per the mandate by the PCI Security Standards Council. This change took place on 9 Aug 2017. This means that browsers and API clients using the older TLS 1.1 or 1.0 protocols will no longer be supported. Older versions of curl or wget may be affected as well.

Drupal.org updates

DrupalCon Calendar syncing

In our last update, we teased a new feature for DrupalCon attendees - the ability to sync your personal schedule to a calendar program. We're pleased to report that this feature made it in time for the event, and was used by attendees throughout the week. If you've already synced your calendar for DrupalCon Vienna, you're already set up to use the same feed for DrupalCon Nashville next April!

Keynote simulcast to Youtube

This year at DrupalCon, in addition to live streaming on Events.Drupal.org itself, we simulcast the keynotes to YouTube. We also embedded the keynote on the Drupal.org homepage - to spread the latest news about Drupal beyond DrupalCon attendees.

In fact, if you couldn't attend DrupalCon or just missed the keynotes, you can watch Dries' update on the Drupal project here:

[embedded content]

Industry Pages promoted in the front page Call-to-Action

We've also made some updates to how the industry pages are promoted. In addition to the dedicated block with icons linking to each industry, we now also promote the industry solutions landing page in the main CTA under the homepage header.

Industry Page CTA

We hope to further encourage users evaluating Drupal to explore some of the tremendous solutions that are already out there, and take inspiration from their success.

First-in/First-out issue sorting

To make sure that issues are reviewed by maintainers in the order they are received, it is now possible to sort the issue queues by when the issue status last changed. This means RTBC issues can be reviewed on a first-in/first-out basis!

This 'status changed' date field is available on the advanced search view for any issue queue. Here's what it looks like for Drupal core:

Issue Status Sort

Project creation analysis

About six months ago we opened up project creation on Drupal.org to allow any confirmed user to create a full project. We've put together a blog post outlining the impact these changes have had on the contrib landscape. In short, we've seen a tremendous increase in the rate of project creation, and the rate of applications for security advisory coverage, and a modest increase in projects receiving stable releases without yet opting in coverage. We're continuing to monitor project creation and work with the Security Working Group and others on next steps.

Displaying orphan dev releases

In last month's update we talked about a variety of changes we made to project pages, to provide better signals about project quality to evaluators. In response to feedback, we've restored the visibility of dev releases, even when they aren't associated with a tagged release.

Dev releases

This is particularly helpful for project maintainers trying to bring visibility to the next major development version of their modules, such as their Drupal 8 module port efforts.

———

As always, we’d like to say thanks to all the volunteers who work with us, and to the Drupal Association Supporters, who made it possible for us to work on these projects. In particular we want to thank:

If you would like to support our work as an individual or an organization, consider becoming a member of the Drupal Association.

Follow us on Twitter for regular updates: @drupal_org, @drupal_infra

Oct 07 2017
Oct 07

About six months ago we made a significant change to the way that modules, themes, and distributions are created on Drupal.org.

In the past, contributors had to first create a sandbox project, and then request manual review of their project in the Project Applications issue queue. The benefit of this community-driven moderation process was that modules were vetted for code quality and security issues by a group of volunteers. Project maintainers who completed this process also received the benefit of security advisory coverage from the Security Team for stable releases of their projects.

Unfortunately, the rate of project applications outpaced what volunteers could keep up with, and many worthy projects were never promoted to full project status, or moved off of Drupal.org to be hosted elsewhere.

To ameliorate this issue, we changed the process so that any confirmed user on Drupal.org may now make full projects.

To mitigate the risks of low code quality or security vulnerabilities we added new signals to project pages: including highlighting which release is recommended by the maintainer, displaying recent test results, and indicating whether the project receives security coverage both on the project page and in the composer 'extra' attribute. We're continuing to work on identifying additional signals of project quality that we can include, as well as surfacing some of this information in Drupal core. We also converted the project applications issue queue into a 'request security advisory coverage' issue queue.

What we hoped to see

We knew this would be a significant change for the project and the community. While many community members were excited to see the gates to contribution opened, others were concerned about security issues and Drupal's reputation for code quality.

Our prediction was that the lower barrier to contribution would result in an increase in full projects created on Drupal.org. This would indicate that new contributors or third party technology providers were finding it easier to integrate with Drupal and contribute those integrations back for use by others.

At the same time, we also expected to see an increase in the number of full projects that do not receive coverage from the security team. The question was whether this increase would be within an acceptable range, or represent a flood of low quality or insecure modules.

The results

The table below provides statistics about the full projects created on Drupal.org in the 5 months before March 17th, 2017 - when we opened the creation of full projects to all confirmed users.

Full projects created from 2016-10-16 to 2017-03-17…

#

% of projects created in this period

… without stable release

431

55.76%

… with stable releases

342

44.24%

… with usage >= 50 sites

237

30.66%

… with usage >= 50 sites and without stable release

68

8.80%

… with usage >= 50 sites and with stable release

169

21.86%

… with an open security coverage application*

18

2.33%

Sub-total with security coverage

342

44.24%

Sub-total without security coverage

431

55.76%

Sub-total with security coverage and >=50 usage

169

21.86%

Sub-total without security coverage and >= 50 usage

68

8.80%

Total

773

* note: full projects that did not have stable releases were not automatically opted in to security coverage when we opened the full project creation gates.

… and this table provides statistics about the projects created in the 5 months after we opened the creation of full projects to all confirmed users:

Full projects created from 2017-03-17 to 2017-08-16…

#

Diff

% of projects created

Diff %

… without stable release

851

+420

69.53%

+97%

… with stable releases

373

+31

30.47%

+9%

… with usage >= 50 sites

156

-81

12.75%

-34%

… with usage >= 50 sites and without stable release

64

-4

5.23%

-6%

… with usage >= 50 sites and with stable release

92

-77

7.52%

+46%

… with an open security coverage application

62

+44

5.07%

+344%

Sub-total with security coverage

182

-160

14.87%

-53%

Sub-total without security coverage

1,042

+611

85.13%

+242%

Sub-total with security coverage and >=50 usage

54

-115

4.41%

-32%

Sub-total without security coverage and >= 50 usage

102

+34

8.33%

+150%

Total

1,224

+451

+58%

As you can see, we have an almost 58% increase in the rate of full projects created on Drupal.org. We can also see a significant proportional increase in two key areas: projects with greater than 50 site usage and no security coverage(up 150% compared to the previous period), and projects that have applied for security coverage(up 344% compared to the previous period). Note: this increase in applications is for projects *created in these date ranges* not necessarily applications created overall.

This tells us that reducing friction in applying for security coverage, and encouraging project maintainers to do so should be a top priority.

Finally, this last table gives statistics about all of the projects currently on Drupal.org, regardless of creation date:

Full projects (7.x and 8.x)

#

% of Total

Rate of change after 2017-03-17

… with the ability to opt into security coverage

8,718

36.15%

-1.33%

… with security coverage and stable releases

8,377

34.74%

-1.49%

… without security coverage

15,396

63.85%

+1.33%

… without security coverage and with stable releases

464

1.92%

+1.04%

… with security coverage and >=50 usage
 

6,475

66.91 / 26.85%

-0.54%

… with security coverage and stable releases and >=50 usage

6,308

65.19 /26.16%

-0.65%

… without security coverage and >=50 usage

3,202

33.09 /13.28%

+0.54%

… without security coverage and with stable releases and >=50 usage

130

1.34 /0.54%

+0.51%

Sub-total with >=50 usage

9,677

40.13%

-1.72%

Total

24,114

From the overall data we see approximately what we might expect. The increase in growth of full projects on Drupal.org has lead to a modest increase in projects without security coverage.

Before the project application change, all full projects with stable releases received security advisory coverage. After this change, only those projects that apply for the ability to opt in(and then do so) receive coverage.

What has this meant for security coverage of projects hosted on Drupal.org?

1.92% of all full 7.x and 8.x projects have stable releases, but do not receive security advisory coverage. It is likely no accident that this translates into 464 projects, which is nearly equivalent to the number of projects additional projects added compared to our old growth rate.

Of those only 130 of those projects report more than 50 sites usage(or .54% of all 7.x and 8x full projects).

Next steps

From this analysis we can conclude the following:

  1. The opening of the project application gates has dramatically increased the number of projects contributed to Drupal.org.

  2. It has also increased the number of projects without security coverage, and the number of applications for the ability to opt in to coverage among new projects.

In consultation with the Security Working Group, we recommend the following:

  • For now, leave the project creation projects as it stands today - open to contribution from any confirmed user on Drupal.org.

  • Solve the problem of too many security advisory coverage applications. The security advisory application queue has the same problem that the old project applications queue had - not enough volunteers to manually vet all of the applications - and therefore a significant backlog of project maintainers waiting on the ability to opt into coverage.

    • Recommendation: Implement an automated best practices quiz that maintainers can take in order to be granted the ability to opt into security advisory coverage. If this process is as successful as we hope, we may want to consider making this a gate on stable releases for full projects as well.

We look forward to working with the Security Working Group to implement this recommendation and continue to improve the contribution experience on Drupal.org, while preserving code quality and security.

Sep 19 2017
Sep 19

Read our Roadmap to understand how this work falls into priorities set by the Drupal Association with direction and collaboration from the Board and community.

Announcement

TLS 1.0 and 1.1 deprecated

Drupal.org uses the Fastly CDN service for content delivery, and Fastly has depreciated support for TLS 1.1, 1.0, and 3DES on the cert we use for Drupal.org, per the mandate by the PCI Security Standards Council. This change took place on 9 Aug 2017. This means that browsers and API clients using the older TLS 1.1 or 1.0 protocols will no longer be supported. Older versions of curl or wget may be affected as well.

Almost time for DrupalCon Vienna

DrupalCon Vienna

DrupalCon Vienna is almost here! From September 26-29 you can join us for keynotes, sessions, and sprinting. Most of the Drupal Association engineering team will be on site, and we'll be hosting a panel discussion about recent updates to Drupal.org, and our plans for the future.

We hope to see you there!

Drupal.org updates

8.4.0 Alpha/Beta/Release Candidate 1

On August 3rd, Drupal 8.4.0 received its alpha release, followed on the 17th by a beta release, and on September 6th by the first release candidate. Several new stable API modules are now included in core for everything from workflow management to media management. Core maintainers hope to reach a stable release of Drupal 8.4 soon.

Improvements to Project Pages

We made a number of improvements to project pages in August, one of which was to clean up the 'Project information' section and add new iconography to make signals about project quality more clear to site builders.

Project information improvements

In the same vein, we've also improved the download table for contrib projects, by making it more clear which releases are recommended by the maintainer, providing pre-release information for minor versions, and displaying recent test results.

Download table improvements

Metadata about security coverage available to Composer

Developers who build Drupal sites using Composer may miss some of the project quality indicators from project pages on Drupal.org. Because of this, we now include information about whether a project receives security advisory coverage in the Composer 'extra' attribute. By including this information in the composer json for each project, we hope to make it easier for developers using Composer to ensure they are only using modules with security advisory coverage. This information is also accessible for developers who may want to make additional tools for managing composer packages.

Automatic issue credit for committers

Just about the last step in resolving any code-related issue is for a project maintainer to commit the changes. To make sure these maintainers are credited for the work they do to review these code changes, we now automatically add issue credit for committers.

Performance Improvements for Events.Drupal.org

With DrupalCon coming up in September we spent a little bit of time tuning the performance of Events.Drupal.org. We managed to resolve a session management bug that was the root cause of a significant slow down, so now the site is performing much better.

Syncing your DrupalCon schedule to your calendar

A long requested feature for our DrupalCon websites has been the ability to sync a user's personal schedule to a calendar service. In August we released an initial implementation of this feature, and we're working on updating it in September to support ongoing syncing - stay tuned!

Membership CTA on Download and Extend

We've added a call to action for new members on the Drupal.org Download and Extend page, which highlights some great words and faces from the community. Membership contributions are a crucial part of funding Drupal.org and DrupalCon, but much the majority of traffic we receive on Drupal.org is anonymous, and may not reach the areas of the site where we've promoted membership in the past. We're hoping this campaign will help us reach a wider audience.

Membership CTA on the Download page

DrupalCI sponsorship

DrupalCI is one of the most critical services the Drupal Association provides to the project, and also one of the more expensive. We've recently added a very small section to highlight how membership contributions help provide testing for the project - and in the future we hope to highlight sponsors who will step up specifically to subsidize testing for the Drupal project.

Infrastructure

More semantic labels for testing

In August we added more semantic labels for DrupalCI test configuration. This means that project maintainers no longer have to update their testing targets with each new release of Drupal, they can instead test against the 'pre-release' or 'supported' version, etc. More information can be found in the DrupalCI documentation.

Semantic Labels for Testing

Started PCI audit

In August we also began a PCI audit, and developed a plan of action to reduce the Drupal Association's PCI scope. Protecting our community's personal and financial information is critically important, and with a small engineering team, the more we can offload PCI responsibility onto our payment vendors the better. We'll be continuing to work on these changes into the new year.

———

As always, we’d like to say thanks to all the volunteers who work with us, and to the Drupal Association Supporters, who made it possible for us to work on these projects. In particular we want to thank:

If you would like to support our work as an individual or an organization, consider becoming a member of the Drupal Association.

Follow us on Twitter for regular updates: @drupal_org, @drupal_infra

May 24 2017
May 24

Read our Roadmap to understand how this work falls into priorities set by the Drupal Association with direction and collaboration from the Board and community.

DrupalCon Baltimore logo Apr 24-28

At the end of April we joined the community at DrupalCon Baltimore. We met with many of you there, gave our update at the public board meeting, and hosted a panel detailing the last 6 months worth of changes on Drupal.org. If you weren't able to join us for this con, we hope to see you in Vienna!

Drupal.org updates

DrupalCon Vienna Full Site Launched!

DrupalCon Vienna logo Sep 26-29 2017

Speaking of Vienna, in April we launched the full site for DrupalCon Vienna which will take place from September 26-29th, 2017. If you're going to join us in Europe you can book your hotel now, or submit a session. Registration for the event will be opening soon!

DrupalCon Nashville Announced with new DrupalCon Brand

DrupalCon Nashville logo Apr 9-13 2018

Each year at DrupalCon the location of the next conference is held as closely guarded secret; the topic of speculation, friendly bets, and web crawlers looking for 403 pages. Per tradition, at the closing session we unveiled the next location for DrupalCon North America - Nashville, TN taking place from April 9-13th in 2018. But this year there was an extra surprise.

We've unveiled the new brand for DrupalCon, which you will begin to see as the new consistent identity for the event from city to city and year to year. You'll still see the unique character of the city highlighted for each regional event, but with an overarching brand that creates a consistent voice for the event.

Starring Projects

Users on Drupal.org may now star their favorite projects - making it easier to find favorite modules and themes for future projects, and giving maintainers a new dimension of feedback to judge their project's popularity. Users can find a list of the projects they've starred on the user profile. Over time we'll begin to factor the number of star's into a project's ranking in search results.

Starring Projects

At the same time that we made this change, we've also added a quick configuration for managing notification settings on a per-project basis. Users can opt to be notified of all issues for a project, only issues they've followed, or no issues. While these notification options have existed for some time, this new UI makes it easier than ever to control issue notifications in your inbox.

Project Browsing Improvements

One of the important functions of Drupal.org is to help Drupal site builders find the distributions, modules, and themes, that are the best fit for their needs. In April, we spent some time improving project browsing and discovery.

Search is now weighted by project usage so the most widely used modules for a given search phrase will be more likely to be the top result.

We've also added a filter to the project browsing pages to allow you to filter results by the presence of a supported, stable release. This should make it easier for site builders to sort out mature modules from those still in initial development.

Better visual separation of Documentation Guide description and contents

Better Documentation Guide Display

In response to user feedback, we've updated the visual display of Documentation Guides, to create a clearer distinction between the guide description text and the teaser text for the content within the guides.

Promoting hosting listings on the Download & Extend page

To leverage Drupal to the fullest requires a good hosting partner, and so we've begun promoting our hosting listings on the Download and Extend page. We want Drupal.org to provide every Drupal evaluator with all of the tools they need to achieve success—from the code itself, to professional services, to hosting, and more.

Composer

Sub-tree splits of Drupal are now available

Composer Façade

For developers using Composer to manage their projects, sub-tree splits of Drupal Core and Components are now available. This allows php developers to use components of Drupal in their projects, without having to depend on Drupal in its entirety.

DrupalCI

Automatic Requeuing of Tests in the event of a CI Error

DrupalCI logo

In the past, if the DrupalCI system encountered an error when attempting to run a test, the test would simply return a "CI error" message, and the user who submitted the test had to manually submit a new test. These errors would also cause the issues to be marked as 'Needs work' - potentially resetting the status of an otherwise RTBC issue.

We have updated Drupal.org's integration with DrupalCI so that instead of marking issues as needs work in the event of a CI Error, Drupal.org will instead automatically queue a retest.

Bugfix: Only retest one environment when running automatic RTBC retests

Finally, we've fixed a bug with the DrupalCI's automatic RTBC retest system. When Drupal HEAD changes, any RTBC patches are automatically retested to ensure that they still apply. It is only necessary to retest against the default or last-used test environment to ensure that the patch will work, but the automatic retests were being tested against every configured environment. We've fixed this issue, shortening queue times during a string of automatic retests and saving testing resources for the project.

———

As always, we’d like to say thanks to all the volunteers who work with us, and to the Drupal Association Supporters, who made it possible for us to work on these projects. In particular we want to thank:

If you would like to support our work as an individual or an organization, consider becoming a member of the Drupal Association.

Follow us on Twitter for regular updates: @drupal_org, @drupal_infra

Apr 18 2017
Apr 18

Read our Roadmap to understand how this work falls into priorities set by the Drupal Association with direction and collaboration from the Board and community.

DrupalCon Baltimore logo Apr 24-28

The Drupal Association team is gearing up for DrupalCon Baltimore. We're excited to see you there and we'll presenting a panel giving an update on our work since Dublin, and our plans for the coming months.

Drupal.org updates

Project application revamp

As we announced in mid-March, new contributors on Drupal.org can now create full projects and releases! Contributors no longer have to wait in the project application queue for a manual review before they are able to contribute projects.

This is a very significant change in the Drupal contribution landscape, and it's something we approached carefully and will continue to monitor over the coming months. Drupal has always had a reputation for a high quality code, and we want to make sure that reputation is preserved with good security signals, project quality signals, and continued incentives for peer code review.

That said, we're very excited to see how this change opens up Drupal to a wider audience of contributors.

Please note that the removal of project applications to create full projects and releases means a change in the security advisory policy (see below for details).

Security Advisory Opt-in and new Security Signals for Projects

Are you responsible for the security of your clients' Drupal sites?

Please note that Drupal's security advisory coverage policy has changed. Security advisory coverage for contributed projects is now only available for projects that have both opted in to receive coverage and made a stable release. You can see which projects have opted in by checking their project pages. If you have questions, please contact [email protected].

Because users may now create full projects and releases without opting in to security advisory coverage, it's critically important that we provide good security signals to users evaluating projects on Drupal.org. This is why we've added a security coverage warning to projects that aren't opted in to coverage.

We've also:

2017 Community Elections Update

The 2017 elections for the community-at-large seat on the board were held successfully in March. Drupal Association community board elections are conducted with the Instant Runoff Voting system. This voting methodology requires that voters rank their preferred candidates on their ballot, and we've heard that this system has been somewhat unwieldy in the past.

Each year we try to improve the voter experience and so this year we deployed a new drag-and-drop ballot.

Drag and Drop Ballot

Finally, we want to congratulate our newest board member Ryan Szrama!

Better international datetime support throughout Drupal.org

Drupal.org has grown organically over the course of more than a decade, and as features have been built out they were not always consistent in their display of datetime information. While it sometimes makes sense to have a few different formats for displaying date and time, many of the formats in use were simply arbitrary historical decisions.

As a quality of life improvement, especially for users outside of the USA, we've standardized the datetime format used on Drupal.org. That format is: DD MMM YYYY - hh:mm (UTC±h). For example: 11 Aug 2016 - 16:42 (UTC+8)

DrupalCI

CSS Lint check style results

DrupalCI logo

When we implemented coding standards testing in DrupalCI in February we were not able to add CSS Lint testing until the CSSLint configuration file in core was fixed. That issue was fixed in late February and so we added CSSLint to support coding standards testing for CSS at the beginning of March.

Cleaning up coding standards results

The addition of coding standards results to DrupalCI means that Drupal.org is now storing even more test data about the code we test on Drupal.org. Our initial implementation of coding standards testing did not include clean up of older results, and so to preserve database space and testing resources, we implemented some clean-up routines in March. In particular we are now:

  • Cleaning up all results for closed issues
  • For custom one-off tests, keeping results for 30 days to match what is shown on project’s automated testing tab
  • For tests triggered on a schedule or commit, keeping the most recent per-environment per-branch, and keeping anything less than 24h old

Infrastructure

Protecting Git services

We experienced some minor Git outages in March, due to malicious authentication attempts. To mitigate these issues in the future, we've implemented fail2ban rules to protect Git authentication. This should improve the stability and uptime of Git services for all developers on Drupal.org.

We want to thank Drupal.org infrastructure volunteer mlhess for his assistance with this.

Community Initiatives

Contrib Documentation Migration

New tools for Documentation have been available on Drupal.org for more than half a year. While most of the core documentation has been migrated to the new system, we are still encouraging Contrib maintainers to migrate their docs.

To make it easier for contrib project maintainers to migrate their documentation to the new documentation tools, we've made two improvements:

———

As always, we’d like to say thanks to all the volunteers who work with us, and to the Drupal Association Supporters, who made it possible for us to work on these projects. In particular we want to thank:

If you would like to support our work as an individual or an organization, consider becoming a member of the Drupal Association.

Follow us on Twitter for regular updates: @drupal_org, @drupal_infra

Apr 11 2017
Apr 11

Workflow iconIn October of last year the Technical Advisory Committee was formed to evaluate options for the developer tools we use on Drupal.org. The TAC consists of Angie Byron, Moshe Weitzman, and Steve Francia, acting as advisors to Megan Sanicki, the Executive Director of the Drupal Association.

The TAC's mandate is to recommend a direction for the future of our tools on Drupal.org. Megan will evaluate this recommendation, make a decision, and prioritize that work in the development roadmap of the Drupal Association engineering team.

What is the motivation behind looking at our developer tools now?

Close followers of the Drupal project will have noticed a trend in the last several months. From Dries' announcement of easy upgrades forever, to the revamp of the project application process, to the discussion about making tools for site builders— there is a unifying theme: broadening the reach of Drupal.

This is the same motivation that underlies this evaluation of our developer tools, and defines the goals and constraints of this initiative:

  • Adopt a developer workflow that will be familiar to the millions of developers outside our community
  • Preserve those unique elements of how we collaborate that have made the Drupal project so successful
  • If possible, leverage an expert partner who will help keeping our tooling up to date as open source collaboration tools continue to evolve

This means looking at a number of elements of the Drupal.org developer tool stack:

  • The underlying git service
  • How we tag and package releases
  • The contribution workflow (patch vs. pull request)
  • Project management workflows (the issue queues and tags)
  • CI integration
  • Maintainership
  • Project pages

If this looks like a tremendous undertaking - that's because it is. But there are some things we already know:

  • Drupal.org should continue to be the home of project pages
  • We should adopt a pull request workflow (and ideally we want to be able continue to accept patches as well, at least in the interim)
  • We should move contrib projects to semver, following core's lead
  • We want to preserve our familiar understanding of maintainership
  • We want to avoid forked code and forked conversation
  • We want to ensure the security team still has the tools they need to provide their service to the community

We also know that whatever decision is made, these changes cannot happen all at once. We'll need to take a progressive approach to the implementation, and focus on the parts of the stack that need to change together, so that we don't bite off more than we can chew.

What options are being considered?

At this time, the technical advisory committee is considering three options as they prepare to make their recommendation. The options are: GitLab, which offers both self-hosted and SaaS options; GitHub, which has recently been adding long-requested new features; or continuing to evolve our custom-built tooling, perhaps via issue workspaces.

GitLab

GitLab is the up-and-comer among Git hosts. GitLab can be self hosted using either their community or enterprise editions, or repositories can be hosted at GitLab.com. Though they recently stumbled, they have been notably open and transparent about their efforts to build a leading collaboration platform.

Gitlab is itself open-source, and has just released its 9.0 edition. GitLab has aggressively pursued the latest in development tools and workflow features, including project management tools, a ui for merge conflict resolution with in-line commenting and cherry-picking, docker registries for projects, integration with CI tools, and even Gitter, an IRC alternative for real-time collaboration.

GitHub

For quite some time, GitHub was the only real player in git repository hosting outside of rolling a custom solution (as we did for Drupal.org). Over the years it has become the home of many open source projects, and while most of those don't match the sheer scale of Drupal in terms of codebase size and number of contributors, there are certainly other major projects that have made their home there.

However, for all of its presence and longevity in the open source world, there are very few options for customizing its toolset for our needs, and we could no longer self-host our canonical repositories. The Drupal project would need to adapt to GitHub, rather than the other way around.
 

That said, in recent months, GitHub has been putting a strong focus on feature development, adding a number of new features including: automated licensing information,  protected branches, and review requests.

Custom tooling

We also must consider that the tools we have now are what built Drupal into what it is today. A great deal of work has gone into our existing developer tools over the years, and we have some unique workflows that would have to be given up if we switched over to a tooling partner. An idea like the issue workspaces proposal would allow us to achieve the goal of modernizing our tools, and likely do so in a way that is better tailored to those unique things about the Drupal workflow that we may want to preserve. However, doubling down on building our own tooling would come at a cost of continuing to be unfamiliar to the outside development community, and dependent on an internal team's ability to keep up with the featureset of these larger players.

Each of these three options would be a compromise between reaching outward and creating familiarity, and looking inward to preserve the Drupal specific workflows that have brought the project to where it is today.

What have we learned so far?

The TAC has conducted their own internal evaluation of the options as well as worked with Drupal Association staff in a two day exploratory session at the end of last year. The primary focus was to identify and triage gaps between the different toolsets in the following areas:

  • Migration effort
  • Project management
  • Code workflow
  • Project handling
  • Testing
  • Git Back-end/Packaging
  • Integrations beyond tools

This initial study also looked at the impact of each option on Drupal community values, and some key risks associated with each.

What comes next?

The next step for the TAC is to make their formal recommendation to the Executive Director of the Drupal Association. At that point, she will direct staff to validate the recommendation by prototyping the recommended solution. Once the recommendation has been validated, Megan will make a final decision and prioritize the work to fully implement this option, relative to other Drupal Association imperatives.

In the comments below, we would love to hear from the community:

What aspects of the way the Drupal community collaborates are the most important to you? What workflow elements do you feel need to be preserved in the transition to any new tooling option?

Mar 17 2017
Mar 17

Any user on Drupal.org who has accepted our Git usage policy may now create full projects with releases. This is a big change in policy for the Drupal project, representing an evolution of the contribution ecosystem in the past half a decade.

What was the Project Application Process?

Ever since the days when Drupal's code was hosted in CVS there has been some form of project application process in the Drupal Community. To prevent duplicate, low-quality, insecure, or otherwise undesirable projects from flooding Drupal, users would submit sandbox projects to an application queue to be reviewed by a group of volunteers.

After resolving any issues raised in this review process, the user would be given the git vetted role, allowing them to promote their sandbox to a full project, claim a namespace, and create releases. Once a user had been vetted for their first project, they would remain vetted and be able to promote any future projects on their own, without submitting an additional application.

The Problem

Unfortunately, though the project application process was created with the best of intentions, in the long term it proved not to be sustainable. Drupal grew too fast for a group of volunteer reviewers to keep up with reviewing new projects, and at times there were applications waiting in queue for 6 months to 1 year, or even more. That is much too slow in the world of software development.

This put Drupal in a difficult situation. After years of subjecting new projects and contributors to a rigorous standard of peer review, Drupal has a well-deserved reputation for code quality and security. Unlike many open source projects, we largely avoided the problem of having many duplicate modules that exist to serve the same purpose. We unified our community’s effort, and kept up a culture of collaboration and peer review. At the same time, many would-be contributors were unable or unwilling to navigate the application process and so simply chose not to contribute.

The question became, how could we preserve the emphasis on quality while at the same time removing the barrier to contribution that the application process had become?

Constraints on a solution

Opening the contribution gates while retaining strong signals about code quality and security was a tricky problem. We established three constraints on a solution:

  1. We need to welcome new contributors, and eliminate the walls that prevent contribution.
  2. We need to continue to send strong signals about security coverage to users evaluating whether to use modules from Drupal.org.
  3. We need to continue our strong emphasis on quality and collaboration through changes to project discovery that will provide new signals about code quality, and by providing incentives and credit for peer review.

The Solution

In collaboration with the community, the security team, members of the board, and staff we outlined a solution in four phases:

Phase 1: Send strong signals about security advisory coverage.

  • We updated project pages to include messaging and a shield icon to indicate whether a project received security advisory coverage from the security team.
  • We now serve security advisory coverage information in the Updates status information provided by Drupal.org, and we're working on a patch to display that information directly on the updates page of users' Drupal sites.

Here are some examples of what these security signals look like on project pages:

If a project is not opted in to security advisory coverage, this message will appear at the top of the project page:

Warning at the top of Project pages

And this one will appear near the download table:

Warning above download table

If a project has opted in, this message will appear near the download table:

Project opt in notice

And covered releases will show the coverage icon (note how the stable 7.x release has coverage and the 8.x release candidate does not):

Release coverage icon

Phase 2: Set up an opt-in process for security advisory coverage

  • Previously any project with a stable release would receive security advisory coverage from the security team. As we opened the gates for anyone to promote full projects, the security team needed an opt in process so that they could enforce an extra level of vetting on projects that wish to receive advisory coverage.
  • We agreed to repurpose the project application queue to be a queue for vetting users for the ability to opt their projects in to receive security advisory coverage. Now that this process has been decoupled from creating full projects, the security team may revise it in future–in collaboration with staff and the community.
  • Now a project maintainer must opt in their project to receive advisory coverage and make a stable release in order to receive security advisory coverage from the security team.

Once a maintainer has been vetted by the security advisory opt in process, they can edit their project and use this field set to opt-in:

Project opt-in field

Phase 3: Open the gate to allow users to create full projects with releases without project applications.

This is the milestone we've just reached!

Phase 4: Provide both automated code quality signals, as well as incentives for peer review of projects - and factor these into project discovery

  • We are working on this phase of the project in the issue queues, and we appreciate your feedback and ideas!

What is the new process?

So in the end - what is the new process if you want to make a contribution by hosting a project on Drupal.org?

  1. You must have a Drupal.org account, and you must accept the git terms of service.
  2. You can create a sandbox or a full project
  • Note: We still strongly recommend that project maintainers begin with sandbox projects, until they are sure they will be able to commit to supporting the project as a full project, and until the code is nearly ready for an initial release.
  • That said, you can promote a sandbox project to a full project at any time, to reserve your name space and begin making releases.

At this point, you will have a full project on Drupal.org, and will be able to make releases that anyone can use on their Drupal site. The project will not receive security advisory coverage, and a warning that the project is not covered will appear on the project page and in the updates information.

If you want to receive security advisory coverage for your project, you will need to take these additional steps:

  1. You must apply for vetted status in the security advisory coverage queue.
  2. Members of the security team or other volunteers will review your application - and may suggest changes to your project.
  3. Once feedback is resolved, you will be granted the vetted role and be able to opt in this project, and any future projects you create, to receive security advisory coverage.
    • Note: Only *stable* releases receive security advisory coverage, so even after opting your project in you will not receive the advisory coverage shield except on stable releases.

What comes next?

Now that the project application process is no more, the gates are open. We are already seeing an uptick in projects created on Drupal.org, and have seen some projects that had migrated to other places (like GitHub) migrate back to Drupal.org. We can expect to see contributions from some great developers who previously felt gate-kept out of the community. We will also see an uptick in contributions that need work, from new developers and others who are still learning Drupal best practices.

That is why our next focus will be on providing good code quality signals for projects on Drupal.org. We want to provide both automated signals of code quality, and new incentives for peer review from existing members of the community. We're outlining that plan in the issue queues, and we welcome your feedback and contributions.

We also still have work to do to communicate this well. This is a big change for the Drupal community and so we want to make people aware of this change in every channel that we can.

Finally, after such a significant change, we're going to need to monitor the contrib ecosystem closely. We're going to learn a lot about the project in the next several months, and it's likely there will be additional follow ups and other changes that we'll need to make.  

Special Thanks

There are many, many contributors on Drupal.org who have put in time and effort to help make the contribution process better for new contributors to Drupal - the deepest thanks to all of you for your insight and feedback. We'd also like to specifically thank those who participated in the Project Application Revamp, including:

Mar 09 2017
Mar 09

Read our Roadmap to understand how this work falls into priorities set by the Drupal Association with direction and collaboration from the Board and community.

Drupal.org updates

Industry Pages Launched

After a great deal of preparation, user research, and content development we've launched the first three 'Drupal in your Industry' pages. These first three pages highlight the power of Drupal in Media and Publishing, Higher Education, and Government. Each of these pages uses geo-targeted content to reach audiences in: the Americas; Europe, the Middle East, and Africa; and the Asia Pacific, Australia and New Zealand regions.

These pages are targeted at evaluators of Drupal in these specific industries. From our research, we've found that these evaluators typically have Drupal on their short list of technology choices, but are not familiar with how a complete solution is built on Drupal, and they're eager to see success stories from their industry peers.

We'll be expanding on this initiative with additional industry pages as time goes on.

Project Application Revamp

In February we completed phases 1 and 2 of the Project Application Process Revamp. This has meant polishing up the security advisory coverage messages that are provided on project pages, adding a new field for vetted users to opt-in to advisory coverage for their projects, and adding security advisory coverage information to the updates xml served from Drupal.org. With these issues complete we'll be able to move forward with Phase 3 (opening the project promotion gates) and Phase 4 (improving code quality signals and incentivizing peer review) as we roll into March.

[Author's note] The project application revamp hit a major milestone in early March with the completion of Phase 3. Now, any user who has accepted the git terms of service may now promote sandbox projects to full projects with releases, and the application process has been re-purposed for vetting users who want the ability to opt into security advisory coverage for their projects. Look for more information in our upcoming March post.

2017 Community Elections are Live

On February 1 we opened self-nominations for one of the two community-at-large seats on the Drupal Association Board of Directors. At the time of this post, self-nominations have closed and now it's time to vote!.

Each year we make incremental improvements to the elections process. This year we've allowed each candidate to present a short 'statement of candidacy' video - and we've updated the ballot to allow easy drag-and-drop ranking of candidates.

Voting closes on March 18th, so make sure to vote soon!

Documentation polish, and new "call-out" templates

As the migration of content into the new documentation system continues, we've continued to polish and improve the tools. In February we made a few small improvements including: help text for maintainers and fixes for links to the discuss page in email notifications. We also made one large improvement: Call-out templates for highlighting warning information or version-specific notes within a documentation page. These templates are available using the CKEditor Templates button when editing any documentation page.

The documentation editor may select from the 'Warning note' template, which will highlight cautionary information in a visually distinct orange section on the page, or the 'Version-specific note' template, which allows users to highlight information that may only be relevant to a specific minor release of Drupal.

Here are two examples of what the call-outs will look like to a documentation reader.

DrupalCI

Coding standards testing

DrupalCI logo

DrupalCI continues to accelerate the pace of Drupal development as we make the system more efficient and add new features. In February we enhanced the coding standards testing that was added DrupalCI in January. Using PHPCodeSniffer, ESlint, and CSSlint coding standards results are available in the test results' Build Artifacts directory, including automatically generated patches to fix found issues. We've also begun displaying summary information about coding standards testing on Drupal.org test results. Again we'd like to thank community contributor mile23 for his work on this feature.

More useful error output

We also made DrupalCI's error output more detailed, to make it more immediately clear to developers what the issue with a particular patch might be. Developers will now see messages on the test result bubbles, for example a 'patch failed to apply' error rather than a generic 'CI error' message.

Community Initiatives

Contrib Documentation Migration

We want to continue to encourage Project maintainers to create documentation guides on their projects using the new documentation content types. Maintainers can then migrate their old documentation content into these new guides, or create new documentation pages. For more information about this process, please consult our guide to contrib documentation.

Help port Dreditor features to Drupal.org

Are you a Drupal.org power user who relies on Dreditor? Markcarver is currently leading the charge to port Dreditor features to Drupal.org, and invites anyone interested in contributing to join him in #dreditor on freenode IRC or the Dreditor GitHub.

Infrastructure

Special note: Drupal Association seeks Infrastructure Services vendor

We'd also like to announce a Request for Information. The Drupal Association seeks an infrastructure services vendor to help us manage the underlying infrastructure that supports Drupal.org, our sub-sites, and the services we maintain. Our internal engineering team will continue to manage the sites and services themselves, while this vendor will help us with systems administration, virtual machine management, monitoring and pager responsibilities, disaster recovery, etc.

For more details about this request for information, please see our post on the Association blog.

———

As always, we’d like to say thanks to all the volunteers who work with us, and to the Drupal Association Supporters, who made it possible for us to work on these projects. If you would like to support our work as an individual or an organization, consider becoming a member of the Drupal Association. Follow us on Twitter for regular updates: @drupal_org, @drupal_infra

Feb 14 2017
Feb 14

Read our Roadmap to understand how this work falls into priorities set by the Drupal Association with direction and collaboration from the Board and community.

Drupal.org updates

Recognizing more types of contribution in the Drupal.org Marketplace

We were very pleased to announce an expansion of the issue credit system into a broader contribution credit system which recognizes more than just code contributions for the purposes of ranking organizations in the marketplace.

We now calculate the following 4 types of contribution into overall contribution credit:

User research for the upcoming industry pages

In a previous blog post on Drupal.org, we talked about our increasing focus on the adoption journey and our plans to create industry specific landing pages on Drupal.org. In January we did extensive user research with people in media and publishing, higher education, and government, which will be the first industries we promote. We're hoping to launch these pages very soon, so keep an eye on the home page.

Preparing for community elections for the Drupal Association board

Elections

The elections process for the community seats on the Drupal Association board kicks off with self-nominations in February each year. This means that we dedicated some time in January to making small refinements and improvements to the nomination process. In particular we've added more in-context educational materials about the board to the self-nomination form, including a video by executive director Megan Sanicki. We've also refined our candidate questions to help candidates express their unique qualifications.

If you're interested in bringing your perspective to the Drupal Association board, please nominate yourself.

Membership history messaging

To make it easier for members to understand their membership history, we've added new messaging to the membership join and renew pages. Users who go to join or renew their Drupal Association membership will now see a message indicating their current membership expiration date, their last contribution amount, a link to contribute again, and their auto-renewal status.

Membership messaging

Migration of Drupal Association content to Drupal.org

Drupal Association

In January we also migrated the majority of content from assoc.drupal.org to a new section on Drupal.org itself. This effort is part of our larger content restructure initiative. By moving Drupal Association content into Drupal.org we hope to increase discoverability of information about the DA, and create a tighter integration between Drupal Association news and the front-page news feed.

DrupalCI

Checkstyle results now available on the DrupalCI dispatcher

DrupalCI logo

Thanks to community member mile23, DrupalCI now supports automated code style testing. To see checkstyle results for any test on Drupal.org, click on the test result bubble and then click the 'view results' link to view the detailed test results on DrupalCI's jenkins dispatcher.

We're still gathering input and feedback for this initial release of the checkstyle feature, as we decide how to integrate the checkstyle results more tightly with Drupal.org. If you have feedback or suggestions please leave your comments in this issue: #1299710: [meta] Automate the coding-standards part of patch review.

Updated testing environments

DrupalCI supports testing code against a matrix of php and database versions. In January we updated the php environments that DrupalCI supports, so that you can test against the minimum supported versions or the latest point releases. Our 5.X containers have been upgraded to the latest version for each minor release (5.3.29, 5.4.45, 5.5.38, 5.6.29). The singular PHP 7 environment that we were using was following the 7.0.x branch of php7. This has now been expanded into four php 7 environments, 7.0 (7.0.14), 7.1 (7.1.0), 7.0.x, and 7.1.x.

The dev versions of php are primarily intended for Core to sense upstream changes to php before they become released, as our comprehensive test suite often finds unanticipated bugs in php7. Additionally some missing features in the php7 containers were added, specifically apcu.

Local testing improvements

DrupalCI has always supported local testing, in order to allow developers to test changes on their own machines. This is helpful for several reasons: it allows people to test on their own machines before triggering one of the DrupalCI test bots, it lets users troubleshoot failing tests, and it helps to eliminate the 'works on my machine' problem where code appears to work in a local environment, but fails on the test bots.

To make local testing even easier, DrupalCI now automatically generates a vagrant environment for local testing. To use this functionality simply clone the drupalci_testrunner.git repo and then run $ vagrant up from within the directory. Furthermore, DrupalCI can download a build.yml file from a dispatcher.drupalci.org url to replicate any test that has been run on Drupal.org. More information about this will be added to the DrupalCI documentation soon.

Adding test priority

DrupalCI runs thousands of tests of the Drupal codebase for core and contrib modules every month. These tests include commit and patch testing for the active development which may be occurring at any time day or night, as well as the hundreds of daily regression tests run for both core and contrib projects. To help make testing more responsive, we've added a notion of testing priority. When there is a queue of waiting tests, Drupal 8 core patch tests will take priority; followed by D8 branch tests; followed by D8 contrib tests; followed by Drupal 7 patch, branch, and contrib tests.

Community Initiatives

Project Applications Revamp

Our primary community initiative priority for the first quarter of the new year is the Project Application Revamp. There are four phases to the revamp: 1) preserving security advisory coverage signals about projects, 2) transitioning security advisory coverage to an opt-in process, 3) opening the gates to allow any user to promote a project to full and create releases, 4) building new tools to incentivize code review and provide code quality signals on project pages. One of the changes we made as part of phase 1 was to adjust the way recommended releases are highlighted on Drupal.org project pages.

Contrib Documentation Migration

Project maintainers are now able to create documentation guides on their projects using the new documentation content types. Maintainers can then migrate their old documentation content into these new guides, or create new documentation pages. For more information about this process, please consult our guide to contrib documentation.

Help port Dreditor features to Drupal.org

Are you a Drupal.org power user who relies on Dreditor? Markcarver is currently leading the charge to port Dreditor features to Drupal.org, and invites anyone interested in contributing to join him in #dreditor on freenode IRC or the Dreditor GitHub.

———

As always, we’d like to say thanks to all the volunteers who work with us, and to the Drupal Association Supporters, who made it possible for us to work on these projects.

If you would like to support our work as an individual or an organization, consider becoming a member of the Drupal Association.

Follow us on Twitter for regular updates: @drupal_org, @drupal_infra

Feb 09 2017
tvn
Feb 09

I am happy and sad to share the news today that I am leaving the Drupal Association for an exciting new adventure.

I've been volunteering for Drupal.org for a while before I joined the Association as a staff member in April 2012. Before that I never actually thought you can be paid to do something you volunteer your free time for because you enjoy doing it so much.

It was also my first remote job. Back then it was not as common. I clearly remember first talking to Angie, and my initial response was "You are crazy. I am literally on the other side of the world." Which she didn't find to be of any consequence.

It's been a wonderful (almost) 5 years. I've learned a lot, and grew. I've traveled a lot and met lots of fantastic people. I watched the Association grow. Our engineering team grew as well from 2 to 11, which was exciting but also challenging to be a part of. We've done some great things on Drupal.org. Of course, there are so many many more I wish we'd done. :)

The time has come however to move on to the new challenges.

Thanks to all the volunteers who worked with me during all these years. You were so helpful and generous with your time.

Thanks to the staff members for being wonderful human beings, for your support, and laughs, for becoming a family.

I am incredibly sad to leave our staff, and especially the Engineering team. They are a smaller team now, and I know just how hard it will be to maneuver all the requests coming from all the different parts of the community with the limited resources they now have. So please be nice to them.

As for the next steps, I am looking forward to joining the DA alumni club, which includes some of my closest friends. And I will be around. You will probably see me at the next DrupalCon. Come say hi. And no, I will not fix your webmasters' queue issue. 
 

---
Photo from Flickr.

Jan 25 2017
Jan 25

Read our Roadmap to understand how this work falls into priorities set by the Drupal Association with direction and collaboration from the Board and community.

Our December update comes to you a bit later than our usual monthly posts, for all the usual practical reasons: holidays, vacations, and our staff retreat in early January. But also, because we've been reflecting on the past year, and planning for the year to come. You'll soon hear about our initiatives for 2017, but for now— let's dive into what we did in December.

Drupal.org updates

DrupalCon Baltimore

DrupalCon Baltimore logo Apr 24-28

At the beginning of December we launched the full site for DrupalCon Baltimore, which is coming up April 24-28. For the first time, we launched the full event site including the call for papers, scholarship applications, and registration all on the same day.

Early bird pricing is available for a limited time, so we encourage you to register today.

Stable release of the Composer Façade

Project Composer logo

Drupal.org's support for Composer has been in development since the beginning of last year. We released the public alpha of our composer endpoints at DrupalCon New Orleans, and then entered beta over the course of this past summer. After a period of feedback, bug fixes, and further refinement with the help of core and contrib developers we announced the stable release of Drupal.org's composer support on December 21st.

We'd like to thank the following community members for their help with this initiative: seldeak, webflo, timmillwood, dixon_, badjava, cweagans, tstoeckler, and mile23. We'd also like to thank Appnovation for sponsoring our initial Composer support work.

Improved messaging for new users

One of the innovations of Drupal.org's online community that we introduced about 2 years ago, is the process by which new users get confirmed by trusted users. As a user of Drupal.org, you know that when you see a new user with a 'confirm' button under their user icon, you can check their recent activity and help confirm for us that they're a real user (not a bot or spammer who managed to slip through).

However, we received some feedback from recently registered users, that this process was too opaque. New users did not have enough guidance to understand that they can only perform a sub-set of site activities until another user confirms them.

After hearing this feedback, we spent some time in December improving the messaging tonew users when they first sign up on Drupal.org— so they can better understand how to become confirmed.

DrupalCI refactored and updated to use composer

DrupalCI logo

In December we also completed a refactor of DrupalCI and updated the testing system to use Composer when testing Drupal. This means we can now test projects with external composer dependencies on Drupal.org. Other new features and bugfixes include: more available test artifacts; dependency changes can now be submitted in patches to composer json; the test runner produces a build file that can be downloaded and run locally to re-execute any test verbatim. There are more added features as well..

This work has continued into January, particularly around making more testing environments available, and adding new test types (such as code sniffer). Look for additional updates in the upcoming January report.

Special thanks to mile23 for collaborating with us on this work.

Jenkins upgraded to better manage our EC2 Instances

The cost of automated testing for the Drupal project is a significant expense for the Drupal Association. In December we updated Jenkins and several of the plugins that are used to orchestrate the creation and management of DrupalCI testbots, and now our enforcement of instance limits is much more reliable. In December this saved us nearly 50% on our testing bill, without a significant increase in testing wait times. In January we are projecting a similar savings.

The work of community member fabianX might also provide similar savings for the project, so we encourage contributors involved in core to help review: #2759197: [D7] Improve WebTestCase performance by 50% and #2747075: [meta] Improve WebTestCase / BrowserTestBase performance by 50%

HTTP/2 Support enabled

HTTP/2 is the next generation network protocol that decreases latency in page loads by using better data compression, pipelining, and server push. In December we enabled HTTP/2 support for Drupal.org, improving performance for all users with modern browsers that support the standard.

Community Initiatives

Preparing for the Project Applications Revamp

In November the Drupal 8 User Guide went live, so in December we prepared for the next community initiative on our roadmap - the Project Application Revamp. Over the course of the last several months we've been doing pre-work around this initiative to ensure that the appropriate signals about security advisory coverage and recommended releases are provided on project pages. This pre-work will help ensure that Drupal users still have good signals to project quality, even as we open up the creation of full projects.

Initiatives need your help

Are you a Drupal.org power user who relies on Dreditor? Markcarver is currently leading the charge to port Dreditor features to Drupal.org, and invites anyone interested in contributing to join him in #dreditor on freenode IRC or the Dreditor GitHub.

Is the written word your domain? Consider putting your skills to use by becoming a maintainer of Drupal documentation. If you are a developer interested in contributing code to the new documentation system, please contact tvn.

———

As always, we’d like to say thanks to all the volunteers who work with us, and to the Drupal Association Supporters, who made it possible for us to work on these projects.

If you would like to support our work as an individual or an organization, consider becoming a member of the Drupal Association.

Follow us on Twitter for regular updates: @drupal_org, @drupal_infra

Jan 12 2017
Jan 12

Within weeks of introducing the contribution credit system on Drupal.org we realized we had created something powerful. Like all open source projects, Drupal has a behind-the-scenes economy of contribution in which individuals, organizations, and end users work together to maintain the software as a public good. That behind-the-scenes economy was brought to the fore when we chose to rank the Drupal Marketplace by issue credits. For the first time, Drupal.org gave businesses a direct financial incentive to contribute code.  

Being good stewards of these incentives is a sobering responsibility, but also a great opportunity. We can use this system to recognize the selfless effort of our community volunteers, to reward the organizations that sponsor their employees' time to give back to the project, and to connect end-users with the organizations that are the biggest contributors.

But as we often say in this community—contribution is more than code. It is the time provided by dedicated volunteers; the talent of community organizers, documentation maintainers, and developers; and the treasure provided by organizations that sponsor Drupal events and fund the operations and infrastructure that maintain the project.

What are we changing?

We’re updating the ranking algorithm for Drupal.org’s Marketplace of service providers and list of all organizations in the Drupal ecosystem. We've expanded on the issue credit system to create a more generic contribution credit system which lets us recognize more types of contribution. Each type of contribution is now weighted to give the organization an overall amount of contribution credit. We've built this system so that we can continuously evolve the incentives it creates by adjusting the weight given to each type of contribution as the project's needs change. To prevent gaming, we will not be publishing the exact weights or total contribution score, but those weights have been reviewed by the Association Board and Community Working Group.

We've carefully chosen a few new types of contribution to factor into the ranking. These were selected because they create incentives to reach specific goals: encouraging organizations to sponsor development of Drupal, gathering more Drupal 8 success stories that can be used to promote Drupal adoption, and recognizing the financial contributions that promote the fiscal health of the Drupal association.

We now calculate the following 4 types of contribution into overall contribution credit:

What about other types of contribution?

Of course, these new factors still don't include all types of contribution. This iteration aims to add measurable factors that reward the behavior of organizations that are good Drupal citizens, and incentivize some of the most important contributions that have a big impact in moving the project forward. But there are other factors we'd like to include in the future! We're keeping track of these additional kinds of contribution, such as sponsoring local user groups, organizing training days, writing documentation, and more, in this issue: #2649100: Improve contribution statistics on user and organization profiles.

There are two factors in particular that we are not yet including that we'd like to address.

The first is project application reviews. These reviews are a critical part of the lifecycle of a new project on Drupal.org, but because we are making the Project Application Revamp a key priority for the first part of 2017, this was not our focus in this initial update. We may revisit this factor as the Project Application Revamp initiative gets underway.

The second is camp organization. We know that there are many individuals and organizations who invest heavily in Drupal Camps, and this has been a critical part of the project's success. However, at this time our data about the individuals and organizations who participate in camp organization is purely self-reported, and therefore too vulnerable to manipulation to include in the algorithm at this time. In the future we hope we can find a responsible way to measure and credit this kind of contribution.

We’ll continue to look for other good factors to add, and do our best to weigh them fairly.

How often will the algorithm change? Who governs these changes?

As this is our first major change to the marketplace ranking system since the launch of issue credits, we may need to make some small adjustments in the first weeks following the launch. However, we know that too frequent changes to the incentive structure will be frustrating for the individuals and organizations who are contributing to the project. Therefore, after the initial tuning we intend to update the marketplace ranking system on a roughly 6 month cycle.

While the primary responsibility to manage the contribution credit system is ours, we have committed to vetting these and future changes with members of the Drupal Association Board and Community Working Group.

Dec 21 2016
Dec 21

Drupal.org Composer Logo

Drupal.org's Composer endpoints have been available in beta for some time now, and in that time we've begun to see many, many people use Composer to manage Drupal modules and themes. We first launched these repositories before DrupalCon New Orleans as an alpha release, and move into beta a few months later. After receiving your feedback and bug reports we've made updates, and are ready to call this service stable.

What is Composer?

Composer is a tool for dependency management in PHP. It allows you to declare the libraries your project depends on and it will manage (install/update) them for you.

… Composer is strongly inspired by node's npm and ruby's bundler." - Source

In a nutshell, Composer allows you to declare the dependencies of your project in a composer.json file in the root of your PHP project. Those dependencies, which you then install through Composer, can have their own composer.json files and their own dependencies—all of which will be automatically managed and installed by Composer. When you need specific control over the versions of dependencies, you can use a composer.lock file.

You can read more about Composer at GetComposer.org.

How do Drupal.org's composer repositories work?

Drupal.org offers two Composer repositories—one for Drupal 7, and one for Drupal 8. Composer requires that packages adhere to semantic versioning, which Drupal 8 core does, but Drupal 8 contrib, and Drupal 7 core and contrib, don’t. To solve this problem, we've created a Composer façade, which takes all of the metadata about projects on Drupal.org and translates them into a format Composer can understand—including translating the Drupal-specific versioning for Drupal 7 and contrib into semantic versioning.

By creating this façade, we've made sure that Drupal.org is still the canonical source for metadata about Drupal.org projects, and that we can update this translation layer as the versioning schema changes. (Learn more about the effort to move Contrib projects to semantic versioning).

In addition to providing endpoints for building projects, Drupal's automated testing suite— DrupalCI—now uses Composer to test Drupal core and contributed projects. This allows developers to test any external dependencies.

How do I use Drupal.org's Composer repositories?

To begin using Drupal.org's Composer repositories, you'll need to update your composer.json file to include the appropriate Composer repository for the version of Drupal. To use Composer with Drupal 7, use the repository url:

https://packages.drupal.org/7

. To use Composer with Drupal 8, use the repository url:

https://packages.drupal.org/8

, as in this example.

After setting up composer, simply run the command:

$ composer config repositories.drupal composer https://packages.drupal.org/8

And your project's composer.json should be updated to look like the following:

{ 
    "repositories": { 
        "drupal": {
            "type": "composer",
            "url": "https://packages.drupal.org/8" 
        }
    }
}

Once you've made that change, you should be able to use Composer for Drupal modules and themes as you would for any other PHP package, using the drupal/ namespace:

$ composer require drupal/<modulename>

There is one caveat about the pattern: there are some namespace collisions among modules, and so it is on our roadmap to update Drupal.org project pages to specify the exact namespace to use to require a given project.

To learn more about how to use Drupal.org's Composer repositories, and for some troubleshooting tips, read the Project Composer documentation.

What about licensing?

All the projects hosted on Drupal.org are licensed GPLv2 or later or have an entry in the packaging whitelist. This means that you can rely on Drupal Core and contributed modules and themes to be licensed under the GPL or compatible. And if you need to redistribute your code created with Drupal projects, it must be redistributed as GPL-2.0 or GPL-3.0, but please note that Drupal.org will only host GPL-2.0 licensed projects.

However, because Composer is a tool that can manage packages in the wider PHP ecosystem, you might find that you want to require a non-GPL package in your project. Using GPL-licensed Drupal projects with external packages that are GPL compatible is fine. Just be aware that if you redistribute that code, you will have to redistribute under a GPL license.

We cannot provide legal advice for your use of open source software. If you use Composer to install packages that are not compatible with the GPL alongside GPL-licensed projects like Drupal, you may use that software together, but per the terms of the GPL you may not copy, distribute, or modify that software.

"Activities other than copying, distribution and modification are not covered by this License; they are outside its scope. The act of running the Program is not restricted…" GPL 2.0 Section 0.

For more questions about Drupal and the GPL you can read the Licensing FAQ.

The Composer "licenses" command

If you need to check what licenses are in use by your Composer-installed dependencies, you can use the licenses command:

$ composer licenses

This will list the licenses of all the packages your project is using. However, we recommend that all users double check licensing information as some of these packages may be dual-licensed or have other licensing concerns that may not be immediately apparent from this command.

What's next?

At this point, the Drupal.org Composer service is stable and you can use it to manage modules and themes in your production websites. That said, we do have a roadmap of additional features that we'd like to add. And your contributions are welcome!

  • As development on Drupal.org's Composer service continues, we want to focus on the following features:
  • Supporting Composer-based workflows for distributions and install profiles
  • Providing sub-tree splits of Drupal Core
  • Updating project pages to provide information about using Composer with any given Drupal.org hosted project
  • Adding features to the updates service, to collect statistics about projects installed with Composer, and to explore providing update alerts about external dependencies
  • We also hope to work with core maintainers to add the Drupal.org Composer repositories to Drupal Core's composer.json file

If you're interested in learning more about our roadmap for Composer, or contributing to this service on Drupal.org, you can learn more in the Composer plan issue.

How you can help

If you’re interested in helping to improve Drupal.org's support for Composer workflows, please take a look at the issue above, find us on irc in #drupal-infrastructure, or send us a volunteer proposal.

Thanks to our Community Initiative contributors

We'd like to thank the individuals who worked with us as part of this Community Initiative.
In particular, we'd like to thank:

We'd also like to thank Appnovation, who sponsored the initial development of Drupal.org's composer endpoints.

To these volunteers and sponsoring organizations—it is your expertise, your insight, and your affirmation of our work that make these Community Initiatives successful. Thank you!

Dec 08 2016
Dec 08

Read our Roadmap to understand how this work falls into priorities set by the Drupal Association with direction and collaboration from the Board and community.

The engineering team at the Drupal Association had much to be thankful for in November. With the support of the wonderful volunteers in our community and the contributions of our Supporting Partners we were able to deliver some great tools for the project. Let's dive and see what's new.

Drupal.org updates

Promoting Drupal by Industry

In November we finished the technical scaffolding for the upcoming industry pages, and began working with the wider Association team on content development for these pages. Because we were ahead of our internal targets for this page and we felt it would add significant value, we've also added the ability to geotarget content on these industry pages.

This is the first instance of geo-targeting on Drupal.org, and we'll be using it to help connect Drupal evaluators with regionally appropriate content and partners on these pages. Work on the industry pages is ongoing, but we're excited to bring them to you soon.

Developer Tools Evaluation

During November the engineering team also had a two day retreat here in Portland, OR with webchick - one of the members of the Technical Advisory Committee. We used this retreat to do a deep dive into the current state of developer tools on Drupal.org, and to evaluate our options to continue evolving the tools we offer to the community.

We gave a summary of our exploration along with some next steps to the Drupal Association Board on November 22nd. You can find the minutes and a recording here.

Core release packaged with --no-dev composer dependencies

Starting with the Drupal 8.2.3 release, we are now packaging full releases of Drupal core with --no-dev composer dependencies. This means that packages downloaded will not include extraneous developer extras that should not be used in production sites, and that the release packages will be smaller. We will continue to package dev releases with the dev dependencies.

Feature branch testing support

Drupal.org allows maintainers to create feature branches for issues by using the name format [issue#]-[short-description]. Any commits made to a branch in this format will appear in the sidebar of the associated issue. To improve the utility of these feature branches, DrupalCI patch file tests now also run on push to these branches.

Feature branch testing UI on issues

To add tests, users can simply click on the 'add test' link beneath the git branch in the issue sidebar, or click on the existing test result bubble to re-test or add a new test. Since this feature was introduced we've run over 200 issue branch tests.

Project maintainers can add Documentation Guides

UI for relating guides to projects

Display of related projects on Guides
We're continuing to support the migration of documentation to the new documentation system, and we've now enabled Project Maintainers to add related documentation guides to their projects. Once added, the related projects will appear on the documentation guides, in the sidebar.

Documentation Maintainers can find their Guides

Many community volunteers have stepped up to become maintainers of the new documentation guides. We want to make sure we're giving them the tools they need to do the work of maintaining those guides and the pages within them.

Your Guides tab on user profile

We've added a 'Your Guides' section to the user profile which will list all of the guides that a user maintains, as well as the pages within those guides. This should allow maintainers to see when pages have been recently changed or added, and to easily keep their guide content curated and up to date.

Infrastructure

Virtualization and Improved Config Management

In November, we completed the majority of two major infrastructure projects. Firstly, we've virtualized the majority of the infrastructure and standardized on Debian 8 images. Secondly we've updated our configuration and user management from Puppet 3 + LDAP to Puppet 4 + Hiera. This is a significant milestone for our infrastructure, and gives us a more portable and maintainable infrastructure to manage moving forwards.

Community Initiatives

Community initiatives are a collaboration; with dedicated community volunteers building improvements to Drupal.org with the architectural guidance and oversight of the Drupal Association engineering team.

Drupal 8 User Guide Launched!

We're very happy to say that the Drupal 8 User Guide is now live on Drupal.org! This documentation guide is carefully curated to provide all the information a new user needs to become skilled at managing a Drupal 8 site. We want to give a special thanks to jhodgdon for all her work on the User Guide project.

Initiatives need your help

Are you a Drupal.org power user who relies on Dreditor? Markcarver, who is currently leading the charge to port Dreditor features to Drupal.org, has invited anyone interested in contributing to join him in #dreditor on freenode IRC or the Dreditor GitHub.

Is the written word your domain? Consider putting your skills to use by becoming a maintainer of Drupal documentation. If you are a developer interested in contributing code to the new documentation system, please contact tvn.

———

As always, we’d like to say thanks to all the volunteers who work with us, and to the Drupal Association Supporters, who made it possible for us to work on these projects.

If you would like to support our work as an individual or an organization, consider becoming a member of the Drupal Association.

Follow us on Twitter for regular updates: @drupal_org, @drupal_infra

Nov 11 2016
Nov 11

Read our Roadmap to understand how this work falls into priorities set by the Drupal Association with direction and collaboration from the Board and community.

The Drupal Association team has been getting back to work after coming back from DrupalCon Dublin in September. For the engineering team, October has been focused on some back-end services and infrastructure that support the Drupal project, while we continue to move forward on some longer term front facing initiatives.

Drupal.org updates

Promoting Drupal by Industry

Last month we talked about the new homepage we released for Drupal.org, and using those editorial tools to build a membership campaign. We hinted that additional changes will be coming soon. While we're not ready to launch this new content - we can talk about it in some greater detail.

Dries Buytaert, the project founder, has called Drupal the platform for ambitious digital experiences. That phrase expresses the incredible power and flexibility of Drupal, but also encapsulates an aspect of Drupal that can be difficult for newcomers. It can be very hard for newcomers to Drupal to understand how to take a base install of Drupal core, and extend that to achieve that ambitious vision.

We want to help close that gap in understanding—to help evaluators see how Drupal achieves these ambitions. To do this, we'll be creating a series of landing pages that focus granularly on how Drupal creates success stories in particular industries. Look for more on this topic in coming months.

DrupalCon Vienna Site Launched

Vienna 2017 Logo

As is tradition, during the closing session of DrupalCon Dublin we announced that the next DrupalCon in Europe will be held in Vienna! We launched the splash page announcing the event at vienna2017.drupal.org and we have information about sponsorship and hotel reservations already available.

DrupalCon Vienna will happen from the 25th to 29th of September 2017, and we'll hope to see you there!

More flexible project testing

We've made a significant update to how tests are configured on the Automated Testing tab of any project hosted on Drupal.org. Automated testing, using the DrupalCI infrastructure, allows developers to ensure their code will be compatible with core, and with a variety of PHP versions and database environments. In October, we updated the configuration options for module maintainers.

Maintainers can now select a specific branch of core, a specific environment, and select whether to run the test once, daily, on commit, or for issues. Issues are limited to a single test configuration, to ensure that the code works in a single environment before being regression tested against multiple environments on on-commit or daily tests.

Better database replication and reliability

Behind the scenes, we've made some updates to our database cluster - part of our infrastructure standardization on Debian 8 environments managed in Puppet 4. We've made some improvements to replication and reliability - and while these changes are very much behind the scenes they should help maintain a reliable and performant Drupal.org.

Response to Critical Security Vulnerabilities

DirtyCow Public Domain Logo

When it rains, it pours—a maxim we take to heart in Portland, Oregon—and that was especially true in the realm of security in October. The most widely known vulnerability disclosed was the 'DirtyCow' vulnerability in the Linux kernel. A flaw in the copy-on-write system of the Linux kernel made it possible, in principle, for an unprivileged user to elevate their own privileges.

Naturally, responding to this vulnerability was a high priority in October, but DirtyCow was not the only vulnerability disclosed, as security releases were also made for PHP, mariadb, tar, libxslt, and curl. We mitigated each of these vulnerabilities in short order.

Community Initiatives

Community initiatives are a collaboration; with dedicated community volunteers building improvements to Drupal.org with the architectural guidance and oversight of the Drupal Association engineering team.

Drupal 8 User Guide

The Drupal 8 User Guide is getting very close to being available on Drupal.org. We are working closely with contributor jhodgdon to resolve some perplexing inconsistencies between what we're seeing in our development environment and in our initial production deployment.

Dreditor

markcarver who is currently leading the charge to port Dreditor features to Drupal.org, has invited anyone interested in contributing to join him in #dreditor on freenode IRC or the Dreditor GitHub issue queue.

Documentation Maintainership

Finally, we want to continue to encourage the community to become maintainers of Drupal documentation. If you are a developer interested in contributing code to the new documentation system, please contact tvn.

———

As always, we’d like to say thanks to all the volunteers who work with us, and to the Drupal Association Supporters, who made it possible for us to work on these projects.

If you would like to support our work as an individual or an organization, consider becoming a member of the Drupal Association.

Follow us on Twitter for regular updates: @drupal_org, @drupal_infra

Oct 21 2016
Oct 21

Republished from buytaert.net

Nasdaq chooses Drupal 8

I wanted to share the exciting news that Nasdaq Corporate Solutions has selected Drupal 8 as the basis for its next generation Investor Relations Website Platform. About 3,000 of the largest companies in the world use Nasdaq's Corporate Solutions for their investor relations websites. This includes 78 of the Nasdaq 100 Index companies and 63% of the Fortune 500 companies.

What is an IR website? It's a website where public companies share their most sensitive and critical news and information with their shareholders, institutional investors, the media and analysts. This includes everything from financial results to regulatory filings, press releases, and other company news. Examples of IR websites include http://investor.starbucks.comhttp://investor.apple.com andhttp://ir.exxonmobil.com -- all three companies are listed on Nasdaq.

All IR websites are subject to strict compliance standards, and security and reliability are very important. Nasdaq's use of Drupal 8 is a fantastic testament for Drupal and Open Source. It will raise awareness about Drupal across financial institutions worldwide.

In their announcement, Nasdaq explained that all the publicly listed companies on Nasdaq are eligible to upgrade their sites to the next-gen model "beginning in 2017 using a variety of redesign options, all of which leverage Acquia and the Drupal 8 open source enterprise web content management (WCM) system."

It's exciting that 3,000 of the largest companies in the world, like Starbucks, Apple, Amazon, Google and ExxonMobil, are now eligible to start using Drupal 8 for some of their most critical websites. 

Oct 20 2016
Oct 20

Read our Roadmap to understand how this work falls into priorities set by the Drupal Association with direction and collaboration from the Board and community.

This month's update comes to you a couple weeks late, but only because we were on site at DrupalCon together with the community to move the project forward! DrupalCon Dublin was a great event, with the entire Drupal Association staff engaged to make DrupalCon the best place to develop your Drupal skills, learn what's coming for the project, and sprint on core and contrib. We are tremendously thankful to the community that joins us for DrupalCon, and to the incredible volunteers that help us put on the event. If you couldn't join us in person, you can still review the session recordings.

Now, on to the updates!

Drupal.org updates

New homepage

Screenshot of the new homepage

Certainly the most visible change to Drupal.org in September was the refresh of our home page. As the front door of our community home, the front page needs to be inviting to both existing community members, and people new to Drupal who are just beginning their adoption journey. The changes are more than aesthetic. We also put in place new editorial tools to give us greater flexibility with the front page itself, and with future landing pages that we hope to create in the same highly-designed, attractive style. In addition to these structural and editorial changes we made some content changes as well, cleaning up our news feed, and giving DrupalCon a new, more prominent position on the home page.

And there are more updates to come! Using the same editorial tools we'll soon be rolling out additional content for Drupal evaluators - promoting proven solutions built using Drupal in specific industries. Look forward to this in the coming months.

Membership campaign

We used the same editorial tools that built the new homepage to build a landing page for our fall membership campaign. This campaign showcases how Drupal Association members make community cultivation grants possible - and the stories that those grants create.

"We wanted to bring international speakers and sponsors to the first DrupalCamp in Mexico City. Getting the grant we needed to build a Drupal ecosystem in a metropolis like Mexico City was terrific."

These community stories run to the heart of our mission - enabling our global community build connections on the local level, and extending Drupal's reach across the world.

Case studies on organization profiles

In September we also made a small but significant update to organization profiles. We've moved the often unwieldy index of people associated with an organization to a subpage, in order to make room for listing the case studies that an organization has created. We want to encourage Drupal organizations of all kinds to share their stories of success, especially around Drupal 8.

If your organization has never created a Drupal case study before, we have some materials to teach you how to create a case study on Drupal.org.

Issue Credit Updates

The issue credit system has had a remarkable impact on the community. Being able to quantify the contribution of organizations to Drupal's codebase has lead to an unprecedented level of healthy competition between organizations who support the project—each trying to outdo the other with their contributions. It has been amazing to see how generous these organizations are, sponsoring the work of committed community contributors to advance the project.

To maintain this system in a healthy way, we need to monitor it carefully and make small adjustments to ensure that we're creating the right incentives for true contribution, and not a system to be gained for self-promotion. We've made a few small tweaks in september to reduce spurious re-opening of issues in order to 'reset the clock' on credits, and we have a few more fixes on the plate to keep this ecosystem healthy.

We're also looking to expand the kinds of activities that receive contribution credit - so look forward to further updates on that front in the coming months.

Community Initiatives

Finally, here are some updates on our active community initiatives. Community initiatives are a collaboration; with dedicated community volunteers building improvements to Drupal.org with the architectural guidance and oversight of the Drupal Association engineering team.

Documentation Migration

The migration of Drupal.org documentation to the new documentation content types is well under way. Tremendous thanks to tvn and eojthebrave for spearheading this effort and recruiting additional volunteers to help maintain the new documentation guides and move the community over into the new system.

We still need your help! We need community volunteers to take on small sub-sets of documentation to maintain, and make sure they're cleaned up post-migration.

If you don't want to commit to maintaining a guide, you can still help out by doing some of the pending tasks for any of the documentation pages.

Lastly, if any Drupal developers are interested in contributing code to the new documentation system to clean up a few minor bugs and features, please contact tvn.

Drupal 8 User Guide

As outlined in our previous update, the Drupal 8 User guide is a special subset of documentation that's been produced in a highly curated, editorially controlled way - to create a guide to Drupal 8 that rivals the standards of an industry publication. All of the components needed to publish this guide to Drupal.org are now in place, so our final step will be to coordinate some last tweaks and bug fixes with jhodgdon, and then to begin linking it prominently on Drupal.org.

Dreditor

In the weeks leading up to DrupalCon Dublin there was a small crisis in the contributor community. Because of changes in the browser add-on validation process, the incredibly valuable and popular Dreditor browser extension, first developed by sun, and currently maintained by markcarver, andcottser has reached its end of life—or has it?

After a tremendous outpouring from the community a new plan was made, and now Mark is working on porting the features of Dreditor directly to Drupal.org. Work is still ongoing, but as it proceeds, users will be able to optionally enable these features component by component on their user profiles.

———

As always, we’d like to say thanks to all the volunteers who work with us, and to the Drupal Association Supporters, who made it possible for us to work on these projects.

If you would like to support our work as an individual or an organization, consider becoming a member of the Drupal Association.

Follow us on Twitter for regular updates: @drupal_org, @drupal_infra

Oct 18 2016
Oct 18

Workflow iconAt DrupalCon Dublin, I spoke about The Association’s commitment to help Drupal thrive by improving the contribution and adoption journeys through our two main community assets, DrupalCon and Drupal.org. You can see the video here.

One area I touch on was my experience as a new code contributor. Contributing my patch was a challenging, but joyous experience and I want more people to have that feeling—and I want to make it as easy as possible for others to contribute, too. It’s critical for the health of the project.

At the heart of the Drupal contributor community are our custom development tools, including the issue tracker, Git repositories, packaging, updates server, and automated testing. We believe there are many aspects of Drupal’s development workflow that have been essential to our project's success, and our current tooling reflects and reinforces our community values of self-empowerment, collaboration, and respect, which we seek to continue to uphold.

It’s time to modernize these developer tools. To support the Association with this objective The Drupal Association created a Technical Advisory Committee (TAC). The TAC consists of community members Angie Byron, Moshe Weitzman, and Steve Francia, who is also our newest Drupal Association board member. The TAC acts in an advisory role and reports to me.

Building off of the work the community has already done, the TAC is exploring opportunities to improve the tools we use to collaborate on Drupal.org. The crux of this exploration is determining whether we should continue to rely on and invest in our self-built tools, or whether we should partner with an organization that specializes in open source tooling.

Our hope is that we will be able to bring significant improvements to our contribution experience faster by partnering with an organization willing to learn from our community and adapt their tools to those things we do uniquely well. Such a partnership would benefit both the Drupal community—with the support of their ongoing development—and potentially the broader open source community—by allowing our partner to bring other projects those aspects of our code collaboration workflow.

The TAC will use a collaborative process, working with staff and community to make a final recommendation. The TAC has already begun the process and has some very positive exploratory conversations. The TAC and staff will be communicating their progress with the community in upcoming blog posts.  

Sep 21 2016
Sep 21

As you can see we've put a fresh coat of paint on Drupal.org - but the changes run below the surface. This latest iteration of the front page brings the key concepts of our design system to the forefront: Clean, Modern, Technical.

Preview of the new home page design

This change also brings new editorial tools for Drupal.org content editors. The new home page provides us more flexibility with content and presentation, and so you'll see more frequent updates, more information about DrupalCon, and more editorial flexibility on the home page than you've seen in the past. These tools are also helping us to build cleaner, modern landing pages - like you've just seen with our Fall Membership Campaign.

We've previewed this work with several key members of the community and the board, and we want to say thank you to everyone who's given us their feedback on this first step for our new home page. We also want to give an extra special thank you to dyannenova for her contributions to this effort.

This is just the beginning - very soon we'll have a new visual look for the case studies that are featured on the home page, and then shortly after that we'll begin promoting solutions to Drupal evaluators in specific industries, like Higher Education, Media & Publishing, and Government.

If Drupal.org is the home of the community, then the front page is our front door. We want to welcome new users and evaluators of Drupal, highlight the project's strengths, and promote news and happenings from throughout the ecosystem.

We hope you like the changes, and we think you'll like the upcoming iterations even more. We'd love to hear your feedback!

Sep 13 2016
Sep 13

Read our Roadmap to understand how this work falls into priorities set by the Drupal Association with direction and collaboration from the Board and community.

Our latest update about Drupal.org comes as the Drupal Association has moved out of our central office in Portland, OR, and gone to an all-distributed team. A move of that sort always creates some upheaval but amidst the move we've continued to push forward on several initiatives to improve Drupal.org.

At the same time we've been pushing forward towards DrupalCon Dublin at the end of September- and we hope to see you there!

Drupal.org updates

A new home page, coming soon

As we recently previewed on the Drupal.org blog, some changes are coming to the home page. We're building some new editorial tools to allow for more flexibility with the home page content, and to enable an increased focus on the adoption journey for visitors to Drupal.org. You'll see styles reminiscent of the Drupal 8 release announcement pages, and a continued modernization of theme.

The launch of the new home page is coming soon, but as a precursor we've been making some small improvements. The new user menu which we launched in July has been updated for better keyboard accessibility, and to show a user picture as an indicator that a user has logged in. We've also moved the search feature into an icon in the top navigation. This gives us more flexibility with the header, which can be customized per-page type or per-section with the overall site search box still being present. For example, the header in the new documentation section features search box specific to this particular section, so while you are there you can search for other documentation without having to go through the full-site search and then filtering down. Lastly, we've merged the 'Get Started' and 'Download & Extend' pages. 90% of the content on these pages was duplicated with each other - and the new page presents a cleaner experience with the essential details needed for getting started with Drupal.

The new front page is beginning editorial review, with the help of DA staff, a marketing task-force from the Drupal Association board, and a few key community members.

We've also just launched our fall membership campaign, and we've used this opportunity to beta test some of these new editorial tools to build the campaign landing page. Your support makes our work possible. Thank you!

Documentation

There's some news to report on the documentation front as well. Firstly, as mentioned above, we've updated the header of the documentation section to default to a documentation-specific search box. While not so important for other areas of the site,, we want to preserve and improve the highly-visible, in context search for Documentation.

We've also made some updates to the new system for Documentation maintainers. Authors of new documentation guides will now automatically become maintainers of those guides and automatically 'follow' the guide content so that they will receive notifications of activity in that guide. Any user following a guide can modify notifications settings at any time from their user profile. Within the notification settings a user can select their prefered method of receiving updates - via email or via their tracker page.

Tvn has continued to spearhed the migration of documentation from the old book pages, to our new documentation system.

We have completed the migration of the majority of the 'general' documentation. While that is done, there is still a lot of work to do to make the documentation content better using the new tools that are now available.

We need community volunteers to take on small sub-sets of documentation to clean them up post-migration and to maintain going forward.

If you don't want to commit to maintaining a guide, you can still help out by doing some of the pending tasks for any of the documentation pages.

Lastly, if any Drupal developers are interested in contributing code to the new documentation system to clean up a few minor bugs and features, please contact tvn. And if you are going to be at DrupalCon Dublin, consider joining us at the sprints!

Quality of Life Improvements

We also took the time in August to make a few quality of life improvements, both for our end users, and for our own team. Firstly, we've made it easier than ever to download a copy of your invoice for DrupalCon. Any user can now log into events.Drupal.org and any time, go to "My Account" -> "Orders" and download a pdf of their invoice for any past event. If your company is sending you to DrupalCon, this makes the process easier than ever. (And if they're not, here are some tools to convince your boss!)

Behind the scenes, we've made some additional improvements to our sophisticated spam prevention system, which focuses on preventing bad actors from even registering on Drupal.org in the first place. For those few bad actors that do get through, the system is also tuned to allow us to prevent those users from making multiple account registrations, as one of the primary methods for targeting Drupal.org in the past has been to make a large number of 'sleeper' account registrations that can be later updated with spam links. Unfortunately, on rare occasions this tool can make it difficult for legitimate users to register an account, so we've updated the system with a whitelisting system that allows legitimate to register, without opening the floodgates to the bad actors.

Infrastructure

Virtualization and better Drupal.org dev sites

On the infrastructural side we've been focused on improving the maintainability, stability, and portability of our infrastructure with our smaller engineering team. In particular we've been focusing on virtualizing all the components of our infrastructure.

In August in particular we completed the virtualization of pre-production services. We've optimized the snapshotting and whitelisting process that allows us to create staging and development environments to make that process more efficient and easier to manage. We've also replaced our drupal.org dev site architecture with a new architecture that is no longer vulnerable to docker-fs faults which have multiple times resulted in data loss on our development environments. Drupal.org contributors who've been affected by dev site fragility should find dev sites to be much more robust moving forward.

Community Initiative Updates

Finally, here are some updates on our active community initiatives. Community initiatives are a collaboration; with dedicated community volunteers building improvements to Drupal.org with the architectural guidance and oversight of the Drupal Association engineering team.

Drupal 8 User Guide

The Drupal.org user guide is an effort lead by jhodgdon and a number of other contributors to create a highly produced, tightly editorially controlled guide to using Drupal 8. This user guide has been written to the standard of an industry publication, and uses a custom editorial workflow with git + asciidoc. Jhodgdon has been building out functionality to publish the user guide to a Documentation guide on Drupal.org.

Security

A few interrelated initiatives are in progress to improve how information about project security is displayed on Drupal.org. Mlhess has been working on a new security advisory content type for Drupal.org, which will allow security advisory content to be more easily related to project releases, among other things.

With the input and collaboration of quite a few community members, including the security working group, we've also deployed an update to project pages.

This update adds a shield icon next to stable releases. This shield icon indicates which releases are covered by the security advisory policy. This small change is also part of the groundwork for a project application revamp.

Community initiatives are not work that the Drupal Association can tackle on our own. Our mandate requires us to remain focused. That said, whenever the community has arrived at a strong plan and individual volunteers are ready to contribute code, the engineering team can provide architectural advice, code review, and deployment support.

———

As always, we’d like to say thanks to all the volunteers who work with us, and to the Drupal Association Supporters, who made it possible for us to work on these projects.

If you would like to support our work as an individual or an organization, consider becoming a member of the Drupal Association.

Follow us on Twitter for regular updates: @drupal_org, @drupal_infra

Aug 30 2016
tvn
Aug 30

One of the biggest content areas on Drupal.org—and one of the most important assets of any open source project—is documentation. Community-written Drupal documentation consists of about 10,000 pages. Preparations for the complete overhaul of the documentation tools were in the works for quite some time, and in the recent weeks we finally started to roll out the changes on the live site.

Background

Improving documentation on Drupal.org has been a part of a larger effort to restructure content on the site based on content strategy we developed.

The new section comes after a few we launched earlier in the year. It also uses our new visual system, which will slowly expand into other areas.

Goals and process

The overall goal for the new Documentation section is to increase the quality of the community documentation.

On a more tactical level, we want to:

  • Introduce the concept of "maintainers" for distinct parts of documentation
  • Flatten deep documentation hierarchy
  • Split documentation per major Drupal version
  • Notify people about edits or new documentation
  • Make comments more useful

To achieve those goals, we went through the following process:

First, we wrote a bunch of user stories based on our user research and the story map exercise we went through with the Documentation Working Group members. Those stories cover all kinds of things different types of users do while using documentation tools.

We then wireframed our ideas for how the new documentation system should look and work. We ran a number of remote and in person usability testing sessions on those wireframes.

Our next step was to incorporate the feedback, update our wireframes, and create actual designs. And then we tested them again, in person, during DrupalCamp London.
Incorporated feedback again, and started building.

The new system

So, how does the new documentation system work exactly? It is based on two new content types:

  1. Documentation guide: a container content type. It will group documentation pages on a specific topic, and provide an ability to assign 'maintainers' for this group of pages (similar to maintainers for contributed projects). Additionally, users will be able to follow the guide and receive notifications about new pages added or existing pages edited.
  2. Documentation page: a content type for the actual documentation content. These live inside of documentation guides.

Documentation guide screenshot
Example of a new documentation guide

All of the documentation is split per major Drupal version, which means every documentation guide or page lives inside of one of a few top level 'buckets', e.g. Drupal 7 documentation, Drupal 8 documentation.
It is also possible to connect guides and pages to each other via a 'Related content' field, which should make it easier to discover relevant information. One of our next to-do’s is to provide an easy way to connect documentation guides to projects, enabling 'official' project documentation functionality.

More information on various design decisions we made for the new documentation system, and the reasons behind them, can be found in our DrupalCon New Orleans session (slides).

Current status

Right now, we have the new content types and related tools ready on Drupal.org.
We are currently migrating existing documentation (all 10,000 pages!) into the new system. The first step is generic documentation (e.g. 'Structure Guide'), with contributed projects documentation to follow later.

While working on the migration, we are recruiting maintainers for the new guides. If you are interested in helping out, sign up in the issue. Please only sign up if you actually have some time to work on documentation in the near future.

There is a lot of work to be done post-migration (both by guide maintainers and regular readers/editors). The content is being migrated as-is, and it needs to be adapted for the new system. This means almost every single page needs to be edited. New fields (such as Summary) filled out with meaningful text (to replace text automatically generated by the migration script). A lot of pages include information for both Drupal 7 and Drupal 8, but this content needs to be split, with Drupal 8 information moved to pages in the appropriate version of the guide. These are just some of the steps that need to happen once the documentation has been migrated into the new system.

Next steps

As staff, we have a few follow-up tasks for minor improvements to the content types and tools. However, the bulk of the work is editing and improving the actual documentation, as I described above. This is in your hands now. Not only do we not have enough staff members to edit every single documentation page in a reasonable amount of time, we are also not subject matter experts for many of the topics, and so can't provide meaningful edits. The tools are ready, now it is up to the community to pick them up and write great documentation.

Documentation page screenshot
Example of a documentation page

Thank you

Lastly we want to say thanks.

Thanks to all the community volunteers who wrote those 10,000 pages over the years. Thanks to the Documentation Working Group members for their expertise, insight, and patience.

And, of course, thanks to staff. Unfortunately due to recent changes for the Engineering team, this will be the last section we'll have resources to work on for a while. This was a fun and important project to work on, and we are glad that we got to finish it. It is a beautiful legacy of the work we did together with some of our former colleagues: DyanneNova, japerry, and joshuami. Thank you!

Aug 24 2016
Aug 24

In recent weeks we've been making several small changes to Drupal.org: precursors to bigger things to come. First, we moved the user activity links to a user menu in the header. Next, we're moving the search function from the header to the top navigation. These changes aren't just to recover precious pixels so you can better enjoy those extra long issue summaries—these are the first step towards a new front page on Drupal.org.

As the Drupal 8 life-cycle has moved from development, to release, to adoption, we have adapted Drupal.org to support the needs of the project in the moment. And today, the need of the moment is to support the adoption journey.

As we make these changes you'll see echoes of the visual style we used when promoting the release of Drupal 8.

  • The Drupal wordmark region will help to define Drupal, and promote trying a demo.

  • A ribbon will promote contextual CTAs like learning more about Drupal 8.

  • The news feed will be tweaked.

  • DrupalCon will have a permanent home on the front page.

  • Community stats and featured case studies will be carried over(but may evolve).

  • The home page sponsorship format may change.

  • We'll be phasing in a new font throughout the site: Ubuntu - which you've already seen featured in the new Documentation section.

Here's a teaser

… a sneak preview of some new page elements and styles you'll see in the new home page.  

Our first deployment will introduce the new layout and styles. Additional changes will follow as we introduce content to support our turn towards the adoption journey. Drupal evaluators beginning their adoption journey want to know who uses Drupal, and what business needs Drupal can solve. We will begin promoting specific success stories: solutions built in Drupal to meet a concrete need.

What's next?

We're continuing to refine our content model and editorial workflow for the new front page. You'll see updates in the Drupal.org change notifications as we get closer to deployment.

Wondering why we're making these changes now? This turn towards the adoption journey is part of our changing priorities for the next 12 months.

Aug 12 2016
Aug 12

Read our Roadmap to understand how this work falls into priorities set by the Drupal Association with direction and collaboration from the Board and community.

The Drupal Association engineering team has been continuing to refine our focus for the next 12 months. In July, we worked through the details of setting new priorities for our work, after the organizational changes earlier this summer.

As part of this prioritization process, we've set up a technical advisory committee: a collaboration between a few members of the staff, a representative from the board, and two members from the community. This committee will help us refine the roadmap for Drupal.org for the short term—while the Association is focused on fiscal health and sustainability—and will provide strategic vision for the long term, as our fiscal stability improves.

As a result of these changes, you'll begin to see our updates in this blog series evolve. Expect a greater focus on:

  • The adoption journey for users evaluating Drupal.
  • Systematic improvements to make maintenance of critical Drupal.org services less labor intensive and more affordable.
  • Community initiatives, where we're working together with community contributors who want to help us improve Drupal.org.

So without further ado, let's talk about what we did in July.

Drupal.org updates

User Menu

We've moved the user activity links (Login/Register, My Dashboard, My Account, etc.) to a user menu in the top navigation. This change is live on www.Drupal.org and all of the sub-sites that use the Bluecheese theme. The immediate effects of this change are a better look and feel and more vertical space for content on every page. But these weren’t the primary motivation. The larger reason for making this change is that it’s the first incremental step towards upcoming editorial changes on Drupal.org.

More incremental changes will follow in August, including accessibility improvements to this new user menu and a new search icon to replace the embedded search box in the header.

Better Packaging Behavior

One of the basic features of Drupal.org's project hosting is packaging the code committed to our git repositories and providing tar.gz and zip files of releases. The packaging process, while generally reliable, has had its share of infrequent but persistent quirks and race conditions. In July, we fixed several aspects of packaging to eliminate race conditions and reduce the need for human intervention if it runs off the rails. The changes we made were:

Taken together, these changes have made packaging faster, more efficient, and less prone to race conditions that require staff time to fix.

Supporting Drupal 8.2

Drupal 8.2 is coming soon, scheduled for release on October 5th. The beta period for this point release began on August 3rd, and so towards the end of July we spent some time supporting the Core developers who were trying to get their features ready for inclusion in the beta period. In particular, we updated PhantomJS to version 2.1.1 in our DrupalCI containers, to allow Core developers to test javascript interactions for file uploads—part of the new quick edit features targetted for this point release.

Deprecated unstable releases

In July, we also deprecated the use of the “unstable” release tag for projects hosted on Drupal.org. Per our naming conventions, the unstable tag was intended to represent a release without a stable codebase, api, or database schema. However, this definition is largely redundant with the alpha tag and/or simply using dev releases. Beyond that, “unstable” is not a standard tag in semver, and is thus not supported by tools that rely on the semver standard, such as Composer. Existing releases tagged “unsable” on Drupal.org weren’t affected by this change, but no future releases with this tag will be packaged.

Drupal.org Composer repositories beta period continues

We’re still observing how the community uses the Drupal.org Composer repositories, and collecting feedback and issues as we move towards designating the service stable. We encourage you to begin transitioning your Composer-based workflows to use Drupal.org's Composer façade. Package names are stable, and downtimes will be planned and announced. For more information on how to use Drupal.org's Composer repositories, read our documentation.

Sustaining support and maintenance

Outage follow-ups

A raid array failure in our data center resulted in a brief outage in July. Fortunately, we were able to mitigate the issue and restore service until the affected array could be replaced. The rebuilt array increased our redundancy to avoid future outages when we experience multiple disk failures.

Backups

We also updated our backup process, and are now using a combination of Borg and rsync.net. The combination of borg for data deduplication and encryption and rsync.net's resilient cloud platform gives us an efficient and economical solution for backup and selective restoration.

Community initiative updates

These are initiatives to improve Drupal.org, driven by members of the community in collaboration with Drupal Association staff for architecture, review and deployment.

Documentation migration

Migration into the new documentation content types that began in June continues. The first sections of documentation being migrated are the Drupal.org docs and the Understanding Drupal guide. More volunteers to help migrate documentation are welcome!

if you are interested in helping, or sign up as a maintainer for some of the new documentation guides.

———

As always, we’d like to say thanks to all the volunteers who work with us, and to the Drupal Association Supporters, who made it possible for us to work on these projects.

If you would like to support our work as an individual or an organization, consider becoming a member of the Drupal Association.

Follow us on Twitter for regular updates: @drupal_org, @drupal_infra

Jul 15 2016
Jul 15

Read our Roadmap to understand how this work falls into priorities set by the Drupal Association with direction and collaboration from the Board and community.

In June the Drupal Association had our annual staff retreat, where the remote team members joined the Portland, OR team for a three day retreat. This year's retreat was particularly important as we found our feet as a smaller, leaner team, and focused on our organizational roadmap for the next twelve months.

For the engineering team in particular, our focus will be on maintaining the critical systems that make project successful: issue queues, updates, testing, packaging, etc, while at the same time finding new ways to support and enable Drupal's evolution.

These were some heady days, but even as we worked through the best ways to continue serving the Drupal community on a strategic level in June, we also found the time to keep making Drupal.org a better home.

Drupal.org updates

Documentation Migration

A long running initiative this year has been the creation of a new Documentation system for Drupal.org, a topic we've touched on in many prior updates as it has begun to come online. We are very happy to say that we are moving to the next stage of the documentation project: moving from development to migration.

In June tvn recruited several volunteers to join our documentation migration team, and to become some of the first maintainers for the new Documentation Guides. General documentation, such as Understanding Drupal, Structure Guide, etc. will be migrated first. Documentation for contributed projects will follow in the coming weeks.

Documentation Preview

Maintainers of contributed projects, who currently have their documentation on Drupal.org, will be added as maintainers to respective documentation guides and are encouraged to clean/tidy up their documentation post-migration.

if you are interested in helping, or sign up as a maintainer for some of the new documentation guides.

Composer Repositories are now in Beta

Drupal.org Composer Logo

Drupal.org's Composer repositories allow developers building sites with Drupal to use the Composer command line tool for dependency management. In June we collected feedback from a variety of users, as well as the community volunteers who assisted us with the Composer Community Initiative.

We spent the month iterating quickly on the alpha implementation: fixing bugs and rebuilding the meta data to ensure that users get consistent and expected results. Because of those fixes, and after gathering yet more feedback from the community, we were able to move the Drupal.org Composer repositories to beta.

We encourage you to begin transitioning your composer based workflows to use Drupal.org's composer facade. Package names are stable, and downtimes will be planned and announced. For more information on how to use Drupal.org's Composer repositories, read our documentation.

Better issue credit tools for maintainers

The Drupal.org issue credit system is a unique innovation of our community. By allowing users to attribute their contributions as volunteers, to their employers, or to client customers, we have an insight into the contribution ecosystem for Drupal that is unparalleled among open source projects. We've also already seen the impact of incentivizing organizations to give back to Drupal, by using the credit system as the basis for organization rankings in the marketplace.

Credit Others

Credit comment generated

In June we added two new tools for maintainers to improve how they grant credit to users. Firstly, maintainers can now deselect the automatic credit attribution for users who have submitted patches. This change was important to prevent gaming the credit system. Secondly, we've given the maintainers the ability to credit users who have not commented in the issue. Whether that help was provided in IRC, Slack, on a video call, or in a sprint room, maintainers can now ensure that those users who helped resolve an issue receive credit for their contributions. Any user who is credited this way can edit their credit attribution if they want to extend that attribution to a supporting organization or customer.

Friendly path aliases for release nodes

We also made a relatively small change that will have a big impact. Path auto is now enabled for project releases, so you for any project a specific release can now be found at:
drupal.org/project/[project_name]/releases/[version]
And you can also find a list of all the releases for a project at:
drupal.org/project/[project_name]/releases/

Take, for example, the Token module:
https://drupal.org/project/token/

You can find the complete index of releases for this project at: https://www.drupal.org/project/token/releases and individual releases now have friendly urls, like this one: https://www.drupal.org/project/token/releases/8.x-1.0-alpha2

Spam Fighting Improvements

Fighting spam on Drupal.org is a never ending battle, but in June we deployed a refinement to our spam fighting tools that helps us to find patterns in registration behavior and prevent spam registrations before they've even started. After flipping on our latest iteration of this spam fighting tool we saw an immediate and dramatic drop-off in suspicious account registrations. With the additional data we've been able to collect we already see ways to improve this even further, so we hope to continue make Drupal.org a cleaner home for the community.

Highlighting Supporting Technologies

Drupal is many things to many different people, but one central function of Drupal is to be the hub of interconnected and complementary technologies. Several of the companies that build these technologies have chosen to support the Drupal project by becoming supporters. To better highlight some of these supporting technologies that work well with Drupal, we've added a supporting technologies listing to the marketplace.

Sustaining support and maintenance

DrupalCon

DrupalCon Dublin Logo

DrupalCon Dublin is coming up soon, from September 26 - 30th. This year we smashed all our previous records for session submissions, and the caliber of speakers and topics is higher than ever before.

In June we opened registration for the event. We encourage you to buy your tickets now! Early bird registration will end soon.

Infrastructure

Infrastructure is the bedrock of Drupal.org - and we're continuing to tune the infrastructure for efficiency, economy, and performance. Alongside the launch of registration for DrupalCon Dublin, we implemented APDQC to improve the performance of the Events website under heavy load.

We've also been upgrading our configuration management from Puppet 3 to Puppet 4, and continuing to standardize our configuration across all of our environments to make our infrastructure durable, consistent, and portable.

———

As always, we’d like to say thanks to all the volunteers who work with us, and to the Drupal Association Supporters, who made it possible for us to work on these projects.

If you would like to support our work as an individual or an organization, consider becoming a member of the Drupal Association.

Follow us on Twitter for regular updates: @drupal_org, @drupal_infra

Jun 08 2016
Jun 08

Why test?

The goal of automated testing is confidence: confidence in application stability, and confidence that new features work as intended. Continuous integration as a philosophy is about speeding the rate of change while keeping stability. As the number of contributing programmers increase, the need to have automated testing as a means to prove stability increases.

This post is focused on how the automated testing infrastructure on Drupal.org works, not actually writing tests. Much more detail about how to write tests during Drupal development can be found in community documentation:

Categories of testing

DrupalCI essentially runs two categories of tests:

Functional tests (also called blackbox testing) are the most common type of test run on DrupalCI hardware. These tests run assertions that test functionality by installing Drupal with a fresh database and then exercising that installation by inserting data and confirming the assertions complete. Front-end tests and behavior driven tests (BDD) tend to be functional. Upgrade tests are a type of functional tests that run a full installation of Drupal, then run upgrade commands.

Unit tests run assertions that test a unit of code and do not require a database installation. This means they execute very quickly. Because of its architecture, Drupal 8 has much more unit test coverage than Drupal 7.

These test categories can be broken down further into more specific test types.

What testing means at the scale of Drupal

Drupal 8, with its 3,000+ core contributors and 7,288 contrib developers (so far), needs testing as a means to comfortably move forward code that everyone can trust to be stable.

Between January and May 2016, 90,364 test runs were triggered in DrupalCI. That is about 18,000 test runs requested per month. Maintainers set whether they want tests to run on demand, with every patch submitted, or nightly. They also determine what environments those tests will run on; there are 6 combinations of PHP and database engines available for maintainers to choose from.

The majority of these test runs are Drupal 8 tests at this point. (19,599 core tests and 47,713 contrib project tests were run during those 5 months.) Each test costs about 12 cents to run on Amazon Web Services. At the time of writing this post, we averaged around $2,000 per month in testing costs for our community. (Thank you supporters!)

An overly simple history of automated testing for Drupal

Automated testing first became a thing for Drupal contributed projects during Drupal version 4.5 with the introduction of the SimpleTest module. It was not until Drupal 6 that we started manually building out testbots and running these tests on Drupal.org hardware.

In Drupal 7, SimpleTest was brought into Drupal Core. (More information about what that took can be reviewed in the SimpleTest Roadmap for Drupal 7.)

In Drupal 8, PHPUnit testing was added to Drupal Core. PHPUnit tests are much faster than a full functional test in SimpleTest—though runtest.sh still triggers a combination of these test types in Drupal 8.

The actual implementation of automated testing was much more complicated than this history suggests. The original testbot infrastructure that ran for 7 years on Drupal.org hardware was manually managed by some fiercely dedicated volunteers. The manual nature of that maintenance led to the architecture of DrupalCI, which was meant to make it easier to test locally at first and later focused on autoscaling on powerful hardware that could plow through tests more quickly.

DrupalCI's basic structure

In The Drupal.org Complexity, we could see the intricate ways that Drupal's code base interacts with other parts of the system.

Representation of the relationships between services and sites in the Drupal.org ecosystem.

We could further break out how systems like DrupalCI are interrelated.
Highlighted relationships between testing and other services.

DrupalCI is a combination of data stored on Drupal.org, cron jobs, drush commands, and most importantly a couple of Jenkins installations to manage all the automation.

Jenkins is the open source automation server project that makes most of the system possible. We use it for automating our build process and deploying code to our dev, staging and production environments. It automates just about anything and is used by companies small and large to run continuous integration or continuous deployment for their applications. It's considered a "best practice" solution alongside options like Travis, CircleCI, and Bamboo. They all have slightly different features, but automation is at the core of most of these DevOps tools.

To provide continuously integrated tests, you need to trigger those tests at a moment when the tests will have the greatest value.

The three triggers for running a test job are when a patch is added to an issue comment, when code is committed to a repository or daily on a cron. Maintainers can specify which triggers are associated with which branches of their projects and which environments should run those tests.

For core these settings look something like this:

Screenshot of the automated testing settings for Drupal Core.

This detail allows for specific tests to run at specific times per the Drupal.org Testing Policy for DrupalCI.

To make this automation happen, we have an installation of Jenkins (Infrastructure Jenkins below) that is polled by Drupal.org once per minute with testing jobs to be queued.

These jobs live in a database record alongside Drupal.org.

The infrastructure jenkins instance polls Drupal.org once per minute looking for new jobs to queue.

Infrastructure Jenkins speaks to the CI Dispatcher (also Jenkins) where the testing queue regularly passes those jobs to available testbots. CI Dispatcher uses an Amazon Web Services EC2 plugin to spin up new testbots when no existing testbot is available. Each testbot can spin up Docker containers with the specific test images defined by the maintainer. Theses containers pull from DockerHub repositories with official combinations of PHP and database engines that Drupal supports.

The CI Dispatcher maintains the queue of jobs to run. When a job is ready, it uses an EC2 plugin to use an existing testbot or spin up a new bot as needed.

After a testbot is running, the CI Dispatcher is in constant communication with the bots. You can even click through to the console on CI Dispatcher and watch the tests run. (It's not very exciting—perhaps we should add sound effects to the failures—but it is very handy.)

Once the testbot has been spun up, the CI Dispatcher listens to it for results.

Once per minute, Drupal.org polls the CI Dispatcher for test status. It responds with pending, running, failed or passed. Failed and passed tests are then pulled back into Drupal.org for display using the Jenkins JSON API.

 pending, running, failed, passed. All the results are pulled back into Drupal.org using the Jenkins' JSON API.

Tests can also be run on demand at the patch, commit or branch level using the handy "add test" and "retest" links.

Why did we build this ourselves? Why not use [insert testing platform here]

Lot's of people have asked why we don't use TravisCI, CircleCI or some other hosted testing solution. The short answer is that most publicly available testing systems require Github authentication and integration.

Additionally, our testing infrastructure is powerful because of its integration with our issue queues. Read the aforementioned The Drupal.org Complexity for more information.

Another reason to run our own testing is scale. To get through all of the core tests for Drupal 8 in an acceptable amount of time (about 44 mins on average), we run very large testbots. These bots have 2 processors with 8 hardware cores. With hyperthreading, that means we have 32 hardware execution threads—about 88 EC2 compute units. They are not exactly super computers, but they are very performant.

We average nearly 18,000 test runs per month. During our peak usage we spin up as many as 25 testbot machines—though usually we cap at 15 bots to keep costs under control. This helps us plow through our testing needs during sprints at DrupalCons and large camps.

We have explored using an enterprise licensed version of either Github or CircleCI with our own hardware to tackle testing. That same consideration has been given to SauceLabs for front-end testing. Right now, there is not a cost savings to tackle this migration, but that does not rule it out in the future. Testing services continues to evolve.

Accelerating Drupal 8

In my first months as CTO, I was told repeatedly that the most important thing for us to work on was testing for Drupal 8. In those early days as I built out the team, we were mostly focused on catching up from the Drupal 7 upgrade and tackling technical debt issues that cropped up. In DrupalCon Austin, I had members of my team learn how to maintain the testbot infrastructure so that we could take over the process of spinning up bots and dealing with spikes in demand.

By early 2015, we had optimized the old testbots as much as they were going to be optimized. We moved them to AWS so we could spin up faster machines and more bots, but there were features that were waiting on the new DrupalCI infrastructure that were blocking key development on Drupal 8.

In March of 2015, we invited all the community developers that had helped with DrupalCI to the Drupal Association offices in Portland and sprinted with them to figure out the remaining implementation needs. The next couple of months involved tweaking DrupalCI's architecture and cutting out any nice to have features to get something launched as soon as possible.
It is no coincidence that from the time of DrupalCI's launch until the release of Drupal 8, progress was rapidly accelerated.

I am immensely proud of the work of all the community members and staff that worked directly with core maintainers to unblock Drupal 8 development and make it faster. This work was critical.

Thank you to jthorson, ricardoamaro, nick_schuch, dasrecht, basicisntall, drumm, mikey_p, mixologic, hestenet, chx, mile23, alexpott, dawehner, Shyamala, and webchick. You all made DrupalCI. (And huge apologies to all those I'm undoubtedly leaving out.) Also thank you to anyone who chimed in on IRC or in the issue queues to help us track down bugs and improve the service.

What's next for testing Drupal

Most of the future state of testing is outlined in the Drupal.org Testing Policy for DrupalCI.

Key issues that we still need to solve are related to concurrent testing improvements and new test types to support. While we have PhantomJS integrated with the test runner, there are optimizations that need to happen.

Testing is not an endpoint. Like much of our work, it is an ongoing effort to continuously improve Drupal by providing a tool that improves how we test, what we test, and when we test.

Pages

About Drupal Sun

Drupal Sun is an Evolving Web project. It allows you to:

  • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
  • Facet based on tags, author, or feed
  • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
  • View the entire article text inline, or in the context of the site where it was created

See the blog post at Evolving Web

Evolving Web