On Drupal's Leadership

My DrupalCon Amsterdam Core Conversation on Managing Complexity has generated quite a bit of follow-up discussion. That's good; it's a conversation we as a community really need to be having.

There are a few points, though, that I feel bear clarification and further explanation as I fear the point of the talk has gotten lost in the details.

Before continuing, if you haven't yet I urge you to watch the session video as well as the background resources linked from the session page. This is not a new conversation; it's the latest chapter in a very long-running discussion that is larger than the Drupal project, and it behooves us all to be aware of the history and context around it.

First of all, a number of people have reacted very negatively to my comment that Initiative Leads and, to a lesser extent, subsystem maintainers are "glorified secretaries" (despite the fact that I was referring to myself, derisively or not). There are probably some fascinating sociological reasons why that word carries far more intense negativity for some, and certainly far more than I intended, but that's a discussion for another time.

To the point at hand, I was specifically referring to the official, in-writing job descriptions of Initiative Leads. The only written, referenceable description of what Initiative Leads do is, again:

Initiative Leads have the following responsibilities:

• Coordinate work on the initiative.
• Communicate the plans, progress, and needs of the initiative to the community and the branch maintainers.

"Coordinate" and "communicate". Those are both fundamentally clerical tasks (hence "secretary", a clerical position). That's not to say unimportant; Good communication and coordination is critical for any medium to large scale endeavor, software or otherwise. (Having worked on projects without a project manager before, I will never do so again.) But "just" communicating does not in any way imply "leadership".

When I asked in my session who the Technical Lead was for Drupal 8, someone jokingly called out that I was. In all modesty it's true that I have had an enormous impact on Drupal 8's architecture and direction, some of it under the aegis of WSCCI, some not. But none of that came from being an initiative lead; it came from being an outspoken, persuasive, stubborn git.

My fellow initiative lead, Gábor Hojtsy, has responded to the discussion by discussing the benefits of "soft power": The ability to persuade rather than coerce, advocate rather than force, etc. To be sure, "soft power" has its benefits, and it's well established that "people skills" are just as important as applied skills in whatever the task at hand is (technical or otherwise). I also don't want to diminish Gábor's unquestionable skills in those areas. However, Gábor contrasts it with "hard power" by saying that "hard power" is about ordering people what to do and throwing around one's "authority", a concept that doesn't fit well with volunteer organizations.

That is, quite simply, a straw man.

Soft power, for all its benefits, is ultimately a form of informal structure. Informal structure, as outlined by Jo Freeman and as I quoted extensively in my session, is by its nature very fluid and opaque. It is inherently complex, because different responsibilities and relationships overlay on top of each other in unpredictable and undocumented ways. Informal structure is inherently not transparent. Anyone who values transparency should be wary of informal structure (and, by extension, over-reliance on "soft power") as it runs directly contradictory to that goal.

A good formal structure offers something that an informal structure cannot: Clarity. If a new contributor wants to ask someone in a "leadership" position if what they're proposing is consistent with the project direction, goals, or software design, who should they ask? Me? Gábor? catch? xjm? Daniel Wehner? Tim Plunkett? Angie Byron? Alex Bronstein? chx? Dries himself? Depending on the question the answer could be very different; even among those "leaders" I doubt there's consensus on which of them to ask, much less what the answer should be. That's actively hostile to new contributors, because then they don't know if what they're suggesting (be it big or small) would even be accepted until it's committed (and occasionally not even then). That is, it makes most contributions spec work, even for those in de facto leadership positions.

It also contributes to the belief that I've heard from numerous Drupal-savvy but not core-involved people that contributing to core involves posting a patch; waiting a few months; getting 300 comments; someone new rewriting the patch; refiling as a duplicate of another issue two or three times; reopening the issue; and then a patch sitting at "needs review" forever because no one can make a decision. (Yes, I have heard that story given by people that could be contributing to core, but aren't because the process barrier to entry is too high.)

With all that informality, it's quite easy to simply ignore the informal structure case-by-case when it's convenient. That's very disrespectful to those who have earned positions within the informal hierarchy through whatever means, and it prevents people from being able to "step back gracefully" as our Community Code of Conduct calls for. If you don't know if you've stepped up, how do you know to step down? As I noted in the session, Paris Liakos and I completely rewrote the logging system for Drupal 8, without the blessing or even awareness of the maintainer of the two logging modules in core. Of course, he wasn't there to tell us no so, and no one else suggested involving him. So did we do it right or were we backstabbing him? In all honesty I don't know.

I do know, however, that elbowing out de facto leaders who don't want to be elbowed out is a very painful and ugly process, one that has hurt the Drupal community and several initiatives over the Drupal 8 cycle; it was worse in 2011 and 2012 than recently but still, it is harmful to people. That process drove Jacine Luisi (HTML5 Initiative Lead) out of Drupal. Yet that is the primary way that de facto informal soft power transitions.

Drupal's extensive documentation and coding standards, for all the seemingly unnecessary anality, do have the distinct advantage of of being objective and transparent. There's very little in Drupal you can say that about.

Another failing of soft power is that it privileges those with an excess of time or an excess of persuasiveness, neither of which necessarily correlate in any way to an excess of skill or expertise at the task at hand. You want people in positions of authority who know what they're talking about, who understand the problem space, are connected with the various threads of work involved, and, yes, have the people skills to persuade and coordinate. But they also need to have the backing to make hard decisions if for whatever reason "consensus" doesn't work (as it often does not). They need to be understood as the domain expert in whatever the domain in question is, and that understanding needs to be transparent. If it's based entirely on soft power and who remembers that so-and-so wrote a particular piece of code then it is anything but transparent.

Fears of "hard power" people just throwing their weight around are also exaggerated. It has been my experience (in three different organizations, Drupal being the 3rd) that when you give someone with soft-power a more formal position with responsibility and the authority to be able to follow through on it one of two things happens:

1. They become mad with power and throw their weight around, trampling all over those that get in their way.
2. They adopt a broader perspective and become more collaborative, more open to discussion, and more amenable to reasonable disagreement than they ever were before because their position in the informal de facto pecking order and ability to influence direction is not under constant (real or perceived) threat. This is often an entirely unconscious distraction before hand that is neatly removed through formal recognition.

That is, if you are in an informal soft de facto position of leadership, part of your attention is always taken up by whether or not you still are at any given time. (Not necessarily in a defensive way, but that can happen too even without anyone being "power hungry".) But that's the last person you want distracted by court politics.

Naturally you want to have a lot of people from group 2 and avoid people in group 1. But it is self-defeating to let the fear of group 1 blind you to the benefits of group 2. Yet the Drupal project has largely let its fear of group 1 and it's zeal for the "one hour, one vote" of Do-ocracy hamstring our ability to leverage and capitalize on the benefits that formal structure can provide (transparency, clarity, efficiency, accountability, etc.). Again, quoting Jo Freeman:

If the movement continues deliberately to not select who shall exercise power, it does not thereby abolish power. All it does is abdicate the right to demand that those who do exercise power and influence be responsible for it. If the movement continues to keep power as diffuse as possible because it knows it cannot demand responsibility from those who have it... it insures that the movement is as ineffective as possible.

Formal structure can certainly be taken too far as well; requiring all patches to be filled out in triplicate with a cover letter attached to the TPS report would be lunacy. But as software passes Norris numbers and communities pass Dunbar numbers formalities need to be put into place to allow both to scale. It's a "right tool for the job" question, and the larger the problem space the more right a formal structure becomes.

Just as procedural code is fine up to a certain level of complexity and then needs the increased rigor of more formal methods like OOP, dependency injection, functional programming, data types, and so on in order to be sustainable so too can a community rely solely on "rough consensus and working code" only up to a point, and then needs to build in more formal structures and processes to allow it to grow.

Ultimately, this debate should not be about hard power vs. soft power. It's not an either-or proposition. A good, sustainable structure happens at the intersection of hard power and soft power; at the healthy combination of formal and informal structures, in a way that is appropriate to the scale of the organization or community.

Right now, Drupal leans far, far too much on soft power and informal structure, and that is harming us, as a community and as a code base. We need to bring that into balance in a way that recognizes just how big Drupal has become.

What sort of formal structure should we add? That's open for healthy debate. Sam Boyer has proposed one model. I've proposed one, taking inspiration from the Linux project. Lee Rowlands has proposed one, taking inspiration from a number of other projects. These are the discussions we should be having.

We should also look outward and dig into how other "super-large" Open Source projects handle these issues. There are only 8 others besides Drupal; it shouldn't take that long to research.