Nov 30 2018
Nov 30

In software just about all project management methodologies get labeled one of two things: Agile or Waterfall. There are formal definitions of both labels, but in practice few companies stick to those definitions particularly in the world of consulting. For people who really care about such things, there are actually many more methodologies out there but largely for marketing reasons we call any process that’s linear in nature Waterfall, and any that is iterative we call Agile.

Classic cartoon of a tree swing being poorly because every team saw it differently.Failure within project teams leading to disasters is so common and basic that not only is there a cartoon about it but there is a web site dedicated to generating your own versions of that cartoon (http://projectcartoon.com/).

Among consultants I have rarely seen a company that is truly 100% agile or 100% waterfall. In fact I’ve rarely seen a shop that’s close enough to the formal structures of those methodologies to really accurately claim to be one or the other. Nearly all consultancies are some kind of blent of a linear process with stages (sometimes called “a waterfall phase” or “a planning phase”) followed by an iterative process with lots of non-developer input into partially completed features (often called an “agile phase” or “build phase”). Depending on the agency they might cut up the planning into the start of each sprint or they might move it all to the beginning as a separate project phase. Done well it can allow you to merge the highly complex needs of an organization with the predefined structures of an existing platform. Done poorly it can it look like you tried to force a square peg into a round hole. You can see evidence of this around the internet in the articles trying to help you pick a methodology and in the variations on Agile that have been attempted to try to adapt the process to the reality many consultants face.

In 2001 the Agile Manifesto changed how we talk about project management. It challenged standing doctrine about how software development should be done and moved away from trying to mirror manufacturing processes. As the methodology around agile evolved, and proved itself impressively effective for certain projects, it drew adherents and advocates who preach Agile and Scrum structures as rigid rules to be followed. Meanwhile older project methodologies were largely relabeled “Waterfall” and dragged through the mud as out of date and likely to lead to project failure.

But after all this time Agile hasn’t actually won as the only truly useful process because it doesn’t actually work for all projects and all project teams. Particularly among consulting agencies that work on complex platforms like Drupal and Salesforce, you find that regardless of the label the company uses they probably have a mix linear planning with iterative development – or they fail a lot.

Agile works best when you start from scratch and you have a talented team trying to solve a unique problem. Anytime you are building on a mature software platform you are at least a few hundred thousand hours into development before you have your first meeting. These platforms have large feature sets that deliver lots of the functionality needed for most projects just through careful planning and basic configuration – that’s the whole point of using them. So on any enterprise scale data system you have to do a great deal of planning before you start creating the finished product.

If you don’t plan ahead enough to have a generalized, but complete, picture of what you’re building you will discover very large gaps after far too many pieces have been built to elegantly close them, or your solution will have been built far more generically than needed – introducing significant complexity for very little gain. I’ve seen people re-implement features of Drupal within other features of Drupal just to deal with changing requirements or because a major feature was skipped in planning. So those early planning stages are important, but they also need to leave space for new insights into how best to meet the client’s need and discovery of true errors after the planning stage is complete.

Once you have a good plan the team can start to build. But you cannot simply hand a developer the design and say “do this” because your “this” is only as perfect as you are and your plan does not cover all the details. The developer will see things missed during planning, or have questions that everyone else knows but you didn’t think to write down (and if you wrote down every answer to every possible question, you wrote a document no one bothered to actually read). The team needs to implement part of the solution, check with the client to make sure it’s right, adjust to mistakes, and repeat – a very agile-like process that makes waterfall purists uncomfortable because it means the plan they are working from will change.

In all this you also have a client to keep happy and help make successful – that’s why they hired someone in the first place. Giving them a plan that shows you know what they want they are reassured early in the project that you share their vision for a final solution. Being able to see that plan come together while giving chances to refine the details allows you to deliver the best product you are able.

Agile was supposed to fix all our problems, but didn’t. The methodologies used before were supposed to prevent all the problems that agile was trying to fix, but didn’t. But using waterfall-like planning at the start of your project with agile-ish implementation you can combine the best of both approaches giving you the best chances for success.  We all do it, it is about time we all admit it is what we do.

Cartoon of a developer reviewing all the things he's done: check technical specs, unit tests, configuration, permissions, API updates and then says "Just one small detail I need to code it."Cartoon from CommitStrip
Feb 12 2017
Feb 12

This is first of a series of blogs to support traditional project managers I am coaching. To help get their bearings in deep and murky waters of Agile projects and Scrum teams.

Before the scrum purists amongst you vehemently shake your heads or berate me on the title, consider being pragmatic. In the Professional Services world there is always a project manager to manage complexity and facilitate the Scrum team(s). My remit is to facilitate and empower the role to help the team, business and customers succeed, rather than debate its applicability and existence.

“Obey the principles without being bound by them.”
– Bruce Lee

I’ll be deep diving into a PM’s role in context to specific Scrum ceremonies in upcoming posts, however its seems apt to start with some health warnings.

Toxic Behaviour for a Scrum team

Toxic Behaviour for a Scrum team

A) This is not a guide for you to try and replace the Scrum master (you cannot) or the product owner (you cannot), or both! (you cannot). Nor is it a reference for you to justify imposing your will on the team (you cannot). It’s a guide to enable you to add ‘value’ to the ‘Scrum team’ and fulfill your purpose of managing risk on complex engagements.

B) Please don’t try to fake it till you make it! you will be caught out and the team will loose respect for you. if you don’t know, embrace not knowing and work to change that. Learn, up-skill, ask for help, do a pre-project retrospective on your own experience and discuss it with your Scrum master and/or Agile coach (if you have one). If you go in waving your strengths and weaknesses we will respect you for your courage and openness… they are part of our value system

C) Own your failures and reflect on them with the scrum master and/or an Agile coach. Don’t look for patsies, the team
will see through it and you will be called out on it. If you are failing, own it, be courageous and open, respect the knowledge and skills your Scrum team has (and you don’t) and in turn you will earn their respect. If you blame someone else for your shortcomings so that you can hide behind them you do not have the DNA to be in an agile environment. 

Agility, Scrum it’s a culture thing!

Agile Culture

Agile Culture

In order for you (a PM) to facilitate a Scrum team (yes your role is one of facilitation only) it is essential that you understand and embrace an agile culture, not just follow parts of a framework (Scrum). Toe-dipping is not going to work, you’re either committed or not.. its time to quit being a chicken and start living like a Pig

……………………

Lastly, If you got value from what I have shared please consider giving back by contributing to @BringPTP, you can follow, broadcast or donate.

Peace Through Prosperity (PTP) works to improve the environment for peacebuilding by nurturing prosperity in conflict affected communities. We work to alleviate poverty and secure livelihoods through empowering micro-entrepreneurs with knowledge, skills and increasing their access to income and opportunities. We support small businesses, owned/managed by vulnerable and marginalised individuals/groups in society.

Apr 28 2016
Apr 28

What do the best agile Drupal support teams, plasma laser systems,

and flocks of birds have in common? 

 They’re self-organizing. And you’ve got to see the big picture to get it. I’ll show you.

agile drupal support

First off, I suppose should tell you that I’ve been studying self-organization as a scientific discipline for the last decade. It’s a good thing I like this stuff, right? During this 10 year journey I’ve jumped into various subjects: plasma laser systems, educational technology, robotics and computer simulations, Drupal project management, and of course: organizational dynamics.

What I found is that when you get the depths of each of these topics, it’s too easy to see the similarities. Each allows for a systemic view—the big picture.

Download: Continuous Discovery White Paper (PDF)

But before we get in too deep here, let’s cover some basics. A system is a collection of interacting components, each having its purpose. The what doesn’t matter; it’s about the how. It could be plasma particles in laser-plasma interactions, it could be a team working on a project, it could be functional groups interacting to form an organization, it could be a flock of birds flying together constantly maintaining distance. The list is endless.

What we’re talking about here are fixed systems. System that:

(A) Can produce the same outcomes in different ways in the same environment and different results in the same and different environments

(B) Can not only learn and adopt but also create

Agile Drupal support teams are great examples.

They are in effect flexible, scaleable systems that have to naturally adapt to function in a changing environment or perform within a constant environment. Agile facilitates evolution and adaption, meaning fully-agile Drupal teams can do this.

So how does this science work with agile Drupal teams?

Agility in Drupal project management is a simple way to look at project teams as self-organizing nonlinear systems. This system is a network of interacting autonomous entities, all working towards the project’s goal. Each team member is autonomous, and the goal is value-driven.

Download Drupal Staff Augmentation Case Study

As an example, Scrum which is an agile model with its three pillars of transparency is a framework that allows the teams to work towards the objective that makes the team most productive. Through this, team productivity is far more important than individual productivity. So while autonomy is taking place, collective achievement is the end-result.

The systemic view of agile teams also explains why clients are expected to be an active participant in the dynamics. The client is a component of the whole system which impacts the input flux, the project’s requirements, and output flux, the project’s releases, by defining priorities and needs.

Every kind of system needs a set of guiding rules.

Scrum, one kind of an agile technique framework for things like agile Drupal QA, is nothing but a set of rules that the project team as a system agrees to play by to meet their goals. Scrum isn’t a methodology—it’s a set of steps that have to be followed, like a manual.

The objective of any system is to improve continuously and emerge. Emergence here means achieving something that wouldn’t be possible independently. It takes an integrated team.

If we look at a project team, scrum does allow agile Drupal retrospectives as a tool or technique for continuous improvement. The main purpose of any environment where any system operates is to design interventions towards improved performance of the system.

Self-Organizing Drupal Support Teams?

So in this way, the best Drupal team can be a self-organizing system, and why not? The science does not limit the volume or size of the system. Perhaps a better word would be a self-organizing dynamic system, through constant retrospection and agility. You can bring stability to chaotic situations—or steep and changing demands or requirements—to meet a shared goal or organizational vision.

Today the discipline of System Dynamics is being used to model the software development life cycle, SDLC, processes we commonly use today. Or at least, that’s what we’re using successfully with our agile Drupal support teams. It works.

You can make this happen.

Applying agile and enabling self-organization might cause some growing pains at first. At Axelerant, we’re transitioning our content department to agile frameworks. It takes time. But for many organizations—perhaps yours?—the way teams go about creating things like dynamic software needs to change.

How?

Agencies with flexible Drupal services, centered on agile practices and self-organizing principals, can help bring this to you. Because enabling self-organization may not be an in-house option right now. But one thing is absolutely for certain—whether you partner with a team like ours of if you go about self-organizing yourself. Things will never be the same.

Want an agile Drupal support on your side?

This article was originally published on November 11, 2015. It has since been updated.

Karuna Batra

Karuna Batra

Karuna is Axelerant's Director of Delivery and a strategy team member. She focuses on evangelizing, institutionalizing, and auditing best practice processes via open source tools and appropriate governance structures for delivery based organizations. She's more importantly a loving aunt and Indian foothills trekker.

Nov 03 2014
Nov 03

I’ll be giving a talk at BADCamp called “Mistakes I have Made: Collected Project Management Failures.” It’ll be funny, and true, and probably reference a few different stories from the past. However, when I look at what the real truth is to a talk about project management mistakes, I go to the source. What are the hardest things about project management?

If you google this, you’re likely to get the answer of ‘everything’.

For me, when I look at this, I separate this out into five different areas of hard:

  1. Team
  2. Clients
  3. Services
  4. Alignment / Mission  
  5. General Sucking: Hard decisions

1. Team

Around the area of team, I think this is one of the biggest things that you’re hired to watch over as a project manager. It’s your job to make sure that you’re engaging the team to make sure that the problems get solved, that you’re building what you set out to build. It’s so common in my own work that when I feel like there is friction or struggle, I have to lean back and ask myself if I’ve actually worked on really engaging the team – or if I’ve just charged ahead full bore.  (Mistake #543)

Do I have the right team?

I have definitely been on projects or working with teams that just didn’t have the right fit. There had never been a conversation about if the people working on that particular ‘thing’ were right for it. (Mistake #324) Or they were being used in ways that didn’t suit them personally, they were being asked to use their weakest skills in a really strong way, and it was burning them out. (Mistake #221)

Did I give everyone enough time?

This is where people will use Agile to its best advantage. Agile, when you’re actually tracking story points and estimating, will give you enough space to be able to understand if you’ve crammed too much in. (Mistake #112) Did we ask too much in that space from people? (Mistake #14) Are we working under a really silly timeline (Mistake #87) and did we not find out until too late? (Mistake #98.) Even more to the point, once we know the problem, are we not willing to correct it? (Mistake #465)

2. Clients

Do we have the right clients? Are we, the people building things for them, the right people to do it? Do we understand their mission? Did we do enough to make sure they understood when, how, and where we were going to build? Do they understand what we’re not doing? Are they ok with their role here? Do they understand ‘scarcity’ in action? Does that shock them?

The list goes on, but in order to be the best technology partner, and you’re helping to lead everything to the finish line, everyone has to agree.

3. Services

This is the day-to-day stuff and where new project managers that are client facing get tripped up. Answering for the developers when you shouldn’t. “That should be easy.” Making estimates with having no idea what you’re doing. Estimating things in general. Making the expectations that you’re going to be available all the time. Or not setting the expectations at all.

Services isn’t the hardest-hardest part of this, but it generally adds to it.

4. Alignment / Mission

There’s a statement running around the internet right now that working on things that aren’t aligned with your values is just stressful. Working on things that are is called passion. You’ll notice when it’s just not working, because you can’t get rid of that pit in your stomach, that sinking feeling. Not listening to it is a big mistake.

5. General Sucking: Hard decisions

Part of your job is being the person that says no, that continually tries to find a way to make the project a success. You’ll sometimes be in the place where everything, all of these 4 things above are collapsing around you. And it sucks. And it happens to a lot of us that do this, because you are the one that’s pulling the threads together.

It gets better.

You are not always going to have these weeks. I refer to them as ‘hell weeks’. Please keep your hands and arms inside the hell week until it comes to a full and complete stop. In these weeks, it’s even more important for you to pay attention to you. Get some sleep. If you can’t get some sleep, talk to your team about how you are feeling, and figure out how to articulate what the problem is so that everyone can solve it together. (Also, ask yourself how you’re working with your team!)

Internalize the idea that you are no good to us dead, and take care of what you can. Projects end, one way or another, and you’ll get through it – one email, one day at a time. These things are hard, projects are hard, building things is hard – but hard things are worth doing.

Hope to see you at BADCamp for my session “Mistakes I have Made: Collected Project Management Failures.” Check out all of the Phase2 thought leaders at BADCamp!

Sep 06 2013
Sep 06

In Episode 25 of the Drupalize.Me Podcast, Kyle Hofmeyer takes time to gather some Lullabots to discuss "developer workflow". He is joined by Lullabot's Senior Technical Project Manager, Jerad Bitner, Senior Developer, Joe Shindelar, and first-time project manager, Emma Westby. In this episode the four bots share their thoughts on how they approach working in a team and the tools they use to make the process of sharing code easier. Everything from Git to peer reviews, to branch naming conventions, and establishing a workflow with your team is discussed. Lullabot has thought a lot about making the developer experience more efficient--in this podcast one of our internal pull request builders, Tugboat, is also discussed. But this podcast is really just the tip of the iceberg. If you're excited to learn more about optimizing your own processes, Joe and Emma will be leading a workshop in Prague (there are still a few spots open).

Mar 06 2013
Mar 06

Lullabot is a 100% distributed company: here's how we've learned to keep things running smoothly!

In the beginning

When I tell new acquaintances that I work for a distributed company, they inevitably ask: “You do all of that from home? How?” We discuss this how on a regular basis, but haven't taken these conversations outside our figurative halls. In this article, all will be revealed!

Lullabot is a distributed company with clients all around the world—we don’t have a single office and almost all of us work from home. You might think that means we’re disconnected (not at all!), that we are lonely (never!), or bored (ha!). You might even think we’re nerds! You’d be right on that count, but that’s irrelevant to our working arrangements.

Being distributed requires a more deliberate approach to communication and getting “in the zone.” Here are some of the highly effective habits of our intercontinental team:

Get dressed like you’re going to work

Nine out of ten Lullabots agree (this is a made-up statistic, but keep reading) that their productivity is boosted when they prepare for the day as if they’re leaving the house to go to work. That means pants! And shoes! You don’t have to bust out a suit, but readying yourself for a purpose helps you get into a focused work mode. (The fact that you don’t actually have to leave the house just makes it a little bit happier.)

Have a schedule

To paraphrase our Director of Operations, Seth Brown: “Find your ideal schedule or pattern for productivity. For me, it involves working 8 to 6 with two hours for a long lunchtime workout. This restores me. For someone else, it might be about starting work at 10, having a nap in the afternoon, but working in the evening when they feel most productive. The key is finding your groove. We’re all unique. There are morning people, evening people, and everything in between. We have the flexibility to choose, but it’s important to be consistent with that choice so others can plan around you.”

Getting up early is the most productive time of the day (by far) for Karen Stevenson, Senior Drupal Architect. Karen's daily schedule: get to work early (usually around 4 a.m.), get done early and enjoy a walk or some family-time, and come back later if need be. For Sean Lange, a front-end developer, having set work hours and a daily routine is what works. The importance of a schedule seems to be key for all of us, no matter how offbeat that schedule might be. Have an idea where your time will be spent each day, stick to a pattern that feels comfortable and repeatable, and let your team members know when you’ll be in your groove. Otherwise, time tends to slip away and you find yourself “working” all the time with reduced productivity.

Draw a boundary

As I was doing interviews for this article, many Lullabots mentioned the importance of boundaries. Senior Drupal Architect Jeff Eaton argues, “The valuable thing about getting dressed like you’re going to work, and similar rituals, is that it puts a distinct dividing line between schlepping around and serious business. Because so many of the ‘Bots work on things they find interesting and are passionate about, it’s easy to stay half-connected 24/7. I’ve found that explicitly disconnecting makes it easier to focus when I am connected, regardless of the mechanism used to draw that dividing line.”

Angus Mak, Developer, says: “I never run straight to the computer after waking up. I always take some time to make coffee and let the dog out, so I don’t feel like I’m working before I’m even awake.”

Have a dedicated workspace (and/or device)

Having a home office is crucial for me. I need a desk with a proper chair to really direct my attention to work projects. Otherwise, I fear I’d be floating about the house, distracted by home duties and shiny objects. Having a dedicated workspace helps keep me organized and efficient. Jeff Robbins, Lullabot CEO & cofounder agrees: “It’s good to have a home base where both you and your family know that you’re working. Not only is this a good physical, personal reminder, but it also sends a message (Don’t bother Dad!) to your family without needing to interrupt your work to explain it. Working at home is living at work. Do what you can to differentiate work and leisure time.”

“I also use a different computer with nothing work related on it for entertainment,” says Angus Mak. “When I’m done with work, I switch to my personal computer. I still check work emails on the phone, but at least I don’t see any work related files, programs, code, etc. on my personal computer that might suck me back into work.”

If a separate computer isn’t in the cards for you, you can create separate accounts on your computer. Using one for work, and personalize the other one with your home life in mind.

While we’re talking about work devices, Jeff Robbins advises: “Spend the extra money for a quality microphone headset. If you work virtually, it isn’t just an adjunct communication device. It’s how you’ll be hearing all of your clients and colleagues, and it’s also how they’ll be hearing you. There’s nothing worse than being that guy that people can’t quite understand on the conference call. Never use speakerphone or your computer’s built-in microphone on conference calls. Ever.”

Be active

If there’s one thing we’ve learned, it’s that Lullabots can “walk the talk," or at least, walk while they talk. I was surprised to hear how many of us walk, hike, or stand when we’re on long phone calls. According to Seth Brown: “Walking on phone calls is critical and gives me the endurance to get through the day. When I’m in front of the computer I don’t listen well—there are too many distractions.” Karen Stevenson advises, “Stand up while you’re on phone calls, at least the important ones.” Not only does this keep us focused on the phone call (instead of being distracted by other online tasks), it keeps our bodies and minds alert during a time when our attention could easily wander.

Another great piece of advice comes from Senior Drupal Architect Andrew Berry, “I try to keep snacks and drinks stored away from my office. Otherwise, it’s too easy to stay sitting at the desk all day: snacks are just a long reach away from your desk. Keeping them elsewhere means an extra minute or two, but also gives your eyes and body a rest.”

“I love going for a walk in the middle of the day,” says Matt Westgate, president and cofounder. “I usually reserve the hardcore gym exercise until the end of the day to release stress. My usual transition from work life to personal life is to prepare dinner while I’m having an end-of-day wrap up with Jeff or Seth. Cooking is a creative outlet for me, and it has the byproduct of being (usually) delicious!”

Get out

Leaving the house during a workday is often a good idea. A coffee run, a class at the gym, or some errands around town can help relieve cabin fever. If you’ve ever had the experience of working at an office, you know how relaxing a lunch away from the office can be. The same principle applies when you work at home! Take a step away, focus on something you need to do in your personal life, reboot, and return.

Jeff Robbins advises: “If you really want to bear down and work, get out of the house. Go to a cafe, get a big caffeinated drink, and put on headphones. The peripheral activity of the staff and other patrons energizes me. Because I don’t know anyone there, I stay isolated and can really focus on my work without distraction. Their chairs usually aren’t as comfortable as mine, so I don’t stay there all day, just a couple of hours. Then I can head home and appreciate my comfortable chair that much more!”

Use being at home to your advantage

Just because you’re working doesn’t mean you have to forego the comforts of home. Do the things you can’t do at an office -- put on some music, work from the patio, or open the windows!

Jared Ponchot, Creative Director, has his afternoon relaxation time down. “I try to schedule some sort of relaxing activity for the afternoon. I typically hit a wall between 2pm and 4pm, so I try to listen to that and either take a walk, go play with my kids, make a cup of tea, or do something for at least 15-30 minutes that’s completely relaxing. That moves all my brain power from my prefrontal cortex back into a more balanced right/left brain, I think.”

Having multiple places to work in your home is another advantage over a typical office. Jerad Bitner, Senior Technical Project manager, mixes it up between a traditional desk, treadmill desk, a lazy boy, and a standing desk. “Sometimes I like the couch,” he says. “It’s a great way to help prevent repetitive stress injuries, which often result from working long hours in the same position.”

Shower

It surprised me just how many Lullabots mentioned the shower when I was researching this article! From Nate Haug, Senior Drupal Architect, “In the midst of all the phone calls, naps, and lunch, I usually take at least one shower in the middle or end of the day. To me the shower is the absolute most productive place in my entire ‘office.’ There’s no problem so difficult that it can’t be solved with a good shower.”

Jared Ponchot also had an opinion on the shower. “I fully relate to the idea that being showered and dressed before work can help orient one’s self. However, I’ve actually changed my routine, intentionally waiting to shower until late morning or midday. I’ve found that it’s a powerful tool for ‘creative pause’ and cranking alpha waves. I want it to disrupt my ultra focus, a zone I get into when I’m working on for more than a few hours. There are so many times when I scramble to get out of the shower because I’ve suddenly understood something or had an important idea during that break.”

Work. Rinse. Repeat.

Be interactive (in work and other aspects of your life)

“Connecting with people face-to-face during the day is really important” says Matt Westgate, and many other Bots agree. “That connection could be family, a local group, or going to the coffee shop or the gym. I find those face-to-face connections remind me that I’m also making human connections when I’m on the phone or writing email. It helps me better empathize.”

Being interactive during work could mean contacting a client or colleague using videochat via Skype, Google Hangout or GotoMeeting rather than shooting off an email. Lullabot uses several means of communication for our team, but nothing feels quite like face-to face time.

Outside of work, “it’s important to talk about non-work things, to make up for the social interaction you’re losing by staying at home,” cautions Karen Stevenson. It would be incredibly easy to turn into a hermit. Many Lullabots recommend becoming involved in a community or interest group outside of home.

“The biggest thing that helps me keep the balance is being in a group.” Angus Mak continues, “I train dogs a few times a week at a dog club, and that forces me to leave the house at 6 p.m. Having something outside of work that I am passionate about really helps.”

“If you’re a designer working from home, you need to join some sort of group in your local area and force yourself (against your will if you’re wired like me!) to go hang out at their events,” says Jared Ponchot. “I’m a part of the Atlanta Web Designer group. I’ve had to miss it a few times and keenly feel the loss.”

Speaking of family, it can be hard to work at home with family present, as they are (and should be) Priority Number One. Finding the work/life balance in regards to family is something that I’ve had to work hard at. It’s helped to get to a point where I can say firmly, “Right now I am working!” and stick to it. After work, if I’m as diligent giving my family attention as I am at giving work attention, we all feel good.

This is also an important issue for Seth Brown who has three young daughters. “For me,” he says, “the act of shutting my door is symbolic. It says, I’m at work, don’t bother me. But if a child falls off a bunk bed or there’s some other emergency, like sick kids at home, you have to step out of work mode. I feel like it’s a blessing and a curse. It’s a blessing to be there for your family when they really need it, but sometimes it can come up at the most inopportune times. It's stressful trying to listen and participate in a conference call while trying to calm a crying child.”

Work at NOT working

Working from home with a distributed team comes with a few challenges that wouldn’t exist in an office. Different time zones, different work schedules, and spotty Internet connections can all work against you. The biggest issue seems to be the same wherever we are: the tendency to overwork. How can you leave work at work when you work from home?

Blake Hall, Senior Developer: “For me it’s always key to remember (and sometimes force myself) to carve out time where I have no Internet access. I inevitably get sucked into something interesting (even just Yammer) if I don’t.”

“It’s easy to get me to read an e-mail or message,” explains Nate Haug, “but I’ll only respond when I’m at my desk. The variability of my days means that I always schedule phone calls and meetings in advance, preferably not the same day. I usually don’t accept direct phone calls and I use conference lines or Skype as much as possible to prevent clients or even co-workers from using my phone number and breaking that scheduled time.”

Now, for the summary

The end all, as they say, is simple: don’t be afraid to try new things, even if they seem odd. Don’t be afraid to explore, find the schedule and best practices that work for you, and work that way. There’s no “right” or “wrong” way to work from home, if it works for you.

I think James Sansbury, our Development Manager, said it best: “It’s working from home, not homing from work, so the point everyone’s making is to create a clear line for what work is and what home is. If that means you get dressed before going to work, get dressed. If that means setting specific work hours, set those hours. If that means taking a long lunch, take a long lunch. The easy part is determining those things; the hard part is actually doing it. DO IT. DO IT!”

Dec 01 2012
Dec 01

Yesterday I took part in a Lullabot podcast on Project Management. The Bots are well known for their excellent courses and dialogue - it is my hope that we don't disappoint. Addison Berry of Lullabot did a great job of herding the herders keeping us on track. So, who participated?

Seth Brown - Lullabot's Director of Operations, Jerad Bitner - a Lullabot Technical Project Manager, Drew Harteveld - Martha Stewart's VP of Digital Operations, and Me. The conversation showed a large degree of consensus on methodologies, tactics, and tools.

The process of getting the podcast set up was pretty cool.

  • Addi approached me at BADCamp, asking if I'd be willing to participate. I'm always game for contributing where I can and I was also humbled to be asked.
  • We started an email chain that brainstormed our topic.
  • The top items were added to a google doc where we were all encouraged to comment
  • We chose a few topics we wanted to explore, Addi added another, and finally there were questions that came from social-media-verse
  • 15 minutes before the podcast we did some sound checks and talked amongst ourselves. Addi explained how it was going to go down.
  • When we started, it was pretty free form - we didn't really know what questions would be asked when. I would say a few minutes into the recording, we were basically just having a friendly conversation.

This was thoroughly a good time. I hope I'm asked to participate again in the future.

  • Free Tagging:
Nov 30 2012
Nov 30

Listen online: 

In this episode Addison Berry is joined by four project managers to explain what project management is all about, and the impact of working with Drupal projects. Join Seth Brown, Lullabot Director of Operations (sethlbrown), Jerad Bitner, Lullabot Sr. Technical Project Manager (sirkitree), Matthew Saunders, COO of 5 Rings Web and a Principal of Vintage Digital (MatthewS), and Drew Harteveld, VP of Digital Operations at Martha Stewart Living Omnimedia, as they give some insight into their crystal ball secrets.

Project Management

If you want to suggest ideas for podcasts, or have questions for us to answer on a podcast, let us know:
Twitter
Facebook
Contact us page

Release Date: November 30, 2012 - 10:00am

Album:

Length: 60:55 minutes (34.86 MB)

Format: stereo 44kHz 80Kbps (cbr)

Nov 30 2012
Nov 30

Listen online: 

In this episode Addison Berry is joined by four project managers to explain what project management is all about, and the impact of working with Drupal projects. Join Seth Brown, Lullabot Director of Operations (sethlbrown), Jerad Bitner, Lullabot Sr. Technical Project Manager (sirkitree), Matthew Saunders, COO of 5 Rings Web and a Principal of Vintage Digital (MatthewS), and Drew Harteveld, VP of Digital Operations at Martha Stewart Living Omnimedia, as they give some insight into their crystal ball secrets.

Project Management

If you want to suggest ideas for podcasts, or have questions for us to answer on a podcast, let us know:
Twitter
Facebook
Contact us page

Release Date: November 30, 2012 - 10:00am

Album:

Length: 60:55 minutes (36.81 MB)

Format: mono 44kHz 84Kbps (vbr)

Nov 23 2012
Nov 23

Listen online: 

  • insert-content-here-05.mp3

Jeff Eaton, Marc Danziger, and Andrew Berry discuss NBC's Juno Project, a new take on the corporate knowledge base that combines sales and project management tools with long-term organizational history. Topics covered include workflow challenges, the importance of iterative prototyping when dealing with complex content, solving "findability" problems, and more.

Release Date: November 23, 2012 - 10:00am

Album:

Length: 45:28 minutes (25.64 MB)

Format: mono 44kHz 78Kbps (vbr)

Oct 31 2012
Oct 31

If you haven't read it already, there is a lot of really good meat in the first article on this topic by Seth Brown. Seth does an excellent job of digesting our process of working with our clients during the decomposition of a project to figure out just how much this baby is gonna cost.

Estimating

One thing that has evolved since the writing of Seth's article, is the spreadsheet that we use for our work breakdown, and the exact method in which we go about estimating the various deliverables. The spreadsheet is full of notes on how we go about our various estimates, and the things what we calculate automatically.

new-spreadsheet-img

First we have at least three developers who are going to be working on the project conduct a blind estimation line by line with white text on a white background and then do a reveal all at once. During the reveal, if we find that there ends up being large discrepancies in the estimates, we talk about the assumptions each developer made in order to make sure we're all on the same page with what is expected for said deliverable and give them a chance to adjust their estimates given any new information. We then average these as the total estimated developer hours.

There is also an added "pm factor" for our project manager which is a .25 multiplier. This multiplier is based on Lullabot's historic data for our projects where we've found that project managers end up spending about one quarter of the time developers spend on a project overall.

Our updated spreadsheet also introduces a concept of a risk or uncertainty factor which is helpful in calling out the unknown aspects of a project. The higher the risk, the more unknown that element of the project is. Measuring risk also helps measure the difference between a fixed-bid VS a time and materials approach to the project. Risk is measured on a scale from 0-5 and we calculate additional developer hours by taking the sum of the developer average plus the pm factor and multiply by the risk rating. Finally we multiply again by point one (=sum(Average+PM Factor)URating.1). We're basically asking our developers how uncertain they are about their estimates and why. It's usually because of some unknown factors, and we try to determine if these factors are something that the client can help us aleviate. If so we go back to make sure that expectations are clearer and give our developers the opportunity to reduce the risk and adjust their estimates. But sometimes it's a factor that is simply not controllable and if the risk remains high it increases the overall estimated hours.

Resourcing

We've also added a new sheet which helps us to figure out what resources we need on the project in order to accomplish the estimated deliverables and to help us to align our resources with the timeline of the project and the estimated amount of work. This helps our project mangers to get an idea of which deliverables they should schedule for each sprint. For instance, if they know that they have 80 man hours during Sprint 1, they also know that they can schedule X amount of deliverables for that first sprint based on how many hours were estimated.

resource-img

This also gives us a chance to think about dates going forward and account for any scheduled vacation times for the developers who are on the project and holidays can also be factored into the timeline. In the case of a fixed bid project, we can either add resources or if we have no more available resources, advise the client that we'll need to cut scope in order to meet their budget.

We continue to revise our approach to this process as we find better ways of trying to come up with more accurate estimates in order to meet our deadlines, budgets and to set client expectations appropriately. So far our latest version feels like the most accurate we've gotten to date, but we'd love to hear what you think about it! How has project estimation changed for you?

Jun 20 2012
Jun 20

We've tried a lot of project management systems over the years. In some way, they have always seemed lacking, confusing or just a pain in the rear end. If they had good tools for project managers, they were confusing to developers. If they were useful for developers, designers complained about the eye-sores. No one system ever seemed to satisfy the team.

We recently started using GitHub for project management after the developers started raving about how much they loved it for managing code. To our surprise, GitHub has proven a solid option for project management. Our designers have even started using it for some of their projects, which I think says something about GitHub's aesthetics. With a little bit of something for each role, GitHub is starting to come out on top as the tool of choice for hosting code, managing projects, and facilitating project communication.

Project Introductions

GitHub is pretty developer-centric. As such, the first thing a developer sees when they open a project, is a view of the code repository. Below that, GitHub automatically renders the README file found in the root of the code base. It's a very typical practice for software projects, especially open source software projects, to have this file in place. The README can be in various formats, but a favorite of mine is Markdown. Simply giving the README an extension of .md tells GitHub to render your README.md using the Markdown syntax. Even better, GitHub has it's own flavor of markdown. Since the developers of your project see the README first, this is a great place for information that will get them up and running with the project as quickly as possible. Be concise. If you need to write more than a few sentences, chances are, you should be linking off to more in-depth documentation in your project's wiki. Here's a quick guideline of some of the things that you might want to include in your README.

  1. A quick project overview.

    Provide a short description of the project's goals and a bit of background. Any links that you frequently access are also good to include up at the top as well, for easy access. Everyone loves easy access.

  2. Information about the directory structure.

    Typically we have more than just Drupal in our repository root, so it's helpful to have a brief description of what is in there. We typically have a drush folder for aliases and commands, as well as a patches directory with its own README.

  3. How to get started developing.

    Tell the developers what the best way to jump into the project might be. Things like, "clone this repository, create a feature branch, run the installer, download a copy of the database, etc.. Whomever reviews the pull request should also do things like remove the remote branch from the repository once it is merged."

  4. Code promotion workflow.

    It's a good idea to outline your development process, as it may change from project to project. Do you want people to fork your repository and send pull requests; create feature branches and send pull requests; or just go ahead and commit to master? Let the developers know up-front, so there's no confusion.

  5. Environments.

    Outline information for your dev, staging and live environments, if you have them. Also, outline the process for getting things to the various places. How do I make sure my code is on staging? What is the best way to grab a database dump? We like to setup drush aliases for each environment ahead of time as a means of outlining this information and giving developers a good starting point. This document contains some example commands for doing some typical operations. Here's an example.

  6. Links to where to find more information.

    Typically this is our wiki, where we keep more detailed documentation and notes on things; project details like the original proposal's SOW, credentials to environments, Scrum Notes, Pre-launch checklists, etc.

We've attempted to create a drupal-boilerplate, of sorts, for our Drupal projects which we're continuously re-using for new projects and modifying when we find things that work better. Take a look, and if you find it useful, please feel free to use it! If you find anything missing, or have ideas on improving it, please fork it and send us a pull request!

Working with GitHub Issues

GitHub has a pretty simple issue management system for bug tracking, but it is flexible enough to be a pretty powerful tool for managing entire projects, large and small. It has issues which can reference each-other; labels for attaching meta data to your issues; methods for attaching code to your issues; and even milestones for grouping and focusing your issues within time blocks.

Referencing and Association

Issues can be associated with each other by simply throwing an #issue-number (ex: #3) within the body of another issue. This is useful in many ways. Firstly, it keeps the relationship simple. We don't have to worry about what kind of relationship it is (parent/sibling/child), just that it's related. Nevertheless, there are a couple of tricks that make this a little more useful if you understand how it works. Let me give you an example.

Let's say you typically create an issue for a content type, and one of the fields on that content type is a taxonomy vocabulary. You probably want to break that vocabulary creation out into it's own issue. So you create the issue for the news content type news-content-type

and then you create an issue for the taxonomy vocabulary and, within your description, link to the news issue. tags-taxonomy

Just by putting in the #ticket-number (in this case #4) GitHub creates a link to the news issue AND it places a back-link reference within the news issue to your tags issue! news-tags

As a part of this reference you will notice that it also gives you the status of the referenced issue. Very handy for whomever is assigned this news issue. They can easily see the status of it's 'dependency'. I use that term loosely because it is a dependency in this instance, but not always.

Issue Labels

Tags are a simple and effective way to add metadata to your issues. A lot of systems tend to create fields and categories with various values in an effort to allow you finite control of the metadata for an issue. I've found the simple tagging system that GitHub employs to be very efficient and more flexible.

GitHub comes with a few labels by default: bug, duplicate, enhancement, invalid, question, and won't fix. These give you a good idea of how to start using labels. For example, "bug" is a type of issue, while "won't fix" is more of a status. Tags can be anything, and if chosen wisely, can give any developer an immediate clue as to what sort of ticket it is, what section it might apply to, or what status it is in at a quick glance.

While they're useful for developers, they're also good for the organizer of the project in that they serve as a great filtering mechanism as well. For instance, just by selecting various labels, I can see all of the issues that are "migration" issues for "taxonomy", or "content types." content-migration

Attach Code to an Existing Issue

Pull requests are an amazing tool for code collaboration. If you're new to the concept, check out this pull request demo. It's a quick and easy way for developers to basically create a copy of the code base (by either forking or branching) and suggest modifications to the existing code, or contribute new code. It allows the other members of the project to then review that code, make their own suggestions with in-line commenting, and then make a decision as to whether to merge it into the main code base or not. We've found the in-line commenting with pull requests to be immensely useful since they keep everyone in the project in the loop with changes that are happening. code-comments

Pull requests in general are a great means of peer review and have helped to keep the quality of our code up to everyone's standards. There's a bit of overhead in that it may take a little longer for some new piece of code to be merged in, so plan accordingly. But this also means we find bugs sooner, typically before they're actually introduced into the master branch of the code.

I had one gripe with pull requests: when you create one through GitHub's web interface, it basically creates a new issue. Though you can certainly reference a particular issue within your pull request, it's still a separate issue. However, through a nice command-line tool called Hub, we've found there's a way to turn issues into pull requests! Very handy for keeping your discussions and code all in one place and not having to deal with multiple issues about the same thing.

Milestones

GitHub has a mechanism for milestones that is actually quite typical of many project systems these days. When you create a new milestone, it simply has a title, description, and a date choosing mechanism. milestone

You can have a nice overview during the time-boxed iteration that gives you the percentage complete. closed

We tend to only plan one sprint ahead, but there is a milestone created for each iteration up until the end of the project. open

We grab these tickets from the Backlog, which is essentially just any ticket that is not in a Sprint. backlog

Huboard

GitHub's issue tracking system lacks a mechanism for prioritizing your issues. You could probably come up with labels for High, Medium and Low priorities, but I tend to prefer an ordered list with the highest priority things on top.

Enter Huboard, which gives you a nice Kanban-style interface (similar to Trello) right on top of the GitHub api. You're looking at your GitHub issues, but with a different interface. The instructions for setting this up are quite sufficient, so I'll not re-iterate those, but I've found that it's quite easy and quick to setup on Heroku with very little maintenance overhead. With Huboard, we now have a means of seeing what the priority tasks are for the week and it gives developers an easy way to see what they should work on next. huboard

Logins

Some project management systems require a new login for every instance of the software. For instance, if you have two different clients using the same project management software you may have to remember two different username and password combinations and your authentication will not transfer from one to the other. Github allows users to access all the projects and repositories you have permission to without multiple authentication.

Github is lean and spare, and you may find there are features missing that you're accustomed to. Luckily, the team over at GitHub is continually making improvements to their product and they blog about it often.

In summary, GitHub is great for the technically-minded person, but less tech-savvy people may not find it as attractive. I'm still working on ways to report on progress to project stakeholders in a more visual way and when I find one I like, I plan to update you all.

If you have any suggestions on things we might also do to improve our process, or would like to share with us some of the exciting things you're doing with your own processes, please hit us up in the comments section! We'd love to hear from you! And remember, Lullabot loves you!

Fork me on GitHub */
Mar 28 2012
Mar 28

This past week at DrupalCon Denver, I attended a wonderful BoF on project management. The session started as a conversation about the complexities of Drupal project management. As we collected a bulleted list, one item in particular from our list stuck with me:

In Drupal development, fixed budget projects exist in an instantaneous sense, but in reality evolve constantly.

One can never account for every unknown on any given project, even when given an infinite amount of time. This idea captures the typical project process exceptionally well as Drupal (and its community) rapidly changes and improves, but we often stumble on the "two steps forward, one step back" phenomenon as a part of this. How can we account for these external risks and explain this phenomenon to our collaborators in the project process? Without firm understanding of this concept, the frustration is likely to mount for all parties.The illusion of moving forward by several paces only to jump back detracts (sometimes entirely) from the fact that progress has been made. This often leaves development teams discouraged and clients, stakeholders and collaborators neglected. As project managers, we're posed with bridging the gap. But how?

Because sometimes modules become abandoned, patches never get merged, and other dominant features crop up, it's best not to guarantee the use of any one particular module, installation profile, distribution or any other kind of feature, but rather an encompassing functionality. For example, instead of specifying the use of the ShareThis module, specify the functionality for users to share pages with members of their social networks. Not only does this get at the meat of the problem we're trying to solve, it explains it in language that our stakeholders will understand, and it won't lock the development team into a dead-end in terms of solutions architecture.

At this point, I can hear site builders and business developers alike saying to themselves, "You expect us to account for building this feature from scratch when there's already something out there that does what I want? That will multiply the size our estimate!" The answer is: not necessarily. Obviously, the Views module is not going to be abandoned any time soon (and if it somehow is, there better well be an jaw-droppingly awesome successor). It really depends on the strength and the state of the feature the developers wish to incorporate, and how cutting edge the stakeholder wants to be. The most cutting edge stakeholder will likely be understanding of the two-steps-one-step phenomenon, so it's a moot point. Those who aren't looking for the shiniest tool in the toolshed should be happy to settle with something that is tried and true. As for those who fall in between, consulting on a feature-by-feature basis will likely get you what you need to accomplish the project: "We could go wtih module X which is currently the standard, or we can try module Y which seems to be up and coming for such and such reasons, but may have risks involved as it's still in its early stages."

A graph of Everett Rogers Technology Adoption Lifecycle model. Drawn in OmniGraffle and then trimmed in Apple Preview. http://en.wikipedia.org/wiki/File:DiffusionOfInnovation.pngAnother great resource to determine where a particular feature is, and where it is going is by consulting the good old technology product cycle diagram (More about the product life cycle on Wikipedia). The development team should be able to place where the feature falls in the product cycle. If they can't, then it's probably not a safe bet. Anywhere at the tail end of a growth phase or in the maturity phase is ideal. This diagram will speak to many clients, stakeholders and collaborators in a way that they will understand. If they see that the module they're interested in is in a decline or an early growth phase, they should be able to understand the consequences with little explanation.

A cry I hear coming from the coding side of my psyche is: "Modules and other functionality can't be swapped out so simply-- they aren't interchangeable. What if the module we intended to use is one I'm familiar with, but the one we end up going with is completely foreign to me?" This begins to get into internal risk mitigation and recognizing the skill set of your team. Skipping over the complexities of that (enormous) topic, before selecting a module and getting gung-ho about it, give your developers time during the design phase to familiarize themselves with the options, and gain confidence in their ability to manipulate them to suit the needs of the project. If they're not confident in this before the design is solidified, they may as well be building the feature themselves. How this affects budget, especially with regard to an increase, will need to be discussed with the client, and the budget balanced accordingly.

In the end, by avoiding specifics and focusing on goals (while still emphasizing the feasibility of the target) in the sales process gives the design and development team the freedom to accomplish the goals in the best way possible in the moment. If uncertainties arise out of product life cycle changes, the path forward should be determined by the stakeholders, so long as they understand the consequences and the available options fit within budget (which could encompass an entire blog post on its own).

All in all, my mind is still churning the ideas exchanged during the brief 15 minutes in this one hour BoF in the three day conference and I can't wait to contribute again to the DrupalCon experience (next time, at home base!) in Portland in 2013. Looking forward to seeing you all next May!

Image: A graph of Everett Rogers Technology Adoption Lifecycle model. Drawn in OmniGraffle and then trimmed in Apple Preview. http://en.wikipedia.org/wiki/File:DiffusionOfInnovation.png

Jul 21 2011
Jul 21

When will Drupal 8 be released?

When will Drupal 8 be out? It's a question that's asked in many forms.

Sometimes it's phrased as: when will I have to upgrade from Drupal 6? Or, how long will Drupal 7 be supported?

Laura Scott and I were reviewing Dries Buytaert's 2010 State of Drupal and the estimates on when the new Drupal release candidate would be virtually frozen. The graph below is his summary graph of the trends in April 2010.

Dries Buytaert
Fig 1 — Dries Buytaert's April 2010 Prediction of Drupal Release.

It turned out that the trends were not linear, and it was Laura who suggested that bug fixes aren't linear: they tend to be hyperbolic or (for fellow math majors) follow "power functions" that have a logarithmic aspect to them.

This got us thinking. What is the history of time between Drupal releases (TBDR)? In the slide below, Dries provided some this data in his slide from the State of Drupal at the March 2011 Drupalcon Chicago.

Dries Buytaert
Fig 2 — Dries' March 2011 graphic showing Drupal's code size historically.

All that was missing were the dates, which Laura looked up from the Changelogs of each Drupal n.0 release, and from this I created the graph below, which is consistent with a power function. That is, geometric!

graph
Fig 3 — TBDR of core size versus release dates. Graph by Kate Lawrence [graph redrawn from same data, September 2012].

What this means is that the more the code, the longer it takes, and in this case (at least thus far) it is obeying a consistent pattern. Interestingly, in the period from of Drupal 4.0.0 to Drupal 4.6.0, the size of Drupal was pretty constant, especially as compared to the recent growth of Drupal core ... and the TBDR was about 6 months! What do today's longer release cycles mean for Drupal? I say this only because we are hearing that Joomla has a 6 month time horizon between releases.

If Developers were Horses...

What would it look like to have 90 developers contributing, only contributing really, really fast? Is there a marginal utility curve, like we learned in Econ 200, in there someplace? That is, at what point do you have so many people involved, that they are tripping over one another? Throwing more people at the problem does not get quicker results, and may in fact, slow things down. cf. The Mythical Man Month.

The Mythical Man Month premise is: Assigning more programmers to a project running behind schedule will make it even later.

So a question to ponder is to what extent can the growth curve in the above slide be changed?

The final slide from Dries that I think is relevant to this pattern appears below.

Dries Buytaert
Fig 4 — 30 contributors were responsible for 50% of D7 Patches. Graphic by Dries Buytaert.

Most of us have heard of the "80/20 Rule." The rule is attributed to the findings of Italian economist Vilfredo Pareto. Sometimes its the 90/10 Rule or 95/5 Rule, or even 920/30 rule (see Figure 4)!

The challenge may be in making the ski-jump curved line more linear (while pinning the line on the y-axis). The more linear it becomes, the steeper the TBDR curve's slope becomes — meaning that releases can be more rapid.

In short, holding the intersection point on the left axis, as the graph becomes more linear, the faster the TBDR. The challenge we face as individuals within the community is helping in making that happen.

Jul 11 2011
Jul 11

Everyone does virtual differently. Having worked in bricks-and-mortar previously, setting up a virtual work system as been something of a mix of experimentation with solutions recommended by others. Here are some of the ways we manage as a virtual company.

Client communications

Email is a place to lose things. We don't depend upon email for critical project communications. We have our own Open Atrium intranet set up for all of our clients. Here we post research documents, wireframes, flow charts, use cases, design mockups, questions, general messages, and the ever essential support tickets. Our clients can post, comment on and otherwise interact with all of these items, and all the archives are there for review later. No spam filters or non-inbox-zero obstacles!

Team communications

What can I say? We like Skype. It's stable, it's pretty much peer-to-peer, it is very good at drilling through firewalls, it's encrypted, it stores archives, we can do conference calls, share screens, etc. That said, we are eyeing other possible solutions, including google+. The main need is to have robust real-time chatrooms, with encryption, and easy voice communications.

Ticket/time tracking

Like most design and development companies, we've been pretty frustrated with a lot of the project management software solutions out there. We use Liquid Planner (affiliate link), which provides ticket-level time tracking, high/low estimation, schedule and cost projections, and some nice graphs. And unlike some other systems, we've had no problems with performance slowdowns — it's a fairly consistently moderately fast web app, given its complexity. Having tried Basecamp, Harvest, Unfuddle, Freshbooks, ActiveCollab, Trac, Jira, Rally and several desktop apps over the years, I think Liquid Planner is the closest thing to the bee's knees.

Design process

We meet with the client if at all possible. We talk. We draw pictures. We point at things. We work to understand the client as best we can. This means travel, if there's room in the budget. It ends up paying off in the end through implementation of a project better aligned with the client's needs. When not in person, we communicate through Atrium.

As for amongst ourselves, we use Dropbox (affiliate link for extra GBs of space) as an easy way to share physical files that are not suited for version control. And we talk in Skype. Sometimes we'll do mark-up on artwork or wireframes, but most of the creative collaboration happens via voice. It's somehow more human that way.

Code repository

Like many, we use GitHub. Can I just take a moment here to say how much I love Git? Escaping from Subversion was transformative. (Drupal.org's escaping from CVS will be even more so.) The biggest problem with svn is all those hidden folders. They're like pox riddled throughout your code, buried in each folder, lurking. Replace a folder with a fresh one and svn freaks out, "What did you do with my hidden folder, you malfeasant!" Git doesn't do this. Git is nice. Swap out things in your code and Git doesn't mind. Git loves it when you do things like that. Go ahead, swap out a folder, swap out 10 folders! Tell Git and Git says, "Got it!" Shiny!

Development and staging

Configure globally, code locally. As soon as we start the first development sprint for a project, we set up a staging server on a VPS. Right now we are really liking Linode (affiliate link) for the smaller sites and AWS for the bigger ones requiring more flexibility on resources. Each of us codes locally, on our own machines, merging our code changes with Git. We deploy committed changes several times a day to the staging server using Git pulls from GitHub, which serves as our canonical central repository. Configuration changes, with the exception of Featurized configurations, are all done on the staging site, so that we don't have to mess with database conflicts. (Bricks-and-mortar teams can all connect to one canonical database on the same network, but that just won't work for virtual teams. The latency is not just fatal, it's fatal on a mass extinction scale.) We periodically pull database backups from the staging site to use locally, so we're all referencing the same basic configurations.

The exceptions to this relate to Features and exportables. We love Features, but find that they can get in the way at times. We use them for special cases, especially when doing iterative development, where it's nice to be able to deploy code and configuration changes through version control. On the other hand, exportables (e.g., Views and Contexts) are nice ways of moving sometimes extensive configuration work from one machine to another.

Sales pipeline

Here I'm afraid we're amateurs. We have our inquiry form. We have phones. We have email. We do things the old fashioned way. That means that, no doubt, our sales process is probably woefully inefficient at times. We're not huge, not even big enough to have a dedicated sales person, so a robust CRM really does not seem to make sense — in fact, it would probably just make everything harder to manage, spending all our time trying to get the CRM to record things in ways that are findable. If you fill out our inquiry form, or call us, or email us, it's we at the other end who are fielding your inquiry, no salespeople or account managers. Consider it the personal touch.

Drupal community communications

Anyone in the community can tell you, the ways to communicate are: on the *.drupal.org websites, in IRC, and on the email lists.

And at Drupal events!

 Blue Angels in formation
Smooth coordination can be a challenge for any team. The Blue Angels set the standard. Photo by Joshua Davis Photography (Creative Commons).

The common thread through all of this is communication. It's not everything, but it's the most important thing.

Syndicated on BlogHer.com
Jun 30 2011
Jun 30

Too busy doing the doing to talk about the doing?

As a strategy and design studio, we're all over communications. It's an uninterrupted conversation with our clients, hashing out ideas, working iterations of goals definition, research, wireframes, designs. But when it comes to development, when we put our heads down and get to work, it's all too easy for us in our focused implementation state to just do it and not communicate enough about what we're doing. Of course, this can be disconcerting to the client who has no idea how things are going. And that's a ripe situation for fear, uncertainty and doubt to creep into the relationship, and suddenly a successful project-in-progress can feel troubled, even if it's not.

The simple solution is to give regular status updates, which is something we've always done. Even just saying, "Things are going well" can help. But on long projects, or even in 3-week iterations, repeating that same message ends up being not that helpful. Better to share what you're thinking with regards to planning and project management, and provide updates on development of the various features/backlog items worked out in the strategy and design phases. This transparency in the project management process is even better when updates are more specific.

We have been using Open Atrium, the powerful intranet CMS built in Drupal, developed initially by our friends at Development Seed, now maintained by Phase2 Technology, for our intranet to handle all client-facing communications, documentation, file sharing and issue tracking. It's been a great tool for this. No missed emails lost in the spam filter. No critical documentation lost on someone's hard drive. No mega-emails containing two dozen specific items. (Well, those still happen on occasion, but we now can push those into the intranet to continue discussion.) We love it.

And now we love it even more.

Rethinking Open Atrium

We had been using almost all of Open Atrium's various features: messages, uploads, case tracker, books for documentation, and of course the groups functionality. Almost all of the features. The part that didn't really fit in with our communications strategy was the OA notion of "Projects". Until now.

You see, in Open Atrium, the basic architecture is thus: You have Clients, which are defined by groups, and you have Projects, which are ways of grouping tickets within the group by project. It can make sense for a large organization using OA for internal communications to manage several projects. But for a professional services firm like a strategy and design company, or a software company, or any other kind of company that defines work by time-boxed iterations or sprints, breaking up stuff by "project" may not be that useful. Projects projects projects blah! At PINGV, we rarely have a series of small projects with a client where it would make sense keeping them all together in one group; we tend to do bigger projects, projects that have phases of strategy, design and planning, and iterations of development. And when we have a new large-ish project with an existing client, we just spin up a new Atrium group. Project was that dunsel feature — the thing that for us had no real use. But—

What if we use "Project" nodes to define phases and iterations?

Wash that nomenclature right out of OA

Ignoring nomenclature, and just looking at what "Project" as a functionality in Open Atrium does for us, it provides a way to group tickets into clumps that you can define by anything. If we define those clumps as iterations, perhaps, or phases, we can organize and parse through backlog items, features, questions and other tickets by iteration, while providing some ready clarity as to how and when these items will be taken on. (NB: It's an approach that works only with Tickets, not other node types, but in our mind that's a minor limitation.)

What's really convenient is that Open Atrium provides via UI the means to rename "Project" to whatever you prefer to call it (e.g., "Iteration"). These string replacements, configured in the group feature settings available to site administrators from within each group's space, work almost everywhere in OA. (The "Add Project" button seems to be unaffected even by string replacements defined in settings.php. Something to investigate later.)

Once we saw this potential for grouping issues into iterations, we ran with it.

Defining iterations in Open Atrium

Open Atrium screenshotUsing "projects" in Atrium, we can define iterations by close date, and assign tickets to iterations.

As you can see, the Iteration title is compressed down to an abbreviation or acronym in the icon in the tickets display. "Iteration 1" becomes "I1", "General" becomes "GENERA", and so on. After briefly considering using iteration numbers, which we rejected as too arbitrary, we opted to name iterations by the Iteration release date, or end date (e.g., 08/17 or even 11/10/23), which can be abbreviated in a meaningful way in the Tickets views. This gives our clients added insight into our timetable planning at a glance.

Adding a sortable Iteration column to the tickets view involves a simple override to the default ticket View. (If ever we need to revert that Feature during a site update, these changes are easily re-implemented in a few minutes. No actual data is affected.)

Open Atrium screenshotYou can view all of the iterations planned, and see their descriptions. Each iteration is clickable for detailed viewing.

This approach allows us also to focus just on the tickets that are relevant now, without having to sort through all the other tickets that are further down in the timetable, and without having to mark dozens (hundreds?) of individual tickets as "deferred" and then change those as they come available.

Open Atrium screenshotSingle iterations of backlog items can be viewed as well, and this is stock functionality for Open Atrium.

The bulk of communications still happens in the individual tickets. Questions, information, updates, concerns, notes can be tied to the ticket itself. New questions and bug reports can also be slotted for attention in a scheduled Iteration. So even though we're grouping our backlog items together, we still avoid the one-ticket-for-8-issues kind of communications that can lead to confusion, frustration and missed messages.

Open Atrium screenot
On any comment update, you can change the iteration assignment of a ticket.

comment status change
All changes are tracked in the comment metadata displayed right there in the thread.

We like this system so far because of this fact that all changes are tracked and visible in each ticket's thread. If we move up a backlog item to a higher priority, we simply do it with a comment and change in iteration designation. This flexibility with traceability provides a nice balance for communicating our planning.

And it's all using stock Open Atrium!

Plan before communicating, and vice versa

I should add that this is not how we do our actual project management. Open Atrium is a great tool for project-related communications. We currently use Brian's recommendation, we tried Liquid Planner (disclosure: affiliate link; go to liquid planner dot com to skip it if you like) for our internal planning. Liquid Planner has an API, and we're considering developing an integration with Open Atrium to allow us to push and pull info between the two. Might be a nice convenience. We do the translation manually now, and while it can take up time on occasion, it also provides an occasion to rethink and re-evaluate, so the manual process does have value.

Feature think, Feature do

Now for the road not taken:

When first considering the challenge of better communicating our iterations plans, I imagined up a very simple Features-based solution using nodes and node reference, which resulted in Open Atrium Iterations. This module works as a drop-in extension to Open Atrium. Not touching any existing Atrium Features, OA Iterations defines a new content type with reference fields to call up tickets (cases), messages (blog posts) and documentation (books) that collectively define and describe an iteration of work. A dedicated view restricts the available referenced nodes to the current group's content.

On Friday afternoon, this seemed like a fine way to communicate our iteration planning. Tickets, as well as blog posts and book nodes, could live on their own, with their own revisions and threads, while the Iteration node would group them into an iteration. Done. Time to relax and prepare for a weekend cleaning the garage.

On Monday morning, however, when I sat down to start defining the iterations for one of our projects, I immediately questioned my own game plan. Open Atrium Iterations was an easy way to define iterations, but making changes would be difficult. What if I wanted to move a ticket from one iteration to another? Yes, this project was never intended for iteration planning, just iteration communication, but even just for communications this approach would mean editing at least two Iteration nodes just to "move" one backlog item. What's more, without implementing Reverse Node Reference and adding that block to one or more Contexts (and thus overriding stock Open Atrium Features), there was no way to look at a backlog item and see what iteration it's slotted for. The simple solution was not turning out to be so simple after all.

And that's when the flicker of the idea described above came to mind. In hindsight, I can smack myself for not thinking of this before. I've been working with Atrium for two years. But I think I got stuck on nomenclature.

Contrib status

Meanwhile, despite our not using it as originally planned, Open Atrium Iterations will remain live on Drupal.org to see if it proves helpful to others. If the Feature can be improved upon to make it more broadly useful to serve more use cases, I look forward to suggestions in the issue queue.

May 12 2011
May 12

Any time a given Drupal core development cycle is more than six months old, people start asking "When will Drupal X be released?" The answer, of course, is always "When it's ready. Please help us finish it." That's well and good, and I don't propose that we change that position. Volunteer labor and fixed release calendars do not mesh well.

However, knowing when Drupal will come out, at least in a vague sense, is increasingly important for core developers. The web is changing rapidly, even more rapidly than it has in the past, and knowing what the web will look like around when Drupal 8 is released is critical. Not only so that we can target the right features to get ahead of the curve, but so that we even know what our available tools are.

For Drupal 7, we had a rare opportunity. Thanks to the GoPHP5 project (spearheaded by a bunch of Drupal people, remember), we knew long before Drupal 6 was even completed that Drupal 7 would require PHP 5.2. That meant that even before Drupal 6 was in the can, we could start planning for what tools we'd have available for Drupal 7: Modern Date and Time handling, actually useful OO, PDO, and so forth. All of those were critical for making Drupal 7 the release that it is.

Even then, we were well behind the curve. PHP 5 had been out for years already and Drupal was somewhat late to the game. For Drupal 8, it's a bit more complicated. We need to predict what tools will be in widespread use when Drupal comes out, so that we can target those. That requires having at least a vague idea of how long the development cycle will be.

For example...

HTML5

Just how much should Drupal 8 embrace HTML5? HTML5 is certainly the way the web is moving, but as always older browsers lag behind and just won't die. There are ways around that, though.

If Drupal 8 is going to come out within a year, then a conservative approach is warranted. While all modern browsers now support HTML5 to a decent degree, I don't know that a majority of people can be said to be using modern browsers. (IE9's requirement of very recent Windows versions doesn'thelp here.) In a year, I don't think we'll have killed off IE 7 yet. That means we need to target a hybrid environment and assume that half our user base will not have support for HTML5, CSS 3, and so forth.

If, on the other hand, Drupal 8 will be out toward the end of 2012 or later, then a conservative approach is suicide. While HTML5 support may not be all that ubiquitous now, it's not at all unreasonable to expect it to be supported by a majority of browser installs by the end of next year, and it absolutely will be during Drupal 8's lifetime. In that case, Drupal 8 should be built HTML5 from the ground up, without question. The non-HTML5 case would be the exception, and that's easily handled via well-known techniques.

Do we really want to find ourselves as the only major non-HTML5 platform in 3 years? Or would we rather be the leading HTML5 platform in 2 years? That's a decision we have to make now, so we can start building that support into Drupal 8 today. But it depends on what we expect the market to look like when Drupal 8 actually ships.

PHP 5.3

The web hosting world is, as always, slow on the uptake. PHP 5.3 has been out for nearly two years, and the PHP internals team is talking about the release schedule for PHP 5.4. PHP 5.2 was officially retired late last year. Every major Linux distribution bar none now ships PHP 5.3 out of the box. (Yes, even Red Hat.) Shared web hosts, of course, are still running the deprecated PHP 5.2.

At least, they are for now. If Drupal 8 comes out within the next 12 months, it's unlikely that the majority of web hosts will have migrated to PHP 5.3. By the end of 2012, though, a fair number could be. An even longer dev cycle? PHP 5.2 will be dead and buried in 3 years.

This is a question that directly affects everything we do for the next however-long. Are we going to require PHP 5.3? If so, then we should be talking about how to leverage namespaces. If not, then that entire thread should be marked postponed and we shouldn't waste time on it. But given how much code would be different in a namespaced world, that's something we need to know now, not in two years when we're getting ready to release.

Or what of anonymous functions and closures? Can we use those in core? Do we have to build support for using them in various ways? If Drupal 8 will be out by the end of this year, definitely not. By the end of 2012, probably yes. If the end of 2013, absolutely yes, and we should be thinking about PHP 5.4, too. But that's something for us to decide now, because that's years worth of development that will be different either way.

Another advantage to knowing far in advance: We can give web hosts fair warning. With GoPHP5, we pushed the entire web hosting world to make a change for the better by drawing a line in the sand, giving ample time, and sticking to it. For PHP 5.3, we can do the same. If web hosts know that the CMS that powers over 2% of the web is going to require PHP 5.3, they'll sit up and take notice, and the good ones will take steps to be ready for us. But only if we give them enough advanced warning.

Shall we be conservative and cautious, or shall we again be a leader in the PHP world? That requires careful timing.

jQuery

We have almost no control over what browsers will be available when. We have only indirect pressure to exert over what versions of PHP will be in use when Drupal 8 is released. But we can have a major impact on our sister project, jQuery, if we know how to plan ahead.

Drupal has a lot of custom Javascript, much of it written before jQuery implemented certain features. Of course, jQuery now implements many of those and there's a push to drop our custom implementations in favor of what jQuery and others have implemented in the meantime. That's all well and good, but wouldn't it be better to not have to write our own versions in the first place?

jQuery is a very fast moving project, and like Drupal it allows for API breakage between versions (although nowhere near to the extent we do). It is difficult to predict what jQuery or jQuery UI will be doing in a year or two... but as with Drupal, there is an easy way to find out: get involved.

If Drupal 8 is coming out by the end of 2011, we probably can't rely on being able to get new functionality into jQuery core or jQuery UI. We need to work with what we have now, build on top of it, and hope it doesn't break. That's what we've done for the past five years. But what if we knew we had time to work with jQuery core directly?

Instead of living in our own little world, plan ahead. Work with the jQuery folks directly build, say, something akin to Drupal.behaviors into jQuery itself. Get it reviewed by Javascript experts who are five times as skilled as any of we Drupalers are, get it good, and get it into jQuery itself. Then we can leverage it in Drupal with less custom code for us to maintain, plus we get the good karma of working directly with fellow open source projects. Everybody wins.

We can only do that, though, if we know how much time we have, and if we can take the time to wait for the next release of jQuery before we start building on the new stuff we worked with them on.

It's all about planning

If we don't know how much time we have, then every issue is in emergency "OMG gotta get this right!" mode because we don't know if it will be our last. That's not a healthy way to develop.

No, we don't need a fixed release date. We don't need any dates. But we do need a sense of how long we want to give ourselves, as that will directly dictate what we do, years before Drupal 8 ships. That provides a guideline for how ambitious we can and should be.

And there's a lot of ambition in the Drupal 8 world right now. :-) How long do we have, Drupal?

Bookmark/Search this post with

Mar 13 2011
Mar 13

At DrupalCon Chicago this past week, there was a "Core Conversations" session track, made up of sessions pitched by contributors to the core Drupal project. A wide range of topics were covered from the Butler project (a new system for context and blocks), to the built-in Help system, to Deployment strategies, to redesigning the issue queue. These sessions were shorter presentations followed by a discussion period for the attendees to give input on the topics.

The final conversation on Thursday by Dries Buytaert (Drupal's project lead) focused on discussing the development process for Drupal 8. (It was also exciting to witness the Drupal 8 branch being created live during the session!) During the presentation portion, Dries described in more detail the process changes he had suggested during his keynote. He then opened the floor up for everyone to bring up issues they felt needed some attention/discussion.

Discussion points

Here is the list of discussion points (that Dries noted during the talk) we core enthusiasts came up with:

  • More structured sprints - project management
  • Sandboxes (aka. Samboxes, since Sam Boyer was a huge contributor to Drupal's CVS to git migration) and locations
  • Timeline of release cycle
  • Hybrid development cycle
  • Ubuntu model?
  • Gate details; performance testing, etc.
  • Roles of initiative owners
  • Backporting
  • Non-initiatives / small changes / bugfixes -- different process?
  • Tools for usability/accessibility
  • Process around new initiatives / proposal
  • Documentation gate - workflows

Many, but not all points were discussed, and as we progressed through the conversation, I began to see parallels between some of the process changes we've implemented here at Affinity Bridge and what's going on in the Drupal development process. When I first began my position as Project Manager here, the first task I was assigned was to figure out how to make our development process work better, especially for larger, ongoing projects. This was sparked by the pain of working on very large projects, with huge issue backlogs and many feature requests, and no set launch date. 

After a lot of reading and research, our team started experimenting with a more agile process. Picking smaller chunks of work, completely finishing them, picking a new chunk of work, completely finishing it, etc. helps to make progress faster and at a better quality. Many times now, the benefits of this process have proven themselves, and I am beginning to see how some of the main points we discussed at the core conversation could be addressed with some incremental changes towards a more agile development process. 

A sidenote, as I was discussing this over the weekend, Bruno De Bondt from DeWereldMorgen.be pointed me at a blog post from one of his old coworkers from Krimson (a development shop in Belgium), Kristof De Jaeger (aka. swentel). The post is from about a month and a half ago, and outlines a proposal for an agile Drupal 8 development process. He goes into much more detail about the process and different roles involved than I will here, but it is well aligned with my thoughts (and potential long term vision), so I recommend you read it!

Specific ideas about sprints and agile for core

Before I delve into details here, I would like to preface this by saying I am not a religiously devoted agile follower. So far, what I've seen work is using elements of the agile methodology that fit for the team and project. Some devoted capital-A "Agile" followers might find that to be a flaw, but I don't think it's necessary to shoehorn people into a process. It's important that anyone adopting this feel comfortable with it, and not forced into it. Also, speaking briefly with Dries after the core conversation, he noted that it's important with such a large community not to try and undertake too much change at once (referring to the git migration and also the new idea of core "initiatives"). I very much believe those are wise words, and so am more keen to suggest some practices that the community can experiment with. If these are successful, then perhaps they can be applied more overarchingly.

Now, to go through a few of the main discussion points and others that came up, and relate them to how a more sprint-based/agile process might help address them.

New opportunites from sandboxes and git

The move to git is obviously going to open up many doors as far as more collaborative work and changes to workflows. Any Drupal.org user can now have their own development sandbox, and as Dries mentioned anyone can now fork core (ie. make a duplicate version and add improvements to it), though it's not always recommended! 

As many others noted in the session, this is going to allow multiple people to easily work on different parts of core and then merge their work back together. It could also allow multiple people working on different approaches to the same parts of core, to then compare and combine the best pieces. In any case, it will allow for a lot more safe experimentation and collaboration.

Keeping the criticals count below 15

This was something Dries brought up in both his keynote and the session. The Drupal 7 release cycle was very drawn-out because of the large quantity of criticals accrued through the three years of work. Longer release cycles tend to lead to burn-out and work that doesn't get fully completed.

Having the sandboxes and git will allow for a totally new approach here, one which is common to a more "agile" process, which prioritizes keeping the master branch clean and ready to launch at any time. To work like this, the master branch for core would be kept in a launchable state, and any changes or new work would be done in branches/sandboxes which would have to be completely done to be merged back into core. The "gates" would protect the master branch from bugs or incomplete work, so that we could be relatively certain that core is always ready to ship.

As a result, we wouldn't be aiming for under 15 criticals, but rather a constant of zero criticals in the master branch.

Small changes and bug fixes

Small changes and bug fixes would also be done in branches, but merged back in more frequently. There would be no necessity to save up batches of small fixes, though they could be done in batches related to different pieces of functionality. 

We would want to aim to avoid incurring technical debt, ie. fix problems as we go, as much as possible. Never delay working on bugs that will need to be fixed before launch. If that's the case, the work is not "done" and should not go into the master branch. This may mean more refactoring, but it helps keep the master branch ready to ship at all times.

Gates

The "gates" Dries talked about are still being defined, but the general idea is that there would be some steps that need to be completed before any branch would be merged into the master. Examples of what gates there might be include:

  • Documentation: making sure all of the code/API documentation is complete/updated, and ideally make sure the online documentation is also complete/updated. Possibly could go through review by Documentation Team members or other developers.
  • Accessibility: making sure that basic accessibility standards are met. Possibly could go through review by Accessibility Team.
  • Testing: making sure automated tests are written and pass. Possibly could go through manual testing as well.
  • Performance: making sure performance meets certain standards.
  • Design/UI: having design or usability reviews.

Making this concept of "gates" work will require defining a set of requirements and standards, and possibly finding people willing to do frequent reviews. But it will also mean that this additional work is really done before merging into the master branch, and that "done" refers not only to code being complete, but a more holistic interpretation of when work is ready to ship.

Timeline of release cycle

Dries suggested a shorter release cycle for Drupal 8, with likely a more focused and smaller set of initiatives (which he's outlined as areas of focus that will have initiative leaders). Keeping the master branch ready to ship at all times will mean that the release cycle can be as short or long as we want, and that we are not limited by half-finished functionality in the master branch (since only finished functionality would ever be merged in).

More structured sprints and phases

A lot of these goals and ideas lend extremely well to working within more structured sprints, and using phased development for larger initiatives. My initial suggestion (Kristof's post suggests two month sprints) would be for one month sprints. Despite the issue of working with a spread out team of mostly volunteers, I find that shorter sprints lead to better momentum. I also feel that it will be easier to pick out clear sets of issues to work on per sprint if they are relatively short.

Another benefit to shorter sprints would be the potential to attract more help from people who aren't interested/able to work on core more consistently. It's a lot easier for someone to commit to working on some functionality for a month rather than for three years. I would bet that once someone helped with one sprint, they would be far more likely to end up helping on another one down the road.

For larger pieces of work, we'd want to work in phases that are several sprints long. Each phase would have a set of functionality that can be merged with the master branch when complete. And the end of each phase would be a great place to work through the "gates". It would be nice to think we could go through the gates at the end of each sprint, but I don't think we currently have the resources to do this. It might take many phases to get an initiative fully complete, but the length of the initiative wouldn't necessarily delay a potential release.

A note on infrastructure...

Structured sprints would be much easier to do if we were able to add "sprint" and/or "phase" fields to the issues, but even without this we can always organize what is in each using tags, as was done with the git migration.

It would also be ideal to be able to relate issues in a richer way like in some issue tracking systems such as Unfuddle (see image below). Despite Unfuddle not being created specifically for agile development, it's easy to use it this way. I tend to create "parent" tickets for meta-issue/discussions and then create specific work item tickets as "children". You can also mark "related" issues, which aren't part of the batch of tickets, but are somehow related. And you can have parent tickets that have child tickets that are parents to others, so it's possible to create a hierarchy of what needs to be done to move onto another piece of work. Then, I create a "Milestone" for each sprint, with start and end dates, and add and prioritize tickets in the milestone appropriately after discussing priorities with the product owner and the development team.

If this interests you, there are issues filed for redesigning the issue queue and creating functionality to support meta-issues that you can add your opinions to.

Project management

This all seems to beg the question: does the Drupal project need a dedicated project manager (or scrum master)? 

At this point, I would say no. Between Dries (who tends to fill the product owner role), the core maintainer he appoints (who tends to act somewhat as a project manager, and does a lot of QA), and now the new Initiative leads, I don't feel that this would be necessary during a period of more experimental adoption.

If teams and individuals try this method, and find it works well enough that it could be adopted fully for all core development, it would be a very good idea. A much more structured process takes a lot of work to keep organized and stick with, and I believe it's best to keep developers' time free to be dedicated to development as much as possible. So at this point we would want to create some overall structure in the issue queues to be able to manage sprints efficiently, and have someone who can oversee the project's organization as a whole.

Realistically, I don't think this would be necessary very soon. For now, I would suggest anyone interested start with:

  • Running set sprints (recommending one month). Meeting at least once per sprint (on IRC or by voice) to review and align priorities amongst whoever is working on a given issue/set of issues/initiative. Communicating frequently on IRC (like we always do!).
  • Defining work to be done at the start of the sprint, and doing your best not to introduce new work into the in-progress sprint. In the beginning, underestimate how much can be done, until you get a feel for the momentum possible in a single sprint. If some things don't get finished, they go back into the "backlog" (or main queue of issues) and are reevaluated in regards to whether they will be in the subsequent sprint.
  • Making sure at the end of the sprint (or phase for larger initiatives), that you pass through the "gates" and deliver functionality that is completely "done" and ready to merge into the master branch.
  • Making sure that no bugs get into the master branch so it can always be ready to ship. As a result being able to make the release cycle any length, and launch on short notice if desired.
  • Avoiding technical debt (things that you put off that will need to be done later); making time for refactoring or bug-fixing as you go. Possibly even having a technical debt recovery sprint early on.
  • Defining the "gates" clearly and deciding who is responsible for signoff before merging into the master branch of core.
  • During this experimental phase, making sure there are some resources that teams and individuals can refer to and learn more about these methods. Possibly looking for project managers in the Drupal community who are familiar with agile development, and open to informal advising on IRC.
  • Remembering that this is a pilot project of sorts. Not putting too much pressure to follow capital-A Agile strictly. Letting this be an organic process to see what fits the community best, while leading to a more efficient, clean, and smooth process.

I was extremely happy to hear how open and interested the other attendees of Dries' core conversation were, and hope this helps clarify some ideas. This should be an open discussion rather than anything too prescriptive, so I would love to continue the conversation and hear your feedback, questions, and concerns below.

(ps. The Drupal 8 wordmark used with this post on the homepage is from Dries' slides, just for the record!)

Jul 14 2010
Jul 14

A few weeks ago, I and several others helped some friends of ours pack up their apartment into a truck in preparation for moving cross-country from Chicago to New York. It was, as such moments generally are, bitter sweet. It's always a good feeling to help out a friend, but when you're helping them get further away from you it's not as pleasant.

Of course, me being me, what struck me most about the whole process was how well it served as a model for software development and project management in general.

There were about a half-dozen of us all told, packing up stuff in a 2nd floor apartment, loading dollies, taking them down the elevator ($deity bless elevators...), out to the truck, and finally loading the truck. (Curiously, half the people involved were named "Dave". Insert joke here.) Throughout the few hours we spent working we evolved through various assembly line approaches, rotated positions, gave up and went with an "every man for himself" approach, took breaks to snack, and so on. If you've ever helped someone move then you know the drill.

Let's look at the whole process from a project management perspective, however.

The box model

The naive approach, of course, would say that if one person can carry a box down in 5 minutes and there's 100 boxes, then we just need 100 people and we'd be done in 5 minutes, or one person would be done in 500 minutes (a little over 8 hours). I certainly hope that everyone sees just how stupid that over-simplification is, but there are still software projects that try to run on that model: When in doubt, add manpower. Clearly they've never heard of Brooke's Law. (And if you haven't, you have no business managing a software project until you've read The Mythical Man Month. Go read it. Now.)

Why is that naive approach useless? Many reasons.

First and foremost, carrying boxes is only one part of the overall packing process. The boxes themselves first need to be packed and their contents organized in some sane fashion. The truck needs to be loaded. Boxes need to be grouped onto dollies in a way that won't fall over (and then of course does anyway).

There are also many constraints on the process besides manpower. We had only 3 dollies. We had only one elevator, and it could hold only so much at a time. It started to rain while we were packing the truck. (Doesn't it always?) Donuts didn't arrive until later.

And of course, not all movers are created equal. Some are stronger than others. When packing the boxes themselves, not everyone knew what went where or what should be thrown out instead of packed. When packing the truck, some people played more Tetris as a kid, and not everyone knew what was packed where and therefore what could be placed on top of what without it breaking.

Software is complicated, m'kay?

And so it is with software projects.

Writing code is only one part of the process. Long before you get there, you need to spec out the site you're building. You need to know what functionality you're going to need and what you will leave out or postpone for phase 2. You need both wire-frames and a design comp, both of which can dramatically alter the functional spec and vice versa. With some more troublesome clients that can take longer than actually implementing the functional requirements. Get the architecture wrong and you're doomed before you've configured one module or written one line of custom code. The QA, deployment, testing, and support phases cannot be forgotten, either.

A project's progress may be constrained by far more than the number of code monkeys one can throw at it. Servers need to work, and work quickly. Network connections need to be stable and fast. You may not have the content for a site when you start, or only a portion of it. You may need to connect to a 3rd party service that just doesn't want to cooperate. Someone is going to get sick at the worst possible time. And of course your client contact's supervisor is going to request a last minute change that completely unravels everything you've done to date, because that's what client contact supervisors are for.

While it totally screws up the spreadsheet, we also cannot escape the fact that not all developers are created equal, either. Some are just flat out better at what they do than others. Some write great PHP but have no visual eye to do CSS or vice versa. Some may have the necessary skills with some system (Views or CCK or Panels in Drupal-land, or some 3rd party API) that would take another days or weeks to acquire if they are even up to the task at all.

And, most damningly, you can't reasonably document where everything is in the truck. There are certain parts of a project that cannot be divided and conquered separately. Having one person, or two working together the entire time, handle all packing of the truck creates a consistent, cohesive packing plan, even if it wasn't planned from the outset. Those dedicated people are able to adjust organically (dare I say, "be agile") when a box doesn't quite fit between the table and couch after all without being constrained by having to explain it to everyone else, and then know exactly where everything is when unpacking it 2000 miles away. Which means that yes, the person packing the truck *must* be one of the people who will be going with the truck.

Similarly, having one person take a week on a highly complex new Views plugin is going to be better, and faster, than trying to have 5 people get it done in a day. The problem space does not divide into 5 parts. Trying to force it into 5 brains is going to make it take more than a week, not less, and the resulting code will have more bugs, less flexibility, and will probably fall over and break your good china when you round a corner in Pittsburg.

In fact, I'd argue that dividing and conquering in the right place is the single most important part of project planning. Figure out what your smallest reasonable chunks of functionality are, and which would be better if the same person's informal knowledge carried from one to the next. Figure out which would benefit from having two or more people involved for the entire chunk, because the box is just too heavy for one person. Then you know where, and when, and if, you can add and remove people to the overall project. That's being agile.

Action items

So what's the take-away for a successful software project?

  1. Always pack your boxes in advance before the movers get there. It may take longer than moving them, but it will avoid having to put the silverware in with the cat food because you already put the towels in the truck.

    Design and architect first, before you build. As a professor of mine once said, "Prior Planning Prevents Piss Poor Performance."

  2. Make sure the people packing the boxes know everything that is being packed; or at least that someone is coordinating the packing who knows everything that is being packed, even if other people actually put books into boxes. Tip: Most of your moving team is completely unqualified for this task, so make sure you have the right person (or people) organizing boxes.

    You need a good architect, and not everyone is a good architect. You need to give them enough time and space to do the job right. Yes, this is billable time.

  3. Know what is in each box before you start moving it, and which can stack on which. You cannot add more people to the project than that, no matter how much the client may demand you do so or how far behind schedule you are. All it will do is make it worse.

    Adding people to a late project makes it later, unless you've already figured out, in advance, where you can add them and you do so before it's too late.

  4. Make sure the strong people with a gentle touch carry the glass figurines. That's not everyone.

    Not all developers are created equal. That's OK. Allocate people to where their skills are going to be most effective and accept that will be different for different people. You may even end up with some people idle part of the time as a result. That's OK, too.

  5. Don't try to carry a box in pieces. That just rips cardboard and is impossible to tape back up.

    Don't break a problem down too far. Sometimes you need to just let someone deal with an entire problem space on their own. Get out of their way and let them do it, however long that takes. The alternative is always worse in the long run.

Quite a lot to learn from a moving van, isn't it?

If a van leaves Chicago...

In the end we managed to get everything packed up into the truck with only a few inches to spare. We had leftover donuts and bagels and the whole process took about 5 hours, including pizza. Of course, that's not counting the moving couple spending the previous two weeks packing on and off so that we could blitz through the truck-stuffing in a long morning, nor the two day drive to New York.

Of course, that doesn't answer the biggest question: Why would anyone want to move from Chicago to New York?

Bookmark/Search this post with

Jun 03 2010
Jun 03

Today at Open Source Bridge, Amye Scavarda and I co-presented a session on Thinking Like a Pirate (aka creating solid requirements docs, and how requirements docs help projects succeed).

Our slides are available here:

(As an aside, for those of you who don't know about Open Source Bridge, I'm glad you asked! It's an awesome, open source focused conference that occurs yearly in Portland, Oregon. Read more on their site.)

In our session, we fielded a question about some of the specific tools we use to build requirements docs. One of the things we use is an intake survey focused on design-related topics, and a participant asked if we'd be willing to share it out, to which I replied, "Sure!"

Which in turn meant I needed to make it pretty.

Intake surveys are incredibly useful tools. We generally use them as a way to start communication rolling on a project, and as a way of accomplishing some or all of the following goals:

  • Learn about the organization's design aesthetic and needs;
  • Begin to define a common vocabulary for web design and development;
  • Start the conversations about the relationship between design, site architecture, and user experience;
  • Begin to create the relationship between the communication leads for both stakeholders;
  • Get a sense of the level of research that has gone into the project prior to beginning development; and
  • Get a sense of how the organization responds to requests for information.

Below, I have included the general questions and structure of our intake survey. We vary the questions depending on the nature of the project; for example, if there is no desire to incorporate content rating, we don't ask the client to look at sites that feature content rating.

Design Elements

The following questions ask you to look at specific elements of web
pages. For each element, pick out 2-3 sites with the following elements
you like, and one site you dislike.

Logo

    Header Graphic

      Primary Navigation Links

      The primary navigation links are generally located at the top of the
      screen/page

        Navigation Buttons

          Navigation Menus

            Sidebar Blocks

              Footer

                Font

                  User Dashboard/Profile

                    Functionality/Workflows

                    User Registration Workflow

                      Content Rating Workflow

                        Content Creation Workflow

                          Web Sites

                          The previous section required an examination of different elements of
                          a web site in isolation. This section asks you to look at all of the
                          elements as they work together in the web site.

                          Two sites that just stand out as good

                            2 sites that are just painful to look at

                              2 sites that are comparable to what you’d like to build

                              Compare the sites that are similar/related to what you would like to
                              accomplish. Highlight the similarities and differences, and how your
                              organizations approach/goals/emphasis compares to the sites included
                              below.

                                If you have requirements capture methodologies or resources that have been effective for you, please share them, either on your blog or in the comments. The means by which software development occurs should evolve over time, and we love hearing about how other people are solving similar issues.

                                Mar 11 2010
                                Mar 11

                                Yesterday I saw this post with a video on the new P2 theme for Wordpress and how Automattic* is using that as a real time microblog like tool for status updates in their company. And it got me thinking...

                                For some time now we've been using Open Atrium in Pronovix for project management. But until today all of our real time communication takes place through Skype. At some point we tried to replace Skype with XMPP (because Skype didn't properly work on Linux) but once Skype worked again for all the colleagues Jabber got abandoned.

                                One of the reasons why we were interested in Jabber was to integrate activity updates into our communication tool (e.g. posting comment, case, commit messages). So that we would be able to have a discussion and have a bot post newly created cases into our chat channel.

                                Right now our workflow is as follows:

                                • while in a Skype chat realize there is a new issue
                                • go to Open Atrium
                                • open a case submission form
                                • fill it out and submit it
                                • copy the created url
                                • post it in Skype

                                Wouldn't it be cool if you could have 1 interface with the activity/chat river of your team where you could create new cases without having to go into another context and then have that case submitted into the activity stream where your colleagues can react on it, either through inline comments (if you want it to be stored in the ticket history) or just chat messages?

                                Today we did a short brainstorm on that in the office. We examined the following issues:

                                • scalability: using nodes for each submitted message is not going to work for big groups (unless you throw hardware at it of course). Doing a pull system with a full Drupal bootstrap for every user's request, every second or so will eat even a decent server if you have enough users.
                                • immediacy: Drupal bootstrap time and network latency will slow down the interaction.
                                • security: you want to be able to make sure only people with the right permissions can see your content.

                                The schema below explains how we are now thinking of solving these questions:

                                • For scalability we want to make a hybrid Drupal/XMPP system: this way it doesn't matter how many users are subscribed to a certain group.
                                • The XMPP server has channels for all groups in the Open Atrium install.
                                • Users with the permission to view content of certain groups get subscribed to the respective channels.
                                • The client for the XMPP messages is a webpage in Drupal.
                                • The page has a submit box for posting chat messages that get sent to the appropriate channel on the XMPP server.
                                • Javascript shortcuts let you perform some of the most common operations like selection of next/previous messages; creation of the different content types available in Open Atrium.
                                • Node submission forms are loaded in an overlay with the node form (this could be implemented using the modal frame API as it's done in Drupal 7).
                                • A prominent context box (that's always visible) displays the last group context you viewed a message from (possibly other context too). Using prepopulate this context gets transfered to new forms you open.
                                • Upon submission Drupal sends a message with the new information to a Jabber channel (this would probably be JSON)
                                • XMPP messages get parsed and displayed on the webpage (comments on nodes could be displayed in a threaded way).
                                • A set of filters lets you filter for streams from only certain groups.
                                • A message that hasn't been selected yet, is coloured differently to indicate that it hasn't been read yet.

                                This is a first draft of the specification, there are still a couple of things that need to be worked out and tested. For example:

                                • We still need to figure out how we will archive the stream so that users can go back in time and see messages/activities that got posted before they opened the stream page.
                                • We also need to do a thorough security review of the whole system.
                                • We'll also be able to add add private messaging to the interface...

                                PS: one thing that is getting me even more excited about this feature is the possibility to display activity streams from groups on other Open Atrium sites: with this system we could make it possible to have federated activity streams from all the Open Atrium sites you are registered on...
                                PPS: This model will of course also work for non-Open Atrium Drupal sites

                                Bookmark/Search this post with:

                                Feb 02 2010
                                Feb 02

                                We’ve just recently made the switch from Active Collab to Open Atrium in the office for our project management and ticketing system. We made the decision to switch based on our needs and the fact that we know Drupal so if we need something that OA doesn’t have, we can build it.

                                One of those things is time tracking. We first took a look at what was out there for time tracking in Drupal. Of the few modules that seemed to be in development, none really hit the target for what we needed or how we really felt time tracking should work. We felt that time should always be contextual and it should fit into the flow we already have set up.
                                 

                                We felt that time should always be contextual and it should fit into the flow we already have set up.

                                So... introducing time_tracker!. We've just started work on this module so functionality is sparse at the moment, but we'll be working hard on it until it's done. It's built with the case_tracker module and OA in mind, but will work with any nodes that have commenting enabled and have the time_tracker bit set.

                                On top of tracking your time, you can also report on it. We've included one all encompassing view to filter all time data, but have plans to build out more specific views for reporting. Graphing anyone?

                                Our long term goal for both time tracking and OA is a desktop Air app for handling some basic functionality of OA. Notifications of new tickets and a time built in for adding time entries. Beautiful.

                                If anyone has any ideas for this module head to http://drupal.org/project/time_tracker and add a feature request.

                                Dec 08 2008
                                Dec 08
                                Now that I have been a full-time Drupal Developer for a couple of years, and a part-time Drupal developer for a year before that, I am beginning to recognize recurring patterns and problems in Drupal projects and people's experiences with Drupal. This is a report on my experiences, and a summary of learned lessons and recommendations for those entertaining the idea of a financially-driven Drupal project.

                                Executive Summary

                                1. Don't undertake a Drupal project without, at the very least, someone on the team or available to the team, to check solutions are architected the Drupal way, monitor development for employment of good practices, look for learning opportunities for the less-experienced Drupal developers on the team, and be available in a timely manner to answer the question What would Drupal do?.
                                2. Don't underestimate the cost of learning Drupal. It is almost always greater than the difference in price between experienced and inexperienced Drupal shops or developers.
                                3. If you do not have an experienced Drupaler on your project's team, expect it to go several times over budget, and/or be under-delivered. Most of the time it will be cheaper to have the developers use the tool they already know.

                                Introduction

                                As more and more people discover and decide to use Drupal, the more and more popular it becomes. Given the way a crowd-attracts-a-crowd, especially with free and open source software, Drupal is quite clearly set to become the dominant open source web CMS in the professional and business-oriented web-development markets – if not the dominant CMS of both open source and proprietary options. It's also likely that Drupal will dominate in the amateur and hobbyiest markets too, when usability bugs get ironed out with a little more time and a couple more Drupal versions.

                                Many non-Drupal web development shops are recognizing this, and are beginning to realize that it's too expensive to maintain their own proprietary CMS, or – worse – maintain skills in multiple different CMS, when one could rule them all, and even do a better job.

                                The Problem

                                Businesses and individuals seeking websites also recognize this. In fact, many clients looking for a web shop have already decided they want their website to use Drupal. Instead of shopping for a general solution provider, they often look for a Drupal developer or Drupal shop.

                                I have seen this many times now, always with one of the two following results;

                                • They find an experienced Drupal developer, recognize the value and importance of siad Drupal developer's experience and role in the project, and get them involved.
                                • They don't find (or find, but can't afford) a Drupal developer or shop, fail to recognize the importance of this, and contract web developer or web shop with little or no experience.

                                Unfortunately – because of the high demand for experienced Drupalers – most end up in the last category. This is understandable, given that most clients seeking someone to build their website do so because they lack the expertise to do it themselves – they are therefore not likely to understand that – even with software, including Drupal – a tool is only as useful as the skills and experience of it's user.

                                This always – with no exception – ends up leaving the developers running frustratedly in circles, and the client burned with an under-delivered and over-budget product.

                                I have seen a number of non-Drupal web development shops talked in to using Drupal for the project because it was what the client wanted. The developers or shop often tell clients that "Drupal is a PHP application, and we're a PHP shop, so we can do Drupal". Even the developers often fail to recognize that Drupal is worthless to them until they have learned "the Drupal way", have experience with Drupal, have become self-dependent Drupal learners and can confidently answer the question What would Drupal do?. For most web development shops this is a significant amount of resources to commit. In fact for most run-of-the-mill shops, which are typically small businesses of 5-15 persons, it's too-large an investment.

                                The Solution

                                And indeed it is a large investment – after working full time with Drupal for a couple of years now, I'm still learning new things about Drupal just about daily. And there is no possible way any person or even small group of people can recall, understand and know how to use even half of the almost-4000 contributed modules.

                                The solution is to develop mentoring relationships. Every Drupal project needs at least one person who is sufficiently experienced with Drupal such that he or she can guide other developers through the project, showing them what Drupal would do. Finding this mentor or Drupal developer is key to the success of any Drupal project with any non-trivial amount of configuration and code.

                                It is the responsibility of project stakeholders to find these people. It is also the responsibility of web developers to avail themselves in a mentoring capacity to less-experienced Drupal developers.

                                About Drupal Sun

                                Drupal Sun is an Evolving Web project. It allows you to:

                                • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
                                • Facet based on tags, author, or feed
                                • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
                                • View the entire article text inline, or in the context of the site where it was created

                                See the blog post at Evolving Web

                                Evolving Web