Jan 21 2020
Jan 21

Bloomington Public Schools website

From chaotic and disjointed to streamlined and cohesive

Bloomington Public Schools (BPS) has been a client since 2011. We migrated them from SchoolFusion back then and helped them design, architect and develop The Hub, a parent and student portal in 2013. Not only have we been keeping their district and accompanying school websites updated and secure, we’ve been adding features to The Hub. In 2019, they decided it was time to re-architect their district and school sites to be user focused and consolidated under one domain. So we helped them with their strategy, redesigned the site and migrated to Drupal 8.

“We already had a good working relationship with TEN7. They know our organization, they know our systems,” said Andrea George, BPS Communication Specialist. “When that’s the case, you get a different level of collaboration and outcome, versus when you just start to work together. We didn’t know how big of a benefit that would be.”

The redesign was going to be a big job, as the sites had issues on many fronts:

  • Content: Things had gone a bit “wild west.” There were 3,000 content items on the site (!), and a large chunk of them were expired news content or dead pages created for one-off purposes.
  • Usability: User feedback (parents and staff) overwhelmingly said that it was too hard to find information they needed. The navigation wasn’t intuitive, was different on each school site and was located in different places on school pages, and in general the page designs just weren’t user friendly.
  • Search: The search didn’t work well, and was bringing up really old content.
  • Design: The sites were over seven years old—a lifetime in internet years—and the design was starting to show its age. Each of the 15 schools lived on its own subdomain, had its own visual identity and there was no visual design relationship between schools, or between the schools and district pages.

Our assignment: cull content and design a user-focused district-wide site

When you redesign legacy sites, the best place to start is with content strategy and a content audit. “Since we were truly starting from scratch, and we were not going to automatically migrate content over, we were committed to looking at EVERYTHING,” said Andrea.

Lynn Winter, our Content Strategist, went through two content purge rounds. The first identified content that should absolutely be removed, like outdated content, and “dead” pages no one was visiting. The second round identified content that could be combined or removed, but that needed BPS review. When it was all over, Lynn and BPS had pared 3000 content items down to about 1000.

To avoid the same content chaos on the new site, BPS wanted to be more intentional about allowing content on the site. Webmasters from each school would be gatekeepers for content, and there would be an approval process for any potential new site content. To assist with this, Lynn ran a workshop with the BPS marketing department to teach them how to rigorously evaluate content, both to determine what should be recreated on the redesigned site, and to help them determine criteria for adding content to the website in the future.

On the infrastructure side, we rebuilt the sites on Drupal 8, got rid of the subdomains they were using and unified the district site and all the school sites under a single domain. We set up the Drupal Group module to provide relationships between the content on the site and group entities. The module also manages permissions and roles for each school, which would help in the future content approval process. We implemented SOLR for search on the back end, with rules to make search results only show current school year content and prioritize recent content. We also added a keyword field to each page, so it could show in results even if a desired keyword isn’t part of the organic page content.

With the content and infrastructure nailed down, the design challenge was: how do we create one design to accommodate the branding distinction of individual schools? We decided the district brand should be the strongest, and we kept a tight shared design aesthetic between the district and school branding. To keep brand impressions consistent between schools, all the schools have a consistent design, with differences occurring in their unique logo, and a distinctive color palette, chosen from the BPS “super style guide” that we created. If a new school is added down the road, they can again choose from the approved palettes. We also made sure that the design was responsive and looked good at multiple screen widths.

The district branding/navigation is omnipresent with a mega menu at the top of any site page. School site navigation is found in a consistently located left sidebar, and navigation options there are related to each school.

A few months before launch, we trained the school webmasters in using Drupal 8’s component-based pages and were there to answer any questions they had on using the new site. Our designer also attended the training to get feedback on whether the content was working as intended with design—and then tweaked the design where it wasn’t.

Outcome: a streamlined, fresh and cohesive site

BPS launched the redesigned district site in mid-January 2020, and they’ve already received great feedback from staff and other stakeholders. “We were worried about how schools would react, but it’s been really positive,” said Andrea. Kate Martin, District Marketing and Communications Manager, added, “They like the use of color. It’s brighter, fresher, and feels more modern. Everyone is just really excited. From a design and style standpoint, the site has achieved all our goals.”

Jan 09 2020
Jan 09

Your browser does not support the audio element. TEN7-Podcast-Ep001-D7-to-D8-Migration.mp3

Summary

Ivan Stegic and Tess Flynn discuss migrating Drupal 7 sites to Drupal 8.

Guests

Tess Flynn, TEN7 DevOps, and Ivan Stegic, TEN7 Founder and President

Transcript

Tess: You're listening to the TEN7.com audiocast. We're here to discuss Drupal, migration, technology, and I'm here with -

Ivan: Ivan Stegic

Tess: And

Jonathan: Jonathan Freed

Tess: And I'm Tess Flynn otherwise known as Socketwench and... so, we have some questions to go through today, Jonathan. Why don't you get us started?

Jonathan: Thank you very much, Tess. Let's see, our first question: why are we launching this informational program of an audiocast?

Ivan: Well, we're doing it because we don't want to do a podcast and have audio out there the web and in the iTunes store that lingers and never gets repeated so we're going to try this, we're going to experiment with short audiocasts because we know our listeners' attention span is very short. And we're going to hopefully put them up and transcribe them, and have them as enhancements to our blog posts.

Jonathan: For our second question, what are the key feature differences between Drupal 7 and Drupal 8?

Tess: Where do we start with this one? Because there are so many of them. There's a lot of backend changes, there's a lot of technology changes. Some have compared Drupal 8 to removing an entire house from the foundation and then putting it down on a new foundation. And that's not necessarily inaccurate. That's a pretty good description. We went through a huge amount of effort to rework Drupal 8 to be based on newer technologies such as PHP 7 so it's faster and more memory efficient; with Composer so that we can bring in more efforts from the external -- outside Drupal PHP community, with twig for better theming -- the list goes on and on and on.

Ivan: And just some user interface changes in the upgrade between D7 and D8. We added in place editing, which never happened before, so you can now change content on the page that you're on just by -- just when you're looking at it. We designed a theme so that it is mobile first. We also designed and implemented multilingual first, as well. There's internationalization: that's a part of core. Accessibility has been improved, and was a core initiative. And actually, I've read that there are fewer modules now involved in Drupal 8 compared to the original core of Drupal 7. So, fewer modules, bigger punch, and we've brought in additional functionality from contrib as well.

Jonathan: And the last question: what are the factors that a website owner should consider when trying to decide if they really need to do a Drupal 8 upgrade?

Ivan: In my mind, the biggest factor is how old your current site is, and what technology it's currently based on. If you have a Drupal 6 site, you are more at risk from a security standpoint than you are in Drupal 7 and Drupal 8, mostly because Drupal 6 is no longer supported. So, if you're in Drupal 6 you should absolutely be considering an upgrade to Drupal 8. And then, depending on how old your Drupal 7 site is, if it's anything less than a year or a year-and-a-half old -- so if it was built at any point starting in 2016 -- you probably don't have a good case to upgrade to Drupal 8. But, anything older than 2016, there's a definitely a reason to be considering Drupal 8 and an upgrade to Drupal 8, especially if you feel like there is a user-experience issue with your current site.

Jonathan: Thanks, Ivan! And Tess, do you have anything to add to that?

Tess: No! I think Ivan covered that really, really well.

Ivan: Thank you.

Jonathan: Excellent! Well that brings us to the end of our first audiocast and I would like to thank Tess and Ivan for sharing their insights today. Please visit us at TEN7.com and keep an eye out on the TEN7 blog for future audiocasts. This is Jonathan Freed and thank you for listening!

Jan 08 2020
Jan 08

Your browser does not support the audio element. TEN7-Podcast-Ep-079-Best-of-2019.mp3

Summary

2019 comes to a close, and with it, another year of fascinating podcast guests that taught and inspired us. We’ve selected some of our favorite clips from this year’s guests for you to enjoy. Listen to the full podcasts if you like what you hear.

Guest Clips

  • Episode 50: Dries Buytaert, Founder of Drupal, on his evolving role in the Drupal community
  • Episode 74: Rob Harr, VP at Sparkbox, on how Sparkbox alums are still part of their story
  • Episode 56: Lynn Winter of Manage Digital on burnout
  • Episode 66: Jeff Archibald of Paper Leaf on the “false hustle”
  • Episode 67: Alison Paris of Parisleaf on G-RICE, their five values
  • Episode 63: Carl Smith of Bureau of Digital on scaling intimacy
  • Episode 53: Heather Schrock of Bonneville Environmental Foundation on their water stewardship business engagements
  • Episode 64: Hans Bjordahl of Culture Foundry on Conscious Capitalism
  • Episode 77: Jeff Geerling, Software Developer and Creator of the Pi Dramble on writing books on LeanPub
  • Episode 78: Gareth van Onselen, South African Journalist, Political Analyst, Author on the similar narratives of Jacob Zuma and Donald Trump

Transcript

IVAN STEGIC: Hey, everyone. You’re listening to The TEN7 Podcast, where we get together every fortnight, and sometimes more often, to talk about technology, business and the humans in it. I’m your host Ivan Stegic. We’re starting off the new year with a retrospective.

We published 29 episodes last year, which is slightly more than once every two weeks, or as we say it here, every fortnight. That’s a great deal of content, all of which was created by an amazing team here at TEN7. So, let me begin with some thanks. I’d like to thank our transcriptionist Roxanne, who works tirelessly transcribing the audio for every episode, so that we can publish the text online. Thank you! To Charlene, who edits each of the transcripts and comes up with the summaries, links and social media posts, thank you! And of course, to Jonathan, the producer of the show, who masterfully records, edits, mixes, schedules, uploads and makes my life magically easy to record this show. Thank you! None of the almost 30 episodes of 2019 would be possible without you.

So we’re going to look back and play some of our favorite excerpts of the year. Before each clip, I’ll tell you who the guest is, what episode it came from and I’ll try to set it up with additional context. And don’t worry, today’s podcast webpage has links to the episodes, so you can always go back and listen to them all.

We started off 2019 with our 50th episode, and who better to have as a guest than the founder of Drupal himself, Dries Buytaert. I asked him about how his role and how it had changed over the years, since starting the project and writing the initial code in his dorm room in Belgium.

DRIES BUYTAERT: I think my role has evolved a lot over time. I mean, in the early days I would write 100% of the code, and I would spend a lot of my time building Drupal.org. I would help run the servers behind Drupal.org. I would organize the DrupalCon events or help organize them, like intensively. And over time I’ve scaled more and more. Drupal Association would be one example of that, as a step in evolving my role, which put in place an entity, a non-profit entity specifically, that could take over the organization of DrupalCon which now is a serious event. It costs a few million dollars to put on and takes a whole team of people to organize. Same thing with managing our website and the underlying hardware infrastructure.

It’s now being managed professionally by people at the Drupal Association and again, also with the help of people in the community, just like DrupalCon. But these are examples of how I’ve scaled my role. Obviously on the technical side, I went from being the sort of single core committer, to now having teams of core committers for each of the major releases, having committees and task forces around different aspects of the project, like a technical working group that defines coding standards. We have release managers and product managers and framework managers, all these kinds of roles to subsystem maintainers that are responsible for different aspects of Drupal core. And so, these are all examples of me scaling my role over time, and we continue to make governance changes all the time and to scale the project as needed.

I think that’s the right thing to do. As projects or organizations get bigger, you need to put the kind of organizational structure in place. You also need to scale the culture of the project and so, I try to help with that through my keynotes. Actually, last year at this time, I helped write Drupal’s Values and Principles document, that’s a way to help scale our culture. So, it takes a lot of effort and different people to maintain and run the Drupal project today.

IVAN: After I had been to the ManageDigital conference, I was enamored with Rob Harr’s keynote and his leadership of a company called Sparkbox. His style is squarely focused on all the people that are connected to his company, whether that’s current employees, clients, partners or even former employees. In Episode 74, that published in November of 2019, I asked him about this focus.

IVAN: I love the friendly disposition you have, and how inviting you are, and it doesn’t just extend to clients and partners and current team members. It also extends to Sparkbox alums. When you gave your keynote at Manage Digital, I was really impressed with how you talked about the wider Sparkbox community, not just the people that are working there right now, but people who have worked and humans that have been with you in the past. What’s your philosophy around that and how do you get those people together, if you do?

ROB HARR: My philosophy around all of this is I got into this for the humans, and I believe that we have to treat them well. I think that your former employees are the ones that actually control your reputation. I want to make sure everybody leaves well, and things end well for people. I think so much of how people view a relationship, and that’s really what employment is, it’s defined by how it ends. A perfectly good, even successful relationship, if it ends poorly, can feel horrible and can color the whole thing, and that’s not at all what I want to do. I think that what we look for is for people who, during at least a season—because there’s seasons for all things—where our seasons line up and we can kick a bunch of butt together on projects, and that season will probably end, and that’s okay.

And I think that’s really healthy to think about life that way that, “Hey, our goals lined up and we did a bunch of great work together, and now our goals don’t and we’re going to go our own ways.” That doesn’t remove that person or alum from our story. It doesn’t take us out of their story. There’s a lot of good stuff that happened. And why not respect that by throwing a party on the way out?

I think the other part that is so fundamental is, I can’t talk about caring for humans the way that I do and believe what I believe and then only live that when it benefits me, when they’re employees. I think that’s fundamental. If you really care about the people, then it has to transcend when it makes sense, when it’s convenient. I invite people to come in and talk to me about what they want to do next, and I’ve written letters of recommendation to help people find new jobs when they’re current employees.

That’s all good stuff. Sometimes people grow. Well, all the times we hope people grow. That should be a common thing that we want out of people. Our alums are part of our story. I still talk to them. If I end up in a city where there’s a Sparkbox alumni and I haven’t seen that person in a while, doing dinner or taking them out, or just saying “thank you” is totally commonplace.

IVAN: Why do you call your company a “studio?” It’s a subtle distinction from being an agency or a firm or a dev shop. I think it’s worth explaining and exploring.

ROB: I hate the word “agency.”

IVAN: Yeah, I do too. [laughing]

ROB: It makes me think of Don Draper, with the brown liquor sitting in his office, smoking, and the whole power dynamic that comes with that. I think the word “agency” has been overloaded a lot and doesn’t make clients think of partnership they have. It thinks like, “Hey, we’re going to throw some work over the wall and it’s going to come back and you’re going to be our agency of record.” It’s everywhere. I like the word “studio” because I think it speaks better to the creative problem-solving work that we want to do in partnership with our clients, and it invokes the right feeling of that.

IVAN: Rob was one of the keynote speakers at Manage Digital, that’s the annual project managers conference in Minneapolis put on by our good friend Lynn Winter. She was on the show as part of Episode 56, in March. And at the conference, she was giving a mini-keynote about burnout, so I asked her about it. She had some wise words to share with us.

IVAN: Let’s talk about burnout. How prevalent do you see that? Do you have any data to show? What have you noticed and what are you hoping to achieve with your mininote?

LYNN WINTER: In general, stats, if you start looking at people across the industry, or working people, and then specifically, across the industry, there’s so much pressure and change of environment from, where is the line between home and work, from working at home or having technology accessible all the time? It’s really blurred the line of when do you go home? When do you put things down? When do you turn off? And it’s becoming a problem, and honestly this talk came out of my own personal challenges with burnout, but as I started talking about it with people, I heard these stories over, and over and over. I’d be speaking at conferences and people would talk to me about, “How do you deal with this?” and “How do you deal with that?” and it was all about boundaries, and all about taking ownership of the path that you actually want to go on, versus letting the path take you.

I think it’s a thing that PMs deal with more in this industry, because of maybe the role and how it’s often positioned at agencies. So, what I mean by that is, a lot of times it’s not as valued as a developer, for instance. Like, when you’re going to hire, “Oh, we can get any old PM, they don’t really need to be a PM, they can be anyone, they can do that, they’re just scheduling meetings,” but hire a developer, it’s, “We’ll get a recruiter and it’s going to be harder.” And, it’s hard for different reasons, to hire different roles, but there’s just oftentimes that situation setup or less responsibility or value placed on it.

Some PMs aren’t even allowed to talk to clients. They have to go through account managers, they’re not given the responsibility or the value in that role. And so, there’s some positioning at certain places, and then there’s also that impact of how you personally come to your role and for me, I personally have an issue, balancing, l like, don’t take on too much, and stop running the hamster wheel kind of situation.

So, I had a kind of career change back in 2017, and I’ve been really trying to force myself to step back and try to realign what are my goals in my career and what are my personal goals, and things like that. So, since I’ve started talking to people about it, I’ve gotten, honestly, a lot of feedback that has been positive, as well as a lot of sad stories. So, it’s something we need to talk about.

IVAN: In Episode 66, from the end of July, I talked to Jeff Archibald, CEO of Paper Leaf, about two articles he wrote, one about the false hustle and the other about the 60-hour work week. I asked him about what motivated him to write them.

IVAN: It segues into this article you wrote on Fast Company about the false hustle, how keeping busy is just a way of not getting things done.

JEFF ARCHIBALD: Right.

IVAN: And you’ve also written about humble bragging, about how a 60-hour workweek is actually symptomatic of larger problems that you might have as a company. I’ve talked to Lynn Winter in a previous episode about burnout, and it just seems to come up more and more. Like we are trying to be cognizant of what we’re doing, of what our employees are doing, so that we’re not detrimental to our own mental health, and so that we’re happy in the work we do, so that we have careful focus in work and life and home. So, the question is, what motivated you to write those two articles? The false hustle one and the one about the 60-hour work week?

JEFF: The 60-hour workweek one was just a bit of a direct response to what I was exposed to, and what I’m sure virtually every listener and yourself has been exposed to as well, through social media, and just is that humble bragging, right? “Oh, man. Crazy week. Finally shutting the laptop down. It’s 10 p.m. on Friday.” It’s indirectly talking about how important we are—and I’m totally guilty of this in the past as well—how important we are, how busy we are and how successful we are. But in reality, that’s not sustainable. It’s symptomatic of not having enough process in place, or not having enough revenue coming in, or just symptomatic of a host of potential flaws with the business model.

So, instead of bragging about it, it’d be great if we were bragging about how everybody in our shop worked a 25-hour work week and it was super stoked, and we’re hitting 30% profit margins, and everyone is getting paid properly. Those are the things we should be bragging about. And I get that, especially when you’re starting a business, a start-up or an agency or whatever, there is an inordinate amount of time that needs to be put in to get to the point where you have enough clients and you can support bringing on somebody to help ease that workload. I understand that. But, if it’s continually what we’re toting as success, then I think we have it totally backwards.

The false hustle article, that was more just like, I suppose, a moment in self-awareness for myself. I’m a productive person, I can get a lot of stuff done very quickly, but I have a tendency to overvalue the volume of tasks I complete, versus the importance of those tasks. I could sit down and crank out 12 things in a day, but did it actually move the needle anywhere? I was working really hard and I was being really busy, but it’s the equivalent on some level of, in that article the analogy I drew to Sammy Sosa’s sprinting from the dugout to the outfield, in between innings, but then jogging after a fly ball. [laughing] It’s the same kind of thing.

For me I just wrote that more to remind myself that I need to make sure and understand what I really need to be working on. What’s truly important. And apply tools like the Eisenhower Matrix to understand what needs to be done now versus what can be delegated and plan my week out a little bit better. Or else you can get to the end of the week and think that you’ve moved the needle because you did a lot of stuff. But it doesn’t mean you actually have moved the needle.

IVAN: How do you keep yourself on track for that? It sounds like a great idealistic way of living, and I would love to do it myself, but how do I actually do it? How do you do it?

JEFF: It’s relatively straightforward, to be honest. It is, I suppose, a series of processes. So, we use OKRs here: objectives of key results. I set one or two objectives for sales and marketing, which is primarily my focus here for every quarter, and then I list out the key results. If you’re a listener and you’re wondering what that means, just google OKRs and you’ll find a whole bunch of really interesting methods and information about it. But I set up those results and those are the things that I really try to focus on. Those are the things that are going to move the needle.

So, I map those out. So if I have an objective to increase revenue for the next quarter, then some of my key results might be to pitch three new projects every month. It might be bid on $1.5 million of work. Just key results like that. So that’s where I start, and then at the start of every week, I have a reminder in my calendar and about a 30-minute window to actually plan and block out my week, and it says right in there, review your OKRs, figure out what you should be working on, and then I’ll go through my calendar and I’ll see what time I have available, that hasn’t been booked for meetings or whatever else.

And I’ll block in time to focus on this particular sales objective, or this key result. Or, this particular proposal that I know was due by the end of the week. So, for me, those OKRs ultimately are making sure I’m working on the really, really important stuff that’ll move the needle, and then that weekly calendar reminder and the subsequent blocking out of my time on a weekly basis, is how I make sure that stuff actually gets done.

IVAN: How many companies have you worked for that have gone to the trouble of figuring out your own personal love language? Well, Alison and Chad Paris, who run Paris Leaf, were my guests in Episode 67, and in this clip Alison talks about their company’s values: gratitude, responsibility, integrity, candor and excellence.

IVAN: Ali, what is G-RICE?

ALISON PARIS: G-RICE stands for our five values. And I do want to expand on that a little bit, what Chad was saying, because I think it’s important to talk about using your purpose and ambition statement and also your core values, to make strategic hiring—and sadly, firing—decisions, I think by evolving our core, kind of brand messaging, and really owning it and metabolizing it for ourselves, it strengthened our ability to make really strategic decisions about who we bring on our team, and as a result, the conversations are so much easier, making decisions as a collective are so much easier because we just bounce it off of our values, which are gratitude, responsibility, integrity, candor and excellence. So, that’s where the G-RICE comes from.

IVAN: How do you know when you’re not living up to those standards and who’s policing them?

ALISON: I think we’re quite a self-policing team. We are very open with one another, and we really turn to one another to hold each other accountable, so if we feel like one another is kind of out of line with our values, we’re very quick to bring that up, in a really kind and empowered way. So, there’s not necessarily one person with the policing badge on, making sure we’re having these as our core values. I think it’s somewhat easy.

And I’ll knock on wood as I’m saying it, it’s easy when you hire and fire for your core values, because the teammates who are ultimately on your team end up embodying those values. So, it’s not as much as a policing as it is like We’re a team doing this together.

IVAN: I met Carl Smith at Owner Camp in Bend, Oregon in the Spring. I went to this Bureau of Digital event not knowing anyone, but came home feeling like I had a new, intimate set of friends and peers I could talk to. In Episode 63, I asked Carl how he planned to scale this wonderful thing he was in charge of.

IVAN: So, I’m going to ask you what your daughter asked. How do you scale that good feeling? How do you make it even bigger?

CARL SMITH: It’s tough right? Because in a way you’re scaling intimacy. Lori Gold Patterson said that to me. Lori Gold Patterson runs Pixo out of Urbana, Illinois and we were at Owner Summit in San Diego a few years back, and she came up to me. She goes, “You’ve got a huge challenge.” I was like, “What is it?” She goes, “How do you scale intimacy?” She was like, “Most of these people know each other because they’ve met, but there are going to be so many people that want in, and you’re not going to be able to do events for everybody,” and all this sort of stuff.

So now, the goal is to find ways to do more stuff online in smaller groups. To have opportunities for people who have met in person to reconnect. And for us to be the facilitator and the organizer and handle the logistics of that.

Because we can do so much more online, that as long as there is some level of in-person connection once a year, maybe twice a year, then the online supplements really well. So, that’s a lot of it. We can’t just have events get bigger, and we can’t just put on more events. It’s going to have to be finding ways, using the means that we have—and a lot of it’s technology—to make sure that everybody’s feeling connected, more frequently. And, even voice over just pixels, right?

Even just like you said, “It’s so good to hear your voice.” So, having that happen. So, we’re working on an idea called Bureau Circles. We’ll see how it can play out. We don’t want to overcommit and then blow it. But the idea of eight to ten people that we help coordinate to get together once a month, it’s kind of like a NEO, kind of like a Mastermind, but it’s super focused on what they do and their role. We called them role calls in the past when there were more people, but this idea is getting it down to a smaller group that is there for each other all the time.

And one of the things, Ivan, that spurred this was we have a Biz Dev Camp. So many camps, dude! I can’t even remember them to list them. But Biz Dev Camp, everybody said don’t do it. Nobody is going to share. Nobody’s going to be willing to share. And we said it when we brought people in.

The thing was, it sold out with seven on the waiting list. And I told them, “We’re going to have an optional show and tell on the final day, where if you show up you have to show. You have to show your pitch deck, or you have to show a presentation, you have to show something.” But what was amazing is when we left that first Biz Dev Camp, monthly calls started.

And it was just hilarious, they started their own monthly call. They called it an “accountability” call. It is just unbelievable. So in a lot of ways, that’s become the model for us as we’re looking at how do we move forward to scale the intimacy, and it becomes, people have said, compared to AIGA or things like this, “I like the grass roots feel,” and, “I like the idea that in those circles there is somebody who connects with us,” but they’re also at liberty to make sure that they’re doing what the group needs, so that we’re continuing to support, but we don’t have to participate in every single thing.

IVAN: In 2019, TEN7 started to purchase renewable energy certificates, called RECs and water restoration credits, called WRCs for every full-time employee. It’s our way of trying to offset the carbon footprint that the company creates by being in business. In February, for Episode 53, Heather Schrock from the Bonneville Environmental Foundation joined me on the show. In this clip, she talks about the projects they’re involved in and how they come about.

HEATHER SCHROCK: Yeah, so I’ll just start with water, because that is, again, our more in-house piece. Because the water projects that we support are all directly—you know, if we’re going to create WRCs it’s from a project that we, as an organization, personally support and are putting some energy behind. Whereas with offsets and RECs, we’re on the retailer side of that relationship. So, we do vet and choose the projects carefully.

For RECs there’s Green-e, the certification for RECs, and then for offsets there’s several certifications that we mainly work with four or five of them, third-party verifiers, and we just like to have a nice array of those projects that we think people would be interested in. Currently in our offset portfolio we have various things from landfill gas to forestry projects. We have an interesting marine project, where it’s a project that makes marine shipping vessels more efficient and also protects some of the sea life. So, always some interesting stuff there.

But the water projects—and I’ll send you the link later—we have a project bank on our businessforwater.org website. So, we have a separate website that we send people to, that really, specifically talks about our water stewardship business engagements, and there’s a project bank, which we started because we were finding with water projects it was getting a little more difficult, whereas with carbon offsets and RECs, there’s almost an infinite supply.

But with water projects, it’s a little more difficult to match the needs of the funder or the person who wants to buy those water WRCs or invest in that type of water project. Trying to matchmake between the funder and the project can be quite difficult, trying to get something in their region, something where they have an impact, something that aligns with their goals and their ideals around what a water stewardship project should look like. So, we started this project bank and there’s now, just scrolling through it, several, maybe dozens of water projects that are in various stages of development.

So, some of them are done, some of them are in the middle, some of them still need funding, have funding gaps, but there’s some great—if you go to that link, again, which I’ll send you later—you can see examples of all these amazing water projects. And, again, we started this bank so that we could solicit water projects from other organizations who have projects that need funding, and then, so we can come in and help matchmake the funders to these projects.

IVAN: That’s wonderful. That’s businessforwater.org and we’ll link to that in the transcripts on our website. Is there one particular project that sticks out for you? Do you have a favorite?

HEATHER: A favorite water project?

IVAN: Yeah.

HEATHER: Oh gosh, there’s some really cool ones. Let me think about that. So, most of our water projects, historically, have been very much your traditional type water leasing type projects, where we’re buying water leasing rights to keep the water in the stream, instead of having it all go to irrigation. Those are very common types of water restoration projects, irrigation upgrades, things like that. We’ve recently branched out into more urban and green infrastructure-type projects, so there’s one we’re undertaking in Los Angeles, which is a green infrastructure project that’s a collaboration with Los Angeles City Council.

There’s a neighborhood association and also an organization called Heal the Bay, and we’re all working together to create this park that will have also an impact on the neighborhood. It will be a safe place for people to play and move, and there will be fitness stations, but the park itself will serve as a water quality improvement project in the Los Angeles River Watershed. So, that’s really exciting, that we’re going to be able to save hundreds of gallons of potable water that will serve this neighborhood, but also creating this safe and fun space for them to interact with. It’s turning what was just a block that had nothing on it, that was not a safe place to be, into something that will be a neighborhood gathering spot.

IVAN: When Hans Bjordahl, CEO of Culture Foundry, attended a talk on conscious capitalism at SXSW, little did he know it would make him question everything he’d been taught about doing business. You can’t just tack conscious capitalism onto a company, it must be baked into the leadership. In this clip from Episode 64, I’ve asked Hans to give us a definition of conscious capitalism. Listen.

HANS BJORDAHL: Conscious capitalism is a model that seeks to unlock the potential for social good and humanistic good in business. Not just do that as a feel good, let’s have a corporate giving program when we’re done counting our money,

but baking that concept right into the core of your company, and doing it in a structured way that actually yields better financial results at the end of the day. When I was first exposed to this system, I was at SXSW. John Mackey, who is one of the founders of this organization and an early champion and ongoing champion, was giving a talk about a book he wrote called Conscious Capitalism, and I’m like, This is interesting. It’s a fairly aggressive assertion that the way we’ve been doing business in this country is wrong, and that we need to evolve it to fit this different, and frankly, more difficult model of how business should work.

But it’s not just profit and loss. It’s things about core purpose and things about stakeholder orientation, conscious culture, conscious leadership. That these things can be defined. That they can be, to some degree, quantified, and that they can, through research, be shown to have not just an incrementally better return on investment, but an exponentially better return on investment. The research around it is still emerging, still evolving. The ideas have really caught fire because the hunger for this in business is acute. And everywhere I come across it, people who feel they have to be one person at home, and bring a different person to work, that they can’t integrate who they are with what they do, that’s a fundamentally unhealthy place to be.

And creating business environments where you can bring your entire genuine self to work and do that, hopefully, where there’s some social good that comes out of that, that’s a very compelling thing. That idea’s been around a long time, but it hasn’t been super structured. Now it’s getting more structured, and it’s getting more structured through conscious capitalism, through B-Corporations, through some of the work that Nick Hanauer is doing. He’s a Seattle billionaire who is making a very aggressive run at redefining what successful economics look like.

So, a lot of people are approaching this idea from slightly different directions. Conscious Capitalism was my introduction into that approach to business. Some of those tenets from the book I picked up directly and plugged them right into Culture Foundry. It’s been very useful to us in terms of how we want Culture Foundry to be, but I also think there’s an imperative worldwide, I won’t even stop at the borders of the U.S., but worldwide, to really think about business in this way, and very aggressively counter the traditional model that you only have an obligation to your shareholders and everyone else can take a hike, that everything else is a commodity.

This model is a broader stakeholder model where you need to accommodate clients, employees, partners, environment, community, and shareholders, but we’re going to slice this pie up many different ways, not just give everything to the shareholders and leave everyone else holding the bag.

IVAN: In my head, Jeff Geerling is the Pi Dramble guy. So, getting a chance to sit down with him and talk about Raspberry Pis and how he’s automated them, clustered them and bent them to his will was a lot of fun for me. In addition to being a lover of software and all things open source, he’s also an author. In Episode 77, I asked him about his latest book and what he’s currently working on.

IVAN: Now, in addition to all the hobbies you have, side projects, and so on, you are also an author, and I would love to hear about your latest book and the book that you’ve written on Kubernetes. What are you working on right now?

JEFF GEERLING: I love writing. I don’t know how many million words I’ve written in my life, on my blog and on other blogs and things, but I love writing. In 2013 or so, I think that’s when I started, I’ve always wanted to write a book my whole life, I want to write a book sometime. I think part of that was jealousy because my brother, when he was a kid, wrote a book and his book, you know, the 15 minutes of fame, his book caught fire and was a local very popular book. He sold maybe 15,000 copies or something. It was pretty cool being the little brother to the brother who wrote that cool book.

But I was also a little jealous, like, I want to do that too. But I also just love writing. I’ve always loved English and literature growing up, and I love reading and I love writing. So I put that together with the fact that Ansible didn’t have a book in 2014. I started in 2013, but in 2014 I’m like, There’s still no book for Ansible, and it’s really popular.

So I decided to start writing it with a goal that I would write 100 pages and sell 200 copies. And it was funny because I started writing it on a platform called Leanpub where you can publish it while you’re writing it and sell it while you’re writing it. And by the time I had written about 40 pages I already had sold 200 copies [laughing]. And then fast forward these many years later, it’s 2019, so it’s been in print for five years now, and I now sell it on Amazon and other places and it’s called Ansible for DevOps. And that book has sold over 22,000 copies and it’s now 480 pages, including a chapter on Kubernetes and a chapter on Docker and a couple examples that do Drupal.

One of them was inspired by the Raspberry Pi Dramble cluster. So, that was my first book effort, and it went incredibly well, and I was floored. There’s no word to describe, when you're like, I want to do this thing my whole life, and this is my goal. And then your goal is surpassed by 50 times over, and you get to meet awesome people because of it. It’s just so many cool things happened because of that book. It also helped my family.

We’ve wanted to remodel our kitchen and after writing the book and making some profit off of it, I was able to remodel the kitchen four years earlier than we thought we might be able to. That’s a huge change for our life, because our old kitchen was kind of hard with three kids, and the way that we live our life and stuff at home, especially since I work remote. And I’m at home all the time. We had an old cramped little kitchen, and we were able to get it better.

So, the book was just awesome. I don’t expect to have the same level of success, but who knows. You never know where it's going to lead. But I’m working on another book. I actually just finished the first chapter a few nights ago, and I have a structure for the rest of it, and I’m working on examples and chapters.

The next book is going to be called Ansible for Kubernetes, and maybe if Ansible is around in five years and there’s another game-changing cloud infrastructure thing, it’ll be Ansible for that and I’ll have a whole series out. But I’m working on that book. I haven’t published it yet. I probably will pretty soon. Even though it’s not finished, I’ll publish in-progress updates on Leanpub, but both of those books, if you go to ansiblefordevops.com or ansibleforkubernetes.com, those are the book sites. I love writing them.

And one of the best things about writing them in progress is for both books I’ve had a lot of interaction with the people who read it, and they can help me. If they’re interested in something, I can write about that. Or if they are like, “Your example didn’t work on my computer,” I can improve it before I actually make a published printed version that people will buy.

IVAN: As some of you may know, I grew up in South Africa only to immigrate to the United States at the turn of the millennium. As a result, my childhood friends are South African, and I was able to interview at least one of them for the show. Gareth van Onselen is a South African journalist, political analyst and author who joined me for Episode 78, right at the end of the year.

We try to stay clear from politics, but in this episode, I was all in. We mostly talked about South African politics, but in this clip, Gareth talks about former South African president Jacob Zuma and the parallels to politics in the United States.

IVAN: I’m glad now that I didn’t read it [van Onselen’s book on Jacob Zuma] back in 2014, because I don’t think I would’ve appreciated it as much as I do now. And mostly it shocks me how much of a parallel there is between Zuma and the current president here in the United States. At some points I was reading the book, and I wasn’t sure if I was reading something that was directed at Zuma, or something that was directed at Trump.

There’s this one place in the book where you say, “He can fill a vacuum with empty rhetoric, but once it is all done, you’re left wondering whether he has said anything at all.” Then you also go on to describe him as something of an ethical black hole. How did such a man become elected? Not once, but twice?

GARETH VAN ONSELEN: I would agree with you 100%. I think Jacob Zuma’s tenure is a brilliant template for what’s happening in the U.S. with Donald Trump. I think they are very similar in a lot of key respects. Not just in terms of the way in which they use or abuse popular sentiment to serve what is essentially an entirely personal political agenda, but in terms of the grand narrative of the entire tenure in office, and how these demagogues tend to affect the way in which society responds to them.

What actually happened with Jacob Zuma—and I think it’s not absolute but in a lot of ways very similar is happening in the United States—is essentially you get elected on a wave of popular appeal, which takes various different forms and has various different causes, but that’s the outcome. There’s some kind of populist zeitgeist that manifests, and you’re swept along by it if you’re this demagogic leader.

There’s then a process of the shattering of the illusion. And that shattering doesn’t happen to the opposition who never had any doubt as to your unsuitability for office. It happens to sections of your own support base. The nature of public office starts to reveal who you are, the demands of making hard, often neutral and magnanimous decisions which you’re kind of incapable of doing, because you’re demagogic, and biased in a certain way, start to reveal your true character.

Then there’s some fundamental problem, either a case of corruption or unethical behavior onto which your opponents then latch as a means to remove you from office. And it becomes a defining battleground along that particular issue. In the case of Jacob Zuma, it was his homestead in Nkandla, this abuse of taxpayers’ money. In the case of Trump, it’s Russia and impeachment and the various things that go about it. But that becomes the mobilizing point.

You then suck in all of civil society and independent institutions to help you deliver the outcome you want, the judiciary and so on and so forth. And how that plays out is yet to be known in the U.S., but that’s the way in which the narrative tends to unfold. As I’m saying all this, I’m realizing I’m going slightly sideways from the original question you asked.

IVAN: No, please keep going.

GARETH: You triggered a thought with regard to the overlap between the two, and I think Jacob Zuma is a real case study for democrats and conservatives in the U.S. to look at, because the narrative is playing out in hugely similar ways on a lot of different levels.

IVAN: Well, that’s it for 2019. Thank you for listening to our little podcast. We’re so glad that you join us as often as you do. We’ve got more in store for you this year, and hope you’ll join us as we continue to talk to interesting people from around the world. If you have a second, send us a message and tell us what you would like to hear in the coming year. Or, just send an email to say hi! Our email address is [email protected].

Until next time, this is Ivan Stegic. Thanks for listening.

Dec 06 2019
Dec 06

Lutheran Social Service Contact Directory

HELPING LSS CLIENTS CONNECT DIRECTLY WITH THE SERVICES THEY NEED

Lutheran Social Service of Minnesota (LSS) is one of the state’s largest private nonprofit social service organizations, providing a vast array of services in all 87 Minnesota counties for children and families, older adults and people with disabilities.

Finding the right contact person for LSS services on their website was not as easy as the organization wanted it to be. On the lssmn.org site, while each service has its own page within the navigation, there was no consolidated contact directory. The main contact page contained a single web form which would send an email to the LSS marketing department (usually one person). This resulted in a bottleneck of incoming requests for numerous services. The team noticed that the contact form was consistently in the top 10 most-visited pages in the site’s analytics. Since it’s important for LSS to provide prompt and efficient service, they required a solution that would allow people seeking service to connect directly with program staff.

Though external visitors are the primary audience for the contact directory, LSS knew that internal users would benefit from the tool as well. The organization’s intranet has a directory, but that focuses on people results, not services: if you know someone’s name, you can look them up. But if you need to find the main contact for disability services in Hinckley, MN, this is much harder to find using the internal tool.

To quote our favorite infomercial, “THERE’S GOT TO BE A BETTER WAY!”

OUR ASSIGNMENT: BUILD A CONTACT DIRECTORY FOR BOTH INTERNAL AND EXTERNAL AUDIENCES

LSS knew that building a consolidated contact directory was going to be a top priority when we took them on as a TEN7Care support client. They had a head start on gathering contact information and putting together design options, thanks to the good work that was done during a recent website redesign. In addition, a Contacts content type had already been set up in Drupal.

We first exported the existing contact data out of Drupal into a spreadsheet, so we could see all the data in front of us. We worked to streamline the contact information, getting rid of duplicate listings and deleting unused fields. LSS Digital Marketing Manager Tom Lany was once again our hands-on partner in the project. Tom worked in concert with other LSS Marketing and Communications Managers in a multi-layered process to have all departments review their contact information for accuracy.

As the data was being updated, we worked to figure out the best hierarchy for the content. Each contact had two important facets: location and service category.

Granularity of location was crucial. Each LSS service operates at a different level of scope. Some are very local physical locations. Some resources are tied to counties, which are grouped into regions. Finally, there are statewide services, those that have no locality. They can be accessed from anywhere. For the most part, the system contextualizes every contact on a local, regional or statewide level. We used two off-the-shelf Drupal modules, Geocoder and Geolocation Field, to get the latitude and longitude based on the contact address, and to save it to the contact.

We also needed to assign each contact to a service category. In addition to the Contacts content type, the LSS site also had an existing Service content type. We had hoped to use this in conjunction with the Contacts content type for the content directory, but the Service content type represented the marketing pages for the services in the navigation structure. Since we didn’t want to be disruptive to their existing content, we created a new content type, Contact Directory Service, that would associate individual contacts with a service grouping.

To organize and display the search results, we used the Views module, which is part of Drupal core. It allows us to compile listings, add filters and perform searches. However, a little tweaking was needed. “The Drupal Views module doesn’t easily allow us to take three different queries and merge them into a single result set,” said Les Lim, Technical Project Lead, “so we wrote custom code that allowed us to dynamically change the underlying database query that built the results, according to filter inputs we were given.”

OUTCOME: INTERNAL AND EXTERNAL AUDIENCES CAN FIND INFORMATION QUICKLY

Site visitors, whether internal staff or external clients, can now quickly find the information they’re looking for. If a site visitor uses the zip code field, the system will provide local results first (including the distance from the zip code entered), then regional, then statewide. A site visitor can also use the category boxes to drill down to a list of results. If a zip code is not provided, the system will show statewide results first, then regional listings. We figured people providing a local zip code will want to see these specific results first, where people who do not have a specific location in mind will likely want to see general statewide results first.

Lutheran Social Service Contact Directory

We launched the new contact page on October 15, 2019, and it’s been well received. A month after launch, Tom Lany presented the contact directory project to a leadership team at LSS, and it got a round of applause. “People here really recognize how this tool will add a lot of value to what we’re doing,” Tom said.

Tom notified us that the contact directory launch made a difference right away. “We’re getting half of the form submissions we were before the directory went live,” explained Tom, “which means people are contacting the services directly. This saves our central staff time and means people are getting faster responses from the people they need to reach.”

As a self-sufficient client, LSS will maintain the contact database themselves in Drupal.

Dec 04 2019
Dec 04

Your browser does not support the audio element. TEN7-Podcast-Ep-077-Jeff-Geerling-Developer-Author.mp3

Summary

If this was a certain 90s show, this episode would be called, “The One About The Raspberry Pis.” Jeff Geerling, software developer, author, and lover of all things open source is our guest. It's fun to listen to Jeff and Ivan geek out.

Guest

Jeff Geerling, software developer, author and creator of the Pi Dramble

Highlights

  • Here’s the part where Ivan and Jeff geek out over their love of Raspberry Pis
  • Living in flyover country (Missouri)
  • How philosophy relates to programming
  • Crohn’s disease and its treatments (and the benefit of living in St. Louis!)
  • Remote work lessens the impact of Crohn’s disease on Jeff
  • Brambles, Drambles and Druplelets, oh my!
  • Evolution of Jeff’s Dramble
  • The upside of cloud storage systems (load balancing)
  • Kubernetes: complicated to start, then smooth sailing
  • Running slower systems lets you find problems you’d never notice on faster setups
  • Putting Kubernetes on ARM
  • Can you run Flight Deck-powered Kubernetes on a Pi cluster?
  • The Drupal file system problem (and Jeff’s solution on Raspberry Pi)
  • How to keep your Raspberry Pis cool
  • When your brother writes a book
  • Writing on Leanpub
  • Jeff’s Ansible books

Links

Transcript

IVAN STEGIC: Hey everyone! You’re listening to the TEN7 podcast, where we get together every fortnight, and sometimes more often, to talk about technology, business and the humans in it. I’m your host Ivan Stegic. My guest today is Jeff Geerling, author and software developer, who is deeply involved in numerous open source development communities. He describes himself as a creative person who builds great software. I first saw his name in the Drupal community, but he’s really active in Ansible and is also the creator of Drupal VM for local Drupal development, as well as the Raspberry Pi Dramble. Hey Jeff, welcome. It’s a great pleasure to have you on the podcast.

JEFF GEERLING: Thank you very much. I’ve been looking forward to talking to you guys on TEN7’s podcast.

IVAN: I’ve been looking forward to talking to you as well. I absolutely love Raspberry Pis. I have, I think, at least 10 of them over here. So I don’t know if that’s catching up to how many you have, [laughing] but it’s definitely something I have tinkered with for the last number of years. And so getting an opportunity to talk to you is very exciting for me.

JEFF: Ten is a good start. You’re going to have to up your game a little bit though [laughing].

IVAN: [laughing] How many do you have Jeff? Come on, let’s be honest.

JEFF: Just looking over in my parts bin I have seven that are not in any active use. I have four in my cluster. I have five hot spares for other purposes and three that are running household projects right now.

IVAN: So, close to 20, if not more.

JEFF: Yeah.

IVAN: That’s a lot of Raspberry Pis.

JEFF: It fluctuates depending on what projects I’m working on.

IVAN: That’s awesome. I want to spend the whole interview talking about Raspberry Pis and the Raspberry Pi Dramble that you have and that you’ve been working on recently, and that’s been going for over four years now, I think. But before we dive deep into that, I’d like to get an understanding just a little bit, about Jeff Geerling, the man. Where do you live in the world? Where did you go to school? What do you do in life? You’re not just playing with Raspberry Pis all the time.

JEFF: [laughing] No. I live in St. Louis, Missouri, which is pretty near the geographical center of the United States. Some people call it flyover country, but because of that we actually now have a Drupal Camp named Flyover Camp.

IVAN: Yes, we do. [laughing] It’s awesome.

JEFF: I got to meet one of TEN7’s employees, Tess Flynn there. She had a really good talk on Kubernetes. But I love living in the Midwest. One of the best advantages is I have a lot of family here. The cost of living is lower than coastal cities, and I get to still do a lot of remote work. The coolest thing is, any flight anywhere in the U.S. is only about two, two and a half hours, whereas a lot of times I get somewhere and they’re like, "Oh, I just got off a six or seven hour flight from the other coast,” so it’s nice to be able to get around anywhere in the U.S. and still have family and all those benefits.

I actually studied philosophy, which you could say it has nothing to do with programming or nothing to do with tech, but really, a lot of the philosophy I studied was logic and reason—which, if you’re a programmer—that’s pretty much all you deal with on a day-to-day basis.

Actually, if you’re dealing with Kubernetes, logic is out the window.

IVAN: [laughing] Yes, it is. It’s a different kind of logic, right?

JEFF: [laughing] Yeah, deep down, there’s ones and zeros somewhere underneath, and you have to understand that. So, I studied philosophy. I actually went to a seminary to be a priest for a few years.

IVAN: Oh.

JEFF: I ended up finding out that was not what I was meant to do. I wanted to get married. So, I ended up leaving the seminary, getting married, having three kids and a wife, and I'm happy to be here in St. Louis. One of the things other than that in my life that’s really affected a lot, especially how I work and the jobs I’ve worked, is my Crohn’s Disease. It’s a chronic illness that I have that has required a few surgeries. It’s required a lot of time in the hospital. In fact, I was just in the hospital a couple weeks ago for it, but because of that I’ve always been extremely grateful to have the skills that I have in tech because I can work. There’s been many times where I’ve been in the hospital, but nobody would even know it because I’m still doing my work, since I can do everything I need to on a computer or an iPad or whatever and work remote.

So, I’m extremely happy and feel very blessed to be able to do that kind of work. In open source everything is asynchronous, so that benefits me, because if I had to do a job where I had to be at a desk or be at a place doing manual labor of certain hours in a day and I had to miss a lot of work, I wouldn’t be a very good employee. But I could be a good employee doing remote work and open source work, even though I have Crohn’s Disease, which impacts my life and a lot of my friends’ lives pretty deeply.

IVAN: I don’t know very much about Crohn’s Disease. I know it’s related to your digestive system and I think to your intestines.

JEFF: Yep.

IVAN: Is it something that either you’re genetically predisposed to have? Is it something that’s hereditary? How did you find out?

JEFF: It’s something that’s still under investigation. One of the difficult things is that Crohn’s is in a family of immune-related diseases and for pretty much all of them, it’s hard to say right now with the research we have, if it’s more genetic or more environmental. There’s a lot of theories, and one other interesting thing—and a good reason why I stay in St. Louis—is we have one of probably the top five or 10 Crohn’s research facilities at our local Washington University Hospital and Medical School.

I’ve been able to talk to some of the best Crohn’s doctors in the world, and all of them have theories, and there’s a lot of research but not a lot of concrete facts other than, throw these medicines at it and hope for the best. Unfortunately, in my life, I’ve tried literally every medicine. I’m not joking. Every medicine that’s ever been made for Crohn’s, I’ve been on it. One of them paralyzed me from the waist down, so I got off that medicine.

IVAN: Oh my God.

JEFF: So, I'm not paralyzed anymore, so that's good. Another one caused anaphylaxis so I’m like, Okay, I’m not going to take that one. So, all these drugs, some of them are life changing, but a lot of them don’t last very long because the way that they operate, your body starts reacting to them no matter what. So, once you hit that, you start doing surgeries—that’s the stage I’m in right now—but the good thing is, in my mind, if you have a positive outlook on it, if you could find ways to work with it, you can bounce back. I’m generally a very optimistic person. I think that’s also how I can survive in the open source world where things can get a little bit dicey sometimes.

IVAN: [laughing] They certainly can. Well, I’m glad that you’re able to talk about Crohn’s so openly, and I hope that your last bout wasn’t a terrible one, a bad one, and I hope you’re on the upswing.

JEFF: I was only in the ICU for one day. That’s a good thing. [laughing]

IVAN: Oh my gosh. Well, I’m glad to be talking to you about it. So, you’re in the Midwest. Working remotely is amazing. You might know we became remote in 2017, and we’re not looking back. It’s definitely opened up so many different avenues for us. So, I’m glad that you have that as well to be able to do what you do at home.

The pidramble.com site—I want to start at the end and then kind of work back to how you are hosting pidramble.com in your basement on a cluster of Raspberry Pis. And they run Ansible and Kubernetes and Drupal and NGINX and all kinds of things, and to someone who heard a whole bunch of jargon right now, and doesn’t understand why this isn’t a really cool thing, can you describe what pidramble.com is?

JEFF: So, most of us who have a Drupal site or WordPress or whatever kind of website we’re running, most of us have a cloud hosting provider, and you never have to worry about things like servers and power and networking and all that kind of stuff, you just go online, buy a service, and you update a Drupal site to it and log in and edit your stuff. I’ve always wanted to have as much control as I can over my hosting and my sites and my performance and all that. So, a few years back when the Raspberry Pi came out, it was interesting.

The first couple versions of the Raspberry Pi were super underpowered, and if you installed Drupal—6 or 7 was the current version at the time—even that version, you’d install Drupal on it, it would take a minute or two for a page to load. They were super slow, and didn’t have much memory or anything. But as time when on, the Raspberry Pi foundation kept introducing newer, faster models, and somewhere around 2014—I think, I don’t remember exactly—they introduced the Raspberry Pi 2, and that model had enough memory to actually run Drupal pretty well. So, I bought a few of those to see if I could set up a Drupal cluster and run my own Drupal website in my own house using a cluster of them. 

You could do it on Raspberry Pi, but because of the fact they use micro SD cards which are not as fast or as good as hard drives for longevity, much less SSDs that most of us have nowadays. Because of that, if you just run it on one Raspberry Pi, it’s kind of flaky and it might just blow up and die one day, and then you have to reinstall it and stuff. So, I wanted to see if I could make a robust little cluster of computers to run a Drupal site. And, there was a secondary thing that I was trying to do too, to make a little cluster of computers to demonstrate running Drupal or other applications like it in high availability with multiple servers and have it be able to be in a little box that I could bring with me somewhere.

So, I started doing that. I brought it to Drupal Camp St. Louis, I brought it to DrupalCon. I think the first one was Austin that I brought it to, and Dries [Buytaert] actually took a picture of it. This was before I worked for Acquia—and actually I was hired by Acquia soon after DrupalCon, completely unrelated but it was just an interesting aside. Dries came up and he’s like, “Oh, what is that?” I was like, “Oh, it’s my Raspberry Pi cluster.” He’s like, “Oh, that’s really cool. I gotta take a picture.” So, he and I now have shared more stories about pictures and things. He’s actually a pretty decent photographer.

IVAN: Photographer, right?

JEFF: Yeah. And, I’ve always loved photography. I used to do it semi-professionally with some photojournalism, but don’t get to do it as much nowadays. However, some of my other Raspberry Pi projects do have to do with cameras. 

But, getting back to the Pi Dramble, over the years I’ve made it more robust, I’ve done some more work to automate the setup process, I’ve documented everything in excruciating detail. There’s videos on the website of how to do the first version of the cluster. I haven’t set up videos for the current version.

But in 2017 when Kubernetes was getting to be a lot more popular and I was starting to use it for some things, I thought, Maybe it could run on the Pi. It ran, but it just barely ran. I had a lot of trouble because the Raspberry Pi only had one gigabyte of memory, and that was the absolute minimum you could run Kubernetes on. So, I got it working. It was kind of janky and kind of fell apart sometimes, and I was getting really frustrated. But in 2018 the Raspberry Pi—no, was it 2018 or 2019?—was that this year that the Pi 4 came out?

IVAN: Pi 4 came out this year, I believe it was in June. I think they were talking about it for months, but no one ever knew when it would come out.

JEFF: That was a lot more recent that I remembered. But the day it comes out, it’s really hard to get Raspberry Pis right after they’re released, because they are very popular for makers and hackers and people who have fun with computers. And so I placed an order from a company in the UK for one, because most of them have limits on the orders for the first few weeks. So, I did one from the UK, one from a place in the U.S., I got two from Micro Center, they’re usually the best place to get them in the U.S.

IVAN: Yes, they are.

JEFF: So, I got those four together, and I got the 2 gigabyte models, and with 2 gigabytes of RAM, Kubernetes actually ran pretty well. So, that’s the current state of my cluster. I have Kubernetes running, I have Drupal running on Kubernetes, I have the 2 gigabyte Raspberry Pi 4s and I power them using power over ethernet, which means I only have one cable I have to plug into each Pi. If you go to pidramble.com, the picture there shows you how they look and what they do.

IVAN: Now, what the heck is a Dramble? I know what a bramble bush is, but let’s take it down. What’s a Dramble?

JEFF: A Dramble, is, I guess it’s called a portmanteau, when you put together different words. A bramble is a bush of raspberries. So, when you see a raspberry bush, I usually call it a raspberry bush, but a lot of people call it a bramble.

IVAN: I knew that was the case. Bramble is something I knew. [laughing]

JEFF: Yeah. [laughing] So, traditionally from the beginning of the Raspberry Pi, people who made clusters of them called them brambles. That’s not as much the case anymore since Raspberry Pi has gotten super popular. But early on everybody would say, “I have a bramble of Pis.” But my bramble of Pis was running Drupal, and so I took Drupal and bramble and put them together and it came up with "Dramble." Little did I know at the time, that was also when I registered the domain for it, that the word Dramble, if you looked it up and found it on certain urban dictionary-type sites, it had a very different definition. But I think now if you search on Google, my Pi Dramble site might have a higher ranking for it.

IVAN: I just did that. [laughing] I typed in "dramble" because I want to know what it means on urban dictionary, and the first hit was pidramble.com.

JEFF: Drupal SEO is so good after all these years.

IVAN: Anybody listening out there needs Drupal SEO, Jeff is the guy. [laughing]

JEFF: Yeah. So, that was the original reasoning, but somebody asked me more about it one time and I was like, I’ll look up a little more information, and I found out that the generic term, so, Raspberry is our bramble, that’s their type of bush, but all the different types of berries that are like it like raspberries, blackberries, blueberries, all those things, actually grow in clusters, they call them aggregates of drupelets.

IVAN: What!

JEFF: This is like providence, drupelets? So, it’s a Dramble of drupelets. It’s spelled d-r-u-p-e-l-e-t-s, but that is the official, biological term, I guess.

IVAN: You’re making that up. That can’t be possible. Drupelet. That’s basically Drupal.

JEFF: [laughing] Go to Wikipedia and search for "drupe." It has a whole article about drupelets.

IVAN: Oh my gosh, look at that. A stone fruit.

JEFF: Yeah. I don’t think I would want to eat a stone fruit, that doesn’t sound very tasty.

IVAN: No. That does not sound tasty. How interesting. So, what a great name for this little cluster of Drupal technology. Right?

JEFF: Yeah.

IVAN: So, you talked about creating the cluster. Can we go back to the first version of the Raspberry Pi cluster, the Dramble that you created? What were the versions? Was it Raspberry Pi 2 or 2+ or something like that? I’d like to hear about the stack. I think the stack of technology has changed over the years.

JEFF: Yeah. The first version was a very traditional cluster of web server technology. It was based on LAMP, and actually the first ever version which I never actually put up on GitHub, the first version that I had locally was running Linux, Apache, MySQL and PHP. The way I had it set up was, the top server was running Apache, and Apache was set up to redirect web requests to the two PHP servers, server number two and three. Then I had two database servers, there was the database primary or master server, and the secondary server that was set up to replicate the master in case the master goes down.

There’s still a lot of people that run those kind of setups where there is a server for each one of those purposes. You might be in Amazon or somewhere else, or if you’re using a cloud hosting provider for Drupal, some of them still set up their service that way where you have dedicated servers for a database and all that. The cool thing about it is I used Ansible to do everything, and I actually used almost the same kind of Ansible playbook that I use for Drupal VM to manage all those individual servers and get them set up. I timed it one time. In about 35 minutes it took from the time that you have the hardware plugged in to the time it was serving up a Drupal website with a fully redundant, highly available, relatively high performance, because we’re talking about Raspberry Pis, a cluster of computers.

So, that was the first version way, way back. Then soon after that I switched to using NGINX, just because it was a little easier to configure NGINX to be that load balance earlier, and also NGINX has built-in caching. So, instead of having a Varnish server I could use NGINX for that purpose, because it had the basic caching I needed. Varnish is way better for some things, but I switched to NGINX mostly because you just change a couple settings, and it caches images more easily.

IVAN: How many total versions have there been now if we fast forward to the Raspberry Pi for the cluster that you have now? How many versions have there been, and what are the biggest lessons you’ve learned in the evolution of this cluster?

JEFF: There've been four major versions. It hasn’t been like each Pi miles a new version, it’s more like each technological shift has been a new version. The first version had that first two database servers setup and all that. The second version I actually ditched the replica MySQL server for two reasons. One is, it’s not that easy to maintain a replica setup like that with MySQL, especially for people who are newer to the whole LAMP stack and management and things. And even with automation it can be a little difficult.

So, I ditched that just because that was never a problem in the real world for me, and the most important thing was just to have a backup of the primary server, so I have a nightly backup that it does. The second thing was just streamlining things, making it a little simpler, and the first versions I had had a lot of configurability and things, but really you want the server to be easy to set up and easy to manage, And the more complexity you add to a project like this, especially if it’s a hobby project and I’m not earning tons of income from it—in fact, I’ve never earned a dime from it.

IVAN: Not yet.

JEFF: I have gotten to go to a couple Drupal camps and things, and it’s been great to learn from it, but making it simpler is better. So I think version 2 is just simplifying the architecture, making it so that you could use four or five Raspberry Pis instead of requiring 6, things like that. Version 3, might’ve been the first Kubernetes version. I could be wrong there. That was about a year ago that I came out with 3, and I started working on getting everything into Kubernetes which makes it easier to scale up or scale down if you want to.

You could have 100 Raspberry Pis or two Raspberry Pis or four, depending on how redundant you want everything, how scalable, because Kubernetes lets you say, instead of this server is MySQL and this server is PHP, you could say, I want to have three PHP things and two of MySQL things and one load balancer thing and it just puts them on servers. For MySQL I don’t want the two MySQLs to be on the same server because that would be bad for performance.

So, Kubernetes sorts all that out for you and you just say, Hey, Kubernetes, I have these five things I want to run, and I have these four servers, go do your thing. And Kubernetes manages that for you. That’s the cool thing about Kubernetes. It’s more complicated than that in the real world, but it’s getting easier as time goes on. Kubernetes is one technology that started out crazy complicated and has gotten a lot easier over the years as they refine it, as they make things more robust and a little easier to get started with, and as people understand it more.

IVAN: Is everything in the cluster now orchestrated with Ansible, and is everything virtualized as containers inside Kubernetes, or do you still have an ingress that’s a NGINX or on a separate Raspberry Pi, for example?

JEFF: Almost everything. Right now, the ingress is actually running inside of Kubernetes and the way that I have it set up is you just point your DNS at one of the Raspberry Pis. That’s obviously not wonderful. If that one Raspberry Pi goes down, you've got to point your DNS at another Raspberry Pi. So, that’s one slight weakness of the architecture, but as an alternative you could have another Raspberry Pi being in the ingress and be a load balancer, but if that Raspberry Pi goes down you have a problem. That’s one case where having a cloud hosting provider, like Amazon or Google, is really nice because you can have their cloud load balancers. They take care of all the really complicated stuff in terms of when you get a request for your website. What happens if one of the servers that is routing those requests goes down?

All those cloud systems, they kind of self-heal automatically with DNS and with all their different things. When you’re running a website in your basement, you have one IP address and it’s not very reliable usually, especially if you’re using most ISPs in the U.S.A., and you don’t get any more IP addresses. And if that route goes bad and one server goes down, you can have a lot of issues. So, that’s one area. One of the main reasons why I would say if you have a website that sells things or you generate a lot of revenue off your website, you probably don’t want to run it inside your house like the Pi Dramble website, because it does have 10-20 minutes every few days of downtime, when my ISP is like, yeah, You guys aren’t getting internet for right now. Too bad. [laughing] That happens a lot.

IVAN: What do you think the biggest lesson is that you’ve learned going from your first version to the latest one that you have?

JEFF: I think the biggest thing is that managing Kubernetes is difficult, it’s still not easy, but managing a cluster of application stuff with Kubernetes is a lot easier than it was when it was just individual servers. Because you used to have to manage each application on each server and it would take a lot of time to get things set up and to tweak things and make sure all the backups were good. When you standardize in Kubernetes and have everything run in a container, it is more complicated at the start when you’re learning it, because containers mean that you have to build the container and you have to have a place to store the container and all that. But once you have that set up, everything is automated, like out of the box. You don’t have to spend time worrying about, How do I get this to go here and how do I change the configuration? You just say, Deploy this version, and it’s there and it’s happy. And Kubernetes does it all for you.

That was the biggest lesson; sometimes the complexity does save a lot of hassle if you need it. Obviously, there are a lot of people listening to this probably thinking You might not need to be doing your own hosting stuff. I don’t technically need to, but I do like to because a lot of the work that I do does have to be more complex. So, doing these fun side projects for me teaches me things. Another cool lesson was, when you’re running on a server that’s slower, there’s a lot of things you learn to worry about that will save your skin when you run something on even faster servers when they’re under heavy load.

One of the biggest instances of that is, when you have really slow hard drive access, when it takes a long time to write files on the Raspberry Pi—which it does a lot—you start bumping into weird issues that you never see if you only ran your tests on your local computer that has an Intel Core 2 Duo or whatever, the last i9 chip or something. If you’re doing that with an SSD all the time, you’re not going to run into these weird issues. But when you’re on a cloud hosting provider—which most companies do use those—most sites are on cloud hosting now, disk access can sometimes go crazy, and the error message you see and the behaviors you see can get confusing, because you never really notice that and you can’t replicate it locally. But I’ve been able to replicate some of those weird things on a Raspberry Pi, just because it’s so slow, the disk access.

I actually found a bug in Twig, with the way that Twig renders files. If you have multiple computers writing to an NFS storage device and that device is writing slowly, I found in Drupal 8.0, the first version, right around the beta timeframe was when I found the issue and the issue was still there in 8.0, but they fixed it. It was a race condition, when you have multiple servers writing to slow shared storage, and this was good because other people—it’s not just Raspberry Pis—if you’re using cloud storage, a lot of cloud storage providers have throttling, and you can run into this throttling sometimes, so instead of the site going down, it might just be slow. It’s not all for naught and for fun. There was that issue I found that was an actual bug in Drupal that we fixed.

IVAN: It’s amazing the kinds of things you learn when you try to scale down hardware or scale up requests and scale up bandwidth. You just don’t see things in general unless you do something. So, that’s always interesting to me as well. One of the things that I just realized as you were talking was that you actually had to come up with a way to install Kubernetes on Raspbian, and that means that you had to basically either compile Kubernetes for ARM or they’re already precompiled packages, and that never actually dawned on me. So, how hard was it to put Kubernetes on ARM infrastructure?

JEFF: So, it was perfect timing for me to be getting into Kubernetes on Pi around 2017. There was, I think it was a teenager, I don’t remember the guy's name, but there was somebody who just loved Raspberry Pis and loved Kubernetes, and he spent all this time making sure that the Kubernetes build system built and tested ARM architecture stuff for all the Kubernetes releases. And it just got into Kubernetes 1.8 or 1.9, or something like that, right before I got started. So, when I was looking into it, it’s like, Oh, you just install Kubernetes, just like everything else.

However, Kubernetes is one part of the equation when you’re talking about running a cluster that runs Kubernetes, because you also need something to run the containers, and that’s usually Docker, and Docker for ARM is a little bit different. You have to install it differently and there are some things you manage differently. The versions are sometimes older and there’s weird specific version things that cause trouble. So, I ran into some issues there, and I actually created a separate Ansible role for Docker ARM from my normal Docker one, because the ARM one is more complicated and convoluted. But it’s running pretty well now.

Then another thing that you find out if you’re using Raspberry Pis. Since the chip is an ARM processor, it’s not this typical processor in most of our laptops and servers, which is AMD64 or X86. Since it’s a different architecture, everything has to be compiled for that architecture and container images often have to have a specific version of the image built for it.

So, a lot of times you’ll be like, Oh, I want to deploy this to the Raspberry Pi, you needed to play it in the Raspberry Pi, and Kubernetes is like, I’m not going to deploy that. You’re like, Why aren’t you going to do that? So, I was like, Yeah, it’s not for ARM. Then you have to figure out, is it important enough for me to rebuild this image for ARM? So, I actually build and maintain a set of PHP and Apache base images for the Raspberry Pi, and they’re on my Docker Hub account under geerlingguy. Those are the ones that I use on the Raspberry Pi and build on top of. And a lot of images now are supporting ARM because you can get ARM servers from Amazon and from other companies.

IVAN: But it’s also a little more complex than just ARM though, if I’m not mistaken. There are different versions of ARM. Some are 32-bit, and some are 64-bit, and you can say ARM but it’s even harder because you have to know what architecture you’re building for to make sure that when you build it, it actually runs on that ARM processor. So, my guess is, you actually had to update your images when you went from two to three to four.

JEFF: Yeah, so, Raspberry Pi has its own OS that’s kind of the official one, called Raspbian, and it’s based on Debian, but it’s a 32-bit OS. And there’s some initiatives to upgrade to 64 bit, especially now that the Raspberry Pi 4 has more RAM and can use more of that 64-bit power. But, a lot of the things are either ARM V6, ARM V7, or ARM 64, and you have to always figure out which one is which. And I think, the ones that I’m building right now are ARM V7 because that’s what works on Raspbian.

But if you run a different OS on the Pis which you can, there’s a Ubuntu and actual Debian, and Fedora, and some other versions that you can run on Raspberry Pis, you might have to get a different version of a container which may not exist again. It’s not for the faint of heart to get things running on the Pi all the time. Every year it gets easier, because more and more people support Raspberry Pi stuff.

It’s an interesting thing since I do write a lot about Pis, I found out about a lot of companies who you wouldn’t even think about it, who do remote control systems, they do signage, they do radio frequency stuff, logging systems, all kinds of things that are industrial and commercial applications that use Raspberry Pis for everything. So, there’s a lot more support for it nowadays than there was five or 10 years ago when it was only a bunch of people playing around with stuff in their houses. Nowadays the Raspberry Pi is a more serious computing platform.

IVAN: What a great success story, isn’t it?

JEFF: Yeah.

IVAN: That’s cool. So, you mentioned that you had seen Tess’s talk at Flyover Camp, and so you’re probably familiar with Flight Deck and the hosting that we’ve been talking about, hosting live sites using Drupal on Kubernetes. She’s done some great work in developing that for us, and as you know we’ve open sourced all of it. But one of the things it requires is a S3 block storage like the ones from Spaces from Digital Ocean, for example. I wanted to talk to you about two ideas. One, how hard would it be to get Flight Deck-powered Kubernetes hosting onto your cluster? And two, since it requires S3 block storage, have you thought about implementing block storage on your Dramble?

JEFF: That’s an interesting question, because probably of all the issues there are in hosting Drupal on Kubernetes, or any kind of, what we call now cloud-native hosting environments, one of the main issues is always, how do you store files for Drupal? It’s complicated, because in WordPress or a lot of other systems, usually when you ask that question, you’re just talking about media uploads like, I’m creating a blog post and I upload a picture to put in the blog post, where do I put the picture? That’s part of Drupal’s problem.

That one is very perfectly solved by block storage and works great with that, and you can integrate with CDNs and things, there’s a lot of different solutions for that. What I do on the Raspberry Pi right now is I use NFS, which is not block storage, but it’s just a networked file system that is shared among all the servers, and Kubernetes mounts it into Drupal so Drupal can write to it. But when you’re talking about Drupal, you’re also talking about things like the Twig cache files. Every time that you load up a Drupal 8 website it has to write a bunch of Twig cache files that are like compiled PHP, and that currently writes them by default into your public file system. That’s a lot of files that are read, at least once per server. And so having a slow storage solution can cause problems with that and having slow writes can cause problems with that.

Then you also are talking about CSS compilation and JavaScript compilation. So, there are more complicated things with Drupal. And so, I’ve seen some people do S3 block storage, and as you say as long as you have a provider that is compatible with the Amazon S3 kind of API for writing files, it works so you can do it on Digital Ocean. There are open source block storage software that you can do, and if I were to put it on a Raspberry Pi, I’d probably use one of those open source packages and install it on the Pi. Or, if it’s an internet site, available through the internet, you can even use Digital Ocean Spaces or Amazon S3 even though you’re hosting it locally. So, those are options for it. I think you guys might even use Flysystem or something like that to make Drupal integrate with it?

There’s some PHP level stuff that you can do to write files in different places, but that is probably the number one thing that people ask about and debate about, and I’ve gone a hundred which ways. I think I built it five or ten different ways in the real-world clusters that I built that aren’t on Raspberry Pis. So, that is the million-dollar question right now. There’s even a couple issues on Drupal.org exploring how can we make it easier to use a file system in Drupal that’s shared but doesn’t have to make it so complicated.

IVAN: You’re actually right about Flysystem. That’s exactly how Tess implemented the file storage in the solution we have for hosting Drupal on Kubernetes with Flight Deck, and it’s possible to do if you set up the infrastructure so that you have enough caching, those first-time hits don’t seem as bad. And we run our own live sites and other clients sites live, in production with Flight Deck and with block storage. So, we use Flysystem in Drupal 8, and then we’ve also got Drupal 7 sites that we’re running on the Kubernetes infrastructure. But I don’t recall the name of the module that we’re using. It’s not Flysystem because it came along only in Drupal 8.

JEFF: I know I’ve used in the past s3fs as well.

IVAN: Oh, yeah, that’s it. I think that’s what it is.

JEFF: It’s one of those things where for most sites it’ll work perfectly fine, but there’s always going to be some site that’s doing some weird thing and you’re like, you know. 

IVAN: There’s nothing like beta storage.

JEFF: There’s no good answer. When you have control over the sites that’s the best. I have control over the Pi Dramble site. So, if you look at it, you’ll see I intentionally kept that thing super simple, because even under load the Raspberry Pis can handle that. If I had commenting in accounts, and all kinds of crazy things going on, and real-time chat and who knows what else that I’ve seen on peoples’ sites, it’s going to fall over. 

IVAN: [laughing] Yeah. Well, how taxed do those Raspberry Pi 4s end up being? I’ve seen that you’re writing about the Ice Tower, the need for active cooling on the processors now. How is the cluster handling traffic? And tell me a little about Ice Tower, it sounds so cool?

JEFF: It’s funny to ask about that. Right now, I’m looking at my Mac in front of me, just doing this podcast, the CPU’s at 80% and it’s dying, and the fans are on. And it’s funny because the Pi is sitting over there, the fan's not even on because it’s not hot enough. The Raspberry Pi generally speaking, if you’re not mining bitcoin or something on it, it’s not going to use a whole lot of CPU, and it’s not going to need a whole lot of cooling power. However, when I do these projects I always try to go the extra mile and then an extra ten miles after that, and I always tax them until I can break them, basically. So, I do a lot of performance benchmarking on them, and I found that the Raspberry Pi 3B+ and the Raspberry Pi 4 both, they are not quite as good at keeping their cool.

The Raspberry Pi 4 was even worse than the Raspberry Pi 3, and it was found pretty early on that the reason for that was they had a little flaw in the way that they implemented USB. They added a new USB controller that makes it USB 3, which is awesome. It’s way faster than the old Raspberry Pis with USB 2. You can get an external hard drive and things are a lot faster connection, has gigabit ethernet, that’s awesome, because the old Pis were also constrained to 200 megabits or 100 megabits, way slower. So, it’s really awesome, but the first version of the Pi for ROM or Bootloader, it had a flaw in it that would keep the USB chip in high power mode all the time. So the Raspberry Pi 4, even if you just turned it on and left it sitting there, it would just get hotter and hotter.

If you left it out in the open—which is a bad idea for dust and things like that—if you drop something on it and you short out a circuit or something, so everybody should have a Pi in a case if you’re going to have it running for any period of time. If you have it in a case, it just sits there and turns into a little oven and it cooks itself, and it starts throttling the CPU because it gets so hot. I did an early article about this months ago right after it was released on how it gets super hot, and it’s really bad because if you put it in a case it’ll just cook itself, and that’s without much load. If you put load on it, it just starts throttling right away.

So, I said you basically have to have a fan on the thing. I still recommend having a fan on the new Pi. But they released a firmware update a few week ago that sets the USB chip into the correct mode. It very slightly reduced the performance of USB, but not really in any huge meaningful way for most peoples' usage. But it makes the Pi 8° Celsius cooler, which is hugely significant. That’s a huge difference. I posted one blog post a week or so ago on this, and I’m going to be doing another one because I tested with a couple more cases how that affects the Pi's cooling. Basically, the general thing that I’d say now is, you still need to have some method of cooling the Pis.

So, for my cluster, I have the Raspberry Pi HAT, the power over ethernet HAT that has a fan built into it, and all that fan does is blows a little bit of air onto the processor. That’s enough to keep it cool. You just need some convection over that processor. If you put it in a case, there’s no convection because the case is just going to hold that air inside of it and heat it up. If you have it open air or if you have a fan blowing on it, it will make some convection, it’ll take the air and move it over the processor and take that heat away.

So that’s all that’s really needed. But, a company called Seeed Studio, S2 Pi, they sent me this colossus heat sink, it’s not colossus when you compare it to—I don’t know if anyone ever built a Pentium 2 rig back in the day—you have those giant coolers that are the size of a computer nowadays, and a giant fan or two fans on either side of it, blowing air through it, or If you ever saw the Power Mac G5 quad core. I saw one of those one time, and the cooling system is bigger than everything else on the computer.

It’s kind of the equivalent of that for Raspberry Pi, because Raspberry Pi is a size of a credit card, and this Ice Tower is the size of the whole board, but super tall as well. If you use it, you can’t put it in a case, but it does do an unbelievably good job at cooling Raspberry Pi. It cools the Raspberry Pi so cool that it’s almost like you don’t even have it running when you have the fan on the thing. It does what it says it does, it keeps it ice cold, but at a bit of an expense you can’t use Pi HATs with it, if you use that cooler, you can’t fit it in most conventional cases.

IVAN: You can’t put it in a cluster.

JEFF: Yeah, if you were doing web hosting it wouldn’t be a bad idea to build a custom little case for these things, because that would keep the CPU so cold that it would operate a little better. It would make the Pi operate a little better.

IVAN: Does it have contact with the CPU?

JEFF: Yeah. It comes with a little thermal pad. It’s like a little piece of rubber but it’s thermal rubber and yet you just wedge it between the CPU and the cooler. It has one copper heat pipe that goes up and down, and that copper attaches to cooling fins. The cooling fins distribute the heat. And even if you run it without the fan, it’s going to keep it way cooler than just blowing the fan over a processor. It does a really good job at cooling and the fan makes it do even a better job. The difference was without the Ice Tower it was 60° Celsius, with the Ice Tower it was 30° Celsius. It was a huge, huge difference.

IVAN: Wow. That’s a giant difference. Well, it’s unfortunate you can’t use it in your whole cluster, because that little compact package you have, I would assume the Ice Tower just doesn’t allow for it.

JEFF: Yeah, luckily with the Tower since I switched to the Pi HAT, the power ethernet HAT for it, since it has those built-in fans that’s just enough to keep it from overheating. With Kubernetes running, Kubernetes does take up some CPU constantly, so it does get hotter over time, but it stays under the throttling threshold.

Another option, I actually bought one because I kept having people tell me, “You should try the Flirc case.” I’m like, Ok, whatever. I’ll buy one. So, I bought one this week and I tested it and it doesn’t do as good a job as the Ice Tower. It’s basically a giant aluminum heat sink case for the Pi. The case actually attaches to the CPU, and it does do a pretty good job cooling it. It keeps it down at 40-45° Celsius, which is still way better than just having an open air, having a little fan blowing on it. But, having a cluster of these, they’d probably still get super hot. [laughing] You got to have some sort of fan blowing the heat out otherwise it’s just going to turn into a little oven.

IVAN: Now, in addition to all the hobbies you have, side projects, and so on, you are also an author, and I would love to hear about your latest book and the book that you’ve written on Kubernetes. What are you working on right now?

JEFF: I love writing. I don’t know how many million words I’ve written in my life, on my blog and on other blogs and things, but I love writing. In 2013 or so, I think that’s when I started, I’ve always wanted to write a book my whole life, I want to write a book sometime. I think part of that was jealousy because my brother, when he was a kid, wrote a book and his book, you know, the 15 minutes of fame, his book caught fire and was a local very popular book. He sold maybe 15,000 copies or something. It was pretty cool being the little brother to the brother who wrote that cool book.

But I was also a little jealous, like, I want to do that too. But I also just love writing. I’ve always loved English and literature growing up, and I love reading and I love writing. So I put that together with the fact that Ansible didn’t have a book in 2014. I started in 2013 but in 2014 I’m like, There’s still no book for Ansible and it’s really popular.

So I decided to start writing it with a goal that I would write 100 pages and sell 200 copies. And it was funny because I started writing it on a platform called Leanpub where you can publish it while you’re writing it and sell it while you’re writing it. And by the time I had written about 40 pages I already had sold 200 copies [laughing]. And then fast forward these many years later, it’s 2019, so it’s been in print for five years now, and I now sell it on Amazon and other places and it’s called Ansible for DevOps. And that book has sold over 22,000 copies and it’s now 480 pages, including a chapter on Kubernetes and a chapter on Docker and a couple examples that do Drupal.

One of them was inspired by the Raspberry Pi Dramble cluster. So, that was my first book effort, and it went incredibly well, and I was floored. There’s no word to describe, when you’re like, I want to do this thing my whole life, and this is my goal. And then your goal is surpassed by 50 times over, and you get to meet awesome people because of it. It’s just so many cool things happened because of that book. It also helped my family.

We've wanted to remodel our kitchen and after writing the book and making some profit off of it, I was able to remodel the kitchen four years earlier than we thought we might be able to. That’s a huge change for our life, because our old kitchen was kind of hard with three kids and the way that we live our life and stuff at home, especially since I work remote. And I’m at home all the time. We had an old cramped little kitchen, and we were able to get it better.

So, the book was just awesome. I don’t expect to have the same level of success, but who knows. You never know where it’s going to lead. But I’m working on another book. I actually just finished the first chapter a few nights ago, and I have a structure for the rest of it, and I’m working on examples and chapters.

The next book is going to be called Ansible for Kubernetes, and maybe if Ansible is around in five years and there’s another game-changing cloud infrastructure thing, it’ll be Ansible for that and I’ll have a whole series out. But I’m working on that book. I haven’t published it yet. I probably will pretty soon. Even though it’s not finished, I’ll publish in progress updates on Leanpub, but both of those books, if you go to ansiblefordevops.com or ansibleforkubernetes.com, those are the book sites. I love writing them. And one of the best things about writing them in progress is for both books I’ve had a lot of interaction with the people who read it, and they can help me. If they’re interested in something, I can write about that. Or if they are like, "Your example didn’t work on my computer," I can improve it before I actually make a published printed version that people will buy.

IVAN: I very much appreciate knowing about Ansible for Kubernetes. I didn’t know that was what you were working on. I think we actually bought Ansible for DevOps if I’m not mistaken.

JEFF: Thank you.

IVAN: When Tess first started working for us, we wanted to make a big change to how we were doing things at TEN7 and the custom scripts we had, and we wanted more automation. And Tess was very interested in Ansible being the thing, and we had to learn it. So, you’ve certainly provided a great deal of information and helped TEN7 in that manner as well. We wish you all the best for your new book Ansible for Kubernetes, and we’ll be linking to it from the show notes on the podcast episode as well.

JEFF: That’s great. Thank you.

IVAN: You’re very welcome. Well, I think that’s a wrap. We should say goodbye, and thank you so much for being on the show, it’s been a great pleasure talking to you. I hope to talk to you again soon. And I’m going to go to Micro Center right now on a number of different visits, because you can’t buy more than one at a time and buy some more Raspberry Pis, so I have more than you do. [laughing] I don’t know what I’m going to do with them thought. [laughing]

JEFF: You can always get a bunch of Raspberry Pi Zeros and stick them in peoples stocking stuffers, that kind of thing.

IVAN: Oh, that’s a good idea. Stocking stuffer. Five-dollar Raspberry Pi Zero. Awesome. Thank you so much for spending your time with me today. It’s been great.

Author and software developer Jeff Geerling was my guest today, and you can find him across the web using the handle @geerlingguy. You can also find him online at jeffgeerling.com and of course the Raspberry Pi Dramble is at pidramble.com. If you’re interested in either of those books you should go to ansiblefordevops.com or ansibleforkubernetes.com.

You’ve been listening to the TEN7 podcast. Find us online at ten7.com/podcast. And if you have a second, do send us a message. We love hearing from you. Our email address is [email protected]. Until next time, this is Ivan Stegic. Thank you for listening.

Nov 14 2019
Nov 14

As part of being a distributed company, we've found that using Know Your Team on a regular basis helps us communicate outside of projects and tasks and deadlines. Every Friday we respond to the question “What’s something you figured out this week?” and here we’re collating our best of.

NUMBER 2–OCTOBER 2019

Chris

I learned how to work with json arrays to store and manage data efficiently in a database field.

Jason

I learned about Event Subscribers and how to intercept every request and do stuff to change responses that are sent for certain requests.

Jill

CSS Scroll Snap controls the location on a page the user scrolls to. This is helpful if a page has defined sections. The ten7.com site is an example of a site containing pages with sections where scroll snap may improve the UI.

<div class="page">
  <div class="snap"></div>
  <div class="snap"></div>
</div>

.page {
  scroll-snap-type: y;
}
.snap {
  scroll-snap-align: start;
}

Les

When debugging email events in SendGrid [our cloud email provider], it's difficult to get the Activity log to show you exactly what happened when. It's easier to export the CSV version of the log and go through it that way. Also, SendGrid's knowledge base didn't tell me anything about cancelling deferred emails. But their chat support was prompt, fast, and helpful.

Lex

This week I figured out that Apache Solr configuration in Flight Deck was causing my local environment to be extremely slow. I had been dealing with the slowness for quite some time and thought it was just Docker on Mac, but it turns out we needed to make adjustments!

Oct 31 2019
Oct 31

 

St. Catherine University

St. Kate’s Website Redesign: A Fresh Site and an Empowered Team

St. Catherine University (known as St. Kate’s) is a private Catholic liberal arts university in St. Paul and Minneapolis, Minnesota. St. Kate’s was one of the first colleges specifically for women in the Midwest but today also welcomes men into graduate and associate programs. It has a student population of around 5,000 annually and welcomes students of all ethnicities and ages.

St. Kate’s website is their biggest marketing tool, and it needs to give site visitors—from prospective students to internal St Kate’s workers—a great user experience on desktop and increasingly, on mobile.

One of the biggest issues with the website was that it was effectively two sites, one internal (Kateway) and one external. People didn’t always know where to go to find information, and information duplicated in both sites could get out of sync. Beyond that, the search functionality of the site was poor—when users couldn’t find what they needed, they resorted to using Google to search the site, or started helpdesk tickets.

But bigger process issues lurked behind the scenes. The site was built on ModX, a content management system that was not supported internally. This required working with an outside vendor when more than simple content updates were needed. In addition, any content updates, regardless of department, had to be approved and made by the Marketing & Communications department, who already had their hands full with their own work, so there was often a backlog of requests.

Our Assignment: Build a New Website and a Self-Sufficient Dev Team

The site redesign had several high-level goals: create a new site in Drupal 8 from the ground up that was more flexible and able to be supported in-house, combine the internal and external sites to one site with a single sign on, and refresh the site design and content to better support more robust marketing efforts. Once the site was supported in-house, it would be easier to implement another goal: share ownership of the site content across campus, instead of under Marketing & Communications. 

We spent a great deal of time on the content audit and content strategy on this project, in order to know what content to keep and how to organize old and new content effectively. This process can seem long if you’re not used to it, but it’s absolutely essential in a site redesign, where we need to tailor content to specific audiences who all need to be communicated to in different ways. St. Kate’s really did their due diligence to gather information and ensure all voices were heard; they sent out surveys, did presentations, met with folks. We performed user interviews with all levels of stakeholders and users, from members of the web advisory group to department heads, teachers and students. 

We believe in getting stakeholders (IT, marketing and business units) involved as early as possible in the review process, so as soon as we’d constructed the new site’s navigation, we tested it with both internal and external audiences and made changes based on those results.

We made the following changes to the St. Kate’s site:

  • CMS & Infrastructure: We moved the site hosting to Pantheon, a development platform tailored for Drupal, which also supports Apache Solr, the solution we implemented to improve St. Kate’s search functionality. We also liked that Pantheon’s workflow enforces the development > test > live workflow.
  • One Site, Home for All Content: Instead of the old internal site and the old external site being separate, there is now one integrated site with a single sign-on. All content can be reached from the home page.
  • Content Migration and Streamlining: Most of the content had to be recreated manually on the new site, because there wasn’t an easy way to get most of the data out of the old site (we did manage to get news articles out). We consolidated information where possible. For example, there used to be five mission and impact pages, now there is just one, and we streamlined the news and events content together.
  • Content Types: We implemented Drupal paragraphs to enter content, and whittled the number of content types from 60-80 on the old site to a more manageable 10-15.
  • Improved Search: We implemented Apache Solr, which allows us to index content in different ways, and to assign higher weight to different fields of data. For example, we can give the TITLE field a higher weight, so if someone searches for “admissions,” the Admissions page is likely to be served up higher in search results.
  • Integration Work: We integrated many external systems into the site. For example, faculty content used to be hosted on the old site, but now there’s BEPRESS integration that uses the Drupal Feeds module to bring data on to the site. We also integrated BANNER, the Student Information System (SIS), D2DL (Desire to Learn), and SOPHIA (library software).

Empowering Developers

When St. Kate’s was looking for someone to do the redesign, they were really looking for a partner, a vendor who would collaborate with the in-house dev team, and help them become self-sufficient in supporting the site after the project was complete. “We didn’t have any dev process. Nobody understood how ModX worked,” said St. Kate's Austin Allman, Senior Programmer Analyst. “We didn't have any practices or prejudices. We were very malleable to TEN7’s practices.” We got them used to the Agile software process, including the use of version control, JIRA issue tracking, and GitFlow to test their feature branches before going into production.

“Any major site like this has a major learning curve, especially if you’re dropping a new platform on them,” said Les Lim, TEN7 Technical Project Manager. “We tried to be as transparent as possible with what was being built in Drupal 8, so the St. Kate’s dev team could see things evolve over time, see the work in progress. That’s an instructive thing, more so than just seeing the finished product.”

Outcome: A Fresh Website and Positive Intangible Changes

The new website launched in the summer of 2019, and the feedback has been very positive. People say that the site is so much easier to navigate, all the information is in one place, and they like the visuals. There is now site content for every audience, accessible with their own set of links. The number of help desk tickets looking for information has gone done significantly, meaning people are finding what they need.

Departments can now make content changes to their own site area, freeing the Marketing & Communications Department to focus on their core responsibility, promotion of the University, and raising money through donor programs.

The website redesign process has also changed their dev team. “I’ve seen this dev team go from group of people who were scared to death to make a decision to seeing leadership come out,” said Lisa Gariepy, Information Technology Consultant with St. Kate’s. “It’s a whole new mindset of being able to see a problem, and know that they should take action, and know they have the power to do that—they don’t have to send triplicate emails and wait for a thumbs up. It was helpful for them to keep hearing Les and Jason [Cote, TEN7 Front End Developer] saying ‘LET’S JUST DO IT,’ and accepting that sometimes they’ll make the wrong choice. And if that happens, all we have to do is fix it!”

This has also changed the dynamic between the dev team and the user base. “Before, when users would ask for changes, we would have said ‘Nope we can’t do it,’ because we didn't know how to do it,” said Austin. “Now, users are more emboldened to ask for changes, because we know we can we make a change.” 

Oct 21 2019
Oct 21

As part of being a distributed company, we've found that using Know Your Team on a regular basis helps us communicate outside of projects and tasks and deadlines. It periodically sends us questions and on Fridays at noon it asks, “What’s something you figured out this week?” This gently reminds us to think about the week that just happened, and to reflect on what we’ve learned.

People share everything from a life hack anyone can use to a deep concept in Drupal that only a handful of people will appreciate. We're planning on sharing the best of what we've learned with you.

Number 1—September, 2019

Charlene

I was going to rent a Uhaul cargo van to pick up the boxes I shipped cross country. Unfortunately the boxes were 50 miles away, and Uhaul charges $19.95 + .59 cents a miles = $78.90. I discovered you can rent cargo vans from Enterprise, and it’s cheaper: $19.95/day and 100 miles FREE! I saved myself $58.95!

Chris

A cool thing I learned this week is the power of using usort to sort a given array by a non-intrinsic arbitrary order: Let's say you have a data source that provides a list of items by date shorthand M, T, W, Th, F but the list is not always in that order. Sometimes it's alphabetical. What you do is create template representing your arbitrary order $template = ['M','T','W','Th','F']; And the order provided by your data source is $source_array = ['T','Th','W','F','M'];

Enter the usort method. Usort uses a binary stepping method where it compares each item to the next one one by one until everything is evaluated and properly positioned. Any item in the $source that isn't in the $template will end up being at the beginning.

asort($source_array, function($a, $b) use ($template)
   {
      //get the position of the first comparison item in the template array
      $pos_a = array_search($a, $template);
      //get the position of the second comparison item 
      $pos_b = array_earch($b, $template);
      //are we positive or negative and order accordingly.      
      return $pos_a - $pos_b; 
});
return $source_array;

Jason

  • I learned that you probably want array_shift instead of array_pop. I mean, when are you ever taking the last thing and not the first? (Obviously, sometimes. But not very often.)
  • I really miss D8 when forced to work on a D7 site.
  • I built a custom configuration page so client could add custom text to a thing. Then you just get that custom text by using \Drupal::config('thing')->get('fieldname'). Super easy.
  • When you create a new service you use the Drupal service registry to access your class, you don't instantiate a new instance of the class. Which, I'm pretty sure, is the whole point of registering a service.
  • Creating a custom block plugin is super easy. And you can even get started using Drupal console. drupal generate:plugin:block

Jill

FOUC, Flash of Unstyled Content, is when a web page loads without styles for less than a second. One possible method of correcting this is by placing scripts at the bottom of the page.

Les

  • You can change the default local IP address blocks that Docker Compose assigns to your projects, which is useful to avoid conflicts with other services that might want to use those same IPs. https://serverfault.com/questions/916941/configuring-docker-to-not-use-t...
  • Confluence has an undocumented bulk-issue creation mechanism, if you are right-click-creating from a basic table.

Tess

Ansible does not treat localhost as special, but rather like any other fully qualified domain. Furthermore, when a host is in multiple groups in an inventory, they groups are not applied separately, but together as a merged item. To combine inventories while keeping localhostspan> separate, the trick is to create a fake mysite.localhost inventory item, with ansible_connection=local to skip any domain lookups.

Oct 11 2019
Oct 11

We recently completed a special data integration project for the University of Minnesota, Office of the Executive Vice President and Provost, Faculty and Academic Affairs. Faculty Affairs, as they are referred to, uses a product called Activity Insight from Digital Measures (referred to internally as “Works”) to capture and organize faculty information. Works acts as both a departmental LinkedIn and their internal performance review tool. They needed help getting faculty data out of Works in order to display it in web profiles on departmental websites. Many universities use Digital Measures to manage their faculty information, so when we took on the project, we wanted to be able to open source the solution we came up with and make the code available to those universities who wanted to do similar projects. Luckily, Faculty Affairs was a willing partner in working towards that goal!

The Digital Measures Migration

Once the project was kicked off, our DevOps Engineer Tess Flynn took a peek under the covers. “Digital Measures posed a significant challenge to open sourcing anything,” said Tess, “because it doesn’t really have a set data schema. It’s a REST-based application for storing structured data. It’s not as specific as a database; it’s somewhere between a document store and a database. All Digital Measures lets you do is say: this is a data object that we want to store, and these are the fields it can have. The University creates the structure, and the data can then be accessed through an API.”

For this project, we created one module specific to Digital Measures (Digitalmeasures Migrate) to access the Digital Measures API from within Drupal. This, in turn, allows data extracted from the API to be leveraged through Drupal 8 migrations. Due to the volume of data, we write it to a “staging” database table. This way, we have more flexibility in processing it, without bringing down the Digital Measures API, or encountering rate limitations. Once we get the XML out of the API and store it in the database, then we can do the transform part. That’s where the bulk of the work is done.

The other seven modules (process plugins) solve specific migration problems in the transform step of the Extract-Transform-Load (ETL) process. We wrote a series of Drupal migrations for Faculty Affairs which relied on these plugins.

The Framework

Here are the modules we created and contributed to Drupal.org:

  • Digitalmeasures Migrate
    Provides a method to access Digital Measures API through Drupal.
  • Migrate Process XML
    One of the most-used modules when writing migrations for Faculty Affairs. It reads XML and allows you to extract particular key sections using XPath.
  • Migrate Process S3
    This is the second-most useful module. It allows you to download objects from an S3 bucket as a file to your Drupal site. For Faculty Affairs, this was used to download a professor’s profile photo, resume/CV, and other attached files. Once downloaded, this is then attached to the professor’s profile so that direct linking to S3 is not required.
  • Migrate Process URL
    Allows you to manipulate URL values that are provided within the data.
  • Migrate Process Regex
    While a small module, it’s useful in a surprising number of cases! This module provides a way to use Regular Expressions in a Drupal migration. This allows for matching and text replacements where XPath alone wouldn’t be enough. “This is a tiny thing, but it’s really handy in migrations,” said Tess. “For example, when you have two quotes next to each other that should have been one quote.”
  • Migrate Process Vardump
    Often used for debugging, this module takes any data given to it and dumps it to the terminal output and then passes it on. “It doesn’t sound very useful, unless you’re writing migrations, but then it’s the best thing EVER!” joked Tess. Every time Tess has written migrations, she said she had to write a custom version of this. So having this as a standard module will help many users. This module currently has the most users on Drupal.org out of all the modules in the framework.
  • Migrate Process Skip
    When processing lists of data, the migration system has very specific ideas of what’s considered “empty” or not: zero = false, empty string, and NULL, which of these means ”empty”? Drupal has a “skip_on_empty” migration process plugin, but it may not give the desired result if you don’t know what you’re doing. This module provides a few different mechanisms to define what is “empty” and should be skipped.
  • Migrate Process Trim
    Often, we need to remove leading or trailing characters—such as spaces—after extraction. This module provides a quick and simple means to achieve this in a Drupal migration.

We Open Sourced the Tools, Not the Solution

“Originally I really wanted to open source a solution, but it didn’t end up working that way,” remarked Tess. While multiple colleges and departments within the University of Minnesota (and outside of it) are all using Digital Measures, they all have their own customized implementation of it. Because of this, there will always be custom work required to display the Digital Measures data in each department’s web profiles. So the best you can do is to release tools to get to the solution. The framework we created will give each department’s IT team a head start on the process. “The tools we created are like socket wrenches to work on a car,” said Tess. “Every car is different, but the socket wrenches make the work easier.”

If we’d open sourced the Faculty Affairs solution, it would have been specific to them, and not transferable to another school. But what we can open-source are the tools to create a solution.

“There are mechanisms to retrieve types of data, but the types of data are up to the user of the API to decide,” Tess continued. “Faculty Affairs created the schema and data formats. All we did was used their existing forms, and tweaked it here and there as we needed to, and then wrote tools to create the necessary things to import data from Digital Measures into Drupal. Digital Measures is so customizable, you can’t do anything but release tools.”

This framework has broader applications than just Digital Measures. “Each module solves a specific issue that can be used for Digital Measures.” said Tess, “But the specific problems they solve are common to migration processes in general. With the exception of Digitalmeasures Migrate, these modules are fairly generic and can also be used for a variety of other migrations.”

If you want to use the framework yourself on a Digital Measures migration, the place to start is with the Digitalmeasures Migrate module. “That gets you the data,” said Tess, “but it doesn’t necessarily put it into a form that you can immediately use in Drupal, because unfortunately there’s no way of telling how the data is structured, so it’s up to you to make that part.” Tess did include a rough process for using the framework in the README for the Digitalmeasures Migrate module.

Got a Data Migration Problem? We Can Solve It!

Are you using Digital Measures and wishing you could pull the data into your Drupal deployment? Do you have any other Drupal problem we can help you solve? Drop us a line!  

Oct 09 2019
Oct 09

Your browser does not support the audio element. TEN7-Podcast-Ep-072-Joe-Shindelar-a-Passion-for-Open-Source.mp3

Summary

Podcast guest Joe Shindelar of Osio Labs chats with Ivan about making interactive sculptures, snowboarding and his long history with open source software.

Guest

Joe Shindelar, Osio Labs  

Highlights

  • First tech memories
  • Kids today! They don’t know how hard we had it with technology
  • How debugging software is akin to interactive sculpture
  • Working at Lullabot
  • Drupalize.Me
  • Lullabot Education becomes Osio Labs
  • Snowboarding instructor at Blizzard
  • Snowboarding vs. skiing
  • How not to pick an online handle

Links

Transcript

IVAN STEGIC: Hey Everyone! You’re listening to the TEN7 podcast, where we get together every fortnight, and sometimes more often to talk about technology, business and the humans in it. I am your host Ivan Stegic. My guest today is Joe Shindelar, a lead trainer and lead developer at Osio Labs, whose mission is to empower anyone to build websites using open source tools. Joe is passionate about open source technology and has a rich and interesting past, having started out at a small agency here in town called Triangle Park Creative, moved on to Lullabot, Drupalize.Me and now Osio. He’s also a snowboard instructor, teaching kids how to snowboard. I have lots of questions about pronunciations as well. So, good morning Joe. Welcome. It’s so nice to have you on the podcast. 

JOE SHINDELAR: Good morning. So, pronunciation. My last name is Shindelar, so close, but not quite. Then Osio, we say Osio (pronouncing "OH-SEE-OH") Labs. It’s technically a made-up word, so I don’t know that there’s necessarily a correct pronunciation, but we all say Osio Labs. It stands for Open Source Inside and Out.

IVAN: So, it’s actually an acronym. I love that that’s the name. I think the reason I said Osio (pronouncing "AH-SEE-OH") is because the I kind of looked like an l to me and I must’ve thought Oslo.

JOE: Sure.

IVAN: Osio. Okay. Open Source Inside and Out. I love it. And, I actually was going to say "Shind-e-lar," but I thought I heard someone say your name "Shindler," without the e, and I thought, Oh, that must be the way you say it.

JOE: There’s a very good chance that you have heard someone say it that way. It’s a common thing. For the most part, unless I need to, I don’t even bother correcting people, because I know that they are talking to me, and it’s fine.

IVAN: I feel the same way. What’s the etymology of the last name?

JOE: It’s Czech.

IVAN: Wow, cool. So, let’s see—you’re lead trainer, lead developer at Osio Labs, but I want to go back a little further to you being at Osio, and I kind of want to figure out where life started for you, and how you arrived where you are today. So, where did you grow up?

JOE: I grew up in Stillwater, Minnesota. So, not too far from where I live now in Minneapolis. Maybe you’re familiar with the area. I grew up just north of downtown Stillwater in a house, kind of right on the riverbanks, with a vacant lot across the street. Spent a lot of time running around by the St. Croix River and playing in the woods outside in Stillwater.

IVAN: Wow. That must’ve been absolutely beautiful there.

JOE: It was awesome. I loved growing up there. I also loved that when I was 14 or 15, we moved to the cities and there were things to do. So, I was like, I want to go to the movies or an arcade, or any of those things, and there just wasn’t a lot of that to do in Stillwater.

IVAN: Did you end up at Stillwater High School?

JOE: I went to Stillwater High School for half a year, and halfway through my first year there, we moved to Minneapolis.

IVAN: Was that in the nineties or a little sooner than that?

JOE: Yeah, in the nineties. We moved in 1998 or so.

IVAN: So, you’re kind of a kid of the nineties?

JOE: Yeah.

IVAN: What’s your first memory of technology and the internet?

JOE: I kind of vaguely remember the first computer that we had at my house. My Dad worked at the Science Museum in Minnesota at the time, and they had gotten a bunch of Mac computers for the exhibits department to experiment with. They didn’t really know what they were going to do with them, whatever, my Dad got to bring one home.

So, he would bring it to our house on the weekend and we’d set it up in the living room. And kind of like today, you might set up a TV or something in the living room, and the whole family gathers around to watch a movie. My Dad would set this Mac up and me and my sisters would all gather around and watch my Dad play games. Mostly he would play this game called Crystal Quest, which every once in a while I try to find a copy of so I can play it now. But, it’s basically like little blobs floating around on the screen, collecting crystals.

So, I remember that part of it, and then I remember my Mom really got into playing Tetris. I remember this rivalry between my Mom and our babysitter at the time. Greta would play Tetris and get the high score. Then my Mom would come home from work, and she’d sit down at the computer and immediately start playing Tetris in order to beat Greta’s high score. Then the next day Greta would show up to take care of us, and she’d be like, “All right, you guys can go outside and play.” And she’d sit down and start playing Tetris to try and beat the high score. This went on forever. And that kind of stuck with me in some ways. I played a lot of Tetris as a kid, and I still do occasionally now.

IVAN: Did you ever try to beat your Mom’s score?

JOE: I’m sure I tried to. I don’t really recall how that all played out. I imagine today I would probably win, because I don’t think she plays Tetris much anymore.

IVAN: [laughing] You could beat your Mom now. Okay. Nice, Joe. Mom, if you’re listening, I think that’s a challenge. That’s great. And so, that was the family computer. I guess that wasn’t connected to the internet in any way right?

JOE: No, it wasn’t. My first memories of being connected to the internet are later. We had another Mac at the house, and I remember getting connected to AOL Instant Messenger. Specifically my cousins had come to visit, and they had AOL, and I think we were able to dial into their account while they were at my house. And I was like, This is awesome. So then when they left, we had one of those free month trial discs, and we signed up.

I remember spending a lot of time going into the AOL chatrooms and chatting with my cousins. That’s my earliest memory of doing things on the internet, and eventually that led to looking at websites and such. But mostly I just remember going into random chatrooms and saying hi and trying to figure out who else was in the chatroom with you. In retrospect I’m like, That is awkward and terrifying. Like, I wouldn’t do that now.

IVAN: I feel the same way. Did you have one of those external US Robotics modems?

JOE: Yeah, we totally did.

IVAN: And you could tell when it was connecting. I would always have trouble getting it to connect. Boy, memories, huh? Geez. That was awesome. But you learn so much about how the internet works having those issues, because you’d really have to try and figure out what the heck’s going on.

JOE: Totally. In comparison to now, where it’s like, you connect to Wi-Fi and maybe it goes down every once in a while, but rarely. It’s just sort of like every device you have is connected all of the time and there’s none of this, I have to turn on the internet and then wait for it to connect and hope that my sister doesn’t try and make a phone call at the same time. It’s amazing how much this stuff has changed.

IVAN: It is. It’s really amazing. I remember my parents getting really mad at how much time I was spending on the internet, but mostly because local calls in South Africa were not free. So, internet service you’d have to pay for, but to dial in you’d have to pay for the time that you were on the landline as well. And, the first month’s bill, like anyone with the original iPhone knows, right? You use all that data that you don’t know that you’re using. [laughing]

JOE: [laughing] Right.

IVAN: Yeah. That was bad.

JOE: How could I possibly use more than 1 gigabyte of data? Then you’re like, Oh, apparently I can do that in a day.

IVAN: [laughing] That’s awesome. So, Stillwater High and then what happened after that? What did you do for education after that?

JOE: When we moved to Minneapolis, I went to South High School for about half a year. A kind of a funny story there, where, we moved from Stillwater to Minneapolis in the middle of the school year, and Stillwater High School operates their school year on quarters, and South High School operates on trimesters. So, we moved at the end of the quarter from Stillwater, but I couldn’t get into South High until the beginning of the second trimester. So, there was this almost one-month period of time where, as like a 16-year-old boy, I didn’t have to go to school in the middle of my school year.

Which was amazing, except for in order to make this all work, my Mom had orchestrated this deal with South High School that was like, “All right, we’ll give him credit for the full year even though he’s missing a month. But in order to do that, he needs to spend a bunch of time doing some research and writing a report while he’s on this break.” So, I opted to write a report about U-boats, and so I did all this research on German U-boats and wrote a report, made paintings of U-boats and collected all of this and put it together. Gave it to my Mom so she could hand it into the school, started school, went on from there.

Like 10 years later, I learned that my Mom made this whole thing up. There was no requirement to write the report. [laughing] My Mom was like, “Well I can’t have Joe and his sister just sitting around doing nothing for three weeks, so I guess I’ll make them write reports.” And, she dug it out of some box in the basement and was totally like, “Oh, yeah. That’s right. This thing.” I’m like, “Are you kidding me?” [laughing]

IVAN: [laughing] That’s amazing. So, you ended up studying art in high school?

JOE: I did. There’s a school here in Minnesota that’s called the Perpich Center for Arts Education.

IVAN: Oh, I know that school. That’s a good school.

JOE: Yeah. So, I went to Perpich for two years. I was in the media program there, studying media arts. It ended up being really fascinating for me, because about the time that I was there, like 1999/2000, the program was transitioning from a lot of analog media stuff. So, photos in a dark room, creating video using VHS and two editing decks, and transitioning to do a lot more. The school had just gotten its first digital cameras, and first digital video recorders, and there was an iMac that had the very first version of Final Cut Pro on it. So, it was like, my first year there I did a ton of stuff in the dark room, and the second year there, all of a sudden, we were using Photoshop and computers, and I was like, This is cool. This is what I want to do.

IVAN: And so, you knew you wanted to be involved in computers right away in high school. So, what was the next step? I know that you went and spent some time in Rhode Island.

JOE: When I graduated high school, my initial plan was to move to New York City and become an ActionScript developer and make millions of dollars writing Flash applications.

IVAN: Of course. I mean, yeah, Flash, absolutely. [laughing]

JOE: [laughing] But, that didn’t pan out. I never actually left for New York, and I decided I should go to college instead. Then I bounced around a bunch. I went to a handful of different schools because I wasn’t entirely sure what I wanted to do. I started out as a computer science major at University of Wisconsin Stout. So, I did that for a semester, and then decided This is way too hard and I’m not good at it, so then I switched to math and I was going to major in math, then I decided I didn’t like that either.

So after a semester I dropped out, and didn’t go to school at all for about a year. I followed a girl out to Rhode Island, and I ended up going to the community college out there and taking a bunch of art classes, and, kind of falling in love with art again after having not done it for a few years. Then I moved back to Minneapolis and finally about eight years later, graduated from the University of Minnesota with a degree in Bachelor of Fine Art with a focus in sculpture.

IVAN: Wow. A BFA with a focus in sculpture.

JOE: Yep. [laughing]

IVAN: That’s insane. And it’s so interesting, all the people we’ve had on the podcast, all of the different paths that lead to Drupal.

JOE: Yeah, totally.

IVAN: And, it seems like a lot of them go through the fine arts degree of some sort. You know, journalism. So, sculpture eventually turns into Drupal, or was Drupal happening at the same time you were at the University? Because I know there’s all these overlaps as well.

JOE: I was starting to do Drupal at the same time I was going to school. I had a job working for a company here in town, Triangle Park Creative. I was working there pretty much full time while also going to school. And so, I was learning to build websites on the job at Triangle Park, and at school I was studying sculpture and in a lot of ways, trying to figure out ways to incorporate technology into the sculptures that I was creating. So, microprocessors and sensors and that kind of stuff.

IVAN: That was hard so long ago, right? Things weren’t small.

JOE: Oh my gosh. I was talking to somebody about this the other day, where, not that it’s by any means easy to do this stuff today, but, there is a plethora of different microchip boards you can buy, like Arduino and Raspberry Pi and all of these kits that you can get with sensors, and assemble them together, relatively easy in comparison to even like 10 years ago. You could do all that stuff back then, but you had to do a lot more of the behind-the-scenes work to flash your program onto the memory of the board and get it to run and solder things together.

All of these things that, as an art student, I was like, I have no idea how to do this, but what I really want is for this light to turn on when somebody walks past my sculpture. And, so, I spent a lot of time tinkering with hardware stuff like that in order to incorporate it into the art that I was making. I had this idea that I wanted to make sculptures that would evolve by virtue of the fact that someone was viewing or interacting with the piece. So, you go and see the sculpture and instead of it just being a piece of concrete, or something that never changed…

IVAN: Like a sculpture that’s static of David, right?

JOE: That the fact that you had been there to visit that sculpture would reflect in the art for the next person that saw it. That was the dream. I would say 80% of the projects that I worked on were like duct tape and bubble gum and the fact that it held together long enough for a critique was always the highlight of the sculpture. I was like, Wow, it didn’t fall apart. [laughing]

But, I had a lot of fun doing that. I learned a ton about computers and programming and problem solving and debugging. Just things I think about a lot today, even working on writing PHP for a Drupal module, and you’re trying to figure out why a particular aspect of it isn’t working. That the process that you go through to debug it isn’t all that different than trying to figure out why your mechanical switch doesn’t actuate when somebody walks by. And, initially you’re like, Those are two totally different concepts, Joe. It’s like, Yeah, but the debugging part of it is roughly the same, trying to isolate what the problem is, and figure out what’s causing it, and see if you could replicate the problem, and so on.

IVAN: Are you still making art these days?

JOE: Not really. Not in any kind of professional capacity. I have two little kids. I do a lot of coloring and painting with them, but I wouldn’t call it art, per se.

IVAN: Yeah, well I kind of miss doing that as well. It’s definitely something that rejuvenates and, kind of, re-energizes oneself when you do it.

JOE: So, then I was also working at Triangle Park, building websites. And initially the goal there was I needed a way to pay for school. I had a job and more and more people kept asking us to build them websites, and I kept getting paid to do it. So, I thought, Well, I’ll keep doing this until I grow up and get a real job. [laughing] And here I am, 15 plus years later, and I still make websites.

IVAN: Still waiting to grow up. [laughing]

JOE: Right. I found Drupal through all that initially. I think what a lot of us have done, and have been doing this for a long time, you went through the process of trying to craft your own website, and maybe your own content management system, and in my case, learning PHP along the way. I would go and look at the code and things like WordPress and Drupal. Mostly I would copy and paste wholesale things from Drupal into my code, and then eventually I was like, This is kind of dumb. I should probably just pick one of these and use it. And I did. For various reasons it meshed well with me. It solved the problems that I had at the time and it continued to evolve, and here I am.

IVAN: It continues to pay the bills.

JOE: Like, 15 years later, instead of being a starving artist, I build websites with Drupal and teach other people how to do it.

IVAN: It’s a good story. So, eventually you left Triangle Park and started with Lullabot, a completely distributed company.

JOE: Yeah.

IVAN: What was that like? Why the change?

JOE: I was looking for an opportunity to work with other people who were doing Drupal. At Triangle Park at the time, there was just a couple of us, and we were doing Drupal, but mostly we were doing it in kind of isolation within Triangle Park, and I was interested in figuring out ways to participate more in the larger community. I was also looking to have an opportunity to work with people who were better at this than I was.

At Triangle Park, I think we all kind of grew up and evolved together. It was an awesome opportunity to learn, but you were never working with someone who had been doing this for a few years longer than you had. I was looking for some of that. Then the opportunity at Lullabot—I wasn’t specifically leaving Triangle Park to go to Lullabot—it was more one of those scenarios where I was in a fortunate position to be able to say If I don’t quit here, I’m never going to put the work into finding what’s next.

So, I left Triangle Park and sort of had some ideas of what I wanted to do next, but nothing concrete lined up. And the opportunity to do some contract work as a trainer for Lullabot came up, so, I jumped on that. I applied for the position and didn’t hear anything back for three months, and I was like, Well, I guess that didn’t go anywhere. My recollection is that I didn’t even get an email in response to the fact that I had applied. Like, “Hey, thanks for your application.” It was just like radio silence. Then one Thursday afternoon I got an email and they were like, “Hey are you still interested in helping do some training? Can you be in New York tomorrow?” [laughing] I was like, “Yes, I can. Sure.”

IVAN: Wow.

JOE: Again, I was in a fortunate position to be able to make that work, and it did. I started doing a bunch of contract work for them, then more and more over the course of about half a year, and eventually that got to the point where I just said, “Look, I’m working for you two and a half, three weeks out of the month, and it’s making it hard for me to find any other work. I need to either come on full time or I need to go do something else.” So, they brought me on full time. Then, I was working from home full time which, I know you know this is, it’s quite an adventure and a transition to make.

IVAN: It is. It’s been the best adventure and transition for us over the last two years, when we started doing it two years ago. So many good things about doing it. For me it's just the ability to see my kids and my family as much as I can. It gets a little long toothed sometimes in the summer when the teenagers are all around, but otherwise it’s just amazing. So, yeah, it’s really great. I didn’t realize that you jumped into work with Lullabot directly as a trainer. I just thought that that kind of evolved out of your being a developer at Lullabot, but actually not.

JOE: It’s the other way around.

IVAN: The other way around. Wow.

JOE: I started out doing training. So, back in the day, Lullabot used to do a lot of in-person training workshops. We would do trainings at DrupalCon and Camps, but we had also organized and hosted events in various different cities. We would say, “Hey, we’re going to be in Portland this week, for these two days, and we’re going to teach theming. Then we’re going to be in Boston for these couple of days and we’re going to teach advanced module development, and if you want to attend, here’s the cost. Come to the workshop.”

In addition to that, we were also doing those same workshops, privately for clients. At the time that I came on that was really taking off as a business opportunity for Lullabot, and they needed to have additional help to do the training, because they just had so much of it going on at the time. I started doing that with them, and for me, that evolved into—I said earlier that eventually I kind of had to play my card and say, “Look, I need a full-time gig, or I need to move on,” and they brought me on full time. And then it was like, “Well, now what do you do?” The answer was I kept doing a lot of training, and then when I wasn’t working on the training stuff, I mostly worked on internal projects. One of those was what eventually evolved to become the Drupalize.Me website. In addition to doing the in-person training, Lullabot produced a bunch of DVDs for training.

IVAN: I heard about this.

JOE: Yeah.

IVAN: Yes, I’ve heard this story from someone, who shall remain nameless at this time. [laughing]

JOE: [laughing] We made a bunch of DVDs, and they were awesome and then everyone was like, “Why would we want DVDs when you could just have your training on the internet?”

IVAN: Right.

JOE: I think they did fairly well for a year, and then I was like YouTube came around and people were like, “You could put video on the internet. Why would I buy a DVD?” So, Drupalize.Me kind of started out as a, “Well, let’s take the content of the DVDs and put it online, and charge people that way.” Which is pretty much what we did. Very early iterations of Drupalize.Me, when you would go to watch a video of our training, you would sit down and it would be like, “Here’s a four-and-a-half-hour video that you could watch on how to build a module.” [laughing] We literally ripped the DVDs and put them on the internet.

IVAN: Were there any FBI warnings on those DVDs? [laughing]

JOE: [laughing] I don’t recall. There are still a bunch of them in existence. We could probably find out. When you make DVDs to sell, you have to get a certain quantity of them printed in order for it to be economic, and so we printed out thousands of these DVDs and then sold hundreds, and there are now still hundreds of them in boxes.

IVAN: Like in Jeff’s basement, or in Matt’s basement somewhere.

JOE: I don’t actually know if this is true anymore, but for a while, they were in storage. We had worked with this drop ship-type company where they store the DVDs for you, and you send them a message saying "Ship these three to Ivan." The contract that we had with them sort of stipulated that if we wanted to cancel our contract and get our remaining inventory back from them, it was going to be way more expensive than it was to just leave the DVDs there, and keep paying the monthly fee. So, for a long period of time those DVDs just remained in storage, and I guess you could call them, and they would ship one, but I’m not sure if they’re still there or not.

IVAN: I will ask Jeff the next time I talk to him. [laughing] What’s going on with that? So, Drupalize.Me then became quite successful, and then at some point it split off and it was no longer a part of Lullabot. And it was kind of run by, I think I remember, was it called Lullabot Education?

JOE: Yeah, that’s correct. So, within Lullabot we had kind of the education department, and we were responsible for Drupalize.Me and for all of the client training and in-person workshops and stuff that Lullabot was doing. Over time that business changed a bit. We were doing less and less in-person training and more and more working on Druaplize.me.

For the most part, other than myself, who was kind of full-time in that department, people at Lullabot would work on Drupalize.Me in between client projects. Which, of course, is like also synonymous with never. [laughing] But the idea was when you’re not working on client projects, you’ll have time to help work on Drupalize.Me. It meant that Drupalize.Me, while it was showing signs of being able to be really successful, was struggling a bit because it just couldn’t get the attention that it needed to get over that next hurdle.

IVAN: When it’s a services company running a product within your services business. Right?

JOE: Exactly. There’s this idea that your service is kind of ebb and flow and in between you’ll be able to work on a product, but a product needs constant attention. You can’t just work on it now and then. Especially something like Drupalize.Me where ultimately what you’re paying for is ongoing content. Like, yes, there’s a technical aspect of like, we have a website and you can watch the videos online, and maybe that part doesn’t change a lot, but we do have to continue to produce new content.

IVAN: Drupal 8 comes out, Drupal 9 comes out.

JOE: I’ve made videos about how to create a view in Drupal for Drupal 6 and Drupal 7 and Drupal 8. Oh my God, I've got this down. So, ultimately the decision was in order for Drupalize.Me to really be successful you need to push it out of the nest and teach it how to fly. Let it learn on its own. So, we did. The company was split off from Lullabot into—what was at the time, Lullabot Education—and it was split off on paper, so technically for legal reasons, we were separate companies, but we still had the same ownership.

We were still all in the same Slack channel. We still all went on the same company retreat. It was just from an accounting perspective and taxes and that kinds of stuff, we were a separate company. That was years ago, now. And it has continued to evolve since. The ownership has changed a little bit, there’s overlap between the two, but it’s not identical anymore. Lullabot Education, which is now Osio Labs, has grown. We still attend the same retreat—sometimes, but not always. In some ways there’s some logistic stuff about it that makes that really easy for us to do.

And we just recently started our own Slack team, so we’re no longer sharing the same Slack room with Lullabot. We now have our own. I guess over time what’s happened is, as Osio Labs has grown and we’ve hired new people, there are now a number of people who don’t have that shared history, and trying to figure out how to balance that has been an ongoing thing. Those of us who were part of Lullabot at one time, it’s often difficult to be like, “Ah, I miss that,” or “I still am part of that. These are still my friends.” But then the people that never there don’t have that history, and it just makes no sense. It’s like, Lullabot who? Who are we talking about? What’s going on here? It’s like trying to figure that all out. 

IVAN: And, it’s also may be influenced by the evolution of the things you’re teaching as well, because you’ve gone from teaching purely Drupal, and spinning that off as another company, and Osio Labs, now, you guys are teaching Gatsby and Node as well as Drupal, and this is kind of more of a generalized company that isn’t just going to take on Drupal.

JOE: Yeah, I think that’s right. I think it’s like part of the change of name, and really, it’s that going from Lullabot Education to Osio Labs is like, we’ve done a bunch of paperwork and we’ve changed our name. Everything else is still the same. It’s an indication of our evolution and growing up as a company. And, wanting to branch out into things other than Drupal and establish a name for the company.

We’re looking at things in other communities like Gatsby or Node or other open source areas that we might want to get into. We don’t necessarily need that association with the Lullabot name, where Drupalize.Me has been, and continues to be, super beneficial. Then we wanted a name that was a bit more reflective of the fact that we’re doing more than just Drupal at this point. Barely. But we’re just starting to do more than Drupal, but the long-term plan is to take all of the lessons that we’ve learned, teaching people how to do things with Drupal and the success that we’ve had with Drupalize.Me, and being members of the Drupal community and finding other open source communities that, as a business, we can participate in, and provide training, provide support for the community, help the community grow. All of those things.

IVAN: I love that you’re expanding and applying the things that you’ve learned in Drupal to the rest of the world and the rest of the community. I think that’s admirable, and I think you have a ton of value that you can bring to the open source community. I just wish you the very best of luck in doing that, and I’m excited to see that grow.

JOE: Yeah. It’s going to be a lot of fun. I think.

IVAN: I think so. And I love that it’s in the name, as well.

JOE: Yes.

IVAN: Just awesome. So, you’ve been teaching for a long time. I think you alluded to the fact that it’s been probably about 15 years or so, and it’s not just technology is it? You’re kind of a beast on the ski slopes, aren’t you?

JOE: [laughing] Yeah, so I teach snowboarding as well, and I’ve actually been doing that for quite a bit longer than I’ve been teaching people Drupal. And I continue to do that today, but not nearly as much. Teaching snowboarding does not pay the bills for me, [laughing] but I still enjoy doing it.

IVAN: When did you start that? Was that around the same time as you got to the city, moving away from Stillwater?

JOE: I was in college.

IVAN: Oh, so you didn’t do one of those Saturday camps where you get up at 7:00 a.m. and spend the day out on the ski slopes and Mom and Dad pick you up later?

JOE: I did not participate in one, but I teach for one now. But I was never a part of it. I teach for a ski school here in Minnesota called Blizzard. One of my friends had been in Blizzard when she was younger and we were both in college. She and her family had remained connected with the people that owned Blizzard at the time. Snowboarding, as a thing that students were interested in, was really taking off and they were basically desperate for snowboarding instructors and they were like, “Joe, Katie, you know how to snowboard right?” We’re like, “Yeah.” They were like, “You want to be teachers?” We were like, “No, I have no idea how to be a teacher.” They were like, “That’s fine. Mostly we just need warm bodies on the hills.”

IVAN: Usually the case. [laughing]

JOE: [laughing] It’s really more like we need someone to make sure this group of kids returns safely at the end of the day. And if they happen to learn something along the way, I guess that’s a nice side effect.

So, I was like, “Sure, I can do that.” And I got into teaching through that, and that kind of evolved over time too. After a couple of years as a pseudo-instructor, I found some training that I could take to help learn be a better snowboarding instructor, and then got certified through a program called The American Association of Snowboard Instructors

Over time I continued to learn how to be a better instructor through that. I think part of what happened there is I had spent a number of years teaching snowboarding, getting certified, getting to a point where I was doing training for other people to become certified as snowboard instructors, and this opportunity—I mentioned earlier Lullabot was looking to hire trainers and I applied for the job—and one of the questions is, “Do you have any previous teaching experience?” I was like, “Yeah, I do actually. Not formal in a school sense, but I had all of this background from Blizzard teaching and helping with certification programs, and that kind of stuff.”

In retrospect, I think that was a big part of why they eventually called me back. Like we just said, everyone in the Drupal community at the time, it was like, “Do you know how to teach people?” I was like, “No.” They’re like, “Oh, that’s fine. We just need warm bodies to stand up here and flip through the slide deck.” [laughing]

IVAN: [laughing] Right.

JOE: I was like, “I could do that.” It’s awesome.

IVAN: My son loves Blizzard. He’s been skiing for about eight years. He’s been skiing since he was three or four years old, and this last winter was the first time he did the Blizzard program. He’ll be thirteen pretty soon. He loved it. He complains about getting up early on a Saturday morning. You know, “Oh, I have to get up at 6:30 and be there at 7:00.” But he comes back with the best experience and people, new people he’s met. And, you know, they have a rating system on which ski slopes have the best refreshments and fries.

JOE: So, how it works is Blizzard is a traveling ski school. So, we go to different ski areas around the Twin Cities Metro area instead of just going to the same one every time. And you show up at one of these places, like you show up at Trollhaugen and as a snowboarder I’m like, This is awesome. We’re at a new place. I can’t wait to get out and go snowboarding. And then, as a teenage kid, they’re all like, “Hmm, what videogames do they have at Trollhaugen?” [laughing]

IVAN: [laughing] Yeah, the perspective is a little different.

JOE: One of the things I enjoy most about being a snowboard instructor is the opportunity to take my passion for snowboarding and display that in a way that someone else gets to see it and sort of be like, “Wow, he’s really excited about this and he’s having a lot of fun. I want to do that too.” And, especially working with kids, you have this opportunity to like, yes you can teach them the mechanics of how to make a toeside turn, but also you can have a lot of fun. So, then the next week, when they’re getting ready to come back, they’re excited to come back, and they’re excited to go snowboarding again. And hopefully they grow up to be adults that are excited to go snowboarding.

IVAN: I appreciate that work that you do as well. I know Cooper evolved from not wanting to get up and try and go to Blizzard to like, “Yeah, don’t mess with my Saturdays in the winter because I have Blizzard to go to.” Yeah, it’s just been great to see him do that. Now, I’m going to ask you maybe a little bit of a controversial question. So, this is a heads up.

JOE: Okay.

IVAN: Have you biased your kids one way or another, ski versus snowboard?

JOE: Yes, I’m sure I have. [laughing] I have not intentionally told my children, “You will snowboard, and you will never ski.” But at the same time there are no skis in my house. My kids are really young. I have a three-year-old and a one-year-old. So, neither of them are skiing or snowboarding yet. We did get Wesley a snowboard last winter, no bindings, just a plastic thing with a rope on the front, and I’ll pull him up and down the alley, and he thinks that’s pretty fun.

We’ve been talking about maybe starting them on skis, in part because, generally kids can start skiing a lot younger than they can snowboarding. Snowboarding just requires a different amount of muscle. You need more physical strength to control a snowboard, initially, than you do with skis, and to be able to do things like stop. Whereas for younger kids, it is relatively easier to put on a pair of skis and make a wedge, pizza, French fries, pizza, French fries.

IVAN: I started skiing in my late thirties. I’m very familiar with pizza. [laughing]

JOE: [laughing] Right. That’s about the extent of my skiing skills too. So, if I was to teach my kids how to ski, it would mostly be, “Here’s some skis, here’s a hill, I’ll see you at the bottom.” [laughing]

IVAN: [laughing] I think that’s awesome.

JOE: I will absolutely teach them skiing or snowboarding. As far as I’m concerned, they should choose one that’s exciting to them. I would rather them be really excited about a winter sport, than me dictating what it is. Ultimately, I just want us to have things that we can do that get us outside in a Minnesota winter, and enjoy being outside instead of cooped up inside all of the time.

IVAN: Totally agree. My kids were in the Buck Hill program when they were three and four. My wife would take them out, and here’s this South African immigrant [Ivan] who has never skied ever, sitting in the chalet watching them. I thought to myself, What am I doing? I got to get out there as well. It’s never too late to learn anything. They’re going to be much better than I am, and still are, but so what, we’re out there. That’s what counts. That’s awesome. Okay. One final question. I want to talk about your handle. I’ve always thought it was the greatest handle that someone has on Twitter. So, it’s eojthebrave. What’s the etymology of that?

JOE: Eojthebrave, eoj is "Joe" spelled backwards, and then "the brave," with all the spaces removed. And it’s a funny thing, because I now have this online moniker that everyone knows me as that was never intended to be something that people could pronounce. It was supposed to be just a bunch of characters that you saw on the screen. I never thought about would someone be able to say my name. But now I attend all of these Drupal events and open source community events, and people are like, “You’re eojthebrave,” and I’m like “I guess you could say it that way.” It’s kind of like, is it OH-sio or AH-sio? I don’t know.

IVAN: It doesn’t matter. It’s important that we’re talking.

JOE: So, years ago I was really into playing Warcraft 2, and you could play online, and I needed to have a nickname that would inspire fear in the hearts of my enemies, [laughing] especially in a medieval war game. What better than something like "Joe the Brave." And I was like, online you’re supposed to be anonymous. People shouldn’t know that my name is Joe. So, if I just spell it backwards, no one will ever know.

IVAN: Clever.

JOE: And then it evolved over time. Initially I think it was like eoj <space> the <space> brave, and then I probably signed up for something where you couldn’t have spaces in your name so I used underscores. And then I probably signed up for something where you couldn’t have underscores in your name. So, it just kind of all merged together. And now, here I am, twenty years later being like, “I should’ve just picked Joe.” [laughing] But, this works fine.

IVAN: [laughing] And, do you have all of the domain names as well. Eojthebrave.com and all that?

JOE: No, I don’t.

IVAN: Oooh Alright. I’m buying it right now. It’s going to be a little more valuable in a little bit here.

JOE: Yeah, totally. [laughing]

IVAN: [laughing] That’s awesome. Well, thank you very much for giving that explanation and talking with me. It was just really wonderful to find out more about you and what you’re doing. Yeah, come back on the show, and maybe we’ll talk about some big ideas in the next episode.

JOE: That’d be fun. I appreciate the opportunity to chat with you and just catch up.

IVAN: Yeah. That was awesome. Joe Shindelar is passionate about open source technology and is a lead trainer and lead developer at Osio Labs. You can find him on Twitter and on Drupal.org where his handle is @eojthebrave. You can also check out his personal chunk of the internet at dreamformula.com. You’ve been listening to the TEN7 Podcast. Find us online at ten7.com/podcast. And if you have a second, do send us a message. We love hearing from you. Our email address is [email protected]. Until next time, this is Ivan Stegic. Thank you for listening.

Oct 07 2019
Oct 07
Screenshot of LSS Home Page

Becoming Their Go-To Drupal Experts

Lutheran Social Service of Minnesota (LSS) is one of the state’s largest private nonprofit social service organizations, providing a vast array of services in all 87 Minnesota counties for children and families, older adults and people with disabilities.

Our Assignment: Audit LSS Sites and Take Them On as a TEN7Care Support Client

LSS hired us to support and maintain their flagship website (lssmn.org) and seven smaller websites. Minnesotans find and interact with LSS services through their websites (over 1,200 pages), including performing time-sensitive tasks like submitting timesheets and payments, so it’s crucial the websites perform optimally with very little downtime. The sites were already well-organized, designed and written to meet the organization’s needs—a testament to hard work done previously. LSS just needed help maintaining the technology and continuing to enhance the site.

TEN7Audit & TEN7Improve

Before TEN7 takes on any support client, we take them through our comprehensive onboarding process to get an overall lay of the land and identify any lurking security or performance issues. As a result of the TEN7Audit, we implemented best practices to update the LSS sites and keep them backed up and as secure as we could possibly make them:

  • Updates: Updated Drupal core, Drupal modules and PHP to current official versions (including disabling non-essential and performance-intensive Drupal modules).
  • Security: Ensured there were no security vulnerabilities.
  • Site backups: We recommend (and LSS wanted) offsite backups, something their current host didn’t provide. We set up redundant offsite backups using Tractorbeam at Amazon’s AWS and in Google Cloud. 
  • Update Code: Replaced custom code with Drupal core functionality or Drupal modules.
  • Caching: Implemented HTTP caching, database query caching, and Drupal caching to provide a faster site experience for site visitors.

TEN7Care

In the TEN7Care phase, we focus on adding new features and optimizing the site and the processes that surround it.

  • Issue backlog: In the support and maintenance phase, new clients usually bring us a small task backlog. But the super-prepared client of the year award goes to Tom Lany, Online Marketing Manager at LSS, who started the TEN7Care process with a list of 60 backlogged tasks compiled from stakeholder feedback, prioritized and categorized in a spreadsheet we could feed directly into our issue tracking program. Thanks to Tom’s great organization, we assigned and blew through many of those tasks in short order!
  • Deployment process: We discovered the website deployment process was a pain point. Various features (like menus) would frequently break during deploys. We fixed the root cause of the menu issue and implemented some of our software development best practices around deploys: greater reliance on issue tracking and version control, and a deploy schedule that mirrored the sprint schedule. At end-of-sprint meetings, we review candidates for release, and then do a release after that meeting. LSS now has confidence in and connection to a predictable and testable deployment process.

We’re Now Caring for Lutheran Social Service’s Sites

Lutheran Social Service of Minnesota is now an official TEN7Care support client. We’ll continue to update, optimize, and secure their sites, and add new features as needed. We look forward to a long and fruitful relationship.

“TEN7 helps us solve technical challenges that we can’t solve ourselves within Drupal, and they keep our systems updated and secure. TEN7’s extensive expertise with Drupal means we get answers quickly, and that we don’t have to worry about missing security and stability updates, because their team carefully tracks all of this.“ —Tom Lany, Online Marketing Manager, Lutheran Social Service of Minnesota

Sep 25 2019
Sep 25

Your browser does not support the audio element. TEN7-Podcast-Ep-071-Kevin-Thull-Drupal-Archivist.mp3

Summary

If you've ever watched a Drupal Camp or Con session from the comfort of your home, you likely have our guest Kevin Thull to thank. Thull has recorded almost 1700 Drupal sessions, and he keeps looking for more ways to contribute to the Drupal community. Bonus: If you're a gear nerd you'll love hearing about the evolution of the recording system!     

Guest

Kevin Thull, Drupal Developer and Archivist

Highlights

  • Life in Chicago (Cubs fan forever!)
  • Almost artificial limbs
  • Yet another person that joined Drupal because of the helpful, welcoming community
  • How Kevin got started recording sessions
  • Failing is important!
  • Going on the road to record Camps
  • Giant red button debut
  • Iterating to find the perfect sound and audio recording setup
  • Ad hoc recording becomes the Drupal Recording Initiative
  • Midwest Open Source Alliance (MOSA) and fiscal sponsorship and insurance
  • Aaron Winborn Award
  • Contributing to community without coding

Links

Transcript

IVAN STEGIC: Hey everyone! You’re listening to the TEN7 podcast, where we get together every fortnight, and sometimes more often, to talk about technology, business and the humans in it. I am your host Ivan Stegic. My guest today is Kevin Thull, a freelance frontend developer and President of the Midwest Open Source Alliance. You may know him as the guy whose session recording kits are omnipresent at Drupal events across the globe. He’s also the 2018 recipient of the Aaron Winborn Award, an award that is presented annually to an individual who demonstrates personal integrity, kindness and above and beyond commitment to the Drupal community. Hey, Kevin. Welcome to the podcast. It’s a great pleasure to have you on.

KEVIN THULL: Thank you. It’s great to be here.

IVAN: I have so many questions. [laughing] I feel like there’s so much to explore. So maybe you’ll consider coming back if we don’t get to it all?

KEVIN: Definitely.

IVAN: Awesome. Okay, I thought we’d start with some background. So right now, you live in Chicago, and you went to the University of Illinois at Chicago. Are you a lifelong Chicagoan?

KEVIN: I am. Born and raised.

IVAN: Born and raised. So, where did life in school start for you?

KEVIN: I’m on the northwest side of Chicago. So, I went to schools in that area. As far as UIC I went there for Bioengineering. My dream was to create artificial limbs, and then I learned I’d pretty much be in school for the rest of my life. I said nope! [laughing]

IVAN: [laughing] Wow. Bioengineering. That’s amazing. What was the motivation for artificial limbs?

KEVIN: It just seemed an interesting and really useful career. At the time, I graduated college in 1989/90, so there wasn’t a whole lot of advancement at that point. So, it just seemed like a really interesting and rewarding career to go into.

IVAN: Yeah. I’ve seen those artificial limbs that are 3D printed these days.

KEVIN: Yeah, it’s incredible.

IVAN: It really is. Do they use Raspberry Pis, I think, in some cases? Or, I don’t know what it is, but it looks like there’s a really inexpensive way to get things done these days.

KEVIN: Yeah. There was an event at my old job sponsorship conference and one of speakers was one of the inventors of that 3D printing, or some of the innovators of it, and it’s just an incredible story.

IVAN: It really is. It’s kind of what technology and the internet, the original idea behind it was trying to accomplish. Right? Something that can bring the masses. Something that’s cheap and life-changing as a technology, whether it’s hardware or software, it doesn’t matter.

KEVIN: Right. Yeah.

IVAN: So, I have to ask you, since you’re in Chicago. White Sox or Cubs?

KEVIN: I grew up on the northwest side, so Cubs fan forever.

IVAN: Cubs fan. Yes.

KEVIN: I don’t really follow sports at this point anymore.

IVAN: No? Well recent World Champions, I have to give it to you that it’s probably okay to stop following it. Right?

KEVIN: Yeah. And I actually lived near Wrigley Field when that happened.

IVAN: What a beautiful ballpark.

KEVIN: Yeah. It’s wonderful. I hope they don’t change it.

IVAN: Yeah. I’m just so excited about baseball these days, given that the Twins are now number one in the league, and have the best average. We really needed that. I feel like it’s a good omen that that’s what happened to the Cubs who went to the World Series, and now maybe the Twins can do it.

KEVIN: Yeah, that’d be amazing.

IVAN: Definitely would be amazing. [laughing] So, let’s talk a little bit about Drupal. You’ve been in the Drupal ecosystem for more than 10 years with many different areas of interest and expertise from being a site builder to developer, to being involved in the community. Do you remember your first experience with Drupal?

KEVIN: Yeah. Vividly. [laughing] Drupal 6 was just shiny and new, and I was using a product to essentially build a static site, so I was using an early static site generator, just this Perl script that let me create both a car parts website and a couple different product websites, we’ll put it at that. Since it was UI-based the site kept timing out during the rebuilds for the owners of their sites, and telling them, “Oh, you can log in through SSH and run it there.” It was not an option. So, I started evaluating other systems and it was really down to between Joomla and Drupal.

IVAN: Ooh, Joomla.

KEVIN: Right. But feature set, similar, they looked equally capable on paper. So, I looked through support forms, because I’m not a coder by trade, I guess you can say. I can't code my way out of a paper bag, is how I’ll define my programming skills. I’m good with CSS and SaSS, but in terms of the rest, even though I went to engineering school, you’d think I’d be better at it. I looked at the community forums for both, and Joolma’s answers were, “Sure we’ll help you for a bounty,” and Drupal’s answers were “Sure we’ll help you. Can I move in with you to help you build this thing?” Sort of that feel. So, at that point I went with Drupal.

IVAN: Yeah, it was the community that got you hooked, it sounds like.

KEVIN: Absolutely. Then I struggled because there were no migration scripts at that point, so I had to find some custom PHP to brute force it into the database, which worked, data tables were a whole lot easier in Drupal 6.

IVAN: Of course.

KEVIN: Yeah. Then I didn’t quite understand the whole contrib cycle. I was like Drupal 5 versus Drupal 6. Well Drupal 6 is new I’ll use that, and then realized I was sort of stuck waiting for contrib to follow-up. I ended up doing an Ubercart site, struggled with a make/model/year selector, and my first community event, because I had learned a lot through videos. You found videos on archive.org from past events, and that got me a long way. But then I was stuck. 

I was very, very introverted, very shy at the time. I still am a little bit. So, I committed to going to an in-person meetup. I was living in the suburbs at the time, and there was a meetup posted and it’s like, “Well ask your questions and we’ll have Jeff Eaton there because he wrote the book on building Drupal," or he’s one of the writers. I’d been listening to Lullabot podcasts. I was having celebrity anxiety. So, he showed up, and I asked my question, “How could you do this?” He’s like, “Oh, I’m so sorry, 'cause basically there is no solution for that right now.” At least I felt good that it wasn’t me. And if he can’t figure it out, then...

IVAN: Is that code still around that you wrote?

KEVIN: No. The site owner ended up migrating out to BigCommerce at some point. He had several different sites. But we had it going for a while, doing lots of imports and CSV files. So, it was a pretty intense project.

IVAN: So that interaction with Jeff Eaton, was that your first in-person involvement in some sort of a community event?

KEVIN: Yeah. It was the very first Drupal Fox Valley meetup.

IVAN: Drupal Fox Valley meetup. Is that still around?

KEVIN: They are. Yeah. I’m sad I don’t live in the suburbs. That’s one of the reasons I’m sad I don’t live in the suburbs, because it’s a pretty far west suburb but it was a great, great group. I met a lot of wonderful people there, and I count that as one of the reasons that I am where I am today, being part of that group in that community.

IVAN: So that was your first exposure to the community. Is it also the first time you started participating and organizing events as well?

KEVIN: More or less. I did some light volunteering at the Chicago Drupal Camp when it was around, but we ended up as a suburban group. It’s a decent commute. There’s a good community in the suburbs so we decided to have our own DrupalCamp Fox Valley. That was October, 2013. That’s also when I decided I was going to record the sessions, because at the time where I worked, we hosted a marketing conference where I basically was involved in recording sessions. So I’m like, well, a) I learned when I started Drupal from session recordings; b) I do this for work, so it was a no brainer in my mind to do that for events that I’m organizing.

IVAN: So, 2013, there’s the first set of sessions that you decide to record and it’s at a meetup? Or it’s at the Fox Valley Camp?

KEVIN: Yeah. We had our Fox Valley DrupalCamp, or DrupalCamp Fox Valley.

IVAN: So, did you go into that camp thinking, “Okay, I’m going to record every single session?” Or did you say, “Let’s iterate. Let’s choose one room and see how it goes?” [laughing]

KEVIN: No. I figured, I do this for work, so, we’re going to get them all, and the method was have a camcorder in the back of the room just to see when slides change, get the slide presentation from the presenter, make stills of each slide, and then kind of rebuild what would be a screen share. Because that was the process that I did at work, but it was for marketing conference.

IVAN: I see.

KEVIN: So, there were maybe 30 slides or so. At the work event, it was a union hotel, so we brought in AV to do the keynote as a live video production, but in the breakout rooms, to cut costs we just got an audio file from them. So, I would get their deck and any videos that they were playing and kind of rebuild it based on the audio and just what I call the "reference record" to see where those slides change.

IVAN: So, you actually had to rebuild every session there was on any live capture?

KEVIN: Correct.

IVAN: Alright. So that’s kind of version one?

KEVIN: Yeah. That was terrible.

IVAN: Was it? [laughing]

KEVIN: Well, there was one talk, it was like a 45-minute talk and over 100 slides, so it took like three hours to rebuild that.

IVAN: Boy, that was really time intensive.

KEVIN: Yeah, and you know, demos were lost. It’s just a completely different medium. It’s funny because friends of mine at the time were like, “Why are you investing so much time in this, in the post-production? You know, nobody’s going to watch these.” I’m like, “It’s important.”

IVAN: Yeah, and it really is important. Thank you for investing the time. It’s such an asset to the community now, I can’t even imagine what it would be without it.

KEVIN: Yeah. I never once imagined it would be what it is today. [laughing]

IVAN: I would love to know about what the next iteration was after you decide, “I can’t handle doing four hours and 100 slides for a 45-minute talk.” What’s the next iteration?

KEVIN: So, shortly after that event was the first MidCamp. That was March of 2014. We were fortunate enough to get the Drupal Association recording kits. So, the same laptops and splitters that they use at DrupalCon. Because apparently if your event falls in the window of when they’re not needed to be shipped or in shipping or en route, you can just pay the FedEx cost to borrow the equipment. So, you get this giant Pelican case, you could fit a body in it because it’s stuffed with laptops and equipment. And that was also a pretty horrible experience, because it was just a lot of setup. They didn’t work terribly well. Every once in a while, in the recording there would be a dropped frame, so you just see a one-second blue frame. So, of course, I edited those out. It was pretty low res and at the end of the event we were exhausted because it was our first one, it’s like, Oh now we have to drag this giant case and find a FedEx to send it home.

IVAN: So, were the laptops themselves doing the recording,  and you had to have your presentation on the laptop?

KEVIN: No, there was a splitter. So, the presentation computer fed into a splitter that split to the projector and to the recording MacBook. So, the MacBook was basically running a capture software. And to this day that’s the same type of equipment they use at Drupal Con. So, if you go into a session at DrupalCon you’ll see off to the side a table with a laptop on it that has a note saying, “Recording. Do not Touch.”

IVAN: Really.

KEVIN: And it’s just always on.

IVAN: I always wondered about that. Okay, so that’s version 2. So that’s 2014. So, what happens after that? You’ve reduced the amount of time you spend on post-production but you’re still not happy. You still want something better.

KEVIN: Yeah. So, I think it was, DrupalCon Austin was that year, so March is when we did the laptops. Went to DrupalCon Austin. We actually met with the people who produce the videos and I was brainstorming with a fellow organizer, there’s got to be some sort of solution that’s lightweight, inexpensive, device agnostic and no drivers. We kind of came up with this base requirements list and started looking, and it was really difficult to find stuff, because it turns out this is a very lucrative industry, recording events. They don’t want to give away their methods. Even the prosumer-level equipment is really expensive. So, I found this device that the intended market is to record your console gameplay, so it’s HDMI in and out, it records to a thumb drive and it’s got an audio mixer so it can pick up the gameplay audio from your console and then also your commentary through a headset. And it has a standalone mode, because lot of those console gameplay systems require you to either hook to a console where there’s some sort of interface or attach to a PC so you can run it through software. But it’s the only one that had a standalone mode. So, I’m like, well let me try it. So, I bought one and it worked. So, the second DrupalCamp Fox Valley, which was then later in 2014, was where that kit first debuted.

IVAN: Wow. And what was the cost of the kit at the time? Do you remember?

KEVIN: At that time, they came with a lav mic. So, just the unit itself was, I want to say, $180.00 plus dongle, so maybe low $200 per kit, which is really cheap.

IVAN: That’s really reasonable, yeah. Plus, you have to supply the thumb drive, right?

KEVIN: Yeah. When I think of cost per kit, that’s equipment plus dongles plus recording gear. But we have issues where if you had multiple presenters, you’re handing around this lav mic which is not a great way to deal with it, and every once in awhile there was no audio on the record. So, you’ve got the screen recording, no problems, but it’s silent. So those were lost. That’s when I decided to add in the zoom voice recorder which serves as the mic, but also records to an SD card.

IVAN: So that’s the omnidirectional mike that’s hooked up and right next to the console?

KEVIN: Yeah.

IVAN: Okay, so that’s the next version after the Fox Valley Camp?

KEVIN: Yeah. That was all exciting. It was promising. I got most of the recordings for Fox Valley. I was going to BADCamp that year, so that was September 2014, BADCamp was San Francisco, would’ve been late October. I just wanted to show off the kits and they’re like, “Well can you actually record some sessions?” I’m like, “Okay, sure.” I probably had two or three at the time. So, I brought them with me, they’re compact, and I recorded sessions and failed miserably.

IVAN: Really?

KEVIN: I think I caught maybe two out of 20 or 30 that I tried.

IVAN: What was the main issue?

KEVIN: So, this time they were bus powered. So, it would plug into the presenter's laptop.

IVAN: So, the assumption was there’s enough juice coming out of the laptop that’ll actually give you consistent power to power it.

KEVIN: Well there’s juice, but if that power gets interrupted before the file is written, then you get a zero K file.

IVAN: Oh, no!

KEVIN: Right. And generally, with equipment it loses the connection; it writes the file, powers down. Not so with this one. So, then that was wonderful and terrifying. It’s like, Okay, good, failing is important.

IVAN: Absolutely.

KEVIN: Right. Because if it’s working you don’t know how to break it, and therefore you don’t know how to fix it.

IVAN: Exactly.

KEVIN: So that was late 2014. March 2015 MidCamp No. 2 is coming up and I was a little scared, because it worked and then it failed, and here we go again. And, MidCamp was a success. So, it’s like, okay, great. What this tells me is I need to take these things on the road and just get more variables into the equation. So, shortly after MidCamp, I sent out a Tweet saying, “Hey Camps, if you’ll cover my airfare and hotel, I’ll record your Camp.” And St. Louis and Twin Cities took me up on it right away.

IVAN: Yeah, we did. It was like a no brainer for Twin Cities DrupalCamp. We were like, Oh, Kevin’s going to record it?” I think I remember voting yes on that request. I’m like, “Yeah, absolutely. Bring him. We’ll do it.”

KEVIN: Yeah, so that was also terrifying, because like, Oh now this someone else’s money. But by and large it worked really well. In St. Louis, I had 100% capture. So, like, great, this is good. But over time just various variables helped me to iterate on the kit, or whether that’s documentation—because BADCamp there was one year, there’s no time between sessions, and you’ve got six rooms over four buildings, and you’re the only one doing it. It’s like, “Okay, I guess I’m going to make instructions and put them at the podium because I’m not going to be there.” And it worked mostly.

Kevin Thull

IVAN: The giant red button, when did that make its debut?

KEVIN: The red button was part of what I call the beta kit, the 2014 Fox Valley version. So, it was the camcorder, there was the Drupal laptops, the DA laptops and then the Big Red Button. So that was early on the process and it’s just been a matter of smoothing out the whole bit.

IVAN: So, you took it on the road, you got different variables for the kit. Did the kit stay very similar after your beta process or did you change anything major after you were done with Twin Cities and St. Louis?

KEVIN: The bulk of the changes were adding in redundancies and taking out other variables. So, I added in the digital voice recorder, but then I got a remote for it. So, you hit the red button on the video, you’d hit the button on the audio record, and then when you’re done you stop both. Then so many times people forget to do the audio record, and then I realized, why don’t I just leave this thing to record all day long and take that out of the equation. One less thing for presenters to think about. And that worked.

There have been times, and sort of been the bane of my existence for a bit, because invariably someone would bump the power, or they’d turn off the power strip that everything’s attached to. Well, the video recorder powers on automatically. The audio recorder has to be turned on once it’s got power. So, I would lose it that way. Now there’s batteries in there, so there’s a failover. I now discovered there’s a hold switch so you can’t accidentally stop the recording, which has happened before. So, audio has become pretty solid in terms of capturing it.

Then just accidents. I had a four-port USB power, because one of the AV guys, when BADCamp failed miserably, he’s like, “Maybe there’s not enough USB power for some of this stuff.” For the laptops we’ll get a separate powered hub. So, I did but I thought I had to plug it into the laptop, and so, if the laptop went to sleep, it turned off power to the recorder. In one session I accidentally forgot to plug it in, and I think I know who the person is who, what I call happy accidents, her laptop went to sleep, and her recording continued. I'm like, “Oh, it’s because it’s not plugged in.” Because it’s not plugged into the laptop. So, it didn’t perceive that as a signal loss. Yeah, so, just lots of documentation and happy accidents throughout the years.

IVAN: And are you now at a final version of the right or do you have additional changes you want to make for the future?

KEVIN: The issues are currently, for whatever reason, if it’s not Mac OS, even though there’s a voice mixer, an audio mixer in the unit, there’s still no audio on the screen record. So, I don’t know if somehow rather than dubbing the non-audio from the presentation plus the spoken audio from the presenter, rather than mixing them, it’s completely wiped out. So, I had some time before someone’s session who historically had no audio, so I’m like, “Let’s look at your audio system and pick whatever is not chosen.” I assumed that it was choosing HDMI, and we would have to set it to headset. But it was set to headset. I’m like, “Let’s choose HDMI.” Then it worked. So, it’s like, “Oh, that’s cool.” But it’s still not 100%. Either you choose it, there will still be no audio, or it’ll be bad audio that has to be replaced. But it’s improved.

IVAN: I’m just amazed at the speed at which you get these sessions turned around and available online. What’s the secret to doing that?

KEVIN: The unit records to an .mp4 file on the thumb drive that’s attached to it. So, assuming you’ve got good audio, you already have a compressed file to upload to YouTube. So, as long as I go in, like any large break I’ll swap out media, see what I’ve got, fix anything that needs fixing and upload it, assuming then it has decent internet.

IVAN: So, really, you’re not doing any postproduction. That rig does it all for you.

KEVIN: Ideally, right. Yeah. When it works, it works really well. There are some small fixes. I’ve got enough experience where I’ve gotten quick at it. I think my most challenging was last year at DrupalCamp Montreal, it was a completely French spoken session that had no audio. So, trying to time that was tough.

IVAN: Oh, yeah. [laughing]

KEVIN: [laughing] I don’t speak French. So, eventually I figured it out.

IVAN: And what do you think the Achilles heel is with the whole system?

KEVIN: People. And that’s my next focus. When I’m doing it, I get close to 100% capture, pretty consistently. When others are doing it, it’s generally 80% or less, which I’ve learned is still okay. Because it means there are other people doing it, I’m not the blocker. But also, it’s just a matter of presence and making sure that everything’s being checked and rechecked. That a) you’re connecting the presenter laptops; and b) when sessions start, you’re verifying that the recording is recording. You still may lose the one or two that way, but it’s really just a matter of finding the people who care enough to make sure that it’s as successful as possible for any event that they’re managing equipment.

IVAN: How many sessions do you think you’ve recorded since you started in Fox Valley?

KEVIN: I do keep track.

IVAN: You do? So, this is not a guess. Okay. How many. What are we up to?

KEVIN: 1,646 total.

IVAN: Wow.

KEVIN: Although there’s more than that, because I don’t have numbers from Chattanooga. That includes sessions I’ve captured plus sessions—I call them proxy captures. So I now will send equipment to Camps through FedEx. And with instruction documentation. If needed I’ll do a video call with them to kind of go over how the kit works, troubleshooting stuff.

IVAN: And are the kits still around $250? Or has that changed?

KEVIN: So, by adding in the voice recorder that all totals about $450 per setup, which is still relatively affordable. I can get eight of them into a carryon-size Pelican case.

IVAN: That’s great.

KEVIN: They’re portable. They’re lightweight.

IVAN: Yeah. Wow. The quality on the recordings are nothing to be ashamed of. They’re all HD, the audio’s great. I don’t know how you get such great audio. You even get the questions from the auditorium as well.

KEVIN: That’s the audio recorder. I just have it set to multichannel, I think the auto gain and meeting is the setup. The preset. So, it does a good job of it.

IVAN: Yeah. I’m just so proud and amazed and you should be commended at every chance you can get, because this is such an amazing service and such high quality. It’s just amazing to see.

Am I right in saying that you started something called the Drupal Recording Initiative?

KEVIN: Yes.

IVAN: Tell me about that. What is that?

KEVIN: Yeah, this is a funny story. DrupalCorn Camp in Iowa last year, I was very happy to be able to record it. It was one of the first non-Chicago Camps I went to in 2014-2015, but they always had a way to record their sessions. So, I was never going to record theirs, even though I wanted to. This last year they reached out, I guess they didn’t have their typical contact and they wanted me to record it. So, I’m like, “Yes, absolutely.”

Then Matt Westgate of Lullabot, he gave a keynote and either right before or after that, he just nonchalantly asked me, “So, how’s the recording initiative going?” In my mind I’m like, “Oh wow, you just named this thing.” Because forever I was just like, “I’m just recording stuff.” So, it immediately got an upgrade. So, I had to kind of figure out what that was.

So I tried to do a year-end blog post to say how it’s gone for the year, do a little reporting, and the DA [Drupal Association] reached out to me after that, because this past year I realized that a) because this is bigger than just me, I need to start mentoring people, and so they offered to let me do a guest blog post on the DA’s blog so that it would amplify that. So, I’m like, great what am I going to write?

So, I came up with the initiative. Basically, broke it down into various buckets, like training and mentorship, expanded coverage, improved documentation, funding organization, content discoverability, and that was just basically December of last year. Now it’s just a matter of a three to five-year roadmap.

IVAN: So, this is quite recent. So, this is the end of last year you’re through about six months of it. How’s it going?

KEVIN: Surprisingly well. I think it just goes to show when you create a plan, you’ll start…

IVAN: …you’ll start working on the plan.

KEVIN: Yeah. If you don’t have a plan, you’re not going to achieve results. If you have a plan, you have a roadmap and things to shoot for.

IVAN: And how do we find out more about the Drupal Recording Initiative?

KEVIN: One of the items was open accounting, and in order to do that I put it on Open Collective, whether it links through show notes or something. But if you search "Drupal Recording Initiative," you’ll pretty much find it on open collective and I’ve got the entire initiative spelled out there.

IVAN: Excellent. We’ll link to it in the transcript and the show notes of this podcast episode, so keep it tracked there. But, it’s on opencollective.com, and as you said if you do a search for "Drupal Recording Initiative" it should be one of the first results. And I think it was for me.

KEVIN: Excellent. So, it’s working. [laughing]

IVAN: [laughing] So, it’s working. This is actually a really good segue into a question I had about the Midwest Open Source Alliance. It did say on the recording initiatives webpage that it’s hosted by the Midwest Open Source Alliance. What is MOSA? I’m sure that’s what you call it. Right?

KEVIN: It is what we call it.

IVAN: Okay. What is MOSA?

KEVIN: Yeah. MOSA was born out of the fact that the Drupal Association used to provide fiscal sponsorship for events, primarily in the U.S. They ended that program with the recommendation transfer over to Open Collective because they can be your fiscal sponsor. The DA took 10% which went to the Drupal project. Great. Going to Open Collective was going to take 10% and fund Open Source in general, also good but they were going to take a 10% on that initial deposit in addition.

So, as an event, we had already paid our 10% to the DA, so we were going to lose another 10% just to transfer. I wasn’t okay with that, especially because we didn’t know anything about Open Collective. So, that felt like a big jump to me. And there’s still issues like insurance, and getting sales tax exemption in Chicago is an issue.

So, some of the issues that we had when the DA was running this sponsorship program were going to not be fixed by moving the Open Collective. So, some of the MidCamp organizers got together, and we had been talking about this for a while, and that was the impetus to form our town nonprofit.

IVAN: And so, the Midwest Open Source Alliance is a federally recognized nonprofit, and you behave the same way that the Drupal Association did. You are fiscal sponsors for camps. I know that Twin Cities DrupalCamp uses you right now.

KEVIN: Yeah. It was primarily a solution for MidCamp, but we realized that if we could fix this for one, we could fix it for more. We tried to keep the scope smaller, geographically by Midwest, but also open the scope and just call it open source.

IVAN: And are you the fiscal sponsor and the insurance and everything else that a camp needs? Like the Open Collective and like the Drupal Association was to us?

KEVIN: That’s the intent. We’re still working on the insurance part. For any camp to be part of MOSA we have to designate an at large board member. So, in this case that was Dan Moriarty. So, he then is a representative of MOSA, so he can sign insurance using MOSA’s name. So, it’s not his name or his company. I didn’t even know that was a problem until I heard about event organizers being sued, because of something on their website.

IVAN: That’s awful.

KEVIN: Yeah. So, this is important. Even with the DA, at one point they provided insurance, and then they realized they couldn’t because it really wasn't part of their structure.

IVAN: The liability.

KEVIN: Yeah. So then here I am buying event insurance under my own name.

IVAN: Ouch.

KEVIN: Which is terrifying.

IVAN: Yeah, that is terrifying.

KEVIN: You do what you can to get your Camp out.

IVAN: Right. And how is MOSA funded? Is it also through a percentage that the members paid?

KEVIN: Yeah. So, we’re taking 5% from events and that’s been enough because it’s all volunteer run. We take 0% from initiatives, so donations. The Recording Initiative I do pay 5% platform fee to open collective but no additional cost. Because Open Collective itself is not a fiscal sponsor. There are fiscal sponsors on Open Collective. MOSA is now one of those, and the fiscal sponsor decides what percent they’ll take. So, for camps we don’t organize that through Open Collective, so that way we can get 5% to help keep the lights on. But for initiatives, we don’t need to take anything.

IVAN: And you talked about a plan for the Drupal Recording Initiative. What kind of a plan is there for MOSA? What are you guys hoping to achieve in the next few years?

KEVIN: We’ve got a project board on GitHub mostly to sort of finish. We’re building the bike as we’re riding it. It’s like, oh we have to create a nonprofit and run an event. And, oh, Twin Cities is actually going to be our next event, so now we have to figure that out. So, we’re getting documentation and things of that nature, hashing out insurance. We want year-long insurance for Board members, but also how to cover all volunteers of an event during the phase of the event. So, in theory, events don’t need event insurance. MOSA’s insurance would cover it, in theory. There’s a lot of time to talk to a lot of people to get a lot of quotes.

IVAN: Well, you guys are doing a wonderful job, so I wish you all of the best of luck for MOSA. I know that it felt like TC Drupal was looking for something like MOSA, and I’m just glad that we’re in the Midwest, and we’re able to take advantage of the Open Source Alliance.

KEVIN: Yeah, I’m glad it worked out.

IVAN: So, I think the last thing I want to talk to you about is the Aaron Winborn Award. Last year in 2018 in recognition of this incredible service you’ve been providing to our community, you received the Aaron Winborn award. What an honor to receive that. How did that make you feel?

KEVIN: It was incredibly humbling. I’m definitely not here for anything but to give back. So, to have to stand up and thank people. I understand that people really appreciate what I’m doing, but I’m not here for that. I’m here to just make videos available. So, it’s hard to go up there. I’m a very much behind-the-cameras kind of guy. It was wonderful.

IVAN: It was wonderful to see you accept that.

KEVIN: Thanks. Yeah.

IVAN: Did it change your approach to how and what you’re doing? Did it make it more intense? Did it change anything about your approach?

KEVIN: I don’t think so. I think, if anything, more people know me. [laughing] So I’m now Drupal famous.

IVAN: [laughing]

KEVIN: But aside from that, I’d say no.

IVAN: Well, it’s just wonderful to see. You’re just such a great example of how you can contribute to the community without writing a single line of code. Right?

KEVIN: Well, that’s the whole point.

IVAN: You’re a front end developer. You’ve written code. You’ve got patches in there, but you get an award for not writing code. So, that’s just a testament. So, what do you think your advice would be to those who just joined the Drupal community, or even to any open source community who maybe are not developers or who are young developers, or who just started writing code, maybe they’re afraid to show what they’ve written? What would your advice be to them about wanting to contribute?

KEVIN: Just, if you’re passionate about giving back to a community that you’re getting benefit from, don’t let the fact that you’re not maybe working on core module development, don’t let that stop you. There are so many ways that are either technical-lite or non-technical to give back. Documentation would be a great example for Drupal, because it’s still a sticking point. Plenty of opportunity to contribute there. But, at events you always need "day of" volunteers. There are plenty of non-standard ways to get involved. And also especially to bring in any past experience you have. I did video work, that’s not at all Drupal related, but look how big of an impact it’s made.

IVAN: Kevin, thank you so much for spending your time with me today on the podcast. It’s been a pleasure talking to you.

KEVIN: Well, thank you for having me.

IVAN: Kevin Thull is a freelance frontend developer and President of the Midwest Open Source Alliance. You can find him on Twitter @kevinjthull and on Drupal.org @kthull. And we'll have those in the show notes and in the transcription on the website. You’ve been listening to the TEN7 Podcast. Find us online at ten7.com/podcast. And if you have a second, do send us a message. We love hearing from you. Our email address is [email protected]. And don’t forget, we’re also doing a survey of our listeners. So, if you’re able to, tell us about what you are and who you are, please take our survey as well at ten7.com/survey. Until next time, this is Ivan Stegic. Thank you for listening.

Sep 18 2019
Sep 18

Your browser does not support the audio element. TEN7-Podcast-Ep-070-Using-Kubernetes-for-Hosting.mp3

Summary

After months deep in the weeds of Kubernetes, our DevOps Tess Flynn emerged with the best practices for melding Docker, Flight Deck and Kubernetes to create a powerful open source infrastructure for hosting Drupal sites in production (powered by our partner, DigitalOcean). Ivan and Tess take a deep dive into why we chose this combination of tools, our journey to get here, and the nitty gritty of how everything works together.    

Guest

Tess Flynn, TEN7 DevOps

Highlights

  • Why offer hosting ourselves now?
  • Differences in hosting providers
  • The beauty of containerization, and the challenge of containerization
  • The best container orchestrator
  • What’s with hosting providers and their opaque pricing? (and why we like DigitalOcean)
  • Kubernetes’ highly dynamic environment: updated with just a code push
  • Flight Deck, the genesis of our journey to Kubernetes
  • Docker enables consistent environments
  • Flight Deck + Kubernetes + DigitalOcean
  • You can do this all yourself! (or we can help you with our training)
  • It all runs on Drupal OR other platforms
  • In order to enjoy Drupal + Kubernetes, you must let go of your local file system and SSH, and reevaluate your email system
  • Complex files vs. static files and S3
  • Kubectl! (it sounds cuter when you say it out loud)
  • Cron jobs run differently in Kubernetes
  • A Tess talk isn’t complete without a car analogy: Kubernetes is like a garage that comes pre-stocked with all the tools you’ll need to work on your car

Links

Transcript

IVAN STEGIC: Hey everyone! You’re listening to the TEN7 podcast, where we get together every fortnight, and sometimes more often, to talk about technology, business and the humans in it. I am your host Ivan Stegic. We’ve talked about DevOps at TEN7 on the show before. We’ve done an episode on why we decided to expand our hosting offering to Linode back at the end of 2017. We’ve talked about why we think it’s important to have a good relationship with your hosting company. And, we’ve written about automation and continuous integration over the years as well.

For the last year or so, we’ve been working on our next generation of hosting service, and our DevOps Engineer, Tess Flynn, has been deep in the weeds with Kubernetes. Today, we’re going to spend some time talking about what we’ve done—and how you could be doing it as well,—given that we’ve open sourced all of our work.

We’re also rolling out training at BadCamp this year, that’s in October of 2019, and we’ll be at DrupalCorn as well, in November. So, we’ll talk about that and what you might learn by attending. So, joining me again is our very own Tess Flynn. Hello, socketwench.

TESS FLYNN: Hello.

IVAN: Welcome, welcome. I’m so glad you’re on to talk shop with me. I wanted to start with why. Why are we hosting our own sites and those of our clients? There are so many good options out there for WordPress, for Drupal: you've got Acquia and Pantheon, Blue Host, and others. We typically use the provider that makes the most sense, based on our clients’ needs.

We’ve had a close relationship with ipHouse and their managed hosting services for a long time. But why start hosting now? For us, as an organization, it’s kind of been the perfect storm of circumstances, from the technology being mature, to the cost of it, and the availability of it, to where we are as an organization from a developmental point of view, to even being more conscious of vendor lock in and actively trying to avoid it.

So, I want to talk about technology a little bit more with you, Tess. What’s so different now than it was a few years ago? Why is it suddenly okay for us to be hosting ourselves?

TESS: There’s been kind of an explosion over the last few years of managed Kubernetes hosting providers. Now, we’ve had managed hosting providers forever. We’ve had things that are called Infrastructure as a service (IaaS) provider; that’s going to be things like AWS and Google Compute Cloud, as well as other providers, including DigitalOcean, but also say, Linode and other ones, which just provide raw hardware, virtual machine and root login. Lately, however, a lot of people would rather break up their workloads into containers, using something that’s similar to Docker. And I’ve talked about Docker before, but Docker is an alternative take on virtualization technologies, which works on taking applications and putting them in their own individual, virtual environment. I’m glossing over so many things when I say that, but it gets the general point across, with the two minutes before everybody else falls asleep.

IVAN: Right.

TESS: What’s really nifty about putting applications into a container is that now the container doesn’t really care where it is. You can run it on your system, you can run it somewhere else, you can run it on a hosting provider. And, the great thing about these containers is that you can download ones that other people have created. You can modify them, make your own, and you can string them together to build an entire application service out of them. And that’s really, really great. That’s like infrastructure Legos.

But the problem is, once you get the containers, how do you make sure that they’re on the systems, on the actual hardware where they are supposed to be, in the number of copies that there’s supposed to be, and that they can all talk to each other? And the one’s that aren’t supposed to talk to each other, can’t? That’s a lot trickier. For a long time the problem has been that you really only have two solutions: you do it yourself, or you use something like Docker Swarm. I don’t have the greatest opinion of Docker Swarm. I have worked with it before in a production environment, it’s not my favorite.

IVAN: It’s a little tough, isn’t it? We’ve had a client experience on that.

TESS: It’s a little tough, yeah. It’s not really set up for something like a Drupal workload. It’s set up more for a stateless application. A prototypical example is, you need to calculate the progression of matter within the known galaxy, factoring a certain cosmological constant. Take that variable, set it into a compute grid and go, “Hey, tell me what the results are in 15 years.” But you don’t really do that with Drupal. With Drupal, you’re not just going to send off one thing and always get the same thing back. There’s going to be state, which is preserved. That’s going to be in the databases somewhere, and there are going to be files that are uploaded somewhere. And then you have to get load balancing involved, and then it gets really complicated, and it’s like ugh. I really didn’t like how Swarm did any of this stuff. It was very prescriptive. It was, you do it their way, and nothing else.

IVAN: No flexibility.

TESS: No flexibility at all. It was really, really not fun, and it meant that we had to do a lot of modification of how Drupal works, and incur several single points of failure in our infrastructure, in order to make it work in its form. That whole experience just did not get me interested or excited to make a broader Swarm deployment anywhere else.

Then I ran across Kubernetes, and Kubernetes has a very different mentality around it. Kubernetes has more different options for configurations, and you can tailor how Kubernetes manages your workload, rather than tailoring your workload to work with Docker Swarm. That’s why I really liked it. What's really nifty is, once you have Kubernetes, now you have an open source project, which is platform agnostic, which doesn’t care about which individual hosting provider you’re on, as long as you have containers, and you can send configuration to it somehow, it’s fine, it doesn’t care.

A lot of managed hosting providers are going, “Hey, you know, VMs [virtual machines] were kind of nifty, but we really want to get in on all this container stuff now, too.” “Oh, hey, there’s a container orchestrator,” which is what Kubernetes is, and what Docker Swam is, as well, a container “orchestrator” which does all of the making sure the containers are on the right systems, are running, they can talk to the containers they're supposed to, and can’t talk to containers they're not supposed to.

That made a lot of infrastructure providers go, “This is not really a Platform as a service anymore. This is another form of Infrastructure as a service. As such, that is a segment that we can get into."

So, first it started with Google Kubernetes Engine, which is still considered today the defacto version. Amazon got into it, Azure got into it. And all of these are pretty good, but a lot of these huge cloud service providers, you can’t get clear pricing out of them to save your life.

IVAN: Yeah. That’s so frustrating, as a client, as a business owner. How do you do that? It’s insane.

TESS: I mean, the only way that it seems that is deterministic, in order to figure out what your bill is going to be at the end of the month, is to spend the money and hope that it doesn’t kill your credit card. [laughing]

IVAN: Yeah, right, and then try to figure out what you did, and ways of changing it, and then hell, you’re supposed to be just charged that every month from now on, I suppose.

TESS: It’s just a pain. It wasn’t any fun, whatsoever. So, an alternative approach is, you could actually install Kubernetes yourself on an Infrastructure as a service provider with regular VMs.

IVAN: And, we considered that, right?

TESS: Oh, I considered it, and I even spun that up on a weekend myself. It worked. But the problem is, I’m a colossal cheapskate and I didn’t want to spend $30.00 a month for it. [laughing]

IVAN: [laughing] If only there was a supporting ISP that had free Kubernetes support, and just charged you for the compute engines that you used.

TESS: I was really kind of sad that there wasn’t one, until six or eight months ago, when DigitalOcean announced that they have in beta (now it’s in production) a Kubernetes service, where the pricing was incredibly clear. You go to the cluster page, you select the servers that you want to see (the nodes as it were). I know, Drupal nodes, infrastructure nodes, it’s really confusing. Don’t even get physics people involved, it gets really complicated. [laughing]

IVAN: No, please. No, don’t. [laughing]

TESS: But you select which servers that you want to have in your Kubernetes cluster, the sizing, and the price is just listed, right there, in numbers that you can understand! [laughing]

IVAN: Per month, not per minute.

TESS: I know, per month, not per minute.

IVAN: It’s just the small things. Crazy.

TESS: And, it really targeted the kind of market that we are in for a hosting provider, and it made me really excited, and I really wanted to start putting workloads on it, and that’s what started the entire process.

IVAN: It really was, kind of a fortuitous series of events, and the timing kind of just really worked out. I think one of the biggest things for us, for me, is that with Kubernetes, we don’t have to worry about patching and security updates, and monitoring them, and these large hardware machines that we have to keep patched and updated. Essentially, it’s updated every time we do a code push, right? I mean, we’re still concerned with it, but it’s a much easier burden to bear.

TESS: Right. Now what’s going on is that, every time that we do a push, we’re literally rebuilding every system image necessary to run the underlying application. Which means that if we need to push a system update, it’s really just a matter of updating the underlying container's base image to the newest version. We’re already using Alpine Linux as our base containers, which already is a security-focused minimal container set.

IVAN: So, this is actually a good segue to what I wanted to talk about next. A few years back (as opposed to six to nine months back), which is how we kind of got down the road to get to Kubernetes was, I think the origin of all this really is, Flight Deck, and the desire for us to make it easy for developers who work at TEN7—and anyone else who uses Flight Deck, honestly—to have the same development environment locally. Basically, we wanted to avoid using MAMP and WAMP and different configurations so that we could eliminate that from any of the bug-squashing endeavors that we were going into. So, let’s talk about this started with Docker and led into Flight Deck, and what a benefit it is to have the same environment locally as we do in staging and production.

TESS: So, there’s a joking meme that’s been going around, and DevOp cycles, of a clip of a movie where, I think a father and son are sitting and having a very quiet talk on a bench somewhere in a park, where the kid is saying, “But it works on my machine.” And then the Dad hugs him and says, “Well, then we’ll ship your machine.” [laughing] And, that’s kind of what Docker does. But joking aside, I wanted to get that out of the way so I’m not taking myself too seriously. [laughing]

So, one of the problems with a lot of local development environments—and we still have this problem—is that traditionally we’ve used what I consider a hard-installed hosting product. So, we’re using MAMP or WAMP or Acquia Dev Desktop, or if you’re on Linux you’re just installing Apache directly. And all of those work fine, except when you start working on more than one site and more than one client. So, suddenly you have this one problem where, this one client has this really specific php.ini setting, but this other client can’t have that setting. And MAMP and WAMP work around this through a profile mechanism which, underneath the covers is a huge amount of hyperlinking and weird configurations, and spoofing, and like eww, it makes me shutter.

IVAN: Yeah, it makes me cringe just to talk about it, yeah.

TESS: And, the problem is that, every time you have to do this, every developer has to do this themselves, they can’t just standardize on it. So, if somebody has an individual problem on their system, that only happens on their system at 3:45 on a Thursday, after they’ve had chili for lunch or something or other, then you can’t really reproduce it. So, the solution really is, you need to have replicatable, shareable, consistent development environments across your entire team. And that’s what Docker does.

Docker provides that consistency, that shareability, and makes sure that everybody does, in fact, have the same environment across the board. That’s the entire point of that, and that’s where the whole joke about, “Well, then we’ll ship your machine,” [laughing] because that is in essence what containers are. They are system images that run particular bits of software. Now, once we moved everyone to Docker for development, we now had a consistent environment between all of our systems, so that now we didn’t have to work about a number of different problems.

Another good example is, this site uses PHP 5, this site uses PHP 7—a little out of date now, but it was very relevant two years ago—in which case, how do you make sure you’re on the right version? Well, with Docker, you change a text file, and then you boot the containers up, and that’s it.

IVAN: And that text file lives in a code repository, right? So, everybody else gets that change?

TESS: Mm hmm, because you are literally sharing the same environment; you are enforcing a consistent development environment across your entire team for each individual project. And, if you use that strategy, you have something that is flexible, yet at the same time incredibly consistent.

IVAN: And this is really important across all of our developers, and all of our local development that we do, but the challenge then becomes, how do you consistently replicate this in a staging or in a test environment, and even in production? So, that’s kind of the genesis of how we thought Kubernetes could help us here, right?

TESS: Right.

IVAN: So, the challenge to you from me was, how do we make this work in production?

TESS: So, the nice thing about Flight Deck is, it was always designed with the intention of being put into production, But the orchestration component just wasn’t there, and the hosting component wasn’t there. Kubernetes showed up, and that solved the orchestration component, and then, eventually DigitalOcean showed up and now we have the hosting component. So, now, we have all the pieces together to create a consistent environment that is literally the same containers, from the first time someone starts working on the project, to when it gets deployed to production. That is the height of continuous integration ideals, to make sure that you have consistency across all of your environments. That you don’t have different, weird shared environments along the way, that everything is exactly the same so that you know that it will work.

IVAN: I want to stop right there, just so our listeners can appreciate the power of what you just said. You basically said, “I’m going to be working on a website, or a web application locally, with some sort of stack of required server components, whose version numbers and installation profile is configured in a certain way. My teammate is able to replicate that environment exactly, to the version, simply by using the same repo, and by using Flight Deck.

Moreover, all of those version numbers and the stack that is being used, is actually also the same now in staging and, most amazingly to me, in production. So, we can guarantee that what container is functioning in production on the Kubernetes cluster, is actually on staging and on everyone else’s machine. We’ve totally eliminated any variability and any chance that the environment is going to be causing an issue that one person may be seeing that another isn’t.

TESS: That’s correct.

IVAN: That’s pretty amazing!

TESS: It’s a really difficult thing to do, but starting with the containers and building that from the base up actually makes it a lot easier, and I don’t think that any other local development environment, even container based local development environment such as DDEV and Lando are doing this quite yet. Last I heard, I think DDEV was working on a production version of their containers, but it’s not the same containers, whereas with Flight Deck, it literally is the same container.

IVAN: It’s the same configuration. Everything is the same. That’s pretty amazing. I’m still kind of really impressed with all of the stuff that we’ve done, that you’ve done. And, honestly, this is all open source too. This is not like TEN7’s proprietary product, right? We’ve open sourced this, this is all on the web, you can download it yourself, you can figure it out yourself, you can do this as well. You can start your own hosting company.

TESS: That’s correct. The key item which puts all this together is, the Ansible role called Flight Deck Cluster. What Flight Deck Cluster does is, it will create a Flight Deck-flavored Kubernetes cluster and it works perfectly well on DigitalOcean. There’s no reason why it can’t work on say, Google Kubernetes Engine or AWS or anyone else. The architecture that Flight Deck Cluster uses is meant to be simple, durable and transportable, which is something that a lot of other architectures that I’ve seen just don’t have.

IVAN: So, we’ve designed a lightweight set of Docker containers called Flight Deck that you can use locally. We’ve evolved them so that they work with Kubernetes, which you can deploy anywhere in staging and production. We’ve open sourced them. And, the fact that it runs Kubernetes, all you need is a service that supports Kubernetes and you should be able to run all of this in those other locations.

So, we’ve talked about how we started with Docker and how that evolved, and I talked about how we've open sourced it and it’s available to you. I want to spend a little bit of time getting into the details, into the nitty gritty of how you would actually do this for yourself. Is there an app I download? Is it all the YML, all the YML files that we’ve open sourced? What would someone who wants to try this themselves, what would they have to do?

TESS: The first thing that I would probably do is, start running Flight Deck locally. Because you don’t need to pay any extra money for it, you just need to use your local laptop, and it’s also a good experience for you to learn how to interact with Docker by itself. That looks good on a résumé and it’s a good skill to actually have.

I have a talk that I used to give about Docker, and I know that there’s a blog post series that I posted somewhere a long time ago, about how Docker actually works under the covers. Both of those are going to be invaluable to understand how to get Flight Deck working on your local environment, and once you have it working on your local environment, then the next problem is to figure out the build chain. Now the way that our build chain works is, that we have another server, which is a build server, and what the build server does, is it’s going to receive a job from Gitlab and that job is going to take all of the files that constitute the site, it will build them into a local file system, and then it will put those inside of a container which is based on Flight Deck. Then it will upload those to a container registry somewhere else. So that we already have a few additional pieces of technology involved. But the nice thing is, Gitlab is open source, Ansible is open source, and all of our build processes are run through Ansible, and the Docker registry is also open source. It's just a container that you can run somewhere. There’s also services that you can buy that will actually provide you a container registry on a fee basis. All of those are definitely options. Once you have the container in a registry somewhere, then you can run Flight Deck Cluster to build out the rest of the cluster itself.

IVAN: You make it sound so easy. [laughing]

TESS: I make it sound easy, but it’s a lot of code, but it is all open source and it is all there for you to use. Right now, our cluster is based on a development version of Flight Deck, which I’ve been calling Flight Deck 4, and this version is intentionally natively designed for a Kubernetes environment. But it still works perfectly fine under Docker Compose locally, and it is literally the containers that we are using in production right now, at this minute. All of those containers have been thoroughly documented. They have nice readmes which describe exactly how you configure each individual container. And the Flight Deck Cluster role on GitHub also has an extensive readme document which describes how every individual piece is supposed to work.

IVAN: So, the easiest way to get to all that documentation into the repo is to simply go to flight-deck.me. That will redirect you to a blog post about Flight Deck on the ten7.com website, and at the bottom of that post you’ll see links to the GitHub repos and all of the other information that you’ll need to get to that.

So, I wanted to talk about the fact that the hosting itself, the Kubernetes hosting that we have, is optimized for Drupal right now—I kind of struggle to say "optimized for Drupal." It’s just configured for Drupal. There’s no reason that Kubernetes is, and what we’ve released, is locked into Drupal. We are hosting our own React app on there. We have a CodeIgniter app that’s running, we even have a Grav CMS site on it. There’s no reason why you couldn’t host WordPress on it, or ExpressionEngine or any other php, MySQL, Apache, Varnish, Stack on it. Right? There’s nothing innately that forces you to be Drupal on this, right?

TESS: Nope.

IVAN: And that’s also from a design perspective. That was always the intention.

TESS: It’s intended to be run for Drupal sites. However, it always keeps an eye towards being as flexible as possible.

IVAN: So, I think that’s an important thing to mention. Let’s talk about some of the challenges of running Kubernetes in a cluster in production. It’s not like running a server with a local file system, is it?

TESS: [laughing] No, it isn’t.

IVAN: [laughing] Okay. Let’s talk about the opportunities of things to learn.

TESS: The biggest, scariest thing about Kubernetes and Drupal is, you have to let go of your local file system. That is the most scary thing that I have to tell people about Kubernetes.

IVAN: So, no file system, huh?

TESS: No file system.

IVAN: Does that make it slow?

TESS: Well, not really. Let me describe why. The problem is, that— and I’ve had this in my Return of the Clustering talk—is that we’re used to something which is called “block storage.” Now, block storage is pretty great. It is a literal attached disk to the server. So, it is mounted on the server, you have direct access to it, and you can store all kinds of things to it. And it’s fast, and it’s right there. It has no failover, it can’t be shared across the systems, but ehhh, whatever, we have one big server, who cares about that.

Then, if you do try building a traditional server cluster, well, you can’t quite do that. So then you get network file system involved, NFS. And then now, all of the file reads and writes occur over the network to some other centralized server. Okay, it still looks like a local block storage, it still works like block storage, so, okay, sure. But the problem with that is that network file systems, by their base nature, introduce a single point of failure.

Now, that’s not good by itself. If the NFS server goes down, your entire site no longer looks or functions correctly. But the problem is, that it also doesn’t scale either. There’s a natural limitation between the number of different replications for frontend server, servers that intercept the actual requests from people, and then send them to the Drupal backend for processing, and then push back their responses. There’s a natural limitation between those systems and those that can access NFS. And as soon as you have too many accesses, suddenly NFS is not going to be keeping up with you and your performance drops to the floor.

Also, NFS is kind of persnickety. You have to tune it. You have to make sure that it has enough RAM, enough bandwidth. You have to make sure it’s physically proximate to the rest of the servers. And, all of this is because it’s trying to replicate block storage. Now, block storage is great for a whole bunch of data, but in a cloud architect's perspective, there are really two different kinds of data. There’s complex data and static data.

And when I tell people about this, they go, “Well, what’s a complex file?” A lot of people will say, “Well, we have a whole bunch of files which are all linked together, that’s complex, right?” Nope. “Well, we have some Excel documents that’s on an NFS file, that’s complex, right?” Not really. So, what is a complex file? 

I spent hours, tried to squeeze an answer [laughing] out of the internet for this, and eventually arrived at the answer from a cloud architect's perspective: “complex files, such as the files which constitute the actual underlying disk storage for say, a MySQL database.” Data, which is written sparsely and seemingly randomly in multiple locations at multiple times with strict concurrency requirements. Now when I say that, does that sound like anything that we actually upload in a Drupal site?

IVAN: Nope.

TESS: Nope. None of it does. Block storage is required for complex data. But for static data, which is virtually everything that a Drupal site hosts, we don’t need it, it’s too much. It’s way too complicated. And, it doesn’t scale. So, what’s the solution? The solution really is, we need to treat the file system like an API. We need to treat the file system like a database. We don’t care where the database is, as long as you have an IP, a login and the correct credentials to actually get to the database, and then we have multiple readers, multiple writers. That’s what we want for a file system, right? Well, it turns out, there’s a thing that does that already, it’s called S3.

IVAN: Yes, AWS, hello. [laughing]

TESS: And the nice thing about S3 is, it’s perfect for static data. It’s API accessible and it can be made internally redundant. So, it has its own high availability built in that we don’t need to worry about. The nice thing that’s even more than that, is when we say S3, most people go, “Oh, Amazon.” No. S3 is, in fact, a standard. It is not just Amazon’s implementation of S3. There are multiple implementations of S3. So, I usually like saying an S3-compatible hosting provider. And that’s going to include anybody who runs any kind of S3-compatible service. And there’s actually an open source product called Ceph that actually provides an S3 frontend for file storage. And that is actually a service that DigitalOcean also provides. They have DigitalOcean spaces, which provide an S3-compatible static file interface, that’s actually powered by a Ceph cluster underneath the covers. So, open source all the way down to the core.

IVAN: Well, I didn’t know that spaces was Ceph underneath the covers. That’s cool.

TESS: It’s just buried in there. You could find it though.

IVAN: Cool. So, file storage is a challenge, but we fix that by using S3.

TESS: Yep, because Drupal 7 and 8 actually have very good S3 support. There’s S3 FS, that particular module which is excellent for doing Drupal 7 sites. We’ve been using Fly System for Drupal 8 for a few different reasons, but there are reasons that are a little bit easier for us. But your mileage may vary.

IVAN: And, if you’re going to host something that’s not Drupal related, you would need to find some other S3-compatible layer module, right?

TESS: Like for the CodeIgniter application, we are currently looking at implementing that as well.

IVAN: And, there’s a React app as well that we’ve deployed. That uses the underlying Drupal site, though, doesn’t it?

TESS: Yes, that doesn’t actually need a local file system.

IVAN: There’s no SSH access to a cluster of Kubernetes, is there?

TESS: Yes, that’s the other thing. It’s like after I already brutalized you with saying, “No, you can’t have a local file system,” now I take your SSH away as well. [laughing]

IVAN: [laughing] But there is something to use to replace it, right?

TESS: There is. The problem is that, you really, really, really, really, really, really, really shouldn’t use SSH in Kubernetes. SSH is a very dangerous thing to have running anywhere, because it is a potential security access point that can be used and abused, both internally and externally. You really don’t want to have to run it, because if you want to run SSH in Kubernetes, you have to run it in a container. And if you run it in a container, you’re running it as root. And if you’re running it as root, you’re running it as root on the underlying hardware that’s powering the cluster, and that’s bad. [laughing] You don’t want to do that.

So, instead you want to access what is typically called “the backplane.” The backplane is going to be access to the workload via the orchestration system. So, for Kubernetes, the backplane access comes in the form of a command line application called Kubectl or “Kube control” or “Kubey control” or “Kubectl” or like 15 other different names. [laughing] I always thought of Kubectl, that’s my favorite.

IVAN: Let's spell it out. [laughing] I like that one too. k-u-b-e-c-t-l

TESS: And this application not only lets you interact with the orchestrator, but also allows you to directly access individual containers as well. Although getting to an individual container is a little bit more difficult, once you’ve done it a few times, it’s not that hard. Because Kubernetes is so popular, there’s a lot of other command line environments, which will have auto completion assistance for Kubectl as well. So, for me, if I enter in a parameter to Kubectl, say for name space, I can hit tab and it will give me a list of the name spaces that I have. So I don’t actually have to type it out.

IVAN: Pretty slick.

TESS: I use Z Shell (ZSH) but that’s me, I’m weird. Some people like using Fish or some other shell. And I’m sure there’s auto completion mechanisms for your favorite shell somewhere.

IVAN: There’s not a whole lot of challenges then, with Kubernetes. You’ve kind of mentioned a few that are surmountable. Is there anything else, a budding developer, a budding DevOps person should know about, that are looking to start to explore hosting for themselves?

TESS: Well, they should also keep in mind that email is a problem.

IVAN: Yes! We discovered that in the last few weeks, didn’t we?

TESS: Yes, we did.

IVAN: So, we decided that we were going to use an external, transactional email provider. We ended up on SendGrid. But you don’t think of these things once when you’re working on a cluster that’s managed because, hey, these machines all have SendMail on them.

TESS: Yup, and that’s one thing that you really can’t rely on when you start working with a container-based workload. It exposes a lot of these things. But, we’re not where we were two or three years ago where this would’ve been a huge, scary, problem. These things have existing solutions, which are not that difficult to implement, even today.

IVAN: And there are some free tiers as well that you can use, especially if you don’t have a high volume of emails that you’re sending out.

TESS: If you’re only sending 500 emails a day, you can configure your G Suite email as the SMTP provider.

IVAN: Exactly. What about cron? Isn’t that a problem too?

TESS: Cron is a little bit different in Kubernetes. So, the thing with cron is that, in Kubernetes, cron isn’t just something that runs a command. In a traditional server workload, cron is some background process that exists in the system, and when a certain time shows up, it runs a certain command that you tell it to. And, it assumes that you’re running it on literally the same exact system that is running everything else, your web workload. Right?

IVAN: Right.

TESS: That’s not quite the case in Kubernetes. In Kubernetes, a cron job actually runs a container. So, when you actually have your web workload, you’re going to have one container, say, for Apache, somewhere, which is running your site. Then you have a cron job in Kubernetes, and that cron job will literally spin up a completely separate container in order to actually run that process.
So, that’s a bit different.

Now, the only real part of that which gets really confusing is, if you don’t have a nice separation of all of the different infrastructure we just finished talking about, if you don’t have any local disks that you need to worry about, if you don’t have SendMail you have to worry about, if you don’t have any of this stuff and you can scale out your web container to 10 or 20 or more, and not have a problem because they all rely on external API-based providers, then it doesn’t really matter what you do with cron. You just literally run the same container that you run for your web workload, with the same configuration and everything else, but you only tell it run a particular command, instead of "Run Apache." And that’s it. That’s what we do. And, it’s actually not very hard.

IVAN: What’s your favorite thing about Kubernetes? I’m only going to give you five minutes at the most. [laughing]

TESS: [laughing] I think the thing that I like the most about it, is probably the ability to easily scale things. Once you actually have solved all the underlying infrastructure problems, you basically have just a container-based workload that you can say, “I need to run three of these.” Then you can tell it and it will run three of them, and it will just run it, that’s it, you don’t need to worry about it. It already load balances it for you. How can I describe this? Well, let’s go back to the infamous car analogies again.

IVAN: They work.

TESS: They work, but you know they work within a US cultural context of a certain decade period, of a certain geographic location, but let’s put that aside for a second.

So, a car analogy. Let’s say you have a car, and you want to do some work on it. And you go to your garage and what do you see? The car and an empty garage. That’s often what a lot of other systems look like. When you have to do traditional clustering with regular virtual machines, or even self-hosted physical machines, you have to go over to your local hardware store, buy all the tools, buy the car jack, buy an engine lift, buy an air compressor and a whole bunch of other stuff, in order to do your car stuff, and it’s a lot of work and a lot of investment.

With Kubernetes, it’s more like, Okay, I go to my garage and I have Kubernetes. So I have all the tools already. All the tools are just there on the walls, right now. I can just start working. That’s what I really like about Kubernetes. It provides me a room with all the tools for me to actually make this workload do what I want it to do, rather than having to go and grab yet another thing, then another thing, then another thing. Then try to make compromises to make two things, which aren’t the thing that I can’t get right now, but they’re the two I have, to work together.

IVAN: I love the analogy. [laughing] I think that works, Tess. So, what about training? Wouldn’t it be great if, instead of trying to figure this all out yourself (like we did), you could just have us show you how to do it?

TESS: Gee, wouldn’t it? [laughing]

IVAN: Wouldn’t it be great? Well, guess what? That actually exists. We’re going to be doing some free trainings at BadCamp and then at DrupalCorn as well. We’ll be at BadCamp next month, the beginning of October. Now, they’re free trainings, but there is a cost of use to attending the training itself, so I think you have to register and it’s $20, or $10 at DrupalCorn. They’re free as far as we’re concerned.

Can you talk through, just a little bit about the format of the training that we have set up? What are you going to learn and who is it for?

TESS: So, we’ll briefly touch upon different kinds of Kubernetes hosting providers, as well as what Kubernetes actually is and what it does, and what it gives you. Then afterwards, we’re going to start containerizing your particular application. So, we’ll start working with containers, putting them onto Kubernetes, getting used to how to use Kubectl, how to work with individual definitions within Kubernetes, and making all of these pieces work together.

IVAN: And, it’s a four-hour workshop, it’s half a day, you get to spend time with Tess, and I think I’ll be there too. It’s going to be great. So, if you want to contribute to Flight Deck, or to Kubernetes, the Kubernetes Flight Deck Cluster that we have, we’d love it. It’s all online. You can visit ten7.com, and you’ll find it there on the what we give back page and you can also visit us on github.com/ten7, and you’ll see all the repos there. We’d love your help. Thank you, Tess, so much for spending your time with me today. This has been truly great.

TESS: Not a problem.

IVAN: So, if you need help with your own hosting, or figuring out what makes most sense to you, we’d love to be there to help you, whether you’re a developer or a large university, or a small business, it doesn’t matter. We’re happy to provide consulting, whether that means deploying your own Kubernetes or having us do it for you, or even selecting another vendor that makes the most sense to you.

Just send us an email and get in touch. You can reach us at [email protected]. You’ve been listening to the TEN7 Podcast. Find us online at ten7.com/podcast. And if you have a second, do send us a message. We love hearing from you. Our email address is [email protected]. And don’t forget, we’re also doing a survey of our listeners. So, if you’re able to, tell us about what you are and who you are, please take our survey as well at ten7.com/survey. Until next time, this is Ivan Stegic. Thank you for listening.

Sep 18 2019
Sep 18

Your browser does not support the audio element. TEN7-Podcast-Ep-070-Using-Kubernetes-for-Hosting.mp3

Summary

After months deep in the weeds of Kubernetes, our DevOps Engineer Tess Flynn emerged with the best practices for melding Docker, Flight Deck and Kubernetes to create a powerful open source infrastructure for hosting Drupal sites in production (powered by our partner, DigitalOcean). Ivan and Tess take a deep dive into why we chose this combination of tools, our journey to get here, and the nitty gritty of how everything works together.    

Guest

Tess Flynn, TEN7 DevOps Engineer

Highlights

  • Why offer hosting ourselves now?
  • Differences in hosting providers
  • The beauty of containerization, and the challenge of containerization
  • The best container orchestrator
  • What’s with hosting providers and their opaque pricing? (and why we like DigitalOcean)
  • Kubernetes’ highly dynamic environment: updated with just a code push
  • Flight Deck, the genesis of our journey to Kubernetes
  • Docker enables consistent environments
  • Flight Deck + Kubernetes + DigitalOcean
  • You can do this all yourself! (or we can help you with our training)
  • It all runs on Drupal OR other platforms
  • In order to enjoy Drupal + Kubernetes, you must let go of your local file system and SSH, and reevaluate your email system
  • Complex files vs. static files and S3
  • Kubectl! (it sounds cuter when you say it out loud)
  • Cron jobs run differently in Kubernetes
  • A Tess talk isn’t complete without a car analogy: Kubernetes is like a garage that comes pre-stocked with all the tools you’ll need to work on your car

Links

Transcript

IVAN STEGIC: Hey everyone! You’re listening to the TEN7 podcast, where we get together every fortnight, and sometimes more often, to talk about technology, business and the humans in it. I am your host Ivan Stegic. We’ve talked about DevOps at TEN7 on the show before. We’ve done an episode on why we decided to expand our hosting offering to Linode back at the end of 2017. We’ve talked about why we think it’s important to have a good relationship with your hosting company. And, we’ve written about automation and continuous integration over the years as well.

For the last year or so, we’ve been working on our next generation of hosting service, and our DevOps Engineer, Tess Flynn, has been deep in the weeds with Kubernetes. Today, we’re going to spend some time talking about what we’ve done—and how you could be doing it as well—given that we’ve open sourced all of our work.

We’re also rolling out training at BadCamp this year, that’s in October of 2019, and we’ll be at DrupalCorn as well, in November. So, we’ll talk about that and what you might learn by attending. So, joining me again is our very own Tess Flynn. Hello, socketwench.

TESS FLYNN: Hello.

IVAN: Welcome, welcome. I’m so glad you’re on to talk shop with me. I wanted to start with why. Why are we hosting our own sites and those of our clients? There are so many good options out there for WordPress, for Drupal: you've got Acquia and Pantheon, Blue Host, and others. We typically use the provider that makes the most sense, based on our clients’ needs.

We’ve had a close relationship with ipHouse and their managed hosting services for a long time. But why start hosting now? For us, as an organization, it’s kind of been the perfect storm of circumstances, from the technology being mature, to the cost of it, and the availability of it, to where we are as an organization from a developmental point of view, to even being more conscious of vendor lock in and actively trying to avoid it.

So, I want to talk about technology a little bit more with you, Tess. What’s so different now than it was a few years ago? Why is it suddenly okay for us to be hosting ourselves?

TESS: There’s been kind of an explosion over the last few years of managed Kubernetes hosting providers. Now, we’ve had managed hosting providers forever. We’ve had things that are called Infrastructure as a service (IaaS) provider; that’s going to be things like AWS and Google Compute Cloud, as well as other providers, including DigitalOcean, but also say, Linode and other ones, which just provide raw hardware, virtual machine and root login. Lately, however, a lot of people would rather break up their workloads into containers, using something that’s similar to Docker. And I’ve talked about Docker before, but Docker is an alternative take on virtualization technologies, which works on taking applications and putting them in their own individual, virtual environment. I’m glossing over so many things when I say that, but it gets the general point across, with the two minutes before everybody else falls asleep.

IVAN: Right.

TESS: What’s really nifty about putting applications into a container is that now the container doesn’t really care where it is. You can run it on your system, you can run it somewhere else, you can run it on a hosting provider. And, the great thing about these containers is that you can download ones that other people have created. You can modify them, make your own, and you can string them together to build an entire application service out of them. And that’s really, really great. That’s like infrastructure Legos.

But the problem is, once you get the containers, how do you make sure that they’re on the systems, on the actual hardware where they are supposed to be, in the number of copies that there’s supposed to be, and that they can all talk to each other? And the one’s that aren’t supposed to talk to each other, can’t? That’s a lot trickier. For a long time the problem has been that you really only have two solutions: you do it yourself, or you use something like Docker Swarm. I don’t have the greatest opinion of Docker Swarm. I have worked with it before in a production environment, it’s not my favorite.

IVAN: It’s a little tough, isn’t it? We’ve had a client experience on that.

TESS: It’s a little tough, yeah. It’s not really set up for something like a Drupal workload. It’s set up more for a stateless application. A prototypical example is, you need to calculate the progression of matter within the known galaxy, factoring a certain cosmological constant. Take that variable, set it into a compute grid and go, “Hey, tell me what the results are in 15 years.” But you don’t really do that with Drupal. With Drupal, you’re not just going to send off one thing and always get the same thing back. There’s going to be state, which is preserved. That’s going to be in the databases somewhere, and there are going to be files that are uploaded somewhere. And then you have to get load balancing involved, and then it gets really complicated, and it’s like ugh. I really didn’t like how Swarm did any of this stuff. It was very prescriptive. It was, you do it their way, and nothing else.

IVAN: No flexibility.

TESS: No flexibility at all. It was really, really not fun, and it meant that we had to do a lot of modification of how Drupal works, and incur several single points of failure in our infrastructure, in order to make it work in its form. That whole experience just did not get me interested or excited to make a broader Swarm deployment anywhere else.

Then I ran across Kubernetes, and Kubernetes has a very different mentality around it. Kubernetes has more different options for configurations, and you can tailor how Kubernetes manages your workload, rather than tailoring your workload to work with Docker Swarm. That’s why I really liked it. What's really nifty is, once you have Kubernetes, now you have an open source project, which is platform agnostic, which doesn’t care about which individual hosting provider you’re on, as long as you have containers, and you can send configuration to it somehow, it’s fine, it doesn’t care.

A lot of managed hosting providers are going, “Hey, you know, VMs [virtual machines] were kind of nifty, but we really want to get in on all this container stuff now, too.” “Oh, hey, there’s a container orchestrator,” which is what Kubernetes is, and what Docker Swam is, as well, a container “orchestrator” which does all of the making sure the containers are on the right systems, are running, they can talk to the containers they're supposed to, and can’t talk to containers they're not supposed to.

That made a lot of infrastructure providers go, “This is not really a Platform as a service anymore. This is another form of Infrastructure as a service. As such, that is a segment that we can get into."

So, first it started with Google Kubernetes Engine, which is still considered today the defacto version. Amazon got into it, Azure got into it. And all of these are pretty good, but a lot of these huge cloud service providers, you can’t get clear pricing out of them to save your life.

IVAN: Yeah. That’s so frustrating, as a client, as a business owner. How do you do that? It’s insane.

TESS: I mean, the only way that it seems that is deterministic, in order to figure out what your bill is going to be at the end of the month, is to spend the money and hope that it doesn’t kill your credit card. [laughing]

IVAN: Yeah, right, and then try to figure out what you did, and ways of changing it, and then hell, you’re supposed to be just charged that every month from now on, I suppose.

TESS: It’s just a pain. It wasn’t any fun, whatsoever. So, an alternative approach is, you could actually install Kubernetes yourself on an Infrastructure as a service provider with regular VMs.

IVAN: And, we considered that, right?

TESS: Oh, I considered it, and I even spun that up on a weekend myself. It worked. But the problem is, I’m a colossal cheapskate and I didn’t want to spend $30.00 a month for it. [laughing]

IVAN: [laughing] If only there was a supporting ISP that had free Kubernetes support, and just charged you for the compute engines that you used.

TESS: I was really kind of sad that there wasn’t one, until six or eight months ago, when DigitalOcean announced that they have in beta (now it’s in production) a Kubernetes service, where the pricing was incredibly clear. You go to the cluster page, you select the servers that you want to see (the nodes as it were). I know, Drupal nodes, infrastructure nodes, it’s really confusing. Don’t even get physics people involved, it gets really complicated. [laughing]

IVAN: No, please. No, don’t. [laughing]

TESS: But you select which servers that you want to have in your Kubernetes cluster, the sizing, and the price is just listed, right there, in numbers that you can understand! [laughing]

IVAN: Per month, not per minute.

TESS: I know, per month, not per minute.

IVAN: It’s just the small things. Crazy.

TESS: And, it really targeted the kind of market that we are in for a hosting provider, and it made me really excited, and I really wanted to start putting workloads on it, and that’s what started the entire process.

IVAN: It really was, kind of a fortuitous series of events, and the timing kind of just really worked out. I think one of the biggest things for us, for me, is that with Kubernetes, we don’t have to worry about patching and security updates, and monitoring them, and these large hardware machines that we have to keep patched and updated. Essentially, it’s updated every time we do a code push, right? I mean, we’re still concerned with it, but it’s a much easier burden to bear.

TESS: Right. Now what’s going on is that, every time that we do a push, we’re literally rebuilding every system image necessary to run the underlying application. Which means that if we need to push a system update, it’s really just a matter of updating the underlying container's base image to the newest version. We’re already using Alpine Linux as our base containers, which already is a security-focused minimal container set.

IVAN: So, this is actually a good segue to what I wanted to talk about next. A few years back (as opposed to six to nine months back), which is how we kind of got down the road to get to Kubernetes was, I think the origin of all this really is, Flight Deck, and the desire for us to make it easy for developers who work at TEN7—and anyone else who uses Flight Deck, honestly—to have the same development environment locally. Basically, we wanted to avoid using MAMP and WAMP and different configurations so that we could eliminate that from any of the bug-squashing endeavors that we were going into. So, let’s talk about this started with Docker and led into Flight Deck, and what a benefit it is to have the same environment locally as we do in staging and production.

TESS: So, there’s a joking meme that’s been going around, and DevOp cycles, of a clip of a movie where, I think a father and son are sitting and having a very quiet talk on a bench somewhere in a park, where the kid is saying, “But it works on my machine.” And then the Dad hugs him and says, “Well, then we’ll ship your machine.” [laughing] And, that’s kind of what Docker does. But joking aside, I wanted to get that out of the way so I’m not taking myself too seriously. [laughing]

So, one of the problems with a lot of local development environments—and we still have this problem—is that traditionally we’ve used what I consider a hard-installed hosting product. So, we’re using MAMP or WAMP or Acquia Dev Desktop, or if you’re on Linux you’re just installing Apache directly. And all of those work fine, except when you start working on more than one site and more than one client. So, suddenly you have this one problem where, this one client has this really specific php.ini setting, but this other client can’t have that setting. And MAMP and WAMP work around this through a profile mechanism which, underneath the covers is a huge amount of hyperlinking and weird configurations, and spoofing, and like eww, it makes me shutter.

IVAN: Yeah, it makes me cringe just to talk about it, yeah.

TESS: And, the problem is that, every time you have to do this, every developer has to do this themselves, they can’t just standardize on it. So, if somebody has an individual problem on their system, that only happens on their system at 3:45 on a Thursday, after they’ve had chili for lunch or something or other, then you can’t really reproduce it. So, the solution really is, you need to have replicatable, shareable, consistent development environments across your entire team. And that’s what Docker does.

Docker provides that consistency, that shareability, and makes sure that everybody does, in fact, have the same environment across the board. That’s the entire point of that, and that’s where the whole joke about, “Well, then we’ll ship your machine,” [laughing] because that is in essence what containers are. They are system images that run particular bits of software. Now, once we moved everyone to Docker for development, we now had a consistent environment between all of our systems, so that now we didn’t have to work about a number of different problems.

Another good example is, this site uses PHP 5, this site uses PHP 7—a little out of date now, but it was very relevant two years ago—in which case, how do you make sure you’re on the right version? Well, with Docker, you change a text file, and then you boot the containers up, and that’s it.

IVAN: And that text file lives in a code repository, right? So, everybody else gets that change?

TESS: Mm hmm, because you are literally sharing the same environment; you are enforcing a consistent development environment across your entire team for each individual project. And, if you use that strategy, you have something that is flexible, yet at the same time incredibly consistent.

IVAN: And this is really important across all of our developers, and all of our local development that we do, but the challenge then becomes, how do you consistently replicate this in a staging or in a test environment, and even in production? So, that’s kind of the genesis of how we thought Kubernetes could help us here, right?

TESS: Right.

IVAN: So, the challenge to you from me was, how do we make this work in production?

TESS: So, the nice thing about Flight Deck is, it was always designed with the intention of being put into production, But the orchestration component just wasn’t there, and the hosting component wasn’t there. Kubernetes showed up, and that solved the orchestration component, and then, eventually DigitalOcean showed up and now we have the hosting component. So, now, we have all the pieces together to create a consistent environment that is literally the same containers, from the first time someone starts working on the project, to when it gets deployed to production. That is the height of continuous integration ideals, to make sure that you have consistency across all of your environments. That you don’t have different, weird shared environments along the way, that everything is exactly the same so that you know that it will work.

IVAN: I want to stop right there, just so our listeners can appreciate the power of what you just said. You basically said, “I’m going to be working on a website, or a web application locally, with some sort of stack of required server components, whose version numbers and installation profile is configured in a certain way. My teammate is able to replicate that environment exactly, to the version, simply by using the same repo, and by using Flight Deck.

Moreover, all of those version numbers and the stack that is being used, is actually also the same now in staging and, most amazingly to me, in production. So, we can guarantee that what container is functioning in production on the Kubernetes cluster, is actually on staging and on everyone else’s machine. We’ve totally eliminated any variability and any chance that the environment is going to be causing an issue that one person may be seeing that another isn’t.

TESS: That’s correct.

IVAN: That’s pretty amazing!

TESS: It’s a really difficult thing to do, but starting with the containers and building that from the base up actually makes it a lot easier, and I don’t think that any other local development environment, even container based local development environment such as DDEV and Lando are doing this quite yet. Last I heard, I think DDEV was working on a production version of their containers, but it’s not the same containers, whereas with Flight Deck, it literally is the same container.

IVAN: It’s the same configuration. Everything is the same. That’s pretty amazing. I’m still kind of really impressed with all of the stuff that we’ve done, that you’ve done. And, honestly, this is all open source too. This is not like TEN7’s proprietary product, right? We’ve open sourced this, this is all on the web, you can download it yourself, you can figure it out yourself, you can do this as well. You can start your own hosting company.

TESS: That’s correct. The key item which puts all this together is, the Ansible role called Flight Deck Cluster. What Flight Deck Cluster does is, it will create a Flight Deck-flavored Kubernetes cluster and it works perfectly well on DigitalOcean. There’s no reason why it can’t work on say, Google Kubernetes Engine or AWS or anyone else. The architecture that Flight Deck Cluster uses is meant to be simple, durable and transportable, which is something that a lot of other architectures that I’ve seen just don’t have.

IVAN: So, we’ve designed a lightweight set of Docker containers called Flight Deck that you can use locally. We’ve evolved them so that they work with Kubernetes, which you can deploy anywhere in staging and production. We’ve open sourced them. And, the fact that it runs Kubernetes, all you need is a service that supports Kubernetes and you should be able to run all of this in those other locations.

So, we’ve talked about how we started with Docker and how that evolved, and I talked about how we've open sourced it and it’s available to you. I want to spend a little bit of time getting into the details, into the nitty gritty of how you would actually do this for yourself. Is there an app I download? Is it all the YML, all the YML files that we’ve open sourced? What would someone who wants to try this themselves, what would they have to do?

TESS: The first thing that I would probably do is, start running Flight Deck locally. Because you don’t need to pay any extra money for it, you just need to use your local laptop, and it’s also a good experience for you to learn how to interact with Docker by itself. That looks good on a résumé and it’s a good skill to actually have.

I have a talk that I used to give about Docker, and I know that there’s a blog post series that I posted somewhere a long time ago, about how Docker actually works under the covers. Both of those are going to be invaluable to understand how to get Flight Deck working on your local environment, and once you have it working on your local environment, then the next problem is to figure out the build chain. Now the way that our build chain works is, that we have another server, which is a build server, and what the build server does, is it’s going to receive a job from Gitlab and that job is going to take all of the files that constitute the site, it will build them into a local file system, and then it will put those inside of a container which is based on Flight Deck. Then it will upload those to a container registry somewhere else. So that we already have a few additional pieces of technology involved. But the nice thing is, Gitlab is open source, Ansible is open source, and all of our build processes are run through Ansible, and the Docker registry is also open source. It's just a container that you can run somewhere. There’s also services that you can buy that will actually provide you a container registry on a fee basis. All of those are definitely options. Once you have the container in a registry somewhere, then you can run Flight Deck Cluster to build out the rest of the cluster itself.

IVAN: You make it sound so easy. [laughing]

TESS: I make it sound easy, but it’s a lot of code, but it is all open source and it is all there for you to use. Right now, our cluster is based on a development version of Flight Deck, which I’ve been calling Flight Deck 4, and this version is intentionally natively designed for a Kubernetes environment. But it still works perfectly fine under Docker Compose locally, and it is literally the containers that we are using in production right now, at this minute. All of those containers have been thoroughly documented. They have nice readmes which describe exactly how you configure each individual container. And the Flight Deck Cluster role on GitHub also has an extensive readme document which describes how every individual piece is supposed to work.

IVAN: So, the easiest way to get to all that documentation into the repo is to simply go to flight-deck.me. That will redirect you to a blog post about Flight Deck on the ten7.com website, and at the bottom of that post you’ll see links to the GitHub repos and all of the other information that you’ll need to get to that.

So, I wanted to talk about the fact that the hosting itself, the Kubernetes hosting that we have, is optimized for Drupal right now—I kind of struggle to say "optimized for Drupal." It’s just configured for Drupal. There’s no reason that Kubernetes is, and what we’ve released, is locked into Drupal. We are hosting our own React app on there. We have a CodeIgniter app that’s running, we even have a Grav CMS site on it. There’s no reason why you couldn’t host WordPress on it, or ExpressionEngine or any other php, MySQL, Apache, Varnish, Stack on it. Right? There’s nothing innately that forces you to be Drupal on this, right?

TESS: Nope.

IVAN: And that’s also from a design perspective. That was always the intention.

TESS: It’s intended to be run for Drupal sites. However, it always keeps an eye towards being as flexible as possible.

IVAN: So, I think that’s an important thing to mention. Let’s talk about some of the challenges of running Kubernetes in a cluster in production. It’s not like running a server with a local file system, is it?

TESS: [laughing] No, it isn’t.

IVAN: [laughing] Okay. Let’s talk about the opportunities of things to learn.

TESS: The biggest, scariest thing about Kubernetes and Drupal is, you have to let go of your local file system. That is the most scary thing that I have to tell people about Kubernetes.

IVAN: So, no file system, huh?

TESS: No file system.

IVAN: Does that make it slow?

TESS: Well, not really. Let me describe why. The problem is, that— and I’ve had this in my Return of the Clustering talk—is that we’re used to something which is called “block storage.” Now, block storage is pretty great. It is a literal attached disk to the server. So, it is mounted on the server, you have direct access to it, and you can store all kinds of things to it. And it’s fast, and it’s right there. It has no failover, it can’t be shared across the systems, but ehhh, whatever, we have one big server, who cares about that.

Then, if you do try building a traditional server cluster, well, you can’t quite do that. So then you get network file system involved, NFS. And then now, all of the file reads and writes occur over the network to some other centralized server. Okay, it still looks like a local block storage, it still works like block storage, so, okay, sure. But the problem with that is that network file systems, by their base nature, introduce a single point of failure.

Now, that’s not good by itself. If the NFS server goes down, your entire site no longer looks or functions correctly. But the problem is, that it also doesn’t scale either. There’s a natural limitation between the number of different replications for frontend server, servers that intercept the actual requests from people, and then send them to the Drupal backend for processing, and then push back their responses. There’s a natural limitation between those systems and those that can access NFS. And as soon as you have too many accesses, suddenly NFS is not going to be keeping up with you and your performance drops to the floor.

Also, NFS is kind of persnickety. You have to tune it. You have to make sure that it has enough RAM, enough bandwidth. You have to make sure it’s physically proximate to the rest of the servers. And, all of this is because it’s trying to replicate block storage. Now, block storage is great for a whole bunch of data, but in a cloud architect's perspective, there are really two different kinds of data. There’s complex data and static data.

And when I tell people about this, they go, “Well, what’s a complex file?” A lot of people will say, “Well, we have a whole bunch of files which are all linked together, that’s complex, right?” Nope. “Well, we have some Excel documents that’s on an NFS file, that’s complex, right?” Not really. So, what is a complex file? 

I spent hours, tried to squeeze an answer [laughing] out of the internet for this, and eventually arrived at the answer from a cloud architect's perspective: “complex files, such as the files which constitute the actual underlying disk storage for say, a MySQL database.” Data, which is written sparsely and seemingly randomly in multiple locations at multiple times with strict concurrency requirements. Now when I say that, does that sound like anything that we actually upload in a Drupal site?

IVAN: Nope.

TESS: Nope. None of it does. Block storage is required for complex data. But for static data, which is virtually everything that a Drupal site hosts, we don’t need it, it’s too much. It’s way too complicated. And, it doesn’t scale. So, what’s the solution? The solution really is, we need to treat the file system like an API. We need to treat the file system like a database. We don’t care where the database is, as long as you have an IP, a login and the correct credentials to actually get to the database, and then we have multiple readers, multiple writers. That’s what we want for a file system, right? Well, it turns out, there’s a thing that does that already, it’s called S3.

IVAN: Yes, AWS, hello. [laughing]

TESS: And the nice thing about S3 is, it’s perfect for static data. It’s API accessible and it can be made internally redundant. So, it has its own high availability built in that we don’t need to worry about. The nice thing that’s even more than that, is when we say S3, most people go, “Oh, Amazon.” No. S3 is, in fact, a standard. It is not just Amazon’s implementation of S3. There are multiple implementations of S3. So, I usually like saying an S3-compatible hosting provider. And that’s going to include anybody who runs any kind of S3-compatible service. And there’s actually an open source product called Ceph that actually provides an S3 frontend for file storage. And that is actually a service that DigitalOcean also provides. They have DigitalOcean spaces, which provide an S3-compatible static file interface, that’s actually powered by a Ceph cluster underneath the covers. So, open source all the way down to the core.

IVAN: Well, I didn’t know that spaces was Ceph underneath the covers. That’s cool.

TESS: It’s just buried in there. You could find it though.

IVAN: Cool. So, file storage is a challenge, but we fix that by using S3.

TESS: Yep, because Drupal 7 and 8 actually have very good S3 support. There’s S3 FS, that particular module which is excellent for doing Drupal 7 sites. We’ve been using Fly System for Drupal 8 for a few different reasons, but there are reasons that are a little bit easier for us. But your mileage may vary.

IVAN: And, if you’re going to host something that’s not Drupal related, you would need to find some other S3-compatible layer module, right?

TESS: Like for the CodeIgniter application, we are currently looking at implementing that as well.

IVAN: And, there’s a React app as well that we’ve deployed. That uses the underlying Drupal site, though, doesn’t it?

TESS: Yes, that doesn’t actually need a local file system.

IVAN: There’s no SSH access to a cluster of Kubernetes, is there?

TESS: Yes, that’s the other thing. It’s like after I already brutalized you with saying, “No, you can’t have a local file system,” now I take your SSH away as well. [laughing]

IVAN: [laughing] But there is something to use to replace it, right?

TESS: There is. The problem is that, you really, really, really, really, really, really, really shouldn’t use SSH in Kubernetes. SSH is a very dangerous thing to have running anywhere, because it is a potential security access point that can be used and abused, both internally and externally. You really don’t want to have to run it, because if you want to run SSH in Kubernetes, you have to run it in a container. And if you run it in a container, you’re running it as root. And if you’re running it as root, you’re running it as root on the underlying hardware that’s powering the cluster, and that’s bad. [laughing] You don’t want to do that.

So, instead you want to access what is typically called “the backplane.” The backplane is going to be access to the workload via the orchestration system. So, for Kubernetes, the backplane access comes in the form of a command line application called Kubectl or “Kube control” or “Kubey control” or “Kubectl” or like 15 other different names. [laughing] I always thought of Kubectl, that’s my favorite.

IVAN: Let's spell it out. [laughing] I like that one too. k-u-b-e-c-t-l

TESS: And this application not only lets you interact with the orchestrator, but also allows you to directly access individual containers as well. Although getting to an individual container is a little bit more difficult, once you’ve done it a few times, it’s not that hard. Because Kubernetes is so popular, there’s a lot of other command line environments, which will have auto completion assistance for Kubectl as well. So, for me, if I enter in a parameter to Kubectl, say for name space, I can hit tab and it will give me a list of the name spaces that I have. So I don’t actually have to type it out.

IVAN: Pretty slick.

TESS: I use Z Shell (ZSH) but that’s me, I’m weird. Some people like using Fish or some other shell. And I’m sure there’s auto completion mechanisms for your favorite shell somewhere.

IVAN: There’s not a whole lot of challenges then, with Kubernetes. You’ve kind of mentioned a few that are surmountable. Is there anything else, a budding developer, a budding DevOps person should know about, that are looking to start to explore hosting for themselves?

TESS: Well, they should also keep in mind that email is a problem.

IVAN: Yes! We discovered that in the last few weeks, didn’t we?

TESS: Yes, we did.

IVAN: So, we decided that we were going to use an external, transactional email provider. We ended up on SendGrid. But you don’t think of these things once when you’re working on a cluster that’s managed because, hey, these machines all have SendMail on them.

TESS: Yup, and that’s one thing that you really can’t rely on when you start working with a container-based workload. It exposes a lot of these things. But, we’re not where we were two or three years ago where this would’ve been a huge, scary, problem. These things have existing solutions, which are not that difficult to implement, even today.

IVAN: And there are some free tiers as well that you can use, especially if you don’t have a high volume of emails that you’re sending out.

TESS: If you’re only sending 500 emails a day, you can configure your G Suite email as the SMTP provider.

IVAN: Exactly. What about cron? Isn’t that a problem too?

TESS: Cron is a little bit different in Kubernetes. So, the thing with cron is that, in Kubernetes, cron isn’t just something that runs a command. In a traditional server workload, cron is some background process that exists in the system, and when a certain time shows up, it runs a certain command that you tell it to. And, it assumes that you’re running it on literally the same exact system that is running everything else, your web workload. Right?

IVAN: Right.

TESS: That’s not quite the case in Kubernetes. In Kubernetes, a cron job actually runs a container. So, when you actually have your web workload, you’re going to have one container, say, for Apache, somewhere, which is running your site. Then you have a cron job in Kubernetes, and that cron job will literally spin up a completely separate container in order to actually run that process.
So, that’s a bit different.

Now, the only real part of that which gets really confusing is, if you don’t have a nice separation of all of the different infrastructure we just finished talking about, if you don’t have any local disks that you need to worry about, if you don’t have SendMail you have to worry about, if you don’t have any of this stuff and you can scale out your web container to 10 or 20 or more, and not have a problem because they all rely on external API-based providers, then it doesn’t really matter what you do with cron. You just literally run the same container that you run for your web workload, with the same configuration and everything else, but you only tell it run a particular command, instead of "Run Apache." And that’s it. That’s what we do. And, it’s actually not very hard.

IVAN: What’s your favorite thing about Kubernetes? I’m only going to give you five minutes at the most. [laughing]

TESS: [laughing] I think the thing that I like the most about it, is probably the ability to easily scale things. Once you actually have solved all the underlying infrastructure problems, you basically have just a container-based workload that you can say, “I need to run three of these.” Then you can tell it and it will run three of them, and it will just run it, that’s it, you don’t need to worry about it. It already load balances it for you. How can I describe this? Well, let’s go back to the infamous car analogies again.

IVAN: They work.

TESS: They work, but you know they work within a US cultural context of a certain decade period, of a certain geographic location, but let’s put that aside for a second.

So, a car analogy. Let’s say you have a car, and you want to do some work on it. And you go to your garage and what do you see? The car and an empty garage. That’s often what a lot of other systems look like. When you have to do traditional clustering with regular virtual machines, or even self-hosted physical machines, you have to go over to your local hardware store, buy all the tools, buy the car jack, buy an engine lift, buy an air compressor and a whole bunch of other stuff, in order to do your car stuff, and it’s a lot of work and a lot of investment.

With Kubernetes, it’s more like, Okay, I go to my garage and I have Kubernetes. So I have all the tools already. All the tools are just there on the walls, right now. I can just start working. That’s what I really like about Kubernetes. It provides me a room with all the tools for me to actually make this workload do what I want it to do, rather than having to go and grab yet another thing, then another thing, then another thing. Then try to make compromises to make two things, which aren’t the thing that I can’t get right now, but they’re the two I have, to work together.

IVAN: I love the analogy. [laughing] I think that works, Tess. So, what about training? Wouldn’t it be great if, instead of trying to figure this all out yourself (like we did), you could just have us show you how to do it?

TESS: Gee, wouldn’t it? [laughing]

IVAN: Wouldn’t it be great? Well, guess what? That actually exists. We’re going to be doing some free trainings at BadCamp and then at DrupalCorn as well. We’ll be at BadCamp next month, the beginning of October. Now, they’re free trainings, but there is a cost of use to attending the training itself, so I think you have to register and it’s $20, or $10 at DrupalCorn. They’re free as far as we’re concerned.

Can you talk through, just a little bit about the format of the training that we have set up? What are you going to learn and who is it for?

TESS: So, we’ll briefly touch upon different kinds of Kubernetes hosting providers, as well as what Kubernetes actually is and what it does, and what it gives you. Then afterwards, we’re going to start containerizing your particular application. So, we’ll start working with containers, putting them onto Kubernetes, getting used to how to use Kubectl, how to work with individual definitions within Kubernetes, and making all of these pieces work together.

IVAN: And, it’s a four-hour workshop, it’s half a day, you get to spend time with Tess, and I think I’ll be there too. It’s going to be great. So, if you want to contribute to Flight Deck, or to Kubernetes, the Kubernetes Flight Deck Cluster that we have, we’d love it. It’s all online. You can visit ten7.com, and you’ll find it there on the what we give back page and you can also visit us on github.com/ten7, and you’ll see all the repos there. We’d love your help. Thank you, Tess, so much for spending your time with me today. This has been truly great.

TESS: Not a problem.

IVAN: So, if you need help with your own hosting, or figuring out what makes most sense to you, we’d love to be there to help you, whether you’re a developer or a large university, or a small business, it doesn’t matter. We’re happy to provide consulting, whether that means deploying your own Kubernetes or having us do it for you, or even selecting another vendor that makes the most sense to you.

Just send us an email and get in touch. You can reach us at [email protected]. You’ve been listening to the TEN7 Podcast. Find us online at ten7.com/podcast. And if you have a second, do send us a message. We love hearing from you. Our email address is [email protected]. And don’t forget, we’re also doing a survey of our listeners. So, if you’re able to, tell us about what you are and who you are, please take our survey as well at ten7.com/survey. Until next time, this is Ivan Stegic. Thank you for listening.

Sep 12 2019
Sep 12

The Hub - No Question It Can't Answer

The Bloomington Public Schools District in Bloomington, MN is a leader in harnessing technology to support student learning. We’ve been working with them since 2012, when we helped them move their K-12 site from a proprietary content management system that was expensive, inflexible and not mobile-friendly to a Drupal-powered site that provided more control over the site structure and content. We also added multi-site capabilities for the district’s 17 schools and departments, so they could manage their own content while retaining the broader district branding.

BPS has always been an early adopter of technology for school systems. However, back in 2012, there weren’t any student information systems like we have today; schools were still using paper planners to track homework. So they challenged us: build an online portal for teachers, students and their parents to access school data that was only accessible internally. The goal was to surface the data and make it easily consumable by everyone.

Nothing like this existed at the time, even in the Drupal world. The TEN7 development team wrote custom code (effectively creating original Drupal modules) to accomplish the task. The end result was a separate password-protected web application called “The Hub.” The Hub compiled data from third-party sources such as TIES, hundreds of Google calendars of students and teachers, data from school and district websites, as well as the district’s internal database.

The first release of The Hub was simple: a dashboard showing the students' class schedules and news feeds. However, the secure portal soon became more than a homework planning resource—The Hub has grown into a student, school and district information portal.

Bloomington Public Schools

New Features

  • Pathways to Graduation graph, which allow students (and their parents) to track their progress toward their post-secondary goals
    Bloomington Public Schools Pathways to Graduation
  • Data Warehouse, which holds lots of student info, such as student testing results, and even health records
    Bloomington Public Schools HUB Grades
    Bloomington Public Schools Hub Health Records
  • The Hub Digest emails, which give students and parents an overview of the upcoming week in their classes
  • Notes, which lets students, parents and teachers notes to any event or activity in the Hub
  • Students can add their own calendar events

“We have weekly meetings with TEN7, and they’re very innovative in coming up with creative solutions for the crazy ideas and problems we bring them. We serve 14,000 parents, almost 11,000 students and 1500 staff members—we have a huge backlog of crazy ideas. There’s always more we can give our stakeholders. With off-the-shelf software, you get what you get. But because we can do whatever we want with The Hub, it’s become the solution for lots of gnarly problems in our system. There are always more gnarly problems to solve and The Hub is almost always the answer.”
—Katrina Mezera, Digital Learning & Data Manager

The Hub is considered an indispensable tool by parents, students and staff. On any given day, thousands of students (up to 50% of the student body) use The Hub. Other school districts have reviewed and admired the functionality of the portal. Bloomington Public Schools presented The Hub at the 2015 Google in Education Conference in Mankato, MN, and other school districts inquired about having the same system implemented for their district.

TEN7 continues to collaborate with Bloomington Public Schools to add features and refine the user experience.

Aug 13 2019
Aug 13

Our Family Wizard Home page

Website Redesign: A Professional Face for a Market Leader

When parents divorce or separate, the child often becomes the unwilling intermediary for communication. Our Family Wizard (OFW) is a mobile application for co-parenting exes that facilitates and tracks communication, helps coordinate child duties and stores important information. The app includes a shared calendar, messaging component (with a “read” stamp), school and medical information and an expense log. 

People are usually referred to the site by judges, attorneys and mediators during separation or divorce proceedings. Often a judge or mediator requires that the parents use the application to document any child-related communication. 

Our Family Wizard Mother and Child

Our Family Wizard is the market leader in their vertical, but they felt their website did not exhibit the gravitas, credibility and professionalism that it should. TEN7 had been working with Avirat (the parent company of OFW) since 2015, supporting the existing OFW site, and they hired us to completely make over their website in both content and form. We went through a complete discovery process to ensure we completely understood the needs and desires of the client before we launched into design and content strategy for the new site.

The site redesign focused on the following goals:

Focus the Site to Drive Signups

The entire purpose of the ourfamilywizard.com website is to get the site visitor to sign up for the service and download the app. The new site has been streamlined to focus on two clearly defined calls to action: learn more and get started/sign up. The new design also makes use of a persistent “sticky bar” on the homepage with these two prompts. 

Redesign the Site to Make it More Modern and Professional 

In describing the desired site look and feel, the client mentioned pharmaceutical sites. Pharma sites tend to have a very clean design, large images and clear messaging. But more than that, pharma sites serve two audiences: professionals (doctors) and users (patients). Our Family Wizard also needs to market to two distinct user groups with distinct needs. 

Whereas the old site had a very “default site” feel, the new site feels more alive. Our designer Eva Lovisa paid attention to details big and small—from adding bigger photos and hero images to creating a brand library of icons. She created a bold color scheme with a lot of blues which served two purposes: blues tend to be calming (for the stressed out parents) and dark navy blues tend to have a regal, legal feel (for the professional visitors). She implemented a varied text style palette, choosing more distinctive fonts and creating things like quote and bullet styles to add more visual interest to what could have become a boring text-heavy site.

We gave a lot of consideration to the home page flow. Whereas the old home page felt like a mishmash of information, the new homepage is composed of clean, modular “stripes” of information. This solution is more than just good-looking; the stripes are created using the Paragraphs module in Drupal. Paragraphs functionality puts power in the hands of page editors to create paragraph types for common scenarios (image to the left of text, pull quotes, slideshows, etc.) that can be easily moved and edited.

“Instead of one giant HTML palette to work from, [with Paragraphs] we can work in pre-made chunks that look good. We have a lot more flexibility. We have a few content writers on staff, and the pages they’ve been able to create—the whole look and feel compared to the way they were doing it before—the difference is night and day.”
—Jai Kissoon, Avirat CEO

The mobile app was being redesigned at the same time as the website, so both will have the same identity. However, the website was also designed to be responsive, so it looks great on mobile devices.

Consolidate Information and Make It More Searchable

As part of the redesign, we migrated the Drupal 7 site to Drupal 8. A site redesign or migration is a great time to take stock of the organization and usability of your content items. Although this is primarily a signup site, it also houses a library of helpful content and resources for co-parents. With a long-running site like OFW, there can be an overabundance of categories, taxonomies and tags for the hundreds of pieces of content, and this hinders content usefulness and searchability. In addition, content was scattered in multiple areas around the site. 

We thought about which tags and topics we wanted, which URL structures we needed to change, and made decisions about single or multiple tagging. Each blog article had copies in multiple languages, so we had to properly tag and move those over.

We consolidated the content categories down into a manageable set, which we hope will last them for many years. We sorted and grouped related information to make it more easily navigable. For example, content about using the app and site, regional resource directories and other evergreen content was moved to a new Knowledge Center, found only on the website. Additional articles are found in the site blog. 

“Working with TEN7 has been a good fit. Things have been straightforward and easy. Les [Lim, Senior Developer] is a wizard in his own right. It’s his attitude, more than anything—his ability to help us get to resolution we want, which is sometimes in conflict with what’s easiest. We’re very happy.”
—Jai Kissoon, Avirat CEO 

The new site launched in May of 2018.

Aug 07 2019
Aug 07

Website Refresh: The Only Thing Missing is a Purring Sound

The Animal Humane Society (AHS), in Minneapolis, Minnesota is the leading animal welfare organization in the Upper Midwest, helping 25,000 dogs, cats and critters in need find loving homes each year, while providing a vast array of services to the community, from low-cost spay and neuter services to dog training to rescuing animals from neglectful and abusive situations. 

TEN7 has been working with AHS since 2008, making piecemeal updates to their website and finding creative solutions for desired changes with a limited budget. In 2016, the Animal Humane Society wanted to reimagine the animalhumanesociety.org website as not just an adoption source, but a resource, an authority, and an advocate for all things related to companion animals and the community that loves them. 

One of the main goals was to include even more information to support pet owners and animal lovers, including more photos, videos and shareable content. Other goals were to integrate the separate Kindest Cut website (a low-cost spay and neuter clinic) into the main site, and improve functionality of the Lost and Found bulletin boards.

“We wanted the user experience on the site to match the user experience when people come to the shelter. That it would be colorful and emotional and warm and inviting, and that it would give people that same wonderful feeling that they have when they walk in the door at the [shelter] and see the puppies and kittens.”—Paul Sorensen, Director of Brand and Communications, Animal Humane Society

To give AHS the increased functionality they desired (like the enhanced image and video capabilities), we embarked on building a complex Drupal 8 site from scratch. It was more than just a one-and-done update, however. Over a nine-year period, the site had evolved from a manually-updated custom CMS to a new Drupal 5 installation, and later Drupal 6. Additional functionality and one-off customizations to the codebase had created a great deal of technical debt, making the site difficult to maintain and support. 

Drupal 8 functionality allowed us to scrap some custom code, while in other cases we were able to replace custom code with contributed modules developed by the Drupal community. 

Integration with PetPoint (the animal information database) under Drupal 6 was challenging, requiring custom code from beginning to end. We were able to use Drupal 8’s built-in functionality to talk to PetPoint in a more standards-based way, which meant far less custom code.

As we were making these updates, we also followed best practices and implemented coding standards for the new site, which reduce the amount of technical debt that was created.

We launched the site in the summer of 2017, and although there were some hiccups, results were immediate: people LOVED the bold photos, video and shareable content. As a result of the site update, more Minnesotans are:

  • Visiting the website and staying longer. Traffic is up 8.5% from the previous year, and the average visit is over four minutes, up 8.6% percent from the previous year
  • Viewing animal profiles, with nearly 4 million views, leading to 10,751 animal adoptions
  • Sharing and responding to AHS content on social media, with double and triple-digit traffic increases on Twitter, Instagram, LinkedIn and Reddit
  • Donating online, with donations driven by site content up 18.2% from the previous year

We continue to support and collaborate with the Animal Humane Society, adding more functionality we couldn’t squeeze in during the big update, like setting up visitor accounts with the ability to “favorite” animals. And we still have to figure out how to make the site purr.

Jul 26 2019
Jul 26

We’re excited to be hosting the August 2019 TC Drupal monthly meeting, the first in the new "Lunch and Learn" format, which will rotate venues and feature a myriad speakers. See Allie's post about the change on the Twin Cities Drupal.org page. We look forward to them each month!

August Talk

TEN7’s Project Lead Les Lim will be giving a talk titled "Latest Paradigms in Content Editing: Paragraphs, Layout Builder, and Gutenberg" where he'll review the different available paradigms and where they are useful. 

Lunch and Learn Details

There's no registration required, just show up! Space is limited to 50 people though, so show up early!

When: Friday August 16, 12-1 p.m.
Where: Bde Maka Ska Room, Walker Library, 2880 Hennepin Ave, Minneapolis
Lunch: Pizza! (provided by TEN7)
Parking: The Walker Library has a paid parking garage (for a minimal fee). We recommend using it as Walker Library is in the heart of Uptown where street parking is seriously challenging.

More Info

Jul 23 2019
Jul 23

Solhem Companies

Website Refresh - Making It Easy to Decide on Apartments from Afar

Solhem Companies develops, owns and manages award-winning residential and office properties in Minneapolis, Minnesota’s most desirable neighborhoods. TEN7 has been working with Solhem since the beginning. We built their first Drupal site in 2009, and as they added buildings, the site soon evolved into a multi-site installation for their portfolio of boutique apartment properties. The Solhem websites, for existing properties as well as websites for properties under development, are managed with a single theme. In addition to providing cohesive visual branding for all the sites, this approach results in reduced support and maintenance costs, since updates are performed in one place and changes are propagated to all sites. Additionally, it spreads out the investment being made over multiple sites.

Solhem approached TEN7 in 2016, with a wish list that included developing a new theme that would differentiate them from their competitors' sites, that generally use a standard real estate website template, the one that’s provided from their property management software and usually can be quite boring. CLient's goal was to visually identify their buildings as belonging to the Solhem group of properties, by embarking on a theme redesign that would match the Scandinavian ambiance of their buildings, sunny and light, friendly, modern and sustainable.

The design process went through a few iterations, until all of the client stakeholders were satisfied. Both teams were able to collaborate using InVision, a design prototyping platform. Eva, our UX designer, brought their vision to life in digital form, and our talented front-end team implemented the designs with great care. This project is a perfect example of how the design phase can take longer than planned, and how that extra time contributes to a more considered, user-focused site.

Solhem old layout The old theme with a short-scroll home page as seen on the former Soltva site. New Solhavn site with long scroll The new minimalist theme with a long-scroll home page as featured on the new Solhavn site.

Meeting Client's Goals

Solhem said, “One of our goals is to get our websites to the point where someone who lives in LA can get enough information from the site to confidently apply for a unit, without ever visiting us in person.” One feature that helps them fulfill that goal is the custom floor plans page we built for the sites. All sites have both 2D and 3D plans, with photos and virtual tours. On some sites, you can even select a floor and see which floor plan types are available on that floor. 

Solhem Companies floor plans

A Guide to the Neighborhood

Knowledge of the buildings' surrounding neighborhoods is vital. Solhem wanted to feature Instagram content from neighborhood amenities like attractions, food & drink, recreation and shopping. They also wanted to highlight selected Instagram content from the community of residents from each building on an updated version of their blog page. Obtaining these images required getting permissions from the content creators, a sensitive issue that we reviewed carefully with the client. 

A Valuable Editing Tool

In order for the Solhem staff to curate and update Instagram posts, Jason Cote, our front-end developer, created a custom Drupal module that allows site administrators to retrieve and display Instagram images (including information about the Instagram account owner) from within the site’s administration pages. “It was a farfetched idea in the design phase, and I had no idea that could be brought to life! But Jason figured out how to do it,” said Megan Glover, Solhem Companies’ Marketing Manager. “I’m already seeing other people steal this idea from us.”  It took a little extra development time upfront to make it work, but it will save Solhem time and money in the long run.

Living amenities in the North Loop Instagram content on the Solhavn site.

“We have a lot of awesome residents who are good at Instagram, as well as neighborhood businesses with great content. To be able to present this content keeps us totally on-brand, and it’s a WIN-WIN for everyone. We get fresh beautiful content and get to promote people’s Instagram accounts, and get them new followers!” —Megan Glover, Marketing Manager, Solhem Companies

Outcome

TEN7 completed the Solhem redesign project on time and within budget, despite some unanticipated complications. It takes a surprising amount of work to render a clean, minimalist and elegant theme that works for multiple websites, while still being fully responsive!

The work was well worth it.

“Conversion rates and SEO have all been improved with the new design, and I can’t tell you the number of people who have said, ‘I LOVE your website, and it’s so easy to use!’ Oftentimes people come in to tour, and say they made up their mind by just viewing the website.”
—Megan Glover

Onward

So what’s next for Solhem? Tastes and design trends change quickly. We have been working working on a new theme design with updated fonts featuring more vibrant color, just in time for their new Nordeast Minneapolis location, coming in 2020. Stay tuned.

Jun 20 2019
Jun 20

We just moved our Drupal site to DigitalOcean and powered it with fully open-source, Kubernetes infrastructure that you could also be using. This is thrilling for us, and will be for you too!

Kubernetes Blog Header

A Little Background

Our very own Tess has been giving the session, “Return of the Clustering: Kubernetes for Drupal” during the last few months at Drupal camps across the U.S. In it she explains how it’s theoretically possible to deploy Drupal in production on a Kubernetes infrastructure. Drupal and Kubernetes don’t naturally play together, but we’ve done some serious work in devising a solution in which they would. For the last six months, Tess has been deep in research and development testing a solution, and last week we were confident enough to launch our own TEN7 site using it. 

Why is This a Big Deal?

Running a Drupal site on Docker in a production environment using open sourced Kubernetes  object descriptors has never been done before—not until now, and it’s all open source.  

“For the longest time, we’ve had a disjunction between what we use to develop sites locally, test sites locally, and what ends up on production environment,” said Tess. “It happens a lot, it’s difficult to get it to be the same thing. You need many more bits of technology, best practices, and discipline to pull it off. Kubernetes is becoming the de facto clustering standard, and they run Docker containers. We already do that locally. We’ve invested time and effort into developing those containers so they run in production.”

Using Kubernetes also solves a problem with scalability on VPS (Virtual Private Server) and shared hosting. “There’s a happy trough in the price curve for hosting at the far end,” said Tess. “If you don’t need the hosting to do much, you can do a lot for very little. But as soon as you get more traffic and need it to do more things, it gets really expensive.” 

“If your only choice is to make the server bigger, that has exponential cost. If you wanted to avoid that, you can scale out instead of scaling up. If you scale out, you need technology to have servers to coordinate with each other. That’s where Kubernetes comes in: it orchestrates the containers on the servers in a timely and flexible fashion.” 

You used to need hosting companies to do the hard work of infrastructure and container orchestration, but Kubernetes does all of the hard work for you. “You don’t have to build cluster orchestration, and you don't have to solve complex problems,” said Tess.

We still needed a partner to furnish the Kubernetes instance, and we chose DigitalOcean. They were one of the first companies to offer Kubernetes infrastructure. “DigitalOcean support for Kubernetes is impressive,” said Tess. “I didn’t expect a hosting company to offer that amount of flexibility, versatility and support versus the cost. We could do this without DigitalOcean, but we really like them. They’re more approachable than Google.”

Why This is Awesome for You

The success of this solution means we can roll out Kubernetes-based Drupal site hosting for our clients. And because we’ve open sourced the description and setup of the clustering itself, you could do it for yourself and your own clients too!

More Control Over Your Site and Fewer Hosting Limitations

The hosting on which your website code resides is the silent partner of your website, and it can either enable your website to be powerful, or curtail its development. 

Agencies that build websites typically don’t do the hosting for clients; they usually just farm it out to hosting companies with whom they have a business relationship. But your website will pay the price if the hosting isn’t up to snuff. “There are a lot of things that you can’t do with some of the popular hosting companies,” said Tess. “If you want certain capabilities, you might get into the ‘if you have to ask you can’t afford it’ category.” 

“Even if the providers let you do certain things, they might make it so you have to modify Drupal in ways to make it work on that particular host,” said Tess. “As a result, it won’t work well elsewhere. We call this ‘vendor lock.’” 

What we’ve done is create an open-source clustering standard that isn’t tied to hosting. “We're using open source containers and orchestration, and Kubernetes has a vendor isolation layer built in,” said Tess. “All this allows you to avoid vendor lock entirely! Our partner DigitalOcean furnishes the Kubernetes instance and they’ve installed the infrastructure around it. There’s nothing vendor-specific about it. We’re just allocating the cluster.” We use their object store called Spaces for file storage, but you could use any other S3 compatible object store on the market.

This Solution Helps Us Help You

The benefit of Kubernetes-based hosting for TEN7 clients is huge: “We can offer purely open source hosting that provides you top-tier hosting functionality, and we can do that at a lower price point than our competitors can,” said Ivan Stegic. “What we’re offering comparatively blows a lot of other agencies out of the water. For a comparable price, we can offer more scalability, bandwidth and versatility, which means we can do more ambitious sites. We’ll also have better tuning and finer control over the environment, allowing us to support our clients better.” 

Using this Kubernetes solution also aligns with TEN7’s love of open source products. “We want to be independent, and using a hosting solution that is supported in the open source and is vendor agnostic, which aligns with our commitment to open-source products and services,” says Ivan Stegic. 

You Could Host It Yourself Too (and We Won’t Be Offended)

We have long relationships with our clients, but if at some point they want to do their own Kubernetes hosting in the future, they can! They can pull the same containers and workflow we’ve been using, and put it on their own Kubernetes host. Since we’ve already architected the hosting, reusing our solution will be easier. “You don’t need to be nearly as much of an architect as I had to be in the first place!” Tess said.

We’d Love to Care for Your Site

We’ll be rolling out this solution to existing TEN7 Care clients over the next few months. In the future, we’ll experiment in setting up a Flight Deck cluster, for people to set up Kubernetes hosting for themselves!

Drop us a note if you’d like TEN7 to care for your Drupal site!
 

Jun 19 2019
Jun 19

Your browser does not support the audio element. TEN7-Podcast-Ep-062-2019-Flyover-Camp.mp3

Our frequent DrupalCamp attender and speaker, DevOps Tess Flynn, returns to the podcast to recap her recent experience at Flyover Camp, a brand new Drupal camp in Kansas City, Missouri.

Host: Ivan Stegic
Guest: Tess Flynn, DevOps Engineer at TEN7

Podcast highlights: 

  • We in the midwest totally own the “flyover” jokes!
  • The continuing diversity in camp talks (business, self-care, human focus tracks)
  • Tess reviews both her talks (Return of the Clustering: Kubernetes for Drupal, and Health Check Your Site)
  • How you should stretch your mind to prepare for all the rapid-fire information you get in the Kubernetes talk!
  • Location, location, location is as important for conference talks as it is for real estate
  • Listen to Ivan and Tess geek out over the Raspberry Pi session

Links:

TRANSCRIPT

IVAN STEGIC: Hey everyone you're listening to the TEN7 Podcast, where we get together every fortnight and sometimes more often, to talk about technology, business, and the humans in it. I'm your host Ivan Stegic. Let's talk about Drupal Flyover Camp 2019, that happened from Friday the 31st of May to Sunday, the 2nd of June, in Kansas City. Joining me to give her thoughts is socketwench. That's wench, not wrench. Welcome back to the podcast.

TESS FLYNN: Hello.

IVAN: Now, did I say it the right way, because I know you always have a specific way of saying it when you give your intro to socketwench.

TESS: Well, that’s pretty close. That works.

IVAN: Close? Okay. Good. So, you were at a Flyover Camp. What's in a name? I just love how Flyover Camp were poking fun at themselves in Kansas. I mean, we're pretty much in flyover land here in Minneapolis too, so I totally get it.

TESS: [laughing] So let's first frame what that is because if we're having international listeners, they might not get what the reference is.

IVAN: Good idea.

TESS: So, the thing that goes with it is, if you're from the Midwest you're considered in flyover country. And the reason why is because the joke goes, that there is nothing in the United States that's of interest unless if you're on either coast, which is actually completely untrue. However, that is what a lot of people tend to think of it. So as a result, if you're in the Midwest you kind of go, Well, you know, what we're going to own that turf.

IVAN: Exactly.

TESS: We're going to go and names things after it and take that world.

IVAN: I love it. I love that they did that. Drupal Flyover Camp in Kansas City, Missouri. And so, this is a brand-new camp, right? This is the first time they've ever done this camp. How great is that? We have a new camp on the schedule.

TESS: Yeah, I was surprised that it was new because they hit everything running. It felt like this was a well-oiled machine for a camp.

IVAN: That's wonderful. It's wonderful to have that on the calendar again. So, well-oiled machine. Did you recognize any of the organizers? Maybe these people have done it before.

TESS: I think that I recognized a few people from…oh, what is their name…VML and YL. What are they called now, because they merged with somebody?

IVAN: I don't know.

TESS: VML Y&R. Wow, that is a mouthful.

IVAN: What?

TESS: Victor, Mike, Lima, Yankee and Romeo.

IVAN: Okay. What are they, a global marketing agency that needs a new name? [laughing]

TESS: [laughing] That is their new name. [laughing]

IVAN: [laughing] Okay. I don't even know how to say it.

TESS: They used to be two different companies that got merged, and this is the resulting name.

IVAN: Oh, it's on their BOF page. If you're looking for something about VML you can still visit the VML website. If you're looking for something about Y&R, don’t sweat, you can still visit the Y&R. So, it's basically like you said, a concatenation of their former names. Maybe it's just temporary. Ok. A little bit of a tangent. [laughing] So, some sort of experience in Flyover Camp organization. Sounds like you said they were a well-oiled machine. It was a three-day camp?

TESS: I believe so. There was a day of trainings which I did not attend, and then two days of sessions, which actually has been bucking the trend lately.

IVAN: Yeah. And also, from what I can tell there were contributions as well on Sunday so, maybe it was a four-day camp, if there were trainings as well.

TESS: Might’ve been.

IVAN: Yeah. So, you were there Friday and Saturday. It looked like they had numerous tracks. So, I thought, usually these camps have five tracks and then you have five rooms and people go to the room for the track that they're interested in. This felt like it had a dozen tracks, but three rooms and it sort of was interspersed track sessions and BOFs as well amongst these three rooms. Is that what it was like? I mean I’m only gauging from the website.

TESS: So, you know the thing with the tracks is that a lot of the time it depends on how promoted they are as their own top-level entity in the data, as it were. And some camps do a very good job of this, that they have this track, this track and this track. I think DrupalCon recently reorganized so that there's only particular tracks that they directly advertise to different audiences, like a business audience, a frontend audience, something like that. Some camps have a lot of tracks and they're not particularly consistently organized, or if they are, it doesn't feel like that when you're attending because you don't tend to notice it, and Flyover Camp seemed to fall into this latter category. That's not bad but it's just a thing.

IVAN: Yeah. And I love that the tracks were so diverse as well, right? There was security, QA, site building, the usual frontend/backend stuff and there was a self-care track as well. I mean, more of that please. More mental health stuff, more business stuff, more human focus sessions. I love it.

TESS: Mm hmm.

IVAN: I love it. I think that's awesome. And, it looked like there were about 30 sessions, so similar to Drupaldelphia and those 30 sessions were spread across two days as opposed to one day at Drupaldelphia.

TESS: Yeah and it seemed to attract a lot of people from the area. I mean I was there from Minnesota and I saw people that usually I see at DrupalCorn there as well. So, it attracted a lot of people from the Midwest.

IVAN: That's wonderful. And there were BOFs as well, and it kind of looked like they were spread out across the two days as well.

TESS: I think there were, but I was so focused on other stuff that I completely missed it.

IVAN: Yeah. This was a heck of a camp for you. I mean it wasn't one session it was two sessions.

TESS: Oh man, it was a double feature. That was hard.

IVAN: I'm sure you absolutely shone on that and I'm sure you did really well. So, let's talk about those two sessions. So, your first session on Friday was the famous cloaked talk, Kubernetes called Return of the Clustering, right? The third part of the trilogy. So that was Friday. And then Saturday you gave a talk essentially about the Healthcheck module, right? What can you do to keep tabs on the health of your Drupal site?

TESS: Well, it was also about site auditing as well, in general.

IVAN: That’s right, and site auditing. So, I guess the critical question here is, did you wear a costume for both talks?

TESS: So, here's the problem with that. I don't have a car. And in order to actually get the costume for that one I would have probably had to rent a car to go to a local thrift store chain called Ax Man surplus and see if I could find like a stethoscope or whatever that little satellite dish head gear thing that they wear, I forget what it's called, and see if I could shove one of those into my luggage. But I didn't have the time to do that. Every weekend that I've had lately has just been completely booked up.

IVAN: Well maybe we'll have to work on that if you get asked to do that talk again and we'll figure out another costume for you.

TESS: Well, rumor has it that's going to happen. [laughing]

IVAN: [laughing] So, comparatively, how were the two sessions attended? Was there a drop of people on Saturday compared to Friday or was it comparable?

TESS: It was actually the other way around. I think a lot of people find the Kubernetes talk is fun, but it can be very intimidating because it seems like, “oh, that's a very devopsy, very technical talk and it's going to be way over my head.” And I was able to attract some people to come to it, especially by making a fool out of myself, by dressing up like a Jedi and standing outside of my door waving a lightsaber to have people come and join the session. But, it was a smaller room and it was still well attended, but the site audit talk actually had a lot more people in it, mostly because it was also in the main auditorium, so a lot of people who were just there were also just there, but there was a lot of people paying attention to it as well, because it tends to be a really fun, engaging talk and it tends to appeal to a much broader audience than the Kubernetes talk, which tends to be more infrastructury devopsy people. Even though I try to make that as broadly appealing as I can.

IVAN: So, location, location, location. Right. You had a wonderful location in the auditorium for that talk.

TESS: Yeah. The only downside is that when you're in an auditorium you're usually on a pedestal or a dais or something like that, and the problem is that it sounds like I'm a T-Rex walking around on stage, because the thing is hollow so the microphone just picks up everything, and I don't tend to stay still when I give a talk, I tend to gesticulate and walk around and do lots of weird things.

IVAN: Jump around I believe you do as well. [laughing]

TESS: [laughing] Yeah. Well, I think the site audit talk, I also fall to my knees at one point, dramatically. [laughing]

IVAN: [laughing] It's a good talk.

TESS: I still remember skinning my knee at DrupalCorn.

IVAN: Well, it’s a good talk. I think it gets valuable as you do that. It certainly reminds people how important it is. Right? So, what do you think the biggest question was that people had from that health check talk, from that audit talk?

TESS: You know, I didn't get many questions. I was actually thinking about this a few days ago. I tend not to get that many questions directly after a talk because usually my talks last the entire amount of time, and afterwards, I have to rush out the door for the next person to start setting up their talk. And usually I don't get many questions, and I do try to anticipate a lot of the potential questions as well within the contents of the talk. So sometimes people will come by and ask me questions later, but that hasn't happened lately.

IVAN: It's a similar sort of thing for both talks then.

TESS: Yeah. I did have a nice conversation with someone, I think they're from the U of Kansas. I forget. I remember their face. I know that they go to DrupalCorn regularly too, but they were telling me about Kubernetes operators and all of that nifty technical stuff and that was a really interesting conversation to have, but it really wasn't a question.

IVAN: Well that actually leads me into my next question. Usually you're the one educating people about whatever you're talking about. What do you think your biggest takeaway from a session was in Flyover camp? What did you learn from each of them?

TESS: Ok. Geez, I’m trying to remember all the sessions I went to because Twin Cities Drupal was last weekend, and now I'm trying to remember any of the sessions I went to.

IVAN: Oh, I think maybe you misunderstood. What I meant was, what did you learn in your talk from the audience?

TESS: Oh. So, one thing that definitely occurred to me is that, when it comes to the Kubernetes talk is, just how much technical knowledge you need, all technical terms you need to pick up very, very quickly to get anywhere with understanding Kubernetes without feeling like you're “drowning,” in technical terms, all of a sudden. And I certainly had that experience myself just trying to learn Kubernetes in the first place and that is after having a very strong background in how containers work and how Docker works and some of the top terminologies I picked up from running production workloads in Dockers form, and I realized that after that talk, like, wow, in 45 minutes I take you from, you kind of, sort of know what Docker is, and you might have heard of Ansible, but you don't know too much about it, to, here’s how you run a Drupal site in production on Kubernetes using a simple effective formula. And that kind of struck me as Wow, not many people are doing that because, wow that can be really complicated.

IVAN: Yeah, it's the bleeding edge of it isn't it?

TESS: It's not just the bleeding edge, it’s just that the underlying design that I went for strives for minimalism and simplicity, and a lot of people find that appealing because it reduces the number of working parts that you have to know. A good example would be memcached. The way that it's presented in the talk is as a stateful set and that works. A lot of people will say, "What you should do is run it as another object called a daemon set." But in order to introduce a daemon set, I'd have to introduce a completely new object type that only works for that, and afterwards it's like, "Is that really necessary to talk about it?” “How often do you add or remove notes?” If you are already thinking about adding and removing notes, you're probably going to look up this stuff for you. So, I don't need to actually tell you about this in this talk. [laughing]

IVAN: Yeah, I love that you're able to educate people in one session even at a very high level. To go from, kind of knowing to, being interested in the technology and in what we're doing and in being interested in continuing to find out more. And, maybe that's a good reason to do a separate podcast just on the talk you gave and the contents of the talk and why are we doing that? Why is TEN7 investing as much as we are in Kubernetes and in Docker and in Drupal, and, you know, sending you to all of these camps, and then putting all of this work into the open source domain? Like, maybe there's enough there to talk about. I mean, just from my perspective, we want to be independent, and using a hosting solution that is supported in the open source that is vendor agnostic. And, if we're doing it for ourselves, there's no reason why we couldn't put it out there and have others learn and leverage from it as well. So, we should probably talk about this a little more in a separate podcast.

TESS: That's not a bad idea at all.

IVAN: I love it. Okay. We'll do that. We'll ask Jonathan to make that happen for us. Okay, so, a little more about Kubernetes. I was looking through the schedule of talks and as you, Tess, know, Raspberry Pis are really near and dear to my heart. I've used them for many different things at home, most recently as an ad blocker for the whole network, but I saw that Jeff Geerling was at Flyover Camp, and he had a talk about the cluster of RPis, or the Raspberry Pis, that he's been building since 2012-2013, something like that, and how it taught him everything he knows about Kubernetes. Did you catch that talk by any small chance?

TESS: I actually did go to that talk.

IVAN: You're kidding?

TESS: Because I was like, Oh that sounds really fun and I'd like to see what he does. Is he going to use straight K8s or is he going to use that K3 that I heard about? And, what was funny to me is that I remember watching a talk that he gave, not about Kubernetes, but about Ansible. Way, way, way back in the day at MidCamp with a very similar block of a Raspberry Pi cluster in a box. And I really wanted to see what he was going to do with this. So, sure, I went to it.

IVAN: And was it everything you wished it could be? I mean, I looked at the slides and there was a shout out to socketwench in one of the slides.

TESS: Yeah, I was like thankful. I was in the front row and no one could see how I was blushing [laughing] the entire time. Like, Oh, stop talking about me please. This is your talk. [laughing]

IVAN: [laughing] That’s great. I mean, you guys are related and connected by Kubernetes, so, how wonderful that that would be the case. So, can you give me a quick synopsis of the talk? What was the nugget that you took out of it?

TESS: So, what was interesting is that Jeff has built a small Kubernetes cluster using a standard distribution Kubernetes to run on, I think it's four or five Raspberry Pis.

IVAN: I think it’s four.

TESS: I think it's four now. Four Raspberry Pis with a single ethernet switch with power over ethernet so that it reduces the amount of additional circuitry and cables he has to carry around to power them altogether.

IVAN: Hold up. Hold up. He's actually powering the RPis now through power over ethernet? That's amazing. Of course, you could.

TESS: You can get an adapter board for that.

IVAN: That's awesome.

TESS: It's not really complicated.

IVAN: That’s so great. I'm sorry. I totally interrupted you there. What was the nugget?

TESS: [laughing] Well he didn't even mention the power over ethernet except for one thing, but I was looking at the screenshots like, Oh, you’re using power over ethernet now. Nice. [laughing]

IVAN: Nice. That’s nice.

TESS: So, a lot of the talk was about how he was running his own personal site, using a Raspberry Pi cluster out of his home network. And, I used to run my own single node server out of a home network way back, many, many years ago. And there's a number of challenges that come with that out of the box. You tend not to get static IPs from most ISPs. They'll get a Dynamic IP, some of them don't like that you have a significant amount of outbound traffic or incoming traffic that's coming from the net and they may block you for that reason, if you're on a particular service tier. Some ISPs are better at that than others, it really depends. But running his own site on a Kubernetes cluster on Raspberry Pis it's like, it reminds me of this meme that I saw passed around Kubernetes Twitter a while ago, which is, the subtext is, I deployed my blog on Kubernetes and it's this big semitrailer and it has a toy truck trailer box in the middle of it, completely dwarfed by the full size trailer. [laughing] That’s kind of like, Yeah that's pretty accurate.

IVAN: Well, I mean if you ever get reddited or slashdotted, I guess maybe it'll survive?

TESS: [laughing] Kind of. There’s a degree of front side caching I think that he also used. This kind of a project always comes across to me as not a serious, You should use this instead of traditional hosting and more like, I wanted to see if I could do that and it would be fun and it's something to do and it's something that lets me learn by doing. And that's you know, a worthy pursuit in its own right.

IVAN: But if you look at the other side of that coin, you're hosting your own website, you own the hardware, you own the software, you own your site, you can see it, you're not putting the risk of hosting in another large company's data center, right? You own it all from top to bottom, and honestly if you have a small blog and you're using your ISPs connection, and you have this overkill of a Raspberry Pi cluster that is powering the static site, you're probably not going to ever get enough traffic to bring that thing down. You're probably fine.

TESS: Probably not.

IVAN: Yeah.

TESS: Although I think Jeff's site is Drupal 8.

IVAN: Oh [laughing] so, not static, not static. Well, I'm very jealous of you getting to see that talk. That must’ve been pretty cool. I'm hoping that maybe we can get Jeff on the podcast to talk about his cluster and what he's been through and how it's evolved soon. So, Jeff if you happen to be listening, watch out for an email from us about that.

Let's talk about diversity at Flyover Camp. What did it look like? Were there the kind of usual cast of people that look like I do, white males, or what did that look like amongst attendees and speakers this year?

TESS: So, there certainly is a large contingent of white straight cis male people there as well. There were a lot of women there as well, and there were several POC as well. I didn't actually take any moment to really do any kind of headcounts on that. It just never crossed my mind to log that kind of information. But I did sit with several people which were really fascinating and really interesting to talk to, and that was really nice.

IVAN: I hope we can have more of that and more attention to that in the future and we'll try to continue to talk about it and bring it up in our podcast as well. What about attendance as a whole at Flyover Camp? Was it comparable to Drupaldelphia or to Twin Cities Drupal Camp? Did you get a feel for what it was like?

TESS: I think that it was more closer to the size of Twin Cities Drupal than Drupaldelphia. Drupaldelphia had a surprising amount of people in it. And it could have been complicated by the space, because it was a smaller space than Flyover Camp or Twin Cities Drupal, but there was certainly a large number of people there.

IVAN: Any particular sessions besides Jeff’s, that were memorable to you?

TESS: Oh geez, I'd have to look it over because so much of it was kind of a blur. I was kind of sad that I missed John Rearick's session about 45 Modules and Forty-Five Minutes. I caught the end of it. But, yeah, that would have been a really interesting talk to go to, because literally every slide has a timer. So, the talk is only 45 minutes long. So every slide is only a minute long.

IVAN: So, it's kind of like an ignite session, where it's 30 seconds per slide, 20 slides, something like that?

TESS: Mm hmm. I saw one by Ria Dixon called CloudWatch-ing, which was all about creating logs and alerts using AWS CloudWatch. That was really fascinating. And, it makes me wonder if there is a way to create similar mechanisms and use similar strategies in a purely open source implementation that doesn't rely on AWS's productized version of that.

IVAN: What is CloudWatch?

TESS: It's kind of an event and log tracking mechanism meant for distributed logging. There's a lot of that I didn't get into because it was mostly a case study about how they implemented it, and how they solved their own problems. There was a lot of additional research that I'd love to circle back to, but it was a really good session and I really enjoyed it.

IVAN: So, is CloudWatch then, kind of similar to Splunk?

TESS: I think that it's a bit similar to Splunk. I know that there might be part of that that's similar to Prometheus and Grafana which is a common Kubernetes logging mechanism.

IVAN: Yeah, Prometheus is pretty widespread as well, isn't it?

TESS: Mm hmm.

IVAN: Yeah. Okay. So, a couple of good sessions. Generally, you had a good time at the camp, gave two wonderful talks. Where was the camp? Was it at the University?

TESS: I believe it was. It was a pretty good location, although because it is in the middle of Missouri, I did have a problem getting to and from the Camp, because I didn't have a car rental. So I ended up walking there and that was a 20 minute walk in Missouri in June, which was a bit warmer than I’m used to. [laughing]

IVAN: Yeah, I guess the flipside of that is it could have been Missouri in December or January.

TESS: I mean I would have been fine with that but that’s me, I like winter. [laughing]

IVAN: [laughing] Well, go figure. I like it too. So, what do you know? Okay. So, the event venue was good. The attendance was comparable to TCDrupal. And before we wrap up, overall impression of the event? If there's another one next year are you going to go?

TESS: I would love to go again. It was a lot of fun to go there. And it’s a lot more interesting than I had expected it to be, which kind of lives up to the name. [laughing]

IVAN: [laughing] That's great. Well, I appreciate the time you spent with us today once again. Thank you so much for being with me. It's really been a pleasure.

TESS: Mm hmm.

IVAN: Well, Tess Flynn or socketwench, is the Devops Engineer here at TEN7, and she was just at Drupal Flyover Camp 2019, where she gave her talk, “Return of the Clustering Kubernetes for Drupal.” Of course, that's the third in a trilogy and the other talk, “Dr. Upal Is In - Healthcheck your Site.” Those slides are all online and a recording of the sessions are also available. Just visit this episode's webpage for those links. You’ve been listening to the TEN7 Podcast. Find us online at ten7.com/podcast. And if you have a second, do send us a message, we love hearing from you. Our email address is [email protected]. Until next time, this is Ivan Stegic. Thank you for listening.

Jun 19 2019
Jun 19

Your browser does not support the audio element. TEN7-Podcast-Ep-062-2019-Flyover-Camp.mp3

Our frequent DrupalCamp attender and speaker, DevOps Tess Flynn, returns to the podcast to recap her recent experience at Flyover Camp, a brand new Drupal camp in Kansas City, Missouri.

Host: Ivan Stegic
Guest: Tess Flynn, DevOps Engineer at TEN7

Podcast highlights: 

  • We in the midwest totally own the “flyover” jokes!
  • The continuing diversity in camp talks (business, self-care, human focus tracks)
  • Tess reviews both her talks (Return of the Clustering: Kubernetes for Drupal, and Health Check Your Site)
  • How you should stretch your mind to prepare for all the rapid-fire information you get in the Kubernetes talk!
  • Location, location, location is as important for conference talks as it is for real estate
  • Listen to Ivan and Tess geek out over the Raspberry Pi session

Links:

TRANSCRIPT

IVAN STEGIC: Hey everyone you're listening to the TEN7 Podcast, where we get together every fortnight and sometimes more often, to talk about technology, business, and the humans in it. I'm your host Ivan Stegic. Let's talk about Drupal Flyover Camp 2019, that happened from Friday the 31st of May to Sunday, the 2nd of June, in Kansas City. Joining me to give her thoughts is socketwench. That's wench, not wrench. Welcome back to the podcast.

TESS FLYNN: Hello.

IVAN: Now, did I say it the right way, because I know you always have a specific way of saying it when you give your intro to socketwench.

TESS: Well, that’s pretty close. That works.

IVAN: Close? Okay. Good. So, you were at a Flyover Camp. What's in a name? I just love how Flyover Camp were poking fun at themselves in Kansas. I mean, we're pretty much in flyover land here in Minneapolis too, so I totally get it.

TESS: [laughing] So let's first frame what that is because if we're having international listeners, they might not get what the reference is.

IVAN: Good idea.

TESS: So, the thing that goes with it is, if you're from the Midwest you're considered in flyover country. And the reason why is because the joke goes, that there is nothing in the United States that's of interest unless if you're on either coast, which is actually completely untrue. However, that is what a lot of people tend to think of it. So as a result, if you're in the Midwest you kind of go, Well, you know, what we're going to own that turf.

IVAN: Exactly.

TESS: We're going to go and names things after it and take that world.

IVAN: I love it. I love that they did that. Drupal Flyover Camp in Kansas City, Missouri. And so, this is a brand-new camp, right? This is the first time they've ever done this camp. How great is that? We have a new camp on the schedule.

TESS: Yeah, I was surprised that it was new because they hit everything running. It felt like this was a well-oiled machine for a camp.

IVAN: That's wonderful. It's wonderful to have that on the calendar again. So, well-oiled machine. Did you recognize any of the organizers? Maybe these people have done it before.

TESS: I think that I recognized a few people from…oh, what is their name…VML and YL. What are they called now, because they merged with somebody?

IVAN: I don't know.

TESS: VML Y&R. Wow, that is a mouthful.

IVAN: What?

TESS: Victor, Mike, Lima, Yankee and Romeo.

IVAN: Okay. What are they, a global marketing agency that needs a new name? [laughing]

TESS: [laughing] That is their new name. [laughing]

IVAN: [laughing] Okay. I don't even know how to say it.

TESS: They used to be two different companies that got merged, and this is the resulting name.

IVAN: Oh, it's on their BOF page. If you're looking for something about VML you can still visit the VML website. If you're looking for something about Y&R, don’t sweat, you can still visit the Y&R. So, it's basically like you said, a concatenation of their former names. Maybe it's just temporary. Ok. A little bit of a tangent. [laughing] So, some sort of experience in Flyover Camp organization. Sounds like you said they were a well-oiled machine. It was a three-day camp?

TESS: I believe so. There was a day of trainings which I did not attend, and then two days of sessions, which actually has been bucking the trend lately.

IVAN: Yeah. And also, from what I can tell there were contributions as well on Sunday so, maybe it was a four-day camp, if there were trainings as well.

TESS: Might’ve been.

IVAN: Yeah. So, you were there Friday and Saturday. It looked like they had numerous tracks. So, I thought, usually these camps have five tracks and then you have five rooms and people go to the room for the track that they're interested in. This felt like it had a dozen tracks, but three rooms and it sort of was interspersed track sessions and BOFs as well amongst these three rooms. Is that what it was like? I mean I’m only gauging from the website.

TESS: So, you know the thing with the tracks is that a lot of the time it depends on how promoted they are as their own top-level entity in the data, as it were. And some camps do a very good job of this, that they have this track, this track and this track. I think DrupalCon recently reorganized so that there's only particular tracks that they directly advertise to different audiences, like a business audience, a frontend audience, something like that. Some camps have a lot of tracks and they're not particularly consistently organized, or if they are, it doesn't feel like that when you're attending because you don't tend to notice it, and Flyover Camp seemed to fall into this latter category. That's not bad but it's just a thing.

IVAN: Yeah. And I love that the tracks were so diverse as well, right? There was security, QA, site building, the usual frontend/backend stuff and there was a self-care track as well. I mean, more of that please. More mental health stuff, more business stuff, more human focus sessions. I love it.

TESS: Mm hmm.

IVAN: I love it. I think that's awesome. And, it looked like there were about 30 sessions, so similar to Drupaldelphia and those 30 sessions were spread across two days as opposed to one day at Drupaldelphia.

TESS: Yeah and it seemed to attract a lot of people from the area. I mean I was there from Minnesota and I saw people that usually I see at DrupalCorn there as well. So, it attracted a lot of people from the Midwest.

IVAN: That's wonderful. And there were BOFs as well, and it kind of looked like they were spread out across the two days as well.

TESS: I think there were, but I was so focused on other stuff that I completely missed it.

IVAN: Yeah. This was a heck of a camp for you. I mean it wasn't one session it was two sessions.

TESS: Oh man, it was a double feature. That was hard.

IVAN: I'm sure you absolutely shone on that and I'm sure you did really well. So, let's talk about those two sessions. So, your first session on Friday was the famous cloaked talk, Kubernetes called Return of the Clustering, right? The third part of the trilogy. So that was Friday. And then Saturday you gave a talk essentially about the Healthcheck module, right? What can you do to keep tabs on the health of your Drupal site?

TESS: Well, it was also about site auditing as well, in general.

IVAN: That’s right, and site auditing. So, I guess the critical question here is, did you wear a costume for both talks?

TESS: So, here's the problem with that. I don't have a car. And in order to actually get the costume for that one I would have probably had to rent a car to go to a local thrift store chain called Ax Man surplus and see if I could find like a stethoscope or whatever that little satellite dish head gear thing that they wear, I forget what it's called, and see if I could shove one of those into my luggage. But I didn't have the time to do that. Every weekend that I've had lately has just been completely booked up.

IVAN: Well maybe we'll have to work on that if you get asked to do that talk again and we'll figure out another costume for you.

TESS: Well, rumor has it that's going to happen. [laughing]

IVAN: [laughing] So, comparatively, how were the two sessions attended? Was there a drop of people on Saturday compared to Friday or was it comparable?

TESS: It was actually the other way around. I think a lot of people find the Kubernetes talk is fun, but it can be very intimidating because it seems like, “oh, that's a very devopsy, very technical talk and it's going to be way over my head.” And I was able to attract some people to come to it, especially by making a fool out of myself, by dressing up like a Jedi and standing outside of my door waving a lightsaber to have people come and join the session. But, it was a smaller room and it was still well attended, but the site audit talk actually had a lot more people in it, mostly because it was also in the main auditorium, so a lot of people who were just there were also just there, but there was a lot of people paying attention to it as well, because it tends to be a really fun, engaging talk and it tends to appeal to a much broader audience than the Kubernetes talk, which tends to be more infrastructury devopsy people. Even though I try to make that as broadly appealing as I can.

IVAN: So, location, location, location. Right. You had a wonderful location in the auditorium for that talk.

TESS: Yeah. The only downside is that when you're in an auditorium you're usually on a pedestal or a dais or something like that, and the problem is that it sounds like I'm a T-Rex walking around on stage, because the thing is hollow so the microphone just picks up everything, and I don't tend to stay still when I give a talk, I tend to gesticulate and walk around and do lots of weird things.

IVAN: Jump around I believe you do as well. [laughing]

TESS: [laughing] Yeah. Well, I think the site audit talk, I also fall to my knees at one point, dramatically. [laughing]

IVAN: [laughing] It's a good talk.

TESS: I still remember skinning my knee at DrupalCorn.

IVAN: Well, it’s a good talk. I think it gets valuable as you do that. It certainly reminds people how important it is. Right? So, what do you think the biggest question was that people had from that health check talk, from that audit talk?

TESS: You know, I didn't get many questions. I was actually thinking about this a few days ago. I tend not to get that many questions directly after a talk because usually my talks last the entire amount of time, and afterwards, I have to rush out the door for the next person to start setting up their talk. And usually I don't get many questions, and I do try to anticipate a lot of the potential questions as well within the contents of the talk. So sometimes people will come by and ask me questions later, but that hasn't happened lately.

IVAN: It's a similar sort of thing for both talks then.

TESS: Yeah. I did have a nice conversation with someone, I think they're from the U of Kansas. I forget. I remember their face. I know that they go to DrupalCorn regularly too, but they were telling me about Kubernetes operators and all of that nifty technical stuff and that was a really interesting conversation to have, but it really wasn't a question.

IVAN: Well that actually leads me into my next question. Usually you're the one educating people about whatever you're talking about. What do you think your biggest takeaway from a session was in Flyover camp? What did you learn from each of them?

TESS: Ok. Geez, I’m trying to remember all the sessions I went to because Twin Cities Drupal was last weekend, and now I'm trying to remember any of the sessions I went to.

IVAN: Oh, I think maybe you misunderstood. What I meant was, what did you learn in your talk from the audience?

TESS: Oh. So, one thing that definitely occurred to me is that, when it comes to the Kubernetes talk is, just how much technical knowledge you need, all technical terms you need to pick up very, very quickly to get anywhere with understanding Kubernetes without feeling like you're “drowning,” in technical terms, all of a sudden. And I certainly had that experience myself just trying to learn Kubernetes in the first place and that is after having a very strong background in how containers work and how Docker works and some of the top terminologies I picked up from running production workloads in Dockers form, and I realized that after that talk, like, wow, in 45 minutes I take you from, you kind of, sort of know what Docker is, and you might have heard of Ansible, but you don't know too much about it, to, here’s how you run a Drupal site in production on Kubernetes using a simple effective formula. And that kind of struck me as Wow, not many people are doing that because, wow that can be really complicated.

IVAN: Yeah, it's the bleeding edge of it isn't it?

TESS: It's not just the bleeding edge, it’s just that the underlying design that I went for strives for minimalism and simplicity, and a lot of people find that appealing because it reduces the number of working parts that you have to know. A good example would be memcached. The way that it's presented in the talk is as a stateful set and that works. A lot of people will say, "What you should do is run it as another object called a daemon set." But in order to introduce a daemon set, I'd have to introduce a completely new object type that only works for that, and afterwards it's like, "Is that really necessary to talk about it?” “How often do you add or remove notes?” If you are already thinking about adding and removing notes, you're probably going to look up this stuff for you. So, I don't need to actually tell you about this in this talk. [laughing]

IVAN: Yeah, I love that you're able to educate people in one session even at a very high level. To go from, kind of knowing to, being interested in the technology and in what we're doing and in being interested in continuing to find out more. And, maybe that's a good reason to do a separate podcast just on the talk you gave and the contents of the talk and why are we doing that? Why is TEN7 investing as much as we are in Kubernetes and in Docker and in Drupal, and, you know, sending you to all of these camps, and then putting all of this work into the open source domain? Like, maybe there's enough there to talk about. I mean, just from my perspective, we want to be independent, and using a hosting solution that is supported in the open source that is vendor agnostic. And, if we're doing it for ourselves, there's no reason why we couldn't put it out there and have others learn and leverage from it as well. So, we should probably talk about this a little more in a separate podcast.

TESS: That's not a bad idea at all.

IVAN: I love it. Okay. We'll do that. We'll ask Jonathan to make that happen for us. Okay, so, a little more about Kubernetes. I was looking through the schedule of talks and as you, Tess, know, Raspberry Pis are really near and dear to my heart. I've used them for many different things at home, most recently as an ad blocker for the whole network, but I saw that Jeff Geerling was at Flyover Camp, and he had a talk about the cluster of RPis, or the Raspberry Pis, that he's been building since 2012-2013, something like that, and how it taught him everything he knows about Kubernetes. Did you catch that talk by any small chance?

TESS: I actually did go to that talk.

IVAN: You're kidding?

TESS: Because I was like, Oh that sounds really fun and I'd like to see what he does. Is he going to use straight K8s or is he going to use that K3 that I heard about? And, what was funny to me is that I remember watching a talk that he gave, not about Kubernetes, but about Ansible. Way, way, way back in the day at MidCamp with a very similar block of a Raspberry Pi cluster in a box. And I really wanted to see what he was going to do with this. So, sure, I went to it.

IVAN: And was it everything you wished it could be? I mean, I looked at the slides and there was a shout out to socketwench in one of the slides.

TESS: Yeah, I was like thankful. I was in the front row and no one could see how I was blushing [laughing] the entire time. Like, Oh, stop talking about me please. This is your talk. [laughing]

IVAN: [laughing] That’s great. I mean, you guys are related and connected by Kubernetes, so, how wonderful that that would be the case. So, can you give me a quick synopsis of the talk? What was the nugget that you took out of it?

TESS: So, what was interesting is that Jeff has built a small Kubernetes cluster using a standard distribution Kubernetes to run on, I think it's four or five Raspberry Pis.

IVAN: I think it’s four.

TESS: I think it's four now. Four Raspberry Pis with a single ethernet switch with power over ethernet so that it reduces the amount of additional circuitry and cables he has to carry around to power them altogether.

IVAN: Hold up. Hold up. He's actually powering the RPis now through power over ethernet? That's amazing. Of course, you could.

TESS: You can get an adapter board for that.

IVAN: That's awesome.

TESS: It's not really complicated.

IVAN: That’s so great. I'm sorry. I totally interrupted you there. What was the nugget?

TESS: [laughing] Well he didn't even mention the power over ethernet except for one thing, but I was looking at the screenshots like, Oh, you’re using power over ethernet now. Nice. [laughing]

IVAN: Nice. That’s nice.

TESS: So, a lot of the talk was about how he was running his own personal site, using a Raspberry Pi cluster out of his home network. And, I used to run my own single node server out of a home network way back, many, many years ago. And there's a number of challenges that come with that out of the box. You tend not to get static IPs from most ISPs. They'll get a Dynamic IP, some of them don't like that you have a significant amount of outbound traffic or incoming traffic that's coming from the net and they may block you for that reason, if you're on a particular service tier. Some ISPs are better at that than others, it really depends. But running his own site on a Kubernetes cluster on Raspberry Pis it's like, it reminds me of this meme that I saw passed around Kubernetes Twitter a while ago, which is, the subtext is, I deployed my blog on Kubernetes and it's this big semitrailer and it has a toy truck trailer box in the middle of it, completely dwarfed by the full size trailer. [laughing] That’s kind of like, Yeah that's pretty accurate.

IVAN: Well, I mean if you ever get reddited or slashdotted, I guess maybe it'll survive?

TESS: [laughing] Kind of. There’s a degree of front side caching I think that he also used. This kind of a project always comes across to me as not a serious, You should use this instead of traditional hosting and more like, I wanted to see if I could do that and it would be fun and it's something to do and it's something that lets me learn by doing. And that's you know, a worthy pursuit in its own right.

IVAN: But if you look at the other side of that coin, you're hosting your own website, you own the hardware, you own the software, you own your site, you can see it, you're not putting the risk of hosting in another large company's data center, right? You own it all from top to bottom, and honestly if you have a small blog and you're using your ISPs connection, and you have this overkill of a Raspberry Pi cluster that is powering the static site, you're probably not going to ever get enough traffic to bring that thing down. You're probably fine.

TESS: Probably not.

IVAN: Yeah.

TESS: Although I think Jeff's site is Drupal 8.

IVAN: Oh [laughing] so, not static, not static. Well, I'm very jealous of you getting to see that talk. That must’ve been pretty cool. I'm hoping that maybe we can get Jeff on the podcast to talk about his cluster and what he's been through and how it's evolved soon. So, Jeff if you happen to be listening, watch out for an email from us about that.

Let's talk about diversity at Flyover Camp. What did it look like? Were there the kind of usual cast of people that look like I do, white males, or what did that look like amongst attendees and speakers this year?

TESS: So, there certainly is a large contingent of white straight cis male people there as well. There were a lot of women there as well, and there were several POC as well. I didn't actually take any moment to really do any kind of headcounts on that. It just never crossed my mind to log that kind of information. But I did sit with several people which were really fascinating and really interesting to talk to, and that was really nice.

IVAN: I hope we can have more of that and more attention to that in the future and we'll try to continue to talk about it and bring it up in our podcast as well. What about attendance as a whole at Flyover Camp? Was it comparable to Drupaldelphia or to Twin Cities Drupal Camp? Did you get a feel for what it was like?

TESS: I think that it was more closer to the size of Twin Cities Drupal than Drupaldelphia. Drupaldelphia had a surprising amount of people in it. And it could have been complicated by the space, because it was a smaller space than Flyover Camp or Twin Cities Drupal, but there was certainly a large number of people there.

IVAN: Any particular sessions besides Jeff’s, that were memorable to you?

TESS: Oh geez, I'd have to look it over because so much of it was kind of a blur. I was kind of sad that I missed John Rearick's session about 45 Modules and Forty-Five Minutes. I caught the end of it. But, yeah, that would have been a really interesting talk to go to, because literally every slide has a timer. So, the talk is only 45 minutes long. So every slide is only a minute long.

IVAN: So, it's kind of like an ignite session, where it's 30 seconds per slide, 20 slides, something like that?

TESS: Mm hmm. I saw one by Ria Dixon called CloudWatch-ing, which was all about creating logs and alerts using AWS CloudWatch. That was really fascinating. And, it makes me wonder if there is a way to create similar mechanisms and use similar strategies in a purely open source implementation that doesn't rely on AWS's productized version of that.

IVAN: What is CloudWatch?

TESS: It's kind of an event and log tracking mechanism meant for distributed logging. There's a lot of that I didn't get into because it was mostly a case study about how they implemented it, and how they solved their own problems. There was a lot of additional research that I'd love to circle back to, but it was a really good session and I really enjoyed it.

IVAN: So, is CloudWatch then, kind of similar to Splunk?

TESS: I think that it's a bit similar to Splunk. I know that there might be part of that that's similar to Prometheus and Grafana which is a common Kubernetes logging mechanism.

IVAN: Yeah, Prometheus is pretty widespread as well, isn't it?

TESS: Mm hmm.

IVAN: Yeah. Okay. So, a couple of good sessions. Generally, you had a good time at the camp, gave two wonderful talks. Where was the camp? Was it at the University?

TESS: I believe it was. It was a pretty good location, although because it is in the middle of Missouri, I did have a problem getting to and from the Camp, because I didn't have a car rental. So I ended up walking there and that was a 20 minute walk in Missouri in June, which was a bit warmer than I’m used to. [laughing]

IVAN: Yeah, I guess the flipside of that is it could have been Missouri in December or January.

TESS: I mean I would have been fine with that but that’s me, I like winter. [laughing]

IVAN: [laughing] Well, go figure. I like it too. So, what do you know? Okay. So, the event venue was good. The attendance was comparable to TCDrupal. And before we wrap up, overall impression of the event? If there's another one next year are you going to go?

TESS: I would love to go again. It was a lot of fun to go there. And it’s a lot more interesting than I had expected it to be, which kind of lives up to the name. [laughing]

IVAN: [laughing] That's great. Well, I appreciate the time you spent with us today once again. Thank you so much for being with me. It's really been a pleasure.

TESS: Mm hmm.

IVAN: Well, Tess Flynn or socketwench, is the Devops Engineer here at TEN7, and she was just at Drupal Flyover Camp 2019, where she gave her talk, “Return of the Clustering Kubernetes for Drupal.” Of course, that's the third in a trilogy and the other talk, “Dr. Upal Is In - Healthcheck your Site.” Those slides are all online and a recording of the sessions are also available. Just visit this episode's webpage for those links. You’ve been listening to the TEN7 Podcast. Find us online at ten7.com/podcast. And if you have a second, do send us a message, we love hearing from you. Our email address is [email protected]. Until next time, this is Ivan Stegic. Thank you for listening.

Jun 19 2019
Jun 19

Your browser does not support the audio element. TEN7-Podcast-Ep-062-2019-Flyover-Camp.mp3

Our frequent DrupalCamp attender and speaker, DevOps Tess Flynn, returns to the podcast to recap her recent experience at Flyover Camp, a brand new Drupal camp in Kansas City, Missouri.

Host: Ivan Stegic
Guest: Tess Flynn, DevOps Engineer at TEN7

Podcast highlights: 

  • We in the midwest totally own the “flyover” jokes!
  • The continuing diversity in camp talks (business, self-care, human focus tracks)
  • Tess reviews both her talks (Return of the Clustering: Kubernetes for Drupal, and Health Check Your Site)
  • How you should stretch your mind to prepare for all the rapid-fire information you get in the Kubernetes talk!
  • Location, location, location is as important for conference talks as it is for real estate
  • Listen to Ivan and Tess geek out over the Raspberry Pi session

Links:

TRANSCRIPT

IVAN STEGIC: Hey everyone you're listening to the TEN7 Podcast, where we get together every fortnight and sometimes more often, to talk about technology, business, and the humans in it. I'm your host Ivan Stegic. Let's talk about Drupal Flyover Camp 2019, that happened from Friday the 31st of May to Sunday, the 2nd of June, in Kansas City. Joining me to give her thoughts is socketwench. That's wench, not wrench. Welcome back to the podcast.

TESS FLYNN: Hello.

IVAN: Now, did I say it the right way, because I know you always have a specific way of saying it when you give your intro to socketwench.

TESS: Well, that’s pretty close. That works.

IVAN: Close? Okay. Good. So, you were at a Flyover Camp. What's in a name? I just love how Flyover Camp were poking fun at themselves in Kansas. I mean, we're pretty much in flyover land here in Minneapolis too, so I totally get it.

TESS: [laughing] So let's first frame what that is because if we're having international listeners, they might not get what the reference is.

IVAN: Good idea.

TESS: So, the thing that goes with it is, if you're from the Midwest you're considered in flyover country. And the reason why is because the joke goes, that there is nothing in the United States that's of interest unless if you're on either coast, which is actually completely untrue. However, that is what a lot of people tend to think of it. So as a result, if you're in the Midwest you kind of go, Well, you know, what we're going to own that turf.

IVAN: Exactly.

TESS: We're going to go and names things after it and take that world.

IVAN: I love it. I love that they did that. Drupal Flyover Camp in Kansas City, Missouri. And so, this is a brand-new camp, right? This is the first time they've ever done this camp. How great is that? We have a new camp on the schedule.

TESS: Yeah, I was surprised that it was new because they hit everything running. It felt like this was a well-oiled machine for a camp.

IVAN: That's wonderful. It's wonderful to have that on the calendar again. So, well-oiled machine. Did you recognize any of the organizers? Maybe these people have done it before.

TESS: I think that I recognized a few people from…oh, what is their name…VML and YL. What are they called now, because they merged with somebody?

IVAN: I don't know.

TESS: VML Y&R. Wow, that is a mouthful.

IVAN: What?

TESS: Victor, Mike, Lima, Yankee and Romeo.

IVAN: Okay. What are they, a global marketing agency that needs a new name? [laughing]

TESS: [laughing] That is their new name. [laughing]

IVAN: [laughing] Okay. I don't even know how to say it.

TESS: They used to be two different companies that got merged, and this is the resulting name.

IVAN: Oh, it's on their BOF page. If you're looking for something about VML you can still visit the VML website. If you're looking for something about Y&R, don’t sweat, you can still visit the Y&R. So, it's basically like you said, a concatenation of their former names. Maybe it's just temporary. Ok. A little bit of a tangent. [laughing] So, some sort of experience in Flyover Camp organization. Sounds like you said they were a well-oiled machine. It was a three-day camp?

TESS: I believe so. There was a day of trainings which I did not attend, and then two days of sessions, which actually has been bucking the trend lately.

IVAN: Yeah. And also, from what I can tell there were contributions as well on Sunday so, maybe it was a four-day camp, if there were trainings as well.

TESS: Might’ve been.

IVAN: Yeah. So, you were there Friday and Saturday. It looked like they had numerous tracks. So, I thought, usually these camps have five tracks and then you have five rooms and people go to the room for the track that they're interested in. This felt like it had a dozen tracks, but three rooms and it sort of was interspersed track sessions and BOFs as well amongst these three rooms. Is that what it was like? I mean I’m only gauging from the website.

TESS: So, you know the thing with the tracks is that a lot of the time it depends on how promoted they are as their own top-level entity in the data, as it were. And some camps do a very good job of this, that they have this track, this track and this track. I think DrupalCon recently reorganized so that there's only particular tracks that they directly advertise to different audiences, like a business audience, a frontend audience, something like that. Some camps have a lot of tracks and they're not particularly consistently organized, or if they are, it doesn't feel like that when you're attending because you don't tend to notice it, and Flyover Camp seemed to fall into this latter category. That's not bad but it's just a thing.

IVAN: Yeah. And I love that the tracks were so diverse as well, right? There was security, QA, site building, the usual frontend/backend stuff and there was a self-care track as well. I mean, more of that please. More mental health stuff, more business stuff, more human focus sessions. I love it.

TESS: Mm hmm.

IVAN: I love it. I think that's awesome. And, it looked like there were about 30 sessions, so similar to Drupaldelphia and those 30 sessions were spread across two days as opposed to one day at Drupaldelphia.

TESS: Yeah and it seemed to attract a lot of people from the area. I mean I was there from Minnesota and I saw people that usually I see at DrupalCorn there as well. So, it attracted a lot of people from the Midwest.

IVAN: That's wonderful. And there were BOFs as well, and it kind of looked like they were spread out across the two days as well.

TESS: I think there were, but I was so focused on other stuff that I completely missed it.

IVAN: Yeah. This was a heck of a camp for you. I mean it wasn't one session it was two sessions.

TESS: Oh man, it was a double feature. That was hard.

IVAN: I'm sure you absolutely shone on that and I'm sure you did really well. So, let's talk about those two sessions. So, your first session on Friday was the famous cloaked talk, Kubernetes called Return of the Clustering, right? The third part of the trilogy. So that was Friday. And then Saturday you gave a talk essentially about the Healthcheck module, right? What can you do to keep tabs on the health of your Drupal site?

TESS: Well, it was also about site auditing as well, in general.

IVAN: That’s right, and site auditing. So, I guess the critical question here is, did you wear a costume for both talks?

TESS: So, here's the problem with that. I don't have a car. And in order to actually get the costume for that one I would have probably had to rent a car to go to a local thrift store chain called Ax Man surplus and see if I could find like a stethoscope or whatever that little satellite dish head gear thing that they wear, I forget what it's called, and see if I could shove one of those into my luggage. But I didn't have the time to do that. Every weekend that I've had lately has just been completely booked up.

IVAN: Well maybe we'll have to work on that if you get asked to do that talk again and we'll figure out another costume for you.

TESS: Well, rumor has it that's going to happen. [laughing]

IVAN: [laughing] So, comparatively, how were the two sessions attended? Was there a drop of people on Saturday compared to Friday or was it comparable?

TESS: It was actually the other way around. I think a lot of people find the Kubernetes talk is fun, but it can be very intimidating because it seems like, “oh, that's a very devopsy, very technical talk and it's going to be way over my head.” And I was able to attract some people to come to it, especially by making a fool out of myself, by dressing up like a Jedi and standing outside of my door waving a lightsaber to have people come and join the session. But, it was a smaller room and it was still well attended, but the site audit talk actually had a lot more people in it, mostly because it was also in the main auditorium, so a lot of people who were just there were also just there, but there was a lot of people paying attention to it as well, because it tends to be a really fun, engaging talk and it tends to appeal to a much broader audience than the Kubernetes talk, which tends to be more infrastructury devopsy people. Even though I try to make that as broadly appealing as I can.

IVAN: So, location, location, location. Right. You had a wonderful location in the auditorium for that talk.

TESS: Yeah. The only downside is that when you're in an auditorium you're usually on a pedestal or a dais or something like that, and the problem is that it sounds like I'm a T-Rex walking around on stage, because the thing is hollow so the microphone just picks up everything, and I don't tend to stay still when I give a talk, I tend to gesticulate and walk around and do lots of weird things.

IVAN: Jump around I believe you do as well. [laughing]

TESS: [laughing] Yeah. Well, I think the site audit talk, I also fall to my knees at one point, dramatically. [laughing]

IVAN: [laughing] It's a good talk.

TESS: I still remember skinning my knee at DrupalCorn.

IVAN: Well, it’s a good talk. I think it gets valuable as you do that. It certainly reminds people how important it is. Right? So, what do you think the biggest question was that people had from that health check talk, from that audit talk?

TESS: You know, I didn't get many questions. I was actually thinking about this a few days ago. I tend not to get that many questions directly after a talk because usually my talks last the entire amount of time, and afterwards, I have to rush out the door for the next person to start setting up their talk. And usually I don't get many questions, and I do try to anticipate a lot of the potential questions as well within the contents of the talk. So sometimes people will come by and ask me questions later, but that hasn't happened lately.

IVAN: It's a similar sort of thing for both talks then.

TESS: Yeah. I did have a nice conversation with someone, I think they're from the U of Kansas. I forget. I remember their face. I know that they go to DrupalCorn regularly too, but they were telling me about Kubernetes operators and all of that nifty technical stuff and that was a really interesting conversation to have, but it really wasn't a question.

IVAN: Well that actually leads me into my next question. Usually you're the one educating people about whatever you're talking about. What do you think your biggest takeaway from a session was in Flyover camp? What did you learn from each of them?

TESS: Ok. Geez, I’m trying to remember all the sessions I went to because Twin Cities Drupal was last weekend, and now I'm trying to remember any of the sessions I went to.

IVAN: Oh, I think maybe you misunderstood. What I meant was, what did you learn in your talk from the audience?

TESS: Oh. So, one thing that definitely occurred to me is that, when it comes to the Kubernetes talk is, just how much technical knowledge you need, all technical terms you need to pick up very, very quickly to get anywhere with understanding Kubernetes without feeling like you're “drowning,” in technical terms, all of a sudden. And I certainly had that experience myself just trying to learn Kubernetes in the first place and that is after having a very strong background in how containers work and how Docker works and some of the top terminologies I picked up from running production workloads in Dockers form, and I realized that after that talk, like, wow, in 45 minutes I take you from, you kind of, sort of know what Docker is, and you might have heard of Ansible, but you don't know too much about it, to, here’s how you run a Drupal site in production on Kubernetes using a simple effective formula. And that kind of struck me as Wow, not many people are doing that because, wow that can be really complicated.

IVAN: Yeah, it's the bleeding edge of it isn't it?

TESS: It's not just the bleeding edge, it’s just that the underlying design that I went for strives for minimalism and simplicity, and a lot of people find that appealing because it reduces the number of working parts that you have to know. A good example would be memcached. The way that it's presented in the talk is as a stateful set and that works. A lot of people will say, "What you should do is run it as another object called a daemon set." But in order to introduce a daemon set, I'd have to introduce a completely new object type that only works for that, and afterwards it's like, "Is that really necessary to talk about it?” “How often do you add or remove notes?” If you are already thinking about adding and removing notes, you're probably going to look up this stuff for you. So, I don't need to actually tell you about this in this talk. [laughing]

IVAN: Yeah, I love that you're able to educate people in one session even at a very high level. To go from, kind of knowing to, being interested in the technology and in what we're doing and in being interested in continuing to find out more. And, maybe that's a good reason to do a separate podcast just on the talk you gave and the contents of the talk and why are we doing that? Why is TEN7 investing as much as we are in Kubernetes and in Docker and in Drupal, and, you know, sending you to all of these camps, and then putting all of this work into the open source domain? Like, maybe there's enough there to talk about. I mean, just from my perspective, we want to be independent, and using a hosting solution that is supported in the open source that is vendor agnostic. And, if we're doing it for ourselves, there's no reason why we couldn't put it out there and have others learn and leverage from it as well. So, we should probably talk about this a little more in a separate podcast.

TESS: That's not a bad idea at all.

IVAN: I love it. Okay. We'll do that. We'll ask Jonathan to make that happen for us. Okay, so, a little more about Kubernetes. I was looking through the schedule of talks and as you, Tess, know, Raspberry Pis are really near and dear to my heart. I've used them for many different things at home, most recently as an ad blocker for the whole network, but I saw that Jeff Geerling was at Flyover Camp, and he had a talk about the cluster of RPis, or the Raspberry Pis, that he's been building since 2012-2013, something like that, and how it taught him everything he knows about Kubernetes. Did you catch that talk by any small chance?

TESS: I actually did go to that talk.

IVAN: You're kidding?

TESS: Because I was like, Oh that sounds really fun and I'd like to see what he does. Is he going to use straight K8s or is he going to use that K3 that I heard about? And, what was funny to me is that I remember watching a talk that he gave, not about Kubernetes, but about Ansible. Way, way, way back in the day at MidCamp with a very similar block of a Raspberry Pi cluster in a box. And I really wanted to see what he was going to do with this. So, sure, I went to it.

IVAN: And was it everything you wished it could be? I mean, I looked at the slides and there was a shout out to socketwench in one of the slides.

TESS: Yeah, I was like thankful. I was in the front row and no one could see how I was blushing [laughing] the entire time. Like, Oh, stop talking about me please. This is your talk. [laughing]

IVAN: [laughing] That’s great. I mean, you guys are related and connected by Kubernetes, so, how wonderful that that would be the case. So, can you give me a quick synopsis of the talk? What was the nugget that you took out of it?

TESS: So, what was interesting is that Jeff has built a small Kubernetes cluster using a standard distribution Kubernetes to run on, I think it's four or five Raspberry Pis.

IVAN: I think it’s four.

TESS: I think it's four now. Four Raspberry Pis with a single ethernet switch with power over ethernet so that it reduces the amount of additional circuitry and cables he has to carry around to power them altogether.

IVAN: Hold up. Hold up. He's actually powering the RPis now through power over ethernet? That's amazing. Of course, you could.

TESS: You can get an adapter board for that.

IVAN: That's awesome.

TESS: It's not really complicated.

IVAN: That’s so great. I'm sorry. I totally interrupted you there. What was the nugget?

TESS: [laughing] Well he didn't even mention the power over ethernet except for one thing, but I was looking at the screenshots like, Oh, you’re using power over ethernet now. Nice. [laughing]

IVAN: Nice. That’s nice.

TESS: So, a lot of the talk was about how he was running his own personal site, using a Raspberry Pi cluster out of his home network. And, I used to run my own single node server out of a home network way back, many, many years ago. And there's a number of challenges that come with that out of the box. You tend not to get static IPs from most ISPs. They'll get a Dynamic IP, some of them don't like that you have a significant amount of outbound traffic or incoming traffic that's coming from the net and they may block you for that reason, if you're on a particular service tier. Some ISPs are better at that than others, it really depends. But running his own site on a Kubernetes cluster on Raspberry Pis it's like, it reminds me of this meme that I saw passed around Kubernetes Twitter a while ago, which is, the subtext is, I deployed my blog on Kubernetes and it's this big semitrailer and it has a toy truck trailer box in the middle of it, completely dwarfed by the full size trailer. [laughing] That’s kind of like, Yeah that's pretty accurate.

IVAN: Well, I mean if you ever get reddited or slashdotted, I guess maybe it'll survive?

TESS: [laughing] Kind of. There’s a degree of front side caching I think that he also used. This kind of a project always comes across to me as not a serious, You should use this instead of traditional hosting and more like, I wanted to see if I could do that and it would be fun and it's something to do and it's something that lets me learn by doing. And that's you know, a worthy pursuit in its own right.

IVAN: But if you look at the other side of that coin, you're hosting your own website, you own the hardware, you own the software, you own your site, you can see it, you're not putting the risk of hosting in another large company's data center, right? You own it all from top to bottom, and honestly if you have a small blog and you're using your ISPs connection, and you have this overkill of a Raspberry Pi cluster that is powering the static site, you're probably not going to ever get enough traffic to bring that thing down. You're probably fine.

TESS: Probably not.

IVAN: Yeah.

TESS: Although I think Jeff's site is Drupal 8.

IVAN: Oh [laughing] so, not static, not static. Well, I'm very jealous of you getting to see that talk. That must’ve been pretty cool. I'm hoping that maybe we can get Jeff on the podcast to talk about his cluster and what he's been through and how it's evolved soon. So, Jeff if you happen to be listening, watch out for an email from us about that.

Let's talk about diversity at Flyover Camp. What did it look like? Were there the kind of usual cast of people that look like I do, white males, or what did that look like amongst attendees and speakers this year?

TESS: So, there certainly is a large contingent of white straight cis male people there as well. There were a lot of women there as well, and there were several POC as well. I didn't actually take any moment to really do any kind of headcounts on that. It just never crossed my mind to log that kind of information. But I did sit with several people which were really fascinating and really interesting to talk to, and that was really nice.

IVAN: I hope we can have more of that and more attention to that in the future and we'll try to continue to talk about it and bring it up in our podcast as well. What about attendance as a whole at Flyover Camp? Was it comparable to Drupaldelphia or to Twin Cities Drupal Camp? Did you get a feel for what it was like?

TESS: I think that it was more closer to the size of Twin Cities Drupal than Drupaldelphia. Drupaldelphia had a surprising amount of people in it. And it could have been complicated by the space, because it was a smaller space than Flyover Camp or Twin Cities Drupal, but there was certainly a large number of people there.

IVAN: Any particular sessions besides Jeff’s, that were memorable to you?

TESS: Oh geez, I'd have to look it over because so much of it was kind of a blur. I was kind of sad that I missed John Rearick's session about 45 Modules and Forty-Five Minutes. I caught the end of it. But, yeah, that would have been a really interesting talk to go to, because literally every slide has a timer. So, the talk is only 45 minutes long. So every slide is only a minute long.

IVAN: So, it's kind of like an ignite session, where it's 30 seconds per slide, 20 slides, something like that?

TESS: Mm hmm. I saw one by Ria Dixon called CloudWatch-ing, which was all about creating logs and alerts using AWS CloudWatch. That was really fascinating. And, it makes me wonder if there is a way to create similar mechanisms and use similar strategies in a purely open source implementation that doesn't rely on AWS's productized version of that.

IVAN: What is CloudWatch?

TESS: It's kind of an event and log tracking mechanism meant for distributed logging. There's a lot of that I didn't get into because it was mostly a case study about how they implemented it, and how they solved their own problems. There was a lot of additional research that I'd love to circle back to, but it was a really good session and I really enjoyed it.

IVAN: So, is CloudWatch then, kind of similar to Splunk?

TESS: I think that it's a bit similar to Splunk. I know that there might be part of that that's similar to Prometheus and Grafana which is a common Kubernetes logging mechanism.

IVAN: Yeah, Prometheus is pretty widespread as well, isn't it?

TESS: Mm hmm.

IVAN: Yeah. Okay. So, a couple of good sessions. Generally, you had a good time at the camp, gave two wonderful talks. Where was the camp? Was it at the University?

TESS: I believe it was. It was a pretty good location, although because it is in the middle of Missouri, I did have a problem getting to and from the Camp, because I didn't have a car rental. So I ended up walking there and that was a 20 minute walk in Missouri in June, which was a bit warmer than I’m used to. [laughing]

IVAN: Yeah, I guess the flipside of that is it could have been Missouri in December or January.

TESS: I mean I would have been fine with that but that’s me, I like winter. [laughing]

IVAN: [laughing] Well, go figure. I like it too. So, what do you know? Okay. So, the event venue was good. The attendance was comparable to TCDrupal. And before we wrap up, overall impression of the event? If there's another one next year are you going to go?

TESS: I would love to go again. It was a lot of fun to go there. And it’s a lot more interesting than I had expected it to be, which kind of lives up to the name. [laughing]

IVAN: [laughing] That's great. Well, I appreciate the time you spent with us today once again. Thank you so much for being with me. It's really been a pleasure.

TESS: Mm hmm.

IVAN: Well, Tess Flynn or socketwench, is the Devops Engineer here at TEN7, and she was just at Drupal Flyover Camp 2019, where she gave her talk, “Return of the Clustering Kubernetes for Drupal.” Of course, that's the third in a trilogy and the other talk, “Dr. Upal Is In - Healthcheck your Site.” Those slides are all online and a recording of the sessions are also available. Just visit this episode's webpage for those links. You’ve been listening to the TEN7 Podcast. Find us online at ten7.com/podcast. And if you have a second, do send us a message, we love hearing from you. Our email address is [email protected]. Until next time, this is Ivan Stegic. Thank you for listening.

Jun 19 2019
Jun 19

Your browser does not support the audio element. TEN7-Podcast-Ep-062-2019-Flyover-Camp.mp3

Our frequent DrupalCamp attender and speaker, DevOps Tess Flynn, returns to the podcast to recap her recent experience at Flyover Camp, a brand new Drupal camp in Kansas City, Missouri.

Host: Ivan Stegic
Guest: Tess Flynn, DevOps Engineer at TEN7

Podcast highlights: 

  • We in the midwest totally own the “flyover” jokes!
  • The continuing diversity in camp talks (business, self-care, human focus tracks)
  • Tess reviews both her talks (Return of the Clustering: Kubernetes for Drupal, and Health Check Your Site)
  • How you should stretch your mind to prepare for all the rapid-fire information you get in the Kubernetes talk!
  • Location, location, location is as important for conference talks as it is for real estate
  • Listen to Ivan and Tess geek out over the Raspberry Pi session

Links:

TRANSCRIPT

IVAN STEGIC: Hey everyone you're listening to the TEN7 Podcast, where we get together every fortnight and sometimes more often, to talk about technology, business, and the humans in it. I'm your host Ivan Stegic. Let's talk about Drupal Flyover Camp 2019, that happened from Friday the 31st of May to Sunday, the 2nd of June, in Kansas City. Joining me to give her thoughts is socketwench. That's wench, not wrench. Welcome back to the podcast.

TESS FLYNN: Hello.

IVAN: Now, did I say it the right way, because I know you always have a specific way of saying it when you give your intro to socketwench.

TESS: Well, that’s pretty close. That works.

IVAN: Close? Okay. Good. So, you were at a Flyover Camp. What's in a name? I just love how Flyover Camp were poking fun at themselves in Kansas. I mean, we're pretty much in flyover land here in Minneapolis too, so I totally get it.

TESS: [laughing] So let's first frame what that is because if we're having international listeners, they might not get what the reference is.

IVAN: Good idea.

TESS: So, the thing that goes with it is, if you're from the Midwest you're considered in flyover country. And the reason why is because the joke goes, that there is nothing in the United States that's of interest unless if you're on either coast, which is actually completely untrue. However, that is what a lot of people tend to think of it. So as a result, if you're in the Midwest you kind of go, Well, you know, what we're going to own that turf.

IVAN: Exactly.

TESS: We're going to go and names things after it and take that world.

IVAN: I love it. I love that they did that. Drupal Flyover Camp in Kansas City, Missouri. And so, this is a brand-new camp, right? This is the first time they've ever done this camp. How great is that? We have a new camp on the schedule.

TESS: Yeah, I was surprised that it was new because they hit everything running. It felt like this was a well-oiled machine for a camp.

IVAN: That's wonderful. It's wonderful to have that on the calendar again. So, well-oiled machine. Did you recognize any of the organizers? Maybe these people have done it before.

TESS: I think that I recognized a few people from…oh, what is their name…VML and YL. What are they called now, because they merged with somebody?

IVAN: I don't know.

TESS: VML Y&R. Wow, that is a mouthful.

IVAN: What?

TESS: Victor, Mike, Lima, Yankee and Romeo.

IVAN: Okay. What are they, a global marketing agency that needs a new name? [laughing]

TESS: [laughing] That is their new name. [laughing]

IVAN: [laughing] Okay. I don't even know how to say it.

TESS: They used to be two different companies that got merged, and this is the resulting name.

IVAN: Oh, it's on their BOF page. If you're looking for something about VML you can still visit the VML website. If you're looking for something about Y&R, don’t sweat, you can still visit the Y&R. So, it's basically like you said, a concatenation of their former names. Maybe it's just temporary. Ok. A little bit of a tangent. [laughing] So, some sort of experience in Flyover Camp organization. Sounds like you said they were a well-oiled machine. It was a three-day camp?

TESS: I believe so. There was a day of trainings which I did not attend, and then two days of sessions, which actually has been bucking the trend lately.

IVAN: Yeah. And also, from what I can tell there were contributions as well on Sunday so, maybe it was a four-day camp, if there were trainings as well.

TESS: Might’ve been.

IVAN: Yeah. So, you were there Friday and Saturday. It looked like they had numerous tracks. So, I thought, usually these camps have five tracks and then you have five rooms and people go to the room for the track that they're interested in. This felt like it had a dozen tracks, but three rooms and it sort of was interspersed track sessions and BOFs as well amongst these three rooms. Is that what it was like? I mean I’m only gauging from the website.

TESS: So, you know the thing with the tracks is that a lot of the time it depends on how promoted they are as their own top-level entity in the data, as it were. And some camps do a very good job of this, that they have this track, this track and this track. I think DrupalCon recently reorganized so that there's only particular tracks that they directly advertise to different audiences, like a business audience, a frontend audience, something like that. Some camps have a lot of tracks and they're not particularly consistently organized, or if they are, it doesn't feel like that when you're attending because you don't tend to notice it, and Flyover Camp seemed to fall into this latter category. That's not bad but it's just a thing.

IVAN: Yeah. And I love that the tracks were so diverse as well, right? There was security, QA, site building, the usual frontend/backend stuff and there was a self-care track as well. I mean, more of that please. More mental health stuff, more business stuff, more human focus sessions. I love it.

TESS: Mm hmm.

IVAN: I love it. I think that's awesome. And, it looked like there were about 30 sessions, so similar to Drupaldelphia and those 30 sessions were spread across two days as opposed to one day at Drupaldelphia.

TESS: Yeah and it seemed to attract a lot of people from the area. I mean I was there from Minnesota and I saw people that usually I see at DrupalCorn there as well. So, it attracted a lot of people from the Midwest.

IVAN: That's wonderful. And there were BOFs as well, and it kind of looked like they were spread out across the two days as well.

TESS: I think there were, but I was so focused on other stuff that I completely missed it.

IVAN: Yeah. This was a heck of a camp for you. I mean it wasn't one session it was two sessions.

TESS: Oh man, it was a double feature. That was hard.

IVAN: I'm sure you absolutely shone on that and I'm sure you did really well. So, let's talk about those two sessions. So, your first session on Friday was the famous cloaked talk, Kubernetes called Return of the Clustering, right? The third part of the trilogy. So that was Friday. And then Saturday you gave a talk essentially about the Healthcheck module, right? What can you do to keep tabs on the health of your Drupal site?

TESS: Well, it was also about site auditing as well, in general.

IVAN: That’s right, and site auditing. So, I guess the critical question here is, did you wear a costume for both talks?

TESS: So, here's the problem with that. I don't have a car. And in order to actually get the costume for that one I would have probably had to rent a car to go to a local thrift store chain called Ax Man surplus and see if I could find like a stethoscope or whatever that little satellite dish head gear thing that they wear, I forget what it's called, and see if I could shove one of those into my luggage. But I didn't have the time to do that. Every weekend that I've had lately has just been completely booked up.

IVAN: Well maybe we'll have to work on that if you get asked to do that talk again and we'll figure out another costume for you.

TESS: Well, rumor has it that's going to happen. [laughing]

IVAN: [laughing] So, comparatively, how were the two sessions attended? Was there a drop of people on Saturday compared to Friday or was it comparable?

TESS: It was actually the other way around. I think a lot of people find the Kubernetes talk is fun, but it can be very intimidating because it seems like, “oh, that's a very devopsy, very technical talk and it's going to be way over my head.” And I was able to attract some people to come to it, especially by making a fool out of myself, by dressing up like a Jedi and standing outside of my door waving a lightsaber to have people come and join the session. But, it was a smaller room and it was still well attended, but the site audit talk actually had a lot more people in it, mostly because it was also in the main auditorium, so a lot of people who were just there were also just there, but there was a lot of people paying attention to it as well, because it tends to be a really fun, engaging talk and it tends to appeal to a much broader audience than the Kubernetes talk, which tends to be more infrastructury devopsy people. Even though I try to make that as broadly appealing as I can.

IVAN: So, location, location, location. Right. You had a wonderful location in the auditorium for that talk.

TESS: Yeah. The only downside is that when you're in an auditorium you're usually on a pedestal or a dais or something like that, and the problem is that it sounds like I'm a T-Rex walking around on stage, because the thing is hollow so the microphone just picks up everything, and I don't tend to stay still when I give a talk, I tend to gesticulate and walk around and do lots of weird things.

IVAN: Jump around I believe you do as well. [laughing]

TESS: [laughing] Yeah. Well, I think the site audit talk, I also fall to my knees at one point, dramatically. [laughing]

IVAN: [laughing] It's a good talk.

TESS: I still remember skinning my knee at DrupalCorn.

IVAN: Well, it’s a good talk. I think it gets valuable as you do that. It certainly reminds people how important it is. Right? So, what do you think the biggest question was that people had from that health check talk, from that audit talk?

TESS: You know, I didn't get many questions. I was actually thinking about this a few days ago. I tend not to get that many questions directly after a talk because usually my talks last the entire amount of time, and afterwards, I have to rush out the door for the next person to start setting up their talk. And usually I don't get many questions, and I do try to anticipate a lot of the potential questions as well within the contents of the talk. So sometimes people will come by and ask me questions later, but that hasn't happened lately.

IVAN: It's a similar sort of thing for both talks then.

TESS: Yeah. I did have a nice conversation with someone, I think they're from the U of Kansas. I forget. I remember their face. I know that they go to DrupalCorn regularly too, but they were telling me about Kubernetes operators and all of that nifty technical stuff and that was a really interesting conversation to have, but it really wasn't a question.

IVAN: Well that actually leads me into my next question. Usually you're the one educating people about whatever you're talking about. What do you think your biggest takeaway from a session was in Flyover camp? What did you learn from each of them?

TESS: Ok. Geez, I’m trying to remember all the sessions I went to because Twin Cities Drupal was last weekend, and now I'm trying to remember any of the sessions I went to.

IVAN: Oh, I think maybe you misunderstood. What I meant was, what did you learn in your talk from the audience?

TESS: Oh. So, one thing that definitely occurred to me is that, when it comes to the Kubernetes talk is, just how much technical knowledge you need, all technical terms you need to pick up very, very quickly to get anywhere with understanding Kubernetes without feeling like you're “drowning,” in technical terms, all of a sudden. And I certainly had that experience myself just trying to learn Kubernetes in the first place and that is after having a very strong background in how containers work and how Docker works and some of the top terminologies I picked up from running production workloads in Dockers form, and I realized that after that talk, like, wow, in 45 minutes I take you from, you kind of, sort of know what Docker is, and you might have heard of Ansible, but you don't know too much about it, to, here’s how you run a Drupal site in production on Kubernetes using a simple effective formula. And that kind of struck me as Wow, not many people are doing that because, wow that can be really complicated.

IVAN: Yeah, it's the bleeding edge of it isn't it?

TESS: It's not just the bleeding edge, it’s just that the underlying design that I went for strives for minimalism and simplicity, and a lot of people find that appealing because it reduces the number of working parts that you have to know. A good example would be memcached. The way that it's presented in the talk is as a stateful set and that works. A lot of people will say, "What you should do is run it as another object called a daemon set." But in order to introduce a daemon set, I'd have to introduce a completely new object type that only works for that, and afterwards it's like, "Is that really necessary to talk about it?” “How often do you add or remove notes?” If you are already thinking about adding and removing notes, you're probably going to look up this stuff for you. So, I don't need to actually tell you about this in this talk. [laughing]

IVAN: Yeah, I love that you're able to educate people in one session even at a very high level. To go from, kind of knowing to, being interested in the technology and in what we're doing and in being interested in continuing to find out more. And, maybe that's a good reason to do a separate podcast just on the talk you gave and the contents of the talk and why are we doing that? Why is TEN7 investing as much as we are in Kubernetes and in Docker and in Drupal, and, you know, sending you to all of these camps, and then putting all of this work into the open source domain? Like, maybe there's enough there to talk about. I mean, just from my perspective, we want to be independent, and using a hosting solution that is supported in the open source that is vendor agnostic. And, if we're doing it for ourselves, there's no reason why we couldn't put it out there and have others learn and leverage from it as well. So, we should probably talk about this a little more in a separate podcast.

TESS: That's not a bad idea at all.

IVAN: I love it. Okay. We'll do that. We'll ask Jonathan to make that happen for us. Okay, so, a little more about Kubernetes. I was looking through the schedule of talks and as you, Tess, know, Raspberry Pis are really near and dear to my heart. I've used them for many different things at home, most recently as an ad blocker for the whole network, but I saw that Jeff Geerling was at Flyover Camp, and he had a talk about the cluster of RPis, or the Raspberry Pis, that he's been building since 2012-2013, something like that, and how it taught him everything he knows about Kubernetes. Did you catch that talk by any small chance?

TESS: I actually did go to that talk.

IVAN: You're kidding?

TESS: Because I was like, Oh that sounds really fun and I'd like to see what he does. Is he going to use straight K8s or is he going to use that K3 that I heard about? And, what was funny to me is that I remember watching a talk that he gave, not about Kubernetes, but about Ansible. Way, way, way back in the day at MidCamp with a very similar block of a Raspberry Pi cluster in a box. And I really wanted to see what he was going to do with this. So, sure, I went to it.

IVAN: And was it everything you wished it could be? I mean, I looked at the slides and there was a shout out to socketwench in one of the slides.

TESS: Yeah, I was like thankful. I was in the front row and no one could see how I was blushing [laughing] the entire time. Like, Oh, stop talking about me please. This is your talk. [laughing]

IVAN: [laughing] That’s great. I mean, you guys are related and connected by Kubernetes, so, how wonderful that that would be the case. So, can you give me a quick synopsis of the talk? What was the nugget that you took out of it?

TESS: So, what was interesting is that Jeff has built a small Kubernetes cluster using a standard distribution Kubernetes to run on, I think it's four or five Raspberry Pis.

IVAN: I think it’s four.

TESS: I think it's four now. Four Raspberry Pis with a single ethernet switch with power over ethernet so that it reduces the amount of additional circuitry and cables he has to carry around to power them altogether.

IVAN: Hold up. Hold up. He's actually powering the RPis now through power over ethernet? That's amazing. Of course, you could.

TESS: You can get an adapter board for that.

IVAN: That's awesome.

TESS: It's not really complicated.

IVAN: That’s so great. I'm sorry. I totally interrupted you there. What was the nugget?

TESS: [laughing] Well he didn't even mention the power over ethernet except for one thing, but I was looking at the screenshots like, Oh, you’re using power over ethernet now. Nice. [laughing]

IVAN: Nice. That’s nice.

TESS: So, a lot of the talk was about how he was running his own personal site, using a Raspberry Pi cluster out of his home network. And, I used to run my own single node server out of a home network way back, many, many years ago. And there's a number of challenges that come with that out of the box. You tend not to get static IPs from most ISPs. They'll get a Dynamic IP, some of them don't like that you have a significant amount of outbound traffic or incoming traffic that's coming from the net and they may block you for that reason, if you're on a particular service tier. Some ISPs are better at that than others, it really depends. But running his own site on a Kubernetes cluster on Raspberry Pis it's like, it reminds me of this meme that I saw passed around Kubernetes Twitter a while ago, which is, the subtext is, I deployed my blog on Kubernetes and it's this big semitrailer and it has a toy truck trailer box in the middle of it, completely dwarfed by the full size trailer. [laughing] That’s kind of like, Yeah that's pretty accurate.

IVAN: Well, I mean if you ever get reddited or slashdotted, I guess maybe it'll survive?

TESS: [laughing] Kind of. There’s a degree of front side caching I think that he also used. This kind of a project always comes across to me as not a serious, You should use this instead of traditional hosting and more like, I wanted to see if I could do that and it would be fun and it's something to do and it's something that lets me learn by doing. And that's you know, a worthy pursuit in its own right.

IVAN: But if you look at the other side of that coin, you're hosting your own website, you own the hardware, you own the software, you own your site, you can see it, you're not putting the risk of hosting in another large company's data center, right? You own it all from top to bottom, and honestly if you have a small blog and you're using your ISPs connection, and you have this overkill of a Raspberry Pi cluster that is powering the static site, you're probably not going to ever get enough traffic to bring that thing down. You're probably fine.

TESS: Probably not.

IVAN: Yeah.

TESS: Although I think Jeff's site is Drupal 8.

IVAN: Oh [laughing] so, not static, not static. Well, I'm very jealous of you getting to see that talk. That must’ve been pretty cool. I'm hoping that maybe we can get Jeff on the podcast to talk about his cluster and what he's been through and how it's evolved soon. So, Jeff if you happen to be listening, watch out for an email from us about that.

Let's talk about diversity at Flyover Camp. What did it look like? Were there the kind of usual cast of people that look like I do, white males, or what did that look like amongst attendees and speakers this year?

TESS: So, there certainly is a large contingent of white straight cis male people there as well. There were a lot of women there as well, and there were several POC as well. I didn't actually take any moment to really do any kind of headcounts on that. It just never crossed my mind to log that kind of information. But I did sit with several people which were really fascinating and really interesting to talk to, and that was really nice.

IVAN: I hope we can have more of that and more attention to that in the future and we'll try to continue to talk about it and bring it up in our podcast as well. What about attendance as a whole at Flyover Camp? Was it comparable to Drupaldelphia or to Twin Cities Drupal Camp? Did you get a feel for what it was like?

TESS: I think that it was more closer to the size of Twin Cities Drupal than Drupaldelphia. Drupaldelphia had a surprising amount of people in it. And it could have been complicated by the space, because it was a smaller space than Flyover Camp or Twin Cities Drupal, but there was certainly a large number of people there.

IVAN: Any particular sessions besides Jeff’s, that were memorable to you?

TESS: Oh geez, I'd have to look it over because so much of it was kind of a blur. I was kind of sad that I missed John Rearick's session about 45 Modules and Forty-Five Minutes. I caught the end of it. But, yeah, that would have been a really interesting talk to go to, because literally every slide has a timer. So, the talk is only 45 minutes long. So every slide is only a minute long.

IVAN: So, it's kind of like an ignite session, where it's 30 seconds per slide, 20 slides, something like that?

TESS: Mm hmm. I saw one by Ria Dixon called CloudWatch-ing, which was all about creating logs and alerts using AWS CloudWatch. That was really fascinating. And, it makes me wonder if there is a way to create similar mechanisms and use similar strategies in a purely open source implementation that doesn't rely on AWS's productized version of that.

IVAN: What is CloudWatch?

TESS: It's kind of an event and log tracking mechanism meant for distributed logging. There's a lot of that I didn't get into because it was mostly a case study about how they implemented it, and how they solved their own problems. There was a lot of additional research that I'd love to circle back to, but it was a really good session and I really enjoyed it.

IVAN: So, is CloudWatch then, kind of similar to Splunk?

TESS: I think that it's a bit similar to Splunk. I know that there might be part of that that's similar to Prometheus and Grafana which is a common Kubernetes logging mechanism.

IVAN: Yeah, Prometheus is pretty widespread as well, isn't it?

TESS: Mm hmm.

IVAN: Yeah. Okay. So, a couple of good sessions. Generally, you had a good time at the camp, gave two wonderful talks. Where was the camp? Was it at the University?

TESS: I believe it was. It was a pretty good location, although because it is in the middle of Missouri, I did have a problem getting to and from the Camp, because I didn't have a car rental. So I ended up walking there and that was a 20 minute walk in Missouri in June, which was a bit warmer than I’m used to. [laughing]

IVAN: Yeah, I guess the flipside of that is it could have been Missouri in December or January.

TESS: I mean I would have been fine with that but that’s me, I like winter. [laughing]

IVAN: [laughing] Well, go figure. I like it too. So, what do you know? Okay. So, the event venue was good. The attendance was comparable to TCDrupal. And before we wrap up, overall impression of the event? If there's another one next year are you going to go?

TESS: I would love to go again. It was a lot of fun to go there. And it’s a lot more interesting than I had expected it to be, which kind of lives up to the name. [laughing]

IVAN: [laughing] That's great. Well, I appreciate the time you spent with us today once again. Thank you so much for being with me. It's really been a pleasure.

TESS: Mm hmm.

IVAN: Well, Tess Flynn or socketwench, is the Devops Engineer here at TEN7, and she was just at Drupal Flyover Camp 2019, where she gave her talk, “Return of the Clustering Kubernetes for Drupal.” Of course, that's the third in a trilogy and the other talk, “Dr. Upal Is In - Healthcheck your Site.” Those slides are all online and a recording of the sessions are also available. Just visit this episode's webpage for those links. You’ve been listening to the TEN7 Podcast. Find us online at ten7.com/podcast. And if you have a second, do send us a message, we love hearing from you. Our email address is [email protected]. Until next time, this is Ivan Stegic. Thank you for listening.

May 23 2019
May 23

Your browser does not support the audio element. TEN7-Podcast-Ep-060-Drupaldelphia-2019.mp3

In this week’s podcast TEN7’s DevOps Tess Flynn aka @socketwench is our guest, giving us her observations of the Drupaldelphia 2019 conference she recently attended, as well as a summary of her helpful session, “Return of the Clustering: Kubernetes for Drupal.”

Host: Ivan Stegic
Guest: Tess Flynn, TEN7 DevOps

In this podcast we'll discuss: 

  • What it takes to get a good Jedi costume
  • Tess’s Trilogy of Talks
  • Summary of Tess’s Kubernetes session
  • How camps are not just about Drupal and not just about developers anymore
  • Short camps = no keynote
  • Help desks at the conference - what a great idea!

Links:

TRANSCRIPT

IVAN STEGIC: Hey everyone you're listening to the TEN7 Podcast, where we get together every fortnight and sometimes more often, to talk about technology, business and the humans in it. I'm your host Ivan Stegic. Let’s talk about the Drupaldelphia 2019 conference that happened recently in Philadelphia. And, joining me to give her thoughts on the camp is Tess Flynn, a seasoned guest here on the TEN7 Podcast. @socketwench welcome back to the podcast.

TESS FLYNN: Hello.

IVAN: I’m going to have to make you the guest host when I’m out of town. [laughing]

TESS: [laughing] Sure, why not. We could do one on terrible movies. [laughing]

IVAN: [laughing] That would be actually awesome. We should totally do that. All right, Drupaldelphia. What’s in a name? So, this is the annual Drupal Camp in Philadelphia, Pennsylvania, and this year it was part of something called Philly Tech Week 2019. And you Tess, were there representing TEN7. What do you know about the relationship between Philly Tech Week and Drupaldelphia?

TESS: Not much, actually. I was so focused on other things during the week, I didn’t have any time to really investigate it deeply. But I noticed that some of the other attendees didn’t know that either, [laughing] so at least I was in good company.

IVAN: Okay. And were there any signs around that referred to Philly Tech Week, or was it kind of just insular?

TESS: No, not that I noticed. I think that it was just part of the longer weeklong event. Like, some organizations will have this kind of co-boosting agreement, where we’ll talk about your event and then you’ll talk about our event, and it’s good for both of us, and we don’t have any other commonality. And there’s nothing wrong with that. That’s good. 

IVAN: That is good. Have you been to Drupaldelphia before?

TESS: I have not. This is my first time. It just showed up across my board, and I went, Huh, I haven’t done that one before, sure, why not?

IVAN: And we have a great talk that you have produced, and so, as a company, we’re making sure you get out and tell that to as many people as possible. So, it was a good opportunity, I think.

TESS: It’s the first talk that I’ve been giving in costume, which is hilarious.

IVAN: Right, I heard about this, and you talk about it in the session, in the recording. So, tell us about the costume.

TESS: So, the thing is that, when you’re doing air travel, it’s always best to only have carry-on luggage, because you never know if you’re going to have a delay, or if your flight is going to get lost, or if your luggage is going to get lost, or something. So, you should restrict yourself to a roll-aboard and a backpack. Now, I’m an old hat toward business travel, so, I’m used to the standard, roll-aboard-backpack modality, but it does limit what you could bring with you when you’re traveling. So, the main concern for coming up with an outfit is: how can I do this so that I minimize the amount of potential luggage space that I take up, while still being evocative of the thing that I’m trying to riff on. And after thinking about this for a while, I actually did come up with something that was rather clever. The easiest thing to do was first find a plastic lightsaber. Not an electronic lightsaber, not one that lights up, not anything else.

The recommendation is that it has to be a lightsaber that’s made of plastic, that fully collapses into the hilt, and has no other electronics. Why is this important? Because air travel. Because we don’t want to have to pull the lightsaber out and put it on the scan deck for everyone else to look at. When it’s just plastic, it’s just plastic, no one cares about it, just put it in the bag and no one cares. Fortunately for me, I found one that was The Last Jedi  branded, but it’s actually a model of Luke Skywalker's lightsaber, that was fully plastic, no electronics, completely collapses into the hilt, on Amazon for 10 bucks.

IVAN: Nice.

TESS: And that was really as far as I was going to take it, because having a prop in the talk isn’t necessarily the worst thing. But then I started getting a little bit ahead of myself and I go, Well, what else can I do? And it occurred to me that one thing that I could do was, I should probably do up the outfit a little bit more. And, one of the things I decided I could do to do up the outfit a little bit more is, what is one thing that a lot of Jedi have going for them? And so, I thought back to all of those hours that I spent watching all of those movies, and all of the comics I read, and everything else, and go, They have this thing for cloaks. That’s it! They have this thing for cloaks and robe-like things. Well, I can’t really do a robe-like thing, because that would require a lot more layers in order to do everything. However, the problem is that, in order to really be evocative, I still need the exterior robe. So, I went again to Amazon, and I found a $20 brown cloak with a hood that looked pretty darn close to what a Jedi would normally wear [laughing], and that’s all I got.

And, so, I am out on stage wearing my normal talk outfit, the infamous skull dress, and wearing this huge brown cloak and carrying a plastic lightsaber. And, I actually stand in front of the room I’m giving the talk in before it starts, ushering people in, acting like a complete fool. Like, everyone going like, What am I getting into here? Why is this person in costume? And I’ll show up to the camp in costume for the entire camp [laughing] because it’s fun and it’s different and it’s weird, and one thing that is really kind of useful is, this is meant to be a fun thing. It’s meant to add more flavor to the talk, and the talk is full of my style of corny jokes. So there’s nothing wrong with me also amping up the corny the entire time. This whole thing about me amping up the corny, actually started in, I think 2015, in DrupalCon where I showed up with a giant inflatable whale, and I literally carried that around with me until the talk started and then I gave it away within the first few minutes. [laughing]

IVAN: I love how involved you get in your talks and how personal your slides are, and the comics that are hand drawn and the characters that you have, they just really make it a joy to listen to you speak. And, I wondered, because I haven’t actually seen you give this talk yet, I will absolutely be going to it at TC Drupal Camp when you give it, and in my mind I wasn’t sure if it was a black cloak or a brown cloak, but I was pretty sure you went with brown, so I’m glad to hear you went with brown. I guess the question I have is, did you have a hoodie?

TESS: I didn’t have a hoodie, but the cloak has a hood on it. So, when I’m outside the room ushering people in, I will put the hood up and start waving the lightsaber around, and usually that’s enough to let people get the idea of what I’m doing. [laughing]

IVAN: That’s really great. Ok, so let’s actually stay on this track. We’ll get to the details of Drupaldelphia in a sec here, but since we’ve already started talking about your session, let’s keep going. So, your session is called "Return of the Clustering, Kubernetes for Drupal", and it’s part of a trilogy, and I’m going to say I think first of all it’s amazing that it’s part of a trilogy. So I’d like you to kind of just tell us what Episode 1 and Episode 2 are about, and my very technical question thereafter is going to be a follow-up, because this is a trilogy of trilogies, but we can get to that in a second. Let’s set up the Episode 1, Episode 2 part of this.

TESS: So the whole, "it’s a part of a trilogy" thing was actually a bit of a joke on myself. One of the first talks that I started doing outside of Minneapolis was on "Ride the Whale: Docker for Drupalists", and the idea was to teach people how to build your own Docker-based local development environment, and how Docker in general worked. And, that was the first start of it. Then the next talk that I gave was “Avoid Deep Hurting! Deployments beyond Git,” which introduced how to build your own continuous integration system using just open source, free stuff. Free stuff like Ansible, free stuff like Gitlab CI, and the idea behind that is now you have these two different pieces that don’t seem related, but become related in this talk.

I kind of sat back and thought about all the talks I’d given over the last few years and realized, Oh, geez, I made a trilogy. Oh, God, what did I do? When that occurred to me, I also knew what the whole theme of the talk was going to be, and because it was a trilogy, I decided, Okay, fine we’ll call it "Return of the Clustering" and we’ll do a whole Return of the Jedi motif, because why not?  [laughing]

IVAN: Exactly. So, we’ve set up the trilogy, now "Return of the Clustering" is a wonderful evolution, in my opinion, from your very beginning of trying to figure out how the heck are we going to set up Drupal in an easy Docker container or set of containers locally, and basically fast forward to the real meat of what we’re trying to do, and that’s run Docker in production? Right? So, give us a high-level description of "Return of the Clustering."

TESS: So, we first start out with describing, Wouldn’t it be great if we ran Docker in production? And we dissect the various problems with that. There are security problems that come inherent with running a Docker-based workload that you have to be aware of, and most people who just use Docker out of the box might not have ever considered these facts. And, it’s different than traditional server management, because there are a few different additional factors that come into play with how Docker executes things. Then you also have to worry about how you’re going to actually get the workload onto the cluster as well, and how you’re going to orchestrate it. One of the biggest problems is, it’s really easy to stand up Docker on a single server and have that single server run an entire workload of multiple containers.

There’s nothing wrong with that. You install Docker, you install Compose, you write a Compose file, you stand it up, you do some security things to make sure everything is loaded correctly, and there you go, you’re done. But, the problem is that your scaling is only vertical. You can only make your server bigger. You really want to make your scaling horizontal and add more servers. And this makes things very complex, because if you have to coordinate multiple Docker hosts together, doing that manually is not fun, and it’s not DevOps either. It’s not automatable in a very easy way. Fortunately, there are container orchestrators out there that know how to do this for you. So we talk about Docker Swarm and what the advantages and disadvantages of that are, and then we introduce Kubernetes. We talk about how the Kubernetes model is different, then we architect out an architecture that will run a Drupal site in Kubernetes. And, this is actually a subtle point that I keep having to remind myself and others, is that the Kubernetes model is so complicated, with so many little details, and so many different things in it, that you really get lost very quickly. And it took me a year to figure out which bits of the Kubernetes’ model work for Drupal, and which bits we don’t need to talk about. Once I understood those parts, it was easy to build an architecture from those minimal amounts of pieces.

So, we review what those pieces are, and we describe them. Then we talk about how to technically implement them within Kubernetes, and why Kubernetes is kind of nifty through its use of YAML. So, what we’re going to do is, we build all of that out. But now we have another problem, which is, if we want to build the dynamic cluster which supports multiple clients, multiple sites, that’s a lot of YAML we have to manage and now we’ve just introduced a problem that I covered in Avoid Deep Hurting, which is, we have introduced humans back into the mix. We need to take the people out of the mix and let the technology handle it for us, because it doesn’t get tired, it doesn’t make typos, it doesn’t need three more coffees in order to get through the day—at least not most days. [laughing] So, we combine Ansible and Kubernetes and Docker to build out an entire cluster in an automated fashion, by just changing a few different variables, and we run that on Digital Ocean.

Then we also found out that the problem is, to really effectively leverage Kubernetes for Drupal, you can’t just put an Apache container out on Kubernetes and then throw the site on it, and then update it in place like you did with traditional server management. It’s not really the way that Kubernetes wants to do things. So, instead, we end up having to build a custom container that contains the site code already, and then run that on Kubernetes directly. And that’s a much more cloud-native workflow, and it’s a bit of a paradigm shift from what a lot of people are used to. But, each one of these pieces works well together to create an effective, open source, minimal Kubernetes production cluster for Drupal.

IVAN: I love that it’s minimal. I mean, you just put what you absolutely need out there. And that’s kind of the philosophy and it reduces your security work, quite honestly, and it also reduces the attack vector. So, I’m so glad to have heard this summary about the talk. I hope that people go out there and watch the recording, and if you have an opportunity to attend TC Drupal Camp or Flyover Camp in Kansas, you’ll be at both of those places giving the talk again.

We went through your talk, let’s talk a little bit more about Drupaldelphia 2019. So, it’s a one-day camp, and it kind of looks like there were six tracks, which is a large number of tracks for a camp.

TESS: It was a surprising amount of things you could do in every time slot.

IVAN: And it feels like the tracks weren’t just Drupal, right? I mean we talked about this with Chris and Dan [organizers of TC Drupal Camp]. It kind of feels like there is this conscious effort in the camps, at least the ones I’ve been paying attention to recently, to be not just about developers, and not just about Drupal. So, what do you think about that?

TESS: I think that’s really a good strategy and it’s a lot more holistic than we’re used to going forward. I think that our industry is still getting used to the idea that there is an internet that we can connect with each other and research things, even though we do it every day. I don’t think that the cultural and emotional impact of that has really entirely sunk in. And a lot of events are changing to reflect that attitude, that it’s not just one piece of technology that we need to deeply investigate anymore. It’s now: how does this one piece interact with a whole bunch of other pieces? And some of those pieces aren’t technology. A lot of those pieces are people.

IVAN: Yeah, the people aspect is so important as well, and I’m so glad to see those tracks are appearing on the local camps. And, so, one day, six tracks, five sessions in the days, so about 30 sessions total in the whole camp, which sounds about exactly the same as what we’re doing in TC Drupal this year on the one day of the camp. I didn’t see a keynote on the schedule, an official keynote. I didn’t see intro and outro, welcoming remarks and ending remarks. Is that right? There was no keynote?

TESS: There was no keynote. There was a brief intro, and then the session started. And I think that actually worked fairly well for this camp, because I think that the shorter the camps, a keynote is less effective. And, this is a bit of an attitude change, because I remember when TC Drupal first didn’t have a keynote, and I kind of missed it. I kind of wanted that back, because one thing that was kind of nifty about having a keynote is that, it’s there to introduce you to the talk, and get you excited, and get you ready for the day. But for a one day camp, it’s not really that necessary, is it?

IVAN: No.

TESS: You’re already there. You already are invested to do the day, and there doesn’t seem to be much of a need to really do that. It might be that keynotes are a thing that are best suited for multi-day events.

IVAN: Yeah, that’s kind of what I’m thinking as well, multi-day events that maybe need higher attendance, and so you almost use the keynote to attract attendees that you hope stay on. So, maybe there’s a celebrity, or someone who’s done an awful lot of contribution that has the keynote. And I’ve always thought of that as an attractor, but it almost eats into the day if it’s a one-day camp.

TESS: Yeah. It usually takes a good hour and a half, and you could get a whole entire slot and a break in otherwise. And from an organizer perspective, attracting a keynoter is often very difficult, because if you’re a keynoter you usually should compensate them in some way, even if it’s just paying for lodging and a flight. But that can be a bit of an impediment, especially if you’re a smaller event, or a less established event, you might not be able to get someone to do a keynote. And, I think it might just be not as necessary anymore. It might just be better to have everything else.

What was also kind of nifty about Drupaldelphia is that they had these help desks, as well. Every time slot had an accessibility lab, a Drupal and Composer help desk, and a contribution workshop room. And I thought that was spectacular, because it allows people to drop out of the sessions and work on other things or get directed help. And a Drupal and Composer help desk, I thought that was a brilliant idea, because wow Composer. Composer can really, really ruin your day. [laughing]

IVAN: Yeah, I agree. So, is that how BoFs and Contributions will handle theirs basically during the whole day and maybe as options and alternatives to the sessions that were already scheduled? Is that how it was handled?

TESS: No, there was also a distinct BoF room, but it did share with that particular room, which I think might not have been necessarily the best choice. I think that it would’ve been a bit better to have a dedicated BoF room, and then a dedicated help desk lab room, because that would allow people to be less interrupted by people who are doing a BoF. And sometimes BoFs end up being sessions that were submitted, but weren’t accepted for whatever reason, and those could be kind of distracting.

IVAN: And animated. Sometimes you get people who show up in cloaks and sabers in those BoF sessions too. [laughing]

TESS: Only if it fits in my carry on. [laughing]

IVAN: [laughing] It sounds like you had a good time there, Tess. That’s great. So, it was on a Friday at the end of the work week but in the middle of the day. What was attendance like?

TESS: There were about 250 people who attended. It was surprisingly packed. I think this is one thing that I noticed that a lot of these one-day events have kind of a side effect that they can concentrate their attendance. A multi-day event actually has kind of the reverse effect. It spreads out the attendance to multiple days and makes it more difficult to actually advocate it. Particularly if one of those days is a weekend. If one of those days is a weekend, a lot of developers, depending on the age

bracket of the segment that you’re going for, and Drupal is tending to get a little bit more middle-aged than it was like 10 years ago.

IVAN: Oh, come on. Come on. [laughing]

TESS: Hey, I’m being honest here. I’ve been to a lot of events and I’ve looked at the age of the people showing up and it’s like, Yeah, I’m seeing it.

IVAN: I know what you’re saying. People are little wiser, a little grayer, a little more family, families are attending. Yeah, it’s true.

TESS: And, as a result, having time on weekends is kind of a difficult call, even a difficult ethical call, because some people will do this for work. And now you’re asking them to do work on a Saturday. That could be a little bit of a difficult call, whereas if it’s a one-day event that’s during the week, then it’s fairly concentrated. And at that point you know that it’s just a workday that’s being used, and if this is aligned with your career, there’s nothing wrong with that. And, then it doesn’t interfere with your weekend and that critical amount of recovery time, that we all need at the end of a work week.

IVAN: I agree. I think it’s good for mental health to do it the way that it was done. What was the location of the camp? What was that event space like? Where was it located?

TESS: I think it was called The Hussian School of Art; I think. Originally when I found the building, I went to the wrong door and some very nice security people at the door were like, “Where are you going and what the hell is Drupal?” [laughing] So I had to give them my spiel really quickly and then they said, “Oh, oh, you want that? It’s probably around the corner. So, go down to the end of the block and take a left.” It was really helpful. And then later on when I went to get some coffee I passed by there and I found a door. I found a sign there that said that you should always go around the corner. So, no one else had to repeat what I did in the morning. [laughing]

IVAN: Okay, good. That’s called iterating, I think. Were you wearing your cloak at the time that the security guards were giving you directions?

TESS: I think that I stuffed it in my bag while I was taking the cab to the event, because the only hotel I could get was actually right next to the airport. [laughing] 

IVAN: Oh, my goodness.

TESS: And so, I had to take a cab ride to go into town and I didn’t want to seem weirder than I usually am to people. So, I stuffed the whole thing in my backpack and fortunately a plastic lightsaber and a cloak does not require that much space intentionally. We covered that already. So, it was kind of handy.

IVAN: So, a successful space then you felt. Was there kind of an atrium or a gathering space for everyone where there was lunch? Or, how did that work?

TESS: There was this semi-open area. We had the quick intro in that semi-open area. And then they pulled out some moveable walls to enclose that to make another room, which was nifty. And there was a nice little gathering spot where you could see the tables, and you could get lunch, and you could interact with different vendors that were at the event. That was all kind of nice, and then there was a couple rooms down some hallways, and yeah, it was a nice, cozy space and I rather liked it.

IVAN: So, overall impression of the event then, seemingly positive?

TESS: Yeah, I really enjoyed it. I thought that was a great event, and I think that more people should go to it.

IVAN: So, definitely a candidate for going to next year. We’ll have to wait and see what the next episode looks like of the series, I guess. And whether there will be a next episode?

TESS: We’ll find out. There might be a case study afterwards where we talk about what we actually did that might be a little bit more interesting to people than an intro to Kubernetes itself, but I think that it’s necessary.

IVAN: You know, that’s a great idea, to do a case study. I know we’re working through actually implementing something live right now, so be taking notes Tess, because it sounds like you already have an idea for the case study talk.

TESS: (laughing) All right.

IVAN: All right, all right. Well, thank you so much for spending your time with me today, again. It’s truly been my pleasure to have you on. Tess Flynn is the DevOps Engineer here at TEN7 and she was at Drupaldelphia 2019, giving her talk, "Return of the Clustering, Kubernetes for Drupal." The slides are online and a recording of the session itself is available as well. Just visit this episode's webpage for the links. The URL to this episode is t7.io/ep60.

And, don’t forget, if you’d like a credit towards your new Digital Ocean account, just go to ten7.com/digitalocean and follow the link on the page to redeem it.

You’ve been listening to the TEN7 Podcast. Find us online at ten7.com/podcast. And if you have a second, do send us a message, we love hearing from you. Our email address is [email protected]n7.com. Until next time, this is Ivan Stegic. Thank you for listening.

May 23 2019
May 23

Your browser does not support the audio element. TEN7-Podcast-Ep-060-Drupaldelphia-2019.mp3

In this week’s podcast TEN7’s DevOps Tess Flynn aka @socketwench is our guest, giving us her observations of the Drupaldelphia 2019 conference she recently attended, as well as a summary of her helpful session, “Return of the Clustering: Kubernetes for Drupal.”

Host: Ivan Stegic
Guest: Tess Flynn, TEN7 DevOps

In this podcast we'll discuss: 

  • What it takes to get a good Jedi costume
  • Tess’s Trilogy of Talks
  • Summary of Tess’s Kubernetes session
  • How camps are not just about Drupal and not just about developers anymore
  • Short camps = no keynote
  • Help desks at the conference - what a great idea!

Links:

TRANSCRIPT

IVAN STEGIC: Hey everyone you're listening to the TEN7 Podcast, where we get together every fortnight and sometimes more often, to talk about technology, business and the humans in it. I'm your host Ivan Stegic. Let’s talk about the Drupaldelphia 2019 conference that happened recently in Philadelphia. And, joining me to give her thoughts on the camp is Tess Flynn, a seasoned guest here on the TEN7 Podcast. @socketwench welcome back to the podcast.

TESS FLYNN: Hello.

IVAN: I’m going to have to make you the guest host when I’m out of town. [laughing]

TESS: [laughing] Sure, why not. We could do one on terrible movies. [laughing]

IVAN: [laughing] That would be actually awesome. We should totally do that. All right, Drupaldelphia. What’s in a name? So, this is the annual Drupal Camp in Philadelphia, Pennsylvania, and this year it was part of something called Philly Tech Week 2019. And you Tess, were there representing TEN7. What do you know about the relationship between Philly Tech Week and Drupaldelphia?

TESS: Not much, actually. I was so focused on other things during the week, I didn’t have any time to really investigate it deeply. But I noticed that some of the other attendees didn’t know that either, [laughing] so at least I was in good company.

IVAN: Okay. And were there any signs around that referred to Philly Tech Week, or was it kind of just insular?

TESS: No, not that I noticed. I think that it was just part of the longer weeklong event. Like, some organizations will have this kind of co-boosting agreement, where we’ll talk about your event and then you’ll talk about our event, and it’s good for both of us, and we don’t have any other commonality. And there’s nothing wrong with that. That’s good. 

IVAN: That is good. Have you been to Drupaldelphia before?

TESS: I have not. This is my first time. It just showed up across my board, and I went, Huh, I haven’t done that one before, sure, why not?

IVAN: And we have a great talk that you have produced, and so, as a company, we’re making sure you get out and tell that to as many people as possible. So, it was a good opportunity, I think.

TESS: It’s the first talk that I’ve been giving in costume, which is hilarious.

IVAN: Right, I heard about this, and you talk about it in the session, in the recording. So, tell us about the costume.

TESS: So, the thing is that, when you’re doing air travel, it’s always best to only have carry-on luggage, because you never know if you’re going to have a delay, or if your flight is going to get lost, or if your luggage is going to get lost, or something. So, you should restrict yourself to a roll-aboard and a backpack. Now, I’m an old hat toward business travel, so, I’m used to the standard, roll-aboard-backpack modality, but it does limit what you could bring with you when you’re traveling. So, the main concern for coming up with an outfit is: how can I do this so that I minimize the amount of potential luggage space that I take up, while still being evocative of the thing that I’m trying to riff on. And after thinking about this for a while, I actually did come up with something that was rather clever. The easiest thing to do was first find a plastic lightsaber. Not an electronic lightsaber, not one that lights up, not anything else.

The recommendation is that it has to be a lightsaber that’s made of plastic, that fully collapses into the hilt, and has no other electronics. Why is this important? Because air travel. Because we don’t want to have to pull the lightsaber out and put it on the scan deck for everyone else to look at. When it’s just plastic, it’s just plastic, no one cares about it, just put it in the bag and no one cares. Fortunately for me, I found one that was The Last Jedi  branded, but it’s actually a model of Luke Skywalker's lightsaber, that was fully plastic, no electronics, completely collapses into the hilt, on Amazon for 10 bucks.

IVAN: Nice.

TESS: And that was really as far as I was going to take it, because having a prop in the talk isn’t necessarily the worst thing. But then I started getting a little bit ahead of myself and I go, Well, what else can I do? And it occurred to me that one thing that I could do was, I should probably do up the outfit a little bit more. And, one of the things I decided I could do to do up the outfit a little bit more is, what is one thing that a lot of Jedi have going for them? And so, I thought back to all of those hours that I spent watching all of those movies, and all of the comics I read, and everything else, and go, They have this thing for cloaks. That’s it! They have this thing for cloaks and robe-like things. Well, I can’t really do a robe-like thing, because that would require a lot more layers in order to do everything. However, the problem is that, in order to really be evocative, I still need the exterior robe. So, I went again to Amazon, and I found a $20 brown cloak with a hood that looked pretty darn close to what a Jedi would normally wear [laughing], and that’s all I got.

And, so, I am out on stage wearing my normal talk outfit, the infamous skull dress, and wearing this huge brown cloak and carrying a plastic lightsaber. And, I actually stand in front of the room I’m giving the talk in before it starts, ushering people in, acting like a complete fool. Like, everyone going like, What am I getting into here? Why is this person in costume? And I’ll show up to the camp in costume for the entire camp [laughing] because it’s fun and it’s different and it’s weird, and one thing that is really kind of useful is, this is meant to be a fun thing. It’s meant to add more flavor to the talk, and the talk is full of my style of corny jokes. So there’s nothing wrong with me also amping up the corny the entire time. This whole thing about me amping up the corny, actually started in, I think 2015, in DrupalCon where I showed up with a giant inflatable whale, and I literally carried that around with me until the talk started and then I gave it away within the first few minutes. [laughing]

IVAN: I love how involved you get in your talks and how personal your slides are, and the comics that are hand drawn and the characters that you have, they just really make it a joy to listen to you speak. And, I wondered, because I haven’t actually seen you give this talk yet, I will absolutely be going to it at TC Drupal Camp when you give it, and in my mind I wasn’t sure if it was a black cloak or a brown cloak, but I was pretty sure you went with brown, so I’m glad to hear you went with brown. I guess the question I have is, did you have a hoodie?

TESS: I didn’t have a hoodie, but the cloak has a hood on it. So, when I’m outside the room ushering people in, I will put the hood up and start waving the lightsaber around, and usually that’s enough to let people get the idea of what I’m doing. [laughing]

IVAN: That’s really great. Ok, so let’s actually stay on this track. We’ll get to the details of Drupaldelphia in a sec here, but since we’ve already started talking about your session, let’s keep going. So, your session is called "Return of the Clustering, Kubernetes for Drupal", and it’s part of a trilogy, and I’m going to say I think first of all it’s amazing that it’s part of a trilogy. So I’d like you to kind of just tell us what Episode 1 and Episode 2 are about, and my very technical question thereafter is going to be a follow-up, because this is a trilogy of trilogies, but we can get to that in a second. Let’s set up the Episode 1, Episode 2 part of this.

TESS: So the whole, "it’s a part of a trilogy" thing was actually a bit of a joke on myself. One of the first talks that I started doing outside of Minneapolis was on "Ride the Whale: Docker for Drupalists", and the idea was to teach people how to build your own Docker-based local development environment, and how Docker in general worked. And, that was the first start of it. Then the next talk that I gave was “Avoid Deep Hurting! Deployments beyond Git,” which introduced how to build your own continuous integration system using just open source, free stuff. Free stuff like Ansible, free stuff like Gitlab CI, and the idea behind that is now you have these two different pieces that don’t seem related, but become related in this talk.

I kind of sat back and thought about all the talks I’d given over the last few years and realized, Oh, geez, I made a trilogy. Oh, God, what did I do? When that occurred to me, I also knew what the whole theme of the talk was going to be, and because it was a trilogy, I decided, Okay, fine we’ll call it "Return of the Clustering" and we’ll do a whole Return of the Jedi motif, because why not?  [laughing]

IVAN: Exactly. So, we’ve set up the trilogy, now "Return of the Clustering" is a wonderful evolution, in my opinion, from your very beginning of trying to figure out how the heck are we going to set up Drupal in an easy Docker container or set of containers locally, and basically fast forward to the real meat of what we’re trying to do, and that’s run Docker in production? Right? So, give us a high-level description of "Return of the Clustering."

TESS: So, we first start out with describing, Wouldn’t it be great if we ran Docker in production? And we dissect the various problems with that. There are security problems that come inherent with running a Docker-based workload that you have to be aware of, and most people who just use Docker out of the box might not have ever considered these facts. And, it’s different than traditional server management, because there are a few different additional factors that come into play with how Docker executes things. Then you also have to worry about how you’re going to actually get the workload onto the cluster as well, and how you’re going to orchestrate it. One of the biggest problems is, it’s really easy to stand up Docker on a single server and have that single server run an entire workload of multiple containers.

There’s nothing wrong with that. You install Docker, you install Compose, you write a Compose file, you stand it up, you do some security things to make sure everything is loaded correctly, and there you go, you’re done. But, the problem is that your scaling is only vertical. You can only make your server bigger. You really want to make your scaling horizontal and add more servers. And this makes things very complex, because if you have to coordinate multiple Docker hosts together, doing that manually is not fun, and it’s not DevOps either. It’s not automatable in a very easy way. Fortunately, there are container orchestrators out there that know how to do this for you. So we talk about Docker Swarm and what the advantages and disadvantages of that are, and then we introduce Kubernetes. We talk about how the Kubernetes model is different, then we architect out an architecture that will run a Drupal site in Kubernetes. And, this is actually a subtle point that I keep having to remind myself and others, is that the Kubernetes model is so complicated, with so many little details, and so many different things in it, that you really get lost very quickly. And it took me a year to figure out which bits of the Kubernetes’ model work for Drupal, and which bits we don’t need to talk about. Once I understood those parts, it was easy to build an architecture from those minimal amounts of pieces.

So, we review what those pieces are, and we describe them. Then we talk about how to technically implement them within Kubernetes, and why Kubernetes is kind of nifty through its use of YAML. So, what we’re going to do is, we build all of that out. But now we have another problem, which is, if we want to build the dynamic cluster which supports multiple clients, multiple sites, that’s a lot of YAML we have to manage and now we’ve just introduced a problem that I covered in Avoid Deep Hurting, which is, we have introduced humans back into the mix. We need to take the people out of the mix and let the technology handle it for us, because it doesn’t get tired, it doesn’t make typos, it doesn’t need three more coffees in order to get through the day—at least not most days. [laughing] So, we combine Ansible and Kubernetes and Docker to build out an entire cluster in an automated fashion, by just changing a few different variables, and we run that on Digital Ocean.

Then we also found out that the problem is, to really effectively leverage Kubernetes for Drupal, you can’t just put an Apache container out on Kubernetes and then throw the site on it, and then update it in place like you did with traditional server management. It’s not really the way that Kubernetes wants to do things. So, instead, we end up having to build a custom container that contains the site code already, and then run that on Kubernetes directly. And that’s a much more cloud-native workflow, and it’s a bit of a paradigm shift from what a lot of people are used to. But, each one of these pieces works well together to create an effective, open source, minimal Kubernetes production cluster for Drupal.

IVAN: I love that it’s minimal. I mean, you just put what you absolutely need out there. And that’s kind of the philosophy and it reduces your security work, quite honestly, and it also reduces the attack vector. So, I’m so glad to have heard this summary about the talk. I hope that people go out there and watch the recording, and if you have an opportunity to attend TC Drupal Camp or Flyover Camp in Kansas, you’ll be at both of those places giving the talk again.

We went through your talk, let’s talk a little bit more about Drupaldelphia 2019. So, it’s a one-day camp, and it kind of looks like there were six tracks, which is a large number of tracks for a camp.

TESS: It was a surprising amount of things you could do in every time slot.

IVAN: And it feels like the tracks weren’t just Drupal, right? I mean we talked about this with Chris and Dan [organizers of TC Drupal Camp]. It kind of feels like there is this conscious effort in the camps, at least the ones I’ve been paying attention to recently, to be not just about developers, and not just about Drupal. So, what do you think about that?

TESS: I think that’s really a good strategy and it’s a lot more holistic than we’re used to going forward. I think that our industry is still getting used to the idea that there is an internet that we can connect with each other and research things, even though we do it every day. I don’t think that the cultural and emotional impact of that has really entirely sunk in. And a lot of events are changing to reflect that attitude, that it’s not just one piece of technology that we need to deeply investigate anymore. It’s now: how does this one piece interact with a whole bunch of other pieces? And some of those pieces aren’t technology. A lot of those pieces are people.

IVAN: Yeah, the people aspect is so important as well, and I’m so glad to see those tracks are appearing on the local camps. And, so, one day, six tracks, five sessions in the days, so about 30 sessions total in the whole camp, which sounds about exactly the same as what we’re doing in TC Drupal this year on the one day of the camp. I didn’t see a keynote on the schedule, an official keynote. I didn’t see intro and outro, welcoming remarks and ending remarks. Is that right? There was no keynote?

TESS: There was no keynote. There was a brief intro, and then the session started. And I think that actually worked fairly well for this camp, because I think that the shorter the camps, a keynote is less effective. And, this is a bit of an attitude change, because I remember when TC Drupal first didn’t have a keynote, and I kind of missed it. I kind of wanted that back, because one thing that was kind of nifty about having a keynote is that, it’s there to introduce you to the talk, and get you excited, and get you ready for the day. But for a one day camp, it’s not really that necessary, is it?

IVAN: No.

TESS: You’re already there. You already are invested to do the day, and there doesn’t seem to be much of a need to really do that. It might be that keynotes are a thing that are best suited for multi-day events.

IVAN: Yeah, that’s kind of what I’m thinking as well, multi-day events that maybe need higher attendance, and so you almost use the keynote to attract attendees that you hope stay on. So, maybe there’s a celebrity, or someone who’s done an awful lot of contribution that has the keynote. And I’ve always thought of that as an attractor, but it almost eats into the day if it’s a one-day camp.

TESS: Yeah. It usually takes a good hour and a half, and you could get a whole entire slot and a break in otherwise. And from an organizer perspective, attracting a keynoter is often very difficult, because if you’re a keynoter you usually should compensate them in some way, even if it’s just paying for lodging and a flight. But that can be a bit of an impediment, especially if you’re a smaller event, or a less established event, you might not be able to get someone to do a keynote. And, I think it might just be not as necessary anymore. It might just be better to have everything else.

What was also kind of nifty about Drupaldelphia is that they had these help desks, as well. Every time slot had an accessibility lab, a Drupal and Composer help desk, and a contribution workshop room. And I thought that was spectacular, because it allows people to drop out of the sessions and work on other things or get directed help. And a Drupal and Composer help desk, I thought that was a brilliant idea, because wow Composer. Composer can really, really ruin your day. [laughing]

IVAN: Yeah, I agree. So, is that how BoFs and Contributions will handle theirs basically during the whole day and maybe as options and alternatives to the sessions that were already scheduled? Is that how it was handled?

TESS: No, there was also a distinct BoF room, but it did share with that particular room, which I think might not have been necessarily the best choice. I think that it would’ve been a bit better to have a dedicated BoF room, and then a dedicated help desk lab room, because that would allow people to be less interrupted by people who are doing a BoF. And sometimes BoFs end up being sessions that were submitted, but weren’t accepted for whatever reason, and those could be kind of distracting.

IVAN: And animated. Sometimes you get people who show up in cloaks and sabers in those BoF sessions too. [laughing]

TESS: Only if it fits in my carry on. [laughing]

IVAN: [laughing] It sounds like you had a good time there, Tess. That’s great. So, it was on a Friday at the end of the work week but in the middle of the day. What was attendance like?

TESS: There were about 250 people who attended. It was surprisingly packed. I think this is one thing that I noticed that a lot of these one-day events have kind of a side effect that they can concentrate their attendance. A multi-day event actually has kind of the reverse effect. It spreads out the attendance to multiple days and makes it more difficult to actually advocate it. Particularly if one of those days is a weekend. If one of those days is a weekend, a lot of developers, depending on the age

bracket of the segment that you’re going for, and Drupal is tending to get a little bit more middle-aged than it was like 10 years ago.

IVAN: Oh, come on. Come on. [laughing]

TESS: Hey, I’m being honest here. I’ve been to a lot of events and I’ve looked at the age of the people showing up and it’s like, Yeah, I’m seeing it.

IVAN: I know what you’re saying. People are little wiser, a little grayer, a little more family, families are attending. Yeah, it’s true.

TESS: And, as a result, having time on weekends is kind of a difficult call, even a difficult ethical call, because some people will do this for work. And now you’re asking them to do work on a Saturday. That could be a little bit of a difficult call, whereas if it’s a one-day event that’s during the week, then it’s fairly concentrated. And at that point you know that it’s just a workday that’s being used, and if this is aligned with your career, there’s nothing wrong with that. And, then it doesn’t interfere with your weekend and that critical amount of recovery time, that we all need at the end of a work week.

IVAN: I agree. I think it’s good for mental health to do it the way that it was done. What was the location of the camp? What was that event space like? Where was it located?

TESS: I think it was called The Hussian School of Art; I think. Originally when I found the building, I went to the wrong door and some very nice security people at the door were like, “Where are you going and what the hell is Drupal?” [laughing] So I had to give them my spiel really quickly and then they said, “Oh, oh, you want that? It’s probably around the corner. So, go down to the end of the block and take a left.” It was really helpful. And then later on when I went to get some coffee I passed by there and I found a door. I found a sign there that said that you should always go around the corner. So, no one else had to repeat what I did in the morning. [laughing]

IVAN: Okay, good. That’s called iterating, I think. Were you wearing your cloak at the time that the security guards were giving you directions?

TESS: I think that I stuffed it in my bag while I was taking the cab to the event, because the only hotel I could get was actually right next to the airport. [laughing] 

IVAN: Oh, my goodness.

TESS: And so, I had to take a cab ride to go into town and I didn’t want to seem weirder than I usually am to people. So, I stuffed the whole thing in my backpack and fortunately a plastic lightsaber and a cloak does not require that much space intentionally. We covered that already. So, it was kind of handy.

IVAN: So, a successful space then you felt. Was there kind of an atrium or a gathering space for everyone where there was lunch? Or, how did that work?

TESS: There was this semi-open area. We had the quick intro in that semi-open area. And then they pulled out some moveable walls to enclose that to make another room, which was nifty. And there was a nice little gathering spot where you could see the tables, and you could get lunch, and you could interact with different vendors that were at the event. That was all kind of nice, and then there was a couple rooms down some hallways, and yeah, it was a nice, cozy space and I rather liked it.

IVAN: So, overall impression of the event then, seemingly positive?

TESS: Yeah, I really enjoyed it. I thought that was a great event, and I think that more people should go to it.

IVAN: So, definitely a candidate for going to next year. We’ll have to wait and see what the next episode looks like of the series, I guess. And whether there will be a next episode?

TESS: We’ll find out. There might be a case study afterwards where we talk about what we actually did that might be a little bit more interesting to people than an intro to Kubernetes itself, but I think that it’s necessary.

IVAN: You know, that’s a great idea, to do a case study. I know we’re working through actually implementing something live right now, so be taking notes Tess, because it sounds like you already have an idea for the case study talk.

TESS: (laughing) All right.

IVAN: All right, all right. Well, thank you so much for spending your time with me today, again. It’s truly been my pleasure to have you on. Tess Flynn is the DevOps Engineer here at TEN7 and she was at Drupaldelphia 2019, giving her talk, "Return of the Clustering, Kubernetes for Drupal." The slides are online and a recording of the session itself is available as well. Just visit this episode's webpage for the links. The URL to this episode is t7.io/ep60.

And, don’t forget, if you’d like a credit towards your new Digital Ocean account, just go to ten7.com/digitalocean and follow the link on the page to redeem it.

You’ve been listening to the TEN7 Podcast. Find us online at ten7.com/podcast. And if you have a second, do send us a message, we love hearing from you. Our email address is [email protected]. Until next time, this is Ivan Stegic. Thank you for listening.

May 23 2019
May 23

Your browser does not support the audio element. TEN7-Podcast-Ep-060-Drupaldelphia-2019.mp3

Summary

TEN7’s DevOps Tess Flynn aka @socketwench gives us her observations of the Drupaldelphia 2019 conference she recently attended, as well as a summary of her helpful session, “Return of the Clustering: Kubernetes for Drupal.”

Guest

Tess Flynn, TEN7 DevOps

Highlights

  • What it takes to get a good Jedi costume
  • Tess’s Trilogy of Talks
  • Summary of Tess’s Kubernetes session
  • How camps are not just about Drupal and not just about developers anymore
  • Short camps = no keynote
  • Help desks at the conference - what a great idea!

Links

Transcript

IVAN STEGIC: Hey everyone you're listening to the TEN7 Podcast, where we get together every fortnight and sometimes more often, to talk about technology, business and the humans in it. I'm your host Ivan Stegic. Let’s talk about the Drupaldelphia 2019 conference that happened recently in Philadelphia. And, joining me to give her thoughts on the camp is Tess Flynn, a seasoned guest here on the TEN7 Podcast. @socketwench welcome back to the podcast.

TESS FLYNN: Hello.

IVAN: I’m going to have to make you the guest host when I’m out of town. [laughing]

TESS: [laughing] Sure, why not. We could do one on terrible movies. [laughing]

IVAN: [laughing] That would be actually awesome. We should totally do that. All right, Drupaldelphia. What’s in a name? So, this is the annual Drupal Camp in Philadelphia, Pennsylvania, and this year it was part of something called Philly Tech Week 2019. And you Tess, were there representing TEN7. What do you know about the relationship between Philly Tech Week and Drupaldelphia?

TESS: Not much, actually. I was so focused on other things during the week, I didn’t have any time to really investigate it deeply. But I noticed that some of the other attendees didn’t know that either, [laughing] so at least I was in good company.

IVAN: Okay. And were there any signs around that referred to Philly Tech Week, or was it kind of just insular?

TESS: No, not that I noticed. I think that it was just part of the longer weeklong event. Like, some organizations will have this kind of co-boosting agreement, where we’ll talk about your event and then you’ll talk about our event, and it’s good for both of us, and we don’t have any other commonality. And there’s nothing wrong with that. That’s good. 

IVAN: That is good. Have you been to Drupaldelphia before?

TESS: I have not. This is my first time. It just showed up across my board, and I went, Huh, I haven’t done that one before, sure, why not?

IVAN: And we have a great talk that you have produced, and so, as a company, we’re making sure you get out and tell that to as many people as possible. So, it was a good opportunity, I think.

TESS: It’s the first talk that I’ve been giving in costume, which is hilarious.

IVAN: Right, I heard about this, and you talk about it in the session, in the recording. So, tell us about the costume.

TESS: So, the thing is that, when you’re doing air travel, it’s always best to only have carry-on luggage, because you never know if you’re going to have a delay, or if your flight is going to get lost, or if your luggage is going to get lost, or something. So, you should restrict yourself to a roll-aboard and a backpack. Now, I’m an old hat toward business travel, so, I’m used to the standard, roll-aboard-backpack modality, but it does limit what you could bring with you when you’re traveling. So, the main concern for coming up with an outfit is: how can I do this so that I minimize the amount of potential luggage space that I take up, while still being evocative of the thing that I’m trying to riff on. And after thinking about this for a while, I actually did come up with something that was rather clever. The easiest thing to do was first find a plastic lightsaber. Not an electronic lightsaber, not one that lights up, not anything else.

The recommendation is that it has to be a lightsaber that’s made of plastic, that fully collapses into the hilt, and has no other electronics. Why is this important? Because air travel. Because we don’t want to have to pull the lightsaber out and put it on the scan deck for everyone else to look at. When it’s just plastic, it’s just plastic, no one cares about it, just put it in the bag and no one cares. Fortunately for me, I found one that was The Last Jedi  branded, but it’s actually a model of Luke Skywalker's lightsaber, that was fully plastic, no electronics, completely collapses into the hilt, on Amazon for 10 bucks.

IVAN: Nice.

TESS: And that was really as far as I was going to take it, because having a prop in the talk isn’t necessarily the worst thing. But then I started getting a little bit ahead of myself and I go, Well, what else can I do? And it occurred to me that one thing that I could do was, I should probably do up the outfit a little bit more. And, one of the things I decided I could do to do up the outfit a little bit more is, what is one thing that a lot of Jedi have going for them? And so, I thought back to all of those hours that I spent watching all of those movies, and all of the comics I read, and everything else, and go, They have this thing for cloaks. That’s it! They have this thing for cloaks and robe-like things. Well, I can’t really do a robe-like thing, because that would require a lot more layers in order to do everything. However, the problem is that, in order to really be evocative, I still need the exterior robe. So, I went again to Amazon, and I found a $20 brown cloak with a hood that looked pretty darn close to what a Jedi would normally wear [laughing], and that’s all I got.

And, so, I am out on stage wearing my normal talk outfit, the infamous skull dress, and wearing this huge brown cloak and carrying a plastic lightsaber. And, I actually stand in front of the room I’m giving the talk in before it starts, ushering people in, acting like a complete fool. Like, everyone going like, What am I getting into here? Why is this person in costume? And I’ll show up to the camp in costume for the entire camp [laughing] because it’s fun and it’s different and it’s weird, and one thing that is really kind of useful is, this is meant to be a fun thing. It’s meant to add more flavor to the talk, and the talk is full of my style of corny jokes. So there’s nothing wrong with me also amping up the corny the entire time. This whole thing about me amping up the corny, actually started in, I think 2015, in DrupalCon where I showed up with a giant inflatable whale, and I literally carried that around with me until the talk started and then I gave it away within the first few minutes. [laughing]

IVAN: I love how involved you get in your talks and how personal your slides are, and the comics that are hand drawn and the characters that you have, they just really make it a joy to listen to you speak. And, I wondered, because I haven’t actually seen you give this talk yet, I will absolutely be going to it at TC Drupal Camp when you give it, and in my mind I wasn’t sure if it was a black cloak or a brown cloak, but I was pretty sure you went with brown, so I’m glad to hear you went with brown. I guess the question I have is, did you have a hoodie?

TESS: I didn’t have a hoodie, but the cloak has a hood on it. So, when I’m outside the room ushering people in, I will put the hood up and start waving the lightsaber around, and usually that’s enough to let people get the idea of what I’m doing. [laughing]

IVAN: That’s really great. Ok, so let’s actually stay on this track. We’ll get to the details of Drupaldelphia in a sec here, but since we’ve already started talking about your session, let’s keep going. So, your session is called "Return of the Clustering, Kubernetes for Drupal", and it’s part of a trilogy, and I’m going to say I think first of all it’s amazing that it’s part of a trilogy. So I’d like you to kind of just tell us what Episode 1 and Episode 2 are about, and my very technical question thereafter is going to be a follow-up, because this is a trilogy of trilogies, but we can get to that in a second. Let’s set up the Episode 1, Episode 2 part of this.

TESS: So the whole, "it’s a part of a trilogy" thing was actually a bit of a joke on myself. One of the first talks that I started doing outside of Minneapolis was on "Ride the Whale: Docker for Drupalists", and the idea was to teach people how to build your own Docker-based local development environment, and how Docker in general worked. And, that was the first start of it. Then the next talk that I gave was “Avoid Deep Hurting! Deployments beyond Git,” which introduced how to build your own continuous integration system using just open source, free stuff. Free stuff like Ansible, free stuff like Gitlab CI, and the idea behind that is now you have these two different pieces that don’t seem related, but become related in this talk.

I kind of sat back and thought about all the talks I’d given over the last few years and realized, Oh, geez, I made a trilogy. Oh, God, what did I do? When that occurred to me, I also knew what the whole theme of the talk was going to be, and because it was a trilogy, I decided, Okay, fine we’ll call it "Return of the Clustering" and we’ll do a whole Return of the Jedi motif, because why not?  [laughing]

IVAN: Exactly. So, we’ve set up the trilogy, now "Return of the Clustering" is a wonderful evolution, in my opinion, from your very beginning of trying to figure out how the heck are we going to set up Drupal in an easy Docker container or set of containers locally, and basically fast forward to the real meat of what we’re trying to do, and that’s run Docker in production? Right? So, give us a high-level description of "Return of the Clustering."

TESS: So, we first start out with describing, Wouldn’t it be great if we ran Docker in production? And we dissect the various problems with that. There are security problems that come inherent with running a Docker-based workload that you have to be aware of, and most people who just use Docker out of the box might not have ever considered these facts. And, it’s different than traditional server management, because there are a few different additional factors that come into play with how Docker executes things. Then you also have to worry about how you’re going to actually get the workload onto the cluster as well, and how you’re going to orchestrate it. One of the biggest problems is, it’s really easy to stand up Docker on a single server and have that single server run an entire workload of multiple containers.

There’s nothing wrong with that. You install Docker, you install Compose, you write a Compose file, you stand it up, you do some security things to make sure everything is loaded correctly, and there you go, you’re done. But, the problem is that your scaling is only vertical. You can only make your server bigger. You really want to make your scaling horizontal and add more servers. And this makes things very complex, because if you have to coordinate multiple Docker hosts together, doing that manually is not fun, and it’s not DevOps either. It’s not automatable in a very easy way. Fortunately, there are container orchestrators out there that know how to do this for you. So we talk about Docker Swarm and what the advantages and disadvantages of that are, and then we introduce Kubernetes. We talk about how the Kubernetes model is different, then we architect out an architecture that will run a Drupal site in Kubernetes. And, this is actually a subtle point that I keep having to remind myself and others, is that the Kubernetes model is so complicated, with so many little details, and so many different things in it, that you really get lost very quickly. And it took me a year to figure out which bits of the Kubernetes’ model work for Drupal, and which bits we don’t need to talk about. Once I understood those parts, it was easy to build an architecture from those minimal amounts of pieces.

So, we review what those pieces are, and we describe them. Then we talk about how to technically implement them within Kubernetes, and why Kubernetes is kind of nifty through its use of YAML. So, what we’re going to do is, we build all of that out. But now we have another problem, which is, if we want to build the dynamic cluster which supports multiple clients, multiple sites, that’s a lot of YAML we have to manage and now we’ve just introduced a problem that I covered in Avoid Deep Hurting, which is, we have introduced humans back into the mix. We need to take the people out of the mix and let the technology handle it for us, because it doesn’t get tired, it doesn’t make typos, it doesn’t need three more coffees in order to get through the day—at least not most days. [laughing] So, we combine Ansible and Kubernetes and Docker to build out an entire cluster in an automated fashion, by just changing a few different variables, and we run that on Digital Ocean.

Then we also found out that the problem is, to really effectively leverage Kubernetes for Drupal, you can’t just put an Apache container out on Kubernetes and then throw the site on it, and then update it in place like you did with traditional server management. It’s not really the way that Kubernetes wants to do things. So, instead, we end up having to build a custom container that contains the site code already, and then run that on Kubernetes directly. And that’s a much more cloud-native workflow, and it’s a bit of a paradigm shift from what a lot of people are used to. But, each one of these pieces works well together to create an effective, open source, minimal Kubernetes production cluster for Drupal.

IVAN: I love that it’s minimal. I mean, you just put what you absolutely need out there. And that’s kind of the philosophy and it reduces your security work, quite honestly, and it also reduces the attack vector. So, I’m so glad to have heard this summary about the talk. I hope that people go out there and watch the recording, and if you have an opportunity to attend TC Drupal Camp or Flyover Camp in Kansas, you’ll be at both of those places giving the talk again.

We went through your talk, let’s talk a little bit more about Drupaldelphia 2019. So, it’s a one-day camp, and it kind of looks like there were six tracks, which is a large number of tracks for a camp.

TESS: It was a surprising amount of things you could do in every time slot.

IVAN: And it feels like the tracks weren’t just Drupal, right? I mean we talked about this with Chris and Dan [organizers of TC Drupal Camp]. It kind of feels like there is this conscious effort in the camps, at least the ones I’ve been paying attention to recently, to be not just about developers, and not just about Drupal. So, what do you think about that?

TESS: I think that’s really a good strategy and it’s a lot more holistic than we’re used to going forward. I think that our industry is still getting used to the idea that there is an internet that we can connect with each other and research things, even though we do it every day. I don’t think that the cultural and emotional impact of that has really entirely sunk in. And a lot of events are changing to reflect that attitude, that it’s not just one piece of technology that we need to deeply investigate anymore. It’s now: how does this one piece interact with a whole bunch of other pieces? And some of those pieces aren’t technology. A lot of those pieces are people.

IVAN: Yeah, the people aspect is so important as well, and I’m so glad to see those tracks are appearing on the local camps. And, so, one day, six tracks, five sessions in the days, so about 30 sessions total in the whole camp, which sounds about exactly the same as what we’re doing in TC Drupal this year on the one day of the camp. I didn’t see a keynote on the schedule, an official keynote. I didn’t see intro and outro, welcoming remarks and ending remarks. Is that right? There was no keynote?

TESS: There was no keynote. There was a brief intro, and then the session started. And I think that actually worked fairly well for this camp, because I think that the shorter the camps, a keynote is less effective. And, this is a bit of an attitude change, because I remember when TC Drupal first didn’t have a keynote, and I kind of missed it. I kind of wanted that back, because one thing that was kind of nifty about having a keynote is that, it’s there to introduce you to the talk, and get you excited, and get you ready for the day. But for a one day camp, it’s not really that necessary, is it?

IVAN: No.

TESS: You’re already there. You already are invested to do the day, and there doesn’t seem to be much of a need to really do that. It might be that keynotes are a thing that are best suited for multi-day events.

IVAN: Yeah, that’s kind of what I’m thinking as well, multi-day events that maybe need higher attendance, and so you almost use the keynote to attract attendees that you hope stay on. So, maybe there’s a celebrity, or someone who’s done an awful lot of contribution that has the keynote. And I’ve always thought of that as an attractor, but it almost eats into the day if it’s a one-day camp.

TESS: Yeah. It usually takes a good hour and a half, and you could get a whole entire slot and a break in otherwise. And from an organizer perspective, attracting a keynoter is often very difficult, because if you’re a keynoter you usually should compensate them in some way, even if it’s just paying for lodging and a flight. But that can be a bit of an impediment, especially if you’re a smaller event, or a less established event, you might not be able to get someone to do a keynote. And, I think it might just be not as necessary anymore. It might just be better to have everything else.

What was also kind of nifty about Drupaldelphia is that they had these help desks, as well. Every time slot had an accessibility lab, a Drupal and Composer help desk, and a contribution workshop room. And I thought that was spectacular, because it allows people to drop out of the sessions and work on other things or get directed help. And a Drupal and Composer help desk, I thought that was a brilliant idea, because wow Composer. Composer can really, really ruin your day. [laughing]

IVAN: Yeah, I agree. So, is that how BoFs and Contributions will handle theirs basically during the whole day and maybe as options and alternatives to the sessions that were already scheduled? Is that how it was handled?

TESS: No, there was also a distinct BoF room, but it did share with that particular room, which I think might not have been necessarily the best choice. I think that it would’ve been a bit better to have a dedicated BoF room, and then a dedicated help desk lab room, because that would allow people to be less interrupted by people who are doing a BoF. And sometimes BoFs end up being sessions that were submitted, but weren’t accepted for whatever reason, and those could be kind of distracting.

IVAN: And animated. Sometimes you get people who show up in cloaks and sabers in those BoF sessions too. [laughing]

TESS: Only if it fits in my carry on. [laughing]

IVAN: [laughing] It sounds like you had a good time there, Tess. That’s great. So, it was on a Friday at the end of the work week but in the middle of the day. What was attendance like?

TESS: There were about 250 people who attended. It was surprisingly packed. I think this is one thing that I noticed that a lot of these one-day events have kind of a side effect that they can concentrate their attendance. A multi-day event actually has kind of the reverse effect. It spreads out the attendance to multiple days and makes it more difficult to actually advocate it. Particularly if one of those days is a weekend. If one of those days is a weekend, a lot of developers, depending on the age

bracket of the segment that you’re going for, and Drupal is tending to get a little bit more middle-aged than it was like 10 years ago.

IVAN: Oh, come on. Come on. [laughing]

TESS: Hey, I’m being honest here. I’ve been to a lot of events and I’ve looked at the age of the people showing up and it’s like, Yeah, I’m seeing it.

IVAN: I know what you’re saying. People are little wiser, a little grayer, a little more family, families are attending. Yeah, it’s true.

TESS: And, as a result, having time on weekends is kind of a difficult call, even a difficult ethical call, because some people will do this for work. And now you’re asking them to do work on a Saturday. That could be a little bit of a difficult call, whereas if it’s a one-day event that’s during the week, then it’s fairly concentrated. And at that point you know that it’s just a workday that’s being used, and if this is aligned with your career, there’s nothing wrong with that. And, then it doesn’t interfere with your weekend and that critical amount of recovery time, that we all need at the end of a work week.

IVAN: I agree. I think it’s good for mental health to do it the way that it was done. What was the location of the camp? What was that event space like? Where was it located?

TESS: I think it was called The Hussian School of Art; I think. Originally when I found the building, I went to the wrong door and some very nice security people at the door were like, “Where are you going and what the hell is Drupal?” [laughing] So I had to give them my spiel really quickly and then they said, “Oh, oh, you want that? It’s probably around the corner. So, go down to the end of the block and take a left.” It was really helpful. And then later on when I went to get some coffee I passed by there and I found a door. I found a sign there that said that you should always go around the corner. So, no one else had to repeat what I did in the morning. [laughing]

IVAN: Okay, good. That’s called iterating, I think. Were you wearing your cloak at the time that the security guards were giving you directions?

TESS: I think that I stuffed it in my bag while I was taking the cab to the event, because the only hotel I could get was actually right next to the airport. [laughing] 

IVAN: Oh, my goodness.

TESS: And so, I had to take a cab ride to go into town and I didn’t want to seem weirder than I usually am to people. So, I stuffed the whole thing in my backpack and fortunately a plastic lightsaber and a cloak does not require that much space intentionally. We covered that already. So, it was kind of handy.

IVAN: So, a successful space then you felt. Was there kind of an atrium or a gathering space for everyone where there was lunch? Or, how did that work?

TESS: There was this semi-open area. We had the quick intro in that semi-open area. And then they pulled out some moveable walls to enclose that to make another room, which was nifty. And there was a nice little gathering spot where you could see the tables, and you could get lunch, and you could interact with different vendors that were at the event. That was all kind of nice, and then there was a couple rooms down some hallways, and yeah, it was a nice, cozy space and I rather liked it.

IVAN: So, overall impression of the event then, seemingly positive?

TESS: Yeah, I really enjoyed it. I thought that was a great event, and I think that more people should go to it.

IVAN: So, definitely a candidate for going to next year. We’ll have to wait and see what the next episode looks like of the series, I guess. And whether there will be a next episode?

TESS: We’ll find out. There might be a case study afterwards where we talk about what we actually did that might be a little bit more interesting to people than an intro to Kubernetes itself, but I think that it’s necessary.

IVAN: You know, that’s a great idea, to do a case study. I know we’re working through actually implementing something live right now, so be taking notes Tess, because it sounds like you already have an idea for the case study talk.

TESS: (laughing) All right.

IVAN: All right, all right. Well, thank you so much for spending your time with me today, again. It’s truly been my pleasure to have you on. Tess Flynn is the DevOps Engineer here at TEN7 and she was at Drupaldelphia 2019, giving her talk, "Return of the Clustering, Kubernetes for Drupal." The slides are online and a recording of the session itself is available as well. Just visit this episode's webpage for the links. The URL to this episode is t7.io/ep60.

And, don’t forget, if you’d like a credit towards your new Digital Ocean account, just go to ten7.com/digitalocean and follow the link on the page to redeem it.

You’ve been listening to the TEN7 Podcast. Find us online at ten7.com/podcast. And if you have a second, do send us a message, we love hearing from you. Our email address is [email protected]. Until next time, this is Ivan Stegic. Thank you for listening.

May 08 2019
May 08

Your browser does not support the audio element. TEN7-Podcast-Ep-059-2019-Twin-Cities-Drupal-Camp_0.mp3

Chris Weber and Dan Moriarty, volunteer organizers for the 2019 Twin Cities Drupal Camp are today's podcast guests. We'll be talking about the changes to this year's TCDrupal Camp and fond memories of previous camps. 

TCDrupal Camp is a three-day conference for open source enthusiasts, designers, hackers, geeks, developers, UI experts, IT managers and anyone else that wants to find out more about Drupal. It’s a great place to learn, code, network and have fun with your fellow Drupalistas.

Host: Ivan Stegic
Guests: Chris Weber, software engineer at The Nerdery and Dan Moriarty CEO and Creative Director at Electric Citizen
Running time: 32 minutes 

In this podcast we'll discuss: 

  • TCDrupal Camp's location at St. Thomas
  • Format changes: three days instead of four, 45-minute talks
  • Fewer days, but just as many parties! And food trucks, board gaming and karaoke
  • Focus on expanding talks to topics outside of just Drupal 
  • The House of Balls, a Minneapolis institution
  • How TCDrupal Camp's spontaneity is what makes it great
  • TCDrupal Camp's history
  • So much Drupal goodness coming to Minneapolis (DrupalCon 2020) how will we manage it all?

Links:

TRANSCRIPT

IVAN STEGIC: Hey everyone you're listening to the TEN7 Podcast, where we get together every fortnight and sometimes more often, to talk about technology, business and the humans in it. I'm your host Ivan Stegic. My guests today are Chris Weber and Dan Moriarty, two of the volunteer organizers of this year's Twin Cities Drupal Camp.

TC Drupal Camp

IVAN: Chris is a software engineer at The Nerdery, and Dan is CEO and Creative Director at Electric Citizen. Hello Chris and Dan. Welcome to the podcast.

CHRIS WEBER: Hello, hello.

DAN MORIARTY: Hey there. Thanks for having us.

IVAN: Oh, it’s my pleasure. Dan Moriarty, I love saying your name. The whole Sherlock Holmes thing, I just love it.

DAN: Yeah, and I will take that anytime. I'm always happy to reference my evil ancestors. [laughing]

IVAN: [laughing] Oh wait! Relation? Are you related to a fictitious person?

DAN: I’ll claim that.

IVAN: [laughing] That's awesome. Well I'm glad that you are on the show with us today talking about Twin Cities Drupal Camp this year. So, Chris, tell us about the camp itself. When and where is it this year?

CHRIS: Well this 2019 version of our camp, is going to be at St. Thomas which is in downtown Minneapolis. We've had it at St. Thomas for a number of years, so it should be familiar to folks that have gone to the Twin Cities Drupal Camp before. It's a really good location, really large open space, very, very lighty and breathy. We’ll be having it on June 6th through June 8th. June 6th is a training day. The 7th will be filled with excellent talks, sessions. And then the 8th will be kind of something a little bit new that we're doing. We're having an unconference on that day, as well as providing a space for people who want to sprint on core contributions. And we’re very excited to have the camp again here in the Twin Cities.

IVAN: And so that's a little different than how we've done it, I would guess, every year since we started, although I don't remember the first year. But that's a Thursday, Friday, Saturday, as opposed to Thursday, Friday, Saturday, Sunday. So, we have one day of sessions as opposed to two. Dan, do you want to talk through kind of the reasoning behind that, and why we decided to do it that way this year?

DAN: Yeah. So, it's something we've talked about off and on for a few years now, and we really as a group decided a couple of things. One is, four days was becoming a fairly large time commitment for a lot of people to participate in the full range of camp activities. And then another reason is we generally saw a bit of a drop off in attendance when we went from the weekday to the weekend. And so, as sort of a trial thing we're doing this year is reducing it to three days, with keeping our focus on the sessions like we've always done on Friday. But then on Saturday, making it a little more open free form, which is the unconference, which we can get into, just to see what that does for our numbers and helps more people participate than on the weekends.

IVAN: So, is the unconference style going to be very similar to the way we did BOFs (Birds of a Feather meetings) in the past? Or, how's that going to be structured?

DAN: You know, that's how I picture it. Although it is still a matter of discussion between various camp organizers, how exactly we're going to do it. But the way I'm envisioning it—and Chris can correct me if I'm wrong—is we're going to largely be in the atrium area on Saturday as opposed to going to the classrooms, and people will sort of self-organize into different groups around that large space just to have informal discussions about whatever topics they would like. And then ideally we'll have a few moderators available, floating around the room to sort of help facilitate conversations and make sure people are in the spaces that they find most helpful.

CHRIS: That's right. In the unconference format, we're looking for interesting things to talk about. Tim likes to bring up that Tim [Ericson] and Wilbur [Ince] were the genesis of this idea, when we were talking about adding it to the camp. He likes to talk about the law of two feet, where if you're in a conversation that isn't providing you with what you need, you could use your two feet in order to find another conversation that is more engaging. Then in that way, kind of like plan your own day out of what talks are engaging you and finding the information you need. But the format is very much like a BOF. Instead of slides and rooms, and a more instructor-led conversation, where one person is just talking for an hour or whatever, it's more of a conversation, sharing of input and allowing more people to provide information than just one person at a time.

IVAN: I love the idea of doing that. It really allows, I think, the community to drive what the topics are and the discussions that are being had. I think that's a good experiment. I'm looking forward to participating and seeing how that affects our camp next year. I was going to ask about Thursday. Usually we have trainings on Thursdays, right? Can you speak to what the trainings are this year?

CHRIS: It looks like Drupal 8 content migrations. This can be a getting started with building sites with Drupal. There is a Drupal 8 crash course for content editors, marketers and project managers. Then intermediate to advanced CSS for practical peoples. I think those are all of our trainings.

DAN: Those are the four trainings, and then on top of the trainings we're also hosting a mini-camp this year.

IVAN: Oh really? Well, tell me about that. I know Backdrop has always got such a great presence at TC Drupal Camp every year. What is that about?

DAN: So, in the past we’ve hosted sessions on Backdrop. Every year we seem to draw some of the leaders behind the Backdrop community, and we'll do that again this year. Particularly Jen Lampton, who's helped lead and create the Backdrop community, she's coming, as well as several other prominent Backdrop contributors. And, what they've decided to do this year in the form of a mini-camp, is hold the day of sort of sessions, all in one room dedicated to Backdrop. And we as a camp decided to provide a room to sit alongside the training sessions for people that are interested in contributing to Backdrop, or learning about it to attend to this free session.

IVAN: I think that's a great idea. And why haven't I reached out to Jen and Nate about Backdrop? We should totally have them on the podcast.

CHRIS: There you go.

IVAN: That's awesome. We will make sure we do that. If you happen to be listening out there, Jen and Nate, please send us an email before we do. But yeah, we'd love to have you on the podcast. So that's great.

So, trainings on Thursday, Backdrop CMS minicamp as well in one of the rooms, and then sessions on Friday. So, I would imagine there is a keynote on Friday? Let's talk about what the day looks like on Friday.

DAN: I can tell you definitively we've got five rooms. So, five tracks if you will, and each track will have six sessions throughout the day, for a total of 30 sessions. We're starting the morning like we typically do with a welcome session at 9 am, going into our first session around 9:45 and continuing with the last session ending around 5:00. We will have a keynote this year during the lunch hour on Friday, and I'm happy to tell you about that.

That is a local group called the Asian Penguins, a Linux user group made of boys and girls grades 6 through 8, and they're based out of Hmong charter school in St. Paul. Their director Stu Keroff, is coming to tell the story about what they do, how their work is helping bridge the digital divide in the metro area. He works with the students to teach them Linux, and they repurpose old computers installing Linux on them and giving them away to families in need. So, we’re excited. He's going to bring some of his students with him too, and they're going to do a presentation for us on Friday.

IVAN: Oh, that's wonderful. That's really wonderful to hear. So, the Asian Penguins. Wow, how did you guys find out about them? Meet them? Involve them? What's the story behind that?

CHRIS: Matthew Tiff could probably give you the full answer of how that connection was formed. We found out about them through him, and then we were able to find out more about their organization. And it just sounds like a really great opportunity. I know you share an interest in making sure that tech is accessible to kids as well, Ivan, and it's really great to hear what they're doing.

IVAN: I'm not surprised that Matthew is involved in that.

DAN: Yeah. Matthew had hosted Stu on his Hacking Culture podcast a few months ago, and then recommended them as a potential keynote speaker. So, we reached out to them and we're just finalizing the details on that now, and it should be up on the site by the time this podcast comes out.

IVAN: Oh, that's great. I'm looking forward to hearing what they have to say. So, keynote over the lunch hour. Sessions in the morning, sessions in the afternoon. Can you tell me a little bit about a session format? Are they the same as last year? How long are they? What do those look like?

CHRIS: Yeah. We actually had quite a bit of a debate on how long our sessions should be. You know Drupalcon has moved to a format of half-hour talks and longer talks more than an hour. Right? It's like an hour and a half. And we were concerned about what is the appropriate amount of duration, so we wanted to make sure that we've got a lot of talks that people can give on Friday. But at the same time we were concerned that a half hour might be too short. We're trying 45-minute talks out this year. We're gonna see how that goes. And as a result, we were able to fit about 30 talks into that Friday.

IVAN: Is that 45 minutes of speaker time, or is that 45 minutes of session plus questions?

DAN: It gives time for questions at the end of sessions, unlike 30-minute sessions. You know that was a common experience at DrupalCon this year, is, there really weren't any time for questions at the end of those 30-minute sessions and speakers are really hard pressed to fit all their content in 30 minutes. So even though DrupalCon experimented with this and I think that's fine, we as a group felt like 45 minutes was a much better time slot, and I wouldn't be surprised if they go back to that at feature DrupalCons.

CHRIS: I wouldn't be surprised with that either.

IVAN: So, the 45 minutes is inclusive of the questions then?

DAN: Right. I mean, the assumption is that the speakers will have time then to answer questions.

IVAN: Got it. Ok. And so, let's talk about the five rooms and the five tracks you have. What are the tracks this year?

CHRIS: Well the tracks are similar. If you look at the website right now, they're almost identical to the tracks that we had last year. But we're making an effort this year to be inclusive of talks that are tangential from Drupal. Not every talk has to be about Drupal. We've got talks about GraphQL, JSON integrations and Ruby on Rails. We wanted to make sure that we've got some talks about mental health. We've got talks on a wide area of topics and not necessarily specifically about Drupal.

IVAN: That's great. I'm looking forward to seeing the list of sessions come out. I'm hoping, fingers crossed, that my session made it. When is that session list going to be published?

DAN: That's a good question. [laughing] The announcements will have gone out to the accepted speakers well ahead of this podcast being released. I don't know that we'll have them accepted and published on the site at that point, but we'll be publishing them hopefully by, what would you say Chris, mid-May at the latest?

CHRIS: Yeah, hopefully earlier, but that is largely based upon how we can contact folks. As our good friend Joe [Shindelar] was telling us, we've never had somebody tell us they either can't make it, or say that they can and don't show up. We've had a really high success rate, and we’d like to keep that going, but there's always the possibility that the worst could happen. If we don't get a hold of somebody, or we have to strategically plan, it's better to have everything figured out before we publish. And so, we're putting the effort in now in order to make sure that can happen.

IVAN: So, if you're listening to this podcast there's a chance the sessions have been published but there's a chance the sessions have not yet been published [laughing]. And if they haven't been published, we promise they will be published in the next week after you listen to this. So, fingers crossed.

DAN: Absolutely.

CHRIS: Let’s hope it works out.

IVAN: Let's hope it does. Ok. So, one of the things I love—I love a lot of things about the Drupal Camp in Minneapolis—is the parties. There's always the speaker party and the sponsor appreciation party, and then there's the Friday night party and the Saturday night party. But if the camp is one day short of four days, does that mean it's one day short of a party as well?

DAN: Absolutely not. [laughing]

IVAN: Oh good. Let's hear about what’s going on there.

DAN: Right. Well we do have a few changes this year. I think one of the big ones is that our Thursday night party, which is the day that camp opens after the training, we're trying something new, sort of inspired by our friends at Midcamp in Chicago, and that is changing the Thursday night party to the welcoming party. And what this means is that we're extending an invitation to anyone that is involved or interested in participating in our conference to come to the welcome party on Thursday.

CHRIS: That's right.

IVAN: And where is that party this year?

DAN: The unofficial plan right now is that we're going to host that at Pizza Lucé in downtown Minneapolis.

IVAN: And what about Friday? The Friday night party?

DAN: Yeah. So, Friday we're going back to the House of Balls.

IVAN: Oh yeah. I love that place.

CHRIS: Yeah, it's a really great place.

DAN: House of Balls, we’ve been at, I think this is our fourth year at this, sort of amazing, eccentric art studio/event space, just off of downtown Minneapolis. And we're going to have some of the same great things we have every year. We're lining up a food truck. We're going to have free food and drinks and most importantly, we’ve lined up karaoke.

IVAN: Oh yeah. That sounds amazing. I'm secretly hoping that Marc [Drummond] is able to give his five-minute talk about hotdish again.

DAN: Yeah, Chris, are we going to try to do any of the lightning talks this year?

CHRIS: Well I don't know. We like to be flexible. We're kind of a spontaneous crowd. We've got a number of events planned for the day. You know, we're gonna have some board gaming, and it seems like the board gaming thing has gotten even stronger here in the Twin Cities community. We're going to have some food and, of course, there will be Foursquare.

DAN: Foursquare.

IVAN: I hope Les [Lim] brings his ball.

Foursquare

CHRIS: We lean on Les for both the rules and the gamesmanship and the setup of that. We should side note, we should double check with him, if he's going to do that again this year, or if he wants one of us to take that on. And, yeah, in years past we've had lightning talks and we've also had karaoke. I do know for a fact that we will be having karaoke again this year. I don't know if we'll have lightning talks, but there's room still, I think Dan, we just need to put a plan into action to see if we can provide equipment and time for that.

IVAN: I'm a proponent of the lightning talks, so if you need votes you have one from me, and if you do need something to help make that happen, please ask, I'll do what I can.

DAN: Great. I think we've got a volunteer to run the lightning talks. [laughing] 

CHRIS: Sounds great. That's how this works.

IVAN: [laughing] Ok, well, if I get to run it that means I get to give one too.

CHRIS: [laughing] Indeed. You can kick us off.

IVAN: Alright. Let’s do that. Let me know the details, and I'll help make it happen.

DAN: Alright. Sounds good.

IVAN: Alright. So that's Thursday and Friday taken care of, and what are we doing Saturday? Are we doing anything Saturday?

DAN: We will. We'll do our traditional post-camp party. It is at a location to be determined. So, you have to stay tuned to the website or the newsletter to find out when and where that's going to happen.

IVAN: Well I'm glad that's still happening even though we don't have the fourth day. Stay tuned on the website. That's tcdrupal.org and subscribe to the email list, I'm sure that'll be mentioned in the email as well. One more question. How do you register for camp and what does it cost?

CHRIS: Well you can go to our website at tcdrupal.org/register and you can register right there. We've got a nice big link for you right there in the top of the page, just click on that, go on over to registration. Registration remains inexpensive, especially compared to other Twin Cities camps which we've been able to look at the cost of camps nearby. Our camp's only $50. We are providing means for people who want to contribute more. Like myself, I tried to come in at the Community Contributor level. How much is that again Dan?

DAN: Yeah. So that's $150, and that includes camp registration, a free T-shirt, and it also means that you are helping support the camp above and beyond, which is really key to us being able to offer all these things, including the parties and the free training and all the sessions and the venue.

So, that's kind of a new, it's not new, but what's new this year is, is we're really trying to emphasize to anyone that uses Drupal professionally and that can afford it, please consider coming in at the Community Sponsor level, Community Supporter level. It really helps us out. But anyone is welcome to come to camp, and as always if anyone wants to come and can't afford it, please contact us, and we would be happy to set you up for free.

IVAN: What's the best way to contact you?

DAN: Yeah, so, go to the website tcdrupal.org, go to the contact page and just shoot us a message, and one of us would be happy to get back to you. Or you can hit us up on Twitter as well.

CHRIS: Yeah. Like Dan said, if you fill out the contact form on site, you're sending an e-mail message to the entire team. Someone's going to see that immediately. And, again, we're available over Twitter just like the rest of the Drupal community. We all kind of hang out there.

IVAN: And, there are sponsors again this year, like there were last year. There always seems to be a plethora of sponsors for camp, which is just so awesome to see for our little community. Are there still opportunities to sponsor? What options are left, if they are?

DAN: Please, please, please, always welcome more sponsors. The more sponsors we get, the more we can do. You know, we really are wanting and planning to offer free lunch to everyone at camp this year on Friday, and getting a few more sponsors really help make that happen. And so, we have some great sponsors so far including TEN7, thank you for that.

IVAN: Yeah!

DAN: And, you know, we have a few platinum sponsor slots still available. We have unlimited slots at the gold and silver level. And so anyone who wants to consider both helping the camp out and maybe getting a table to tell people about your organization or what you do, you're very welcome to do that. And again, just come to the website. There's information about how to become a sponsor,  or to just get in touch with someone.

IVAN: So, that URL is tcdrupal.org/sponsors, and there's a great little button there that you can visit the sponsor page for more information about the benefits of each of the sponsor levels. Yeah, it's been great to see the same companies coming back to the camp and coming back and providing to the community. It’s always a pleasure for us to do it, and I'm sure it is for you too Dan for Electric Citizen and for the others that are also doing that.

I’ve asked this before of members of our community and of members of the organizing team that always puts on this volunteer event. It's volunteers that do it. I'm amazed that it happens every year. But DrupalCon 2020 is in Minneapolis next year, and DrupalCon just happened last month, and we have our camp in close proximity to it. So, has there been any discussion about what, if any effect DrupalCon in Minneapolis is going to have on our camp next year?

CHRIS: So we've had a lot of internal discussions about it, and while we have a lot of energy in the Twin Cities, it seems like the prevailing wisdom is that we want to try to find a couple of smaller events. The work that we anticipate we're going to put in around DrupalCon is really too close to where we would want to have our camp here in the Twin Cities, to make both the contribution we want to put in to make DrupalCon a success and the contribution we want to put in in order to make our camp a success. That said, it's still kind of up in the air.

We haven't had the powwow that we really need in order to come to a firm decision that, “Hey, we're not going to do a Drupal Camp,” or “Hey, we're not going to do a Drupal Camp like later in the fall sometime that day of the year.” So, I guess the answer that we have right now is that, we want to continue to be active. We want to do things in the Twin Cities surrounding Drupal and getting together an event. And I think we've got different ideas on how to accomplish that, but the main thing we want to do is to continue to talk about Drupal, celebrate Drupal and promote knowledge and learning and inclusiveness.

IVAN: So, "Stay tuned. We're evolving the decision as time progresses," is what you’re saying?

CHRIS: Yeah. So, we don't have a good answer yet. We're all so laser focused on getting this year's camp put together and have it be so awesome, that we've postponed any other kind of discussion of what's next, until we're done.

IVAN: And thank you for being so laser focused on the camp, and Dan and Chris and everyone else that's helped organize the camp, Jer [Davis] and Tim [Erickson] and all of the other volunteers. It's just always so amazing for me to see the camp happen and for all those people to contribute and for there to be so much empathy and care that it happens in the most equitable and fun and cheap and value-based event that we can put on, and I think that's great. So, thank you both for doing that and for contributing.

CHRIS: After this is all done, there’s so much gratitude to make sure people get, based upon their efforts that they've been able to put in to make this thing a success. And the thing that we keep talking about, it's really our deliverable at the end of this is our process, because our process has been pretty good. We keep on iterating on it, so that we can have the confidence that, “Hey, we can put together a camp like this,” and we could feel really good about that process.

DAN: And not only that, but I've been involved in many years of camp organizing for TCDrupal, and I feel like every year is good, but the gang's really getting along well this year to where I'm not even daunted by the thought of doing it again next year.

CHRIS: I would love to do it again.

IVAN: You heard it here first. [laughing] We’re already thinking about the following year's Drupal camp. That's great.

CHRIS: So that's the high we're on right now from all the good work we’re doing. We’ll see how we’re feeling after this.

IVAN: [laughing] No, you can’t go back now. You just said that you're not even worried about it. So, let's actually just spend a minute before we close here, and say this is version 9 of the camp, if I'm not mistaken. I think the first one was in 2011, so, this would be version 9, and so the next one is the 10th anniversary. Right? So, we should celebrate that somehow.

DAN: Well, we are. It's called Drupalcon 2020, [laughing] and what better way to cap off 10 years of active community growing, stewardship, caretaking, whatever you want to call it. I myself came to this Drupal community group as a lone wolf developer looking to find some other group of people that I can nerd out about Drupal with. And my story is basically the story of how successful this community has been. Thanks to all of the people who have welcomed me in and made me feel like I belonged. I'm here today helping plan the next one.

IVAN: I love it. I think it's precious and amazing, and I'm always amazed by all of that. So, yeah. I hope I'm right about it being the 10th anniversary, because I feel like there were different incarnations of the camp before 2011, but I think 2011 was the first official one, right?

DAN: It was. Yep, you're absolutely right.

IVAN: Ok, good. Well, thank you both for spending your precious time with me today. It's really been a pleasure talking with you Chris and Dan.

CHRIS: Same here, man.

DAN: Yeah, thanks so much for hosting us.

IVAN: Chris and Dan are two of the volunteer organizers of Twin Cities Drupal Camp happening from June 6th, a Thursday to June 8th, a Saturday, at the University of St. Thomas in downtown Minneapolis. Tickets are still available and they're reasonably priced starting at $50, and we're hoping that includes lunch as well.

So, head on over to tcdrupal.org and register now. You can find the camp on Twitter, Facebook and Instagram. The handle is @tcdrupal. And, of course, the Twin Cities Drupal group is also on groups.drupal.org/twin-cities for other local events that happen outside of camp, and they happen every month, whether it's the happy hour or something else, it is on.

You’ve been listening to the TEN7 Podcast. Find us online at ten7.com/podcast. And if you have a second, do send us a message, we love hearing from you. Our email address is [email protected]. Until next time, this is Ivan Stegic. Thank you for listening.

May 08 2019
May 08

Your browser does not support the audio element. TEN7-Podcast-Ep-059-2019-Twin-Cities-Drupal-Camp_0.mp3

Chris Weber and Dan Moriarty, volunteer organizers for the 2019 Twin Cities Drupal Camp are today's podcast guests. We'll be talking about the changes to this year's TCDrupal Camp and fond memories of previous camps. 

TCDrupal Camp is a three-day conference for open source enthusiasts, designers, hackers, geeks, developers, UI experts, IT managers and anyone else that wants to find out more about Drupal. It’s a great place to learn, code, network and have fun with your fellow Drupalistas.

Host: Ivan Stegic
Guests: Chris Weber, software engineer at The Nerdery and Dan Moriarty CEO and Creative Director at Electric Citizen
Running time: 32 minutes 

In this podcast we'll discuss: 

  • TCDrupal Camp's location at St. Thomas
  • Format changes: three days instead of four, 45-minute talks
  • Fewer days, but just as many parties! And food trucks, board gaming and karaoke
  • Focus on expanding talks to topics outside of just Drupal 
  • The House of Balls, a Minneapolis institution
  • How TCDrupal Camp's spontaneity is what makes it great
  • TCDrupal Camp's history
  • So much Drupal goodness coming to Minneapolis (DrupalCon 2020) how will we manage it all?

Links:

TRANSCRIPT

IVAN STEGIC: Hey everyone you're listening to the TEN7 Podcast, where we get together every fortnight and sometimes more often, to talk about technology, business and the humans in it. I'm your host Ivan Stegic. My guests today are Chris Weber and Dan Moriarty, two of the volunteer organizers of this year's Twin Cities Drupal Camp.

TC Drupal Camp

IVAN: Chris is a software engineer at The Nerdery, and Dan is CEO and Creative Director at Electric Citizen. Hello Chris and Dan. Welcome to the podcast.

CHRIS WEBER: Hello, hello.

DAN MORIARTY: Hey there. Thanks for having us.

IVAN: Oh, it’s my pleasure. Dan Moriarty, I love saying your name. The whole Sherlock Holmes thing, I just love it.

DAN: Yeah, and I will take that anytime. I'm always happy to reference my evil ancestors. [laughing]

IVAN: [laughing] Oh wait! Relation? Are you related to a fictitious person?

DAN: I’ll claim that.

IVAN: [laughing] That's awesome. Well I'm glad that you are on the show with us today talking about Twin Cities Drupal Camp this year. So, Chris, tell us about the camp itself. When and where is it this year?

CHRIS: Well this 2019 version of our camp, is going to be at St. Thomas which is in downtown Minneapolis. We've had it at St. Thomas for a number of years, so it should be familiar to folks that have gone to the Twin Cities Drupal Camp before. It's a really good location, really large open space, very, very lighty and breathy. We’ll be having it on June 6th through June 8th. June 6th is a training day. The 7th will be filled with excellent talks, sessions. And then the 8th will be kind of something a little bit new that we're doing. We're having an unconference on that day, as well as providing a space for people who want to sprint on core contributions. And we’re very excited to have the camp again here in the Twin Cities.

IVAN: And so that's a little different than how we've done it, I would guess, every year since we started, although I don't remember the first year. But that's a Thursday, Friday, Saturday, as opposed to Thursday, Friday, Saturday, Sunday. So, we have one day of sessions as opposed to two. Dan, do you want to talk through kind of the reasoning behind that, and why we decided to do it that way this year?

DAN: Yeah. So, it's something we've talked about off and on for a few years now, and we really as a group decided a couple of things. One is, four days was becoming a fairly large time commitment for a lot of people to participate in the full range of camp activities. And then another reason is we generally saw a bit of a drop off in attendance when we went from the weekday to the weekend. And so, as sort of a trial thing we're doing this year is reducing it to three days, with keeping our focus on the sessions like we've always done on Friday. But then on Saturday, making it a little more open free form, which is the unconference, which we can get into, just to see what that does for our numbers and helps more people participate than on the weekends.

IVAN: So, is the unconference style going to be very similar to the way we did BOFs (Birds of a Feather meetings) in the past? Or, how's that going to be structured?

DAN: You know, that's how I picture it. Although it is still a matter of discussion between various camp organizers, how exactly we're going to do it. But the way I'm envisioning it—and Chris can correct me if I'm wrong—is we're going to largely be in the atrium area on Saturday as opposed to going to the classrooms, and people will sort of self-organize into different groups around that large space just to have informal discussions about whatever topics they would like. And then ideally we'll have a few moderators available, floating around the room to sort of help facilitate conversations and make sure people are in the spaces that they find most helpful.

CHRIS: That's right. In the unconference format, we're looking for interesting things to talk about. Tim likes to bring up that Tim [Ericson] and Wilbur [Ince] were the genesis of this idea, when we were talking about adding it to the camp. He likes to talk about the law of two feet, where if you're in a conversation that isn't providing you with what you need, you could use your two feet in order to find another conversation that is more engaging. Then in that way, kind of like plan your own day out of what talks are engaging you and finding the information you need. But the format is very much like a BOF. Instead of slides and rooms, and a more instructor-led conversation, where one person is just talking for an hour or whatever, it's more of a conversation, sharing of input and allowing more people to provide information than just one person at a time.

IVAN: I love the idea of doing that. It really allows, I think, the community to drive what the topics are and the discussions that are being had. I think that's a good experiment. I'm looking forward to participating and seeing how that affects our camp next year. I was going to ask about Thursday. Usually we have trainings on Thursdays, right? Can you speak to what the trainings are this year?

CHRIS: It looks like Drupal 8 content migrations. This can be a getting started with building sites with Drupal. There is a Drupal 8 crash course for content editors, marketers and project managers. Then intermediate to advanced CSS for practical peoples. I think those are all of our trainings.

DAN: Those are the four trainings, and then on top of the trainings we're also hosting a mini-camp this year.

IVAN: Oh really? Well, tell me about that. I know Backdrop has always got such a great presence at TC Drupal Camp every year. What is that about?

DAN: So, in the past we’ve hosted sessions on Backdrop. Every year we seem to draw some of the leaders behind the Backdrop community, and we'll do that again this year. Particularly Jen Lampton, who's helped lead and create the Backdrop community, she's coming, as well as several other prominent Backdrop contributors. And, what they've decided to do this year in the form of a mini-camp, is hold the day of sort of sessions, all in one room dedicated to Backdrop. And we as a camp decided to provide a room to sit alongside the training sessions for people that are interested in contributing to Backdrop, or learning about it to attend to this free session.

IVAN: I think that's a great idea. And why haven't I reached out to Jen and Nate about Backdrop? We should totally have them on the podcast.

CHRIS: There you go.

IVAN: That's awesome. We will make sure we do that. If you happen to be listening out there, Jen and Nate, please send us an email before we do. But yeah, we'd love to have you on the podcast. So that's great.

So, trainings on Thursday, Backdrop CMS minicamp as well in one of the rooms, and then sessions on Friday. So, I would imagine there is a keynote on Friday? Let's talk about what the day looks like on Friday.

DAN: I can tell you definitively we've got five rooms. So, five tracks if you will, and each track will have six sessions throughout the day, for a total of 30 sessions. We're starting the morning like we typically do with a welcome session at 9 am, going into our first session around 9:45 and continuing with the last session ending around 5:00. We will have a keynote this year during the lunch hour on Friday, and I'm happy to tell you about that.

That is a local group called the Asian Penguins, a Linux user group made of boys and girls grades 6 through 8, and they're based out of Hmong charter school in St. Paul. Their director Stu Keroff, is coming to tell the story about what they do, how their work is helping bridge the digital divide in the metro area. He works with the students to teach them Linux, and they repurpose old computers installing Linux on them and giving them away to families in need. So, we’re excited. He's going to bring some of his students with him too, and they're going to do a presentation for us on Friday.

IVAN: Oh, that's wonderful. That's really wonderful to hear. So, the Asian Penguins. Wow, how did you guys find out about them? Meet them? Involve them? What's the story behind that?

CHRIS: Matthew Tiff could probably give you the full answer of how that connection was formed. We found out about them through him, and then we were able to find out more about their organization. And it just sounds like a really great opportunity. I know you share an interest in making sure that tech is accessible to kids as well, Ivan, and it's really great to hear what they're doing.

IVAN: I'm not surprised that Matthew is involved in that.

DAN: Yeah. Matthew had hosted Stu on his Hacking Culture podcast a few months ago, and then recommended them as a potential keynote speaker. So, we reached out to them and we're just finalizing the details on that now, and it should be up on the site by the time this podcast comes out.

IVAN: Oh, that's great. I'm looking forward to hearing what they have to say. So, keynote over the lunch hour. Sessions in the morning, sessions in the afternoon. Can you tell me a little bit about a session format? Are they the same as last year? How long are they? What do those look like?

CHRIS: Yeah. We actually had quite a bit of a debate on how long our sessions should be. You know Drupalcon has moved to a format of half-hour talks and longer talks more than an hour. Right? It's like an hour and a half. And we were concerned about what is the appropriate amount of duration, so we wanted to make sure that we've got a lot of talks that people can give on Friday. But at the same time we were concerned that a half hour might be too short. We're trying 45-minute talks out this year. We're gonna see how that goes. And as a result, we were able to fit about 30 talks into that Friday.

IVAN: Is that 45 minutes of speaker time, or is that 45 minutes of session plus questions?

DAN: It gives time for questions at the end of sessions, unlike 30-minute sessions. You know that was a common experience at DrupalCon this year, is, there really weren't any time for questions at the end of those 30-minute sessions and speakers are really hard pressed to fit all their content in 30 minutes. So even though DrupalCon experimented with this and I think that's fine, we as a group felt like 45 minutes was a much better time slot, and I wouldn't be surprised if they go back to that at feature DrupalCons.

CHRIS: I wouldn't be surprised with that either.

IVAN: So, the 45 minutes is inclusive of the questions then?

DAN: Right. I mean, the assumption is that the speakers will have time then to answer questions.

IVAN: Got it. Ok. And so, let's talk about the five rooms and the five tracks you have. What are the tracks this year?

CHRIS: Well the tracks are similar. If you look at the website right now, they're almost identical to the tracks that we had last year. But we're making an effort this year to be inclusive of talks that are tangential from Drupal. Not every talk has to be about Drupal. We've got talks about GraphQL, JSON integrations and Ruby on Rails. We wanted to make sure that we've got some talks about mental health. We've got talks on a wide area of topics and not necessarily specifically about Drupal.

IVAN: That's great. I'm looking forward to seeing the list of sessions come out. I'm hoping, fingers crossed, that my session made it. When is that session list going to be published?

DAN: That's a good question. [laughing] The announcements will have gone out to the accepted speakers well ahead of this podcast being released. I don't know that we'll have them accepted and published on the site at that point, but we'll be publishing them hopefully by, what would you say Chris, mid-May at the latest?

CHRIS: Yeah, hopefully earlier, but that is largely based upon how we can contact folks. As our good friend Joe [Shindelar] was telling us, we've never had somebody tell us they either can't make it, or say that they can and don't show up. We've had a really high success rate, and we’d like to keep that going, but there's always the possibility that the worst could happen. If we don't get a hold of somebody, or we have to strategically plan, it's better to have everything figured out before we publish. And so, we're putting the effort in now in order to make sure that can happen.

IVAN: So, if you're listening to this podcast there's a chance the sessions have been published but there's a chance the sessions have not yet been published [laughing]. And if they haven't been published, we promise they will be published in the next week after you listen to this. So, fingers crossed.

DAN: Absolutely.

CHRIS: Let’s hope it works out.

IVAN: Let's hope it does. Ok. So, one of the things I love—I love a lot of things about the Drupal Camp in Minneapolis—is the parties. There's always the speaker party and the sponsor appreciation party, and then there's the Friday night party and the Saturday night party. But if the camp is one day short of four days, does that mean it's one day short of a party as well?

DAN: Absolutely not. [laughing]

IVAN: Oh good. Let's hear about what’s going on there.

DAN: Right. Well we do have a few changes this year. I think one of the big ones is that our Thursday night party, which is the day that camp opens after the training, we're trying something new, sort of inspired by our friends at Midcamp in Chicago, and that is changing the Thursday night party to the welcoming party. And what this means is that we're extending an invitation to anyone that is involved or interested in participating in our conference to come to the welcome party on Thursday.

CHRIS: That's right.

IVAN: And where is that party this year?

DAN: The unofficial plan right now is that we're going to host that at Pizza Lucé in downtown Minneapolis.

IVAN: And what about Friday? The Friday night party?

DAN: Yeah. So, Friday we're going back to the House of Balls.

IVAN: Oh yeah. I love that place.

CHRIS: Yeah, it's a really great place.

DAN: House of Balls, we’ve been at, I think this is our fourth year at this, sort of amazing, eccentric art studio/event space, just off of downtown Minneapolis. And we're going to have some of the same great things we have every year. We're lining up a food truck. We're going to have free food and drinks and most importantly, we’ve lined up karaoke.

IVAN: Oh yeah. That sounds amazing. I'm secretly hoping that Marc [Drummond] is able to give his five-minute talk about hotdish again.

DAN: Yeah, Chris, are we going to try to do any of the lightning talks this year?

CHRIS: Well I don't know. We like to be flexible. We're kind of a spontaneous crowd. We've got a number of events planned for the day. You know, we're gonna have some board gaming, and it seems like the board gaming thing has gotten even stronger here in the Twin Cities community. We're going to have some food and, of course, there will be Foursquare.

DAN: Foursquare.

IVAN: I hope Les [Lim] brings his ball.

Foursquare

CHRIS: We lean on Les for both the rules and the gamesmanship and the setup of that. We should side note, we should double check with him, if he's going to do that again this year, or if he wants one of us to take that on. And, yeah, in years past we've had lightning talks and we've also had karaoke. I do know for a fact that we will be having karaoke again this year. I don't know if we'll have lightning talks, but there's room still, I think Dan, we just need to put a plan into action to see if we can provide equipment and time for that.

IVAN: I'm a proponent of the lightning talks, so if you need votes you have one from me, and if you do need something to help make that happen, please ask, I'll do what I can.

DAN: Great. I think we've got a volunteer to run the lightning talks. [laughing] 

CHRIS: Sounds great. That's how this works.

IVAN: [laughing] Ok, well, if I get to run it that means I get to give one too.

CHRIS: [laughing] Indeed. You can kick us off.

IVAN: Alright. Let’s do that. Let me know the details, and I'll help make it happen.

DAN: Alright. Sounds good.

IVAN: Alright. So that's Thursday and Friday taken care of, and what are we doing Saturday? Are we doing anything Saturday?

DAN: We will. We'll do our traditional post-camp party. It is at a location to be determined. So, you have to stay tuned to the website or the newsletter to find out when and where that's going to happen.

IVAN: Well I'm glad that's still happening even though we don't have the fourth day. Stay tuned on the website. That's tcdrupal.org and subscribe to the email list, I'm sure that'll be mentioned in the email as well. One more question. How do you register for camp and what does it cost?

CHRIS: Well you can go to our website at tcdrupal.org/register and you can register right there. We've got a nice big link for you right there in the top of the page, just click on that, go on over to registration. Registration remains inexpensive, especially compared to other Twin Cities camps which we've been able to look at the cost of camps nearby. Our camp's only $50. We are providing means for people who want to contribute more. Like myself, I tried to come in at the Community Contributor level. How much is that again Dan?

DAN: Yeah. So that's $150, and that includes camp registration, a free T-shirt, and it also means that you are helping support the camp above and beyond, which is really key to us being able to offer all these things, including the parties and the free training and all the sessions and the venue.

So, that's kind of a new, it's not new, but what's new this year is, is we're really trying to emphasize to anyone that uses Drupal professionally and that can afford it, please consider coming in at the Community Sponsor level, Community Supporter level. It really helps us out. But anyone is welcome to come to camp, and as always if anyone wants to come and can't afford it, please contact us, and we would be happy to set you up for free.

IVAN: What's the best way to contact you?

DAN: Yeah, so, go to the website tcdrupal.org, go to the contact page and just shoot us a message, and one of us would be happy to get back to you. Or you can hit us up on Twitter as well.

CHRIS: Yeah. Like Dan said, if you fill out the contact form on site, you're sending an e-mail message to the entire team. Someone's going to see that immediately. And, again, we're available over Twitter just like the rest of the Drupal community. We all kind of hang out there.

IVAN: And, there are sponsors again this year, like there were last year. There always seems to be a plethora of sponsors for camp, which is just so awesome to see for our little community. Are there still opportunities to sponsor? What options are left, if they are?

DAN: Please, please, please, always welcome more sponsors. The more sponsors we get, the more we can do. You know, we really are wanting and planning to offer free lunch to everyone at camp this year on Friday, and getting a few more sponsors really help make that happen. And so, we have some great sponsors so far including TEN7, thank you for that.

IVAN: Yeah!

DAN: And, you know, we have a few platinum sponsor slots still available. We have unlimited slots at the gold and silver level. And so anyone who wants to consider both helping the camp out and maybe getting a table to tell people about your organization or what you do, you're very welcome to do that. And again, just come to the website. There's information about how to become a sponsor,  or to just get in touch with someone.

IVAN: So, that URL is tcdrupal.org/sponsors, and there's a great little button there that you can visit the sponsor page for more information about the benefits of each of the sponsor levels. Yeah, it's been great to see the same companies coming back to the camp and coming back and providing to the community. It’s always a pleasure for us to do it, and I'm sure it is for you too Dan for Electric Citizen and for the others that are also doing that.

I’ve asked this before of members of our community and of members of the organizing team that always puts on this volunteer event. It's volunteers that do it. I'm amazed that it happens every year. But DrupalCon 2020 is in Minneapolis next year, and DrupalCon just happened last month, and we have our camp in close proximity to it. So, has there been any discussion about what, if any effect DrupalCon in Minneapolis is going to have on our camp next year?

CHRIS: So we've had a lot of internal discussions about it, and while we have a lot of energy in the Twin Cities, it seems like the prevailing wisdom is that we want to try to find a couple of smaller events. The work that we anticipate we're going to put in around DrupalCon is really too close to where we would want to have our camp here in the Twin Cities, to make both the contribution we want to put in to make DrupalCon a success and the contribution we want to put in in order to make our camp a success. That said, it's still kind of up in the air.

We haven't had the powwow that we really need in order to come to a firm decision that, “Hey, we're not going to do a Drupal Camp,” or “Hey, we're not going to do a Drupal Camp like later in the fall sometime that day of the year.” So, I guess the answer that we have right now is that, we want to continue to be active. We want to do things in the Twin Cities surrounding Drupal and getting together an event. And I think we've got different ideas on how to accomplish that, but the main thing we want to do is to continue to talk about Drupal, celebrate Drupal and promote knowledge and learning and inclusiveness.

IVAN: So, "Stay tuned. We're evolving the decision as time progresses," is what you’re saying?

CHRIS: Yeah. So, we don't have a good answer yet. We're all so laser focused on getting this year's camp put together and have it be so awesome, that we've postponed any other kind of discussion of what's next, until we're done.

IVAN: And thank you for being so laser focused on the camp, and Dan and Chris and everyone else that's helped organize the camp, Jer [Davis] and Tim [Erickson] and all of the other volunteers. It's just always so amazing for me to see the camp happen and for all those people to contribute and for there to be so much empathy and care that it happens in the most equitable and fun and cheap and value-based event that we can put on, and I think that's great. So, thank you both for doing that and for contributing.

CHRIS: After this is all done, there’s so much gratitude to make sure people get, based upon their efforts that they've been able to put in to make this thing a success. And the thing that we keep talking about, it's really our deliverable at the end of this is our process, because our process has been pretty good. We keep on iterating on it, so that we can have the confidence that, “Hey, we can put together a camp like this,” and we could feel really good about that process.

DAN: And not only that, but I've been involved in many years of camp organizing for TCDrupal, and I feel like every year is good, but the gang's really getting along well this year to where I'm not even daunted by the thought of doing it again next year.

CHRIS: I would love to do it again.

IVAN: You heard it here first. [laughing] We’re already thinking about the following year's Drupal camp. That's great.

CHRIS: So that's the high we're on right now from all the good work we’re doing. We’ll see how we’re feeling after this.

IVAN: [laughing] No, you can’t go back now. You just said that you're not even worried about it. So, let's actually just spend a minute before we close here, and say this is version 9 of the camp, if I'm not mistaken. I think the first one was in 2011, so, this would be version 9, and so the next one is the 10th anniversary. Right? So, we should celebrate that somehow.

DAN: Well, we are. It's called Drupalcon 2020, [laughing] and what better way to cap off 10 years of active community growing, stewardship, caretaking, whatever you want to call it. I myself came to this Drupal community group as a lone wolf developer looking to find some other group of people that I can nerd out about Drupal with. And my story is basically the story of how successful this community has been. Thanks to all of the people who have welcomed me in and made me feel like I belonged. I'm here today helping plan the next one.

IVAN: I love it. I think it's precious and amazing, and I'm always amazed by all of that. So, yeah. I hope I'm right about it being the 10th anniversary, because I feel like there were different incarnations of the camp before 2011, but I think 2011 was the first official one, right?

DAN: It was. Yep, you're absolutely right.

IVAN: Ok, good. Well, thank you both for spending your precious time with me today. It's really been a pleasure talking with you Chris and Dan.

CHRIS: Same here, man.

DAN: Yeah, thanks so much for hosting us.

IVAN: Chris and Dan are two of the volunteer organizers of Twin Cities Drupal Camp happening from June 6th, a Thursday to June 8th, a Saturday, at the University of St. Thomas in downtown Minneapolis. Tickets are still available and they're reasonably priced starting at $50, and we're hoping that includes lunch as well.

So, head on over to tcdrupal.org and register now. You can find the camp on Twitter, Facebook and Instagram. The handle is @tcdrupal. And, of course, the Twin Cities Drupal group is also on groups.drupal.org/twin-cities for other local events that happen outside of camp, and they happen every month, whether it's the happy hour or something else, it is on.

You’ve been listening to the TEN7 Podcast. Find us online at ten7.com/podcast. And if you have a second, do send us a message, we love hearing from you. Our email address is [email protected]. Until next time, this is Ivan Stegic. Thank you for listening.

May 08 2019
May 08

Your browser does not support the audio element. TEN7-Podcast-Ep-059-2019-Twin-Cities-Drupal-Camp_0.mp3

Summary

Chris Weber and Dan Moriarty, volunteer organizers for the 2019 Twin Cities Drupal Camp are today's podcast guests. We'll be talking about the changes to this year's TCDrupal Camp and fond memories of previous camps. TCDrupal Camp is a three-day conference for open source enthusiasts, designers, hackers, geeks, developers, UI experts, IT managers and anyone else that wants to find out more about Drupal. It’s a great place to learn, code, network and have fun with your fellow Drupalistas.

Guests

Chris Weber, software engineer at The Nerdery, and Dan Moriarty, CEO and Creative Director at Electric Citizen

Highlights

  • TCDrupal Camp's location at St. Thomas
  • Format changes: three days instead of four, 45-minute talks
  • Fewer days, but just as many parties! And food trucks, board gaming and karaoke
  • Focus on expanding talks to topics outside of just Drupal 
  • The House of Balls, a Minneapolis institution
  • How TCDrupal Camp's spontaneity is what makes it great
  • TCDrupal Camp's history
  • So much Drupal goodness coming to Minneapolis (DrupalCon 2020) how will we manage it all?

Links

Transcript

IVAN STEGIC: Hey everyone you're listening to the TEN7 Podcast, where we get together every fortnight and sometimes more often, to talk about technology, business and the humans in it. I'm your host Ivan Stegic. My guests today are Chris Weber and Dan Moriarty, two of the volunteer organizers of this year's Twin Cities Drupal Camp.

TC Drupal Camp

IVAN: Chris is a software engineer at The Nerdery, and Dan is CEO and Creative Director at Electric Citizen. Hello Chris and Dan. Welcome to the podcast.

CHRIS WEBER: Hello, hello.

DAN MORIARTY: Hey there. Thanks for having us.

IVAN: Oh, it’s my pleasure. Dan Moriarty, I love saying your name. The whole Sherlock Holmes thing, I just love it.

DAN: Yeah, and I will take that anytime. I'm always happy to reference my evil ancestors. [laughing]

IVAN: [laughing] Oh wait! Relation? Are you related to a fictitious person?

DAN: I’ll claim that.

IVAN: [laughing] That's awesome. Well I'm glad that you are on the show with us today talking about Twin Cities Drupal Camp this year. So, Chris, tell us about the camp itself. When and where is it this year?

CHRIS: Well this 2019 version of our camp, is going to be at St. Thomas which is in downtown Minneapolis. We've had it at St. Thomas for a number of years, so it should be familiar to folks that have gone to the Twin Cities Drupal Camp before. It's a really good location, really large open space, very, very lighty and breathy. We’ll be having it on June 6th through June 8th. June 6th is a training day. The 7th will be filled with excellent talks, sessions. And then the 8th will be kind of something a little bit new that we're doing. We're having an unconference on that day, as well as providing a space for people who want to sprint on core contributions. And we’re very excited to have the camp again here in the Twin Cities.

IVAN: And so that's a little different than how we've done it, I would guess, every year since we started, although I don't remember the first year. But that's a Thursday, Friday, Saturday, as opposed to Thursday, Friday, Saturday, Sunday. So, we have one day of sessions as opposed to two. Dan, do you want to talk through kind of the reasoning behind that, and why we decided to do it that way this year?

DAN: Yeah. So, it's something we've talked about off and on for a few years now, and we really as a group decided a couple of things. One is, four days was becoming a fairly large time commitment for a lot of people to participate in the full range of camp activities. And then another reason is we generally saw a bit of a drop off in attendance when we went from the weekday to the weekend. And so, as sort of a trial thing we're doing this year is reducing it to three days, with keeping our focus on the sessions like we've always done on Friday. But then on Saturday, making it a little more open free form, which is the unconference, which we can get into, just to see what that does for our numbers and helps more people participate than on the weekends.

IVAN: So, is the unconference style going to be very similar to the way we did BOFs (Birds of a Feather meetings) in the past? Or, how's that going to be structured?

DAN: You know, that's how I picture it. Although it is still a matter of discussion between various camp organizers, how exactly we're going to do it. But the way I'm envisioning it—and Chris can correct me if I'm wrong—is we're going to largely be in the atrium area on Saturday as opposed to going to the classrooms, and people will sort of self-organize into different groups around that large space just to have informal discussions about whatever topics they would like. And then ideally we'll have a few moderators available, floating around the room to sort of help facilitate conversations and make sure people are in the spaces that they find most helpful.

CHRIS: That's right. In the unconference format, we're looking for interesting things to talk about. Tim likes to bring up that Tim [Ericson] and Wilbur [Ince] were the genesis of this idea, when we were talking about adding it to the camp. He likes to talk about the law of two feet, where if you're in a conversation that isn't providing you with what you need, you could use your two feet in order to find another conversation that is more engaging. Then in that way, kind of like plan your own day out of what talks are engaging you and finding the information you need. But the format is very much like a BOF. Instead of slides and rooms, and a more instructor-led conversation, where one person is just talking for an hour or whatever, it's more of a conversation, sharing of input and allowing more people to provide information than just one person at a time.

IVAN: I love the idea of doing that. It really allows, I think, the community to drive what the topics are and the discussions that are being had. I think that's a good experiment. I'm looking forward to participating and seeing how that affects our camp next year. I was going to ask about Thursday. Usually we have trainings on Thursdays, right? Can you speak to what the trainings are this year?

CHRIS: It looks like Drupal 8 content migrations. This can be a getting started with building sites with Drupal. There is a Drupal 8 crash course for content editors, marketers and project managers. Then intermediate to advanced CSS for practical peoples. I think those are all of our trainings.

DAN: Those are the four trainings, and then on top of the trainings we're also hosting a mini-camp this year.

IVAN: Oh really? Well, tell me about that. I know Backdrop has always got such a great presence at TC Drupal Camp every year. What is that about?

DAN: So, in the past we’ve hosted sessions on Backdrop. Every year we seem to draw some of the leaders behind the Backdrop community, and we'll do that again this year. Particularly Jen Lampton, who's helped lead and create the Backdrop community, she's coming, as well as several other prominent Backdrop contributors. And, what they've decided to do this year in the form of a mini-camp, is hold the day of sort of sessions, all in one room dedicated to Backdrop. And we as a camp decided to provide a room to sit alongside the training sessions for people that are interested in contributing to Backdrop, or learning about it to attend to this free session.

IVAN: I think that's a great idea. And why haven't I reached out to Jen and Nate about Backdrop? We should totally have them on the podcast.

CHRIS: There you go.

IVAN: That's awesome. We will make sure we do that. If you happen to be listening out there, Jen and Nate, please send us an email before we do. But yeah, we'd love to have you on the podcast. So that's great.

So, trainings on Thursday, Backdrop CMS minicamp as well in one of the rooms, and then sessions on Friday. So, I would imagine there is a keynote on Friday? Let's talk about what the day looks like on Friday.

DAN: I can tell you definitively we've got five rooms. So, five tracks if you will, and each track will have six sessions throughout the day, for a total of 30 sessions. We're starting the morning like we typically do with a welcome session at 9 am, going into our first session around 9:45 and continuing with the last session ending around 5:00. We will have a keynote this year during the lunch hour on Friday, and I'm happy to tell you about that.

That is a local group called the Asian Penguins, a Linux user group made of boys and girls grades 6 through 8, and they're based out of Hmong charter school in St. Paul. Their director Stu Keroff, is coming to tell the story about what they do, how their work is helping bridge the digital divide in the metro area. He works with the students to teach them Linux, and they repurpose old computers installing Linux on them and giving them away to families in need. So, we’re excited. He's going to bring some of his students with him too, and they're going to do a presentation for us on Friday.

IVAN: Oh, that's wonderful. That's really wonderful to hear. So, the Asian Penguins. Wow, how did you guys find out about them? Meet them? Involve them? What's the story behind that?

CHRIS: Matthew Tiff could probably give you the full answer of how that connection was formed. We found out about them through him, and then we were able to find out more about their organization. And it just sounds like a really great opportunity. I know you share an interest in making sure that tech is accessible to kids as well, Ivan, and it's really great to hear what they're doing.

IVAN: I'm not surprised that Matthew is involved in that.

DAN: Yeah. Matthew had hosted Stu on his Hacking Culture podcast a few months ago, and then recommended them as a potential keynote speaker. So, we reached out to them and we're just finalizing the details on that now, and it should be up on the site by the time this podcast comes out.

IVAN: Oh, that's great. I'm looking forward to hearing what they have to say. So, keynote over the lunch hour. Sessions in the morning, sessions in the afternoon. Can you tell me a little bit about a session format? Are they the same as last year? How long are they? What do those look like?

CHRIS: Yeah. We actually had quite a bit of a debate on how long our sessions should be. You know Drupalcon has moved to a format of half-hour talks and longer talks more than an hour. Right? It's like an hour and a half. And we were concerned about what is the appropriate amount of duration, so we wanted to make sure that we've got a lot of talks that people can give on Friday. But at the same time we were concerned that a half hour might be too short. We're trying 45-minute talks out this year. We're gonna see how that goes. And as a result, we were able to fit about 30 talks into that Friday.

IVAN: Is that 45 minutes of speaker time, or is that 45 minutes of session plus questions?

DAN: It gives time for questions at the end of sessions, unlike 30-minute sessions. You know that was a common experience at DrupalCon this year, is, there really weren't any time for questions at the end of those 30-minute sessions and speakers are really hard pressed to fit all their content in 30 minutes. So even though DrupalCon experimented with this and I think that's fine, we as a group felt like 45 minutes was a much better time slot, and I wouldn't be surprised if they go back to that at feature DrupalCons.

CHRIS: I wouldn't be surprised with that either.

IVAN: So, the 45 minutes is inclusive of the questions then?

DAN: Right. I mean, the assumption is that the speakers will have time then to answer questions.

IVAN: Got it. Ok. And so, let's talk about the five rooms and the five tracks you have. What are the tracks this year?

CHRIS: Well the tracks are similar. If you look at the website right now, they're almost identical to the tracks that we had last year. But we're making an effort this year to be inclusive of talks that are tangential from Drupal. Not every talk has to be about Drupal. We've got talks about GraphQL, JSON integrations and Ruby on Rails. We wanted to make sure that we've got some talks about mental health. We've got talks on a wide area of topics and not necessarily specifically about Drupal.

IVAN: That's great. I'm looking forward to seeing the list of sessions come out. I'm hoping, fingers crossed, that my session made it. When is that session list going to be published?

DAN: That's a good question. [laughing] The announcements will have gone out to the accepted speakers well ahead of this podcast being released. I don't know that we'll have them accepted and published on the site at that point, but we'll be publishing them hopefully by, what would you say Chris, mid-May at the latest?

CHRIS: Yeah, hopefully earlier, but that is largely based upon how we can contact folks. As our good friend Joe [Shindelar] was telling us, we've never had somebody tell us they either can't make it, or say that they can and don't show up. We've had a really high success rate, and we’d like to keep that going, but there's always the possibility that the worst could happen. If we don't get a hold of somebody, or we have to strategically plan, it's better to have everything figured out before we publish. And so, we're putting the effort in now in order to make sure that can happen.

IVAN: So, if you're listening to this podcast there's a chance the sessions have been published but there's a chance the sessions have not yet been published [laughing]. And if they haven't been published, we promise they will be published in the next week after you listen to this. So, fingers crossed.

DAN: Absolutely.

CHRIS: Let’s hope it works out.

IVAN: Let's hope it does. Ok. So, one of the things I love—I love a lot of things about the Drupal Camp in Minneapolis—is the parties. There's always the speaker party and the sponsor appreciation party, and then there's the Friday night party and the Saturday night party. But if the camp is one day short of four days, does that mean it's one day short of a party as well?

DAN: Absolutely not. [laughing]

IVAN: Oh good. Let's hear about what’s going on there.

DAN: Right. Well we do have a few changes this year. I think one of the big ones is that our Thursday night party, which is the day that camp opens after the training, we're trying something new, sort of inspired by our friends at Midcamp in Chicago, and that is changing the Thursday night party to the welcoming party. And what this means is that we're extending an invitation to anyone that is involved or interested in participating in our conference to come to the welcome party on Thursday.

CHRIS: That's right.

IVAN: And where is that party this year?

DAN: The unofficial plan right now is that we're going to host that at Pizza Lucé in downtown Minneapolis.

IVAN: And what about Friday? The Friday night party?

DAN: Yeah. So, Friday we're going back to the House of Balls.

IVAN: Oh yeah. I love that place.

CHRIS: Yeah, it's a really great place.

DAN: House of Balls, we’ve been at, I think this is our fourth year at this, sort of amazing, eccentric art studio/event space, just off of downtown Minneapolis. And we're going to have some of the same great things we have every year. We're lining up a food truck. We're going to have free food and drinks and most importantly, we’ve lined up karaoke.

IVAN: Oh yeah. That sounds amazing. I'm secretly hoping that Marc [Drummond] is able to give his five-minute talk about hotdish again.

DAN: Yeah, Chris, are we going to try to do any of the lightning talks this year?

CHRIS: Well I don't know. We like to be flexible. We're kind of a spontaneous crowd. We've got a number of events planned for the day. You know, we're gonna have some board gaming, and it seems like the board gaming thing has gotten even stronger here in the Twin Cities community. We're going to have some food and, of course, there will be Foursquare.

DAN: Foursquare.

IVAN: I hope Les [Lim] brings his ball.

Foursquare

CHRIS: We lean on Les for both the rules and the gamesmanship and the setup of that. We should side note, we should double check with him, if he's going to do that again this year, or if he wants one of us to take that on. And, yeah, in years past we've had lightning talks and we've also had karaoke. I do know for a fact that we will be having karaoke again this year. I don't know if we'll have lightning talks, but there's room still, I think Dan, we just need to put a plan into action to see if we can provide equipment and time for that.

IVAN: I'm a proponent of the lightning talks, so if you need votes you have one from me, and if you do need something to help make that happen, please ask, I'll do what I can.

DAN: Great. I think we've got a volunteer to run the lightning talks. [laughing] 

CHRIS: Sounds great. That's how this works.

IVAN: [laughing] Ok, well, if I get to run it that means I get to give one too.

CHRIS: [laughing] Indeed. You can kick us off.

IVAN: Alright. Let’s do that. Let me know the details, and I'll help make it happen.

DAN: Alright. Sounds good.

IVAN: Alright. So that's Thursday and Friday taken care of, and what are we doing Saturday? Are we doing anything Saturday?

DAN: We will. We'll do our traditional post-camp party. It is at a location to be determined. So, you have to stay tuned to the website or the newsletter to find out when and where that's going to happen.

IVAN: Well I'm glad that's still happening even though we don't have the fourth day. Stay tuned on the website. That's tcdrupal.org and subscribe to the email list, I'm sure that'll be mentioned in the email as well. One more question. How do you register for camp and what does it cost?

CHRIS: Well you can go to our website at tcdrupal.org/register and you can register right there. We've got a nice big link for you right there in the top of the page, just click on that, go on over to registration. Registration remains inexpensive, especially compared to other Twin Cities camps which we've been able to look at the cost of camps nearby. Our camp's only $50. We are providing means for people who want to contribute more. Like myself, I tried to come in at the Community Contributor level. How much is that again Dan?

DAN: Yeah. So that's $150, and that includes camp registration, a free T-shirt, and it also means that you are helping support the camp above and beyond, which is really key to us being able to offer all these things, including the parties and the free training and all the sessions and the venue.

So, that's kind of a new, it's not new, but what's new this year is, is we're really trying to emphasize to anyone that uses Drupal professionally and that can afford it, please consider coming in at the Community Sponsor level, Community Supporter level. It really helps us out. But anyone is welcome to come to camp, and as always if anyone wants to come and can't afford it, please contact us, and we would be happy to set you up for free.

IVAN: What's the best way to contact you?

DAN: Yeah, so, go to the website tcdrupal.org, go to the contact page and just shoot us a message, and one of us would be happy to get back to you. Or you can hit us up on Twitter as well.

CHRIS: Yeah. Like Dan said, if you fill out the contact form on site, you're sending an e-mail message to the entire team. Someone's going to see that immediately. And, again, we're available over Twitter just like the rest of the Drupal community. We all kind of hang out there.

IVAN: And, there are sponsors again this year, like there were last year. There always seems to be a plethora of sponsors for camp, which is just so awesome to see for our little community. Are there still opportunities to sponsor? What options are left, if they are?

DAN: Please, please, please, always welcome more sponsors. The more sponsors we get, the more we can do. You know, we really are wanting and planning to offer free lunch to everyone at camp this year on Friday, and getting a few more sponsors really help make that happen. And so, we have some great sponsors so far including TEN7, thank you for that.

IVAN: Yeah!

DAN: And, you know, we have a few platinum sponsor slots still available. We have unlimited slots at the gold and silver level. And so anyone who wants to consider both helping the camp out and maybe getting a table to tell people about your organization or what you do, you're very welcome to do that. And again, just come to the website. There's information about how to become a sponsor,  or to just get in touch with someone.

IVAN: So, that URL is tcdrupal.org/sponsors, and there's a great little button there that you can visit the sponsor page for more information about the benefits of each of the sponsor levels. Yeah, it's been great to see the same companies coming back to the camp and coming back and providing to the community. It’s always a pleasure for us to do it, and I'm sure it is for you too Dan for Electric Citizen and for the others that are also doing that.

I’ve asked this before of members of our community and of members of the organizing team that always puts on this volunteer event. It's volunteers that do it. I'm amazed that it happens every year. But DrupalCon 2020 is in Minneapolis next year, and DrupalCon just happened last month, and we have our camp in close proximity to it. So, has there been any discussion about what, if any effect DrupalCon in Minneapolis is going to have on our camp next year?

CHRIS: So we've had a lot of internal discussions about it, and while we have a lot of energy in the Twin Cities, it seems like the prevailing wisdom is that we want to try to find a couple of smaller events. The work that we anticipate we're going to put in around DrupalCon is really too close to where we would want to have our camp here in the Twin Cities, to make both the contribution we want to put in to make DrupalCon a success and the contribution we want to put in in order to make our camp a success. That said, it's still kind of up in the air.

We haven't had the powwow that we really need in order to come to a firm decision that, “Hey, we're not going to do a Drupal Camp,” or “Hey, we're not going to do a Drupal Camp like later in the fall sometime that day of the year.” So, I guess the answer that we have right now is that, we want to continue to be active. We want to do things in the Twin Cities surrounding Drupal and getting together an event. And I think we've got different ideas on how to accomplish that, but the main thing we want to do is to continue to talk about Drupal, celebrate Drupal and promote knowledge and learning and inclusiveness.

IVAN: So, "Stay tuned. We're evolving the decision as time progresses," is what you’re saying?

CHRIS: Yeah. So, we don't have a good answer yet. We're all so laser focused on getting this year's camp put together and have it be so awesome, that we've postponed any other kind of discussion of what's next, until we're done.

IVAN: And thank you for being so laser focused on the camp, and Dan and Chris and everyone else that's helped organize the camp, Jer [Davis] and Tim [Erickson] and all of the other volunteers. It's just always so amazing for me to see the camp happen and for all those people to contribute and for there to be so much empathy and care that it happens in the most equitable and fun and cheap and value-based event that we can put on, and I think that's great. So, thank you both for doing that and for contributing.

CHRIS: After this is all done, there’s so much gratitude to make sure people get, based upon their efforts that they've been able to put in to make this thing a success. And the thing that we keep talking about, it's really our deliverable at the end of this is our process, because our process has been pretty good. We keep on iterating on it, so that we can have the confidence that, “Hey, we can put together a camp like this,” and we could feel really good about that process.

DAN: And not only that, but I've been involved in many years of camp organizing for TCDrupal, and I feel like every year is good, but the gang's really getting along well this year to where I'm not even daunted by the thought of doing it again next year.

CHRIS: I would love to do it again.

IVAN: You heard it here first. [laughing] We’re already thinking about the following year's Drupal camp. That's great.

CHRIS: So that's the high we're on right now from all the good work we’re doing. We’ll see how we’re feeling after this.

IVAN: [laughing] No, you can’t go back now. You just said that you're not even worried about it. So, let's actually just spend a minute before we close here, and say this is version 9 of the camp, if I'm not mistaken. I think the first one was in 2011, so, this would be version 9, and so the next one is the 10th anniversary. Right? So, we should celebrate that somehow.

DAN: Well, we are. It's called Drupalcon 2020, [laughing] and what better way to cap off 10 years of active community growing, stewardship, caretaking, whatever you want to call it. I myself came to this Drupal community group as a lone wolf developer looking to find some other group of people that I can nerd out about Drupal with. And my story is basically the story of how successful this community has been. Thanks to all of the people who have welcomed me in and made me feel like I belonged. I'm here today helping plan the next one.

IVAN: I love it. I think it's precious and amazing, and I'm always amazed by all of that. So, yeah. I hope I'm right about it being the 10th anniversary, because I feel like there were different incarnations of the camp before 2011, but I think 2011 was the first official one, right?

DAN: It was. Yep, you're absolutely right.

IVAN: Ok, good. Well, thank you both for spending your precious time with me today. It's really been a pleasure talking with you Chris and Dan.

CHRIS: Same here, man.

DAN: Yeah, thanks so much for hosting us.

IVAN: Chris and Dan are two of the volunteer organizers of Twin Cities Drupal Camp happening from June 6th, a Thursday to June 8th, a Saturday, at the University of St. Thomas in downtown Minneapolis. Tickets are still available and they're reasonably priced starting at $50, and we're hoping that includes lunch as well.

So, head on over to tcdrupal.org and register now. You can find the camp on Twitter, Facebook and Instagram. The handle is @tcdrupal. And, of course, the Twin Cities Drupal group is also on groups.drupal.org/twin-cities for other local events that happen outside of camp, and they happen every month, whether it's the happy hour or something else, it is on.

You’ve been listening to the TEN7 Podcast. Find us online at ten7.com/podcast. And if you have a second, do send us a message, we love hearing from you. Our email address is [email protected]. Until next time, this is Ivan Stegic. Thank you for listening.

Apr 24 2019
Apr 24

Your browser does not support the audio element. TEN7-Podcast-Ep-058-TEN7s-Onboarding-Process.mp3

Are you looking for someone to support your existing Drupal site? Or, are you an agency and you're considering taking on a new client? We think you should date before getting married right away. Trust us, it's better for both parties! In this podcast, Ivan Stegic and our DevOps Engineer Tess Flynn discuss the TEN7 courtship—er—new client onboarding process, which insures that we get to know your site better than you do!

Running time: 39 minutes
Host: Ivan Stegic
Guest: Tess Flynn

In this podcast we'll discuss: 

  • The difference between discovery & design clients and audit–improve–support clients
  • Why we don’t just say “yes” to a client that gives us money to support their site
  • How the TEN7Audit process is like CarTalk (the multiple layers of the audit and troubleshooting process)
  • The simple check that tells Tess how much you love (or neglect) your site
  • The topics of the TEN7Audit: security, infrastructure, UX and theming, content types
  • We get the data, then try to figure out the underlying human story
  • Why we take the time to present our audit findings to you (in three tiers) vs. dumping the PDF in an email 
  • Tess compares your website to a car. Mark your TEN7 Bingo cards!
  • After TEN7Improve, we’re intimate with your site and know whether we want to support it for the long haul
  • How we take backups for TEN7Care so seriously we created a product (Tractorbeam) to do it for us (and you)

Links:

TRANSCRIPT

IVAN STEGIC: Hey everyone you're listening to the TEN7 podcast, where we get together every fortnight and sometimes more often, to talk about technology, business and the humans in it. I'm your host Ivan Stegic. We recently published a blog post called ‘Becoming a TEN7 Support Client, You Can’t Just Give Us Cash’, and we think it’s a good description of the process of becoming a client in which we support a site that we didn’t build. I thought it might be useful to talk about this whole process as well in a podcast, and that’s what we’re going to be focusing on in this episode. To help me flesh out the details, I have Tess Flynn, our DevOps Engineer joining me once again. Hey Tess.

TESS: Hello.

IVAN: I wanted to start by talking about TEN7’s business, and how clients come to us, and how we have distinct categories of clients. And then we’ll kind of get into the nitty gritty of the process of onboarding a client, and that’s where I thought you could help us out.

Essentially, we have two distinct categories of clients.

One category is a new site build, or a new feature build, and this is a client that specifically wants us to create something from scratch or add on to an existing deployment. And usually that process is discovery and design, some sort of strategy, then development and launch, then we support the site we’ve built. On the other side, the other major category of business we do is supporting an existing site. So, that’s supporting a client or a prospect that comes to us that has a Drupal site that someone’s already built for them. And for whatever reason, they need it supported and maintained and looked after for an extended period of time.

The process there is similar. We audit the site, we improve it and then we support it, and the products we have for those services are called TEN7Audit, TEN7Improve and TEN7Care. Now, there’s a natural progression, I think, from learning everything we can about you and what you have, to recommending things we could improve, to then supporting and maintaining that site for you. But the reason we came up with this process was, a particularly difficult client that we had some years ago, And I did something as an owner, that I shouldn’t have done, and that was, say yes to supporting a client whose site I knew absolutely nothing about, whose site no one on our team evaluated or saw, that turned out not to be a brochure site, but a complex, inner connection of many different modules, some custom, some not.

We ended up with a commitment with a client for two years that went on for two years longer than it should have, negatively affecting our team, the morale on our team, resulted in some internal soul searching within the organization. We discovered the DiSC analysis process, we applied it to everyone in the company, we focused on our own mission, our own values. And when we came out on the other side, I basically didn’t want this to ever happen again, right? From our perspective, we wanted to do our best work for the client, and we were being handcuffed, because we didn’t do our due diligence right in the beginning of the engagement. And, of course from the clients' perspective, they expected nothing less, right? They wanted us to do the best work for them, we possibly could be doing, and so we came up with this three-step process that sets us both up for success.

Essentially, we do an evaluation and an audit of an existing Drupal site. We call that the TEN7Audit. Once we’ve evaluated it, we kind of have to get to know the code base a little better, and nothing’s ever perfect, there’s always some amount of work that needs to be done. And so we do that. The next step we call the TEN7Improve step. And once that’s complete, we offer TEN7Care, our support agreement that keeps the client's Drupal site humming, kind of the way it should be. So, it’s three steps, TEN7Audit, TEN7Improve, TEN7Care, and what I’d like to do now, is talk about the details of TEN7Audit.

So, a prospect comes to us, and they ask us if we want to support their site, and we say, “You seem nice, I think we’d like to work with you. Let’s take a look under your hood.” And, that’s what the TEN7Audit really is. Tess, you’re the principal engineer that’s responsible for the TEN7Audit. So, when a prospect comes to us and I say, “Okay, let's do the audit,” it lands in your lap, right? And, you’re then tasked with doing this audit. And we’ve done so many of them we have a good process around what we do, and I want to find out about what exactly happens during the audit? What’s the first thing you do?

TESS: The first thing I usually do when it comes to doing an audit is that I’m going to need access to the underlying infrastructure. So, I’m going to need access to SSH in order to get to the file system on which the site is hosted. And I will also need database access credentials, so that I can get a copy of the website itself. Now, I like doing some audits entirely in place if it’s possible, without copying it to a local environment, but increasingly this is difficult, because depending on the client and depending on the site that might not be possible. For some shared hosting providers, for example, adding an additional tech in place can be a risky proposition, because it can cause some issues with the site, in order to perform the audit. Also, certain platform-as-a-service hosts actually don’t really let you do that very easily, in which case you do have to copy the site off and then run it locally. And then, in some cases, you might have the ability to run additional tools in place, but you’ll have to run it locally anyways, because the actual environment that the site is running on already has something already so fundamentally wrong with it that you can’t even run those tools in the first place.

So, it all comes down to: how do I get my hands on the site code and the site database? That is always the first problem when it comes to doing an audit. And, the thing with doing that is, some clients actually find that kind of intimidating, because they don’t know who you are, they don’t know if they can trust you, and they’re not sure if they really want to give that information over. And, it’s really necessary in order to perform an effective audit in the first place. Otherwise, the best I can do is look around a little bit. Now, if I do get that access, then I can start tearing down how the site itself is built, and this becomes an interesting process.

The very first thing to do is to run several auditing tools. Depending on the site, that could be Healthcheck, or the Site Audit module, or the Hacked! module. There’s a number of different tools that we use in order to facilitate gathering the best information necessary, to see what the red flags are in the site, if there are any. What’s interesting about this process is, already the attempt to run these tools will tell us something about the site itself. Can we run them in place? If we can’t run them in place, why can’t we run them in place? What’s the underlying problem that’s preventing us from doing that? Is it because of the hosting provider? Is it because of the way that the server infrastructure is set up? Is something else wrong that we need to take note of, that is something that we need to remark on our document?

Once we actually run the tools, we examine the output of those tools, and usually this gives us kind of a 1,000-foot perspective of the general health of the site, but it doesn’t allow us to really uncover the underlying causes of some of this. An auditing tool can tell you, say, you’re running out of disc space. But why are you running out of disc space? You might have, well, there’s excessive activity in the database, or there’s excessive CPU draw, well why? What’s really doing that? So, all these tools just paint a more detailed picture, and at some point, you do have to start breaking those down and going after them and investigating them yourself.

IVAN: So, I was going to ask you about the tools that you use, and you mentioned that you usually SSH into the clients website. I would imagine that if there’s already some continuous integration in place, or some codebase in place, maybe you’re using Git to get those files as well. And then you mentioned Site Audit module and the Healthcheck module and Hacked!. And it’s not just modules that you use to determine the health of a website too, right? You are also evaluating the infrastructure, so, is it a shared host? Is there Varnish installed? Is there Memcached? How is the host configured? What kind of access do you have? Those are also other things that we evaluate. Could you talk about the next step? Once you’ve done, kind of the tools evaluation, what are the other things you evaluate?

TESS: It’s kind of like an episode of Car Talk really. A caller calls in, says, “Oh, hey, my site is making this weird sound,” and then after, you should not turn it like that, so it doesn’t make that sound. Then you actually ask, “Okay, when does it make the sound?” “How long has it made the sound?” And you start following the investigative chain. Using the tools really are just the first step in that process, because often with websites, as with a lot of modern vehicles, they are so complicated we don’t really know what’s wrong with them intuitively. We will only be able to know after analysis, and usually that requires an additional amount of technical expertise. So, the tooling basically gives us a rough topology of what the site health is like. And, afterwards, we need to investigate each one of those vectors. And a lot of the time, it comes down to how our audit document is structured, which allows me to investigate that. So, what I tend to do is, I will first start by running the tools and see if there’s any red flags in there.

If there aren’t, then the next thing I do is see if the client has mentioned anything in particular that we want to look at when it comes to the site. Sometimes that can give us a good clue and sometimes that could be a false positive, and sometimes we don’t have that information. So, it depends on what we have available to track down the necessary clues.

With our audit document, we actually break that out to several different sections. There’s security findings, infrastructure findings, UX and theming findings, and content findings. Each one of these is a section by which we can go and do further investigation on how the site health is working. Usually the next thing after I do the first pass is to check when was the last time the site was updated. This sounds like such a simple, easy thing, but it tells you a lot about, not necessarily the technology, but the people around the site, and how they worked with the site and regarded it.

Everything we do in technology is about people, so you have to understand the underlying human story around the technology, and that will allow you to effectively resolve any problems that come up with the technology. So, the first thing that I usually do is, see when it was last updated, and that, ominously, I use the status update page just to check it and see what the security updates are. If there’s a lot of them and if there’s a numerous amount of them, and if there’s several years in the past, since the last update, that tells me a lot about the human management around the site, that it might not necessarily have enough technical people around it, or people don’t know that they have to update it, or a number of different human problems related to that. Then, once I have that information, then it’s down to going through each individual section.

So, I note all the different modules that require an update, which ones need security updates. Sometimes sites will specifically hold back one module or another, several versions, and that doesn’t necessarily speak to neglect, but it might be an intentional holdback, because of some bit of custom functionality built around that module that could not be reimplemented easily with the available skill levels that they have within the organization after doing an update. So, that also tells me something.

Then it comes down to, okay, let’s look at the infrastructure of the site. Are they on a platform as a service provider like Acquia or Pantheon or Platform.sh? Are they on shared hosting? Are they on a virtual private server liker Linode or DigitalOcean? Are they on self-managed hosting? Because some organizations mandate self-managed hostings, particularly governments and schools will have a mandate for self-hosting by default. And each of one of those tells me something.

If it’s on shared hosting, that already tells me about the kind of price tier that they’re looking at, how they regard the amount of performance of their site. Do we need to investigate if they have outgrown that. If they’re on a virtual private server. When was the last time the server infrastructure was updated? What distribution of Linux or Unix are they working under? Do we have access to underlying abilities like accessing root so we can perform even more invasive checks, like disc sizes? What software has been installed? What are the user permissions that are used? Who else is using the server? If it’s on a platform-as-a-service provider, that gets a little bit different. Usually those I tend not to audit for infrastructure too deeply, mostly because they tend to work out pretty well by themselves. They’re intended to actually be fairly ‘use it and forget it’.

So, a quick cursory check is important for those, but unless if something specifically stands out to me, I usually don’t investigate them very deeply. So, we’ve covered security, we’ve covered infrastructure, then I start looking at content. What kind of content types do we have? Are we using content types? That sounds like a ridiculous question to some people, but yes, some sites decide, “I don’t know about this Drupal thing. I’m just going to use our raw table and some code, and slap it in there, that’s good enough for me.”

IVAN: We’ve seen it.

TESS: We’ve seen it, and that comes with pluses and minuses, and it’s important that we bring those forth to the client. That is something else. What’s important through all of these little details that we’ve covered is that, it’s not just noting a thing exists, it’s going why does that exist? Why has that happened? Find the underlying story behind the motivation that lead to this current situation. Everything is really about documenting each one of these finer details, and the interesting thing is that usually as you document these details, you start asking better questions yourself, and then you need to go investigate those questions.

So, with content, you might ask, what kind of content types do you use? Do those make sense with the kind of site that they have? Do you have a number of duplicate content types, like news and blog and press release? Are they the same kind of content really just in different categories? Do you have a large number of fields that are unused? Do you have too few fields that you’re making do too many things? Do you shove entire bits of layout into your content? Trust me, we’ve all done it, it’s okay [laughing], but we need to do better than that. There’s a lot of these little bits of story that come too.

Once we’ve investigated the content types and those structures, usually I try seeing what kind of custom integrations that they have, as well. Do we interact with any third-party APIs or commerce organizations or survey organizations? Do we have any dependencies that can be a bit of a risk for us in order to manage going forward? Because if it’s outside of the realm of Drupal, those can be a little brittle and we do need to actually be careful about how those are implemented. Eventually we do come down to custom functionality. You notice we’ve done all of this other stuff, and now, twenty minutes into the podcast, are we talking about custom functionality. Because custom functionality in general with a lot of Drupal sites that we’ve audited, tends to be a lot less than you expect. Usually a well-managed site has only a minor amount of custom code, just enough to pull the site together. Some sites on the other hand have an enumerate amount of custom code, and that also tells us a story. How much custom code do you have? Do you need that amount of custom code for the site that you’re running? Why did that custom code get used? You have to examine each one of these decisions in order to see what the whole picture of the story is.

IVAN: It’s a lengthy, involved process that we undertake, isn’t it? And, I want to make sure that we are clear about what isn’t in the audit. So, the audit is mostly a health check of your site, your infrastructure, and your processes. We do a cursory look at your analytics and a cursory look at your content, and a cursory look at your accessibility. But as far as doing a deep dive into a content audit, or a deep dive into an accessibility audit, which we have done and which we do, that is not part of the deal here.

The main point is to get to a point where we can give you a report, and a status quo, and a set of recommendations about the things that we think you need to fix. Now, let’s just talk about the audit itself. What do you actually get? You get a PDF and for those of you listening, you can go to ten7.com/audit to see an example of a PDF of one audit that we’ve done. It’s been anonymized so there’s no actual client information in there, but you’ll get the gist from the PDF itself. From when we kick off to when we've created the PDF it usually takes about four weeks and at the end of those four weeks, we have a document, a PDF that we then present to the client. We don’t email the client with this PDF and say, “Hey, take a look at this thing. Tell us what you think.”

And we definitely don’t send the email with the PDF in it to the client before we present it to them. That video conference, that presentation of the TEN7Audit is very important. It’s very important to provide that to our clients in real time. Tess, can you talk about that meeting and what that meeting feels like and look like. What’s the goal of that meeting?

TESS: So, first let’s frame what the document looks like. On average these audit documents run 18-35 pages. That's right, pages. I’m a bit wordy. [laughing]

IVAN: That’s right. It’s a big one. [laughing] Right, I mean, this is a serious audit, right? It’s not going to be a couple pages long.

TESS: And, the problem with a document that size that is that comprehensive is that it’s really easy to get drowned in it. There’s just so much detail. There’s no framing around it. There’s no discussion around it. There’s no opportunity to ask questions, and suddenly you easily forget points and questions that you had three pages ago, because you have new ones that have already filled up your entire internal question queue. So as a result, it’s really important to have this meeting at the same time that we hand over the document, because it allows us to make this a conversation, not just, "Here’s the results." Because no one wants "Here’s the results," we really do want to have a conversation about it.

So, the way that it works is, generally we start with the document itself and we briefly talk about the methodology involved. And because sites are all unique, sometimes we do have to adapt our methodology dependently. We’ll point out if we have to run the audit on a local copy for various reasons, and then we start talking about the actual audit findings. And the way that the audit findings are structured are also important, because at the very front we have critical findings. These are the most important things that would need to be fixed with the site immediately. These are things that are going to be possible security attack vectors, critical updates that have yet to be applied, or other critical infrastructure things that need to be resolved as soon as possible.

All of these things need to be acted on relatively quickly to prevent downtime or possible data destruction. Those are usually the first things that we talk about, and they’re the big, big items. And, the idea and the intention behind this is so that we can stress the things that are the most important to fix right now, before we get to other underlying things that might require a longer-term effort. Basically, we want to make sure that we dampen down the campfire, so it doesn’t start a wildfire.

IVAN: [laughing]

TESS: And once we’ve done that, then we go through every different section that is in the audit document and this can be a long meeting. Usually these meetings take about an hour, and we outline each individual point. We don’t read the document because everyone can just read the document, but we point out the things that are the most important that I found with that and give additional context. If there are questions, we can answer them at that point. That way no one feels that their questions go unanswered or that they forgot them, they can always have them right there and we can answer them right there. We go through each individual section and sometimes we will have a finding that is, I don’t know why it was built like this. There’s probably a good reason for this, but I don’t know what it is, and usually at that point I might ask you, the client, why was it built like this? Because sometimes there is no right answer for some of these things.

Sometimes we find, “Oh, well we used a custom table here because we actually have another integration with a GIS application and somewhere else that requires database access.” “Oh, that makes perfect sense.” “Sure, okay. That’s understandable.” Now I don’t need to worry about that particular issue. Now I know that I don’t need to make a recommendation to fix that underlying issue and make it more Drupal-like, because it was intentionally done that way. So, because this is a two-way process, this is really, really important. Once we get all the way through the different categories, and usually by the time we get to the end of it, we’re talking content and theme and UX and then a brief touch on the analytics findings. Then we talk about recommendations, and our recommendations usually come in three distinct tiers.

The first tier of recommendations are usually things that we want to do right now, in order to make sure that we don’t have a wildfire. Things that fix immediate, most critical issues with the site, applying secure updates, fixing any potential security attack vectors, DDOS possibilities, fixing other underlying configuration problems like, caching was disabled for some reason, or maybe we should look at turning Varnish on, or maybe the setting was incorrect, or why do you have user registration open when you’re a brochure site? [laughing] Things that are really simple and really actionable that can be done generally within a week after giving the audit document over.

Then the next tier of recommendations are things that we want to try to do to maximize the site as it currently exists, without fundamentally changing the functionality of the site. So, that’s going to be things like, “Well, do you think that you can enable this kind of cache configuration with Varnish or Memcache? Maybe you can change the way this functionality works so that this bit of functionality will work better for you going forward. Maybe your theme is a little wonky here and needs some correction.” Sometimes we might make a recommendation to change hosting providers at that point as well, because if you’re on shared hosting you might have outgrown that. If you’re on Acquia or Pantheon, you might need to change your hosting plan. If you’re on a VPS (virtual private you might also need to change your pricing plan to get more vCPUs or more disk space, or more network transfer storage, those kinds of things.

IVAN: Or caching even.

TESS: Or caching. The third tier is going to be things that allow the site to reach its full potential, which may involve fundamentally changing certain aspects of how the site functions. So, we might want to say, “Maybe you should make a new theme. Maybe you should take this bit of functionality that was implemented this way and reimplement it this way instead.” Those tend to be bigger projects that require several weeks to months to implement, depending on the kind of site. And some of those might not be something that you want to work on immediately.

Some of those might be, “Yeah, we were thinking about redoing the entire site in Drupal 8 and we’re on Drupal 7.” That’s one of those recommendations, and doing a site rebuild does take time and that’s where that recommendation goes. These three tiers allow you to prioritize which aspects of the site you want to act on as a client without feeling like, “Oh, geez, my site is terrible, and everything is wrong and on fire.” [laughing] No, we break that up for you so that you can know, “Okay, these are the things that we need to fix now, because you don’t want your wheel to fall off while you’re on the highway. Here’s the things that we should probably fix because that’s not good, winter's going to happen eventually, and you need to replace that heater core in your car, because you’re going to get cold eventually (laughing), and then finally maybe you just need a new car." [laughing] Everything comes under car analogies.

IVAN: Yes, it does. Or tractors, right? So, that was a great summary of that list of recommendations and the three tiers, Tier 1, Tier 2, Tier 3. And you’re essentially cherry picking those recommendations that make sense for your needs, for your budget, for your organization moving forward. And what we do as the next step in our process, so this is step one right TEN7Audit, it’s about four weeks, get out of that with an audit report and a list of recommendations. And once we’ve done that you cherry pick the list of recommendations and those become tasks for us. And with that list of tasks, of things that you want changed, things that you want improved, based on your budget and our recommendations, we package that up into the TEN7Improve contract and the next step of the process. And usually that takes between four to eight weeks of our time, and it’s really dependent on kind of the results of the audit.

TESS: The thing that’s also important in this entire thing that often goes unsaid, is that an audit is a wonderful "get-to-know-you" activity. Because now after the audit, we as an organization as TEN7, know your site and have a lot of knowledge about how your site works, and what your motivations are, and what your perspective of your site is. And also, you know us, and you know our processes, and you know our names and our faces, so that you can actually know who to talk to. An audit is a wonderful get-to-know-you exercise, and I cannot stress the importance of that human connection enough in what is otherwise a very dry technical field.

IVAN: The importance of that human process is not just getting to know each other, but to laying the groundwork and the foundation for a long-term relationship after that audit's happened. And I think the next step, the TEN7Improve step, that’s kind of getting to know your code base, getting to know how it’s configured more deeply, not just one person getting a higher-level view of the site, but more than one person getting a deeper level understanding of the technical debt that’s in the site, the way that things are configured exactly, so that there’s not just one person who knows how your site is configured and deployed, and I think the TEN7Improve process is also a good next step for the relationship, because now we’re spending more time with each other, getting to know how each others' work styles are, what your needs are, what our needs are, so I guess you could say the TEN7Audit is kind of the dating part of the relationship, and the improve step is kind of the engagement part of the relationship? I guess it’s the time when you get to know the deep-down, dirty secrets of the code base [laughing].

TESS: Why does the site always leave the socks on the floor? [laughing]

IVAN: [laughing] Exactly. That’s exactly what the Improve process is, the answer to the socks on the floor question, right? So, four weeks for the Audit, four to eight weeks for the Improve process, the outcome of the Improve process is a site that we now know quite well. We know how much technical debt there is, we know how it’s configured, we’ve improved it, we’ve updated it, it’s in a state now that we would be comfortable saying, "We can support this for you from now on." Don’t you think?

TESS: That’s the entire goal of the Improve process, is to get us to a point where we can start working with it regularly without having to worry that the site's going to completely blow up for whatever reason, be it infrastructure or code or simply lack of knowledge.

IVAN: And so now we know the site, right? So we can offer a support agreement and that’s the last step of the process, TEN7Care. The way that the support agreement is structured is, it’s an annual agreement. We agree to some minimum number of hours that we will have every month with you, and the agreement typically covers things like Drupal site maintenance, so we maintain and update the core and contributed modules that are installed. We provide 24/7 uptime monitoring and response, and so that part is really dependent on the hosting provider that you have. So in some cases Pantheon is already monitoring their sites, we’re monitoring in addition to that, and sometimes we don’t have any control about whether Pantheon is up or down, and so we have to revert to their knowledge and them working on an emergency, and we are simply the conduit for you. The other thing that TEN7Care provides is regular backups and archiving and that’s really important isn’t it Tess?

TESS: I can’t stress that enough—how important a backup is, because life is unpredictable, and you want to make sure that you have a backup just in case life throws something very, very nasty in your direction.

IVAN: And we’ve got a number of blog posts and podcasts that we’ve done where we've talked about backups and details of what you should have. We use Tractorbeam, the open-source solution that we’ve published and provided to the community to do those backups of your website. Remind me again, Tess, it’s daily, weekly, monthly, right? And, two different off-site locations.

TESS: Mm-hmm. Correct.

IVAN: Great. So, AWS (Amazon Web Services) and Google Cloud, DigitalOcean, those are the three different companies, three different places. So that’s covered under your TEN7Care support agreement. And then all of the CI, continuous integration, and automation goodness that Tessa absolutely loves and that I am a huge proponent of, that comes as part of TEN7Care as well, right? So, our regular release process, the use of feature branches, the use of code review, the fact that we can push code and it deploys to numerous different environments, automatically. Do you want to say a couple things about that? I don’t want to prevent you from geeking out here [laughing], so do say something about that.

TESS: Mostly the reason why CI is particularly wonderful is because the problem is that human beings are inconsistent. You’ve had a bad night of sleep or you’ve read something upsetting and you might be distracted, and that can cause real downtime and real outages and real technical problems. The idea behind using CI is so that you remove more human hands from the process, and outsource that to a piece of technology that can do that consistently every time and be a lot more situationally aware of what’s going on when you do the deploy. So, having CI allows us to respond to changes a lot more quickly, a lot faster, and make sure there’s accountability at every step of the process with regards to updates and feature deployment.

IVAN: I couldn’t have said it better myself. Just wonderful, thank you. So, TEN7Care is the last step in the process, preceded by TEN7Improve and, of course, the TEN7Audit right at the beginning. I think we’ve kind of gone through the whole process, right, beginning to end.

TESS: That’s the whole thing.

IVAN: That’s the whole thing. Thanks again for being on the podcast. It’s always such a pleasure to talk to you, Tess.

TESS: Not a problem.

IVAN: So, if this podcast sounded interesting to you, and you think we might be able to help your organization in some way, we’d love to hear from you. Send us an email, send it to [email protected] to start a conversation. You can also find out more on our website at ten7.com/welcome. That’ll take you to our blog post on the whole process, and you’ll see a link to the example of the audit and an example of the support agreement as well.

You’ve been listening to the TEN7 Podcast. Find us online at ten7.com/podcast. And if you have a second, do send us a message, we love hearing from you. Our email address is [email protected]. Until next time, this is Ivan Stegic. Thank you for listening.

Apr 24 2019
Apr 24

Your browser does not support the audio element. TEN7-Podcast-Ep-058-TEN7s-Onboarding-Process.mp3

Are you looking for someone to support your existing Drupal site? Or, are you an agency and you're considering taking on a new client? We think you should date before getting married right away. Trust us, it's better for both parties! In this podcast, Ivan Stegic and our DevOps Engineer Tess Flynn discuss the TEN7 courtship—er—new client onboarding process, which insures that we get to know your site better than you do!

Running time: 39 minutes
Host: Ivan Stegic
Guest: Tess Flynn

In this podcast we'll discuss: 

  • The difference between discovery & design clients and audit–improve–support clients
  • Why we don’t just say “yes” to a client that gives us money to support their site
  • How the TEN7Audit process is like CarTalk (the multiple layers of the audit and troubleshooting process)
  • The simple check that tells Tess how much you love (or neglect) your site
  • The topics of the TEN7Audit: security, infrastructure, UX and theming, content types
  • We get the data, then try to figure out the underlying human story
  • Why we take the time to present our audit findings to you (in three tiers) vs. dumping the PDF in an email 
  • Tess compares your website to a car. Mark your TEN7 Bingo cards!
  • After TEN7Improve, we’re intimate with your site and know whether we want to support it for the long haul
  • How we take backups for TEN7Care so seriously we created a product (Tractorbeam) to do it for us (and you)

Links:

TRANSCRIPT

IVAN STEGIC: Hey everyone you're listening to the TEN7 podcast, where we get together every fortnight and sometimes more often, to talk about technology, business and the humans in it. I'm your host Ivan Stegic. We recently published a blog post called ‘Becoming a TEN7 Support Client, You Can’t Just Give Us Cash’, and we think it’s a good description of the process of becoming a client in which we support a site that we didn’t build. I thought it might be useful to talk about this whole process as well in a podcast, and that’s what we’re going to be focusing on in this episode. To help me flesh out the details, I have Tess Flynn, our DevOps Engineer joining me once again. Hey Tess.

TESS: Hello.

IVAN: I wanted to start by talking about TEN7’s business, and how clients come to us, and how we have distinct categories of clients. And then we’ll kind of get into the nitty gritty of the process of onboarding a client, and that’s where I thought you could help us out.

Essentially, we have two distinct categories of clients.

One category is a new site build, or a new feature build, and this is a client that specifically wants us to create something from scratch or add on to an existing deployment. And usually that process is discovery and design, some sort of strategy, then development and launch, then we support the site we’ve built. On the other side, the other major category of business we do is supporting an existing site. So, that’s supporting a client or a prospect that comes to us that has a Drupal site that someone’s already built for them. And for whatever reason, they need it supported and maintained and looked after for an extended period of time.

The process there is similar. We audit the site, we improve it and then we support it, and the products we have for those services are called TEN7Audit, TEN7Improve and TEN7Care. Now, there’s a natural progression, I think, from learning everything we can about you and what you have, to recommending things we could improve, to then supporting and maintaining that site for you. But the reason we came up with this process was, a particularly difficult client that we had some years ago, And I did something as an owner, that I shouldn’t have done, and that was, say yes to supporting a client whose site I knew absolutely nothing about, whose site no one on our team evaluated or saw, that turned out not to be a brochure site, but a complex, inner connection of many different modules, some custom, some not.

We ended up with a commitment with a client for two years that went on for two years longer than it should have, negatively affecting our team, the morale on our team, resulted in some internal soul searching within the organization. We discovered the DiSC analysis process, we applied it to everyone in the company, we focused on our own mission, our own values. And when we came out on the other side, I basically didn’t want this to ever happen again, right? From our perspective, we wanted to do our best work for the client, and we were being handcuffed, because we didn’t do our due diligence right in the beginning of the engagement. And, of course from the clients' perspective, they expected nothing less, right? They wanted us to do the best work for them, we possibly could be doing, and so we came up with this three-step process that sets us both up for success.

Essentially, we do an evaluation and an audit of an existing Drupal site. We call that the TEN7Audit. Once we’ve evaluated it, we kind of have to get to know the code base a little better, and nothing’s ever perfect, there’s always some amount of work that needs to be done. And so we do that. The next step we call the TEN7Improve step. And once that’s complete, we offer TEN7Care, our support agreement that keeps the client's Drupal site humming, kind of the way it should be. So, it’s three steps, TEN7Audit, TEN7Improve, TEN7Care, and what I’d like to do now, is talk about the details of TEN7Audit.

So, a prospect comes to us, and they ask us if we want to support their site, and we say, “You seem nice, I think we’d like to work with you. Let’s take a look under your hood.” And, that’s what the TEN7Audit really is. Tess, you’re the principal engineer that’s responsible for the TEN7Audit. So, when a prospect comes to us and I say, “Okay, let's do the audit,” it lands in your lap, right? And, you’re then tasked with doing this audit. And we’ve done so many of them we have a good process around what we do, and I want to find out about what exactly happens during the audit? What’s the first thing you do?

TESS: The first thing I usually do when it comes to doing an audit is that I’m going to need access to the underlying infrastructure. So, I’m going to need access to SSH in order to get to the file system on which the site is hosted. And I will also need database access credentials, so that I can get a copy of the website itself. Now, I like doing some audits entirely in place if it’s possible, without copying it to a local environment, but increasingly this is difficult, because depending on the client and depending on the site that might not be possible. For some shared hosting providers, for example, adding an additional tech in place can be a risky proposition, because it can cause some issues with the site, in order to perform the audit. Also, certain platform-as-a-service hosts actually don’t really let you do that very easily, in which case you do have to copy the site off and then run it locally. And then, in some cases, you might have the ability to run additional tools in place, but you’ll have to run it locally anyways, because the actual environment that the site is running on already has something already so fundamentally wrong with it that you can’t even run those tools in the first place.

So, it all comes down to: how do I get my hands on the site code and the site database? That is always the first problem when it comes to doing an audit. And, the thing with doing that is, some clients actually find that kind of intimidating, because they don’t know who you are, they don’t know if they can trust you, and they’re not sure if they really want to give that information over. And, it’s really necessary in order to perform an effective audit in the first place. Otherwise, the best I can do is look around a little bit. Now, if I do get that access, then I can start tearing down how the site itself is built, and this becomes an interesting process.

The very first thing to do is to run several auditing tools. Depending on the site, that could be Healthcheck, or the Site Audit module, or the Hacked! module. There’s a number of different tools that we use in order to facilitate gathering the best information necessary, to see what the red flags are in the site, if there are any. What’s interesting about this process is, already the attempt to run these tools will tell us something about the site itself. Can we run them in place? If we can’t run them in place, why can’t we run them in place? What’s the underlying problem that’s preventing us from doing that? Is it because of the hosting provider? Is it because of the way that the server infrastructure is set up? Is something else wrong that we need to take note of, that is something that we need to remark on our document?

Once we actually run the tools, we examine the output of those tools, and usually this gives us kind of a 1,000-foot perspective of the general health of the site, but it doesn’t allow us to really uncover the underlying causes of some of this. An auditing tool can tell you, say, you’re running out of disc space. But why are you running out of disc space? You might have, well, there’s excessive activity in the database, or there’s excessive CPU draw, well why? What’s really doing that? So, all these tools just paint a more detailed picture, and at some point, you do have to start breaking those down and going after them and investigating them yourself.

IVAN: So, I was going to ask you about the tools that you use, and you mentioned that you usually SSH into the clients website. I would imagine that if there’s already some continuous integration in place, or some codebase in place, maybe you’re using Git to get those files as well. And then you mentioned Site Audit module and the Healthcheck module and Hacked!. And it’s not just modules that you use to determine the health of a website too, right? You are also evaluating the infrastructure, so, is it a shared host? Is there Varnish installed? Is there Memcached? How is the host configured? What kind of access do you have? Those are also other things that we evaluate. Could you talk about the next step? Once you’ve done, kind of the tools evaluation, what are the other things you evaluate?

TESS: It’s kind of like an episode of Car Talk really. A caller calls in, says, “Oh, hey, my site is making this weird sound,” and then after, you should not turn it like that, so it doesn’t make that sound. Then you actually ask, “Okay, when does it make the sound?” “How long has it made the sound?” And you start following the investigative chain. Using the tools really are just the first step in that process, because often with websites, as with a lot of modern vehicles, they are so complicated we don’t really know what’s wrong with them intuitively. We will only be able to know after analysis, and usually that requires an additional amount of technical expertise. So, the tooling basically gives us a rough topology of what the site health is like. And, afterwards, we need to investigate each one of those vectors. And a lot of the time, it comes down to how our audit document is structured, which allows me to investigate that. So, what I tend to do is, I will first start by running the tools and see if there’s any red flags in there.

If there aren’t, then the next thing I do is see if the client has mentioned anything in particular that we want to look at when it comes to the site. Sometimes that can give us a good clue and sometimes that could be a false positive, and sometimes we don’t have that information. So, it depends on what we have available to track down the necessary clues.

With our audit document, we actually break that out to several different sections. There’s security findings, infrastructure findings, UX and theming findings, and content findings. Each one of these is a section by which we can go and do further investigation on how the site health is working. Usually the next thing after I do the first pass is to check when was the last time the site was updated. This sounds like such a simple, easy thing, but it tells you a lot about, not necessarily the technology, but the people around the site, and how they worked with the site and regarded it.

Everything we do in technology is about people, so you have to understand the underlying human story around the technology, and that will allow you to effectively resolve any problems that come up with the technology. So, the first thing that I usually do is, see when it was last updated, and that, ominously, I use the status update page just to check it and see what the security updates are. If there’s a lot of them and if there’s a numerous amount of them, and if there’s several years in the past, since the last update, that tells me a lot about the human management around the site, that it might not necessarily have enough technical people around it, or people don’t know that they have to update it, or a number of different human problems related to that. Then, once I have that information, then it’s down to going through each individual section.

So, I note all the different modules that require an update, which ones need security updates. Sometimes sites will specifically hold back one module or another, several versions, and that doesn’t necessarily speak to neglect, but it might be an intentional holdback, because of some bit of custom functionality built around that module that could not be reimplemented easily with the available skill levels that they have within the organization after doing an update. So, that also tells me something.

Then it comes down to, okay, let’s look at the infrastructure of the site. Are they on a platform as a service provider like Acquia or Pantheon or Platform.sh? Are they on shared hosting? Are they on a virtual private server liker Linode or DigitalOcean? Are they on self-managed hosting? Because some organizations mandate self-managed hostings, particularly governments and schools will have a mandate for self-hosting by default. And each of one of those tells me something.

If it’s on shared hosting, that already tells me about the kind of price tier that they’re looking at, how they regard the amount of performance of their site. Do we need to investigate if they have outgrown that. If they’re on a virtual private server. When was the last time the server infrastructure was updated? What distribution of Linux or Unix are they working under? Do we have access to underlying abilities like accessing root so we can perform even more invasive checks, like disc sizes? What software has been installed? What are the user permissions that are used? Who else is using the server? If it’s on a platform-as-a-service provider, that gets a little bit different. Usually those I tend not to audit for infrastructure too deeply, mostly because they tend to work out pretty well by themselves. They’re intended to actually be fairly ‘use it and forget it’.

So, a quick cursory check is important for those, but unless if something specifically stands out to me, I usually don’t investigate them very deeply. So, we’ve covered security, we’ve covered infrastructure, then I start looking at content. What kind of content types do we have? Are we using content types? That sounds like a ridiculous question to some people, but yes, some sites decide, “I don’t know about this Drupal thing. I’m just going to use our raw table and some code, and slap it in there, that’s good enough for me.”

IVAN: We’ve seen it.

TESS: We’ve seen it, and that comes with pluses and minuses, and it’s important that we bring those forth to the client. That is something else. What’s important through all of these little details that we’ve covered is that, it’s not just noting a thing exists, it’s going why does that exist? Why has that happened? Find the underlying story behind the motivation that lead to this current situation. Everything is really about documenting each one of these finer details, and the interesting thing is that usually as you document these details, you start asking better questions yourself, and then you need to go investigate those questions.

So, with content, you might ask, what kind of content types do you use? Do those make sense with the kind of site that they have? Do you have a number of duplicate content types, like news and blog and press release? Are they the same kind of content really just in different categories? Do you have a large number of fields that are unused? Do you have too few fields that you’re making do too many things? Do you shove entire bits of layout into your content? Trust me, we’ve all done it, it’s okay [laughing], but we need to do better than that. There’s a lot of these little bits of story that come too.

Once we’ve investigated the content types and those structures, usually I try seeing what kind of custom integrations that they have, as well. Do we interact with any third-party APIs or commerce organizations or survey organizations? Do we have any dependencies that can be a bit of a risk for us in order to manage going forward? Because if it’s outside of the realm of Drupal, those can be a little brittle and we do need to actually be careful about how those are implemented. Eventually we do come down to custom functionality. You notice we’ve done all of this other stuff, and now, twenty minutes into the podcast, are we talking about custom functionality. Because custom functionality in general with a lot of Drupal sites that we’ve audited, tends to be a lot less than you expect. Usually a well-managed site has only a minor amount of custom code, just enough to pull the site together. Some sites on the other hand have an enumerate amount of custom code, and that also tells us a story. How much custom code do you have? Do you need that amount of custom code for the site that you’re running? Why did that custom code get used? You have to examine each one of these decisions in order to see what the whole picture of the story is.

IVAN: It’s a lengthy, involved process that we undertake, isn’t it? And, I want to make sure that we are clear about what isn’t in the audit. So, the audit is mostly a health check of your site, your infrastructure, and your processes. We do a cursory look at your analytics and a cursory look at your content, and a cursory look at your accessibility. But as far as doing a deep dive into a content audit, or a deep dive into an accessibility audit, which we have done and which we do, that is not part of the deal here.

The main point is to get to a point where we can give you a report, and a status quo, and a set of recommendations about the things that we think you need to fix. Now, let’s just talk about the audit itself. What do you actually get? You get a PDF and for those of you listening, you can go to ten7.com/audit to see an example of a PDF of one audit that we’ve done. It’s been anonymized so there’s no actual client information in there, but you’ll get the gist from the PDF itself. From when we kick off to when we've created the PDF it usually takes about four weeks and at the end of those four weeks, we have a document, a PDF that we then present to the client. We don’t email the client with this PDF and say, “Hey, take a look at this thing. Tell us what you think.”

And we definitely don’t send the email with the PDF in it to the client before we present it to them. That video conference, that presentation of the TEN7Audit is very important. It’s very important to provide that to our clients in real time. Tess, can you talk about that meeting and what that meeting feels like and look like. What’s the goal of that meeting?

TESS: So, first let’s frame what the document looks like. On average these audit documents run 18-35 pages. That's right, pages. I’m a bit wordy. [laughing]

IVAN: That’s right. It’s a big one. [laughing] Right, I mean, this is a serious audit, right? It’s not going to be a couple pages long.

TESS: And, the problem with a document that size that is that comprehensive is that it’s really easy to get drowned in it. There’s just so much detail. There’s no framing around it. There’s no discussion around it. There’s no opportunity to ask questions, and suddenly you easily forget points and questions that you had three pages ago, because you have new ones that have already filled up your entire internal question queue. So as a result, it’s really important to have this meeting at the same time that we hand over the document, because it allows us to make this a conversation, not just, "Here’s the results." Because no one wants "Here’s the results," we really do want to have a conversation about it.

So, the way that it works is, generally we start with the document itself and we briefly talk about the methodology involved. And because sites are all unique, sometimes we do have to adapt our methodology dependently. We’ll point out if we have to run the audit on a local copy for various reasons, and then we start talking about the actual audit findings. And the way that the audit findings are structured are also important, because at the very front we have critical findings. These are the most important things that would need to be fixed with the site immediately. These are things that are going to be possible security attack vectors, critical updates that have yet to be applied, or other critical infrastructure things that need to be resolved as soon as possible.

All of these things need to be acted on relatively quickly to prevent downtime or possible data destruction. Those are usually the first things that we talk about, and they’re the big, big items. And, the idea and the intention behind this is so that we can stress the things that are the most important to fix right now, before we get to other underlying things that might require a longer-term effort. Basically, we want to make sure that we dampen down the campfire, so it doesn’t start a wildfire.

IVAN: [laughing]

TESS: And once we’ve done that, then we go through every different section that is in the audit document and this can be a long meeting. Usually these meetings take about an hour, and we outline each individual point. We don’t read the document because everyone can just read the document, but we point out the things that are the most important that I found with that and give additional context. If there are questions, we can answer them at that point. That way no one feels that their questions go unanswered or that they forgot them, they can always have them right there and we can answer them right there. We go through each individual section and sometimes we will have a finding that is, I don’t know why it was built like this. There’s probably a good reason for this, but I don’t know what it is, and usually at that point I might ask you, the client, why was it built like this? Because sometimes there is no right answer for some of these things.

Sometimes we find, “Oh, well we used a custom table here because we actually have another integration with a GIS application and somewhere else that requires database access.” “Oh, that makes perfect sense.” “Sure, okay. That’s understandable.” Now I don’t need to worry about that particular issue. Now I know that I don’t need to make a recommendation to fix that underlying issue and make it more Drupal-like, because it was intentionally done that way. So, because this is a two-way process, this is really, really important. Once we get all the way through the different categories, and usually by the time we get to the end of it, we’re talking content and theme and UX and then a brief touch on the analytics findings. Then we talk about recommendations, and our recommendations usually come in three distinct tiers.

The first tier of recommendations are usually things that we want to do right now, in order to make sure that we don’t have a wildfire. Things that fix immediate, most critical issues with the site, applying secure updates, fixing any potential security attack vectors, DDOS possibilities, fixing other underlying configuration problems like, caching was disabled for some reason, or maybe we should look at turning Varnish on, or maybe the setting was incorrect, or why do you have user registration open when you’re a brochure site? [laughing] Things that are really simple and really actionable that can be done generally within a week after giving the audit document over.

Then the next tier of recommendations are things that we want to try to do to maximize the site as it currently exists, without fundamentally changing the functionality of the site. So, that’s going to be things like, “Well, do you think that you can enable this kind of cache configuration with Varnish or Memcache? Maybe you can change the way this functionality works so that this bit of functionality will work better for you going forward. Maybe your theme is a little wonky here and needs some correction.” Sometimes we might make a recommendation to change hosting providers at that point as well, because if you’re on shared hosting you might have outgrown that. If you’re on Acquia or Pantheon, you might need to change your hosting plan. If you’re on a VPS (virtual private you might also need to change your pricing plan to get more vCPUs or more disk space, or more network transfer storage, those kinds of things.

IVAN: Or caching even.

TESS: Or caching. The third tier is going to be things that allow the site to reach its full potential, which may involve fundamentally changing certain aspects of how the site functions. So, we might want to say, “Maybe you should make a new theme. Maybe you should take this bit of functionality that was implemented this way and reimplement it this way instead.” Those tend to be bigger projects that require several weeks to months to implement, depending on the kind of site. And some of those might not be something that you want to work on immediately.

Some of those might be, “Yeah, we were thinking about redoing the entire site in Drupal 8 and we’re on Drupal 7.” That’s one of those recommendations, and doing a site rebuild does take time and that’s where that recommendation goes. These three tiers allow you to prioritize which aspects of the site you want to act on as a client without feeling like, “Oh, geez, my site is terrible, and everything is wrong and on fire.” [laughing] No, we break that up for you so that you can know, “Okay, these are the things that we need to fix now, because you don’t want your wheel to fall off while you’re on the highway. Here’s the things that we should probably fix because that’s not good, winter's going to happen eventually, and you need to replace that heater core in your car, because you’re going to get cold eventually (laughing), and then finally maybe you just need a new car." [laughing] Everything comes under car analogies.

IVAN: Yes, it does. Or tractors, right? So, that was a great summary of that list of recommendations and the three tiers, Tier 1, Tier 2, Tier 3. And you’re essentially cherry picking those recommendations that make sense for your needs, for your budget, for your organization moving forward. And what we do as the next step in our process, so this is step one right TEN7Audit, it’s about four weeks, get out of that with an audit report and a list of recommendations. And once we’ve done that you cherry pick the list of recommendations and those become tasks for us. And with that list of tasks, of things that you want changed, things that you want improved, based on your budget and our recommendations, we package that up into the TEN7Improve contract and the next step of the process. And usually that takes between four to eight weeks of our time, and it’s really dependent on kind of the results of the audit.

TESS: The thing that’s also important in this entire thing that often goes unsaid, is that an audit is a wonderful "get-to-know-you" activity. Because now after the audit, we as an organization as TEN7, know your site and have a lot of knowledge about how your site works, and what your motivations are, and what your perspective of your site is. And also, you know us, and you know our processes, and you know our names and our faces, so that you can actually know who to talk to. An audit is a wonderful get-to-know-you exercise, and I cannot stress the importance of that human connection enough in what is otherwise a very dry technical field.

IVAN: The importance of that human process is not just getting to know each other, but to laying the groundwork and the foundation for a long-term relationship after that audit's happened. And I think the next step, the TEN7Improve step, that’s kind of getting to know your code base, getting to know how it’s configured more deeply, not just one person getting a higher-level view of the site, but more than one person getting a deeper level understanding of the technical debt that’s in the site, the way that things are configured exactly, so that there’s not just one person who knows how your site is configured and deployed, and I think the TEN7Improve process is also a good next step for the relationship, because now we’re spending more time with each other, getting to know how each others' work styles are, what your needs are, what our needs are, so I guess you could say the TEN7Audit is kind of the dating part of the relationship, and the improve step is kind of the engagement part of the relationship? I guess it’s the time when you get to know the deep-down, dirty secrets of the code base [laughing].

TESS: Why does the site always leave the socks on the floor? [laughing]

IVAN: [laughing] Exactly. That’s exactly what the Improve process is, the answer to the socks on the floor question, right? So, four weeks for the Audit, four to eight weeks for the Improve process, the outcome of the Improve process is a site that we now know quite well. We know how much technical debt there is, we know how it’s configured, we’ve improved it, we’ve updated it, it’s in a state now that we would be comfortable saying, "We can support this for you from now on." Don’t you think?

TESS: That’s the entire goal of the Improve process, is to get us to a point where we can start working with it regularly without having to worry that the site's going to completely blow up for whatever reason, be it infrastructure or code or simply lack of knowledge.

IVAN: And so now we know the site, right? So we can offer a support agreement and that’s the last step of the process, TEN7Care. The way that the support agreement is structured is, it’s an annual agreement. We agree to some minimum number of hours that we will have every month with you, and the agreement typically covers things like Drupal site maintenance, so we maintain and update the core and contributed modules that are installed. We provide 24/7 uptime monitoring and response, and so that part is really dependent on the hosting provider that you have. So in some cases Pantheon is already monitoring their sites, we’re monitoring in addition to that, and sometimes we don’t have any control about whether Pantheon is up or down, and so we have to revert to their knowledge and them working on an emergency, and we are simply the conduit for you. The other thing that TEN7Care provides is regular backups and archiving and that’s really important isn’t it Tess?

TESS: I can’t stress that enough—how important a backup is, because life is unpredictable, and you want to make sure that you have a backup just in case life throws something very, very nasty in your direction.

IVAN: And we’ve got a number of blog posts and podcasts that we’ve done where we've talked about backups and details of what you should have. We use Tractorbeam, the open-source solution that we’ve published and provided to the community to do those backups of your website. Remind me again, Tess, it’s daily, weekly, monthly, right? And, two different off-site locations.

TESS: Mm-hmm. Correct.

IVAN: Great. So, AWS (Amazon Web Services) and Google Cloud, DigitalOcean, those are the three different companies, three different places. So that’s covered under your TEN7Care support agreement. And then all of the CI, continuous integration, and automation goodness that Tessa absolutely loves and that I am a huge proponent of, that comes as part of TEN7Care as well, right? So, our regular release process, the use of feature branches, the use of code review, the fact that we can push code and it deploys to numerous different environments, automatically. Do you want to say a couple things about that? I don’t want to prevent you from geeking out here [laughing], so do say something about that.

TESS: Mostly the reason why CI is particularly wonderful is because the problem is that human beings are inconsistent. You’ve had a bad night of sleep or you’ve read something upsetting and you might be distracted, and that can cause real downtime and real outages and real technical problems. The idea behind using CI is so that you remove more human hands from the process, and outsource that to a piece of technology that can do that consistently every time and be a lot more situationally aware of what’s going on when you do the deploy. So, having CI allows us to respond to changes a lot more quickly, a lot faster, and make sure there’s accountability at every step of the process with regards to updates and feature deployment.

IVAN: I couldn’t have said it better myself. Just wonderful, thank you. So, TEN7Care is the last step in the process, preceded by TEN7Improve and, of course, the TEN7Audit right at the beginning. I think we’ve kind of gone through the whole process, right, beginning to end.

TESS: That’s the whole thing.

IVAN: That’s the whole thing. Thanks again for being on the podcast. It’s always such a pleasure to talk to you, Tess.

TESS: Not a problem.

IVAN: So, if this podcast sounded interesting to you, and you think we might be able to help your organization in some way, we’d love to hear from you. Send us an email, send it to [email protected] to start a conversation. You can also find out more on our website at ten7.com/welcome. That’ll take you to our blog post on the whole process, and you’ll see a link to the example of the audit and an example of the support agreement as well.

You’ve been listening to the TEN7 Podcast. Find us online at ten7.com/podcast. And if you have a second, do send us a message, we love hearing from you. Our email address is [email protected]. Until next time, this is Ivan Stegic. Thank you for listening.

Mar 14 2019
Mar 14

TEN7 is a full-service digital firm, and our tagline is “We create and care for Drupal-powered websites.” Creating and building a website is the sexy visible part, but caring for a website over time is the just-as-important maintenance work. Keeping your site code updated, backed up, secure and performing well is the job of a support team.

Most of the clients we design and build websites for also ask us to support their site after it’s built, which we happily do. But clients with sites built by other companies often approach us to support their site as well. However, we don’t take on support clients lightly.

Why You Can’t Just Pay Us to Be a Support Client

If you want to pay us to support your site, why wouldn’t we just do it? Well, when we were just starting out, there was a time when we’d take on whoever came along and offered us cash.

We took on support clients without knowing anything about what was “under the hood” of their site. We had to scramble to figure out how a site worked as support issues came in, which led to increased problem-solving times. This resulted in subpar support of the client, and subpar support led to real unhappiness for our team members.

As you might have guessed, the impetus for a change in the way we did business was a particularly difficult client and their site. It took two years to effectively end the relationship. The experience was a turning point in our company’s existence: it led me to think about what we’d done wrong, and reverse-engineer a way to prevent it from happening again. You can hear more about this experience in my Drupal Camp talk, “Know Yourself First.”

As we’ve grown and matured as a company, we’ve learned the value of recognizing the right clients and building the right process to support their sites. This ensures that we do work that meets our high standards, that our team is respected and happy, and that there’s a good vibe between all parties. Our multi-step onboarding process for new support clients lets us accomplish all these goals.

The Audit → Improve → Care Process

Each step in this process is a separate engagement, and has its own pricing and contract. When a step is completed, either party can quit with no hard feelings. The three steps are:

  1. TEN7Audit: We perform a comprehensive site audit and present a report of findings and recommendations.
  2. TEN7Improve: We implement selected recommendations from the audit report (at minimum the most critical ones) based on budget and need.
  3. TEN7Care: This is the final part of the process, an annual support agreement.

Let’s learn more about each step.

Step 1: TEN7Audit

Before we’ll agree to support your site, we need to know what we’re dealing with.

To figure this out, we perform a complete audit for a flat fee of $2,500. We settled on this cost because while it’s not exorbitant, it’s high enough that it will disqualify anyone who isn’t serious. The entire process can take up to six weeks, from when a client first expresses interest until the day we present our findings report.

Site Audit and Analysis

The site audit starts with us making an exact replica of the site so that it can be examined without affecting its live operation. We run Healthcheck, a Drupal module sponsored and maintained by TEN7, to generate a baseline report for review.

Healthcheck is like your Drupal site's own personal physician and can run continuously after installation to keep tabs on the health of the site over time (hence its name!) It has a user-friendly, action-oriented dashboard that shows you issues, ranked in terms of urgency. We install Healthcheck on sites we support.

In the TEN7Audit process, Healthcheck provides the initial lay of the land and identifies things like whether module updates are required, or whether the site has been hacked or modified. Next, we get humans involved to take a deeper look at where the site is hosted, whether there is a version control workflow, whether continuous integration and automation exist, how the infrastructure is configured and more. For example, a Healthcheck-identified performance issue (slow page load speed) could have numerous causes, from giant images to caching being disabled. Humans have the best chance of ascertaining and reporting the causes.

Audit Results

When the TEN7Audit process is complete, we compile an audit report of findings with a prioritized list of issues and recommendations. We define critical issues that need the most attention, as well as recommendations for repair and optimization:

  • Tier 1 - Critical: Issues that need to be fixed as soon as possible for site security
  • Tier 2 - Best Practices: Improvements that will reduce technical debt, optimize the site, and won’t negatively affect the site if not immediately done
  • Tier 3 - Nice to haves: Long-term improvement goals you should consider for the sustainability and success of your site

Here’s an example of a TEN7Audit Findings Report.

While some companies might just send you the PDF of an audit report, we'll either present the findings and recommendations to you in person or through a Zoom conference. It's a lot of information, and we want to be sure to thoroughly explain it to you and be there to answer any questions that come up. This is also a chance for us to hear from you about how decisions or compromises were made for the site to get to its current state. 

What’s Next?

After a TEN7Audit has been completed, the next step involves determining which recommendations from the report should be implemented, and then doing so to improve your site. Of course, this also gives us an opportunity to evaluate our relationship with you and decide whether it makes sense for us to continue (and if not, no hard feelings!)

Step 2: TEN7Improve

You’ve decided you’d like to have TEN7 work on the improvements identified in the TEN7Audit, and we’ve also decided that we’d like to work with you to help improve your site.

The recommendations in the TEN7Audit report will be accompanied by an estimated number of hours to fix each issue. The total cost for TEN7Improve varies for every site, and is calculated by multiplying the number of hours estimated to fix chosen issues and our hourly rate at the time of the site audit. A budget for TEN7Improve starts at around $3,000 for a site with minor issues and can go up to $10,000 or more for a site in bad shape.

If we’re going to put ourselves in a position to care for your site in the long run, you'll expect consistent, high quality work. We require Tier 1 Critical Issues to be fixed to proceed with the TEN7Improve step, but we also highly recommend fixing Tier 2 Best Practice issues as well (and most clients do).

If you have budget constraints, items in the recommendations list can be cherry picked, and you can determine the order in which they should be addressed. Occasionally there will be issues found in the audit that can’t be fixed at any time—they may be so big, or so entrenched in the site infrastructure that it would take an enormous effort to fix. In such rare cases, a complete site rebuild is warranted and recommended.

What’s Next?

By this point, the hard work has been done—your site has had performance and infrastructure issues fixed, Drupal core and modules are up to date, and the site is as secure as we can make it. We’re now more comfortable with your code; we have a better idea about how the site is built and how it works. Hopefully we’ve been able to make a real difference in the infrastructure, security, performance and usability of your site.

After this longer engagement, we also have more information about what it’s like to work with you: how you communicate, how pleasant or demanding you are, and how quickly you pay your bills. You also know what it’s like working with us: how responsive we are, how important you feel, how much value you’re receiving.

We should now be in a position to offer a TEN7Care Support Agreement, in which we support a site we’ve audited, improved and are familiar with. It gives us the opportunity to take care of a site that we didn’t build, but that we feel comfortable being wholly responsible for. In some cases, we may not offer a support agreement, and of course, you may not wish to pursue one either.

Step 3: TEN7Care

Since we are now comfortable with the inner workings of your site, we can estimate how many hours per year will be required for site monitoring and maintenance. This includes time for periodic site updates, security patches, uptime monitoring and regular backups and archiving.

The TEN7Care Support Agreement starts at five hours per month, billed on a monthly basis. The agreement typically covers:

  • Drupal site maintenance: maintain and update core and contributed Drupal modules
  • 24x7 site uptime monitoring and response
  • Regular backups and archiving: automated nightly backups, weekly and monthly snapshots backed up to two offsite servers
  • Monthly traffic analytics insights
  • Guaranteed availability of your business’ critical paths after updates
  • Installation and use of a versioning system to manage multiple site environments
  • Regular releases of code with automated release notes
  • Support availability during business hours via email, video conference or Slack

Here’s an example of a TEN7Care Support Agreement.

What’s Next?

Yes there is a next, even here! Six weeks before the end of a TEN7Care Support Agreement, we meet internally to review the last year of work: how many hours we billed and how much work your site required. We’ll discuss whether to adjust the hours for the coming year, and whether a rate change is due, amongst other things.

That’s How You Become a Support Client!

As a company, we strive to do our best for our clients, and we set a high bar for quality work. I believe our success is directly proportional to our clients’ satisfaction with our work. Moreover, as a team, we have to be happy doing that work. If either of these two things don’t happen, we aren’t going to do our best, and there won’t be satisfaction in what we’ve created.

I think our TEN7Audit → TEN7Improve → TEN7Care process helps us accomplish all these goals.

Would You Like to Work With Us?

Do you have a Drupal site that needs support? We can help! Fill out this form and we'll get back to you quickly!

Jan 02 2019
Jan 02

Your browser does not support the audio element. TEN7-Podcast-Ep-050-Dries-Buytaert.mp3

In this, our 50th episode, Ivan is joined by Dries Buytaert, the founder of Drupal, an open source software developer, a startup founder, technology executive, father, world traveler and photographer. Subscribe to the podcast.

Here's what we're discussing in this episode:

  • Dries' life and career
  • The creation and emergence of Drupal
  • Growing up in Antwerp
  • Living in Boston
  • Shared love for tennis
  • Yet another flying start with the Commodore 64
  • The power of copy/paste
  • Writing code for his father's medical practice
  • Wonka VM, Linux and Java
  • Juggling multiple careers particularly fatherhood
  • DrupalCons
  • The MTV saga
  • TEN7's decision to go Drupal
  • The move from Belgium to Boston
  • Circumventing the globe
  • Benevolent Dictators for Life
  • Creating thousands of roles out of one
  • Drupal's Values and Principles
  • Drupal as a force for good in the world
  • Drupal's future and Dries' role moving ahead

TRANSCRIPT

IVAN STEGIC: Hey everyone, you're listening to the TEN7 Podcast, where we get together every fortnight, and sometimes more often, to talk about technology, business and the humans in it. I'm your host Ivan Stegic. Well, we've hit 50 episodes. When we started this podcast in April of 2017, we did so by limiting it to only five minutes a time and by calling it an audiocast. And we did that because we wanted to experiment with some marketing that we really weren't familiar with and that we hadn't tried before. And I, of course, thought the best way to do this is to try something short and sweet, that's a quick blast of audio. Because who has time to listen to more than that? Well that didn't last terribly long before we realized that five minutes simply wasn't enough time to tell a story, or to get into the meat of an idea. I guess that's obvious now, in retrospect of course. So I'm glad that this podcast has evolved into what it is today, a hopefully interesting discussion with people that are somehow involved in technology or in business. And what better way to celebrate a milestone like 50 episodes for a Drupal company's podcast then by talking to the founder of Drupal himself. Dries Buytaert has a PhD in computer science and engineering and founded the Drupal project while at university. He's an open source software developer, a startup founder and currently serves as a technology executive, all in addition to being the Benevolent Dictator for Life of the Drupal project. He's a father, a traveler and a hobbyist photographer. He's won numerous awards like Entrepreneur of the Year and Fastest Growing Tech Company for Acquia. I'm honored to welcome to the TEN7 Podcast, Doctor Dries Buytaert, the creator of Drupal. Welcome Dries.

DRIES BUYTAERT: Well thank you. Thanks for having me and congratulations with 50 episodes, that's an exciting milestone.

IVAN: Yes, it is. Thank you. Thank you for the congratulations. You know, 10 years ago I didn't think we'd be here doing a podcast for a Drupal company, and 11 years ago I didn't even consider that I was starting a company that was based on Drupal.

DRIES: Well, neither did I, 18 years ago. (laughing) Makes two of us.

IVAN: (laughing) Right. Well congratulations, I guess, for starting Drupal. Do you know when the anniversary of that is?

DRIES: Starting Drupal, it's a little bit blurry, but I do know that we released, or that I released because I was alone at the time, Drupal 1.0.0, was on January 15th of, I think it was 2001. So, it’s almost 18 years ago.

IVAN: Oh, wow. Yes, that is almost 18 years ago, but that was the first release.

DRIES: That was the first release, yea.

IVAN: You were likely working on that even before that, from 2000, so that’s a long time. Well, congratulations on that. That's coming up very soon here.

DRIES: It is, yea.

IVAN: So, Dries is a contraction of the name Andries which is Andrew in English. Is Andries actually your given name, or is it Dries?

DRIES: It's Dries, yeah, which is effectively as you said, Drew in English. I don't know if people know that, but yeah, Dries is my given name. I don't have any middle names, so it's just you know Dries Buytaert.

IVAN: Dries Buytaert.

DRIES: That's it. Yeah.

IVAN: That's great. I don't have a middle name either, so, I just recently had a shirt monogrammed and it says IS on it. (laughing) So, you I guess, would be database, right?

DRIES: That's right. Yeah. (laughing)

IVAN: (laughing) So, you were born in Antwerp in Belgium in the late seventies. Do you have a favorite thing you remember doing as a child?

DRIES: I’m not sure I have one favorite thing, but I did a lot of different interests. Like, when I was young, I loved Legos, when I was older, I really got into building remote controlled airplanes, actually for some time. And then I got into writing software when I was about 10 or 11 years old. And then later in high school, I loved playing tennis. I think I almost played tennis every day in the summer. There's sort of ebbs and flows in my interest, and I think that's still true today, I have a wide variety of different interests.

IVAN: I can identify with all of those. Are you still playing tennis?

DRIES: I am.

IVAN: You live in Boston, right?

DRIES: I do live in Boston now and I still try to play tennis. I'm probably playing two or three times a month. If I could I would play multiple times a week. But between a lot of travel, I'm on the road quite a bit, it's kind of hard. Tennis is not a very portable sport, because you need to bring gear, and you need somebody to play with, and you often need to be a member of a club too. So, it's not something I can do when I travel. So, I don't get to play as much as I'd like to, but I still try to, as I said, play two or three times a month.

IVAN: Do you know how you're rated. Are you a 4.0 player, or a 5, or a 3.5?

DRIES: I'm a decent player, but again, because of the travel I can’t really play competitively. I have a tennis coach that I play with, and that works well for me, because I only play three times a month, and it can really kind of give me a good workout every time I play. So for me it's more about the workout, and I love the game of tennis too, but I don’t know my rating.

IVAN: I ask because I absolutely love playing tennis, as well, but I can't stand the competitive games and playing sets of tennis. So, I do it as a workout as well. And they usually have groups of players that are similarly rated, and so that's why I was curious. You're a tall guy, so you must love to serve and volley.

DRIES: That's right. Pretty good serve.

IVAN: So, you mentioned you loved Legos and remote-control planes. Do you remember your first computing experience as a kid?

DRIES: Yeah. My dad had a Commodore 64, and one evening or day he bought me a couple of computer books, and these were computer books for kids. So programming for kids and had little code snippets, which you had to enter sort of in the computer and they would execute and be a game. And one of the games was like Snail Race or something and you had to basically pick a number between one and five, and randomly they would move like a character across the screen. And so they would randomly pick a character, and that one would move up a position, but if the character reached the other end of the screen that one would win. It was little stuff like that, it was probably like 20 lines of code. I remember as a kid entering 20 lines of basic code, and it was a lot of fun. You start by literally copy/pasting these things, so to speak, and then over time you dabbled around a little bit and started to try some things on your own. And then I remember in, I’m going to say it was 1994, or something, so my Dad is a doctor, and at the time, a lot of patient information and stuff was managed on paper and he came to me and said, “how do you feel about spending some of your summer vacation trying to digitize some of this paperwork and build me a database, effectively for his patients?" I said yes. He gave me some other computer books using Dbase as a database and using, I think it was Clipper, as a programming language. And I remember spending my entire summer building this patient management system for him. And it was hard for me because, one, I didn't have any training in computer programming, and English wasn't something that I was very good at. It was not even my second language. So, I remember sitting there with a dictionary trying to translate the manual effectively, of the Clipper programming language and trying to make sense of it all. I remember reading things like array, and I had no idea what arrays were. I spent my whole summer trying to figure it out. But I got it to work, which was a major milestone for me, and I would say probably what got me hooked on computer science.

IVAN: And did your dad's practice use the software then, and did it evolve? What happened to that?

DRIES: He used some of it, and then eventually he bought something off the shelf. So, I guess it wasn't a huge success from that point of view, like it literally transformed his business, but he used some of it for a while. The part that he used was like a graphing tool, like you would measure…he’s a gynecologist and an oncologist, so he would measure basically, the baby in the belly, and he would enter the measurements in the software and then the software would plot it on a graph. And this was something that he would have to do manually. He had these sheets of paper and then he would plot the measurements on the graph and then he could tell whether the baby was growing well or not. So, that was very manual and tedious. And so that part he used the graphing portion that I built.

IVAN: And, do you think back to that copy/paste as you described it, which is really reading code from a manual and typing it into your program on that Commodore 64. Do you consider that kind of the beginning of open source software? Because that was actually code that was published for anyone to use and to modify effectively.

DRIES: Yeah that's a good point. I never thought about it, but that’s a form of open source I would say. I don't know if it came with a formal open source license. (laughing) But, yeah, there’s a lot of power in books and enabling people to learn through sharing code. Even when I got into web development, copy/paste was a powerful tool. You would browse around websites, and then you saw something that you liked, you could from within the browser, you could do a view source, and you could learn about how that website was built, and you could reuse some of that code if you wanted to, or kind of use it as a starting point to write your own. I think a lot of web development still happens that way. Maybe not most of it, but at least some of it.

IVAN: And, I think a lot of people who are learning are doing that, and it's less view source now than it is inspect element and using the wonderful tools that these browsers have now. So, that's made it easier for us, I think.

DRIES: Yeah, it would be a shame if we lost that, I would say.

IVAN: I agree. I absolutely agree. So, you were raised in Antwerp, and you got a master's degree in computer science from the University in Antwerp. And, after you graduated from that degree, you worked on something called Wonka VM for a company called Acunia, which sounds an awful lot like Acquia. (laughing)

DRIES: Yeah, that was my first real job, so to speak, as an as a software engineer out of college.

IVAN: And it was Java centric. Correct?

DRIES: Java centric. So the company was a Telematics company. We built hardware and software that would be installed in cars, and that would effectively provide navigation software or other kinds of software capabilities. And I did a couple of things. I helped build a real time operating system, first from scratch actually, and later ported Linux to this hardware platform that we designed. And then once we got the operating system running we decided to build our own Java virtual machine, because the one that Sun had built wouldn't run effectively on embedded software, on a little hardware platform. So, we had to build our own virtual machine, and I was part of the virtual machine group. It was a small group, I think we were four or five people. And so, it was Java, but I was actually writing C code, because the operating system and the kernel and then also our Java virtual machine implementation were all written in C, because we needed to have that raw speed, because we had a very lightweight embedded hardware device effectively. But we did make that virtual machine open source.

IVAN: How did your philosophy and your ideas about open source influence that license that you used to release Wonka VM on? Because, according to my research, you basically started Drupal right around the time that you kind of ended your work at the University of Antwerp for your master's degree. So, there must have been some influence there or some perspective that influenced the Wonka VM.

DRIES: Yeah, so, I was an engineer on the Wonka VM team. I wasn't the team lead, but, I started contributing to the Linux kernel, and I helped a little bit with wireless network drivers of Linux and ended up being a maintainer of the documentation and stuff like that. And so, I got really into Linux. Then I started the Drupal project, and because I kind of came from the Linux world, it just was very natural for me to make Drupal open source. In fact, I literally copied another example of copy/paste I guess, I literally copied the GPL license file from my Linux kernel tree into my website, created a zip file or a tarball and uploaded that and called it Drupal. So, that's why Drupal is GPL, because I was using Linux and contributing to Linux a little bit. And, so then I went to work for the startup, building or helping to build a Wonka VM, and there was kind of inbound interest, I would say, from other companies to also use an embedded Java virtual machine. So, open source at the time was sort of a hot topic but also very premature in other ways. But, for me, it was a natural thing to help make the Wonka VM open source. I forgot what license we picked for it, but yeah, it became open source.

IVAN: I think I read about those yesterday when I was prepping. I think it was a license that was pseudo BSD. No, I'm confusing that with the release of the open JBK implementation. Sorry about that.

DRIES: No worries. I don't recall picking the license, I think that was the CTO of the company, who was in charge of that. But I was just excited to be part of another open source team. So, open source has been a constant in my professional career. All the way from college until today.

IVAN: Do you think you still have any of your code that you wrote for the WLAN drivers or the WLAN project in the kernel?

DRIES: No, no. I don't think so. No I haven't looked, but I didn't write that much to be clear. It was small contributions on the edges, and then I did end up being the official maintainer of the documentation for a while.

IVAN: Follow up question, just since we're on the topic of old code. Is any of your code still in Drupal 8 from the very first release? Or has that all disappeared?

DRIES: That’s a great question. Probably all disappeared. From the original release?

IVAN: Yeah, from the original, it would be interesting to see that.

DRIES: I would say a lot of the concepts have survived. Not all of them were in the initial release but, like we shipped Drupal 1 with RSS feeds, which at the time was groundbreaking so to speak. These things are still in Drupal, and then the hook system and the node system, and there is a lot of things that have survived the many major Drupal releases, but I don't know if any code has survived from the original Drupal 1.

IVAN: That would be interesting to know. So, you spend a few years working as a Java engineer, essentially, right? And, you decide to go and get your PhD, and you spend about five years doing that. And your work for your doctorate is focused on Java, and you're working with the inventor of Java. And at the same time you're working on Drupal and basically Drupal is in its infancy, and then maybe in its angry teen years as well. How do you juggle doing these two things that, I'm sure, are taking up so much of your time?

DRIES: Yeah, I mean I think I've always worked hard. During the day I would work on my PhD, and that was pretty intense. And then at night and on the weekends, I would work on Drupal. So I would use every spare minute of my time really to work on Drupal. I mean, it was exciting for me, and it still as exciting, it’s still very passionate and I'm very passionate about Drupal. It was definitely an interesting time because, as I was doing my PhD, Drupal also started to slowly take off. We had multiple tipping points like Howard Dean started using Drupal, and first Drupal companies were being started. And we decided to start organizing DrupalCon events, and all of that was intense. I mean I remember we would do DrupalCon and I needed to travel for them, but as a PhD student I was being paid by the University, or I guess by the government technically, but I was making like $1500 a month. In the context of Belgium at the time wasn't a whole lot of money, $1500. But like we would be like, “hey, let's do a Drupal conference in San Francisco in Sunnyvale” and I'm like, “sure, let's do that.” But the ticket alone was basically my entire salary for the month. I remember people started telling me you should really get a trademark on the name Drupal. I went to a lawyer and said, “what does it take to get the word Drupal trademarked? And, I think they came back and said, “something like $10,000.” I was like, “whoa, I don’t have $10,000.” So, I decided to do it myself. So, I spent a couple nights or maybe a couple weeks researching how you file a trademark application and how you file it in multiple countries too. So, I still had to pay filing fees, and I ended up paying $3,000 or euros or something just to get the trademarks. But again, $3,000 was two months of my salary. So, I remember by the time I finished my PhD, I had like 400 EUR in my bank account, and I spent it all on Drupal. And so, not only did I spend all of my free time, I also spent a lot of my savings, the savings that I got from Acunia, the startup that we talked about. So I was very passionate about that. I was very busy, periods. I was working easily 12-hour days and also weekends.

IVAN: You mentioned Howard Dean as being maybe one of the tipping points for Drupal. When did you realize that this thing that you made wasn't a pet project anymore? That it was affecting people's lives and that there were some legs to this thing?

DRIES: Yeah, it happened kind of in phases, to be honest. But I remember when I was doing my PhD, I'll give you one concrete example, but MTV had decided to switch to Drupal, and I remembered their sites came down crashing, and I took that very personal. For me, it was very important that these organizations, like MTV which was a very big deal to me, especially at the time. I mean it's hard to believe even like at the time like MTV is using Drupal like my little hobby project, that I would volunteer to spend time with them on the phone at night. And so I would come home from work, from my PhD research, and I would spend my spare time, free of charge, on the phone with MTV trying to troubleshoot their performance and scalability issues. That was important because I wanted them to be successful, because I knew if they were successful, it would be incredible references for others. But at times like that, it's when I realize you know this is for real. These are real companies using Drupal in real ways, and we need to make sure that these kinds of organizations are going to succeed. It's also really when, sort of, the first seeds were planted for Acquia, because I was convinced at the time that for Drupal to succeed, it needed to succeed with these larger organizations that would be incredible brands and references for us. And so, I also realized this is not something I can do in my spare time at night, like there needs to be a company that helps these organizations be successful.

IVAN: You co-founded Acquia correct?

DRIES: Yeah.

IVAN: And that was in about 2007, I think?

DRIES: Yeah, that's right.

IVAN: And when did you complete your PhD? Was that about the same time?

DRIES: Technically I completed my PhD in 2008. So I started Acquia while I was still finishing my PhD. So I’d done all of the research at the time, or almost all of the research, but I still needed to write my dissertation, like your between quotes book (laughing), that summarizes the results of your research. That was a very crazy time because we were working on a major release of Drupal, I was finishing my PhD which was a lot of work, I also decided to co-found Acquia. And our oldest son was born at the time as well. So, I was like juggling a lot of things (laughing). I officially incorporated Acquia in the summer of 2007. So about 8 months before finishing my PhD, I would say. So, that's when we incorporated Acquia, so the idea must have been born several months before. So, I guess Acquia was kind of born about a year before finishing my PhD.

IVAN: That was around the same time that I started TEN7 and was deciding which CMS I was going to hitch my company to. And if I remember correctly, that was around the time that Drupal 4.7 was stable, and I think 5 was going to be coming out very soon.

DRIES: I believe that's right.

IVAN: That's what I can remember. I didn’t look that up yet.

DRIES: I think you're right. Yes.

IVAN: So, at what point did you decide you were going to call the United States your home? Because, you started Acquia, but you decided to make Boston your home just after that, right? I didn't think you were here yet?

DRIES: Right. So, because I was still finishing my PhD I had to be at the university, and then as I mentioned my oldest son was born, and so it wasn't a good time to move. But I decided to cofound Acquia in Boston for a couple of reasons. One, the person that I met, Jay Batson, as well as our first investor, Michael Scott, from at the time North Bridge Venture Partners, I think was their official name. They were both based in Boston, and the vision for Acquia at the time was to be to Drupal what Red Hat was to Linux. We would provide enterprise grade support, SLA based support and a couple of products and services around that. So, as we've established, Drupal predated Acquia by seven years. So Drupal already existed. The original idea of sort of being the Red Hat for Drupal, is a very support intensive business model, a lot of human touch, if you will, and you kind of want to put the company close to where the largest customers are, because you need to interact with them, you need to be on the phone with them, all of these things. So, it made a lot of sense for me to put the company in the US broadly. And then because I met Jay and Michael, my co-founder and our first investor, given that they were based in Boston, that was a very logical choice obviously. Boston is a great city, because obviously there is a lot of venture capital there, a lot of access to money for starting companies. It's the second largest technology city in the United States after San Francisco. There's also great access to talent with universities like MIT and Harvard, and so it wasn't too far from Belgium. It was only a six-hour time zone difference, and I think about a six-hour flight. And so, Boston was a great place, and it allowed me to stay in Belgium for the time being, while we were trying to get Acquia going. And I was a young dad, I guess, finishing my PhD, and then, I did a lot of, sort of, one week in Belgium, one week in Boston, one week in Belgium, kind of back and forth kind of travel. And then eventually a couple years later kind of moved permanently.

IVAN: And now you call Boston your home?

DRIES: Yeah, exactly. It's officially my home and it feels like my home. I still spend a good amount of time in Europe, but, yeah, I love being in Boston, it's a great place.

IVAN: I read somewhere, I think it might have been on your website, that you travel about a quarter of a million kilometers every year. That’s a lot of travel.

DRIES: It's a lot of travel, yeah. I don't know. Yes, that's a lot of travel, it's like what, eight times around the world or something a year?

IVAN: Something like that. So you must get a lot of miles and a lot of rewards on a credit card?

DRIES: I get a lot of miles but not a lot of rewards, actually. I think people overrate these programs and they also romanticize what that amount of travel looks like.

IVAN: It’s tough.

DRIES: It’s very tough. It’s in and out. I don’t get to sightsee. I travel economy class, and I still do. I also get a lot of energy from meeting Drupal users and Acquia customers. It’s tough, but for me, also fun.

IVAN: Well, you'll be visiting Minneapolis in 2020 with DrupalCon here. (laughing) So, I hope you get a chance to sightsee here.

DRIES: You know, DrupalCons, I usually do, because I'm usually there for a whole week, so that allows me to explore a little bit more of the city. But sometimes my travel, I'll fly to the west coast which is about a 7-hour flight, I'll be there for an afternoon and fly back that night.

IVAN: That’s tough.

DRIES: Yeah. But DrupalCons are nice because it's a whole week and there's a lot of social activities at night. It's fun.

IVAN: It is fun. It really is. So, you're one of the 30 or so Benevolent Dictators for Life. Linus Torvalds for Linux is one of them. Mark Shuttleworth for Unbutu, DHH for Ruby on Rails, and you’re effectively the final say, right? In any dispute or argument you’re basically the dad of the Drupal project. (laughing)

DRIES: Yeah. So I guess, yeah.

IVAN: So to me, it’s almost incongruent, or at least it causes some sort of cognitive dissonance in my brain, when there’s a BDFL for a project that is effectively designed to be collaborative and transparent and open. And I read some opinions online about why open source projects end up having this kind of dictatorship. And I don't like the word dictatorship, these are other people's words, but it's kind of the descriptor. How do you see your responsibility for this beautiful thing you've created evolving?

DRIES: Yeah. First of all, I don't love the term BDFL either. It’s a title that's been given to me, not something that I kind of picked myself, just to be clear.

IVAN: Exactly. Of course.

DRIES: I think my role has evolved a lot over time. I mean, in the early days I would write 100% of the code, and I would spend a lot of my time building Drupal.org. I would help run the servers behind Drupal.org. I would organize the DrupalCon events or help organize them, like intensively. And over time I’ve scaled more and more. Drupal Association would be one example of that, as a step in evolving my role, which put in place an entity, a non-profit entity specifically, that could take over the organization of DrupalCon which now is, it's a serious event. It costs a few million dollars to put on and takes a whole team of people to organize. Same thing with managing our website and the underlying hardware infrastructure. It's now being managed professionally by people at the Drupal Association and again, also with the help of people in the community, just like DrupalCon. But these are examples of how I've scaled my role. Obviously on the technical side, I went from being the, sort of single core committer, to now having teams of core committers for each of the major releases, having committees and task forces around different aspects of the project, like a technical working group that defines coding standards. We have release managers and product managers and framework managers, all these kinds of roles to subsystem maintainers that are responsible for different aspects of Drupal core. And so, these are all examples of me scaling my role over time, and we continue to make governance changes all the time and to scale the project as needed. I think that’s the right thing to do. As projects or organizations get bigger, you need to put the kind of organizational structure in place. You also need to scale the culture of the project and so, I try to help with that through my keynotes. Actually, last year this time, I helped write Drupal’s Values and Principles document, that's a way to help scale our culture. So, it takes a lot of effort and different people to maintain and run the Drupal project today.

IVAN: It sure does. Do you spend time thinking about the kind of the ethical implications of the technology that you've created and that you've helped create?

DRIES: What do you mean by that? You have an example?

IVAN: I guess anything can be used for good or bad things, right? And so, there's some sort of ethical implications there. And this little project you created in, I would assume a dorm room, but somewhere in a university somewhere when you released it, it's had a profound effect on the world. I mean, as you’ve written yourself, 2% of the world's websites use Drupal. And I wonder about how you think about not just the positive, but the negative effects that the software has. And how does that affect you?

DRIES: I think Drupal is primarily used for good, right. There are tens of thousands of nonprofits using Drupal to accelerate their mission, whatever their mission is. There's a lot of governments using Drupal, which technically they're doing for good. Drupal has done a lot of good things. Obviously, there is also less good, sort of, implementations of Drupal. I guess it's hard to control. But I take pride in that we have actually made a lot of people's lives better with Drupal. I believe that drastically outweighs some of the negative impact that Drupal has had, as well. I remember at one point, this is now probably 8 years ago, two FBI agents showed up at Acquia, in black suits and were like, “where’s Dries, and does he talk to me?”

IVAN: (laughing) Dries is not here right now, can you leave a message please.

DRIES: Exactly. That's exactly what they told them, because I wasn't there. But they immediately called me and said, “hey, two FBI agents showed up.” I’m like ”whoa, what did I do?” And, they left their phone number. I called them, and there was a site, and I don't know which site or what, but obviously there was something illegal on the site, and they thought it was my site. And they looked at the site, they saw it was Drupal, they found my name, and they thought I was the site owner and developer of the site. Obviously, I had nothing to do with the site. And to this day I don't know which site it was. But I remember educating the FBI about open source and how it all works and why it's not my site. But that would’ve been an example of a site, obviously, that probably did not have a good impact on the world. And it's troubling, obviously, but at the same time I'm not sure what to do about that. So, I focus on how we make things better, and to me that's a huge motivator. The fact that Drupal is now used by 1 out of 30 sites in the world, and if you look at some of the larger sites, I think that number gets closer to 1 out of 10. So, if you think about it, what that means is, everyone uses Drupal as a visitor of the web. As a user of the web, you’re gonna hit a Drupal site. It's hard to not hit a Drupal site. Everybody visits at least 30 websites, I imagine. And so, statistically, you're going to hit a Drupal site. What’s encouraging and powerful to me is that when you make a change to Drupal, when you make it a little bit better, let’s say you make it a little bit more accessible, all of a sudden that touches everybody, everybody that uses the web. So, people sometimes complain that it's hard to make changes in Drupal, because we’re so big, and there’s a lot of governance around it. But at the same time, if your contribution makes it in, there's a good chance that it touches billions of people, which that's incredibly encouraging and rewarding to me.

IVAN: I can't even imagine how rewarding it is to you. I can identify with that, I'm sure it's a small portion of how you feel, and the clients that we help with being a Drupal focused agency and just using Drupal. It does feel good to be able to touch people and to improve things as new versions come out. I have to thank you again for creating Drupal, like this company wouldn't exist. I wouldn't be doing what I love and it's awesome. I think I have two questions and we'll be able to wrap it up. I love your Driesnotes. I love them because you’re always talking about what's coming next and what you think we should be focusing on in the future. Not just from a technical point of view, but from the people that make and use Drupal as well. And I'm curious to know, besides the new features and the attention to the user experience, what are your hopes and dreams for the Drupal community itself over the next few years? And, how do you see yourself facilitating that?

DRIES: Yeah, it's a big question. I don't know, I have a lot of thoughts on how to answer this question. I'm not sure if I have a crisp answer. But first of all, I'm very passionate about making Drupal easier to use for the day-to-day users of Drupal. Like the content creators and the marketers and the typically less technical people, I think it's really important. That was less important when I started Drupal but today that's very important and so I'm very passionate about that, because I think it's incredibly empowering for these people. So, we're doing a great job at that, actually. We're working on Layout Builder, we're working on Media, we're working on a whole bunch of things that will kind of make Drupal easier to use for non-developers. So, that’s super exciting. And I'm very happy that the community has been rallying around that right now. I feel like I’ve been talking about this in my Dries Notes actually for many years, and that wasn't always well-received. At least in the early days it wasn't universally well-received, and that is something that we needed to do. So, I think that cultural change has been a little bit slower than I would have liked to, but I feel like we're finally there. And so, that's pretty exciting to me, and that's exactly what needs to happen. And if you combine that with some of the innovation that we're working on around headless Drupal or decoupled Drupal, the API first initiative, where we're focused on the Rest API and the JSON API, that really kind of propels Drupal into new opportunities, where we're kind of moving beyond just supporting traditional websites, but where we can push content into mobile applications and digital kiosks and even drive voice assistance like Siri or Alexa, push content into augmented reality applications, and all that kind of stuff. I think that's incredibly exciting to me as well. I'm very excited that Drupal communities are also embracing that. I guess the last part of your question was “how do I see myself facilitating that?” Is that right?

IVAN: Yes.

DRIES: For me, I try to think about, I like to optimize what I do for impact. Like I love programming, but I don't do a lot of programming in Drupal because I don't feel it maximizes what I can do for the project. So often, what I do do, is I try to help the broad vision of where I think we should go and try to evangelize that and try to organize groups of people around the different pieces that make up that vision. It’s like, I try to plant the flag, and in my last Driesnote, I could show that image with a flag, and then all of the different initiatives that help us get from A to B. How do we actually get to the flag. So, we need to do all these different things. So, I like to kind of track those things and make sure that people are able to move these initiatives forward, so that the combined progress across all of the initiatives helps Drupal succeed. I also try and spend time unblocking people or empowering people to do things. I try to look after the sustainability of the project. I spend a good amount of my time working with the Drupal Association to make sure that the Drupal Association is well funded, that the DrupalCon events happen, because I believe in bringing people together to build in-person relationships, not just relationships on Slack or issue cues. So, I don't know, I do a lot of different things, the things that I feel will help move Drupal forward. A lot of these things are now a little bit more in the background than they used to be, funny enough, or less in the issue cues than the developer spheres but more around governance, strategy work, that kind of stuff. Long-winded answer.

IVAN: But, a very important answer. I appreciate everything you do for the community and for starting the project and for continuing to shepherd the organization and the direction of the project.

DRIES: I do my best, but it's truly the work of hundreds, if not thousands of people. A lot of people do so much for Drupal. And in many ways my contribution now is just like anyone else's contribution, I contribute a small piece of the bigger collective effort.

IVAN: Dries, thank you so very much for spending your time with me today. I really appreciate it. Would you consider coming back in the future at some point?

DRIES: I will.

IVAN: Maybe on the 100th episode. (laughing)

DRIES: Let's do it. (laughing) Well, thanks for the opportunity and congratulations again.

IVAN: Thank you very much. Dries can be found online at dri.es, that's dries with a dot between the "i" and the "e," where he publishes on a regular basis and syndicates it elsewhere. He is @dries and on Drupal.org where he is also user Number One. We'll have that in the transcript online with links. You’ve been listening to the TEN7 Podcast. Find us online at ten7.com/podcast. And if you have a second, do send us a message. We love hearing from you. Our email address is [email protected]. Until next time, this is Ivan Stegic. Thank you for listening.

Jan 02 2019
Jan 02

Your browser does not support the audio element. TEN7-Podcast-Ep-050-Dries-Buytaert.mp3

In this, our 50th episode, Ivan is joined by Dries Buytaert, the founder of Drupal, an open source software developer, a startup founder, technology executive, father, world traveler and photographer. Subscribe to the podcast.

Here's what we're discussing in this episode:

  • Dries' life and career
  • The creation and emergence of Drupal
  • Growing up in Antwerp
  • Living in Boston
  • Shared love for tennis
  • Yet another flying start with the Commodore 64
  • The power of copy/paste
  • Writing code for his father's medical practice
  • Wonka VM, Linux and Java
  • Juggling multiple careers particularly fatherhood
  • DrupalCons
  • The MTV saga
  • TEN7's decision to go Drupal
  • The move from Belgium to Boston
  • Circumventing the globe
  • Benevolent Dictators for Life
  • Creating thousands of roles out of one
  • Drupal's Values and Principles
  • Drupal as a force for good in the world
  • Drupal's future and Dries' role moving ahead

TRANSCRIPT

IVAN STEGIC: Hey everyone, you're listening to the TEN7 Podcast, where we get together every fortnight, and sometimes more often, to talk about technology, business and the humans in it. I'm your host Ivan Stegic. Well, we've hit 50 episodes. When we started this podcast in April of 2017, we did so by limiting it to only five minutes a time and by calling it an audiocast. And we did that because we wanted to experiment with some marketing that we really weren't familiar with and that we hadn't tried before. And I, of course, thought the best way to do this is to try something short and sweet, that's a quick blast of audio. Because who has time to listen to more than that? Well that didn't last terribly long before we realized that five minutes simply wasn't enough time to tell a story, or to get into the meat of an idea. I guess that's obvious now, in retrospect of course. So I'm glad that this podcast has evolved into what it is today, a hopefully interesting discussion with people that are somehow involved in technology or in business. And what better way to celebrate a milestone like 50 episodes for a Drupal company's podcast then by talking to the founder of Drupal himself. Dries Buytaert has a PhD in computer science and engineering and founded the Drupal project while at university. He's an open source software developer, a startup founder and currently serves as a technology executive, all in addition to being the Benevolent Dictator for Life of the Drupal project. He's a father, a traveler and a hobbyist photographer. He's won numerous awards like Entrepreneur of the Year and Fastest Growing Tech Company for Acquia. I'm honored to welcome to the TEN7 Podcast, Doctor Dries Buytaert, the creator of Drupal. Welcome Dries.

DRIES BUYTAERT: Well thank you. Thanks for having me and congratulations with 50 episodes, that's an exciting milestone.

IVAN: Yes, it is. Thank you. Thank you for the congratulations. You know, 10 years ago I didn't think we'd be here doing a podcast for a Drupal company, and 11 years ago I didn't even consider that I was starting a company that was based on Drupal.

DRIES: Well, neither did I, 18 years ago. (laughing) Makes two of us.

IVAN: (laughing) Right. Well congratulations, I guess, for starting Drupal. Do you know when the anniversary of that is?

DRIES: Starting Drupal, it's a little bit blurry, but I do know that we released, or that I released because I was alone at the time, Drupal 1.0.0, was on January 15th of, I think it was 2001. So, it’s almost 18 years ago.

IVAN: Oh, wow. Yes, that is almost 18 years ago, but that was the first release.

DRIES: That was the first release, yea.

IVAN: You were likely working on that even before that, from 2000, so that’s a long time. Well, congratulations on that. That's coming up very soon here.

DRIES: It is, yea.

IVAN: So, Dries is a contraction of the name Andries which is Andrew in English. Is Andries actually your given name, or is it Dries?

DRIES: It's Dries, yeah, which is effectively as you said, Drew in English. I don't know if people know that, but yeah, Dries is my given name. I don't have any middle names, so it's just you know Dries Buytaert.

IVAN: Dries Buytaert.

DRIES: That's it. Yeah.

IVAN: That's great. I don't have a middle name either, so, I just recently had a shirt monogrammed and it says IS on it. (laughing) So, you I guess, would be database, right?

DRIES: That's right. Yeah. (laughing)

IVAN: (laughing) So, you were born in Antwerp in Belgium in the late seventies. Do you have a favorite thing you remember doing as a child?

DRIES: I’m not sure I have one favorite thing, but I did a lot of different interests. Like, when I was young, I loved Legos, when I was older, I really got into building remote controlled airplanes, actually for some time. And then I got into writing software when I was about 10 or 11 years old. And then later in high school, I loved playing tennis. I think I almost played tennis every day in the summer. There's sort of ebbs and flows in my interest, and I think that's still true today, I have a wide variety of different interests.

IVAN: I can identify with all of those. Are you still playing tennis?

DRIES: I am.

IVAN: You live in Boston, right?

DRIES: I do live in Boston now and I still try to play tennis. I'm probably playing two or three times a month. If I could I would play multiple times a week. But between a lot of travel, I'm on the road quite a bit, it's kind of hard. Tennis is not a very portable sport, because you need to bring gear, and you need somebody to play with, and you often need to be a member of a club too. So, it's not something I can do when I travel. So, I don't get to play as much as I'd like to, but I still try to, as I said, play two or three times a month.

IVAN: Do you know how you're rated. Are you a 4.0 player, or a 5, or a 3.5?

DRIES: I'm a decent player, but again, because of the travel I can’t really play competitively. I have a tennis coach that I play with, and that works well for me, because I only play three times a month, and it can really kind of give me a good workout every time I play. So for me it's more about the workout, and I love the game of tennis too, but I don’t know my rating.

IVAN: I ask because I absolutely love playing tennis, as well, but I can't stand the competitive games and playing sets of tennis. So, I do it as a workout as well. And they usually have groups of players that are similarly rated, and so that's why I was curious. You're a tall guy, so you must love to serve and volley.

DRIES: That's right. Pretty good serve.

IVAN: So, you mentioned you loved Legos and remote-control planes. Do you remember your first computing experience as a kid?

DRIES: Yeah. My dad had a Commodore 64, and one evening or day he bought me a couple of computer books, and these were computer books for kids. So programming for kids and had little code snippets, which you had to enter sort of in the computer and they would execute and be a game. And one of the games was like Snail Race or something and you had to basically pick a number between one and five, and randomly they would move like a character across the screen. And so they would randomly pick a character, and that one would move up a position, but if the character reached the other end of the screen that one would win. It was little stuff like that, it was probably like 20 lines of code. I remember as a kid entering 20 lines of basic code, and it was a lot of fun. You start by literally copy/pasting these things, so to speak, and then over time you dabbled around a little bit and started to try some things on your own. And then I remember in, I’m going to say it was 1994, or something, so my Dad is a doctor, and at the time, a lot of patient information and stuff was managed on paper and he came to me and said, “how do you feel about spending some of your summer vacation trying to digitize some of this paperwork and build me a database, effectively for his patients?" I said yes. He gave me some other computer books using Dbase as a database and using, I think it was Clipper, as a programming language. And I remember spending my entire summer building this patient management system for him. And it was hard for me because, one, I didn't have any training in computer programming, and English wasn't something that I was very good at. It was not even my second language. So, I remember sitting there with a dictionary trying to translate the manual effectively, of the Clipper programming language and trying to make sense of it all. I remember reading things like array, and I had no idea what arrays were. I spent my whole summer trying to figure it out. But I got it to work, which was a major milestone for me, and I would say probably what got me hooked on computer science.

IVAN: And did your dad's practice use the software then, and did it evolve? What happened to that?

DRIES: He used some of it, and then eventually he bought something off the shelf. So, I guess it wasn't a huge success from that point of view, like it literally transformed his business, but he used some of it for a while. The part that he used was like a graphing tool, like you would measure…he’s a gynecologist and an oncologist, so he would measure basically, the baby in the belly, and he would enter the measurements in the software and then the software would plot it on a graph. And this was something that he would have to do manually. He had these sheets of paper and then he would plot the measurements on the graph and then he could tell whether the baby was growing well or not. So, that was very manual and tedious. And so that part he used the graphing portion that I built.

IVAN: And, do you think back to that copy/paste as you described it, which is really reading code from a manual and typing it into your program on that Commodore 64. Do you consider that kind of the beginning of open source software? Because that was actually code that was published for anyone to use and to modify effectively.

DRIES: Yeah that's a good point. I never thought about it, but that’s a form of open source I would say. I don't know if it came with a formal open source license. (laughing) But, yeah, there’s a lot of power in books and enabling people to learn through sharing code. Even when I got into web development, copy/paste was a powerful tool. You would browse around websites, and then you saw something that you liked, you could from within the browser, you could do a view source, and you could learn about how that website was built, and you could reuse some of that code if you wanted to, or kind of use it as a starting point to write your own. I think a lot of web development still happens that way. Maybe not most of it, but at least some of it.

IVAN: And, I think a lot of people who are learning are doing that, and it's less view source now than it is inspect element and using the wonderful tools that these browsers have now. So, that's made it easier for us, I think.

DRIES: Yeah, it would be a shame if we lost that, I would say.

IVAN: I agree. I absolutely agree. So, you were raised in Antwerp, and you got a master's degree in computer science from the University in Antwerp. And, after you graduated from that degree, you worked on something called Wonka VM for a company called Acunia, which sounds an awful lot like Acquia. (laughing)

DRIES: Yeah, that was my first real job, so to speak, as an as a software engineer out of college.

IVAN: And it was Java centric. Correct?

DRIES: Java centric. So the company was a Telematics company. We built hardware and software that would be installed in cars, and that would effectively provide navigation software or other kinds of software capabilities. And I did a couple of things. I helped build a real time operating system, first from scratch actually, and later ported Linux to this hardware platform that we designed. And then once we got the operating system running we decided to build our own Java virtual machine, because the one that Sun had built wouldn't run effectively on embedded software, on a little hardware platform. So, we had to build our own virtual machine, and I was part of the virtual machine group. It was a small group, I think we were four or five people. And so, it was Java, but I was actually writing C code, because the operating system and the kernel and then also our Java virtual machine implementation were all written in C, because we needed to have that raw speed, because we had a very lightweight embedded hardware device effectively. But we did make that virtual machine open source.

IVAN: How did your philosophy and your ideas about open source influence that license that you used to release Wonka VM on? Because, according to my research, you basically started Drupal right around the time that you kind of ended your work at the University of Antwerp for your master's degree. So, there must have been some influence there or some perspective that influenced the Wonka VM.

DRIES: Yeah, so, I was an engineer on the Wonka VM team. I wasn't the team lead, but, I started contributing to the Linux kernel, and I helped a little bit with wireless network drivers of Linux and ended up being a maintainer of the documentation and stuff like that. And so, I got really into Linux. Then I started the Drupal project, and because I kind of came from the Linux world, it just was very natural for me to make Drupal open source. In fact, I literally copied another example of copy/paste I guess, I literally copied the GPL license file from my Linux kernel tree into my website, created a zip file or a tarball and uploaded that and called it Drupal. So, that's why Drupal is GPL, because I was using Linux and contributing to Linux a little bit. And, so then I went to work for the startup, building or helping to build a Wonka VM, and there was kind of inbound interest, I would say, from other companies to also use an embedded Java virtual machine. So, open source at the time was sort of a hot topic but also very premature in other ways. But, for me, it was a natural thing to help make the Wonka VM open source. I forgot what license we picked for it, but yeah, it became open source.

IVAN: I think I read about those yesterday when I was prepping. I think it was a license that was pseudo BSD. No, I'm confusing that with the release of the open JBK implementation. Sorry about that.

DRIES: No worries. I don't recall picking the license, I think that was the CTO of the company, who was in charge of that. But I was just excited to be part of another open source team. So, open source has been a constant in my professional career. All the way from college until today.

IVAN: Do you think you still have any of your code that you wrote for the WLAN drivers or the WLAN project in the kernel?

DRIES: No, no. I don't think so. No I haven't looked, but I didn't write that much to be clear. It was small contributions on the edges, and then I did end up being the official maintainer of the documentation for a while.

IVAN: Follow up question, just since we're on the topic of old code. Is any of your code still in Drupal 8 from the very first release? Or has that all disappeared?

DRIES: That’s a great question. Probably all disappeared. From the original release?

IVAN: Yeah, from the original, it would be interesting to see that.

DRIES: I would say a lot of the concepts have survived. Not all of them were in the initial release but, like we shipped Drupal 1 with RSS feeds, which at the time was groundbreaking so to speak. These things are still in Drupal, and then the hook system and the node system, and there is a lot of things that have survived the many major Drupal releases, but I don't know if any code has survived from the original Drupal 1.

IVAN: That would be interesting to know. So, you spend a few years working as a Java engineer, essentially, right? And, you decide to go and get your PhD, and you spend about five years doing that. And your work for your doctorate is focused on Java, and you're working with the inventor of Java. And at the same time you're working on Drupal and basically Drupal is in its infancy, and then maybe in its angry teen years as well. How do you juggle doing these two things that, I'm sure, are taking up so much of your time?

DRIES: Yeah, I mean I think I've always worked hard. During the day I would work on my PhD, and that was pretty intense. And then at night and on the weekends, I would work on Drupal. So I would use every spare minute of my time really to work on Drupal. I mean, it was exciting for me, and it still as exciting, it’s still very passionate and I'm very passionate about Drupal. It was definitely an interesting time because, as I was doing my PhD, Drupal also started to slowly take off. We had multiple tipping points like Howard Dean started using Drupal, and first Drupal companies were being started. And we decided to start organizing DrupalCon events, and all of that was intense. I mean I remember we would do DrupalCon and I needed to travel for them, but as a PhD student I was being paid by the University, or I guess by the government technically, but I was making like $1500 a month. In the context of Belgium at the time wasn't a whole lot of money, $1500. But like we would be like, “hey, let's do a Drupal conference in San Francisco in Sunnyvale” and I'm like, “sure, let's do that.” But the ticket alone was basically my entire salary for the month. I remember people started telling me you should really get a trademark on the name Drupal. I went to a lawyer and said, “what does it take to get the word Drupal trademarked? And, I think they came back and said, “something like $10,000.” I was like, “whoa, I don’t have $10,000.” So, I decided to do it myself. So, I spent a couple nights or maybe a couple weeks researching how you file a trademark application and how you file it in multiple countries too. So, I still had to pay filing fees, and I ended up paying $3,000 or euros or something just to get the trademarks. But again, $3,000 was two months of my salary. So, I remember by the time I finished my PhD, I had like 400 EUR in my bank account, and I spent it all on Drupal. And so, not only did I spend all of my free time, I also spent a lot of my savings, the savings that I got from Acunia, the startup that we talked about. So I was very passionate about that. I was very busy, periods. I was working easily 12-hour days and also weekends.

IVAN: You mentioned Howard Dean as being maybe one of the tipping points for Drupal. When did you realize that this thing that you made wasn't a pet project anymore? That it was affecting people's lives and that there were some legs to this thing?

DRIES: Yeah, it happened kind of in phases, to be honest. But I remember when I was doing my PhD, I'll give you one concrete example, but MTV had decided to switch to Drupal, and I remembered their sites came down crashing, and I took that very personal. For me, it was very important that these organizations, like MTV which was a very big deal to me, especially at the time. I mean it's hard to believe even like at the time like MTV is using Drupal like my little hobby project, that I would volunteer to spend time with them on the phone at night. And so I would come home from work, from my PhD research, and I would spend my spare time, free of charge, on the phone with MTV trying to troubleshoot their performance and scalability issues. That was important because I wanted them to be successful, because I knew if they were successful, it would be incredible references for others. But at times like that, it's when I realize you know this is for real. These are real companies using Drupal in real ways, and we need to make sure that these kinds of organizations are going to succeed. It's also really when, sort of, the first seeds were planted for Acquia, because I was convinced at the time that for Drupal to succeed, it needed to succeed with these larger organizations that would be incredible brands and references for us. And so, I also realized this is not something I can do in my spare time at night, like there needs to be a company that helps these organizations be successful.

IVAN: You co-founded Acquia correct?

DRIES: Yeah.

IVAN: And that was in about 2007, I think?

DRIES: Yeah, that's right.

IVAN: And when did you complete your PhD? Was that about the same time?

DRIES: Technically I completed my PhD in 2008. So I started Acquia while I was still finishing my PhD. So I’d done all of the research at the time, or almost all of the research, but I still needed to write my dissertation, like your between quotes book (laughing), that summarizes the results of your research. That was a very crazy time because we were working on a major release of Drupal, I was finishing my PhD which was a lot of work, I also decided to co-found Acquia. And our oldest son was born at the time as well. So, I was like juggling a lot of things (laughing). I officially incorporated Acquia in the summer of 2007. So about 8 months before finishing my PhD, I would say. So, that's when we incorporated Acquia, so the idea must have been born several months before. So, I guess Acquia was kind of born about a year before finishing my PhD.

IVAN: That was around the same time that I started TEN7 and was deciding which CMS I was going to hitch my company to. And if I remember correctly, that was around the time that Drupal 4.7 was stable, and I think 5 was going to be coming out very soon.

DRIES: I believe that's right.

IVAN: That's what I can remember. I didn’t look that up yet.

DRIES: I think you're right. Yes.

IVAN: So, at what point did you decide you were going to call the United States your home? Because, you started Acquia, but you decided to make Boston your home just after that, right? I didn't think you were here yet?

DRIES: Right. So, because I was still finishing my PhD I had to be at the university, and then as I mentioned my oldest son was born, and so it wasn't a good time to move. But I decided to cofound Acquia in Boston for a couple of reasons. One, the person that I met, Jay Batson, as well as our first investor, Michael Scott, from at the time North Bridge Venture Partners, I think was their official name. They were both based in Boston, and the vision for Acquia at the time was to be to Drupal what Red Hat was to Linux. We would provide enterprise grade support, SLA based support and a couple of products and services around that. So, as we've established, Drupal predated Acquia by seven years. So Drupal already existed. The original idea of sort of being the Red Hat for Drupal, is a very support intensive business model, a lot of human touch, if you will, and you kind of want to put the company close to where the largest customers are, because you need to interact with them, you need to be on the phone with them, all of these things. So, it made a lot of sense for me to put the company in the US broadly. And then because I met Jay and Michael, my co-founder and our first investor, given that they were based in Boston, that was a very logical choice obviously. Boston is a great city, because obviously there is a lot of venture capital there, a lot of access to money for starting companies. It's the second largest technology city in the United States after San Francisco. There's also great access to talent with universities like MIT and Harvard, and so it wasn't too far from Belgium. It was only a six-hour time zone difference, and I think about a six-hour flight. And so, Boston was a great place, and it allowed me to stay in Belgium for the time being, while we were trying to get Acquia going. And I was a young dad, I guess, finishing my PhD, and then, I did a lot of, sort of, one week in Belgium, one week in Boston, one week in Belgium, kind of back and forth kind of travel. And then eventually a couple years later kind of moved permanently.

IVAN: And now you call Boston your home?

DRIES: Yeah, exactly. It's officially my home and it feels like my home. I still spend a good amount of time in Europe, but, yeah, I love being in Boston, it's a great place.

IVAN: I read somewhere, I think it might have been on your website, that you travel about a quarter of a million kilometers every year. That’s a lot of travel.

DRIES: It's a lot of travel, yeah. I don't know. Yes, that's a lot of travel, it's like what, eight times around the world or something a year?

IVAN: Something like that. So you must get a lot of miles and a lot of rewards on a credit card?

DRIES: I get a lot of miles but not a lot of rewards, actually. I think people overrate these programs and they also romanticize what that amount of travel looks like.

IVAN: It’s tough.

DRIES: It’s very tough. It’s in and out. I don’t get to sightsee. I travel economy class, and I still do. I also get a lot of energy from meeting Drupal users and Acquia customers. It’s tough, but for me, also fun.

IVAN: Well, you'll be visiting Minneapolis in 2020 with DrupalCon here. (laughing) So, I hope you get a chance to sightsee here.

DRIES: You know, DrupalCons, I usually do, because I'm usually there for a whole week, so that allows me to explore a little bit more of the city. But sometimes my travel, I'll fly to the west coast which is about a 7-hour flight, I'll be there for an afternoon and fly back that night.

IVAN: That’s tough.

DRIES: Yeah. But DrupalCons are nice because it's a whole week and there's a lot of social activities at night. It's fun.

IVAN: It is fun. It really is. So, you're one of the 30 or so Benevolent Dictators for Life. Linus Torvalds for Linux is one of them. Mark Shuttleworth for Unbutu, DHH for Ruby on Rails, and you’re effectively the final say, right? In any dispute or argument you’re basically the dad of the Drupal project. (laughing)

DRIES: Yeah. So I guess, yeah.

IVAN: So to me, it’s almost incongruent, or at least it causes some sort of cognitive dissonance in my brain, when there’s a BDFL for a project that is effectively designed to be collaborative and transparent and open. And I read some opinions online about why open source projects end up having this kind of dictatorship. And I don't like the word dictatorship, these are other people's words, but it's kind of the descriptor. How do you see your responsibility for this beautiful thing you've created evolving?

DRIES: Yeah. First of all, I don't love the term BDFL either. It’s a title that's been given to me, not something that I kind of picked myself, just to be clear.

IVAN: Exactly. Of course.

DRIES: I think my role has evolved a lot over time. I mean, in the early days I would write 100% of the code, and I would spend a lot of my time building Drupal.org. I would help run the servers behind Drupal.org. I would organize the DrupalCon events or help organize them, like intensively. And over time I’ve scaled more and more. Drupal Association would be one example of that, as a step in evolving my role, which put in place an entity, a non-profit entity specifically, that could take over the organization of DrupalCon which now is, it's a serious event. It costs a few million dollars to put on and takes a whole team of people to organize. Same thing with managing our website and the underlying hardware infrastructure. It's now being managed professionally by people at the Drupal Association and again, also with the help of people in the community, just like DrupalCon. But these are examples of how I've scaled my role. Obviously on the technical side, I went from being the, sort of single core committer, to now having teams of core committers for each of the major releases, having committees and task forces around different aspects of the project, like a technical working group that defines coding standards. We have release managers and product managers and framework managers, all these kinds of roles to subsystem maintainers that are responsible for different aspects of Drupal core. And so, these are all examples of me scaling my role over time, and we continue to make governance changes all the time and to scale the project as needed. I think that’s the right thing to do. As projects or organizations get bigger, you need to put the kind of organizational structure in place. You also need to scale the culture of the project and so, I try to help with that through my keynotes. Actually, last year this time, I helped write Drupal’s Values and Principles document, that's a way to help scale our culture. So, it takes a lot of effort and different people to maintain and run the Drupal project today.

IVAN: It sure does. Do you spend time thinking about the kind of the ethical implications of the technology that you've created and that you've helped create?

DRIES: What do you mean by that? You have an example?

IVAN: I guess anything can be used for good or bad things, right? And so, there's some sort of ethical implications there. And this little project you created in, I would assume a dorm room, but somewhere in a university somewhere when you released it, it's had a profound effect on the world. I mean, as you’ve written yourself, 2% of the world's websites use Drupal. And I wonder about how you think about not just the positive, but the negative effects that the software has. And how does that affect you?

DRIES: I think Drupal is primarily used for good, right. There are tens of thousands of nonprofits using Drupal to accelerate their mission, whatever their mission is. There's a lot of governments using Drupal, which technically they're doing for good. Drupal has done a lot of good things. Obviously, there is also less good, sort of, implementations of Drupal. I guess it's hard to control. But I take pride in that we have actually made a lot of people's lives better with Drupal. I believe that drastically outweighs some of the negative impact that Drupal has had, as well. I remember at one point, this is now probably 8 years ago, two FBI agents showed up at Acquia, in black suits and were like, “where’s Dries, and does he talk to me?”

IVAN: (laughing) Dries is not here right now, can you leave a message please.

DRIES: Exactly. That's exactly what they told them, because I wasn't there. But they immediately called me and said, “hey, two FBI agents showed up.” I’m like ”whoa, what did I do?” And, they left their phone number. I called them, and there was a site, and I don't know which site or what, but obviously there was something illegal on the site, and they thought it was my site. And they looked at the site, they saw it was Drupal, they found my name, and they thought I was the site owner and developer of the site. Obviously, I had nothing to do with the site. And to this day I don't know which site it was. But I remember educating the FBI about open source and how it all works and why it's not my site. But that would’ve been an example of a site, obviously, that probably did not have a good impact on the world. And it's troubling, obviously, but at the same time I'm not sure what to do about that. So, I focus on how we make things better, and to me that's a huge motivator. The fact that Drupal is now used by 1 out of 30 sites in the world, and if you look at some of the larger sites, I think that number gets closer to 1 out of 10. So, if you think about it, what that means is, everyone uses Drupal as a visitor of the web. As a user of the web, you’re gonna hit a Drupal site. It's hard to not hit a Drupal site. Everybody visits at least 30 websites, I imagine. And so, statistically, you're going to hit a Drupal site. What’s encouraging and powerful to me is that when you make a change to Drupal, when you make it a little bit better, let’s say you make it a little bit more accessible, all of a sudden that touches everybody, everybody that uses the web. So, people sometimes complain that it's hard to make changes in Drupal, because we’re so big, and there’s a lot of governance around it. But at the same time, if your contribution makes it in, there's a good chance that it touches billions of people, which that's incredibly encouraging and rewarding to me.

IVAN: I can't even imagine how rewarding it is to you. I can identify with that, I'm sure it's a small portion of how you feel, and the clients that we help with being a Drupal focused agency and just using Drupal. It does feel good to be able to touch people and to improve things as new versions come out. I have to thank you again for creating Drupal, like this company wouldn't exist. I wouldn't be doing what I love and it's awesome. I think I have two questions and we'll be able to wrap it up. I love your Driesnotes. I love them because you’re always talking about what's coming next and what you think we should be focusing on in the future. Not just from a technical point of view, but from the people that make and use Drupal as well. And I'm curious to know, besides the new features and the attention to the user experience, what are your hopes and dreams for the Drupal community itself over the next few years? And, how do you see yourself facilitating that?

DRIES: Yeah, it's a big question. I don't know, I have a lot of thoughts on how to answer this question. I'm not sure if I have a crisp answer. But first of all, I'm very passionate about making Drupal easier to use for the day-to-day users of Drupal. Like the content creators and the marketers and the typically less technical people, I think it's really important. That was less important when I started Drupal but today that's very important and so I'm very passionate about that, because I think it's incredibly empowering for these people. So, we're doing a great job at that, actually. We're working on Layout Builder, we're working on Media, we're working on a whole bunch of things that will kind of make Drupal easier to use for non-developers. So, that’s super exciting. And I'm very happy that the community has been rallying around that right now. I feel like I’ve been talking about this in my Dries Notes actually for many years, and that wasn't always well-received. At least in the early days it wasn't universally well-received, and that is something that we needed to do. So, I think that cultural change has been a little bit slower than I would have liked to, but I feel like we're finally there. And so, that's pretty exciting to me, and that's exactly what needs to happen. And if you combine that with some of the innovation that we're working on around headless Drupal or decoupled Drupal, the API first initiative, where we're focused on the Rest API and the JSON API, that really kind of propels Drupal into new opportunities, where we're kind of moving beyond just supporting traditional websites, but where we can push content into mobile applications and digital kiosks and even drive voice assistance like Siri or Alexa, push content into augmented reality applications, and all that kind of stuff. I think that's incredibly exciting to me as well. I'm very excited that Drupal communities are also embracing that. I guess the last part of your question was “how do I see myself facilitating that?” Is that right?

IVAN: Yes.

DRIES: For me, I try to think about, I like to optimize what I do for impact. Like I love programming, but I don't do a lot of programming in Drupal because I don't feel it maximizes what I can do for the project. So often, what I do do, is I try to help the broad vision of where I think we should go and try to evangelize that and try to organize groups of people around the different pieces that make up that vision. It’s like, I try to plant the flag, and in my last Driesnote, I could show that image with a flag, and then all of the different initiatives that help us get from A to B. How do we actually get to the flag. So, we need to do all these different things. So, I like to kind of track those things and make sure that people are able to move these initiatives forward, so that the combined progress across all of the initiatives helps Drupal succeed. I also try and spend time unblocking people or empowering people to do things. I try to look after the sustainability of the project. I spend a good amount of my time working with the Drupal Association to make sure that the Drupal Association is well funded, that the DrupalCon events happen, because I believe in bringing people together to build in-person relationships, not just relationships on Slack or issue cues. So, I don't know, I do a lot of different things, the things that I feel will help move Drupal forward. A lot of these things are now a little bit more in the background than they used to be, funny enough, or less in the issue cues than the developer spheres but more around governance, strategy work, that kind of stuff. Long-winded answer.

IVAN: But, a very important answer. I appreciate everything you do for the community and for starting the project and for continuing to shepherd the organization and the direction of the project.

DRIES: I do my best, but it's truly the work of hundreds, if not thousands of people. A lot of people do so much for Drupal. And in many ways my contribution now is just like anyone else's contribution, I contribute a small piece of the bigger collective effort.

IVAN: Dries, thank you so very much for spending your time with me today. I really appreciate it. Would you consider coming back in the future at some point?

DRIES: I will.

IVAN: Maybe on the 100th episode. (laughing)

DRIES: Let's do it. (laughing) Well, thanks for the opportunity and congratulations again.

IVAN: Thank you very much. Dries can be found online at dri.es, that's dries with a dot between the "i" and the "e," where he publishes on a regular basis and syndicates it elsewhere. He is @dries and on Drupal.org where he is also user Number One. We'll have that in the transcript online with links. You’ve been listening to the TEN7 Podcast. Find us online at ten7.com/podcast. And if you have a second, do send us a message. We love hearing from you. Our email address is [email protected]. Until next time, this is Ivan Stegic. Thank you for listening.

Dec 19 2018
Dec 19

Your browser does not support the audio element. TEN7-Podcast-Ep-049-Jeff-Robbins.mp3

In this episode, Ivan is joined by his friend Jeff Robbins, entrepreneur, co-founder of Lullabot, founder of Yonder, executive coach, author, signed recording artist and self-proclaimed philosopher. Here's what we're discussing in this podcast:

  • Discovering fortnight
  • Jeff's background
  • Boston universities tour
  • Starting at O'Reilly Media
  • Gopher & open source software
  • Orbit, one of the first bands on the internet
  • Signing with A&M Records
  • Surviving record labels
  • Intellectual property, algorithms and paradigms
  • The complexity of band management
  • On the Lollapalooza tour with Snoop Doggy Dogg
  • The unintentional result of subliminal intentionality
  • 123 Astronaut
  • The transition from music to Drupal management
  • Lullabot, one of the first fully distributed companies
  • DrupalCon, Vancouver 2006, let the hiring begin
  • Yonder, guide to distribution
  • The future of corporate distribution
  • Writing a new book
  • One Minute Manager, Make Friends and Influence People and Tribal Leadership

TRANSCRIPT

IVAN STEGIC: Hey everyone you're listening to the TEN7 Podcast, where we get together every fortnight and sometimes more often, to talk about technology, business and the humans in it. I'm your host even Ivan Stegic. In this episode of the Podcast, Jeff Robbins of Yonder and 123 Astronaut. Jeff is an entrepreneur, co-founder of Lullabot, a design agency you may have heard of, and most recently the founder of Yonder, an advocate for remote work and an inspiration for leaders of distributed companies like my own, all over the world. He's a business consultant and an executive coach, an author, a signed recording artist, a self-proclaimed philosopher, and in my opinion an all-around nice guy. Jeff welcome to the Podcast.

JEFF ROBBINS: Wow. Thanks Ivan. Thanks for having me. I want to start right off with a question for you, though. What is a fortnight?

IVAN: Two weeks.

JEFF: Two weeks. Ok.

IVAN: Two weeks. It must be a British English thing. We used to talk about fortnight's all the time, when I lived in Africa.

JEFF: No, we don't use that one too much. It shows up, sort of, in archaic writing every now and then. (laughing) Or, maybe it's just British writing, I don't know. Fortnight. Okay. You know, I've heard the term, but I am embarrassed that I don't quite know what it is. But now I do. Now I will start using it, and people will feel uncomfortable around me not knowing what a fortnight. (laughing)

IVAN: (laughing) Well, people might think you're talking about the video game.

JEFF: Or, they might think I was British. (laughing)

IVAN: (laughing) Well, I'm so excited to be talking to you, Jeff. And usually I start out by asking about where life started for you. So, I know you live in Providence, Rhode Island. Is that where life started for you as well?

JEFF: Well , I grew up in Connecticut. So, in the grand scheme of things, not too far away, still southern New England. Yeah. That's where life started for me at least. I lived there through high school and then moved to Boston, again, in the grand scheme of things, not too far away. But at the time it seemed pretty significant. And lived in Boston for, as I say, most of my formative years, and then moved to Rhode Island right around 2000, something like that.

IVAN: So, that was where you went to college, was in Boston?

JEFF: Yes, I went to many colleges and universities (laughing) in the Boston area.

IVAN: (laughing) Well, I know you studied liberal arts and music, right?

JEFF: Well, yeah, I originally went there to go to Berklee College of Music, and after a semester at Berklee I realized like, there's no way I'm going to finish at this school. So I took all of the classes that I thought would be sort of important to me, actually pursuing the musical career that I wanted to pursue. I took a whole bunch of music business classes and stuff like that. Then I bounced around, I went to Emerson College for a while and took some classes at Harvard extension school, but eventually found a job doing technical illustration work, using a program called Freehand, which was sort of the competitor to Adobe Illustrator, what we now call Adobe Illustrator and eventually the companies all merged together and they all became sort of the same thing. But I started working for O’Reilly. We now call them O'Reilly Media. They were called O'Reilly and Associates in the early nineties.

IVAN: I remember that.

JEFF: Yeah. And doing technical illustrations for their books. I mean this was pre-web and started, sort of, they had really good internet there, meaning technical people themselves and writing technical books about technical things and so, I sort of discovered FTP, and a sort of lost technology called Gopher.

IVAN: Which was invented in Minnesota.

JEFF: Oh yeah, in fact, actually, many of the FTP sites, or at least one of them, was at the University of Minnesota as well, now that I think about it, where you go and download open source software and stuff like that so that you could, basically you just download other FTP programs. (laughing) Eventually you could FTP to download Gopher, and then use Gopher to download other programs. So Gopher was kind of post FTP, but pre-HTTP, and it sort of had this, sort of loose, it was like basically, folders and a little bit of hypertext linking but no graphical kind of stuff. But then the web came around, and O'Reilly was kind of first into that, and I worked on the team that ended up building the first commercial website. We were just calling it like, an online magazine at the time. We hadn't really thought about the fact that it was kind of the first commercial website, but we figured that we would probably need to support it in some way, and it being a magazine, advertising seemed like the obvious way to go.

IVAN: What quantified it as the first commercial website? Was it because it was revenue generating.

JEFF: Yeah. Up until then it was all, mostly academic colleges and universities people there, just sort of building a website about usually something scientific, or about technological things. There were a few sort of like arts websites and bands. My band was one of the first bands to be on the web. Not quite the first I don't think. Although we did build a website for our record label when we were the first record label on the web at the time. And it was like there was a website that listed new websites launched today, and it had like three listings on it.

IVAN: Back when it was a finite countable list of websites. (laughing)

JEFF: Just email Bob when you launch a website and we'll put it up on the page. (laughing) It all seemed funny at the time.

IVAN: So, you were in this band called Orbit. Were you one of the founding members?

JEFF: Yep.

IVAN: And there were three of you and you started your own record label? You just eluded to that.

JEFF: Yeah, we did. We'd been playing, the other founder of the band Paul and I, had been playing in Boston bands for years, and we were just kind of frustrated and fed up with kind of the subservient nature of playing in a band and decided, “you know what, screw this we're either going to start our own record label, we're going to go out find other bands that we think are good, we're gonna to put out our own music and stop worrying about getting a record deal and kind of trying to be what we think that bands ought to be to get a record deal.” And of course, that was exactly what record labels were looking for it turns out. So, within about six months of starting the band and starting the record label we got signed to A&M records.

IVAN: Wow. How did you know that you had a sound that you thought people wanted to hear?

JEFF: I didn't. We created the sound that I wanted to hear. Like I said, after playing in all these bands where it just felt like a, sort of compromise, and too contrived, I wanted to do something that just sort of felt authentic and kind of raw and something that I felt like I could at least just sort of be proud of. And, it was a big life lesson for me in general, in all things.

IVAN: So when you signed with the label, did that mean you were officially a full-time employee of the label, and you could tour and write and perform without really having to worry about where's my next paycheck coming from? Is that how it works?

JEFF: No. (laughing). It's more like a book publishing contract or something like that where basically they say, you're not an employee but we'll pay you to create records for us, and ultimately that's a good thing, because if you were an employee then, it's actually kind of how record labels are working a little bit more these days where they want a piece of, when you play shows and when you sell T-shirts, what they call an all-in deal, where basically the label is just there. Which is also kind of how record labels got started way back when, when there really wasn't a difference between the record label and the producer and the manager. It was kind of all one thing. But in the nineties at least, it was more like a book publishing deal where the luck at the time that there were a lot of record labels at the time, but the music industry was really changing. And so we got in a record label bidding war and managed to kind have our pick of a bunch of different record labels, and we really liked the way that A&M was running their label, and so we signed with them for a pretty favorable record deal with a three album, actually it was like a five-album deal or something like that. But, so, we knew that if we continued delivering albums, we would get at least the amount of money that it said in our contract. But in terms of like, making a full living, we still needed to tour and sell records and all that kind of stuff.

IVAN: And, they own all of the rights to the records that you produced in perpetuity? Is that right? Or does that get structured differently? And the reason I'm asking is, it feels like that was in the nineties, and that's a long time ago in technology years. And now there's Spotify and Apple Music and all these things, and I'm sure your songs and albums from Orbit are online and people still listen to them. How does that work?

JEFF: Well, it’s all very kind of complicated,and there are spreadsheets and algorithms that keep track of everything. But, in its simplest form, when you sell a physical copy of a CD or a record, and again I mean a lot of these sort of paradigms kind of go back, then, you know, the band makes a certain amount and the record label makes a certain percentage of that. However it gets more complicated because there's also the rights to the songs on the album. So, sort of the intellectual property of this music. So, that's called the mechanical, is the actual physical copy that you're buying. But then there is the publishing on the music that's on the CD, and then when it gets played on the radio, that's also different stuff. So it's all very complicated. It's very complicated.

IVAN: So, you see a check from Spotify for like five pennies every other year. Is it kind of like that?

JEFF: Yeah. We could do a whole Podcast about how it works. (laughing)

IVAN: (laughing) But we won't get into that.

JEFF: Yeah, but there are basically rights agencies that collect money from Spotify and radio play and stuff. So, I get different checks from different places. All of them for about seven dollars for various things. (laughing)

IVAN: But, it's interesting to know about all of that, I'm sure it formed how you ran subsequent businesses and other aspects of your life as well.

JEFF: Yeah. I learned a lot playing in a band and putting together the people in the band and around the band and kind of, very focused on, you don't really think about the business as a business when you're in a band. It's more about the vibe and all these things that are ultimately kind of branding kinds of things, like who’s it going to feel good to work, who's going to look good to work with, who can I sit in a van with for six weeks at a time, and all that kind of stuff. So, I learned a lot about marketing and branding, but also sort of group dynamics and a lot about contract law and intellectual property as well. (laughing)

IVAN: (laughing) What was your highlight during your time with Orbit?

JEFF: Boy, you know, a lot of it sort of happens in retrospect. So, prior to that, 1994 when we got signed, kind of the previous rock trend had been like hair metal bands. And then, like Nirvana broke through, and a lot of the stuff that I was actually listening to kind of broke through , and it was much more of a feeling like, of trying to be more authentic and a little bit, sort of more of like a blue-collar ethic, like less about having this facade of crazy partying. And so, you know we kind of came in, and being from Boston and all that kind of stuff, there was kind of this like, utilitarian aspect of things, like, we're not going to party our asses off and do all the drugs, and we're going to just get in the van and like go play shows and be the best band that we can be. However, we had some really great success. I fear that at the time we didn't absorb that, and if we didn't celebrate that enough, that we kind of took it as like, “well that's great.” We played on MTV or our video got played on MTV. Okay, what’s the next thing. Or we went and performed on a show on MTV. Okay, what's the next thing? And, we were kind of very on to the next thing. We did the Lollapalooza tour in 1997 and that felt good. And at the time in the nineties there were a lot of these like, radio festival shows, it seemed like each major radio station in each major market would have a festival kind of show, whatever the big outdoor venue was. And bands like mine would just kind of go from these radio festival shows and go on to the next and the next and the next, and we had pockets of popularity. But, I mean, we played one show north of Miami to 20,000 people, and all the people in the front are singing the songs along with us. It was like, it’s hard to kind of get too utilitarian about that. So, that kind of stuff’s the highlight.

IVAN: So, while you mentioned Lollapalooza 1997, I thought, there must be a poster online of what that looked like, and I looked it up and wow, you guys were playing with Prodigy and Tricky and Snoop. I guess he was called Snoop Doggy Dogg back then.

JEFF: Yes, he still was Doggy at that time. (laughing)

IVAN: (laughing) That's wonderful. Wow. Twenty thousand people is certainly a tool. KoЯn, except the poster didn’t have the ability to put the R backwards, maybe it wasn’t backwards back then, (laughing), I don’t know. So, now you're the front man for 123 Astronaut, I feel like there's a theme – Orbit, Lullabot, 123 Astronaut – what’s the genesis of 123 Astronaut?

JEFF: Well, I wish it was more thematic. I mean I think it is may because of its unintentionally it's a sort of subliminal intentionality. I was born in 1969, and my mom brags about propping me up, you know, five months old or whatever I was, to watch the moon landing. She says, “you're going to be able to tell everybody you watched this.” I don't know, that sort, kind of the hope and optimism of the space program, I think, is something that has always been really kind of close to my heart and sort of the melding of that optimism with sort of the technology that's offered by close to my heart, as well. But, 123 Astronaut, my son who's now 14, I think was probably two or three at the time, and he was going to throw something up into the air, and he was doing what he thought at the time was a countdown. But, instead of saying three, two, one blastoff, he said “one, two, three astronaut,” and then threw this thing in the air and I said, “oh, man, I’m going to name a band that someday.”

IVAN: That's awesome.

JEFF: And, in the intervening 12 years, never came up with a better name and didn't really think about its relationship to orbit and space, and all that kind of stuff. And. it really wasn't until about two or three months after we'd been playing out with this band name that I kind of realized, “oh, I have a theme going, I guess that’s okay. Yeah.”

IVAN: I think it’s cool. So, it feels like, at least to me, it feels like 123 Astronaut is you going back to music after this career in technology. So, in my mind, you were part of the music scene in Orbit, and then you founded Lullabot, and done some amazing things in the Drupal world and now you're doing Yonder and coaching, and you're also in a band. So, the question is, how has the open source mentality that you experienced in between these two bands, and how has running numerous companies and products influenced your perspective of your new band?

JEFF: Wow, that's interesting. I need to acknowledge just sort of the kind of disparate nature of these things, that arguably and I might be the one to argue this, but when I sort of think about it, like having credibility in business and credibility as a songwriter seem mutually exclusive, right? (laughing) You hear every so often, that I don't know, what's his name, the co-founder of Microsoft who just died recently?

IVAN: Is it Paul Allen?

JEFF: Paul Allen, right. Paul Allen’s got a band, and you sort of assume like, ehh, (laughing)

IVAN: Right. Like that's the first thing, you think.

JEFF: Yeah, and likewise you probably wouldn't go to like, you know, Slash for business advice, so, it’s been a little bit difficult feeling to try and pull these things together in my life, but they really are related. I really got into the creativity of building a business and, in a particular kind of gathering together groups of people to do great things, I think is something. But, to get back more directly to your question, it's been interesting, cause I kind of took a lot of that, sort of, utilitarian attitude in the building Lullabot, but I also took that lesson of, I need to celebrate the successes, I need to notice them. I feel like I went through this whole experience with Orbit and when all was said and done we'd gotten dropped from the record label in about 2000/2001, and we'd had all the success, but it didn't feel like success. And so I wanted to kind of embrace the success, and learn to kind of build off it. And Lullabot became very successful, certainly much more successful than we would have imagined when we first started the company, although I have a very good imagination. (laughing) And, as such I learned a lot of leadership skills, and I just became sort of more confident. I was also, as I mentioned, a parent and there's a lot about being a parent, that is sort of built in with kind of leadership skills and leading and I think with Orbit I was kind of hesitant to take a leadership position, to be too opinionated, to be too preachy is probably not quite the right word, but it's something in that area. But with this band I know I kind of bring that with me. I think, I'm writing more confidently and kind of presenting a more confident. more sort of opinionated perspective, than what was happening in the nineties.

IVAN: And, you're still being authentic in making this music for yourself, for the way you think that it should be made and not for the audience, which is, I think, a wonderful thread to have in any creative work that you do.

JEFF: Yeah, I think you need to balance that. The word I use in business is sustainability. You need to look after your business well enough so that your business can remain a business, that you may want to be true to your employees and really focus on culture and make it a great place to work. And so, having difficult clients is something that you want to avoid, but you need to have clients, and some of them are going to be difficult. And, so from the point of sustainability if you want to really make your employees happy, they need to continue to be employees. They need to continue to have jobs and you need to find the work for them. And so sometimes you need to kind find sort of creative ways to make it kind of be a win/win and keep the business. And the same thing with making art as a consumable, something that you want to put out there. Ultimately, I think it's best to drive it from yourself to find a perspective where I could get excited about. I like to consume things and go to shows and listen to music and get inspired by it. And so, how can I try to create that, that would inspire that in other people as I'm able to inspire myself. That's the hope at least. So, I think sometimes when you talk about this idea of kind of authenticity and just do it for yourself, it can get kind of, I think it can imply a certain sort of uncommercial reality, sort of inaccessibility to it. And I think both in business and in art, you end up kind of selling yourself short if you don't just acknowledge what people like about what you do or what people could like about what you do.

IVAN: One last question about 123 Astronaut. Are you going on tour anytime soon?

JEFF: (laughing) Well, this is the sustainability, commerciality thing. Nobody knows about the new band yet. In retrospect, I probably should have called it Orbit, creatively at least, I was kind of the creative lead in Orbit and the creative lead in this group. I've always really thought of myself as a collaborator and felt like I really needed to collaborate with people. And so I've been uncomfortable with the idea of like doing us solo thing or something like that, because I really value the input of people around me. But nobody knows about it. If I'd called it Orbit maybe at least we could have built off that, and we don't have a record deal or whatever the current equivalent of it is. I'm still trying to figure it out, having taken a basically 15-year break from the music business. But we're playing around in New England and trying to get down to New York City and stuff like that, and the people that have seen us have been interested and excited about this brand-new band that they've never heard of before. But, it's kind of slow going to gain fans one by one. We've got an E.P. that's out on Spotify, and if anybody who is listening and we've wet their appetite, 123Astronaut.com is where you can hear our music, see our music video and find links off to Spotify and Apple Music and all that kind of stuff.

IVAN: And, the singles called Friction, is it?

JEFF: Yeah. On the EP. That's the title of EP and the first song on the EP, and the video that we did, is all the Friction. But we're working on a new album right now, and I feel like I ought to be sending it out to record labels or sort of putting an infrastructure around the band but, mostly I'm just focused on trying to get the right guitar sounds. (laughing). I feel it's a little bit like navel gazing.

IVAN: Well, I wish you the best of luck with that. It's interesting to talk about how 15 years can make a difference to your perspective, I think.

JEFF: Yeah.

IVAN: So, let's go back to what happened after Orbit. You basically co-founded the company Lullabot, which from what I can tell, is the first fully distributed and remote company that ever existed. Am I going out on a limb by saying that? (laughing)

JEFF: (laughing) Well, it's really hard to gauge those things because, especially at the time, you get like freelancers that are working together, that don't have an office and is that a company? I think we were the first fully distributed agency services business. Automatic started right around maybe a year or two before us, and they were a distributed company, a product company, their product being WordPress, WordPress.com in particular is their commercial product. But obviously people know WordPress is an open source project. But we started in 2006, and I think much like Automatic sort of modeled ourselves around the incredible productivity that we were seeing happening around open source projects and the ways developers, in particular, were collaborating online.

IVAN: So, it was fully intentional to be distributed and remote right from the get-go?

JEFF: Yeah, I met Matt Westgate through Drupal.org and I just got over my head with my first Drupal project as probably most people do. And, I needed someone to help bail me out and I was asking questions in the IRC channels and finding a whole lot of developery snark. And that's very defeating and frustrating to kind of have people just sort of make fun of you as you ask questions. You know, this guy Matt Westgate was, every so often he'd answer a question in IRC and he seemed really helpful and posting also on the message boards, and I think I found it at one point. I did a search through Drupal.org and managed to find out our very first interaction. And I was asking some just sort of random question. It was probably about the ecommerce package in Drupal at the time, which was Matt's project,and he was super helpful and eventually I cornered him and said, “can I get you on the phone? I've done the math and even if I pay you to just answer my questions for me, I've got so many questions, that I think it would…right now I'm just Googling and searching and researching, is most of this project, and so if you could just point me in the right direction, then it would save me a lot of money, and if I gave you some of that money that I'm saving, I would still be saving money.” And that's how we kind of first met, but he lived in Ames, Iowa and I lived in Rhode Island, as I still do, and, it just seemed to be such a revelation, to be able to find this information in the form of Matt Westgate, that I was saying to him the whole time, I'm like, “listen, we need to start a company. People need this information that you have and that you're giving to me. And, I think that if we could help the world understand how to use Drupal, there would be value just simply in that, aside from actually building websites for people.” Just to tell people how to build websites would be valuable, and that's ultimately how Lullabot got started. So, again, to go back and answer your question more directly. Matt lived in Iowa, I lived in Rhode Island. So, we were distributed, and then we started finding more people. We went to our first Drupal conference, kind of as a company in Vancouver in 2006 and like February of 2006 and low and behold, several people came up to us and said, “oh, wow, you guys are really cool what you’re doing. Are you hiring?” And we thought, “what? Hiring?” We hadn't even thought out that far. And then we hired a few people who were also living in other areas of the United States and pretty quickly in Canada, as well. And then that was it. We were a distributed company.

IVAN: What a great origin story. I think I understand now, kind of the desire, to educate people about Drupal through Drupalize Me, because Drupalize Me came out of Lullabot, right? That’s a separate company. You guys started that and there was the best training on the internet for Drupal in my opinion, comes from Drupalize Me. So is that true? Is that part of it?

JEFF: Yeah. We started with kind of several different things first. When we set up the company, we thought of ourselves as a consulting company, that we knew, kind of from experience, that if Matt and I just took projects directly, it would take one project, two projects, before we would be unavailable and not able to help people to do more Drupal. And, so, with this mission of, we need to help people do more Drupal, being developers wouldn't be necessarily the best aim towards that goal. So, we started doing a podcast, we started doing Drupal workshops and then put out this idea of consulting. But, between the podcast, and we would use the podcast to promote the workshops, we really got known pretty quickly as the Drupal education company. And there wasn't really so much reason to talk about our consulting work on the podcast because with that, we were pretty full up. So, we would often times get people approaching us saying, “listen, you guys seem to be really experts in this Drupal stuff and I know that you do all these workshops and you guys do all this Drupal education, but do you think we could like, hire you to just like kind of help guide our project, like kind of like, maybe as a, I'll call it a consultant?” We were like, “well, our marketing is not quite, it’s a little too leaning towards all the education stuff.” Eventually we started doing full-fledged development, basically out of necessity. The Drupal market was growing so quickly and a lot of companies sort of built their foundation off of Lullabot taking this position as, sort of, the technical lead consultant for these projects but the development still needed to happen and so we would kind of go out and find all these Drupal companies and then they would kind of take-off from there. So, as those companies got more busy, it became harder to find companies to do the work for all these projects that we had. So, within a couple of years of starting, we started truly doing development and calling ourselves a development company. But we continue to do workshops. Eventually in 2008 the market made a big shift, in particular, a lot of the companies that we were working with were really big companies and the way that the finances tend to work around these companies is kind of slow and delayed. So, although there was this sort of market crash kind of thing that happened in 2008, a lot of the companies, their budgets were still there, they still had the budgets to do development projects, but they were trying to cut the corner so they wouldn't approve things like budgets for people to travel or to go to education events to learn about it. So, we started doing DVDs. Remember DVDs? So we started creating Drupal training DVDs and selling those and those did pretty well. And then, when the next version of Drupal came out, we realized that our DVDs were all going to be obsolete and we'd need to make an entirely new library of DVDs and with open source moving so quickly and stuff like that, and also the reason that I joke about remember DVDs is because the world has moved to streaming and so did, we. So, we took all our DVDs and that became the original basis of what became Drupalize Me, that became the original library of Drupalize Me, and then we started building additional content on top of that and it didn't need to be an entire DVD, we could just kind of do basically what's a patch video. It’s like, now that you watched that here's the new stuff in this module. Here's the new things in this new version and kind of keep it more evergreen. So Drupalize Me is still out there and still doing great. If anybody wants to keep up their Drupal knowledge, it's a great place to start learning Drupal or just make sure that you're keeping current.

IVAN: Your experience with distributed companies and distributed products, as a result of those companies, is vast. And so, in the last three years or so, you guys started something internally at Lullabot called Yonder. That's now its own company that you founded, and it's really, I think it's changed TEN7, honestly, because I think, I thought for a long time that TEN7 would be a company that was always in an office and would always be together, physically present with each other. But now we're completely distributed, and I think part of it has to do with all the literature that was out there and the podcast and the website and everything that you've been talking about, that really got me seriously thinking about becoming a distributed company. And things have changed right? There are more distributed companies now, than there ever have been. And I think the question I'm kind of leaning towards here is, asking you whether you think there's going to be a critical mass? A threshold of the number of organizations that end up fully adopting being a distributed company. And, I know there's gonna be a continuum, right? There's going to be always physical, and then hybrid where some folk are remote and some folk are physically in the location, and then completely distributed. And I'm curious about whether you think there's going to be a plateau or whether you think there is an extreme? We're going to be all distributed in the future, and we're going to have flying cars, and we'll use those cars when we need to go somewhere, and we won't really need to go anywhere. (laughing) Right? Where do you think we're going for distributed work?

JEFF: I think ultimately, sort of pragmatism will drive us wherever we're going. It’s sort of fun to think about it, kind of like, “oh yeah, everybody should do this.” But, again going back to that sustainability thing, like companies need to stay in business, and oftentimes they do what they know, they do what they know works for them. And so legacy companies, again kind of going back to the Fortune 500 companies, they know what works for them. They've figured it out over all of these years. IBM's been around since the fifties, forties, thirties, something like that. And it's kind of difficult to sort of change. The interesting thing is that these companies know this and they have departments -- the change department – that’s what they call themselves. These people who are like change agents within the company acknowledging that there's sort of this, kind of, I wouldn't call it lethargy but it is inertia, in that inertia also happens for things that are rest want to stay at rest, right? As things were in motion want to stay in motion. Things that are at rest, also want to stay at rest. That is also inertia. And so, what I've found is that the companies that are distributed tend to be younger companies. Companies that oftentimes started this way and are growing as a distributor company, automatic as I mentioned, continues to grow. But there are also companies like Shopify whose main offices in Ottawa, Ontario, Canada. But they have a clear central office, but mostly the companies are distributed, they kind of define themselves as remote first. GitHub is another company that kind of defines themselves as remote first. But now you know GitHub is part of Microsoft. Part of what Microsoft is buying and buying GitHub is the knowledge, the perspective, on how this can work. How do you make remote work, work? So, I don't know. It happens a little bit in fits and starts. The companies that are remote first or fully distributed are usually really excited about it. There are some requirements of the actual mechanics of remote work, the architecture of remote work, kind of requires autonomy. You can't look over people's shoulders in order for them to work. And autonomy requires respect, and the respect requires trust, and so, some companies have tried this without autonomy, respect and trust and they usually fail pretty quickly. But other companies sort of go in with this kind of testing kind of experimental attitude towards it. “Well, I guess we need to trust people. We need to give them autonomy if they're going to work from home.” And they do, and people rise to the occasion and they find that when people are trusted, they become trustable, and when they are respected, they become respectable. It doesn't always work with everyone, but there tends to be a little bit of sifting that happens. And then the people that are doing it, like I said, they get really excited about it. Just from that perspective of self-management and all those kinds of things that go with it. I would like to see more of it happen in the future. I'm available to anyone that wants to talk about figuring it out. This is sort of currently my mission, and the mission certainly of Yonder is helping people to figure that out and talking to people who have figured it out, and are trying to figure it out, and sharing what we are finding with everyone. We do a podcast and post articles on the website and stuff like that. We've got a mailing list. But we haven't hit that tipping point yet, right. I mean we haven't hit that point where all of those Fortune 500 companies are chomping at the bit to try to figure it out. But I do think a lot of it is, sort of generational. The definition of connectedness and productivity have changed, especially for the generation that's growing up with productivity and connectedness in their pocket. And the idea of needing to go into an office so that you can type things to people, like doesn’t really make sense, or that you could talk to people doesn't make sense, or that you could see people even, doesn't quite make sense. Obviously, this doesn't translate for all types of work, all types of jobs, doctors, more tactile professions. However, we are seeing even in that medical realm, there is a fair amount of talking to people about how they can be helped, that can happen well prior to a medical professional needing to lay their hands on someone. And so even some of that kind of stuff is starting to move into a more virtual realm and people can kind of work from wherever.

IVAN: When I think of distributed work and remote work, I always think of it as the antithesis to Gary Cole's character in Office Space. You know, the guy who walked up to the cubes with his suspenders and his tie and his thick glasses and just needed to make sure that everybody was in their cube doing what he was paying them to do. And just like you said, the more you respect your people and the more respectably they will behave and the more autonomy you give them, the better off it is for everyone.

JEFF: Well, it's kind of a human, it's not even a human nature thing. I mean, it goes back to sort of like animal pack behavior, having the leader of the pack, like the lion king right. Mufasa, standing up on the rock to look down over the lions to make sure, “we're all good. Are we all here, this is all good. Okay I'm in charge. I know I'm in charge because I can see everyone”. And then you think about kind of the industrial revolution, these giant factory floors with the management office raised up, so that they could just look and see everyone. It is a calming thing to be able to look over everyone You feel like, okay, I know what's happening. But the truth is, you don't know what's happening.

IVAN: Exactly. And it's maybe calming for the for the boss, for the person in charge. It's just the opposite.

JEFF: Yeah, it's sort of a false effect. Right, and going back to that, this idea of, kind of that factory worker mentality, or Office Space, right. I mean the whole thing with Office Space is that they're not really being productive, they don't really like their jobs, it's about putting on this facade and kind of playing this game in the office of caring, of trusting, of respect, and so you don't have that facade anymore. So, it kind of gets all broken down, and hopefully in trying to rethink it, in order to work remotely, we kind of rethink that relationship, because the truth is people need those jobs. Right. Like, you know there's this kind of, this sort of like resentment of your boss and resentment of your company, when you think about that sort of office space, kind of paradox. But the truth is people need jobs. Companies need sustainability. This goes back to that. We need to continue. You’re not going to get paid if the work doesn't happen, if the company doesn't get paid for the work happening and being done. Things don't progress, don't move forward if there's no productivity, and so to kind of redefine that and get everyone involved and to expose that fact is ultimately, I think, ultimately…That's really what I get excited about. Remote work, I think, is a way of kind of rethinking the relationship of workers to work in the future and the relationship of managers to workers and leadership and all that kind of stuff. I think that stuff will translate to all sorts of workplaces, even the ones that can't go remote, because they've got these more tactile professions and that's the thing that's even more exciting. The real work is kind of the mechanics of it, but kind of rethinking the way that work works is really interesting and exciting.

IVAN: And you're writing a book about it as well. I've been meaning to ask you how is that going?

JEFF: Also in fits and starts. It’s kind of sitting and has been sitting for a while now, as I try to figure out if there's a market for it, if people get excited about it and as I get more excited about playing music it just kind of sits. But, hopefully, in a perfect world I'll finish this new album for this band, and kind of have that obsession out of the way and as the weather starts to warm up for the summer in New England, maybe I'll start focusing back on the book again. I have a feeling it wouldn't take too much to shape it up into something that would be helpful for people.

IVAN: I look forward to reading it. One last question before we wrap up. You've read a lot of books and I would love to hear if you have a recommendation for a single book that I should pick up or maybe that our listeners should pick up and not miss out on?

JEFF: Well, there's so many.

IVAN: There are. Only one please. (laughing)

JEFF: I'm just gonna say the first one that came to mind. I guess I'm going to mention some others though. Three books have kind of come to mind. The One Minute Manager, this is like a classic management book, but ultimately, really kind of teaches the lesson that managers shouldn't really manage. That you need to let your people manage themselves. Then, the other book that came to mind was, even a classicer classic, How to Make Friends and Influence People by Dale Carnegie. Some of these books seem kind of old now to touch, but that one just, I think the real theme of that one is just reminding people as managers, as leaders, as people, that really, we all just want to matter. We want to be important. We want to matter. And, I think realizing that kind of builds a certain empathy and a way of connecting to people that recognizes that. Initially when that book came out, I think it was sort of this feeling that you could kind of start to exploit that understanding as a salesperson or something like that. But ultimately it comes down to authenticity. Neither of those books though were the first book that came to my mind. The first book that came to my mind was the book Tribal Leadership, which is not well-named. I think it becomes a little bit, when you read the book, you understand it at first but ultimately, that book is about finding a higher purpose for yourself and your business, in particular and trying to sort of strive beyond that sort of day to day stuff of being in business. This really rung true to me because playing in a band and making music with the goal of inspiring people, or the goal of trying to connect to people that you may never meet, is in itself sort of a higher purpose. And, I tried to sort of take that attitude to Lullabot when I started that business and find ways to you know make an impact on the world. You know, we should go teach people about Drupal. Drupal should be more popular, is ultimately sort of a higher mission than, “hey let's start a company.” I think people need websites and we can build websites and let's just build websites, as cheap and as awful as possible so that we maximize our return on investment. We will invest as little as possible and charge as much money as we can and try to cheat all of our clients. You know, that's sort of obviously the flip side of having sort of a higher purpose, higher goal. In the end you're building websites either way. But ultimately one, I think, will pull people towards you whereas the other tends to sort of repel people away, and that book is great for kind of reminding us of that and putting some, I don't know, kind of rules and metrics around it and stuff like that. So, I recommend that one. I tend to be an audio book person, being an audio person in general, but I believe that the audio book for that at least a few years ago when I found it, I could not find the audio book on Audible and in searching for it realized that that was because the Tribal Leadership people were keeping that for free if you sign up for their mailing list or something like that. So, for anybody that's interested in that book and particularly if you're an audio book person you can get it for free.

IVAN: Awesome. We'll link to all of the books you mentioned online in the transcript of this episode and we'll try to make sure we link to the free version too. Jeff, thank you so much for spending your time with me. I really enjoyed talking with you and listening to everything you had to say.

JEFF: Well thanks, Ivan. Thanks for having me on. I am never quite sure what I'm going to talk about with any given person, and when you invited me on, I thought, “boy I haven't really been keeping up with Drupal very well. I don’t know if we’re going to talk about Drupal for long,” but I hope all of this information is helpful to people. And, if anybody wants to get in touch with me jjeff.com is my website mostly for my business coaching there, but you can find the contact form to get in touch with me there and you can find me as jjeff on all the various social media.

IVAN: Thank you very much. Jeff is, as he said, online on jjeff.com, and of course, be sure to check out Yonders website at yonder.io and make sure you give his band 123 Astronaut a listen as well. 123Astronaut.com. All of that will be in the show notes in the transcript online. You’ve been listening to the TEN7 Podcast. Find us online at ten7.com/podcast. And if you have a second, do send us a message. We love hearing from you. Our email address is [email protected]. Until next time, this is Ivan Stegic. Thank you for listening.

Nov 22 2018
Nov 22

Your browser does not support the audio element. TEN7-Podcast-Ep-046-Healthcheck.mp3

Today, Tess Flynn and Ivan Stegic discuss Healthcheck, a new Drupal module that will make your site think it has its own personal physician. Here's what we're discussing in this podcast:

  • Healthcheck - a new Drupal 8 module
  • Contributing to Open Source
  • What is Healthcheck
  • Your site's personal physician
  • The need to know the health of a site
  • Healthcheck vs. Site Audit
  • Action modality
  • Hard coded vs. rules based
  • Notifications, webhooks & Zapier
  • A call for patches
  • Written from scratch
  • Drupal 7 implementation in the works
  • The all seeing dashboard

TRANSCRIPT

IVAN STEGIC: Hey Everybody! You’re listening to the TEN7 Podcast, where we get together every fortnight, and sometimes more often, to talk about technology, business and the humans in it. I am your host Ivan Stegic. We spend a fair amount of time at TEN7 building software that makes our clients and their users lives easier. In so doing, we end up creating things for ourselves that make our lives better, faster and less mundane. And, we inevitably end up contributing these things back to Open Source. You could check out ten7.com/wegiveback for more info. on things like Flight Deck and Starbase and other things we’ve built and given back and continue to support. Today, we’re announcing the beta release of a new Drupal 8 module we call Healthcheck. And, joining me now is the principal architect and maintainer of the module, Tess Flynn. Hi Tess.

TESS FLYNN: Hello.

IVAN: We’ve been spending our fair share of time talking to each other on the Podcast lately, haven’t we?

TESS: It does seem to be a running theme lately. I’m surprised you’re not sick of my voice yet.

IVAN: Not at all. I have a lot of fun talking to you. So, another module in the Drupal 8 ecosystem. Can you give me a 30,000-foot view of what Healthcheck is?

TESS: I like saying that Healthcheck is a bit like a doctor for your website. It sits in the background of your website and runs periodic checks on the health of your site.

IVAN: And, why is the health of your site so important?

TESS: Mostly because we spend so much time not looking at it. How often do you actually look at your own website? Do you check to make sure that it’s working correctly? That you didn’t make some change without thinking or in desperation, or when you were in the wrong browser window, thinking you were looking at a different copy of your site, and now something is amiss. How often does that happen? A lot of organizations, a lot of people don’t really spend a lot of time looking at their site. To them it’s just a thing that works until there’s a problem, and then once there’s a problem, we don’t know for how long it has been a problem, because someone had to go through the effort to complain about it. Or, someone internally had to stumble on it themselves, because who knows how many users it had affected in the meantime.

IVAN: I think you’re indirectly answering a question I was going to ask, which is, doesn’t Site Audit module already do this? But, what you’re kind of implying here is that, Healthcheck does this Healthcheck of the site, but it does it repeatedly.

TESS: Yea, Site Audit is more of a point checker. I mean, it’s in the name, Site Audit. And, when you do an audit, you’re not going to have an accountant come in and manage all of your finances. You’re going to have an auditor come in and rifle through all of your paperwork at once, to actually figure out what’s going on, and then once the audit is over, they’re gone, they don’t come back. Unless if you have reason to request or require another audit. So, the whole point of Site Audit is to do a point check, at one point in time. Healthcheck doesn’t really have that kind of mentality. It’s mentality is that it checks continually, on a regular basis, and it is meant to constantly monitor your site.

IVAN: And, unlike Site Audit, it’s a module, right? Site Audit is this drush thing, but Healthcheck isn’t?

TESS: That’s correct. Site Audit 2.x is actually a drush command, and it is meant to run as a point check, whenever you run the command. However, Site Audit 3X is actually supposed to be a Drupal 8 module. Now at the time of the recording of this Podcast, it’s not yet finished. So, I don’t know where they are with the development of that, however, it might be out by the time you listen to this.

IVAN: So, basically, you go to drupal.org, you go to the Healthcheck module, it’s for Drupal 8, you download it, you install it like you would any other module, and then you go to the reports page. Does that sound right?

TESS: That’s correct.

Healthcheck report screenshot

IVAN: And, can you tell me about the reports that it generates. I kind of get the, like a very high level, it gives you thumbs up, thumbs down, and thumbs somewhat down. So, things that are good, things that are bad and things that you should check. Is that about, right?

TESS: No. See, that’s one thing that is also different with Healthcheck. Healthcheck doesn’t really try to go against an idealistic bottle of what a Drupal site should be. Instead, it has kind of an action modality. So, each individual check has to report results on how you should resolve that problem, or not. So, is this a critical problem? Should you drop everything and immediately go and fix it. Is this something that you probably should do something about, but it’s not literally on fire? Is this, “well this might be ok, and you should probably look at it, but we’re not sure if it’s actually that bad. It might be good for your site.” Or, “this is fine, this is perfectly expected. Don’t worry about it.”

IVAN: And, who determines that? Is that something that you decided right at the beginning with the rest of the TEN7 team? Or, is that something that’s configurable? Those different levels.

TESS: So, right now, the checks are actually kind of hard coded. I would actually like to someday, make this a little bit more rules based, where you could configure those thresholds individually. I don’t see why the current architecture couldn’t do that, but it’s not in the module right now. We’re still getting towards that data.

IVAN: And, what kind of checks does it support right now? And, how do those compare to something like Site Audit, for example?

TESS: So, there are a lot of similar checks between Healthcheck and Site Audit. We check a lot of best practice things, like, we check if CSS preprocessing is on, what’s your caching configuration? We do a few other things as well. We check the size of the database, if there’s, oh geez, there’s so many of them that I can’t quite remember them all right now, I’d have to look them up.

IVAN: So, basically, it does a ton of things that you’d expect from a best practices point of view, a Drupal site should be doing. And, then, it does that on a schedule, and I would imagine that’s configurable?

TESS: Yep. The schedule can be configured to run along certain periods. I think by default that’s once every 24 hours, but you can actually configure it to run a Healthcheck on every cron run, and then whatever period you have cron configure to, it can run a Healthcheck.

IVAN: So let’s talk a little bit about how it knows that there’s a difference, compared to the check that it did last. Does it only store the last check?

TESS: No. What it actually does is that, the default mode of Healthcheck is what I call ad hoc reports, and what happens is that, out of the box it doesn’t do any checks in the background, it doesn’t send notifications, all it does is provides a report screen that you could interactively check. Now, there are ways of extending that afterwards. You can actually configure it to run notifications regularly, in which case, additional options show up for how often you want that check to be run, and how do you want to be notified of when that occurs.

IVAN: And what kind of notification system exists right now?

TESS: Right now we have two different notification systems built into the module. We have one for sending email notifications, and the email format is actually configurable, so you can change it to be whatever you want, And we have one that uses webhooks, and we’ve tested it with Zapier, I’m not entirely certain if it will work for other integration providers, but via Zapier, you can get to something like Slack.

IVAN: And that notification system to Zapier is really a JSON object that gets posted to a particular endpoint, and then whatever system you’re using as that webhook endpoint, can process that data and do whatever it wants with it. So, something like Integromat or Zapier, or any of the other competitors will likely work with this endpoint. Ok, so, if I understand this correctly, Healthcheck is a site doctor for your Drupal 8 website. It runs and can run on a schedule, it keeps a history of all the checks that it’s done in the past, and it compares the latest check with the previous one. It will then alert you either via email or via webhook as to if there have been any changes. Does that sound about right?

TESS: About. It doesn’t really check the previous one to the next one. There are mechanisms where if a critical item is found, it will run notifications no matter if that was previously discovered or not. And there are configurations to run regular reports and send an entire digest of all of the findings that have been found in the last Healthcheck to whatever notification system you configure it to.

IVAN: That sounds like a patch request. If someone’s out there listening, you can submit your own patches, we can absolutely add the feature of being able to check from a previous, do a comparison from a previous check, so that you’re only sending notifications out when something changes. Although, I’m not sure that that’s any better than sending it out every time cron runs. If you haven’t dealt with an issue, you probably, especially if it’s critical, you probably want to be reminded that something’s going on.

TESS: Yea, that’s generally kind of where it came down with the situation. It’s not like a phone notification, where once you see it once, you don’t want to see it again. This is an infrastructure notification, and it’s kind of common practice that you don’t want those to get deprioritized. You want to have that priority regularly refreshed, so that you know you have to deal with it.

IVAN: That. That makes a lot of sense. So, this is in beta release. It’s the first beta release for Drupal 8. So, basically, Healthcheck’s been written from scratch. Is that a good thing?

TESS: Some people say that it’s never a good thing to write a thing from scratch. Honestly, in my perspective on Drupal 8 is, you might as well write from scratch. And, the reason why is, Drupal 8 is fundamentally a very different system than Drupal 7, and it is a false dichotomy to assume that they’re really the same product. They’re completely different in my opinion. And, if you don’t take the time to reconceptualize what you’re doing from 7 to 8, you’re going to have a bad time eventually if your module is of any degree complex.

IVAN: So, let’s talk about the Drupal 8 version of this module. I want you to mention, and maybe kind of talk about how the plugin architecture has been designed. The idea behind it is that we are going to release a module here that has a certain amount of capability to do the checks that we really wanted to do. But then, make it possible so that anyone can write their own checks in addition to the ones that we’ve written. Does that sound about what we were thinking?

TESS: Yea. One of the primary concerns for me, when I was architecting this module is, I wanted to make it really, really, really, really, really easy to write your own checks. I don’t want to make that a complex, weird affair, I wanted to make it as natural as possible, for anybody who is used to implementing a Drupal 8 plugin. And, with Healthcheck, we already provide a base class for the plugin. We provide all of the documentation, we provide the plugin types, there are easily findable examples within the module that you can pattern it off of, and really you have to make one object and one function, and that’s it.

IVAN: Is there a limitation to the kind of check you could be writing?

TESS: Well, ok. There are a few things that do come as a limitation with the module-centric approach. One is, if the Drupal environment is completely non-functional, this module probably won’t work. And, that’s going to be a bit of a limitation for some people. Also, you can’t run it as part of your infrastructure hosting. So, you can’t have this as a command floating out there as Site Audit does, where you can configure the infrastructure to regularly run it against any directory that you assume has a Drupal installation somewhere. With Healthcheck, it’s limitation is that it is a Drupal module, so it only knows about your Drupal site, it doesn’t know about anything else.

IVAN: How does that affect whether you have a multi-site installation or not?

TESS: So, you can still install Healthcheck in the module directory, and have it available for all of the sites in the multi-site, but you have to enable it for each.

IVAN: So the beta release that we’re announcing today is for Drupal 8. There’s still a ton of Drupal 7 sites out there, and I know we’re planning on providing support for Drupal 7 as well. Can you speak to how that may be the same, or might be different than the Drupal 8 module?

TESS: So, I would like to write it as similarly as possible, with similar class names, similar interfaces, so that it works more or less the same. I would like to use C-tool plugins, but I also have no experience with C tool plugins, so I have no idea if this is a good decision, or even the right one.

IVAN: But we’re going to find out, because that’s how things evolve, and that’s how we learn. So, I’m looking forward to finding it out. Are there any other options to using C tools?

TESS: Well, you could write your own plugin manager toolkit, but it’s like, that’s a little bit…I’m not looking forward to that task myself if I was tasked with it. (laughing)

IVAN: (laughing) Well, we might have to have you architect it, and we’ll see who else we can get to help us do that. Ok. So, Drupal 7 version coming soon, hopefully in the beginning of 2019. As some of the listeners out there know, we have a service called TEN7Audit, which is usually the first step of onboarding a new client with TEN7. And typically it’s for a client, for a site we didn’t build, but might need Drupal support, and in the past, we used Site Audit to extract useful information about the site, kind of just as a cursory look. The very first thing we do in a TEN7Audit is run Site Audit, we get that initial automated list of things to check, but now we’re using Healthcheck as a drop-in replacement for it. There are obviously things that Site Audit does that Healthcheck simply does not, and I guess the question for you, Tess, is, when is it more appropriate to use one over the other?

TESS: Usually when it comes to running the individual Site Audit, usually I’m only concerned with one site, and so far, I haven’t actually run across any multi-site installs. I’ve run across a few domain access module installs, and also custom implementations which mimic that kind of behavior, but not a multi-site qua multi-site, yet. And even so, it could be run on it, it would just require a little bit more effort. And when it comes to when should you run Site Audit, well, for the moment we don’t have a Drupal 7 version of Healthcheck, so, I would say that anytime you’re doing Drupal 7 stuff, you should probably use Site Audit for right now. Also, if you intend to run this as part of an infrastructure setup, inside of server hosting, or containers that regularly run against your multiple installed sites such as if your….I know that Pantheon does this, I know that you could definitely configure Site Audit to work with Aegir, that is one possible usage of it, those are definitely good options to use Site Audit for versus Healthcheck. Healthcheck is meant to check just one Drupal site.

IVAN: Ok. We kind of started talking about future features in my mind there, so, maybe after the first beta release and the first stable release, we’ll focus on doing some Drupal 7 implementation of Healthcheck. But, maybe a future release, or have you thought about a future feature in subsequent releases to enable some sort of CLI interface for Healthcheck? Or do you feel like you don’t want to take the module down that route? Do you feel like you just want to continue to have the module be a module in the classic sense with the UI and not go down that CLI route?

TESS: The problem with the CLI route is once you go down that path, the question is, whether or not it’s going to maintain being a Drupal module, or if it’s going to be a Drupal orbiting command. Site Audit really isn’t a Drupal module, it’s a drush command. It’s what I would call, a utility, which is in Drupal’s orbit. But, Drupal itself doesn’t need to be functional. And, when you look at Site Audit internally that actually explains a lot of the design decisions that go into it, why it doesn’t rely as heavily on internal services provided by Drupal to do a lot of checks, but Healthcheck does.

IVAN: So, likely, no CLI coming to Healthcheck in the future.

TESS: Well, we can still add one, it’s just that it wouldn’t be a pure CLI approach. It would be an adjacent command which would allow you to run the checks, and even get those responses. I had thought about implementing that too, as a drush 9 command, and a Drupal consult command, But at the moment what you would usually do is, you would run drush cron and that would run the Healthcheck for you.

IVAN: The thing that made me nervous is when you said, “once you add it, you won’t be able to take it out,” and you’ve gone down that path. So, now I’m a little bit hesitant about adding the CLI to Drupal 8 Healthcheck. I kind of like the idea of just running drush cron and having it run Healthcheck.

TESS: I mean that already works. That’s how I was testing it. (laughing)

IVAN: (laughing) So, the other thing that appeals to me is, this idea that notifications or reports can be sent using a webhook, because that opens the module up to being used in so many different places as well. And, the thing that I am looking forward to is this idea that we could potentially have a dashboard where we can see all of the sites that we’re checking for clients of our own, the multi-sites, whatever they may be. All Healthcheck sites in one place, to see kind of the status across the board. Do you think that’s a reasonable expectation?

TESS: I would love to write that functionality because it would allow us to collate a lot of different reports from Healthcheck, and then take them into a single dashboard that we can display using views.

IVAN: I love that idea. And then maybe we could start to service online and other people can sign up to using it, and we will become a service provider then. Anything more you want to say with this beta release of Healthcheck?

TESS: The thing that I really like about Healthcheck, is that, it is intended for people who don’t have a comfortable grasp of the command line environment. They’re used to modules, they’re use to working with the Drupal admin Interface, and Healthcheck really shines there. It’s meant to be accessed and worked from, from that interface, and allow you to do all of your checks from that interface. And that’s what I really, really like about it.

IVAN: And it’s also created by an incredibly talented amazing, developer on the TEN7 staff. Thank you so much for all the work that you’ve put into it, Tess. I’m looking forward to getting a stable release out there soon after the beta, and maybe you’ll join me again when we add another major feature to Healthcheck.

TESS: Yep, sounds good.

IVAN: Healthcheck is a Drupal 8 module, available now in beta at ten7.com/healthcheck. It’s you’re your site’s own personal physician. It’ll periodically run checks on your site to see if something’s changed, gone wrong, or been misconfigured, and it’ll also alert you. Drupal 8 Healthcheck at ten7.com/healthcheck. You’ve been listening to the TEN7 Podcast. Find us online at ten7.com/podcast. And if you have a second, do send us a message. We love hearing from you. Our email address is [email protected]. Until next time, this is Ivan Stegic. Thank you for listening.

Nov 07 2018
Nov 07

Your browser does not support the audio element. TEN7-Podcast-Ep-044-DrupalCamp-Ottawa.mp3

It is our pleasure to welcome once again Tess Flynn, TEN7's DevOps Engineer and DrupalCamp ambassador, to discuss the 2018 DrupalCamp Ottawa.

Here's what we're discussing in this podcast:

  • 2018 DrupalCamp Ottawa
  • Minnesota maple syrup
  • Camp format
  • Ottawa's move to Drupal open source
  • Award for travelling the farthest to attend
  • Camp without BOFs
  • Drupal 101
  • Keynote: “Building Accessible Experiences”
  • Accessibility is a core aspect of the entire design experience
  • Socketwench presents: "Healthcheck your site!"
  • Building software as a service
  • Privacy laws differences between Canada and the US

TRANSCRIPT

IVAN STEGIC: Hey Everyone! You’re listening to the TEN7 Podcast, where we get together every fortnight, and sometimes more often, to talk about technology, business and the humans in it. I am your host Ivan Stegic. In this episode of the Podcast, we’re talking to Tess Flynn about her visit to DrupalCamp Ottawa 2018, that happened on Friday, October 26. Tess, welcome back to the Podcast.

TESS FLYNN: Could you even use fortnight now? Isn’t that copyrighted?

IVAN: (laughing) Well, it’s spelled differently, so I think we might be ok. Yea, good point though. Let’s see, DrupalCamp Ottawa. You just got back from Canada. Did you bring back any maple syrup?

TESS: I did, but the problem is, that some of the maple syrup we get here locally actually tastes a bit better than the kind you get from the touristy travel shops that you get in Canada.

IVAN: Yea, we’re a little spoiled in Minnesota with maple syrup, I agree. So, DrupalCamp Ottawa is a little different in format than DrupalCorn that we talked about last. It’s one day of Camp, it’s a Friday, so 25% the length of the other Camps. How did that feel compared to the extended four days that we talked about last time?

TESS: I think that it actually felt rather appropriate. Mostly because you can’t really talk about this Camp without mentioning the fact it is doing head on comparison competition with BADCamp.

IVAN: Oh, that’s right. I’d forgotten that BADCamp was at the same time. What’s the format for BADCamp?

TESS: BADCamp’s a little bit more like TCDrupal. There’s a day of training, then two days of sessions, then a day of contributions.

IVAN: Do you think that affected attendance in Ottawa?

TESS: Well, I actually was wondering about this, as well. The question whether or not is, if you actually had the choice between the two, would you go to one or the other. And I think that’s kind of a false dichotomy, because from another perspective Ottawa is in a completely different country. Even though it’s not very far from Minnesota, at the same time it is technically a different country. So, there are reasons to actually choose a date that even coincides with one of the biggest regional Camps in the United States.

IVAN: And it’s also on the complete opposite end of the Continent as well.

TESS: Yea, it’s on the Eastern time zone.

IVAN: And, how large was DrupalCamp Ottawa, in terms of number of people?  Just share attendance. Just a guess.

TESS: They said that about 250 people registered. Some of those were going to be sponsors, and a fairly typical pattern is that they’ll register more people than actually shows up. So, I would probably guess maybe 175 at least, probably more like 200 and change.

IVAN: Wow, that’s a whole lot for regional Camp and only one day of programming.

TESS: Well, you know, it’s that other country factor, and there’s a lot to really unpack there, because it’s not just a DrupalCamp somewhere else. There are specific regional concerns that go along with having a DrupalCamp in Canada and using Drupal in Canada.

IVAN: So, let’s talk about that a little bit. Would you guess that most of the attendees were from Ottawa and from Ontario?

TESS: I would probably say so, because Ottawa, from what I recall, is the Capitol. So, there’s a lot of government in Ottawa. A lot. And, Ottawa is trying to pivot towards doing more Drupal open source, and more open source in general. So, the idea that a lot of people would attend this Camp to get more open source information makes perfect sense. And, to put it in the same city that a lot of people work in, also makes a lot of sense.

IVAN: It does make a lot of sense. Now, I heard you received a special award.

TESS: (laughing) There was kind of a joke about that. As a Camp speaker, there’s always kind of a little bit of a joke about, if you were the farthest one to attend the Camp. And, from my knowledge, I might’ve been one of the few Americans to attend the entire Camp, and probably the only one that really needed to take a flight to get there.

IVAN: (laughing) What was the prize? Or was it just a proverbial pat on the back?

TESS: It was more like, “oh, really, I am the furthest away one. Oh, that’s nice.” That was it. (laughing)

IVAN: (laughing) Now, I looked at the schedule and it looked like it was broken up into three tracks for the day, and it loosely seemed to be something along the lines of front end, back end and everything else. And, everything else was kind of like business, strategy, communications, content, which kind of makes sense. Did I get that right? Was that more or less how it was?

TESS: It certainly felt like that. I mean with only one day of Camp, and only about four different session periods, there’s not really that much need to break it up along too many different functional lines. There’s only so many slots available.

IVAN: And no BOFs from what I could tell.

TESS: No. I don’t think they had the room available at the venue in order to do that, but they might have.

IVAN: I see. Nice segue into the location of where the event was, it was at the University of Ottawa. The website says the SITE Building. Could you tell me more about the space?

TESS: That place has just got such an interesting personality. How can I explain this? Like if someone took material design and construction aesthetic and mashed them together, you get this combination of bright colors and metals and all sorts of interesting things. It was really, really, a nifty little venue. It was very visually interesting. And, because the Camp wasn’t particularly big, everything was in one building, so it was very easy to find everything.

IVAN: So, three rooms, all in one building. I would assume lunch in a central place, as we’ve come to expect?

TESS: That’s correct.

IVAN: Right. That’s great. That seems to make quite a cozy atmosphere for attendees. I really like those, when they’re all close together and bunched up. Let’s talk a little about the pre-keynote. It looked like there was a session on the scheduled called Drupal 101, that seemed to be very inviting for beginners, kind of before the keynote happens, if you’re new to Drupal, not sure what a node is. The description says, “bring your coffee and get a quick course in Drupal terminology.” I love this idea of kind of giving an intro before the festivities or the keynote begins.

TESS: Yea, I rather liked how that went because it provides a nice bit of framing, that would’ve otherwise been taken out by a training session on the first day of a multi-day Camp. And, I think it was a nice compromise in order to allow people who have heard about this Drupal thing, and then get a nice introduction, so that they can get value out of the Camp. And, because the Camp was on a Friday, some people might be attending this on their work hours.

IVAN: Yea, I think that’s a great welcoming idea. It would be interesting to talk to the organizers to hear what their take on the motivation behind that was. So then, that rolled into the keynote and the keynote was titled, “Building Accessible Experiences.” And, it was from developer advocacy lead at Shopify. I can’t pronounce Tiffany’s last name, I’m going to try, Tiffany Tse. Any ideas if I got close, Tess?

TESS: No, I don’t think the coffee had quite kicked in, and I think I barely missed her last name too. So, I can’t quite remember the pronunciation either.

IVAN: (laughing) We’re sorry Tiffany, if you’re listening. Call us and let us know how we did. Yea, so Shopify, first of all, I love the fact that the keynote was from someone outside of the Drupal ecosystem.

TESS: I just really appreciated this particular keynote. A lot of keynotes lately, including one that I gave myself, tended to be a lot more broad-reaching, a lot more big ideas and directions and business policies. And this one was a lot more down to earth, a lot more practical, really put you into the pilot seat of, “okay, you’re going to be an accessibility designer. What’s wrong with this?” And, it was just a wonderful experience, because it really sat you down and made you think about what you were looking at, and it was nice to do that as the first thing in a Camp, because it felt very direct.

IVAN: Glad to hear it. So, what do you think your major takeaway from the keynote was?

TESS: Well, I think the general message that I took away from it was that accessibility is not something that you can just bolt on later. It is a core aspect of the entire design experience, and you should consider it very carefully from the very beginning, because a site can be a lot more versatile than say an application can be. And, it has a lot more audiences, and a lot more modalities in which that, it is presented to different users. And, it was really, really, well communicated.

IVAN: And, further to that, the thing that I always want to try to remind everyone we’re working with, and the people that we help with our sites is, not only is accessibility important to think about from the design aspect and right from the beginning, but it doesn’t stop after you’ve launched a site. It’s something that continues, that all members of the team that are responsible for the site have to be aware of and continue to build on. It’s not something that you just launch as a feature, and you’re done. So, I’m glad to hear that was a good keynote. And, it looks like your session was directly after the keynote, in the same room (laughing). So, did you luck out and have a whole lot of people stay?

TESS: I apparently did have a lot of people staying for that session. I was kind of surprised, actually, about the number of people that attended. I think it was some 50 people that I counted right before I started. And, I know that some people came in after I got started as well, that I didn’t get a chance to count.

IVAN: And you gave away all the TEN7 swag at your session.

TESS: Yea. We were running a little bit late because the keynote ran a little bit long. So, when I first set up, I basically put everything out, and anyone who was an early bird I said, “here, come take. Don’t make me take this back through American security.”

IVAN: (laughing) Yea, we were a little light on swag at this Camp, because of the fact that you were traveling internationally. But, I’m sure we had enough to make some people happy there.

TESS: It all vanished anyway.

IVAN: Yea, that’s what we want. Any particularly interesting questions that came up in your session, that maybe you haven’t heard before?

TESS: So, the thing with my sessions is that very rarely do people actually come up with questions, because once I tend to get started, it’s really hard to get a question in edgewise, because I just have (laughing) such a presentation that is just a firehose of nonstop rambling for almost an hour. And it’s really hard for people to just stop and ask questions. Sometimes people do, but my sessions tend not to get a lot of questions.

IVAN: I think you do a great job of explaining things so clearly with analogies and with detail that, that’s maybe why there aren’t any questions. I certainly appreciate attending those. So, just looking at the other sessions on the schedule, a few that peaked my interest, “The New Face of CiviCRM.” CiviCRM still makes me a little scared, so I’m glad that there’s a new face. “Building Software as a Service in Drupal,” another session I thought was something I might have attended had I been at the Camp. And then, “Drupal as the Base of an Inclusive Workplace,” which was Mike Gifford’s session. It’s an interesting idea. I kind of read the description of the session, the fact that Drupal is largely still known as a CMS, and people really don’t realize that it’s much more than that, especially when you think about accessibility and user experience. You went to that session, right?

TESS: I did. But I went in with the expectation, because I didn’t read the description very well, that it was going to be a little bit more culturally focused, and how to build a more diverse team as a result of using Drupal. And, so, when they started going on the technical merits I was like, “Ahhh",  and it’s totally my fault, I didn’t read the session description very well.

IVAN: Oh. So, what was your takeaway then from that session?

TESS: A lot of it reminded me of the keynote, but it also kept pointing out one thing that was really important is that, accessibility doesn’t just benefit those who are disabled, because accessibility is not just going to be for those who have a permanent disability, but a temporal or situational disability as well. And, there was a lot of focus on bringing that into the conversation as well.

IVAN: Mike does a great job of being inclusive, and I imagine that was a wonderful session to attend. Did you go to the “Building Software as a Service on Drupal” session?

TESS: I did go to that one. I also, kind of was hoping this one was going to be a little bit more business focused. It actually was mostly a technical discussion about how to use Aegir, which has been around for the better part of 10 years in Drupal circles, and is still going, and is still a method to provide a Drupal solution as a software as a service. And, the next version of Aegir is supposed to finally support more than just Drupal, and virtually any php application, and possibly any web application that can be deployed.

IVAN: So that’s how you say it?

TESS: What? Software as a service?

IVAN: No. Aegir. I always wondered about that.

TESS: No, I only remember that because I think I listened to. Was it a Drupal easy podcast, like years ago, half a decade ago, about Aegir? And that was like one of the first things that they were going to talk about was, “how do you pronounce this? It’s got a diphthong in it, why?”

IVAN: (laughing) I want to spend some time talking about this building software as a service session. So, from what I understand, Aegir’s basically a way for you to host your own site, and maybe even sell hosting to others as a service, particularly just Drupal sites. And you said that it would, in the next version, be supporting more than just Drupal sites, but PHP applications as well. Is this the basis for Pantheon? Is this where Pantheon started? I have no idea. How is it similar or different to Pantheon?

TESS: I don’t know if Aegir was actually used in Pantheon at the beginning. I do know that they were using their own home brewed containerized solution, possibly using Xen or KVM at some point, and that they recently transitioned to Google Kubernetes engine in order to run most of their container systems. And primarily the product that they have is a web front end and a pricing tier in order to better leverage all of that usage. And, I’m not sure if they ever really utilized Aegir for that or not.

IVAN: It looked like this was a session that was more in the style of a BOF, the way the description was written. It felt like it was going to be more discussion oriented. Did that turn out to be the case?

TESS: It did turn out to be the case. I was really hoping for a lot more perspective from the business perspective, because it felt like it was very technically focused, very capability focused, as in Aegir can do this, Aegir can do this, this is how you do this. Yes, you can run it on your own hardware, why would you want to do that? And, this is where one of the key things that I took away from the entire Camp starting sitting in my mind, is, that, because I’m not in the West, there are different concerns for hosting, and a lot of Canadian companies do not want to rely on any US hosting. And I cannot blame them, considering our utterly lackadaisical privacy laws. And, I’m being generous when I describe it that way.

IVAN: So, what turned out to be the options for Canadian companies for doing hosting, if they’re not going to rely on US technology?

TESS: Well, I think that AWS is now involved, but that’s still a company that’s technically owned and operated from the US, and that might not be as comfortable for people. I actually haven’t had enough time yet, to really investigate the hosting market in Canada. It feels like it needs more development, honestly, is my initial impression. I could be wrong about that. I can guess that there’s probably a lot of on-premise hosting, but not nearly enough, like cloud-based hosting. And, there might be a lot of shared hosting, as well, that is used by a lot of smaller sites. But I’m really concerned that there’s just not enough cloud hosting, that is also hosted in Canada, in order to make sure that the privacy laws still apply, that the local/regional laws still apply, and that these are actually utilized for Canadian sites. And, this may be a hollow argument if a lot of the Drupal market share is government, because they’ll be more likely to self-host than use cloud products. Although, that made me think the following day, why isn’t it that the Canadian government itself, doesn’t form a wholly-owned and operated company that does nothing but hosting an infrastructure providing in a cloud facility. They’ve got to have more than one data center under their ownership already.

IVAN: Yea, that’s a good point. It seems like a market opportunity, that a company like Pantheon or Acquia could certainly take advantage of. But then at the same time, they’re a US company that are operating in Canada, and so, maybe there’s a Pantheon Canada that gets formed, or a company that’s run and operated in Canada by similar or related people to the same US company, and yet they have their own privacy standards and use privacy protocols that are acceptable to the Canadian laws. I think Google has GKE zones that are available in Canada, so in theory, you could potentially do that. I suppose.

TESS: Yea, I think there probably are some GKE zones in Canada as well. I have to look into that to be sure.

IVAN: Maybe we should start a hosting company, Tess.

TESS: I’m all for that.

IVAN: (laughing)

TESS: Ottawa isn’t bad, but I like Toronto more. (laughing)

IVAN: (laughing) Ok. We could be wherever you want.

TESS: This is a thing of mine though. When I used to do business travel a lot, I noticed that I tend to get immediate impressions of places that I touchdown in. It’s really weird, because it doesn’t seem to make any logical sense to me either. But Toronto had a very familiar vibe to what I’m used to in Minneapolis, but there were certain rounded corners that I didn’t have from the same vibe in Minneapolis. And those were probably where a lot more of a Canadian cultural vibe was poking out. And, Ottawa felt very similar to that, but a slower pace. It’s a little bit hard to describe. I wanted to describe it using a music analogy. So, the thing that pops to my mind is that, there is a video game called Undertale that has been around for a while, and towards the end there’s this one area of the game that has very upbeat, fast paced music. But if you take an alternate story path in that game, that same music plays, but in a very slow, lumbering pace instead. And, I didn’t get that exact feeling, but it definitely made me think, “wow, this is the same song, but it’s slightly slower. That’s interesting.” (laughing) I know this is all a ridiculous subjective, but that was something that just kept coming up when I was there.

IVAN: It felt familial and accessible I would argue. So, no direct flights to Ottawa from Minneapolis. Did you fly through Toronto?

TESS: I did fly through Toronto, and that was actually fairly easy. It’s only a two-hour flight from MSP and you could get a direct. The only problem is, once you’re in Toronto, you have to catch a one hour connecting flight to Ottawa. Now, I didn’t have any problems going to Ottawa, but coming back I just kept getting hit with delay after delay, and it was a little bit frustrating, because we left probably 15 minutes from Ottawa, from when we were supposed to. I didn’t mind so much, because I had enough time to account for the difference. But then once I landed in Ottawa, I failed to remember that the international security procedures had changed since I last traveled internationally. And now you have to go through international customs as an American citizen on the international side, rather than the US side. And that was a Kafkaesque experience to say the least. I felt like I was reenacting the movie Brazil a little bit there.

IVAN: (laughing) Yea, we had that same experience flying through Toronto on the way back from Europe this year, and, it actually made me think of, kind of what laws apply on the Canadian side after you’ve cleared US customs. I know that it’s US law that applies, but that just feels wrong.

TESS: Someone explained to me that Canadian transit agency, their equivalent of the TSA, is actually a superset of TSA law which just makes me go, “oh geez.” (laughing) In TSA law, if you know even a little bit about it, is already this nightmarish labyrinth of weird edge cases, and political meddling, and none of it makes any sense anymore, and it hasn’t since about 2007 honestly.

IVAN: Yea, it was pretty insane, certainly Kafkaesque as you said, going and clearing customs in Canada for the US, and then physically being in Canada, but technically being in the US after you’ve done that.

TESS: Well, the real hilarious part is the nature of how this works in the Toronto airport. When you actually go through Canadian security at the Toronto airport, and you first get cleared, you’re opened into this wide foyer and it’s got this giant flower sculpture thing, and the underside of each petal is actually your arrival and departure time screens. It’s really nice. And then afterwards I had to walk through there to go to a completely different concourse, and when you get to that concourse you have to go through security again, then you have to go through customs again, then you have to go through the customs waiting area, because they won’t let you go directly to your gate. Then after you go there then you walk to your gate, and by the time I got all the way to the end of my gate, I was on the other side of a glass window, and on the other side of that glass window that I was looking right through, was the giant flower. And that took an hour and a half.

IVAN: (laughing) Wow. Well, I think we have a little more time to talk about the other session that kind of peaked my interest, and you also I think went to it, was the “Journey through the Solr System.” And, the only reason it peaked my interest was because I thought the title of the session was amazing.

TESS: The slides were also great too. They had a really nice visual style that I really appreciated. It made it very fun, but at the same time it focused on information. And the talk itself was also different than I expected. Now, usually when you think of Solr and Drupal, you’re going to think of, well you’re probably going to use a search API implementation, and it’s going to be one site, and you’re going to configure which entities that you’re going to have going to which in that system, and then you’ll use views in order to make your search pages and yada, yada, yada. Well, they couldn’t do that with this solution. The problem is that they have some two hundred different sites, and they had to have a unified singular search mechanism. And it wasn’t a multi-site either, so you couldn’t kind of cheat and use some of that facility in order to populate a single index. So, either they had to come up with a completely custom solution in which any time content was posted for each individual site it went back to a standard search API server, or they’d have to do something completely different. What they used to use, they used to use a Google search appliance, and this was great because it was on premises, all of the data was local, they owned it. And then, suddenly, those yellow boxes stopped arriving from Google because Google deprecated the entire product line. Now you have to forward all of your search index information to some American server, and this is not comfortable for some people, and that is perfectly fair. So, they could’ve paid for a different solution, or they could’ve went, “well, we’ll just risk the privacy implications,” but instead they decided, “you know what, let’s see if we can try to build one of these ourselves.” So, the solution they came up with was, a high availability Solr configuration with an open source web crawler called Nutch, and it was just a fascinating combination of elements to make, basically, your own Google, but within your own organization, for your own sites, without having to have a direct backend connection.

IVAN: Nice. I really love that name, Nutch.

TESS: That was a really, really fascinating talk, and I wish that I could’ve captured more of the technical details of that, but I was coming right off of doing my session, so I still had a lot of adrenaline in me. (laughing)

IVAN: Yea, and I’m sure that the session video will be posted once it’s available. Yea let’s talk about that a little bit, and then I think we’ll wrap. So, it looks like there were sessions that were recorded again, courtesy of Kevin Thull and his equipment.

TESS: Well, not quite.

IVAN: Not quite?

TESS: Not quite. Kevin Thull was not there, he was at BADCamp.

IVAN: Oh. But his equipment was there.

TESS: Well, from my understanding what happened is that Kevin Thull trained the DrupalCamp Ottawa staff, and provided them a list of the hardware that he uses for his talks. So, they reimplemented all of that under his guidance, and then ran it themselves, independently. So, it was a very familiar experience. Everyone had the big red button that they had to press. So it was very, very familiar. I do know that they have a few gotchas with the session recording, but they had generally had a fairly good capture ratio.

IVAN: That’s wonderful. I do see that on the DrupalCamp Ottawa website they published a playlist on YouTube, and I think there are about six videos on there right now, six sessions that are currently available with the note that they’ll be adding the rest of the sessions in the coming week or so. So that’s great. We’re going to have a recording of your session, and you could probably go back to the Solr session as well and check the details of that one out as well. Well, all in all, a good Camp. Something that maybe I’ll consider going to next year, and maybe we’ll send you again next year. Tess, thank you so much for spending your time with me and talking through DrupalCamp Ottawa 2018.

TESS: No problem.

IVAN: You’ve been listening to the TEN7 Podcast. Find us online at ten7.com/podcast. And if you have a second, do send us a message. We love hearing from you. Our email address is [email protected]. Until next time, this is Ivan Stegic. Thank you for listening.

Oct 24 2018
Oct 24

Your browser does not support the audio element. TEN7-Podcast-Ep-042-DrupalCorn.mp3

It is our pleasure to welcome Tess Flynn to the TEN7 podcast to discuss attending the 2018 DrupalCorn and presenting "Dr. Upal Is In, Health Check Your Site". Tess is TEN7's DevOps engineer.

Here's what we're discussing in this podcast:

  • DrupalCorn2018
  • DrupalSnow
  • Camp scheduling
  • What it takes
  • Unconference the conference
  • Substantive keynotes
  • Dr. Upal is now in
  • The good health of your website is important
  • It takes humans and tools
  • Every website is a bit like a person, it’s a story
  • Docker-based Battle Royale
  • Auditing the theme
  • Mental health and tech
  • Drupal 8 migration
  • A camp with two lunches
  • Loaded baked potatoes and corn
  • Cornhole
  • Catching Jack the Ripper
  • Onto DrupalCamp Ottawa

TRANSCRIPT

IVAN STEGIC: Hey Everyone! You’re listening to the TEN7 Podcast, where we get together every fortnight, and sometimes more often, to talk about technology, business and the humans in it. I am your host Ivan Stegic. In this episode of the Podcast, we’re talking to Tess Flynn, about DrupalCorn 2018, that happened at the end of September, in Des Moines, Iowa. Tess, welcome back to the podcast.

TESS FLYNN: Hello!

IVAN: So, DrupalCorn. I love that it’s not DrupalCamp Iowa.

TESS: (laughing) That’s one thing that I always appreciated with it. They have a sense of humor to match their camp.

IVAN: (laughing) It’s just great. I think we have to come up with something for TC Drupal and I don’t think we can top DrupalCorn.

TESS: I still miss the Cherry logo that we had years, and years, and years ago. Although the snow globe one’s pretty good.

IVAN: So, DrupalSnow, maybe? 

TESS: (laughing) That would be an interesting thing to do, like a one-day event, in the middle of winter. (laughing)

IVAN: (laughing) We’ll see how many people come out to that, huh?

TESS: BOFs and sledding. (laughing)

IVAN: (laughing) That would be fun. That’d actually be fun, actually. So, let’s see, DrupalCorn, this year it was at the Center for Higher Education in Des Moines. Is that on campus somewhere? I’m not familiar with where that is.

TESS: So, it was in the business school, that got bought out. There’s a bit of a story about this. It used to be, I think, an independent business school, and then it got bought out, and brought into the university system, and, yeah, there’s a bit of an interesting story behind that, but that one’s not mine to tell.

IVAN: (laughing) Ok, so, on some sort of a former business school campus, and from what I could tell, the format was exactly the same format that we’ve had at DrupalCamp Twin Cities. So, training on a Thursday, session and keynotes, Friday, Saturday, and then contribution day on a Sunday. That’s a lot of programming for a camp.

TESS: It is!

IVAN: Yea, it is. Do you think it’s overkill?

TESS: I think that it’s up to the camps to really determine if that scales for their audience. And, I did participate in the closing session, for DrupalCorn, which was mostly for camp organizers and volunteers, but I was in the same room, and I decided, oh, I’ll just stay around and how it goes, because they were inviting anyone who wanted to stay for it. And, it sounded like they had lots of people attending the training sessions, that they had no problem filling those out. And, the first two days, seemed perfectly fine. It didn’t seem like there was a massive drop off in individuals or people at either one of those.

IVAN: And, that closing session, was that on a Saturday?

TESS: That was on the Saturday.

IVAN: Ok. I mean, it takes a lot of time, and a lot of money, to plan and execute such an extensive program, and I am always amazed that camps do this, and that it’s all volunteer based, it’s not for profit, people give of their own time. It’s amazing! It’s just amazing!

TESS: Well, DrupalCorn serves a lot of different camps in the Greater Midwestern area in the United States. There were lots of people from Kansas, which just surprised me. I was like, “why are there so many people from Kansas here?” And, “why don’t I see those same people at Twin Cities Drupal camp?” And, a lot of it comes down to, “well, it’s closer.”

IVAN: Yea, it’s closer. Like, you would have to drive twice as far, to the Twin Cities, or fly. So, that does make a difference. Was the size of the camp, kind of what you would expect, 200 or so people? Was it any bigger or smaller, this year?

TESS: It felt like it was a bit smaller this year, mostly because they took last year off, for various reasons. And, there was actually an interesting discussion about, if camps should do that more often, that we could talk about.

IVAN: You mean, take a year off?

TESS: Mm hmm.

IVAN: Yeah, so, was that part of the closing session? Where was that discussion?

TESS: That was part of the closing session.

IVAN: So, what were the thoughts around it?

TESS: So, the reason why they decided not to have one last year is there were some people who were starting families, and had vacations, or had other things that were just happening, and they just couldn’t get their core organizational group together, in order to do that. So, it seemed like it made more sense to just, not have the camp that year. And, at first it was really disappointing, because I remember talking to one of the organizers at Baltimore, and being like, “aw, there’s going to be no DrupalCorn this year, it’s one of my favorite camps.” But, since then, I thought, this is actually not a bad idea, because, I’ve gone to DrupalCorn for the previous two, if not three, years. Then they had the year off, then they had this year, and, one thing that I was kind of surprised by, is that, compared to the last DrupalCorn, there was more energy at this one. There was more focus. There was more drive. There was more enthusiasm. I think that actually is something to be said about taking a year off, occasionally. It lets your organizers rest, so that the next year they can really go at it. And, I’ve been thinking a lot about Twin Cities DrupalCamp, because DrupalCon Minneapolis is in 2020. Should we even have a camp in 2019? That’s a tricky one.

IVAN: Yeah, that is a tricky one. It feels like we’ve been evaluating whether or not, not just whether or not the camp’s going to happen, but, what the format of the camp might be, as well. And, I was at the Twin Cities Open Source CMS Unconference, whew, mouthfull (laughing), this weekend, and, it just struck me that, having an Unconference like format for a camp, makes it so much easier to organize, and to put together people that are kind of basically there to learn, just like what the camp is for. And, I wonder if it isn’t, not just taking a year off, but maybe there's an Unconference like event, might be a thing to consider as well.

TESS: Yeah, and I think those concepts can actually play well with each other, because people who like the Unconference format, might not be the same people who normally run a traditionally and conventionally-focused camp. In which case, why not just do a slightly different Drupal event that year, for the area? That might work too, then you can compare and contrast later.

IVAN: Yeah, that might be a good idea for us. It’s interesting how all of this has evolved in the community, and how Drupal compares with local camps, and WordPress, and other communities. It’s just fascinating that all of this happens the way it does. So, I want to ask about the keynotes. So, I want to ask about the keynotes. There were two keynotes – Tiffany Farris from Palentir.net keynoted, and I think the title of her keynote was, “Learning at Work,” and that was on Friday, and then, Matt Westgate from Lullabot, keynoted, and I love the title of his keynote, “How to Fall in Love with Drupal Again.” What are your take-aways from the two keynotes, maybe either as a whole, or individually? I assume you went to them?

TESS: I did go to them, but, both nights, the previous night I had terrible insomnia problems, I was just not getting used to the hotel bed, and, so, I was real dreary and I couldn’t quite remember, and my nose was already, like on the screen, working on my module project, anyways. So, I actually had a hard time remembering almost anything about those talks right now.

IVAN: Oh, no!

TESS: The caffeine just didn’t kick in yet. (laughing)

IVAN: Well, I would then recommend users who are listening, or listeners, to look at the recordings on the DrupalCorn.org website, because I’m pretty sure that Kevin Thull was there, recording every single session, and most likely, the keynote is included there.

TESS: Yeah, I believe that, I did talk to him after the first day, and there was one keynote session that just didn’t record for other reasons, and the next day, there was 100% capture.

IVAN: Wow! It’s just amazing what he does for the community, isn’t it?

TESS: I always have a special place in my heart for the AV guy. (laughing)

IVAN: (laughing) Yes, absolutely. So, the schedule at DrupalCorn wasn’t just sessions, right? Wasn’t just pre-prepared sessions. There were BOFs (Birds Of a Feather) listed there as well. Did you go to any of them?

TESS: I did not. I didn’t find any of them that were that interesting. I found a lot more sessions, which were interesting, so I really wanted to go to a lot of those, and I do remember that on the first day I was there, Friday, even skipped out on an entire session period, and took over an empty BOF room, just so that I could decompress and go over my talk before giving it in the next period.

IVAN: It was probably a good thing. So, let’s talk about your session. So, you presented "Dr. Upal is in, health check your site". Can you tell us a little bit about your session, and maybe what you were hoping the outcomes of the session would be?

TESS: So, really, the session is about trying to instruct people, how to perform a technical audit of your Drupal site. A lot of the time, people who have Drupal sites, who are individual organizations, or are freelancers, might not necessarily know how to really inventory a site. There’s a variety of different circumstances. Like, you’re looking at doing an upgrade, but you don’t know what all your site has in it; you’re a new employee and you don’t know how to quite wrap your head around the site; the site was just dropped in your lap, and now it’s your problem, and you need to figure out what to do with it. So how do we really get a good sense of what a site is doing? How it’s working? Is it healthy or not? Does it have problems or not? Does it have things that will become problems in a short amount of time? What tools can we use to examine those, and how can we perceive all of this, and make it so that we know how to proceed, to get our sites back onto a track towards being healthy.

IVAN: So, the health of your website is really important, and that’s exactly what your talk sounds to be about. You talk about some tools during the session, but you also point out that there is human interaction that needs to be done to evaluate the site itself. Right? It’s not just about running tools.

TESS: Yeah, no one tool or even the entire toolbox of tools, is going to be able to tell if your site is healthy or not. Every site is a bit like a person, it’s a story. You need to figure out where it begins, how it's changed, what its plot is, what it’s twists and turns are, what characters are involved in its development, and then kind of get that whole sense of what that story is, so you could see where it’s going in the future.

IVAN: I love that idea of your site being a story. It’s kind of a live, breathing thing, that doesn’t change. Or people think that it isn’t a live breathing thing that doesn’t change, but it is, and people add content, and submit forms, and things get out of whack sometimes, so it’s not exactly the same as when you launch it. And, I love the idea that it needs a doctor, just like a human does, as well. That’s a great idea to think about. So, your session went well. I know you usually bring swag, and you give that kind of stuff out. That’s awesome. Now, there were some sessions in the schedule that caught my eye, that I thought were really interesting, that I wished I could’ve gone to, if I had gone to the camp. One was Wilbur's Docker-based Battle Royale, which is a great name for a session. The other one was, a guy by the name of Andy Olson, and he had a talk called “Audit your Theme.” Now, I know you’ve seen Wilbur’s talk, probably at TC Drupal, did you by any chance go to the other one?

TESS: I did go to the Audit your Theme one, because that was really quite interesting. Theming is not my particular specialty, and I actually really don’t know how to approach that entire discipline, of auditing a theme, and this gave me a lot of framing, for how to look at that and what to look at, and that was really, really fascinating.

IVAN: So, were there tools involved in that presentation as well, or was it kind of, a mixture of tools and human interaction? What was the, kind of the crux of that session?

TESS: So, there was a combination of tools and human interaction. We had several different tools, like we had some linters, we had some other performance analysis tools, and other CSS compiling efficiency check tools that I had, quite frankly, never even heard of, or conceived that they existed, but as soon as they were revealed to me, I was like, “of course, they would have something like that. Why didn’t I think about that until now?” And, it was really just an eye-opening experience.

IVAN: So, those were kind of two that caught my eye. What sessions did you go to that were interesting outside of Andy’s?

TESS: Oh man, I’d have to bring up a schedule, because a lot of that just got swapped out of my cache already. (laughing) Let’s see. I went to the gulp session, but the caffeine still had not kicked in yet, so that was just, I don’t remember anything at all from that. And, then that’s where I skipped a period, because I needed to practice.

IVAN: So, there were four tracks?

TESS: Yeah, there were several different tracks. I went to the Mental Health and Tech one. It is a very similar presentation to one that’s been going around, at a variety of camps and cons, but it’s really nice to see an update on that. I think he’s doing good work bringing that talk, repeatedly, to our community. I went to the Legos session, "Building your Legos, a Practitioner’s Guide to Building Reusable Components". There’s a module that’s called Legos. And, I didn’t know about this, and it was actually a fascinating counterpoint to paragraphs only sites. I thought that it was kind of fascinating, but at the same time I was like, “ahh, I have some concerns.” I think actually during the course of that talk, there was some mention about how you can’t get certain paragraphs to display with different view modes, and I know that we’ve done that before at TEN7, using a module-only solution, so. That was an interesting problem. (laughing)

IVAN: I would guess you maybe went to the Migrating Drupal 8 entities talk?

TESS: I did go to that one, and I felt kind of bad after that, because I went to it, knowing that, “ok, let’s see how this guy handles this", because unfortunately I know way too much about migration stuff. I have joked on Twitter that at some point, when you become a Drupal 8 migration expert, you start sounding a bit like a babbling prophet (laughing), instead of a developer. Talking about plug-ins and pipelines, and transformation and ETLs, like, I don’t know what any of this means any more (laughing).

IVAN: (laughing) Well, you certainly are our migrations expert at TEN7, so, I wouldn’t have it any other way.

TESS: It was generally a good talk. What was most impressive is, he smashed an awful lot of content, into a very narrow talk. I considered writing a talk about doing migrations before, because I have a blog series on this, but I decided that it was just way too much to squish in to even a 50-minute talk. It was just going to be like a fire hose, and it was a fire hose in this talk, and I believe he did a really good job of it. He used a different technique for importing paragraphs than I’m used to, so I actually put that forward at the end of the talk as an alternative.

IVAN: Very good. Now, were the sessions at DrupalCorn all the same length, or were there shorter ones and longer ones?

TESS: I think that they were all the same length this year. The big thing that was different this year, was that there was two different lunch periods, which was interesting.

IVAN: Oh, lunch one, lunch two?

TESS: Yeah.

IVAN: Really!

TESS: Lunch is actually one of the things I like the most about DrupalCorn, (laughing) mostly because, one thing that I really tended to like about DrupalCorn is that, because it’s in Iowa, there’s not much in Iowa. Even when you’re in Des Moines, it’s not the same as even being in Minneapolis, it’s definitely a smaller metropolis, and as a result, one thing that’s really kind of nice about DrupalCorn is that it has a generally, very familial vibe. You don’t feel like you’re going to a camp, you feel like you’re going to a dinner, with friends and relatives, and you’re all going to have a good time. And, that’s what it often feels like, and I really enjoy that particular feeling.

IVAN: So, basically, they had two lunches available through an extended period of time, over the course of two sessions. So, you basically don’t have one hour where there are no sessions going on. There are sessions going on throughout the day, and you choose when to take your lunch.

TESS: And you got to bring the lunches in with you to the sessions.

IVAN: Oh, you did! So, there wasn’t a lunchroom, or a common space?

TESS: Not really! There was plenty of places to sit down. There was a kind of common space, but there wasn’t many people using it. In this particular venue, they actually did have a café, that you could sit down and actually eat lunch in, and that’s where lunch was actually being served. And, some people did that, but some people also just, filled up their plates and headed off to the next session, and just sat down, and quietly munched while the session was being given.

IVAN: So, was it like a buffet style lunch?

TESS: Yeah, it usually is with DrupalCorn. One of the lunches is the traditional loaded baked potato.

IVAN: Ooh, that sounds good!

TESS: Oh, it is good! It is very good!

IVAN: Was there any corn? I guess that’s a question I have to ask.

TESS: This year, I think there wasn’t. (laughing) I could’ve just missed it. I remember the first lunch I went to, they nearly ran out by the time I was in line, because I showed up late.

IVAN: So, all in all, a good camp, kind of similar, regional camp would be Twin Cities  DrupalCamp. It feels like it’s the same amount of time, 4 days, about the same number of tracks, 3 or 4, BOFs, lunch, and good people. Any social activities in the evenings?

TESS: Yea, there was a speaker dinner on Thursday night, before the regular sessions were given, and that was wonderful. That was right in town, and it was a welcome respite from being in a car for four and a half hours. (laughing)

IVAN: Oh, my goodness! I’m sure. So, you got to mingle and meet with all the people that were giving sessions. I’m sure some familiar faces there too?

TESS: Mm hmm. And, then, the following night there was another dinner/cornhole event, that was in kind of a, it’s hard to describe what it was, it’s not really a sports bar, it felt like a rented bar/restaurant space. It’s hard to describe, but they had really, really good burgers.

IVAN: Really? So, forgive me for not understanding what cornhole is. Maybe you can give me a description.

TESS: So, first of all, imagine a board that is at an incline of, let’s say, ten to fifteen degrees off from the ground, and about three-quarters of the way up from the edge that is touching the ground, to the top of the board, is a hole, and the rest of the board is very highly glossily painted, so that it’s kind of slippery.

IVAN: Ok.

TESS: Now, you stand away from that board, at the other end of the playfield, a distance of approximately, sorry my eyes are calibrated to meters, so let’s just say 4-5 meters away, and what your job is, you have a beanbag, and you need to toss the beanbag so that it goes into the hole. And that sounds really simple, except that there’s a bit of technique to it. You can’t just toss it, and have it land directly in the hole, it’s best to kind of, put a bit of, either you have to undershoot it, but with force so it goes up the board, and down into the hole, or overshoot it, with just enough force so that once it lands, gravity will actually carry it back down into the hole, because you almost never could throw it directly in.

IVAN: So, this sounds like a game that you would play in bars, and maybe you’d get much better, or much worse, depending on how much beer you’ve had.

TESS: (laughing) That’s generally the idea, yes. I was introduced to this at the first DrupalCorn I went to, and that was a lot of fun. That camp was particularly a lot of fun, because it was in a school, and they had all of, every event, every lunch, every before event and after event, was actually in the same location, so that really made it feel very comfy, like you were just coming home to have some fun, and that was great.

IVAN: That’s awesome. And, so that was the social events Friday night. Saturday and then contribution on Sunday. Nothing on Saturday night then?

TESS: I think there was something on Saturday night, but at that point, I was kind of fried, because I’m kind of an introvert, and I just decided that, “I think I had enough,” and instead kind of staged my own little introverts game night at the hotel lobby and we got pizza, and played a card game where you’re trying to catch Jack the Ripper. It was actually a lot of fun.

IVAN: Oh my! So, there were games as well, at DrupalCorn? Awesome.

TESS: Mm hmm. I think the previous night there was also some board games after the cornhole event, but I was too fried after that too.

IVAN: Sounds like a great camp. I’m glad that you were able to go and report back to us, and I know you’re going to DrupalCamp Ottawa next week.

TESS: Oh yeah, first time going to Canada. Going to be interesting.

IVAN: Well, why don’t you remember as much as you can about the camp, and we’ll have you back on the Podcast to give us a kind of a review of that camp as well.

TESS: I will try my best.

IVAN: Well, thanks so much for spending your time with me, talking about DrupalCorn.

TESS: Mm hmm.

IVAN: You’ve been listening to the TEN7 Podcast. Find us online at ten7.com/podcast. And if you have a second, do send us a message. We love hearing from you. Our email address is [email protected]. Until next time, this is Ivan Stegic. Thank you for listening.

Oct 17 2018
Oct 17

Your browser does not support the audio element. TEN7-Podcast-Ep-041-Steve-Persche.mp3

It is our pleasure to welcome to the TEN7 podcast Steve Persch, lead developer advocate at Pantheon.

Here's what we're discussing in this podcast:

  • Steve's background
  • Celebrating a Drupal birthday
  • Theater background and blogging
  • WordPress experience
  • Improv comedy and Comedy Sports gaining self confidence
  • Experience at Palantir in Chicago
  • Contributing to Workbench
  • Discovering Git
  • Teaching WordPress' Gutenberg editor
  • What the WordPress & Drupal communities can learn from each other
  • The 2018 Twin Cities Open Source CMS Unconference
  • WordPress, Drupal & Joomla
  • Supporting Backdrop
  • Alexander Hamilton
  • Steve Vector (alias)

TRANSCRIPT

IVAN STEGIC: Hey Everyone! You’re listening to the TEN7 Podcast, where we get together every fortnight, and sometimes more often, to talk about technology, business and the humans in it. I am your host Ivan Stegic. In this episode of the Podcast, Steve Persch, lead developer advocate at Pantheon. Steve has been on Drupal.org for at least 11 years, and before Pantheon he was at Palantir in Chicago. Steve! It’s my pleasure to welcome you to the Podcast.

STEVE PERSCH: Well, thanks for having me on! I guess I just missed my Drupal birthday again. The last one I remembered was 10 years. (laughing)

IVAN: Yea, it looks like more than 11 years. That’s a long time!

STEVE: Yea! That makes sense for what I was doing 11 years ago.

IVAN: How many versions ago is that?

STEVE: Well, I got into Drupal with 5. I think 5.0 was the current version when I started working with Drupal.

IVAN: I think that’s when I started too. Let’s see, 11 years ago would be 2007, so that’s when TEN7 was founded, so that would’ve been Drupal 5 for us, as well, I think.

STEVE: Yea. I’ve worked with WordPress a little bit before that, and, the first time I remember distinctly hearing the word Drupal was at an Arts & Business council of Chicago event. I was on a panel speaking about some WordPress sites I had made for a theater company, and one of the questions from the audience was basically, “WordPress seems to work with these blogs that you’ve talked about, what would be used for something more complex?” And, someone else on the panel said, “well, obviously, Drupal is the thing for that.” I made a mental note, started looking into Drupal because I knew I wanted to be building more complex sites.

IVAN: So, tell me more about those WordPress sites that you’d been involved in, prior to that. How did you get introduced to WordPress?

STEVE: I got introduced to WordPress because, well I wasn’t asked, I volunteered myself to build some blogs for a theater company when I was in college, between my junior and senior year I had a summer internship with the Looking Glass Theater Company in Chicago. It’s a prominent theater company that came out of the college that I was going to. I was super into what they were producing, very happy to have an artistic department internship, which mainly meant things like filing headshots, and running auditions, things like that. About three weeks into the summer internship I heard one of the artistic directors just kind of, muse aloud that they really should have a blog, because that was the thing to have back then. In the nineties they had produced a physical 'zine, and this artistic director, I think, was missing that kind of creativity and wanted a blog, and I had just taken a class on web development, so I volunteered myself, and basically, that summer changed from regular artistic department internship to Steve figures out how to build WordPress sites by the end of the summer.

IVAN: And that was WordPress early on, I would imagine, with version 2 maybe?

STEVE: Yea, 2 something.

IVAN: 2 something, wow! So, you’re obviously studying some sort of art degree then?

STEVE: Yea. I was a theater major at the time.

IVAN: Wow! A theater major. And now you are a lead developer advocate. That’s almost as crazy an arc as Drew's.

STEVE:  I know. Well, I think there are a lot of people in the Drupal community with a similar arc of basically being the person at some kind of organization, who somehow became responsible for the website, or volunteered themselves to be responsible for the website, and then just kept going from there. I think for, certainly myself, and I think a lot of people in the Drupal community, the web development career path looked more appealing then whatever was the other career path.

IVAN: So, let’s actually take a step back. I want to find out where you were born, and where you grew up. So, prior to landing in the WordPress development business in liberal arts college, where did you grow up?

STEVE: Yea, so, I spent almost all my life very close to Lake Michigan. So, I was born in Milwaukee, Wisconsin, same home all the way from birth through graduating high school, and then to Northwestern University in Evanston, just North of Chicago, then moved to Chicago for seven years or so, back to Milwaukee for four years, and now Minneapolis.

IVAN: So, tried and tested Midwesterner.

STEVE: Exactly. Yea. Love the midwest.

IVAN: That’s awesome. I love it too! So, you went to high school in Milwaukee, made a move to Northwestern, and studied liberal arts. And, you did an internship at a theater company, and somehow this led to Comedy Sports, right? I saw that in your bio as well.

STEVE: Yea. Comedy sports was even earlier. So, Comedy Sports is an improv company that started in Milwaukee, there are branches all over the country, though. The way I like to explain Comedy Sports for people who aren’t familiar with it, is by referencing Whose Line is it Anyway, pretty much everyone has seen that at some point in time. And, on that show they say everything’s made up and the points don’t matter, but in Comedy Sports, the points do matter. It’s the same kind of short form improv but it’s done as a faux competition. Basically, there are three people on two different teams, the red team and the blue team, and they’re competing for audience laughs and points, and things like that.

IVAN: And this was in high school?

STEVE: It started in high school, yea. I first saw it when I was in grade school and the high school Comedy Sports team came to visit my grade school, and do a show, and I thought that looks amazing, I want to do that. It further confirmed my choice of high school. I decided to go to the same Catholic high school that my parents had gone to. My brothers decided to go to a different Catholic high school in the area, but I wanted to go to the one that had a much stronger arts program. It was also coed, that was a factor in the decision.

IVAN: So, unlike me, you saw them do improv, and you thought, cool  ! I totally want to do that! (laughing) I look at that and I’m like, “Wow, that’s really scary. I don’t know how I could do that. I should probably do that to put myself outside of my own comfort zone.” What was attractive about it?

STEVE: I think what was attractive about it for me was, knowing that it was outside of my comfort zone. That it would push me to do something that wasn’t very natural. There are a lot of people in improv who are, natural, extroverts in doing improv, and it’s just kind of a natural extension of their normal personality. For me, it was more of an effort. It took me outside of my comfort zone, but it was something that could be practiced, it was at least in Comedy Sports, something very structured, so it is, of course, loose and comedic, and you’re making things up, but, especially with short form improv, there is a lot of structure there that I could hang on to and feel like I knew what was going on.

IVAN: The structure to me sounds really interesting to be put around something that’s supposed to be unstructured.

STEVE: Yea. One thing I’ve noticed. I did improv starting in high school, age 15 or something. I was heavily involved in improv really all the way through my time in Chicago, which ended in 2014, and then coming back to Milwaukee in 2014. I did start performing at Comedy Sports again for about a year and trailed off as I started the job at Pantheon and had a baby, basically, other things became more important. And, all through that time I found that people not doing improv assumed that there must be some trick to it, that we must be planning something out in advance. But, it’s not the content that gets planned out in advance. You don’t meet backstage and say “alright, we’re going to do a scene about a baker. And, the baker’s going to go on a journey, or something like that.” You might talk about the pace at which you want to play, you might talk about the styles or techniques, the improv techniques that you want to use, but the content is always made up on the spot.

IVAN: How do you think it’s helped you with your professional career? I’ve seen you give very many talks, and I’m always impressed by how eloquent you are. I’ve seen you on panels before. They must be related to the training you’ve had in improv, at Comedy Sports.

STEVE: Absolutely! The improv training gives me confidence to get on stage without a full plan. And, maybe that’s not always a good thing. Maybe I should be stepping on stage at a conference or something with more of a plan. But, it does help just reduce that anxiety, that I know I would be feeling, and do still feel, speaking at a conference or speaking at different work events. I still feel that anxiety to some degree but knowing that I have that improv background gives me more confidence to just start, then just do it.

IVAN: So, spending time in the theater, you started getting involved in WordPress. How long were you in WordPress before that panel discussion and someone mentioned Drupal, did Drupal become something serious to you?

STEVE: About a year. So, it was the summer of 2006 where I was starting with that WordPress site launched by Fall of 2006. Maintained those sites for a little while, and basically that senior year of college was a transition for me. I finished all my graduation requirements in fall, and just kind of pretended to be an undergrad through June, continued with my improv group, produced and directed a play, did all the undergrad activities, just didn’t go to any classes other than my acting class. That then gave me more free time, to take on web work on the side. So, during that year I maintained a static HTML site, which gave me a great appreciation for server-side languages. Hearing Drew mention maintaining 5,000 static pages, I thought, “oh yea, that was formative for me too, having to maintain  the static HTML website for the theater company, that was however many hundreds of pages, and seeing there are menus that are kind of the same, but inconsistent, presumably because no one was paying too close of attention, and now it’s my job to pay attention to such things, and I’ll put it in PHP includes and eventually just switch it over to Drupal.” So, it was about a year, I guess, of thinking WordPress is the thing to switching to Drupal, as the thing that’s going to help me build what I need to build.

IVAN: And, was that about the same time that you started at Palantir? Or, was there something in between there?

STEVE: Four years in between. So, I was a freelancer for a number of years. I did some subcontracting, started with a company called Mighty Bytes, got to have those pun names. Started there in 2009 and worked on, what for me then, was the biggest website that I had ever worked on. It was a music startup. I don’t think it’s around anymore called Music Dealers. Basically, the idea was if you are an independent music producer, or band or singer, or whatever, you could upload your MP3’s to this site, Music Dealers will then try to sell that to McDonalds, or film producers, basically, anyone making commercials, televisions, movies, and they’ll just give you a direct 50/50 cut of the royalties. That was an education in complex websites, kind of, trial by fire.

IVAN: Yea. So, then you started at Palantir in Chicago.

STEVE: Two years later.

IVAN: Two years later. Ok. And, had you ever contributed to Drupal.org before starting at Palantir? How soon did you start?

STEVE: In bits and pieces. So, in that time at Mighty Bytes, I was “the Drupal guy” and I would go to Drupal meetups, and I spoke at Drupal Camp Chicago in 2008, so I did have some community role and a few minor patches, but I kind of felt intimidated knowing that there were companies out there like Palantir, who had developers who, at least by appearance, were much deeper, and Drupal knew much more than me, and looking back there were things I was working on at Mighty Bytes that I could have, should have contributed back to Drupal.org, but for whatever reason I thought, well maybe this isn’t good enough, I need to work on it more, and would never get contributed. But, at San Francisco, DrupalCon in 2010, that was the first time I distinctly remember getting that really positive, contributor feel. Someone from the White House was there and they showed a list of modules that the White House was using, and one of those modules was a module that I had a patch in, and that feeling of, “oh wow,” I mean, it was a tiny patch in rules module, but just knowing that I had contributed to a piece of something that was inside of the White House was very satisfying, and basically gave me the Drupal bug even stronger than I had it then.

IVAN: Yea, there’s something about being able to contribute code, and it actually being used in production and affecting people’s lives that really makes you stop and think about the real power of open source, and what we’ve created in this ecosystem. And so, you’re at Palantir and Workbench is this thing you start working on. Was that a problem that Palantir was trying to fix, by introducing Workbench