Upgrade Your Drupal Skills

We trained 1,000+ Drupal Developers over the last decade.

See Advanced Courses NAH, I know Enough
Jan 30 2019
Jan 30

Why not just use the .gitignore file?

Kaleem ClarksonPhoto by Tim Wright on Unsplash

As many of you know, I am a huge Pantheon hosting fanboy and can still remember the days during the beta launch of being blown away that I have three different environments out of the box, with dev, test and live. Another great service they added recently is that all sites receive SSL certificates automatically and all you have to do is redirect all traffic to use HTTPS. In Drupal 8 they suggest doing this in your settings.php file.

After adding the redirect code everything works great until you fire up your local environment (I am currently using Lando) and you are getting a blank screen. After further investigation, you notice it’s the redirect to HTTPS that is causing the issue. My first thought was to make sure my settings.local.php file was correctly being used but for the life of me, I could not get that file to override the redirect code in my settings.php file. If you are reading this and have a better idea on to how to accomplish this then let me know in the comments :)

My next thought was to simply add the settings.php file to my .gitignore file but when I went to my production website I was prompted to reinstall my Drupal site. When adding a file to .gitignore the repo pretends it doesn’t exist so therefore Drupal was telling me to reinstall. Whoooops, my production site kind of needs this file hahahah. So I thought to myself,

How can I ignore my settings.php file in my local repo but still have the original file on production?

Now that we understand what git update-index does, the real magic happens with the options that you can add to the command. In this case, the option that Ian Gloude suggested is the --skip-worktreeoption. The Git documentation explains that the skip worktree bit tells the git index to assume the file is unchanged from this point on regardless if there is an actual change. So what does this mean for us? It means you can change your file on your local environment while the original file on your production server remains unchanged.

Here is the command I use prior to uncommenting out the pantheon redirect code.

git update-index --skip-worktree /sites/default/settings.php

When I need to make some changes to the production settings.php file I can tell Git to watch the file again with this command.

git update-index —-no-skip-worktree web/sites/default/settings.php

Anyway, I hope this helps you keep your local and production environments running smoothly while maintaining your settings differently.

Nov 06 2018
Nov 06
Jody's desk

Hardware

After a long run on MacBook Pros, I switched to an LG Gram laptop running Debian this year. It’s faster, lighter, and less expensive. 

If your development workflow now depends on Docker containers running Linux, the performance benefits you’ll get with a native Linux OS are huge. I wish I could go back in time and ditch Mac earlier.

Containers

For almost ten years I was doing local development in Linux virtual machines, but in the past year, I’ve moved to containers as these tools have matured. The change has also come with us doing less of our own hosting. My Zivtech engineering team has always held the philosophy that you need your local environment to match the production environment as closely as possible. 

But in order to work on many different projects and accomplish this in a virtual machine, we had to standardize our production environments by doing our own hosting. A project that ran on a different stack or just different versions could require us to run a separate virtual machine, slowing down our work. 

As the Drupal hosting ecosystem has matured (Pantheon, Platform.sh, Acquia, etc.), doing our own hosting began to make less sense. As we diversified our production environments more, container-based local development became more attractive, allowing us to have a more light-weight individualized stack for each project.

I’ve been happy using the Lando project, a Docker-based local web development system. It integrates well with Pantheon hosting, automatically making my local environment very close to the Pantheon environments and making it simple to refresh my local database from a Pantheon environment. 

Once I fully embraced containers and switched to a Linux host machine, I was in Docker paradise. Note: you do not need a new machine to free yourself from OSX. You can run Linux on your Mac hardware, and if you don’t want to cut the cord you could try a double boot.

Philadelphia City Hall outside Jody's office
A cool office view (like mine of Philly’s City Hall) is essential for development mojo

Editor

In terms of editors/IDEs I’m still using Sublime Text and vim, as I have for many years. I like Sublime for its performance, especially its ability to quickly search projects with 100,000 files. I search entire projects constantly. It’s an approach that has always served me well. 

I also recommend using a large font size. I’m at 14px. With a larger font size, I make fewer mistakes and read more easily. I’m not sure why most programmers use dark backgrounds and small fonts when it’s obvious that this decreases readability. I’m guessing it’s an ego thing.

Browser

In browser news, I’m back to Chrome after a time on Firefox, mainly because the LastPass plugin in Firefox didn’t let me copy passwords. But I have plenty of LastPass problems in any browser. When working on multiple projects with multiple people, a password manager is essential, but LastPass’s overall crappiness makes me miserable.

Wired: Linux, git, Docker, Lando
Tired: OSX, Virtual machines, small fonts
Undesired: LastPass, egos

Terminal

I typically only run the browser, the text editor, and the terminal, a few windows of each. In the terminal, I’m up to 16px font size. Recommend! A lot of the work I do in the terminal is running git commands. I also work in the MySQL CLI a good deal. I don’t run a lot of custom configuration in my shell – I like to keep it pretty vanilla so that when I work on various production servers I’m right at home.

Terminal screenshot

Git

I get a lot of value out of my git mastery. If you’re using git but don’t feel like a master, I recommend investing time into that. With basic git skills you can quickly uncover the history of code to better understand it, never lose any work in progress, and safely deploy exactly what you want to.

Once I mastered git I started finding all kinds of other uses for it. For example, I was recently working on a project in which I was scraping a thousand pages in order to migrate them to a new CMS. At the beginning of the project, I scraped the pages and stored them in JSON files, which I added to git.  At the end of the project, I re-scraped the pages and used git to tell me which pages had been updated and to show me which words had changed. 

On another project, I cut a daily import process from hours to seconds by using git to determine what had changed in a large inventory file. On a third, I used multiple remotes with Jenkins jobs to create a network of sites that run a shared codebase while allowing individual variations. Git is a good friend to have.

Hope you found something useful in my setup. Have any suggestions on taking it to the next level?
 

Dec 13 2017
hw
Dec 13

Today, I came across an interesting bug with composer-patches plugin (it really is a git-apply bug/behavior). TL;DR, it is fixed in the latest version of composer-patches plugin as of this writing – 1.6.4. All you have to do is run composer require cweagans/composer-patches:^1.6.4 to get the fix.

The problem is simple: Patches are not applied even though you might see it in the composer log output. In fact, even a PATCHES.txt file is generated with the correct listing of patches. There are no errors and no indication that the patching failed for any reason. The problem is because git apply fails silently. It does not give any output nor sets an error exit code.

The problem was first documented and fixed in cweagans/composer-patches#165; however, the fix used there relies on git-apply outputting log messages saying patches were skipped. Unfortunately, that behavior was only introduced in 2.9.0. The Docker image I was using only contained version 2.1.4. All I needed was either these “Skipped patch” messages or an exit code and the plugin would fall back on patch command, but neither happened.

Also, this is actually supposed to be git behavior: git apply will fail to do anything when used within a local checkout of a git repository (other than the one for the project the patch is made for), such as if you are patching a module that is within a site that is in Git version control. The suggestion is to use patch -p1 < path/file.patch instead. While an obvious solution was to upgrade git, a simple Google search told me that this problem has happened to others and I decided to dig deeper. All variations of git apply simply didn't work, nor would set the exit code. I followed the debugging steps in one of the comments in the pull request, but I still wouldn’t see any messages similar to “Skipped patch” or applying patch, or any error.

In testing, I found a way to get it working with the ‘--directory‘ parameter to ‘git apply‘. The command executed by the plugin is similar to the following:


git -C package-directory apply -p1 /tmp/patchfile.patch

Instead, this command worked (even if there is no git repository involved at all, even in the current directory):


git apply --directory=package-directory apply -p1 /tmp/patchfile.patch

On digging a bit more, I found another workaround that worked for me in cweagans/composer-patches#175. What’s more, this was already merged and tagged. All I had to do was update to the latest version 1.6.4 and it would work. This change checks for the path being patched and if it actually is a git repository. If the package is not a git repository, it does not attempt a git apply at all, but falls back to patch, which works perfectly.

Nov 05 2009
Nov 05

This blog post is a by-product of my preparation work for an upcoming talk titled "Why you should be using a distributed version control system (DVCS) for your project" at SAPO Codebits in Lisbon (December 3-5, 2009). Publishing these thoughts prior to the conference serves two purposes: getting some peer review on my findings and acting as a teaser for the actual talk. So please let me know — did I cover the relevant aspects or did I miss anything? What's your take on DVCS vs. the centralized approach? Why do you prefer one over the other? I'm looking forward to your comments!

Even though there are several distributed alternatives available for some years now (with Bazaar, git and Mercurial being the most prominent representatives here), many large and popular Open Source projects still use centralized systems like Subversion or even CVS to maintain their source code. While Subversion has eased some of the pains of CVS (e.g. better remote access, renaming/moving of files and directories, easy branching), the centralized approach by itself poses some disadvantages compared to distributed systems. So what are these? Let me give you a few examples of the limitations that a centralized system like Subversion has and how these affect the possible workflows and development practices.

I highly recommend you to also read Jon Arbash Meinel's Bazaar vs Subversion blog post for a more elaborate description of the limitations.

  • Most operations require interaction with the central repository, which usually is located on a remote server. Browsing the revision history of a file, creating a branch or a tag, comparing differences between two versions — all these activities involve communication via the network. Which means they are not available when you're offline and they could be slow, causing a slight disruption of your workflow. And if the central repository is down because of a network or hardware failure, every developer's work gets interrupted.
  • A developer can only checkpoint his work by committing his changes into the central repository, where it becomes immediately visible for everybody else working on that branch. It's not possible to keep track of your ongoing work by committing it locally first, in small steps, until the task is completed. This also means that any local work that is not supposed to be committed into the central repository can only be maintained as patches outside of version control, which makes it very cumbersome to maintain a larger number of modifications. This also affects external developers who want to join the project and work with the code. While they can easily obtain a checkout of the source tree, they are not able to put their own work into version control until they have been granted write access to the central repository. Until then, they have to maintain their work by submitting patches, which puts an additional burden on the project's maintainers, as they have to apply and merge these patches by hand.
  • Tags and branches of a project are created by copying entire directory structures around inside the repository. There are some recommendations and best practices on how to do that and how these directories should be arranged (e.g. by creating toplevel branches and tags directories), but there are several variants and it's not enforced by the system. This makes it difficult to work with projects that use a non-standard way for maintaining their branches and can be rather confusing (depending on the amount of branches and tags that exist).
  • While creating new branches is quick and atomic in Subversion, it's difficult to resolve conflicts when merging or reconciling changes from other branches. Recent versions of Subversion added support for keeping better track of merges, but this functionality is still not up to par with what the distributed tools provide. Merging between branches used to drop the revision history of the merged code, which made it difficult to keep track of the origins of individual changes. This often meant that developers avoided developing new functionality in separate branches and rather worked on the trunk instead. Working this way makes it much harder to keep the code in trunk a stable state.

Having described some downsides of the centralized approach, I'd now like to mention some of the most notable aspects and highlight a few advantages of using a distributed version control system for maintaining an Open Source project. These are based on my own personal experiences from working with various distributed systems (I've used Bazaar, BitKeeper, Darcs, git, Mercurial and SVK) and from following many other OSS projects that either made the switch from centralized to distributed or have been using a distributed system from the very beginning. For example, MySQL was already using BitKeeper for almost 2 years when I joined the team in 2002. From there, we made the switch to Bazaar in 2008. mylvmbackup, my small MySQL backup project, is also maintained using Bazaar and hosted on Launchpad.

Let me begin with some simple and (by now) well-known technical aspects and benefits of distributed systems before I elaborate on what social and organizational consequences these have.

In contrast to having a central repository on a single server, each working copy of a distributed system is a full-blown backup of the other repository, including the entire revision history. This provides additional security against data loss and it's very easy to promote another repository to become the new master branch. Developers simply point their local repositories to this new location to pull and push all future changes from there, so this usually causes very little disruption.

Disconnected operations allow performing all tasks locally without having to connect to a remote server. Reviewing the history, looking at diffs between arbitrary revisions, applying tags, committing or reverting changes can all be done on the local repository. These operations take place on the same host and don't require establishing a network connection, which also means they are very fast. Changes can later be propagated using push or pull operations - these can be initiated from both sides at any given time. As Ian Clatworthy described it, a distributed VCS decouples the act of snapshotting from the act of publishing.

Because there is no need to configure or set up a dedicated server or separate repository with any of today's popular DVCSes, there is very little overhead and maintenance required to get started. There is no excuse for not putting your work into revision control, even if your projects starts as a one-man show or you never intend to publish your code! Simply run "bzr|git|hg init" in an existing directory structure and you're ready to go!

As there is no technical reason to maintain a central repository, the definition of "the code trunk" changes from being defined by a technical requirements into a social/conventional one. Most projects still maintain one repository that is considered to be the master source tree. However, forking the code and creating branches of a project change from being an exception into being the norm. The challenge of the project team is to remain the canonical/relevant central hub of the development activities. The ease of forking also makes it much simpler to take over an abandoned project, while preserving the original history. As an example, take a look at the zfs-fuse project, which got both a new project lead and moved from Mercurial to git without losing the revision history or requiring any involvement by the original project maintainer.

Both branching and merging are "cheap" and encouraged operations. The role of a project maintainer changes from being a pure developer and committer to becoming the "merge-master". Selecting and merging changes from external branches into the main line of development becomes an important task of the project leads. Good merge-tracking support is a prerequisite for a distributed system and makes this a painless job. Also, the burden of merging can be shared among the maintainers and contributors. It does not matter on which side of a repository a merge is performed. Depending on the repository relationships and how changes are being propagated between them, some DVCSes like Bazaar or git actually provide several merge algorithms that one can choose from.

Having full commit rights into his one's own branch empowers contributors. It encourages experimenting and lowers the barrier for participation. It also creates new ways of collaboration. Small teams of developers can create ad-hoc workgroups to share their modifications by pushing/pulling from a shared private branch or amongst their personal branches. However, it still requires the appropriate privileges to be able to push into the main development branch.

This also helps to improve the stability of the code base. Larger features or other intrusive changes can be developed in parallel to the mainline, kept separate but in sync with the trunk until they have evolved and stabilized sufficiently. With centralized systems, code has to be committed into the trunk first before regression tests can be run. With DVCSes, merging of code can be done in stages, using a "gatekeeper" to review/test all incoming pushes in a staging area before merging it with the mainline code base. This gatekeeper could be a human or an automated build/test system that performs the code propagation into the trunk based on certain criterions, e.g. "it still compiles", "all tests pass", "the new code adheres to the coding standards". While central systems only allow star schemas, a distributed system allows workflows where modifications follow arbitrary directed graphs.

Patches and contributions suffer less from bit rot. A static patch file posted to a mailing list or attached to a bug report may no longer apply cleanly by the time you look into it. The underlying code base has changed and evolved. Instead of posting a patch, a contributor using a DVCS simply provides a pointer to his public branch of the project, which he hopefully keeps in sync with the main line of development. From there, the contribution can be pulled and incorporated at any time. The history of every modification can be tracked in much more detail, as the author's name appears in the revision history (which is not necessarily the case when another developer applies a patch contributed by someone else).

A DVCS allows you to keep track of local changes in the same repository, while still being able to merge bug/security fixes from upstream. Example: your web site might be based on the popular Drupal CMS. While the actual development of Drupal still takes place in (ghasp) CVS, it is possible to follow the development using Bazaar. This allows you to stay in sync with the ongoing development (e.g. receiving and applying security fixes for an otherwise stable branch) and keeping your local modifications under version control as well.

I've probably just scratched the surface on what benefits distributed version control systems provide with this blog post. Many of these aspects and their consequences are not fully analyzed and understood yet. In the meanwhile, more and more projects make the switch, gather experiences and establish best practices. If you're still using a centralized system, I strongly encourage you to start exploring the possibilities of distributed version control. And you don't actually have to "flip the switch" immediately — most of the existing systems happily interact with a central Subversion server as well, allowing you to benefit from some of the advantages without you having to convert your entire infrastructure immediately.

Here are some pointers for further reading on that particular subject:

About Drupal Sun

Drupal Sun is an Evolving Web project. It allows you to:

  • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
  • Facet based on tags, author, or feed
  • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
  • View the entire article text inline, or in the context of the site where it was created

See the blog post at Evolving Web

Evolving Web