Solr for Drupal Developers, Part 2: Solr and Drupal, A History

Posts in this series:

Drupal has included basic site search functionality since its first public release. Search administration was added in Drupal 2.0.0 in 2001, and search quality, relevance, and customization was improved dramatically throughout the Drupal 4.x series, especially in Drupal 4.7.0. Drupal's built-in search provides decent database-backed search, but offers a minimal set of features, and slows down dramatically as the size of a Drupal site grows beyond thousands of nodes.

In the mid-2000s, when most custom search solutions were relatively niche products, and the Google Search Appliance dominated the field of large-scale custom search, Yonik Seeley started working on Solr for CNet Networks. Solr was designed to work with Lucene, and offered fast indexing, extremely fast search, and as time went on, other helpful features like distributed search and geospatial search. Once the project was open-sourced and released under the Apache Software Foundation's umbrella in 2006, the search engine became one of the most popular engines for customized and more performant site search.

As an aside, I am writing this series of blog posts from the perspective of a Drupal developer who has worked with large-scale, highly customized Solr search for Mercy (example), and with a variety of small-to-medium sites who are using Hosted Apache Solr, a service I've been running as part of Midwestern Mac since early 2011.

Timeline of Apache Solr and Drupal Solr Integration

If you can't view the timeline, please click through and read this article on Midwestern Mac's website directly.

A brief history of Apache Solr Search and Search API Solr

Only two years after Apache Solr was released, the Apache Solr Search module was created. Originally, the module was written for Drupal 5.x, but it has been actively maintained for many years and was ported to Drupal 6 and 7, with some major rewrites and modifications to keep the module up to date, easy to use, and integrated with all of Apache Solr's new features over time. As Solr gained popularity, many Drupal sites started switching from using core search or heavily customized Views to using Apache Solr.

Seeing this trend, hosted solutions for Solr search were built specifically for Drupal sites. Some of these solutions included Acquia's Acquia Search (2008), Midwestern Mac's Hosted Apache Solr (2011), and Pantheon's Apache Solr service. Acquia, seeing the need for more stability and development in Drupal's Solr integration module, began sponsoring the development of the Apache Solr Search module in April of 2009 (wayback machine).

Search API came on the scene after Drupal 7 was released in 2011. Search API promised to be a rethink of search in Drupal. Instead of tying to a particular search technology, a search framework (with modular plugins for indexing, searching, facets, etc.) was written to plug into the Drupal database, Apache Solr, or whatever other systems a Drupal site could integrate with. The Search API Solr module was released shortly thereafter, and both Search API and Search API Solr were written exclusively for Drupal 7.

Both Solr integration solutions—Apache Solr Search and Search API Solr—have been actively developed, and both modules offer a very similar set of features. This led to a few issues during reign of Drupal 7 (still the current version as of this writing):

• Many site builders wonder: Which module should I use?
• Switching site search between the two modules (for example, if you find a feature in one that is not in the other) can be troublesome.
• Does corporate sponsorship of one module over the other cause any issues in enterprise adoption, new feature development, or community involvement?

These problems have run their course over the past few years, and cause much confusion. Some add-on modules, like Facet API (which allows you to build facets for your search results), have been abstracted and generalized enough to be used with either search solution, but there are dozens of modules, and hundreds of blog posts, tutorials, and documentation pages written specifically for one module or the other. For Drupal 6 users, there is only one choice (since Search API Solr is only available for Drupal 7), but for Drupal 7 users, this has been a major issue.

Hosted Apache Solr's solr module usage statistics reveal the community's split over the two modules: 46% of the sites using Hosted Apache Solr use the Apache Solr Search module, while 54% of the sites use Search API Solr.

So, is Drupal's Solr community destined to be divided for eternity? Luckily, no! There are many positive trends in the current Solr module development cycle, and some great news regarding Drupal 8.

Uniting Forces

Already for Drupal 7, the pain of switching between the two modules (or supporting both, as Hosted Apache Solr does) is greatly reduced by the fact that both modules started using a unified set of Apache Solr configuration files (like schema.xml, solrconfig.xml, etc.) as of mid-2012 (see the Apache Solr Common Configurations sandbox project).

Additionally, development of add-on modules like Facet API and the like has been generalized so the features can be used today with either search solution with minimal effort.

A Brighter Future

There's still the problem of two separate modules, two separate sets of APIs, and a divided community effort between the two modules. When Drupal 8 rolls around, that division will be no more! In a 2013 blog post, Nick Veenhof announced that the maintainers of Search API and Apache Solr Search would be working together on a new version of Search API for Drupal 8.

The effort is already underway, as the first Drupal 8 Search API code sprint was held this past June in Belgium, after a successful funding campaign on Drupalfund.us.

The future of Solr and Drupal is bright! Even as other search engines like Elasticsearch are beginning to see more adoption, Apache Solr (which has seen hundreds of new features and greater stability throughout it's 4.x release series) continues to gain momentum as one of the best text search solutions for Drupal sites.