Apr 25 2019
Apr 25

My former Drupal mentor, László Csécsy (boobaa) often says "If there is a task that you need to perform more than once and it can be easily automated then it should be automated."

We would like to introduce a new tool called Rector that could speed up an upgrade process and can reduce upgrade costs. We run a proof of concept for the possible use cases in which Rector can be beneficial for Drupal developers - including but not limited to the automated update to Drupal 9.

Drupal 9 is coming and according to the promises, the upgrade from Drupal 8 will be easy because Drupal 9 is built in Drupal 8.

What does "the upgrade to Drupal 9 will be easy" promise means precisely for developers? From the drupal.org post, Plan for Drupal 9:

Drupal 9 will simply be the last version of Drupal 8, with its deprecations removed.

Releasing Drupal 9 will only depend on removing deprecated functionality and upgrading Drupal's dependencies, such as Symfony.

The Drupal 8 change records show numerous classes and methods marked as "deprecated" in every release.

Depending on the size of a module's or a site's codebase, removing all deprecated APIs can require from a couple of development hours up to months. The process can be postponed until Drupal 9 is out, or it can happen in smaller batches after every Drupal 8 minor release. Either way, keeping a codebase up to date and ready for Drupal 9 upgrade requires resources.

Identifying deprecated API usages

Yes, there are already several different ways for identifying deprecated API usages in a Drupal 8 codebase. For example, if automated tests are available on a project, developers can run deprecation testing on the codebase. Other alternatives are static code analyzers like Matt Glaman's excellent PHPstan Drupal integration or PHPStorm's built-in code inspection.

PHPstan found deprecated API usages

All these tools are great, but they can only tell what deprecated API usages a codebase contains. They leave the actual work to developers: having to fix these depreciation issues one by one.

Applying the same changes several times is definitely a repetitive task. There is a PHP library that can help to automate these kinds of upgrade processes, it is called Rector.

What is Rector?

Rector is a tool created by Tomas Votruba and according to its description on Github:

Rector is a reconstructor tool - it does instant upgrades and instant refactoring of your code.

Why is Rector awesome? Because it:

  • uses AST (Abstract Syntax Tree) to parse PHP code instead of tokens 1,
  • comes with built-in automation for upgrading to new PHP- or framework versions (ex.: Symfony, Laravel, etc.),
  • can also automatically migrate your code from one framework to another or one test runner to another,
  • can improve your code with automated code style and code quality fixes,
  • is open-source.

If you would like to learn more about Rector before you continue, Tomas created an introduction series:

Introducing Rector for Drupal 8

Rector already supported various frameworks, but it did not have support for Drupal 8 yet. This is the reason why Rector for Drupal 8 was born.

The Rector for Drupal 8 0.1.0 development version provides examples for the following possible use cases.

Instant code quality fixes and PHP 7.1 upgrade

Rector comes with numerous rector rules. Some of these rules are framework specific but most of them can run on any PHP code. The rules can be organized into prepared sets. Rector for Drupal 8 contains two prepared sets with all the built-in rectors that work on Drupal 8 codes. The drupal8 set only contains rectors that work on all PHP versions, including < PHP 7.0 versions that are still supported by Drupal core. For example: enforcing strict value comparison, normalizing conditions, removing dead code, etc. The drupal71 instantly updates any PHP 5 code to PHP 7.1 in a way that it still follows the Drupal 8 coding standards. I may include rectors in the future that make changes that require PHP 7.1 or above, ex.: add type to functions and methods parameters, etc.

Autocorrect framework specific mistakes and ensure best practices

All frameworks have their own list of best- and bad practices, so does Drupal 8. In Drupal 8, it is a best practice to use "const" instead of "define" to define constants, because it is better for performance. The DefineToConstRector can fix all constant definitions according to this best practice. As an example of bad practices, there are several traits in Drupal 8 that define a property and method with the same name. Probably the MessengerTrait is one of the most commonly used trait in Drupal core and contrib modules. Let's use this as an example. The $this->messenger property is not set by default, its value should be initialized in a class' constructor with Dependency Injection. If the property is set, then the $this->messenger() method calls the injected Messenger service. If not, then it calls the global Drupal static service container, retrieves the Messenger service from that and sets the property's value. The problem with this default behavior is that it hides an issue caused by the uninitialized property. If the property is not initialized and a valid method is being called on the property, like $this->messenger->addStatus() it causes a WSOD. Sadly, issues like this only reveal themselves in runtime. The only safe way to avoid this kind of issue is always using the method instead of the property. The PropertyToMethodCallRector rector can automatically fix any code from this kind of common pitfall.

Get rid of deprecated code without writing code

Because Tomas already implemented various kinds of automated code upgrades and framework migrations, Rector already contains rectors that can be leveraged to remove deprecated Drupal 8 API usages from a codebase. They only need to be configured. A built-in rector, called MethodNameReplacerRector, can automatically replace all usage of deprecated methods that used to generate a URL or a link for an entity. This rector can also ensure the entityTypeManager object is used everywhere instead of the deprecated entityManager object. Another common change in Drupal 8 core API is that global constants get replaced with class constants. The ConstToClassConstFetchRector rector that I created can replace all usage of a deprecated global constant with the new class constant.

Architectural refactoring

The majority of Drupal 8 API changes are more complex than simple renames. Classes, interfaces, traits, and methods get deprecated and replaced by new ones. These are the types of architectural changes where Rector can demonstrate its real power. The deprecation of the UrlGeneratorTrait is one of the best examples for a significant architectural change. A trait with a protected property and non-static methods got replaced with a class with static methods. Properly removing UrlGeneratorTrait from a Drupal 8 codebase is a risky task because several steps need to be performed and therefore it requires great concentration from a developer. However, with Rector the complete replacement process can be automated, this is what UrlGeneratorTraitRector demonstrates.

Demo time!

I configured Travis CI to run Rector for Drupal 8 on several Drupal modules. The test builds include Drupal core modules (node, system) and contrib modules like Commerce or the Apigee Edge module. The following screenshots showcase some of the fixes that Rector for Drupal 8 applied on these modules.

Rector replaced method calls to deprecated methods from the UrlGenerator trait and also removed dependency to the trait:

Rector replaced one of the most commonly used deprecated function in contributed modules - drupal_set_message():

Rector eliminated a possible bad flow caused by calling a method on a possible null property:

Rector replaced deprecated global constants with the new class constants:

What is next?

We hope this blog post gave you a good idea about the capabilities of the Drupal8-Rector project. Undoubtedly, its primary contribution to the Drupal community is the automated or at least semi-automated deprecated API removal from Drupal core and from contrib modules. The Rector library provides the framework and some good examples, but it requires a community effort to build new rectors or configure existing ones to address Drupal 8 core deprecations. There is still much work to do until all deprecated APIs since Drupal 8.0 are covered.

Please contribute to the Drupal8-Rector project if you see the potential in this proof of concept.

Apr 20 2017
Apr 20

As part of our work to make Drupal 8 the leading CMS for developer portals, we are implementing a mechanism to import the OpenAPI (formerly known as Swagger) specification format. This is a crucial feature not only for dedicated developer portals, but also for all Drupal sites that are exposing an API. Now that it has become much easier to create a RESTful API service in Drupal 8, the next step is to make it more straightforward to create its API reference documentation. That is why we think our work will be useful for site builders, and not just for technical writers and API product owners.

Swagger is a REST API documentation tool, it provides a specification language for describing the APIs and also a set of support tools. One of those tools is Swagger UI, which generates an appealing and readable layout for API endpoints and methods. The Drupal community is considering using the Swagger specification to document Drupal 8 core web services, and Swagger tool adaptations can be found in several contributed modules. In this blogpost we will introduce some of these modules and explain how we want to go beyond the shallow integration that most of them have done, to take full advantage of Drupal’s architecture. But before diving into the technical details, we want to list the features that we seek in the ultimate API reference CMS.

6 features that make the difference between a good and a great API reference system

The following are 6 technical developer portal features that customers have requested from us in the past 2 years working with Apigee’s developer portal. They provide functionality that go beyond what most API management platforms provide.

overview dev portal components

For an overview of other Developer Portal components (besides the API reference), check out our blog post series on this topic.

Read more

This feature list is based on our investigations of existing developer sites, our practical experience from creating developer portals and architecture workshops we’ve held.

1. Storing multiple API versions: versioning

As API services can have multiple supported versions (e.g. v1, v2) in parallel, a Developer Portal should provide a clear user experience that gives visitors the option to choose which version they would like to read about (but still default to the latest supported stable one).

2. Track changes in the documentation of each API version: revisioning

Most developer portal platforms rebuild the documentation as part of an automated build process, Drupal’s revisioning system allows editors and site maintainers to make and track changes in specific versions of the API documentation. While this is less important for the developers that use the documentation, it is an editorial feature that can be useful for technical writers and site owners.

3. Possibility to attach conceptual content to the API reference

API references are very technical and factual. Sometimes developers need more verbose documentation that provides a longer explanation of the context an API operates in. That is why, several of our customers have asked us to add conceptual documentation to their imported content - about domain language, underlying architecture, data models, or code samples that surround an API call.

4. Access control for individual API methods

In order to restrict the visibility of certain API methods (e.g. for partner APIs), a Developer Portal must allow site maintainers to set granular access permissions/restrictions for specific versions, endpoints or other parts of the documentation.

5. Trying out API calls on the Developer Portal’s UI

Integrating a system with an API service can be accelerated by a Try it out feature, that helps developers to decide which API endpoint with what parameters to use in order to get the expected result.

6. Importing reference documentation from a version control system

Recently technical writers have also started using the online collaboration and versioning tools that developers work with. Documentation is now often committed into code repositories, especially when developers contribute to the writing process. One key problem with this approach is that, apart from the API reference documentation where most teams use the Swagger specification, there is no obvious standard to store the content and layout of documentation. We’ve been working with markdown topics, and manifest files to allow technical writers to store conceptual documentation and their navigation structure (what we used to organize in a book hierarchy in Drupal) separate from the API specification. This way all the documentation can be stored in the version control system (e.g. a GitHub or GitLab repository).

Existing Swagger modules in Drupal

The Swagger API documentation toolset covers the entire publishing process: building (Swagger Codegen), documenting, and visualizing (Swagger Editor, Swagger UI). Existing Drupal modules typically focus on the building and visualization steps.

As usual Drupal.org has some modules that seem to be abandoned, but there are two Swagger docs related modules that have been maintained in the past year. One is the Swagger UI Field Formatter module, it renders fields with valid Swagger source file using the Swagger UI tool. The other is Swagger php module (sandbox only), it can generate JSON formatted Swagger code based on annotations and it can render that code using the Swagger UI.

Both of these modules use the Swagger UI project to generate a human readable output from the specification. Swagger UI only needs a valid source file to generate the output and the ‘Try it out’ section (for sending requests to the endpoints); it is useful if you only need to publish the content, but it has its limitations.

Free and Open Source API Documentation Tools

If you are interested in other free and open source API documentation generators, check out our blog post about other solutions.

Read more

The problem we see with this solution and most other API documentation tools is that API providers usually need access control, search, and conceptual documentation for their API descriptions. These functions demand a different approach.

Don’t just show it, integrate it!

After careful evaluation, we came to the conclusion that the currently existing Swagger tools can’t support the 6 advanced API documentation features our customers request from us. To make the API documentations fieldable, revisionable and to be able to apply custom access control on all components of them in Drupal, a more robust API integration is needed. No open-source module is available for Drupal 8 that does this, so we decided to make it a key contribution we would work on with our team.

Since there are other specification languages (such as RAML or I/O Docs) that are widely used and that store similar information as the OpenAPI format, we take great care to make sure that our architecture would be extendable and reusable.

Mapping Swagger objects into Drupal data types

To get a flexible system that can be extended and altered with proper Drupal 8 solutions, we designed custom Drupal 8 entities and field types for every piece of a Swagger source file. The first step was to observe the individual Swagger specification elements and to decide the most suitable Drupal 8 data types for storing them.

We just finished the planning phase of the entity architecture, the overall structure won’t change much, but there might be some small changes during the implementation period.

The below image describes a small part of the planned entity architecture. We defined a vendor independent API documentation as a content entity (basically the root entity) which might have bundles, providing the ability to extend the base system with vendor specific formats other than Swagger (e.g. I/O Docs or Raml). Based on this concept, each specification language format makes a new bundle with vendor specific fields. By default the Swagger 2.0 specification format bundle is provided. Each piece of content in a bundle represents a different API version, so multiple versions (e.g. v1, v2) can be made available in parallel on the Developer Portal (feature 1).

API documentation

type: content entity

has bundles: yes

label: string

version: string

...

endpoints: ER list (endpoint)

security schemes: ER list (security scheme)

[Not supported by viewer] Swagger 2.0

type: entity bundle

parent: API documentation

description: string

...

[Not supported by viewer] Consumes

type: trait

consumes: ER (MIME types)

[Not supported by viewer]

Uses

Uses Produces

type: trait

produces: ER (MIME types)

[Not supported by viewer]

Uses

Uses MIME types

type: vocabulary

Terms: application/json, application/xml, ...

[Not supported by viewer]

References

References

References

References Endpoint

type: content entity

has bundles: yes

URI: URI/string

[Not supported by viewer]

References

References

All of the documentation components are tied to the properties or references of an API documentation entity. For example API endpoints form another content entity type, which can get referenced from the root (API documentation) entity. Moreover, as we are planning to use fieldable entities, any additional information can be attached to them easily (feature 3).

Thanks to the OOP nature of Drupal 8, reusable properties and methods can be attached to entity classes through traits. For example, base field definitions of the consumes and the produces specification properties can be defined in traits and used in multiple entities, as they can be attached to the API documentation entity or to an API method/operation (overriding the default global settings of these properties). The consumes and the produces properties in the Swagger source are technically MIME types (such as application/json), so they can be collected into a vocabulary as taxonomy terms.

Thinking in traits will also enable us to extend the default API specification with custom properties (e.g. extend Swagger specification objects with ‘x-’ properties). Code snippets could for example be included for different programming languages (such as Java, PHP, Python), these might help the readers to understand the API reference.

With the above architecture we can map specification languages into a Drupal entity system where basic revisioning is supported by default (feature 2). Although custom access control can also be added to any type of entity and its fields (feature 4), it’s not as powerful as Drupal’s node access control system. There is already a Drupal core issue that tries to expand the node access grants system to a generic entity grants system, and we are trying to contribute to it while working on our Drupal Developer Portal.

Importing the data into the Drupal system

For the import process we leverage Drupal 8’s Migrate API to import any type of API specification formats to our custom entities and to store them in a unified way. Source files can be either uploaded in the UI or imported from a Github repository (feature 6) through a documentation importer that we are building to support editorial workflows that rely on code repositories and that automatically publish to Drupal as part of a continuous integration process.

If you are interested in our GitHub importer and Migrate processing solution, join our Developer Portal mailing list to receive notifications about blog posts on the subject.

Why Drupal?

We chose Drupal 8 as the framework for our Developer Portal, because it already had a large number of features that our customers need. With a 10 year long history in Drupal, we are obviously somewhat biased, but even if we disregard our prior expertise, we believe that Drupal is one of the best CMSs for building documentation and developer portals.

That is why we decided to extend the existing solutions, with a sufficiently complex system that would enable us to address all the needs our customers have. Some of our code is still in stealth mode, the developer portal market is a relatively small niche, and we need to make sure we can find a sustainable way to give back to the Drupal community. That is why in parallel to our development, we are working on a new business model for our distribution to make sure we will be able to continue sharing our work with the wider community. We are committed to the open source community and credo, but we want to prevent some of the failures we have seen with previous Drupal distributions, more about that in a later post...

Mar 23 2017
Mar 23

Introduction

Web APIs are not just useful when making headless sites in Drupal: large Drupal sites often hold valuable information that could also be useful outside the site's context. Media companies might want to expose historical media content, community sites could show data about their community activities, e-commerce sites tend to open an API for their affiliates and partners.

While it is possible to use Drupal 7 and Drupal 8 as an API backend, a lot of functionalities that describe a mature API service do not come out of the box. In this post we will explain what key concepts you have to keep in mind when designing an API service, why they are important and how APIgee Edge can make it easier to build a full-featured API webservice in Drupal successfully.

Designing APIs: the API first strategy

In a large part of the software development industry, API first thinking is replacing a user interface design approach. API first design is about planning and documenting your API before it would be implemented in code. If you set up your backend service this way, you can use it with different clients regardless of the way they were implemented. API first strategy allows you to diversify user interfaces: UI developers can work without knowing how your backend service works.

Building good backend services is not easy, there are plenty of pitfalls on the road and most of them only reveal themselves during development. Your responsibilities as a service provider grow with the number of clients:

  • maintaining the security of your services (especially if you are providing paid services),
  • handling compatibility problems between client apps and different app versions,
  • ensuring that your services are able to handle unexpected loads.

You can’t handle all of these tasks without monitoring the services. Especially for monetization, monitoring is crucial.

Features to keep in mind for building good API backend services

Security

Security is one of the most important trust signals of a mature API. A multi-layered protection system should be able to hide your non-public services from the public, handle the authorization processes, and protect the original resources from attackers.

Compatibility

Compatibility issues are the nightmares of service providers: versioning your APIs is your first step to harmony.

Scalability

Successful services have to handle an enormous number of requests every second and your services have to scale with the number of your new clients. Sometimes moving your backend to better hardware does not help, because the root of the problem is in the initial architectural decisions or implementations.

Monitoring

You need exact analytics about the usage of your API: it is indispensable for monetization purposes, plus you could use analytics data to improve your service and to understand your users' behavior.

Documentation

Good documentation is an essential part of the API service, as this is the first line of support for developers trying to understand and learn how to use the API. Developer portals often have different kinds and levels of supporting material from getting started pages to various guides, case studies, playbooks, and tutorials.

Monetization

You will need an authorization and monitoring system to efficiently track and bill customers for using your services. Exposing different resources of your APIs individually or grouped, and setting up usage limits based on these “API products” can be a time consuming task.

Companies specialized in API management solutions

You can choose from many API management technologies to build an API service, but each technology stack has its own limitations. Some companies have specialized to help you solve (a part of) the problems that might occur. Our non-exhaustive list of such companies as an example (company descriptions are from Crunchbase):

  • 3scale’s API management platform empowers API providers to easily package, distribute, manage and monetize APIs.
  • Apiphany provides API management and delivery solutions that enable organizations to leverage the mobile, social and app economy.

  • Layer 7 Technologies provides security and management products for API-driven integrations spanning the extended hybrid enterprise.
  • MuleSoft
 provides integration software for connecting applications, data and devices. MuleSoft's software platform enables organizations to build application networks using APIs.
  • Mashery is a TIBCO company providing API management services that enable companies to leverage web services as a distribution channel.

  • StrikeIron offers a cloud-based data quality suite offering web-based infrastructure to deliver business data to internet-connected systems.
  • Apigee is the leading provider of API technology and services for enterprises and developers.

The rest of this article will focus on Apigee (recently acquired by Google). Disclaimer: Pronovix is an Apigee partner, so we are somewhat biased. However, even if we wouldn’t be partners, we believe they are probably the best API management service provider for Drupal projects. They are not only a market leader in the space, they have also invested in a Drupal integration: Apigee Edge.

Apigee Edge: a Drupal integration for API services

Built in JAVA, Apigee Edge is able to replace or enhance complicated parts of your services. API proxies will protect your services from direct customer access (as they guard the backend code), and add the above mentioned 6 key features to your APIs. Apigee Edge manages these features in a specific way.

Policies

Apigee Edge enables you to control the behavior of your APIs (without writing a single line of code) via policies. A policy is similar to a module that implements a specific, limited function that is part of the proxy request/response flow. You can add various types of management features to an API by using policies.

Traffic management policies

With cache policies you can set up traffic quotas and concurrent rate limits on your API proxies.

Mediation policies

Mediation policies let you do custom validation and send back custom messages and alerts to your clients, independently from your backend services. Moreover, you do not need to implement separate xml serialization in your services to accept requests or send responses in XML, because the JSON to XML and XML to JSON policies are capable to do automatic conversions between these formats.

Security policies

Security policies give access control management to your APIs with different types of authorization methods and protection features.

Extension policies

If you haven’t found an existing policy for a special task, you can implement your own policy in Java, Javascript or Python with the help of the Extension policies, which also contain policies for external logging and statistics collecting.

Developer portals

A great developer experience is crucial for API adoption. Apigee Edge has a developer portal solution that is built in Drupal 7 with API documentation, forums and blog posts that come out of the box. API developer portals done well are your power tools to help the adoption of your API and build strong communities. Apigee's dev portals could be hosted either on cloud or on-premises with Apigee Edge for Private Cloud.

We hope this introduction gave you some insight into building high-performance API web services.

Disclaimer: When we specialised Pronovix in API documentation and developer portals we started a partnership with Apigee, and we do extensive work customising the Apigee developer portal.

Interested in our research on developer portals? Subscribe to our developer portals mailing list and get notified when we publish similar content relevant to API teams.

Subscribe >

*We will not publish, share, or sell your email address in any way.

Feb 10 2017
Feb 10

Traditionally CMSs use SQL databases that are really fast when you need all the information stored together in a record row, but are a bad fit when you need to search for relationship patterns that are not already stored together in your database. A significant performance penalty is incurred for every additional table that needs to be joined for a query. That is why SQL databases are notoriously bad at deducting relationships from datasets. Graph databases, however, are really good at this task.

In a talk we did at FOSDEM ‘17, Tamás Demeter-Haludka and I discuss potential application areas of graph databases in existing open source CMSs like Drupal.

We believe graph databases could make a big difference when used in key areas where traditionally CMSs would fail:

We've created a basic integration that makes it possible for non-developers to work with Neo4j. Combined with Drupal’s content modelling capabilities we believe it could be a powerful tool for people to explore graph databases using a GUI. There is also a case to be made for the use of graph databases in the Drupal ecosystem: sites that already use Drupal could benefit from its capabilities.

CMS as GUI of graph db

What are the possibilities if you combine graph databases with the tools and functionalities CMSs already provide?

For the video recording, click here.

Table of content:

  • Introduction (0:00)
  • Module demo (5:18)
  • CMSs as a GUI for a graph database (9:40)
  • Q&A (19:40)

Interested in AI? Subscribe to our AI & Docs mailing list, where we send notifications when we publish similar content.

Jan 13 2017
Jan 13

Intro to Neo4j

SQL databases are really fast when you need all the information stored together in a record row, but they are a bad fit when you need to search for relationship patterns that are not already stored together in your database. A significant performance penalty is incurred for every additional table that needs to be joined for a query. That is why SQL databases are notoriously bad at deducting relationships from datasets. Graph databases however are really good at this task.

Neo4j, a very popular open source graph database stores its data in a property graph, as described in Graph Databases for Beginners on the project’s website:

  • Property graphs contain nodes (data entities) and relationships (data connections).
  • Nodes can contain properties (tech lingo: key-value pairs).
  • Nodes can be labeled with one or more labels.
  • Relationships have both names and directions.
  • Relationships always have a start node and an end node.
  • Like nodes, relationships can also contain properties.

In combination with a graph query language called Cypher that is optimised for between-record pattern recognition, this storage pattern makes it easy to write queries that search for new patterns in a database and for objects that have a specified relationship with another object.

This makes graph databases excel at a number of tasks:

Why we built the module

We wanted to create a basic integration that makes it possible for non-developers to work with Neo4j. Combined with Drupal’s content modelling capabilities we believe it could be a powerful tool for people to explore graph databases using a GUI. There is also a case to be made for the use of graph databases in the Drupal ecosystem: sites that already use Drupal could benefit from its capabilities.

Drupal already had a module for Neo4j, that was originally built by Peter Arato, a former colleague. The module, developed a few years ago, however used the HTTP-only connector, and not the superior bolt connector. That is why we decided to reimplement the module, under a separate namespace.

With the bolt connector the new modules we created for Drupal 7 and 8 have a better performance. For both Drupal versions, the base module is an API module that provides settings for the connection and that supplies other modules with the connection object. This way developers can use the API module to build their own integrations (e.g. for applications that are contributed to the open source community or for integrations specific to an organisation).

Compatibility with Drupal 7 & Drupal 8

The module has support for both Drupal 7 and Drupal 8, however the versions have slightly different feature sets. Out of the box both versions have a page logging functionality that can help you discover browsing patterns of your visitors. The main difference is currently the rules support, as at this point we found it to be too much work to create a rules integration for Drupal 8. In the long term however there will be more possibilities with the Drupal 8 version, because it offers much richer APIs, allowing module developers to do even more.

At present the Drupal 8 integration only provides the default page logging functionality that stores the browsing journeys of your site’s visitors. It can however also be extended with a custom module to store other data. The Drupal 7 module has this page logging functionality and the Rules integration.

Link & Installation Instructions

The module is available on Drupal.org. For both versions, the PHP library dependencies are specified in a composer.json file. For D7, use Composer Manager, for D8, you can manage the composer dependencies globally. Consult the Drupal documentation on composer for more information.

Integration with the Rules module

Rules is a powerful Drupal module that enables site builders to define actions that will be executed when they get triggered by an event and when their conditions are met. With the help of Rules, it is possible to set up complex site behavior without having to write code. In other words, Rules turns clickers into developers. By building the Drupal connector module with a Rules integration, we’ve added a powerful content modelling tool to the Neo4J ecosystem that empowers non-coders to do complex things with graph databases.

For now the Rules integration in the Neo4J module requires you to execute a custom Cypher query as an action. In the future we could imagine creating a GUI that lets you compose the query so you could select the data sources and the way you want to store them in the database. This is something we could do if there is sufficient demand for it. At this point you can use data sources from Drupal by including the appropriate tokens as query parameters. You could for example use the {node__nid} token to refer to the currently active page context (more details in the example below). Using tokens, the Rules module allows you to hook into a wide range of processes that happen in Drupal, that can trigger almost any data available in the Drupal context to be stored in a graph database.

Try out the module

In our tutorial we’ve used the Drupal 7 module, since it has for now more interesting capabilities.

First, download the module into your local Drupal installation, and install the connector with composer. Download Neo4j, install, start it, open the web-interface on localhost:7474 and set the initial password. After you have enabled the Neo4j module, go to the configuration page, and enter the details for the connection.

After enabling the module, go to the Configuration page and click on Neo4j.

On the “General” tab, enter the credentials for your database. We recommend using the bolt protocol instead of http if it is available.

To enable page logging, check the “Page log” checkbox on the “Logging” tab and click on the “Save Configuration” button.

The page log feature will save 3 types of graph nodes: Page, User and Visit. The Page contains very basic information about the page itself. The User contains the user ID. These two are only recorded once. The Visit node contains data for the specific page visit, like the timestamp or the user’s IP address. The visit nodes are also connected for the same user, storing the user’s path on the site.

To set up the rules example, go to the rules admin page and create a new rule.

Add an event. For this example, use a content view of a content type.

When you add an action, select “Run cypher query”, and in the settings, enter the following query:

MERGE (e:Entity { entity_id: {node__nid}, entity_type: "node" })
WITH e
MATCH (v:Visit) WHERE id(v) = {visit}
CREATE (v)-[:SEEN]->(e)

This query uses two parameters. The first one {node__nid} is generated from Drupal’s replacement tokens. The second one, {visit} is a special pattern that references the id of the Visit node in the graph database (not the Drupal node), so it is easy to connect future nodes in the graph database to the current visit. To get the full list of available parameters (like {node__title} or field values), open the “Replacement patterns” fieldset.

In this example, MERGE (get-or-create) ensures that there is a node in the graph database for the Drupal node that is being visited, and that a connection is created between that node and the current visit.

After visiting a site with this plugin enabled, the graph that is created will look something like this:

How to create an integration module

The following tutorial shows how to create a simple module for Drupal 7. The Drupal 8 version of the API module is very similar, the main difference is that the Neo4j client object is available through a service, not as a global function.

Our example module will recommend content based on user data.

Create a directory called neo4j_hello_world in sites/all/modules/custom, and a file called neo4j_hello_world.info.

name = Neo4j Hello World package = Neo4j core = 7.x dependencies[] = neo4j

In the module file, create the info hook for the block

/** 
* Implements hook_block_info().
*/
function neo4j_hello_world_block_info() {
return [
'neo4j_recommendation' => [
  'info' => t('Neo4j content recommendation'),
  'cache' => DRUPAL_NO_CACHE,
 ],
];
}    

Following that, implement the block view hook

/**
 * Implements hook_block_view().
 */
function neo4j_hello_world_block_view($delta = '') {
 $block = [];

 switch ($delta) {
  case 'neo4j_recommendation':
    $block['subject'] = t('Recommended content');
    $block['content'] = [];
    if (($page_node = menu_get_object()) &&   !empty($page_node->nid)) {
    $recommended_nodes = 
_neo4j_hello_world_get_recommendations($page_node->nid);
      $links = array_map(function ($node) {
        return l($node->title, "node/{$node->nid}");
      }, $recommended_nodes);

      $block['content'] = [
        '#theme' => 'item_list',
        '#items' => $links,
      ];
    }
    break;
  }

  return $block;
}

This block only has content on node pages. The helper function _neo4j_hello_world_get_recommendations() returns the list of node objects, and the list gets transformed into links by array_map().

Finally implement the recommendation function that executes the Cypher query to get the recommended pages from the Neo4j database:

/**
* Retrieves a list of nodes that are usually visited by users who visit node/$nid.
*
* @param $nid
*   Page node id.
* @return array
*   List of recommended nodes.
*/
function _neo4j_hello_world_get_recommendations($nid) {
  $client = neo4j_get_client();

  if ($client) {
    try {
      $res = $client->run("
        MATCH (p:Page)<-[:OF]-(:Visit)-[:SEEN]->(pe:Entity { entity_type: {entity_type}, entity_id: {entity_id} })
        MATCH (p)<-[:OF]-(:Visit)-[prev:PREV*..10]-(:Visit)-[:SEEN]->(e:Entity)
        WHERE NOT e = pe AND e.entity_type = {entity_type}
        RETURN e.entity_id AS entity_id, avg(size(prev)) AS distance
        ORDER BY distance ASC
        LIMIT 10
      ", [
        'entity_id' => $nid,
        'entity_type' => 'node',
      ]);

      $nids = [];

      foreach ($res->records() as $record) {
        $nids[] = $record->get('entity_id');
      }

      $nodes = node_load_multiple($nids);

      $ordered_nodes = [];
      foreach ($nids as $nid) {
        $ordered_nodes[] = $nodes[$nid];
      }

      return $ordered_nodes;
    } catch (Exception $e) {
      watchdog_exception('neo4j hw', $e);
    }
  }

  return [];
}

In this function, first we make sure that the client is available. Then make sure that when you interact with the client object, you wrap the code in a try-catch block.

The most interesting part of this function is the recommendation-query itself. Let's have a look at the graph model again.

MATCH (p:Page)<-[:OF]-(:Visit)-[:SEEN]->(pe:Entity { entity_type: {entity_type}, entity_id: {entity_id} })

First it matches all pages for the Drupal node we're currently visiting which we already have visit nodes in the database.

MATCH (p)<-[:OF]-(:Visit)-[prev:PREV*..10]-(:Visit)-[:SEEN]->(e:Entity)

The second match matches the entities (Drupal nodes) that can be connected to the pages from the previous match with a distance of maximum 10 page visits.

WHERE NOT e = pe AND e.entity_type = {entity_type}

The WHERE statement filters the results from the previous matches, excluding the current entity, and narrowing the results to nodes.

RETURN e.entity_id AS entity_id, avg(size(prev)) AS distance

In the RETURN statement, a simple weight is generated, from an average of the distance between the entities. This weight will be too simple for most real world applications, but to keep things simple, it is sufficient for the scope of this example.

ORDER BY distance ASC
LIMIT 10

Finally order by the distance and take the first 10 items.

In an example graph starting from Entity 1 on the left, all these Entities (2,3,4) on the right would be returned as recommendations:

After the query, a foreach loop takes a list of node ids, and with a node_load_multiple() we load the nodes from the database that are then resorted.

Closing Thoughts

These were very basic examples of how the module can be used to start building a graph database. Please try it out and let us know how you would like to use the module, we would love to hear about any applications you think this could be used for. Also if you run into any errors you can report them in the project’s issue queue on Drupal.org.

Neo4j is already used in production on Drupal sites, but all those implementations were one-off integrations. We hope that this project could be the start of a community effort for a full featured Neo4j integration that enables even complex use cases without much coding.

We at Pronovix are a Drupal consultancy, if you need a deeper integration, or if you would like to extend the functionality of the module please let us know us. We are specialised in documentation systems and developer portals, systems that need to deliver content in the context of a user’s activities and problems.

That is why we started working with Neo4J because we want to use it to deliver personalised help content proactively. We would like to become the go-to company for organisations that need help with their Neo4j project on Drupal. That is why we invest in this module and actively share knowledge about interesting problems graph databases can solve in the CMS world. If you want to get notified when we publish a new blogpost, sign up for our Documentation Automation & Personalisation, AI, And Graph Databases mailinglist.

If you want to chat with us, you can find us on the #neo4j-drupal channel in the neo4j.com/slack. Our company’s site is at https://pronovix.com. You can also contact us through our contact form, or connect with Kristof on Linkedin.

Thanks to my colleague Tamás Demeter-Haludka for co-authoring.

Jun 09 2016
Jun 09

When a problem arises in today’s attention deprived world, we can’t expect our customers to go search for a solution in the documentation. Documentation sites can be complex, they often store the documentation for multiple products and each of the products might have many features. In the best case you will frustrate your customer, in the worst case they might simply give up before finding the information they need.

That is why we developed the Walkhub Help Widget to help site owners guide their visitors to the right documentation content - right inside an application or web site. The simplest way to try out WalkHub Help Widgets is embedthedocs.com, but you can also set up your own widget creation interface easily if you have basic knowledge of web development. In this blogpost, we will guide you through the steps you need to take to replicate the functionality of EmbedTheDocs using Drupal as a content store.

EmbedTheDocs, the free UI for creating Help Widgets

In April we launched EmbedTheDocs.com, a service that lets you create an embeddable Help Widget. Our goal was to make it so simple that you should be able to create one under 2 minutes. Our widget allows users to list a series of online help content (such as technical documentation, articles, blog posts, videos, walkthroughs... anything linkable) relevant to a user’s context, that will open in the same browser window in a separate iframe. Once created on embedthedocs.com, the widget can be placed on any site via its unique embedcode. Anyone can freely try it out on our site after a quick registration.

Step-by-step tutorial: generating widgets from your own Drupal site

EmbedTheDocs is sufficient if you only need to create a few Help Widgets. In case you need something more complex, you want to have your own interface, add custom functionality to your widgets, or generate the content for the widget automatically in reaction to the user’s context, this article is for you.

As the EmbedTheDocs site is written on top of Drupal 8 CMS, it is possible to replicate the widget creation feature on other Drupal 8 sites, too. In this section, you will be guided through the process of replicating the widget creation feature, as on embedthedocs.com.

The following steps can be implemented either on an existing Drupal 8 site, or on a newly installed instance. If you don’t have a Drupal 8 site yet, you can install Drupal 8 on your localhost or a dedicated server (we recommend to test the features first on a local server) by following the standard Drupal installation instructions.

I. Setting up the widget elements with Paragraphs

The WalkHub Help Widget contains 3 different types of elements: groups, links and walkthroughs. Groups can be used as dividers for links and walkthroughs, but it’s not obligatory to add one. Links can target any content on the web. The YouTube video links will receive a YouTube icon in the widget and the video goes full screen when you click on its title. The third element is the WalkHub walkthrough, which can be added to the widget using its unique URL after you created one on next.walkhub.net. It’s free to add as many walkthroughs as you want on this site. Also, as WalkHub is open-source, you can fork or download it from GitHub.
Paragraphs module is a very powerful implementation of Drupal’s entity system. With the pre-defined paragraph types it is easy to compose content (e.g landing pages). So you could create other types of paragraphs that let you add different elements to your widgets (e.g. a video paragraph, with a proprietary video not hosted on Youtube).

Example widget with groups, links and walkthroughs
  1. Install the Paragraphs module with its dependencies (if you have Drush, use drush en paragraphs), go to /admin/structure/paragraphs_type and click on Add paragraph type. Type “Group” in the label field, then click on Save and manage fields. Group paragraph type
  2. Add a new plain text field with the label “Group name”. Then save the settings of the paragraph type. Group name field in group paragraph type
  3. Navigate back to /admin/structure/paragraphs_type where you can now see your first paragraph type listed. Paragraph type list with group
  4. Next create the second paragraph type. This paragraph’s label should be “Link” and will contain 3 fields: description (plain text), link (link) and title (plain text), the last two fields have to be required. The field list of the link paragraph type should look like this: Field list of link paragraph type
  5. The last paragraph type’s label is “Walkthrough” with one “Walkthrough link” field (link), which has to be set as required. Field list of walkthrough paragraph type
  6. Now you have all three necessary paragraph types: Group, Link, Walkthrough. Paragraph type list

Important! When adding the paragraph types and their fields, the machine names play key roles. The system will only work properly if you use the correct machine names for the paragraph types and fields.

II. Creating the Widget content type

  1. Go to /admin/structure/types and click Add content type. The new content type’s name should be “Widget” (the other settings can remain in the default state). Add widget content type
  2. Delete the automatically added “Body” field, and add a new one of the Paragraph type and label it “Widget elements”. The allowed number has to be unlimited so that users can add as many help items to a widget as they want to. Widget content type settings
  3. No other fields are needed for the widget creation, so let’s add a new node of this content type. Go to node/add/widget and feel free to add as many Widget elements as you want. The result should look like the following:
New widget node

III. Making the widget come alive

If you followed the steps in chapters I and II, you should now have a widget content type with one multiple value field for the three paragraph types. In the third chapter of this tutorial we install a complementary module for adding the content processing feature.

  1. Install the auxiliary module from https://www.drupal.org/sandbox/kittiradovics/2738613 into /module folder or in any subfolder of /module. Get more info on installing sandbox projects.
  2. Enable the Help Widget data processor module via /admin/modules, then go back to your previously created test widget node. This module’s first main function is to transform the paragraph elements’ content to a special JSON format, which can be processed by the WalkHub server in order to generate the widget iframe from the widget node. The Help Widgets are generated with the assistance of WalkHub, a microservice written in GO and React.js. The Drupal 8 site generates the resource in JSON, then WalkHub processes and displays it. For more information on the relationship, check our previous blogpost on EmbedTheDocs. You can also set up your own WalkHub server so that you can create walkthroughs on a system you control. This also lets you use your own server to display the widget iframe instead of ours.
  3. The content’s URL looks like /node/{nid} (replacing {nid} with the content’s id). If you modify this url to /json/{nid}, you will get the raw JSON formatted output of the test widget content (processable by the WalkHub server). The JSON output of the example is the following: JSON output of widget node The output is generated by helpwidget_data_processor/src/Controller/JSONOutputController.php. This file fetches the paragraph elements and their fields and pushes the text contents in an array. Then the JSON formatted output is generated by the JsonResponse class. The path for the text is defined in helpwidget_data_processor/helpwidget_data_processor.routing.yml file.
  4. In order to get the WalkHub server to process the JSON formatted output and render the widget, an embed code is needed. Go to admin/structure/block, scroll down to the Content region and click on the Place block button, choose Generated embed code block. You can use the default settings. Place Generate embed code block The block is created by helpwidget_data_processor/src/Plugin/Block/EmbedCodeBlock.php, which composes the embeddable code and also inserts it into the page’s source code so that the user can see the result after every modification in the bottom right corner of the widget node.
  5. The WalkHub server uses CORS headers when receiving the JSON formatted output. Drupal has an excellent module for sending CORS headers. Download and install the CORS module (or use drush en cors) and go to /admin/config/services/cors. Paste the following string in the domains field: *|http://next.walkhub.net|OPTIONS,POST,GET|Origin,Content-Type,Accept,X-CSRF-Token|true CORS module settings
  6. Go back to your widget node, and now you can see the embeddable code and the rendered widget in the bottom right corner. If you copy and paste the embed code into any of your sites’ HTML source, you will get the same button as on the widget node’s page.

If the result is not as expected, try to solve it by clearing the Drupal cache in /admin/config/development/performance. Feel free to extend the features or theme the widget node’s page.

Got stuck somewhere? Do you need help? Contact us for support!

How to choose between Embedthedocs.com and your own Drupal 8 site?

EmbedTheDocs.com is a convenient way to try out what the Help Widgets offer. If what you need are only a few widgets, EmbedTheDocs.com will suffice.
Your own, private Help Widgets system can offer some extras that proprietary and/or confidential materials may even require:

  • Store your help contents on the server of your choice, with your security and permission settings.
  • Have full control over access and management of your custom data stored in the Drupal system.
  • Customize the appearance of the widget nodes (Drupal enables easy theming).
Jun 07 2016
Jun 07

Part 1: Architecture summary
Part 2: Installation & Configuration
Part 3: Video & Playlist Management
Part 4: Including Videos & Playlists in Drupal content

In this multi-part blog post, we are covering the various aspects of the new Brightcove Video Connect module for Drupal 8. The first part consisted of an architecture summary and discussed the technical approach used during the development of the module. The second part detailed the installation & configuration steps required to get the module up and running properly. The third part illustrated the management of videos & playlists and discussed some of the changes compared to the Drupal 7 version of the module. This fourth part demonstrates how to add Brightcove videos & playlists into Drupal content.

This posts reads as a step-by-step guide for the following tasks:

Add Brightcove Video field

We will go through step by step how to add Brightcove videos into Drupal content. The same steps apply to Brightcove playlists.
Once we created the Brightcove Video and Playlist entities (or synchronised Drupal with Brightcove in case of an already-existing Brightcove account), we can include them in any other content type. We can do this via a new field: we add a new field to the chosen content type referring to the Brightcove Video or Playlist Entity.

Navigate to Structure > Content types and choose Manage fields.

Content types

Add field.

Manage fields

From Add a new field select Reference / Other.... Label this new field, then Save and continue.

Manage fields save

From Type of item to reference select Brightcove Video. Choose the Allowed number of values from the drop-down, then Save field settings.

Manage fields save field settings

You can edit further settings and add help text. Once done, Save settings again.
Our new field is ready to go: we can now include Brightcove videos in our content. To do this, we navigate to Content and Add new content (or select the existing content) of the type where we just included the Brightcove Video field.

Autocomplete field

By default, our new field uses the Autocomplete widget - so we can browse videos by starting to type in their names.

What if we have hundreds of videos on our Brightcove account? Then we might have trouble selecting them using the standard autocomplete widget for entityreference fields. Entity Browser module to the rescue! With this module, we can create our custom widget via Views to browse our videos. Let’s see how it works!

Browse Brightcove Videos using the Entity Browser module

First we need to download and enable the Entity Browser module: in Extend, find and choose Entity Browser via its checkbox then Install. You can find instructions on how to install modules in the Drupal 8 documentation.

Now we can set up our custom view that we will use for browsing our videos. Navigate to Structure > Views and Add new view. Give View name. From View settings select to show Brightcove Video then Save and edit.

In this example we will use the Table format and we Add the following fields: Name, Created, Thumbnail and Entity browser bulk select form. To add this Entity browser bulk select form is very important, otherwise we cannot select our videos. The rest of the fields are all a matter of taste, you can add whichever fields you need.
For now, we will use here a mini pager and display 3 items per page.

Click +Add in the Displays section and choose Entity browser, then Save.

Then choose Add in the Fields section and choose Entity browser bulk select form. We are automatically switched to the form's configuration. Once you applied your settings, Save the Display section again.

The next step is to set up the entity browser widget, which we will use instead of the standard autocomplete widget for browsing our Brightcove Videos. This form depends on Chaos tool suite (ctools) module, so make sure it is installed and enabled. Navigate to Configuration and select Entity browsers in the Content Authoring section.

Click Add Entity browser.

Enter a Label for the entity browser. Select Modal from Display plugin. Select the Single widget selector plugin for now - we will discuss another solution in the browse and upload new Brightcove video section!
Choose No selection display as the default Selection display plugin. Feel free to play around with the other options. Click Next.

Next, we get to the configuration settings for the modal display plugin’s width and height - we can go with the default settings. Give a Link text for the button link and then go to the Next step.

There are no available configuration options for the next two steps (Widget selector and Selection display), so just click Next in both cases. In the final configuration step, choose View from the Add widget plugin dropdown. Give it a Label. Finally, select the newly created entity browser from the View display list, then hit Finish.

We are almost done! We still have to tell Drupal that we want to use our entity browser (instead of the default autocomplete) to select our videos for the Brightcove video field (in whichever content type we included it in).
Navigate to Structure > Content types and this time choose Manage form display from Operations on your relevant content type.

Change the widget settings: choose Entity browser instead of Autocomplete for our freshly created Brightcove video field.

There is no entity browser selected yet. We select via the configuration icon of the field.

Select Brightcove video browser widget (or whatever name you chose for your entity browser) as the Entity browser. Choose Entity label for Entity display plugin. Update and then Save settings.

Our Brightcove video browser widget is all set up and ready to be used for adding videos into our Drupal content.
Navigate to Content and Add content (or select an existing one) of the type that includes the Brightcove video field.
We can see the new Select videos button (that is the label we gave), instead of the autocomplete field.

Let’s click on Select videos: we get a pop-up where we can browse our uploaded videos more conveniently. We see the fields we set up in our custom widget view for each video, in this example that was Name, Created, Thumbnail and we get a checkbox for each. You select via checkboxes then affirm by clicking Select videos.

After selecting the videos for the Drupal content, we can Remove and Edit each. To finalize this video addition to the content, Save and publish.

Upload Brightcove Videos from Drupal content using the Inline Entity Form module

It's even possible to upload Brightcove Videos and Playlists into your Brightcove account from Drupal content with the Inline Entity Form module. This feature is already included in the Brightcove module, we just need to do some configuration.
Navigate to Structure > Content types and choose Manage form display from the Operations on the video-containing content type.

Instead of the Entity browser widget, this time we want to choose the Brightcove Inline entity form - Complex widget. It is preconfigured, we only have to Save.

Navigate to Content and Add content, the Brightcove video field now shows the Add new brightcove video button.

Click on it, and we can see the same interface showing up as in part 3's Video upload and edit chapter.

Follow the steps in the “Video upload and edit” section then Save. We are done, the video is included in our content (both in Drupal and in Brightcove, as they are synced).

What if we want to be able to browse existing videos and upload new ones in the same field? Is that possible in D8? Yes, it is. All we need to do is to enable the Entity Browser IEF (Inline Entity Form) submodule, and do some configuration. Let’s see how it works!

Browse and upload from the same field in your Drupal content, using the Entity Browser IEF submodule

Navigate to Extend and enable the Entity Browser IEF submodule.

Navigate to Configuration and select Entity browsers from Content Authoring. It is a prerequisite to have the entity browser, you find its setup guidance in the Browse Brightcove Videos using the Entity Browser module chapter of this blog post. In the following steps we will do some changes to that configuration.
Choose to Edit the entity browser.

Change the Widget selector plugin from Single widget to Tabs.

Hit Next until you get to the last configuration page (before Finish). In the Add widget plugin choose the now available Entity form. Give it a Label then change its Entity type and its Bundle both to Brightcove Video. Click Finish.

Navigate to Structure > Content types and choose Manage form display from Operations on your relevant content type. Change the widget settings to Entity browser, the same way we did in the Browse Brightcove Videos using the Entity Browser module chapter of this blog post.

Now navigate to Content and Add content (or select an existing one) of the type that includes the Brightcove video field and click on the Select videos button.

We can see that the new Upload tab appears next to the Select tab, and if we click on it the same form comes up like in this blog post's part 3's Video upload and edit chapter. We fill the form and then click Select videos in the end.

And that’s it, we can select our existing videos and upload new ones in the same place. We can even do this from CKEditor!

Browse and upload Brightcove Videos from CKEditor

Download and enable the Entity Embed and Embed modules, and of course the CKEditor module if it is not enabled yet.

Navigate to Configuration and in the Content authoring section, select Text editor embed buttons.

Choose to Add embed button.

Enter a Label for the embed button. In Embed type select Entity. In Entity type select Brightcove Video. Select the entity embed display plugins that you want to allow - we will leave it empty to allow them all. In Entity browser select the browser you configured. If it is not created yet, please follow the instructions in the Browse Brightcove Videos using the Entity Browser module chapter of this blog post. Upload Button icon for easier distinction.

Before we continue, we need to adjust our entity browser. We must change the Display plugin from Modal to iFrame. CKEditor dialogs are always modal and currently Drupal 8 cannot handle subsequent modal calls, which is a known issue - until that's fixed, we use the iFrame Display plugin.
Navigate to Configuration > Entity browsers in the Content authoring section and choose to edit the browser widget. Change the Display plugin to iFrame. Click through Next until the final configuration page then hit Finish.

Now go back to Configuration and this time choose Text formats and editors in the Content Authoring section.

Choose to Configure the text format where you want to add the entity embed button. For Text editor set CKEditor. In the Toolbar configuration section drag and drop the previously created button from the Available buttons row to the Active toolbar. You can create a new group for it if you want to, like in this example we added the Entity browsers group. In the Enabled filters checklist opt in Display embedded entities then Save configuration.

Navigate to Content > Add content, choose your content type. Give it a Title. In the Body section, select the Text format where you previously added the Entity Embed button. Click on the embed button (if there are more buttons, hovering over them will reveal their functionality).

A modal display will show up to select videos.

We can browse our existing videos and choose them via their checkboxes.

Alternatively, we can upload new Brightcove videos the same way as we did in the Browse and upload from the same field in your Drupal content, using the Entity Browser IEF submodule chapter of this blog post.

Once the video(s) are selected, we can set the way we want to display it (Display as), the View mode, Align_ment and we can also add a _Caption. The default view mode on these configuration options usually contains all properties of the video. We can customize this by creating a new view mode for the Brightcove Video entity then configuring its display. When everything is set, hit Embed.

Our Brightcove video is embedded in the text field, we can Save and publish our content.

This fourth part concludes the blog series. We hope the information provided will prove useful to everybody setting up Brightcove Video Connect in Drupal 8.

Part 1: Architecture summary
Part 2: Installation & Configuration
Part 3: Video & Playlist Management

May 27 2016
May 27

Part 1: Architecture summary
Part 2: Installation & Configuration
Part 3: Video & Playlist Management
Part 4: Including Videos & Playlists in Drupal content

In this multi-part blog series, we are covering the various aspects of the new Brightcove Video Connect module for Drupal 8. Part 1 consisted of an Architecture summary and discussed the Technical approach used during the development of the module. Part 2 detailed the Installation & Configuration steps required to get the module up and running properly. This third part illustrates the management of Videos & Playlists and discusses some of the changes compared to the Drupal 7 version of the module.

Video listing

Once videos have been “imported” from Brightcove or uploaded through Drupal, a listing becomes available in an intuitive interface (with a similar layout to Brightcove’s own Studio interface) located in a tab right next to Drupal’s main Content tab:

Brigthcove Videos list in admin_content

The new listing pages allow you to manage videos in a streamlined manner - something that wasn’t possible in the previous Drupal 7 versions as the video listings were limited and only accessible through an overlay while Attaching a video to a node.

Video upload and edit

When uploading a new video (or editing an existing one), a major improvement over previous Drupal versions is that the upload/edit screen is no longer presented in an overlay - but in the same layer as the other configuration screens:

Add/Edit Brigthcove Video

As can be seen above, tags now include an autocomplete feature. The Related Link URL now also includes autocomplete, letting you search for Drupal nodes:

Autocomplete related link URL

The video upload section has been improved by allowing a wider variety of audio and video container formats, and both the video and image source files can now be replaced with new files on existing assets:

Video upload section

The Availability section has also been improved with a new calendar dropdown:

Video availability section

Finally, not only can videos be uploaded and edited in an efficient manner, but also deleted (deleting Brightcove videos wasn’t possible in the previous Drupal 7 versions):

Video delete

Playlist listing

A listing similar to the Brightcove Video is also available for Playlists:

Playlist listing

Smart Playlist edit

Tags are also autocompleting in the Smart Playlist edit screen as can be seen below:

Smart Playlist tag autocomplete

Manual Playlist edit

Manual Playlists also feature an autocompleting field - this time for the individual videos to be included in the playlist:

Manual Playlist tag autocomplete

Part 4 of this multi-part blog series will demonstrate how to include Brightcove Videos & Playlists in Drupal content.

Part 1: Architecture summary
Part 2: Installation & Configuration
Part 4: Including Videos & Playlists in Drupal content

May 19 2016
May 19

Part 1: Architecture summary
Part 2: Installation & Configuration
Part 3: Video & Playlist Management
Part 4: Including Videos & Playlists in Drupal content


In this multi-part blog series, we are covering the various aspects of the new Brightcove Video Connect module for Drupal 8. Part 1 consisted of an Architecture summary and discussed the Technical approach used during the development of the module. This second part details the Installation & Configuration steps required to get the module up and running properly.

The only element in common between the 8.x-1.x version and the previous 7.x-6.x version of the Brightcove module is the PHP-API-Wrapper library used to assist in the communication between Drupal and the Brightcove API. Installing the PHP library in the previous 7.x-6.x version required downloading and extracting the library into sites/all/libraries, followed by a “composer install.” This is no longer necessary in the 8.x-1.x version because the PHP library is included as a dependency in the module’s composer.json file.

Installation steps

The following steps require Drush and will install the Brightcove module as well as all of the module’s dependencies, including the PHP library...

1. Download the Brightcove module $ drush dl brightcove

2. Install the library dependency (PHP-API-Wrapper)

The recommended method for installing the library dependency is to reference the Brightcove module's composer.json inside the extra section of the root composer.json file as described in Managing dependencies for a custom project.

After adding the brightcove/composer.json path to the merge-plugin section (in the extra section) of the root composer.json, it should look similar to this:

        "merge-plugin": {
            "include": [
                "core/composer.json",
                "modules/brightcove/composer.json"
            ],
            "recurse": false,
            "replace": false,
            "merge-extra": false
        }

After the extra section has been modified, run composer update in the root folder to install the required libraries. (note: composer is required)

For more information, see Using Composer with Drupal

3. Enable the Brightcove module (this will also install all dependencies) $ drush en brightcove

Configuring permissions

The module includes a list of permissions which should be configured if non-admin users will require privileges to manage the Brightcove API credentials, videos or playlists:

configuring permissions to manage Brightcove API credentials, videos or playlists

Configuring API credentials

Once the module and all it’s dependencies have been installed, you can enter your Brightcove API credentials into the Brightcove API Clients configuration page - which is linked from Drupal’s Configuration page:

Brightcove API Clients configuration page

The Brightcove API Clients page lists the configured API credentials and allows you to add new credentials:

Brightcove API Clients list

After clicking Add Brightcove API Client above, the following screen will appear where you can enter your Brightcove Client ID, Secret Key and Account ID:

Brightcove API Clients new credentials

If the Client ID, Secret Key and Account ID entered above pass verification from Brightcove, then the API Client will be saved successfully:

Brightcove API Clients successfully added new client

Synchronization

The most significant difference between the Drupal 8 and previous versions of the module is that videos are now synchronized between Brightcove and Drupal, using Brightcove as the source of truth. This allows users to “import” all of their existing Brightcove videos into Drupal after the module is configured. Not to worry, the actual video files will not be downloaded from Brightcove into Drupal - only the metadata and images associated with the videos. The synchronization option is available in the Brightcove Status Overview page which is linked from the Drupal Reports page:

Drupal Reports page

The Brightcove Status Overview displays the synchronization status of all the various entities, and also includes buttons for managing the queues:

Brightcove Status Overview via Reports page

Clicking “Sync all” above will synchronize all listed Brightcove entities into Drupal - which is useful in order to do an initial batch sync. Synchronizing the entities manually is not necessary, however, as the synchronization will run in the background through Drupal’s cron.

Brightcove Status Overview via Reports page after sync

Part 3 of this blog series will illustrate the management of Videos & Playlists and discuss some of the changes compared to the Drupal 7 version of the module - especially the new video listing interface (with a similar layout to Brightcove’s own Studio interface) and autocompleting tags feature.


Part 1: Architecture summary
Part 3: Video & Playlist Management
Part 4: Including Videos & Playlists in Drupal content

May 08 2016
May 08

In this multi-part blog series, we will be covering the various aspects of the new Brightcove Video Connect module for Drupal 8. This first part consists of an Architecture summary and the Technical approach used during the development of the module.

Architecture summary

The Drupal 8 version of the Brightcove Video Connect module was written from scratch in order to take advantage of the architectural changes in Drupal 8, especially the new Entity Data Model. Drupal 8 core and its contrib modules handle many of the functions that had to be custom-written for the Drupal 7 module. This new Drupal 8 module actively utilizes the Inline Entity Form module and there is support for Entity Browser, therefore fitting nicely into Drupal 8's entity system architecture. Since Drupal 8 core also includes well rounded Views support out of the box, most of the module development was attributed to the module’s data synchronization via Brightcove’s API.

Designed around the new entity system in Drupal 8, the new Brightcove module seamlessly integrates video publishing into the Drupal editorial workflow and interface. This alleviates the fragmented editorial experience typically associated with 3rd party video hosting services.

Technical approach

The Drupal 8 version of the Brightcove Video Connect module sports a number of config and content entities. The three major building blocks are:

  • API Client config entity: holds all the data needed to connect to the Brightcove API utilizing the PHP wrapper library.

  • Brightcove Video content entity: a required entity reference to the API Client config entity, all the data that is supported by the Brightcove API and the PHP wrapper library but the video file itself, and possibly other Drupal-only fields as this entity type is (Drupal-) fieldable.

  • Brightcove Playlist content entity: a required entity reference to the API Client config entity, all the data that is supported by the Brightcove API and the PHP wrapper library, a multiple-value entity reference field to Brightcove Videos in case of a manual playlist, and possibly other Drupal-only fields as this entity type is also (Drupal-) fieldable.

There are various other smaller content entities, but they're mostly hidden from the user, such as:

  • Custom field content entity: this stores the definitions about the custom fields for each API Client. These fields are dynamically generated on the Video entity’s form, and handled through API calls, like the default value, description or the saved value. The custom field values are also stored on the Video entity.

  • Player content entity: Stores some minimal information about the player config, like width and height. These values are applied to the player display if possible.

  • Text Track entity: Holds all metadata about the text tracks, after the ingestion the uploaded file is removed.

Synchronization of content between Brightcove and Drupal is a two-way thing.

  • When a change is made to a Brightcove-related Drupal entity on the Drupal site/side, the module pushes those changes to Brightcove immediately.

  • The module has a built-in mechanism to "download" changes from the Brightcove site to the Drupal site, which may occur by three different triggers.

    • First of all, the module uses Drupal's cron system to download changes from Brightcove to Drupal when the last change dates are different on any given content entity. When all changes have been downloaded to the Drupal site, the synchronization starts over again.

    • Secondly, the site administrator can initiate finishing the current synchronization in a batch process, initiate a completely new synchronization, or stop the currently running one from the admin interface (see below).

    • Thirdly, when the user wants to edit a Brightcove-related content entity on the Drupal site and the last change date of that entity is newer at Brightcove, a message is shown to the user with a link that downloads the latest data from Brightcove to Drupal before the user makes changes to it. It also checks if the video was deleted from the Brightcove studio, in this case the user can delete the local version or recreate it (however the video file won’t be available).

  • For more detailed information about the content synchronization from Brightcove to Drupal, see the comment for the _brightcove_initiate_sync() function in the code.

For various tasks, several already-existing Drupal (contrib) modules are used.

  • The Time Formatter module is nothing more than displaying a number of seconds (either an integer or a float) in fancy formats like "16h58m17s".

  • The Entity Browser module can be used eg. for selecting videos for a manual playlist.

  • The Inline Entity Form module is used for creating and storing informations about the text tracks, currently only new text tracks can be created or deleted, but it is not possible to edit them for now through Drupal.

The new Drupal 8 module was developed by Tamás Nagy and me, László CSÉCSY.
Jan Mashat was the project manager and László Brindza was responsible for testing.

May 08 2016
May 08

In this multi-part blog series, we will be covering the various aspects of the new Brightcove Video Connect module for Drupal 8. This first part consists of an Architecture summary and the Technical details surrounding the config and content entities.

Part 1: Architecture summary
Part 2: Installation & Configuration
Part 3: Video & Playlist Management
Part 4: Including Videos & Playlists in Drupal content

Architecture summary

The Drupal 8 version of the Brightcove Video Connect module was rewritten from scratch in order to take advantage of the architectural changes in Drupal 8, especially the new Entity Data Model. Drupal 8 core and its contrib modules handle many of the functions that had to be custom-written for the Drupal 7 module. This new Drupal 8 module actively utilizes the Inline Entity Form module and there is support for Entity Browser, therefore fitting nicely into Drupal 8's entity system architecture. Since Drupal 8 core also includes well rounded Views support out of the box, most of the module development was attributed to the module’s data synchronization via Brightcove’s API.

Designed around the new entity system in Drupal 8, the new Brightcove module seamlessly integrates video publishing into the Drupal editorial workflow and interface. This alleviates the fragmented editorial experience typically associated with 3rd party video hosting services.

Technical details

The Drupal 8 version of the Brightcove Video Connect module sports a number of config and content entities. The three major building blocks are:

  • API Client config entity: holds all the data needed to connect to the Brightcove API utilizing the PHP wrapper library.

  • Brightcove Video content entity: a required entity reference to the API Client config entity, all the data that is supported by the Brightcove API and the PHP wrapper library but the video file itself, and possibly other Drupal-only fields as this entity type is (Drupal-) fieldable.

  • Brightcove Playlist content entity: a required entity reference to the API Client config entity, all the data that is supported by the Brightcove API and the PHP wrapper library, a multiple-value entity reference field to Brightcove Videos in case of a manual playlist, and possibly other Drupal-only fields as this entity type is also (Drupal-) fieldable.

There are various other smaller content entities, but they're mostly hidden from the user, such as:

  • Custom field content entity: this stores the definitions about the custom fields for each API Client. These fields are dynamically generated on the Video entity’s form, and handled through API calls, like the default value, description or the saved value. The custom field values are also stored on the Video entity.

  • Player content entity: Stores some minimal information about the player config, like width and height. These values are applied to the player display if possible.

  • Text Track entity: Holds all metadata about the text tracks, after the ingestion the uploaded file is removed.

Synchronization of content between Brightcove and Drupal is a two-way thing.

  • When a change is made to a Brightcove-related Drupal entity on the Drupal site/side, the module pushes those changes to Brightcove immediately.

  • The module has a built-in mechanism to "download" changes from the Brightcove site to the Drupal site, which may occur by three different triggers.

    • First of all, the module uses Drupal's cron system to download changes from Brightcove to Drupal when the last change dates are different on any given content entity. When all changes have been downloaded to the Drupal site, the synchronization starts over again.

    • Secondly, the site administrator can initiate finishing the current synchronization in a batch process, initiate a completely new synchronization, or stop the currently running one from the admin interface (see below).

    • Thirdly, when the user wants to edit a Brightcove-related content entity on the Drupal site and the last change date of that entity is newer at Brightcove, a message is shown to the user with a link that downloads the latest data from Brightcove to Drupal before the user makes changes to it. It also checks if the video was deleted from Brightcove Studio, in this case the user can delete the local version or recreate it (however the video file won’t be available).

  • For more detailed information about the content synchronization from Brightcove to Drupal, see the comment for the _brightcove_initiate_sync() function in the code.

For various tasks, several already-existing Drupal (contrib) modules are used.

  • The Time Formatter module is nothing more than displaying a number of seconds (either an integer or a float) in fancy formats like "16h58m17s".

  • The Entity Browser module can be used eg. for selecting videos for a manual playlist.

  • The Inline Entity Form module is used for creating and storing informations about the text tracks, currently only new text tracks can be created or deleted, but it is not possible to edit them for now through Drupal.

Part 2: Installation & Configuration
Part 3: Video & Playlist Management
Part 4: Including Videos & Playlists in Drupal content

The new Drupal 8 module was developed by Tamás Nagy and me, CSÉCSY László.
Jan Mashat was the project manager and László Brindza was responsible for testing.

Apr 21 2016
Apr 21

The video recordings of the lectures held on the Documenting APIs mini-conference are now available on our dedicated event page.

On March 4, 2016 we helped organise a special whole-day meetup in London on the Write The Docs conference, in which the community discussed the tools and processes used to document APIs.

We recorded the sessions, mashed them up with tweets and summaries and published them.

On the page you will find the following content:

  • James Stewart, Director of Technical Architecture at GDS: Intro and welcome to GDS
  • Rosalie Marshall, Tech Author at GDS: Understanding the needs of API documentation users
  • Ellis Pratt, Technical Communicator and director at Cherryleaf: Aye, Aye, API - What makes Technical Communicators uneasy about API documentation, and what can we do about it?
  • Christian Vogel, RAML: A Quick Introduction to RAML
  • Ole Lensmar, Chairman of the Open API Initiative and CTO of SmartBear Software: Swagger
  • Kristof Van Tomme, CEO Pronovix: Code sample libraries in Drupal, using the GitLab field module
  • Alexander Browne, HMRC product manager: The HMRC developer portal
  • Andrew Marshall, technical author: The potential of API docs to evolve technical architecture and API design (TBC)
  • Paul Mackay, Technical Lead for the DCLG Local Waste Service Standards Project: Local digital standards for waste service in government
  • Chris Smith, lead technical architect at Companies House: Introducing Swaggerly
  • Kyle Fuller, API Blueprint: Documenting APIs for Humans
  • Kim Stebel, Tech Author: Interactive API documentation
  • Roger Sheen, Information Architect & DITA Open Toolkit Collaborator, Documentation Maintainer, Infotexture: “API Docs in DITA”

To stay informed about the London WTD events, click here.

If you think your followers would be interested in the sessions, please, send a tweet, share us on Facebook, or LinkedIn. We are looking for people interested in API documentation for similar future events in the Write the Docs community.

We are planning an event on May 22nd in Portland and another one in Washington DC some time in October. Want to get notified about events like this?

Apr 18 2016
Apr 18

While in the past accessibility was often only perceived as a nice extra, it is quickly becoming the norm: in most countries legislation dealing with accessibility currently only affects web pages, but PDF documents might follow soon. With good cause: worldwide approximately 285 million people are visually impaired and have difficulties to read online documents without assistive technologies.

As an addition to our PDF in Drupal series (PDF in Drupal part 1 and part 2), we researched the subject of accessible PDFs: how can writers create content in a way that it becomes accessible to users with disabilities? We’ll highlight checking mechanisms and tools and end with listing three types of the available assistive technologies that make content accessible.

What is an accessible PDF?

PDF, the Portable Document Format, is a widely used standard document type on the internet. Users can convert electronic documents and scanned images into a PDF file, and distribute it easily via email. When implemented properly, PDF content becomes accessible to users with disabilities such as blindness and low vision or of a cognitive or learning nature (e.g. dyslexia).

How to make PDF accessible

Document authors can choose from a range of possibilities to create accessible PDFs. A few examples.

Tags

Tags contain little pieces of metadata creating a hidden structured, textual representation of the content, thus becoming available for screen reader programs and other assistive software. Tagging doesn’t influence the visual integrity of a PDF file: the look of the document remains the same. By adding tags, the content creator defines the exact order of reading, which makes it easier for users to interpret the document structure and navigate through the content.

Alternate text (alt text)

Alternate text provides text-based information about the hyperlinked web page or the inserted image. If an image is not visible due to software errors or because the user is visually impaired, the alt text will provide a text equivalent to the missing information. The example shows an alternate text that describes an image, taken from the blogpost Publishing a code sample book from Stackoverflow to LeanPub using Drupal and GitLab.

Other tips to improve accessibility:

  • Show the logical structure of a document by inserting headers
  • Add navigation labels, such as headings
  • Choose software that creates accessible documents (if this is for a web application, you could integrate an open source library like iText that supports accessible PDF)

Checking accessibility

Ideally, checking the accessibility of a PDF document should the last phase in its creation process. Most PDF creating software provide this possibility.

Software applications like PAC 2 offer possibilities to check the accessibility state of your document.

According to Adobe's guide about creating accessible PDFs, accessibility issues often appear in

  • documents containing only scanned images of text (and without content available to screen readers)
  • documents without tags
  • documents without alt text on images or on links

Assistive technologies to access and check content

Authors can use the same assistive technologies that are used by their audience to test the accessibility of a created document.

Text-to-speech software provides auditory feedback for documents or website content:

  • The Read and Write for Google Chrome extension helps users with cognitive or learning difficulties access Google Docs documents and webpages.
  • Balabolka is downloadable TTS software. It has a portable version that can save on-screen text as WAV or MP3 files.

Speech-to-text (voice recognition) software takes and transcribes audio content into written words.

Available for Windows 7, 8, 10 and OS X.

Screen magnifier software makes it possible to enlarge screen content and provide auditory feedback.

Windows, OS X and Linux-based operating systems have built-in screen magnifiers.

Conclusion

The goal of accessibility is to ensure that every person, with or without disabilities, has equal chances and possibilities to access your content. Creating accessible PDF files can help to reach that goal. With quickly changing technology the creation process of accessible documents gets more complex, but the need to provide them grows at the same time.

We’d like to highlight some sources providing information and tools:

Apr 15 2016
Apr 15

Even though it has become much easier to create embedded help experiences, many software products still do very little to integrate their help content into their application. In this post we will explain why embedded help leads to better experiences, why we built our Help Widget, and give you an overview of its architecture.

Why you should embed your help content

Printed documentation manuals are a thing of the past, or are they?
Very few projects take full advantage of the new capabilities that a connected world is offering. Most documentation efforts for web applications follow the paradigms developed in a time when documentation was a stand-alone printed deliverable, prepared as a reference source that customers would use to solve their problems.

If your documentation is not readily accessible from your application this is a missed opportunity: your customers are getting a worse experience and your support team is paying the price. Bombarded by an ever increasing torrent of information and impulses, people's average attention span is getting shorter. Users are impatient, expecting just enough information right when and where they need it.

Embedded help anticipates a user’s need for help and supports through examples or a walkthroughs in situ - or just the relevant morsels from the documentation - with the least disruption to workflow and focus. It is only a breadcrumb trail that leads back to the original action.

Actual manuals only come into play once the same users want to learn a complex new skill or process, gain a deeper understanding and confidence by having a reference.

Relevant help content embedded in the interface saves time, improves user experience and reduces support calls.

There are a few paradigms of embedding help content into a web application, you can read our overview about them in this previous blogpost. Take into account that your app will be changing, because it well should and will change: inline help needs rewriting, walkthroughs may break.

The embed help widget we created is your quick and simple solution to your UX problems without/while redesigning your app. A widget that opens on top of your application or website: you can pop up text, walkthroughs, videos etc. You can link in the relevant parts of your evolving documentation, explain complexities without crowding the app's UI. The widget's content is rendered by our Drupal CMS, so while it looks part of your interface the help content can be edited independently on the fly.

Embed any kind of online help resource into your webpage or web application. Serve contextual links from your documentation system.

Why we built the WalkHub Help Widget

We announced this January that Walkhub had been rewritten: it now uses Go and React.js. We also introduced our brand new WalkHub help widget and our intention to set up a Drupal 8 site where registered users will be able to create their own widgets. This site, embedthedocs.com is the result.

We created the EmbedTheDocs freemium community service for the purpose of allowing everyone (even those who are not familiar with web development) to create their custom instances of Help Widget for embedding documentation, including but not at all limited to walkthroughs. With this tool you can generate a widget in no time as it does the dirty work for you.

create: register/log in, create help widget, add content via form, save and test; embed: copy js embedcode, embed in your app, your widget is live; iterate: edit widget's content anytime

You can read more information about the Walkhub Help Widget in our previous blogpost and on EmbedTheDocs.com itself.

If you're curious about how widgets are generated technically with the assistance of EmbedTheDocs or if you'd like to do some fine-tuning, the next section will be interesting for you.

How it works

Drupal 8 provides the content and the Walkhub server processes it

widget created: drupal saves widget form, drupal transforms content into json, every json outpur receives unique url, url is accessible to anyone; code embedded: js code connect with walkhub server, walkhub server receives json output via its url, walkhub server builds widget content from json output; content iterated: widget's content mirrors any changes made in the form

We have separated the widget generation into two fundamental pillars: the user specific content elements (groups, links and WalkHub walkthroughs) and the engine, which generates the widget itself.

After registering and logging into the site, you can freely start adding a custom Help Widget, which is technically a node created of our widget content type in the Drupal 8 system. This content type has a simple multiple value paragraph field, which allows people to add their group-, link- or walkthrough elements as they wish:

How to create a widget

After saving the content, it is processed in the background by our custom module which uses Drupal 8’s impressive controller and routing system to generate a special JSON output of the paragraph field values. Finally the rendered JSON file can be reached through the embedthedocs.com/json/{nid} url. Our module also creates a block with the embed code, which already contains the generated JSON file’s path, so it’s ready for use. After saving the widget content you can instantly see how your widget is going to look like on the bottom right corner of the content’s page.

The WalkHub Help Widget

Once you are finished creating your own widget node on our Drupal site, you only need to copy and paste the generated embed code in your website’s source code. The code also contains the WalkHub Record button by default, so you can create walkthroughs easily on the target site. If you don’t want the Record button to be generated into the code, simply uncheck its box when editing your widget node. You can of course generate as many widgets as you need. In a Drupal system the easiest way to embed the widget’s code is to create a custom block and paste the code into the block’s body (it only works with text formatters that allow div and script tags). Once you are successfully done with this part, you can see the orange “Get help” button in the bottom-right corner of your site. You can also relocate the button easily by altering the data-position attribute of the div tag (bottom-right, bottom-left, top-right and top-left positions are available).

Your help widget is actually generated by the Walkhub engine. The server receives the JSON formatted file from the embedded code in the data-list-url attribute, and creates an HTML iframe object with the assistance of the client.js and embed.js JavaScript files (the generated embedcode already contains them) . Then it extracts the information from the data-list-url attribute for building the content for the widget using React.js components. The browser uses CORS headers for verification.

It should also be noted that thanks to Drupal’s flexible content editing feature, an existing widget's content can be edited anytime on the fly. The widget adapts to content-changes immediately on every site where its code has been embedded: the JSON formatted source file mirrors the widget-node’s content.

How to serve widgets from your own Drupal 8 site

You don’t necessarily need EmbedTheDocs to create Help Widgets, you could also generate them from your own Drupal 8 site on your own server.

This is where things get really exciting! With your own site you could serve documentation content that is targeted for a specific user, or make a widget that automatically recommends relevant content depending either on the page that a user is visiting or on their role. You could add a support chat, or a search function that looks for relevant documentation, ...

The Help Widget is generated using the WalkHub engine which is part of the project that we open sourced previously. We haven’t created a feature with the configurations to make a Help Widget module but this shouldn’t be too much work. If there is sufficient interest we will make a package that explains how it is done.

If you are interested in creating your own server for aggregating and serving your documentation, let us know here!

Apr 06 2016
Apr 06

The paragraphs module makes it easy to arrange content on your site, but it limits your creative freedom. First, we’ll explain how we use the module and then why creative constraints are actually a good thing, because they make you more productive, and paradoxically help you be more creative...

What is the Paragraphs Module ?

Previously, we listed some tips on targeted web content. Here, we want to share what we’ve learned building landing pages using the paragraphs module in Drupal.

“The Paragraphs module comes with a new "paragraphs" field type that works like Entity Reference’s. Simply add a new paragraphs field on any Content Type you want and choose which Paragraph Types should be available to end-users. They can then add as many Paragraph items as you allowed them to and reorder them at will. Paragraphs module does not come with any default Paragraph Types but since they are basic Drupal Entities you can have complete control over what fields they should be composed of and what they should look like through the typical Drupal Manage Fields and Manage Display screens.” (drupal.org)

Paragraphs can be anything you want, e.g. an image with a text block, a headline with a call to action, a slideshow. Paragraphs are predefined and you can use them independently from one another.

Developers create paragraphs depending on the customer’s needs. The number of paragraph types is usually limited to increase workflow efficiency, but can be extended at will.

We’ve been using paragraphs to create long form landing pages, with the kind of single page responsive layouts that gained prominence in the startup world. With this technology it is quick and easy to set up layout formats starting from your textual content.

For Pronovix.com, currently we use these paragraph types to construct a landing page:

Paragraph types for pronovix.com landing pages

Let’s take a closer look at three examples: 1. “Hero shot with a call to action”, 2. “Three Column Benefits”, 3. “Tips”.

  1. The paragraph type “Hero shot with a call to action” helps you to grab the attention of your readers and to activate them. For this paragraph, we can fill out the following fields:Example of a Hero shot with a call to action paragraph from the WalkHub landing page.
  2. The “Three Column Benefits” paragraph type shows three advantages of your product side-by-side. The benefits are brief and to the point, in three columns, combined with catchy icons. For pronovix.com, we’ve built a bootstrap icon library into our paragraphs implementation, you can use any of the glyphicons without having to search and upload them into the site. This single feature saves us a lot of time.Example of a Three Column Benefits paragraph from the WalkHub landing page.
  3. When we select “Tips” we can combine an icon with a headline and a short text to point out specific characteristics of the product. Within this paragraph type we have an icon field - a headline field - a text field. The icon field contains entity references. We can choose icons from the already uploaded icon collection, but it is also possible to create a new icon entity and add it to the existing collection.Example of a Tips paragraph from the WalkHub landing page.

Intelligent Constraints = Increasing Efficiency

When we got the green light for our Apigee Developer Portals Workshop, there were only three weeks left before the event. We had to construct a landing page quickly to be able to announce the workshop in time.

By then we already had our workflow refined for creating a simple landing page:

  1. Collect and organize the relevant information, then fill out our landing page content form.
  2. Pick a paragraph type for each chunk of content on the fly.
  3. Add the design elements required by each paragraph form.
  4. Rearrange/change/edit paragraph entities if needed.

Once we have this, creating the actual landing page is mostly filling forms: the configuration options and required fields for each paragraph type are predefined. Just enough to create a visually pleasing result - all the while dramatically shortening the process by limiting the options, leaving the focus on content. Because of this workflow, all parties involved are less reluctant to run further iterations on the copy and the layout/design elements.

Limited options that lead to a creativity boost

With too many options to choose from, people find it very difficult to choose at all (The Paradox of Choice - TED Talk). Embracing our constraints (Creative constraints can lead to amazing work, Boosting creativity through constraints, Embracing constraints) may indeed boost our creativity. Though even MacGyver would need more than his Swiss knife to make a landing page, the principle remains… Having less options sharpens your mind: it makes you efficient.

Combining possibilities and limitations

The paragraphs module...

  • … helps you to focus. Too much choice can paralyze you - the module guides you through your ideas.
  • ... will help you to meet that deadline.
  • ... fits into the corporate style guide. This saves editing time and fastens the publication process.
  • ... helps you to be efficient.

At Pronovix, we use the paragraphs module almost every day to create new web pages. We continuously evaluate and improve our landing page creation process: we were already able to shorten the feedback process and reduce the number of editing rounds before publishing. Two remarks, though. 1. The paragraphs module claims to be giving back the power to the content editors. But without developer skills editors are not able to change the paragraph types to meet their needs themselves. Close co-operation between developer and editor is still necessary to define and customize the paragraph types that will be available to editors on your site. 2. Post-editing could use a handier interface. Now several of the same paragraph types on one page are indistinguishable in edit mode unless unfolded.

We are working on an audience-focused landing page canvas (that we will publish under a Creative Commons license in the near future) to complement the form we published earlier. To be continued…

Do you also use the paragraphs module? Leave us a comment!

Mar 17 2016
Mar 17

Most SaaS integration services that synchronise user accounts, do a one-way sync that copies over to new accounts. PieSync, from Ghent, Belgium, has developed a two-way sync platform, with various sync settings, so you can sync contact information between your favorite cloud apps. Unlike many other sync account companies, PieSync:

  • offers full sync in both (or multiple) apps
  • weeds out duplicates
  • brings all existing information together, also the data that existed before you started using PieSync
  • automatically reflects changes and updates that occur in one app, in the other(s).

Concrete: By using PieSync, you can integrate all or part of your Google Contacts data in Mailchimp and the other way round. When you make a change in one of these apps, the other app will be updated immediately. No messing around curating data, so you save time.

Syncing between Drupal and Google Contacts

We’ve been talking with Piesync about creating a module to sync user profiles between Drupal and Google Contacts, your CRM and other cloud apps. The tricky part is that in Drupal the user profile can be arbitrarily extended, and when you have additional fields you probably also want to synchronise them. So, a Piesync module will need to be able to map user fields to an object that is synced with PieSync (default PieSync object and you map the fields to that, then two-way syncing). Drupal sites operate in an environment with increasingly specialised SaaS platforms (e.g. invoicing with Xero, contacts in Google Contacts, Salesforce as a cloud CRM) that are not interrelated, so a sync module will save you from manually importing all that data across these apps, ensuring accuracy and saving you time. The added bonus is that this sync will make Drupal a central base that keeps data consistent across your apps.

PieSync already provides syncing possibilities between many SaaS apps:

Syncing with PieSync: connections currently available.

Why use a commercial service for syncing? The APIs (application program interface) of SaaS and Cloud apps change regularly. Even popular projects that integrate with third party APIs, e.g. the Hybridauth module, or Facebook Connect, sometimes break when the module is not updated in time to follow API changes. Ideally it would be the responsibility of the API owner to maintain their Drupal integrations, but not everyone does this, and most site owners don’t have the budget or skills to follow these API changes.

We want to partner with Piesync who will guarantee API functionality.

Another advantage of Piesync is that it becomes easier to change apps, because you can seamlessly sync your data. This would also allow you to postpone choosing certain cloud services that interact with your user data, until after you have finished your site building project.

Would you use PieSync’s Drupal module? Tell us how you would use it and we will let you know as soon as the module is out!

Feb 19 2016
Feb 19

The first week of March I will be in London for 3 events related to documentation, APIs and Drupal Developer portals. If you are working on an API documentation project, you should not miss this opportunity:

A one day event organised by Apigee. "I love APIs Europe brings together European digital leaders who will share their strategies and lessons learned, combined with hands-on educational sessions led by the world's most experienced API Ninjas."

Buy your ticket here

A free half day workshop about Drupal Developer Portals with Apigee (whose Developer Portal is built in Drupal). On the agenda:

  • What is Drupal and why is it a good platform for Developer Portals (Kristof Van Tomme)
  • Overview and demonstration of key functionalities for a Developer Portal (Kristof Van Tomme)
  • Demo of the PoC we developed for an Apigee prospective customer (Kevin Bouwmeester)

Register for free here

A special whole day meetup in collaboration with UK’s Government Digital Services, in which we'll undertake an in-depth exploration of the tools and processes you can use to document APIs. I'm organising the event with Rosalie Marshall, and I will do a lightening talk about Code Sample Libraries in Drupal, using the GitLab field module we developed for iText.

More information and signups on the London Write The Docs meetup group

Feb 11 2016
Feb 11

We built a developer portal for iText that helps them to republish questions, answers, and code samples from Stackoverflow in their documentation. The same Drupal site also has an export function that packages the markdown content from the questions in a format that can be published on LeanPub, where it becomes available as a PDF, EPUB or MOBI document.

Stackoverflow is a great content marketing channel for developer evangelists. iText, the company that created the eponymous PDF library, has used Stackoverflow with great success to gain a strong reputation with developers and to build a sustainable business around their open source software.

In this blogpost we want to share how Bruno Lowagie (the original developer of iText) designed a workflow that allows him to publish questions and answers from Stackoverflow on the iText Drupal site and then export them to LeanPub (utilizing a custom Drupal module):

  1. A user posts a question on Stackoverflow, such as “Can’t get Czech characters while generating a PDF”:

  1. To answer this question, we create a standalone example.The finished sample code is then uploaded to GitLab, the open source code repository that iText uses to manage their project’s source code and example files.
  1. Then - when publishing the example Java file into the Examples section of the iText developers website (as a book node) - we use a custom Drupal module to link to the sample code file in GitLab and display the contents of that file within the node.

  1. Having all the elements in place, we answer the question in Stackoverflow - referring to the example with some information and a link to the source code and the resulting PDFs.
  1. Afterwards, we post the example - containing the code written when answering the Stackoverflow question - into the Q&A section of the iText developers website.

  1. In addition to the content now being accessible through the Q&A and Examples sections of the iText developers website (as well as in the original Stackoverflow post), we are also able to export the structured book content into the format (a ZIP file) required for publishing on LeanPub - utilizing a custom Drupal module developed to handle the Leanpub-specific format: Contents of LeanPub-specific book export from Drupal:

  1. Next, we are able to take the LeanPub-specific contents of the exported Drupal book and publish them on LeanPub:


See also Bruno Lowagie's video tutorial:

[embedded content]
Jan 28 2016
Jan 28

It’s strange but true: seven years after the PDF reference was published as an ISO standard (ISO-32000-1), there are still developers who think that the Portable Document Format is a closed document format owned by Adobe. PDF is often perceived as a strange, impenetrable document format. It’s high time we bury that idea and take a look at what’s going on in the world of PDF today...

In this guest post Bruno Lowagie, the project lead of iText, gives an overview of some of the most important innovations in the PDF standard.

Different ISO committees are currently working on new standards such as PDF 2.0 (to be released in 2017), XFDF, and ECMAScript for PDF. Substandards such as PDF/X, PDF/A, PDF/E, PDF/VT, and PDF/UA are updated on a regular basis.

Up until today, ignorance about these standards and about best practices for PDF prevents smarter use of documents. Creation of PDF is often seen as a one-way process: once a PDF is created, the content is locked inside and no longer accessible. Content management systems treat such PDFs as if it were images. It shouldn’t have to be that way.

PDF is for viewing and printing – an idea from the past

The Portable Document Format was created in 1993 to solve the problem of viewing and printing documents in a platform independent way. The goal was to create a document format that allowed the industry to render documents in a reliable and consistent way regardless of the operating system used.

The ubiquity of Adobe Reader as a viewer that was distributed for free to be used either as a standalone application or as a plug-in of an internet browser, contributed to the success of the format. Content owners could easily create documents in the PDF format and then upload them to their server without having to worry if the consumer would be able to view or print the document. So what’s the problem? Let’s take a look at some of the issues that have emerged over the years:

  • Too many features. The PDF specification has been growing in many different directions. As a result, not all PDFs are created equal. For example, documents can be set to depend on external fonts to reduce the file size, but this results in an illegible or odd-looking page if those fonts are missing on the end user’s machine.
  • Too many tools. As Adobe allowed third parties to use the PDF specification to create software that produces PDF, there are thousands of different tools and applications that generate PDF. Some of those tools cut corners that result in badly formatted PDF. Sometimes a PDF document is nothing more than a bunch of scanned images. Although the human eye sees text, there is no text inside the document when a machine looks at the PDF.
  • Lack of responsiveness. Most PDFs are made for a specific paper size. When viewed on a small screen, a visitor needs to do a lot of panning to read the content. It’s possible to provide reflow capabilities in a PDF by “tagging” the content, but most PDFs aren’t tagged correctly (or aren’t tagged at all).
  • Inaccessible content: Most current PDF documents are by default not accessible. For instance: what we perceive as being a table, is nothing more than a bunch of lines and text added at specific coordinates to a machine. As far as the PDF is concerned, there’s no way of telling if a specific text snippet belongs to a cell in a table or to a paragraph because that information wasn’t added when the PDF was created. As a result it’s really hard to browse through documents with screen readers.

Standards to the rescue!

PDF/A: a standard for archiving PDF

When archiving a document, you want to be sure that you will still be able to read the document in 10, 30, maybe even 100 years. This promise of long-term preservation is hard to keep if the specification allows too much freedom in the document creation process. The PDF/A standard (ISO 19005) is a subset of the PDF specification, introducing restrictions and obligations. For instance: the document may not be encrypted as there is no guarantee that it will be possible to decrypt the document in a distant future. The document also has to be self-contained. For instance: fonts need to be embedded, because there is no guarantee that an external font will still be available in the long term. Another obligation is that the document requires metadata in the XMP format.

The PDF/A standard consists of different parts: PDF/A-1 (2005), PDF/A-2 (2011), and PDF/A-3 (2012). Approved parts will never become invalid; new parts define new, useful features. There are also different conformance levels.

  • Level A: the document needs to be accessible. This is achieved through Tagged PDF.
  • Level B: this is the basic level. The visual representation is guaranteed, but the semantic structure of the content may be missing.
  • Level U: the visual representation needs to be guaranteed, but all text needs to be stored in Unicode (introduced in PDF/A-2).

There is only one difference between PDF/A-2 and PDF/A-3. In PDF/A-2, all attachments need to be conforming PDF/A documents. In PDF/A-3, the attachment can be in any format. For instance: you can add a Word file that was used as the source of the PDF document, you can add an XLS file containing raw data that is explained in the document, and so on.

ZUGFeRD: a standard for invoices

In 2014, the German Forum for Electronic Invoicing (FeRD) released a standard that is built on top of PDF/A-3. To a human consumer, a ZUGFeRD invoice looks just like any other electronic invoice, but thanks to the fact that the document conforms to the PDF/A-3 standard, it can be preserved for the long term. ZUGFeRD also requires the PDF to have a standardized XML attachment that allows machines to read and interpret the content of the invoice. Soon suppliers for government agencies will be required to submit their invoices in PDF/ZUGFeRD. Over time as it gains a wider adoption e-commerce applications will also have to follow.

PAdES: making the document official with PDF Advanced Electronic Signatures

To ensure the integrity of a document (for instance: once an invoice is final, it shouldn’t change anymore), to make sure a document is authentic (for instance: making sure the invoice is genuine and not sent by a third party), and to allow non-repudiation, a PDF document can be signed using a digital signature.

The digital signing mechanism of PDFs is described in an ETSI standard that makes it possible to verify a document independent of the vendor of the software that was used to generate it. Digital signatures in PDF allow for documents to be signed by different people in a workflow and they also support Long-Term Validation (LTV). All of this is currently described in PAdES and will be part of the PDF 2.0 standard (ISO 32000-2) that will be released in 2017.

PDF/UA: universal accessibility

There has been a really large push on the web for a more user accessible internet. When accessibility standards are observed, they enable the blind and visually impaired to understand, navigate and scan through documents in a much more efficient way. PDF/UA implements these accessibility standards by requiring, among others, the already mentioned Tagged PDF functionality. Web accessibility standards have become compulsory for public sector websites in a range of countries. There is now also a push to make PDF/UA obligatory (amongst others in the US).

Conclusion

We’ve only been scratching the surface as far as PDF is concerned. We didn’t even mention the different technologies that are available in PDF with respect to forms and templates. We’ve only listed a couple of the many benefits.

You can view the video of "PDF Is Dead; Long Live PDF ...and Java!" presentation here. It was the part of JavaOne, a Java conference, that took place in San Francisco, October, 2015.

Better PDF for Drupal

Do you think it would be worth the investment to build a PDF daemon that we can use from Drupal? Do you need accessible, human and/or computer readable PDFs? Do you want to be informed when we start an iText PDF implementation? Leave your comments on the feedback form, it will help us make a case for the development work.

This was a guest post by Bruno Lowagie, the project lead of iText, you can find more information about iText on their website.

Jan 26 2016
Jan 26

In the web community, PDF has become synonym for a range of accessibility bad practices. Some people even think that we would all be better off if PDF would finally die, just like Flash and Internet Explorer. As a result PDF is not very sexy in the Drupal and wider PHP community and this has negatively impacted our tooling.

This is a shame: when properly implemented, the PDF standard doesn’t need to suffer from the accessibility issues that a lot of online PDF documents are plagued with. PDF also holds a unique position in the digital world, it is a widely accepted standard that enables a range of applications for which there are no real alternatives.

In this series we will give a short introduction to some of the new PDF standards, explain how you can recognise an accessible PDF and describe how we could up our PDF game in the Drupal community.

Interoperability in a centralising world

More and more, digital services are being performed in walled gardens through online platforms and their APIs. It is easier to develop applications inside a closed system where the data structures are dictated by a single authority and centralisation aggregates information that can be used to train artificial intelligence. It is also easier to monetise SaaS software.

Centralisation however also carries an inherent risk, as it makes systems less resilient. For this reason many organisations require applications that work in a distributed architecture. Open format architectures like email, and HTML enable an ecosystem in which a large number of clients implement a protocol and nobody needs to rely on a central data store to exchange information using these standards.

Benefits of PDF

Correct use of the PDF standards has many benefits:

  • Interoperability: an invoice produced using the ZUGFeRD standard (as explained in part 2 of this series) can be processed by any system that supports the standard. A document that was signed using the PAdES standard can be verified using any system that supports the PAdES standard. Documents that are properly tagged can be read and interpreted by any screen reader that understands the standardized Tagged PDF. As long as the standards are followed, different systems running on different platforms can use PDF as the format of choice to exchange documents.

  • Open data: Several countries have signed into law open data policies that guarantee open access to datasets that were created with government funds. Traditionally a lot of data is shared with the public in PDFs, but most of these are flat PDFs that don’t retain any of the semantic mark-up. As a result data often needs to be extracted by hand and cleaned up before it can be reused. In open data portals visitors sometimes expect to be able to generate PDF reports. If PDF/UA becomes required by law for government agencies the reports will also have to provide their data in this accessible format.

  • Decentralised use: you can use PDF to transmit structured information, you can use PDF to embed movies and sound, you can add different attachments to a PDF, you can seal a PDF and sign it with a digital signature. While most applications are becoming ever more centralised there are however another set of advantages to using PDF as a digital envelope that works in a distributed architecture.

PDF and PHP/Drupal

PDF in Drupal is in a rather sorry state. There are a lot of projects that are not properly maintained and most of these projects were created to export websites, and therefore often lack the ambition to implement all the improvements that have been made in the new PDF standards. PDF implementations in PHP are often also resource hungry, and might cause serious performance issues on a high traffic site. To make nice PDFs in Drupal you will often need to install third party libraries that hosting companies might not allow. (Check out Chris Ward’s blogpost for a run down of the currently available solutions and some of their problems).

The Java community has better PDF generation tools. One such tool is iText, an open source Java library available under a dual license (AGPL for free usage and a paying commercial license for organisations that can’t comply with the AGPL license). As a specialised tool for PDF generation, it implements a broad range of the specialised PDF features mentioned above.

Open source PDF API

I see very strong parallels between search and PDF generation functionalities in Drupal. With Lucene, the Java community had a more powerful, better performing solution for indexing and searching websites. When a plugin was developed for Apache Solr, that uses Lucene, Drupal sites were able to delegate a resource intensive function, and to make use of a superior search technology. I believe that we need something similar for PDF generation and I think that iText would be a great candidate to solve our PDF problem.

Do you think it would be worth the investment to build a PDF daemon that we can use from Drupal? Do you need accessible, human and/or computer readable PDFs? Do you want to be informed when we start an iText PDF implementation? Leave your comments on the feedback form, it will help us make a case for the development work.

This post is based on a discussion I had with Bruno Lowagie, the project lead of iText, an open source PDF library for Java.

Jan 22 2016
Jan 22

We’ve developed EmbedHelp, a Javascript widget that makes it really easy to embed any kind of online help resource into a webpage or web application. We plan to extend this with an open source Drupal module that can be used to curate and manage the content that gets embedded into a web application. We believe it could become a valuable extension for Drupal based documentation systems.

Two and a half years ago we built WalkHub, an open source tool that makes it easy to record Walkthrough tutorials. Recently, as part of a large refactoring, we rebuilt WalkHub in Go to make it more lightweight. We did so because we came to realise that Walkthroughs are just one type of embedded help. Walkthroughs can be great for a certain set of applications and a certain group of people, but not all users like the Walkthrough paradigm. That is why we scaled down WalkHub so that it could be integrated into other, more complete onboarding systems.

Since we launched WalkHub, one of the most common requests we got was for a way to embed additional resources into our Walkthroughs. Many customers wanted to add videos and textual documentation that explained more than the information that can practically be added through a Walkthrough. Another, related, problem is that long Walkthroughs don’t really work, especially for customers that are exploring an application. After 4, 5 steps visitors start losing attention and close Walkthroughs or blindly click through just to get rid of it.

Embedded help: more than Walkthroughs

Another problem we found with Walkthroughs is that they are not a silver bullet for all educational content. They only have a natural fit for 1 of the 3 documentation topic primitives as defined in the DITA standard:

  1. They are great for task-oriented help, and can show a minimal amount of text to explain a step in its context.
  2. They are however less useful to transfer conceptual knowledge: when you need to introduce the underlying assumptions and concepts that you are making, it is a lot easier to do so in a video.
  3. Reference topics that explain how an application works and can be extended will need textual content like code samples and will definitely not work as Walkthrough tutorials.

I believe that it is important to make all 3 types of instructional materials available inside an application. When you make users switch context to access documentation (e.g. by opening a new browser tab or window) there is a high probability that they will get distracted and end up forgetting what they were trying to achieve. A lot of people have multiple browser tabs open, and once you take people outside of their primary focus, they might end up checking their emails, or wake up half an hour later and notice they have been checking their Facebook wall because they wanted to read an interesting article the documentation referred them to on Facebook…

Showing help content embedded into the app could serve as a sort of breadcrumb trail to get you back to the action you were performing.

EmbedHelp: an open source Javascript widget that embeds support resources

Last week we published a prototype for our EmbedHelp widget. You can see a screenshot of it below, and a live example in the right bottom corner of your screen on this blogpost.

screenshot of a closed and open EmbedHelp widget

As you can see the widget contains a number of resources: there are links to web pages (with the link icon) and links that will start a Walkthrough (the green play icon). We will also have a special icon for video resources (red icon for Youtube movies).

The help center opens web resources in an iframe on top of the web page you opened it in

Contextual help links served from your documentation system

Ideally, it should be possible to serve the content for the EmbedHelp widget from your application’s documentation portal. With a little curation it would then be possible to adjust the content in the widget to the page on which it is shown. This way you could show any type of relevant content in the context in which it is required. For logged in users you could even show or hide content on the basis of their role in the application.

This is a use case that Drupal excels at. That is why we have decided to build a back-end for the EmbedHelp widget in Drupal 8. That way we imagine our work could be used to bring the content from documentation portals that combine documentation resources with a social component (like a forum or a FAQ section) into an application.

Embedded help for everybody!

Similar to WalkHub, we are planning to make it as easy and frictionless as possible to get started with an EmbedHelp widget. For that reason we are planning to build a Drupal 8 site where registered users can fill in a form to generate a static EmbedHelp widget for their site. The basic service will be free. If this sounds interesting, we encourage you to fill in the form below. It will help us build a better EmbedHelp widget and it will enable us to send you a notification when we go live with the service.

Jan 13 2016
Jan 13

Last year we decided to change WalkHub from a feature rich, but difficult to maintain Drupal distribution to a microservice written in Go. This was a hard decision: at Pronovix we’ve been a Drupal shop for over 9 years now and we were also slashing quite a few features in the porting process, but we felt it was the right thing to do. In this blogpost I want to explain why we took this step and what the result is.

Ever since we launched WalkHub, one of the key feedbacks we received from our customers was that it was too difficult to start recording Walkthroughs. Installing your own WalkHub instance was a complex process. Many people also complained that they would never be able to justify maintaining a Drupal site for Walkthrough management alone.

On top of that we were running into a lot of technical debt with Drupal: while Drupal was a great choice for the initial prototype for WalkHub, we were seeing significant technical debt using Drupal 7 as the API provider. It is important to note that serving Walkthroughs is not a typical Drupal use case, and we were required to write a lot of custom code to make WalkHub work. As a result our development was being held back because of our technology stack.

Third, we were getting feedback from the market that Walkthroughs are just one type of embedded help. Customers came to us because they wanted to try WalkHub to solve a documentation issue often caused by a bad user experience, but Walkthroughs alone cannot solve that problem. Walkthroughs are great for task-oriented help topics, with a sequence of steps that need to be taken. Walkthroughs are not the right tool for conceptual and reference topics. People kept asking us how they could embed videos and text into Walkthroughs. The job people needed WalkHub to do was to offer help inside of an application, whatever format that help took.

For these 3 reasons we concluded we needed to refactor WalkHub. We decided that this time around we wanted WalkHub to be super lightweight, something that anybody would be able to add into an existing help/documentation system. That is why we chose Go and React.js with a PostgreSQL database. Go allowed us to make a single executable for the back-end that could easily be deployed. And we wanted to be able to run our own WalkHub on the same software as the open source package: this is why we prioritised support for a scalable PostgreSQL database over SQLite which might have been even easier to deploy.

WalkHub still has a Javascript front-end that records and plays Walkthroughs. It appears on your site as a 'Record' button and a 'Help center'. To test WalkHub on a web application, you can add the code snippet from the embed code generator below that will enable Walkthroughs from WalkHub.net on your site (a free service we provide for testing purposes and for reasonably low usage).

In our first Go release, the help center was a WalkHub only thing, but we are about to change this to allow you to make any type of web resource available from an in-app help center. You can see a mock-up of the new help center on the bottom right on this page. In the near future we want to make it available as a free service that makes it really easy to add a help center to your site. Keep an eye open for a post with more information about the help center (you sign up here for the WalkHub mailing list). Spoiler: we are planning to build a Drupal 8 application to run it...

Dec 29 2015
Dec 29

Using Watch and Learn’s video series as a foundation, we built a learning path on Outlearn for front-end developers who’d like to get into theming Drupal 8 sites.

The learning path walks you through creating a Drupal 8 theme from installing Drupal to setting up layouts, views, taxonomy and regions. We extended the original video series with pointers to lessons on using base themes, building a theme from scratch and utilizing frameworks, so that you can get a deeper understanding of developing themes in Drupal. We also made some learning objectives explicit from the videos to help you track your progress.

To get the most out of this path you should have some knowledge in front-end development. Some Drupal 7 theming experience will also help you better understand the lessons, although it's not essential.

Dive into Drupal 8 theme development with Drupal 8: Developing your first themes! And let us know about any topics for learning paths that you would want to see in the future.

What is Outlearn?
Outlearn is a training platform that makes it easy to curate and combine public training materials into a learning path. As the curators for Drupal 8, we used the platform to build learning paths that collect the best content about Drupal 8 that is out there, and ordered them so that by following the learning path, you can easily become more and more experienced with D8. We verified the quality of each video and added tasks to help your learning process.

Dec 22 2015
Dec 22

A few months ago OSTraining did a crowdfunding campaign. Their goal was to develop Drupal 8 introductory videos that would then be made available for free on Youtube. With the funding goal exceeded, the result was a great series of videos. We packaged them into a learning path for Drupal 8 site builders and extracted some learning objectives to make it easy to track your progress with the materials.

Drupal 8 has been heralded as the site builder release, as it introduces a better authoring experience, powerful customizations, more authoring features in core and a commitment to accessibility, all of which empower site builders to boldly go where no site builder has gone before.

When we were thinking about the best content for helping site builders get acquainted with Drupal 8, we immediately thought of OSTraining’s online video course, sponsored by Acquia.

To make the learning process easier, we’ve added explicit learning objectives for many of the videos, and turned them into a learning path on Outlearn to provide a starter package that helps you stay on track. Get started now with the Drupal 8 Beginner Class!

What is Outlearn?
Outlearn is a training platform that makes it easy to curate and combine public training materials into a learning path. As the curators for Drupal 8, we used the platform to build learning paths that collect the best content about Drupal 8 that is out there, and ordered them so that by following the learning path, you can easily become more and more experienced with D8. We verified the quality of each video and added tasks to help your learning process.

Dec 16 2015
Dec 16

To make it easier to process the continuous flow of Drupal 8 content, we reviewed all the articles, videos, and tutorials we could get our hands on, and condensed the best of them into learning paths (a sort of customized playlists). To do so we used Outlearn, a new learning system for developers. Outlearn makes it easy to assemble articles into paths and set learning objectives for yourself or your team. We hope this will make it easier to get familiar with Drupal 8, whether you are a developer, site builder or themer.

What’s in this series?

In the first batch of Drupal 8 tutorials and articles, we prepared learning paths for three different roles:

  • Site builders
  • Themers
  • Developers

Learning path for Drupal 8 developers

We created our first path for Drupal 7 developers who’d like to quickly get on board with Drupal 8. In the learning path titled Drupal 8: Developing your first modules, we cover the basics, bring your attention to changes, then get into details like annotation-based plugins, incorporating 3rd-party PHP and JS libraries, understanding dependency injection through the services, and finding out how to continue your learning process.

We would love to make the path even better with help from the community. If we missed something important or if you have any other feedback, please let us know. Or you could just create a pull request on GitHub. You can also fork the path and create your own customized version and publish it on Outlearn yourself.

The best articles from all over the web

Besides creating the learning paths, our curator team collected the best articles from the web to help you get started with Drupal 8. We have curators from different fields (developers, site builders, technical writer) to make sure that the quality of each article is verified by someone experienced in the topic.

More to come

We are partnering with Outlearn and will be working on quality Drupal 8 content for the community that will be released on a regular basis. We plan to have 3 new learning paths and 10-20 fresh articles each month.

Let us know about any topics for learning paths that you would want to see in the future. You can also publish your own paths. Let us know if you have created something that should be included in the Drupal 8 topic stream.

Stay tuned for our blog posts about the learning paths we created for site builders and themers!

What is Outlearn?
Outlearn is a training platform that makes it easy to curate and combine public training materials into a learning path. As the curators for Drupal 8, we used the platform to build learning paths that collect the best content about Drupal 8 that is out there, and ordered them so that by following the learning path, you can easily become more and more experienced with D8. We verified the quality of each article and added tasks to help your learning process.

Dec 16 2015
Dec 16

As part of our research into content architecture best practices for Developer Portals, we’ve done a small scale investigation to evaluate what URLs are used by API companies.

We looked at a sample of 211 companies that had a Drupal site and an API integration module on Drupal.org. We found that one quarter of the examined companies (51 companies out of 211) had a dedicated section for their API documentation.

Our main goal was to determine what URL patterns are most frequently used by API providers. The pie chart below displays the most popular patterns; as indicated there was a large group of ‘other’, that could not be classified. Among the grouped patterns domain.com/api (7,84 %) and docs.domain.com/api (5,88 %) were the most frequently used.

One way to classify URL patterns is to see if they contain the main domain or a subdomain. We found that 68,6 % of the sites used a subdomain and 31,4 %, used the primary domain.

Most likely this is because companies want to make a distinction between the area of their website dedicated to developers (with a focus on simplicity and transparency), and the area dedicated to decision makers (with more emphasis on business value and product maturity).

Hosting your documentation on a subdomain can be a good idea if there is a clear separation between these two audiences for your product.

When choosing between different URL patterns, SEO will also play an important role. An API site can be easily identified by name if it contains keywords. This also makes it easier to show up in relevant search results.

Not too long ago URLs’ only real purpose was to request information from web servers. Today web services use SEO friendly URLs that contain path aliases, that help Search Engines classify the page content.

The table below encompasses the anatomy of a SEO friendly URL with some tips on creating a great URL.

Source: moz.com

This is the first article in a research series we are publishing about content architecture for API documentation and Developer Portals. Sign up below to get notified about future posts.

Subsribe now

*We will not publish, share, or sell your email address in any way.

Dec 15 2015
Dec 15

APIs are going to transform the world: they not only benefit from network effects caused by specialisation, the data they collect also makes them a prime candidate for the emergence of specialised Artificial Intelligence. Considering how powerful APIs are, it would be an injustice should they remain the privilege of developers.

Not enough developers

According to an Evans Data Corporation’s study (2014) on the global developer population, there are today, worldwide approximately 19 million developers. By the year 2020 this number is expected to climb to 25 million, in a total population of nearly 8 billion people, that means that only 0.31% of the world will be developers. If you take into account that developers also need to build our APIs and all the other software we use, it is clear that there are simply not enough developers that can act as the gateway to API consumption.

Why we need a GUI for the API web

Not too long ago computing used to be an obscure activity that only developers could participate in. GUIs changed this, they turned computing into a mainstream technology. The API web today, is where general computing was a few decades ago. Its promise is clear but it hasn’t yet become a mainstream technology. That is why I believe we again need a GUI to emerge, a mashup tool that makes the API web accessible through a point and click interface.

GUIs make it easier for new users to use software because:

  • GUIs are self explanatory: it’s possible to configure an application with a GUI, just by filling in the forms of its admin interface.
  • GUIs offer constraints that get you through analysis paralysis faster, so that you can focus your creativity on the application model rather than on the implementation.
  • GUIs make it easier to discover the full feature space of an application, just by clicking through the navigation model.

GUI as a service

Several companies have identified this opportunity (e.g. Zapier, IFTTT, IBM's Bluemix, Appgyver,… ) and are working on GUIs that integrate a multitude of services. So far these applications are conceived as centralised cloud services. The advantage of such a setup is that its development and maintenance are executed by a dedicated product team and financed by a large group of users.

The disadvantage is that you come to depend on a single point of failure: even if you use abstraction to make it possible to switch APIs, all the applications you build are bound to a single platform. If, for whatever reason you want to switch providers this could lead to considerable reimplementation costs.

Drupal, an Open source middleware

I have been talking with a number of people in the Drupal community about how unique Drupal is as a site builder tool. Drupal is kind of special as a CMS because you can use it to build really complex behaviour without a single line of code. The rules module for example exposes a Turing complete trigger response system through the UI. With it you can do amazing things that in other CMSs can only be done with custom code.

What is also really exciting is that Drupal already has a lot of integrations with APIs: there are over 3000 modules on Drupal.org that are tagged as third party integrations. A lot of popular APIs already have modules. Some API integrations are installed on thousands of sites. A lot of API companies are currently not really aware of this and overlook the opportunity to market to a really large group of people that could get acquainted with their services.

The case for an open source GUI for the API web

Be it Drupal, or another open project, I believe that there is a case to be made for an open source middleware that can connect and process the services from different APIs. Since standards are arising around API design it might even be possible to create a standard interface that allows you to consume APIs that conform to the Open API Initiative standard (formerly Swagger) or to API Blueprint or one of the other formats.

For API consumers, an open source middleware would guarantee independence and prevent single points of failure. An abstraction layer, that lets you switch out APIs combined with an open source middleware in which you build your applications would prevent lock-in and give companies maximum resilience. I think Drupal is a great candidate for this.

25'000$ prize for a Drupal module

A few months ago, a Drupal module we created won the first runner up prize for the Context.IO Devpost challenge. The module leverages Drupal’s Feeds module and makes it easy for site builders to query email accounts and to map the information into a Drupal data model. It is a great example how you can build complex web applications all from the user interface.

Drupal 8, a crossroad

If we want Drupal to become the GUI for the API web, we need to make sure that we keep supporting site builders. Drupal 8, touted as the site builder’s release, has a lot of improvements that greatly benefit site builders: Configuration management, a mobile editing interface and numerous improvements to the UI make it the best release for site builders so far.

At the same time because of its Symfony foundation, it has become a lot easier in Drupal 8 to extend functionality with some Symfony code. This risks diverting some of the resources that consulting companies typically would invest in reusable Drupal modules towards single use custom code that leverages some of Symfony’s community packages, bypassing the need for a GUI.

Clickers unite

For these reasons I think it’s time for site builders to claim their place in the community as Drupal’s core audience. Today sitebuilders are an audience that is sometimes looked down upon.

Too often site builders experience the following:

person 1: So what do you do?

person 2: I’m a site builder, I use the interface to configure Drupal sites.

person 1: Oh (slightly disappointed), so you are not a developer...

Drupal was built for site builders, it is time for us to unite and claim our voice in the community. If you are a sitebuilder make sure to sign up for Drupal Uncoded.

This article was based on a presentation I did at APIdays Paris December 2015. The recording of my talk can be found at https://www.youtube.com/watch?v=liMbivP3rB4&ab_channel=KristofVanTomme

Nov 18 2015
Nov 18

To celebrate the release of Drupal 8 we are launching a new improved D8upgrade report generator. You no longer need to copy paste your module list, all we need is the URL to your site. We’ve also created an embeddable widget that your site’s visitors can use to generate reports.

We’ve automated the module scanning on D8upgrade.org, it’s now dead simple to request a report. Tell us the URL of your Drupal site and your email address and we send you an upgrade report (form embedded below)...

If you are a Drupal service provider you can generate a personalised embed form on d8upgrade that will help us track who requested a report on your site. Every month we’ll send you a list with the sites that were submitted on your form.

If you have a Drupal 5,6 or 7 site that you would like to upgrade, fill out the below form, and we’ll send you an upgrade report.

Nov 16 2015
Nov 16

We are working on a set of free training materials for Drupal 8. To make sure we build something that others will be able to reuse we would like to get your input on the kind of trainings you would like to use to retrain your team.

With the imminent launch of Drupal 8, a lot of companies in the Drupal community will be looking for cost efficient ways to get their teams up to speed. We’ve had a few colleagues who have been exploring Drupal 8 and we’ve asked them to prepare a series of materials that summarise the most important changes.

To make it easy for teams to reuse and repurpose the training course, we are exploring a partnership with Outlearn, a training platform that makes it easy to curate and combine public training materials into a learning path, similar to a personalized playlist. To test the partnership we are working on a first series of learning paths about the most important topics a developer needs to know to be able to work in Drupal 8.

Outlearn is a really good tool for this kind of problem: Using learning paths it is really easy to glue together the most helpful materials that are already available in the community. A learning path also includes extra context from the curator that makes it greater than the sum of its parts.

To help us choose what to focus on, we need your feedback.

To be the first to hear when we release the course and to be able to make the course better fit your requirements, please take 5 minutes to fill out this survey.

Nov 05 2015
Nov 05

Motivation

Drupal 8 is out the door – well, the release candidate at least, so it's time to get intimate with it. As I'm mostly a backend developer with a focus on coding, I'm mostly interested in what's under the hood: how should we write a new module. As we had our heating system upgraded this summer and I opted for a controller that can be instructed via the web, it was an extra motivational factor: it would be nice to have a heating system #madewithdrupal8. :)

Every proper project should:

  1. set the goals first;
  2. come up with a specification that the client/user can understand;
  3. come up with a specification for the coders which includes the data model;
  4. provide some possible later improvement ideas.

Don't expect a "usual" D8 introduction: there’s going to be no speaking about .yml files and structuring the code – there are quite a few of them already out there. I'm going to speak only about my own experiences.

Goals

My goals weren't too high with this project. Honestly, the biggest one was trying to have one of my interests coincide with leveling up in my daily job.

  • Tinycontrol (TC from now on) reports all that it can to a Drupal 8 site of the user's choosing as TC cannot keep logs itself.
  • Every Drupal user can have up to one private TC log.
  • Diagrams of all logged data can be displayed of the last day, week, month or year, with minimum and maximum data displayed with dates.
  • These diagrams should be visible on my stone-age Android phone's default browser. (This is why I can't rely solely on Thingspeak.com.)
  • Learn about D8 while doing all of these.

Considerations

  • TC's "Remote URL" may be max. 127 characters, and it should start with "GET /tc" (7 chars).
  • For security and authentication reasons, the "Remote URL" should contain a write key. Thingspeak uses 16 chars as write key (user-configurable). TC has up to 25 different values to report.
  • Every TC value is encoded as "#XX", so every field eats up 5 chars + separator: "&f=#XX".
  • Conclusion: with 16 chars as key every user can submit up to 16 values. With 15 chars this limit increases to 17 values: "GET /tc?w=123456789012345&a=#XX&b=#XX&c=#XX&…&o=#XX&p=#XX&q=#XX"

Data model

tc_user table

  • uid: unsigned int NOT NULL DEFAULT 0
  • write_key: varchar(15) DEFAULT NULL
  • settings: blob (serialized) DEFAULT NULL

tc_data table

  • uid: unsigned int(10) NOT NULL DEFAULT 0
  • timestamp: int(11) NOT NULL DEFAULT 0
  • field_id: char(1) NOT NULL DEFAULT 'A'
  • field_value: varchar(7) DEFAULT NULL

Functions

User settings (web UI, form)

Provide UI for user to enter:

  • Write key (15 chars, with a random-generated default)
    • Validate uniqueness amongst users
  • For each of the 17 fields:
    • Enabled
    • Name

Data submit (service, page controller)

Provide callback for TC to submit data:

  • As TC does not have any error handling, the Drupal module simply does not store anything if the W field (write key) is unknown.
  • The Drupal module simply stores the first 7 chars for each field, it does not validate the data.

Data display (web UI, page controller)

Provide UI for user to display every enabled fields' data for the past day, week, month and year, using some charting API. (Probably https://www.drupal.org/project/visualization as it seems to be the only one so far with a #D8CX pledge.)

Later improvements

Data import (done) (web UI, form)

Provide UI for users to import their previous data (preferably using the same syntax Thingspeak uses for export).

Data export

Provide UI for users to export their data (preferably using the same syntax Thingspeak uses).

Expose Chart.js settings

Provide UI for users to change some Chart.js settings.

Views integration

For all the data but the settings.

Rules support

So the site owners can set up some rules/actions eg. send users an email when their TC reports above-threshold temperature.

Connecting to Tinycontrol

Usual setup for a TC is a home/office/lab automation tool, behind a firewall with an ever-changing IP address. There aren't too many free dynamic DNS providers, and this functionality depends on the DSL router/firewall. Dynamic DNS is not really needed if TC is set up to submit data every now and then: the Drupal module could store the last IP the TC has connected from, and the users could provide the port they have set up as a forward on for their TC. If the Drupal module knows the IP, the port and the credentials needed for TC, then it should be possible to control TC from a Drupal site – even through a firewall.

Caveats and gotchas during coding

With a decent D7 coder background I thought I knew the main pain factors. Some of them turned out to be the main gain factors, luckily.

  • Rendering a diagram into a dynamic PNG to have my phone understand it is technically possible, but hey, we're in the 21st century. That's how I found Chart.js, which was easier to integrate into my module than anticipated. (I have even played with an extension of it which promised easier date/time display, but that didn't work on my phone.)
  • As much of what I learned about coding was from Knuth's The Art of Computer Programming when the bounty was under a thousand dollars, getting a bit more intimate with OOP while coding for D8 was a must.
  • I was afraid of the new Twig system, being the new theming layer, but it turned out that for my basic theming needs there was no real change. The Batch and Schema APIs is surprisingly the same as in D7 (at least those parts I have used).
  • There are hardly any contrib modules available for D8, so far.. I have even run into several that might have worked in the past, but weren't functional when I tried them.
  • I had some hard time finding how to do this and that, because my D7 skills weren't always good enough, and D8 is so new there aren't too many usable examples. Additionally reading the code requires some alternative thinking, since procedural and object-oriented stuff should be read with two distinctive mindsets. (And I haven't mentioned those classy factorized trait baits.)
  • Don't even think about trying to write D8/OOP code using vim. That's gonna be an epic fail by itself. I tried. Then my colleague helped me to set up a proper IDE, which was the best idea so far.
  • I still have to learn quite a few things regarding D8. Don't take my code as a standard to follow – only as an example that's working somewhat.
  • The code is available on drupal.org. Feel free to send a patch! :)
Nov 02 2015
Nov 02

We are happy to announce that the Drupal module we built for the Context.IO App Challenge just grabbed the 1st Runner Up prize along with a $25,000 cash prize.

About a month ago we asked for your support in helping our module win the Popular Choice prize in the Context.IO App Challenge. Thank you for all your assistance and interest. Though we didn’t win the community outreach award and won’t be able to use that prize money to sponsor events as we originally planned, we are planning to invest the money we won in further development of the Context.IO module; a new release is already out, we are now in beta.

We are also working on community outreach programs and perhaps even developing a customizable B2B solution package.

Next up are in-depth posts about our upcoming plans and the development of the module, where we address challenges we faced and solutions we found. Soon we will also start work on a Drupal 8 version of the module. Stay tuned!

Oct 26 2015
Oct 26

Today and tomorrow you can come and see us at the WalkHub booth at OSCON. It has a bit of the CMS garden feeling (the smallest ever, as Paul Roeland from the Plone community framed it).

We would like to thank Bert Boerland who helped us organise it and Marc Blumenfrucht, Hans Schwirtz and Peter Grond for helping us man the booth.

You can find us at the end of the sponsor pavilion in the not for profit section next to the Plone booth. If you are in Amsterdam and would like to get a free ticket for the exhibition area, send us a mail.

Oct 20 2015
Oct 20

We are looking for volunteers to help us promote Drupal 8 at OSCON next week. The conference will be Monday 26th - Wednesday 28th October in the RAI in Amsterdam, where we had Drupalcon last year. Now that Drupal 8 finally has a release candidate, this is one of the most important moments to be promoting our project, and you can help.

Two weeks ago I heard about an opportunity to get a free exhibition space at OSCON for open source projects. I checked and since we didn’t have an exhibition space yet for the Drupal community I requested one. I also requested a space for WalkHub. I received both.

Together with a colleague we will be manning the WalkHub exhibition space all conference long. WalkHub will be next to Drupal. So occasionally I could take care of both spaces. It would however be better if at least 1 other person could be around to help out, especially when things get busy.

We have a free ticket for you if you can help with the Drupal delegation. If you are attending but can’t help with the exhibition and you don’t have a ticket yet, use the following discount code to get 25% off: Drupal25.

If you are in town, or willing to come help promote Drupal, send me a mail [email protected].

Sep 16 2015
Sep 16

To get site owners to start thinking about upgrading to Drupal 8 and to build anticipation for the upcoming release we are organising an upgrade Bingo and we want the whole Drupal community to benefit from it. If you want to get leads for site upgrades, Drupal 8 hosting, or paid Drupal 8 module development, you should not miss out.

Join the Bingo to:

  • Make your current and prospective customers excited about upgrading their existing sites to Drupal 8
  • Find out who actually considers an upgrade and what their budget is
  • Be featured as one of the companies in the community that help make D8 happen

What we need from you:

  • Help promote D8 Upgrade Bingo to your customers
  • Help promote D8 Upgrade at the events you are going to be present at
  • Donate at least 2 prizes to the Bingo pot (suggested prizes: upgrade feasibility study, a week of module upgrade work, 2 weeks of D8 prototyping, a D8 hosting package ...)

How it works:

  1. You promote your affiliate link for D8upgrade to your current and potential customers (e.g. at your booth and through your mailing lists)
  2. A site owner uses your affiliate link to submit the modules used in their site
  3. They sign up for the Bingo and we generate them a digital Bingo card (Bingo cards have modules on them that are used in a customer’s site that haven’t been ported yet)
  4. When a module gets an alpha release, or when an alternative upgrade path gets suggested on D8upgrade (e.g. a tutorial for replacing Nodequeue with Drupal core gets added as an upgrade path), that module gets flagged on a site owner’s Bingo card
  5. The first site owner with a full Bingo card wins a prize

You are the only one that gets the leads that have signed up through your affiliate link, we’ll give them to you for free. We ourselves will only contact people that didn’t use an affiliate link to sign up for the Bingo.

Upgrade Bingo explained

A few months ago we launched D8upgrade, a free service for the community that tells site owners when the majority of their contributed modules have been upgraded to Drupal 8. We’ve been test running the site for a while now and have already over a 100 site reports. We’ve made it extremely easy to submit a site, a site owner only needs to copy paste their module list into our form, no modules need to be installed.

As you probably guessed, we started this project to generate leads for our company Pronovix. But from day one we’ve announced that we wanted this to be a community initiative. We want other consultancies to join the effort so they can also get access to the leads that are being generated. So far we didn’t have a mechanism to enable other consultancies to participate. We want to change that now in the 2nd phase of the D8upgrade project.

D8upgrade Bingo! is an online competition in which site owners play a form of Bingo with a series of Drupal 8 related prizes in the jackpot.

To get a Bingo card, a site owner needs to submit a Drupal 6 or 7 site on D8upgrade.org and opt in for the Bingo. Every day we’ll check what modules have been upgraded. Once a module has at least received an alpha release, it can be crossed off from a site’s online Bingo card. We’ll keep on doing that until we run out of prizes. Any module with an alternative upgrade path will also be automatically crossed off the list.

To be eligible, a site will need to have at least a minimum number of modules that don’t yet have an alpha release in D8. We don’t yet have the final game rules, we are still doing some statistics to model the game dynamics, but we expect that there will be at least 2 categories of sites:

  • small sites with only a few contrib modules that don’t yet have an alpha release
  • large sites with a big number of missing contrib modules

How does the lead sharing work?

We are setting up an affiliate system that will keep track of the URL a site owner used to enter the site. When one of your (prospective) customers takes part in D8upgrade Bingo, they will be added to the lead list that we will share with you.

When does it start?

After the Beta to Beta upgrade announcement we expect (or should I say hope?) that there will be a major announcement around D8 at Drupalcon Barcelona. We think it’s the perfect time to make a splash with a cool and fun campaign to get Drupal 6/7 site owners to start seriously contemplating the upgrade to D8.

We first wanted to launch the upgrade Bingo in Barcelona, but we’ve decided to wait until we’ve had a chance to organise a BoF in Barcelona where we can first meet with interested consultancies so we can answer any questions you might have.

Do you want to participate? Do you have any crazy/fun ideas to make this even better?

Sign up for the Bingo participant mailing list and join our BoF in Barcelona.

Sep 08 2015
Sep 08

Voting just opened for the Popular choice prize of the Context.IO App Challenge on DevPost. We’ve submitted a Drupal module for the competition and it would be really great if we could win the Popular choice prize. We are however not asking you to vote to support us, instead we would like you to vote for the Drupal community: if we win the prize we will use it to prove to developer tool companies that Drupal is a market worth investing in. Should we win, we’ll split the 5k$ prize money over 2 sponsorships for upcoming Drupalcamps.

When we learned about Context.IO's App Challenge a month ago, we decided to participate with an alternative for Drupal’s popular mailhandler module. The module has a feeds based interface and exposes a ton of options to import email from your mailbox. Our developers got so excited about the project that we even implemented a UI that lets you parse information from an email (e.g. the project name or issue number from your ticketing system).

Today there are way too many third party integration modules that are maintained by unpaid volunteers. With our submission we want to demonstrate that there is a really big opportunity for these companies to get their product in front of Drupal developers. This is really important for the Drupal community: if we can prove the value of the Drupal market, more companies will spend their marketing dollars on maintaining and promoting their integration modules.

How can you help?

Help us win the popular vote for our submission, to cast your vote:

  • Navigate to our submission page
  • Login or sign up (it’s worth to sign up for DevPost because you’ll get notifications about similar app challenges)
  • Vote in the top right corner next to the video


Know an event that could use the cash?

Tweet about our submission with your event hashtag and a link to the voting page. If we win, we’ll donate 2,5k to the 2 events with the most retweets.

Retweet to support (we’ll add links to any other camps that send out a tweet):

  • BADCamp: Tweet: Vote for #Drupal in the @ContextIO apps challenge - retweet to help #BADCamp secure 2,5k$ sponsor money #B2Dev http://ctt.ec/v25fh+
  • Drupalcon: Tweet: Vote for #Drupal in the @ContextIO apps challenge - retweet to help #DrupalCon secure 2,5k$ sponsor money #B2Dev http://ctt.ec/959oh+
  • Drupalcamp Vienna: Tweet: Vote for #Drupal in the @ContextIO apps challenge - retweet to help #dcvie secure 2,5k$ sponsor money #B2Dev http://ctt.ec/5U01b+
  • Add your own Drupal event: Tweet: Vote for #Drupal in the @ContextIO apps challenge - retweet to help #OUREVENT secure 2,5k$ sponsor money #B2Dev http://ctt.ec/XQ9T3+

Need a mail import solution?

Download and install our module on your website, the higher our stats on Drupal.org, the bigger our chance to win one of the other prize categories. Any money we win with that we’ll use to further improve the Context.IO module and to promote the Drupal opportunity to other developer tool companies.

Sep 07 2015
Sep 07

Building a large Drupal site may be a complicated task, but building small parts of it as prototypes helped us to build Go Ask Alice!, a flagship site of the Columbia University in the City of New York. If you are about to build such a large site, then continue reading why, what and how you should build using prototypes. (This is a transcript of my part from our presentation at Drupalaton 2015 with Kristof Van Tomme.)

To validate design ideas: Projects rarely begin with a clear understanding of needs, or at least what is worth building. Prototyping allows everyone to understand the difference between what they think they need and what they (really) need.

To validate implementation ideas: Once the developers have at least some grip on how to build what the client needs, they can try out different tools for reaching the goal. Most often the tool that seemed to be the best at first sight is not the one that makes the most sense.

To decrease risk: If developers start building based on a misunderstanding of project needs, all parties involved will be disappointed, as they will need to rebuild a large part of the work. This means additional time, additional costs, and lower confidence level in all parties. Some of the developers' ideas may not be as clever as others, so identifying which ones are the most trustworthy allows them to recognize early what’s worth pursuing.

To facilitate communication: Most of the time clients and developers are from a different planet, or at least speak different languages. Proper understanding of each other is a key for effective communication, which builds trust between the parties.

To minimise custom code: Developers can show customers what is already available as a community contributed solution, which is good for at least two reasons. First, when more of the site uses existing code, less development is needed. Second, building more of the site on community-supported solutions lowers maintenance and security risks.

To focus in on minimal viable product: If the scope gets too wide, the outcome may be nice and shiny, but the project in general may still be an epic fail. Stephenson's steam locomotives have validated that public inter-city transportation can be beneficial – and only later iterations introduced some advanced technologies on railways like TGV. Stephenson's project could not have been a success if he had wanted to build TGV.

Prototypes give everyone a hands-on example of a proposed solution, and take very little time to prepare. Concrete examples replace abstract ideas about needs and implementations. Using them in a review session (not just presenting them) facilitates discussions of limitations and opportunities. We build several quick, minimalistic prototypes of alternative approaches, then share our recommendations as clients explore them.

When prototyping, you could for example create a site for every possible type of contact-like form: first with core contact.module, second with Webform, third with Entityform, fourth with MailChimp, etc.

You could also create a site for every possible type of workflow tool the client might make use of: first with Workbench and Workbench Moderation, then with core and Views and VBO and Rules, etc.

…and these are only two example areas with six and ten prototypes, respectively (on our project). Usually there are many-many more of them.

Use free hosting: There are a ton of providers (visit this LevelTen blog post for a comparison). There is no point in paying for sites that the developers will not want to maintain and will not want to use longer than needed. On the other hand, it is worth keeping those prototypes for reference until the end of the development. Finally, developers will not have to explicitly delete the files as most free Drupal hosting providers will delete them after a year or so.

Basically it means developers should fire up a new site for:

  • each (new) idea
  • each feature (or set of features later on)

Prototypes are simple throw-away Drupal sites following the fail early, fail fast principle. Following this principle leads to lower risks and lower cost, as it will reduce the amount of work to be thrown away once other ideas are validated.

Once we start getting close to the understanding of what the Minimal Viable Product means, we can start integrating the already-validated ideas into a cleanroom. (Don't forget that when enthusiastic project participants speak about the MVP, they most likely mean the whole polished product. That is not minimal at all: check Kristof's related presentation.) Cleanroom is considered to have only the validated ideas where developers are 100% certain that they will be part of the final deliverable (let it be the MVP or the full product).

When there is a skeleton, developers can start adding muscles to it to make it move, then flesh to give it a momentum and finally a skin to make it look better. Basically the skeleton is the data model, the muscles are the core functionality that makes the website work, the flesh is additional nice-to-have features like social sharing, and the skin is the theme of the Drupal site.

With a cleanroom already committed to a git repo, the most meaningful way of additional development is creating the new prototype sites as a fork off of that cleanroom. This approach ensures two things:

  1. Firing up forks of cleanroom for the new prototypes not only lets you validate those ideas, it also validates that the tools used for them are compatible with the already incorporated prototypes. If there is a compatibility problem, it's visible fairly quickly, so developers can adjust. It is pretty easy to merge the new functionality back to the cleanroom later. This is only possible if the developers share a common, clear and documented process for building feature modules.
  2. If the forks are simply branches on the same (origin) repo, then merging them can be as easy as a git merge. However, when you encounter a conflict, then keep an experienced senior Drupal developer handy, as resolving merge conflicts in automatically-generated code (like feature modules) without understanding what they actually do would be a nightmare.

As with each new method, there are dangers.

Clear and documented understanding of featurization is a must (we are talking about D7). All developers must share a common model for that. If this document is missing or somebody is not following it, then merging the prototypes might become a nightmare (or a rebuild of the combined stuff from scratch, which is time-consuming and error-prone). If you don't have such a document, read [Kerasai[(http://kerasai.com/blog/2014/04/08/organizing-features-configuration-man...)’s article on the topic.

This whole process of prototyping yields a considerable number of sites (mostly throw-away ones, but anyway). After a certain period of time and/or above a certain number of them there is not a single human on Earth who can keep track of them. The solution is a catalog, which can be either as simple as a (Google) spreadsheet, or just another dedicated prototype as a Drupal site with a content type. The minimal information about a single prototype is:

  • name,
  • URL,
  • purpose/description.
  • Prototyping makes it easy (or at least possible) to work on different parts of the project different teams: the development team can work on the contact prototype while the PM team discusses the workflow with the client and the graphical team is working on how the already-validated parts of the site should look.
  • These efforts can have different velocity: for example it is possible to spend only 2 hours a week on the contact feature and 20 hours a week on the workflow feature.
  • Additionally, these efforts can have different completeness level: most of the developers can focus on the feature with the highest lead time, while a single junior can work on the contact form that's almost complete.
  • Finally, prototyping makes it easy to work on some dark areas of the project even after launching the MVP. Later on, if you have to maintain the site, the maintenance changes can be implemented as new prototypes.

In general, prototyping allows making decisions as late as you want, or at least as late as possible – which is the core premise of agile development.

And there are two more outstanding benefits:

  • With this approach, you can already start working on real, paid Drupal 8 projects: you can work on the parts that are already covered by D8 core and the possibly available contrib, while you can postpone working on the parts that would rely on not-yet-existing D8 contrib modules.
  • This method is more generally applicable, for example to business methods. You can give some new methods a try, while still keeping the old balls in the air.

It was a really good experience to be the pivot point between such experienced Drupal companies like Pronovix and OwnSourcing. This technique of prototyping was brought in by Kay VanValkenburgh, the lead at OwnSourcing, and proved to be a really helpful method while we were building Go Ask Alice!

The fact that I could attend Drupalaton 2015 and do a presentation about this approach with Kristof Van Tomme, Pronovix’ CEO – days before launching such a large project shows that prototyping works.

And some personal words: I would like to thank both companies that I could attend this event, first time as a member of the Pronovix team. When I first saw the site saying Drupalaton is a "perfect balance of work and leisure, sessions, sprints, beach sports, great food", etc., I thought this was just marketing eyecandy. Then it turned out to be the reality: I could do a presentation (despite the fact that I didn't have too much time for preparing for it, this close to the site launch), I could read my book in the morning on the hotel's private island while sunbathing, I could go out paddling with my friend and boss, I could attend others' presentations which I was interested in, I could have great food and free beer on the party boat. It couldn't happen without the help of all parties involved – thanks!

Aug 26 2015
Aug 26

We’ve just released Context.IO, an API module with a Feeds plugin submodule that uses the context.io API to import emails into a Drupal site. If you build something interesting with the module in the next 6 days, you can participate in the Context.IO App challenge and make a chance to win 50k USD.

Context.IO makes email data accessible through a simple REST API. It makes it a lot easier to get data out of mailboxes, it implements oAuth authentication for the major email platforms, so it’s really easy to add email accounts. No need to add the password of your mailbox as a setting in your site, or to go search what the connection details are for an email provider. Just add the email account, and give access through the oAuth authentication process.

Once connected you can use the filters in the Context.IO feeds fetcher to specify what emails you would like to upload to your site. Currently the module lets you filter messages using the following filters:

  • Subject
  • To, CC and BCC email addresses
  • From email addresses
  • Folder
  • Mails sent before and/or after a certain date

With the Context.IO parser you can further refine the content you want to upload to your site. We are planning to make it possible to strip out repetitive content from the email body and to split structured information in email titles into separate fields, so that you could use this to automatically organise emails sent by a project management system or any other application.

The resulting fields can then be mapped to Drupal entities using existing or custom feeds processors. With the Node processor you can map the email fields into the appropriate fields of the content type of your choice. With the User processor you can add users on your site whenever you receive an email that matches certain criteria.

Once you’ve configured a feed, you can define how many emails you want to upload, or set up an automatic job that will regularly upload new emails into your site.

An important downside to the module is that it adds another party that will be processing your email, if you are already using a cloud email provider this might not be such a big change, but it’s important to take this into account when using the module. Context.io is a free service by Returnpath a Canadian company that aggregates, anonymises and then analyses the emails they process to help their customers detect email fraud, optimise emails and get insight into consumer behaviour.

This sounds scary but it is similar to the kind of analysis major cloud email providers do on the mailboxes of their customers. If you use this module to build an application that other people will connect their mailboxes to, you will need to add a warning about this to your terms and conditions.

We built the Context.IO module because we thought it would be fun to participate in Context.IO's App Challenge. It would of course be great to win the 50k USD cash prize, but it also fits really nicely into our portfolio: At Pronovix we’ve done a lot of Drupal integrations for third party software applications. Because of our previous hustling with Drupal startup projects like WalkHub and Graphmind, we’ve tried and tested a lot of techniques to promote new tools in the Drupal community. We don’t want to just write code for our customers, we want to help them engage with the Drupal community in a way that will create win-wins for everybody.

If we can help companies like Returnpath, Brightcove, Gitlab and Livefyre recognise the value of the Drupal community, they will become sponsors at Drupal events and spend money promoting Context.IO that helps us grow our ecosystem.

So how do you win that 50k$ prize money? Context.IO is organising an App Challenge for the best consumer application that uses their API. They have some major players in the VC / startup space (Fred Wilson, Brad Feld, David Cohen, Matt Blumberg, Joshua Baer) and a $125,000 in cash prizes waiting to see what kinds of amazing applications you can build with email data. It was already easy for developers to integrate with their API, but with our Context.IO module it’s now really easy even for site builders to create a project. There is 6 days left, there are plenty of MVPs you could click together even in a single day.

We are releasing the module before the end of the competition so that other Drupal site builders would be able to take part in the challenge. This might sound counter intuitive, since we are enabling our own competition, but we would really love to see loads of submissions from the Drupal community. By doing a submission you are not only making a chance for some serious prize money, you also help us prove that Drupal is an ecosystem worth investing in, so that other SaaS companies would recognise the Drupal community as a valuable target to spend their marketing dollars. All we ask from you, is that you mention our module in your application.

Jul 16 2015
Jul 16

If you are still hesitating if it's worth the effort to sign up for d8upgrade, we wanted to give you an account from one of our users who did already do so. We asked omong_kosong, if we could interview him for a blogpost, what follows are his answers to the questions we sent him.

What do you think about the d8upgrade service? Was it helpful?

omong_kosong: First of all I really appreciate that you set up such a service at all. As the users of the Drupal community may be more than aware, you folks put a lot of effort into your work, and all of that for free. Therefore, it is remarkable that you take the time to establish the service and it shows how serious you are about services rendered by Drupal and that quality is the main focus of your work. That really gives me, an average user, who is concerned about the quality of the products used anyway, a very good feeling about Drupal. And I take it that not only the quality of the product is to be seen in terms of the overall framework, but also security considerations. And that makes it a big point to use Drupal. Well, the upgrade service is just a great uncomplicated way of being prepared for the "big" change. It gives me the possibility to get acquainted with the special requirements of the Drupal 8 structure and to sort out the module structure in lights of Drupal 8 requirements. And, it just fires up curiosity for "the new kid on the block". The website is easy to use, though one or the other flaw may have to be worked on. I’ll get to those points later. It is also great that you can manage multiple sites by this service. Makes life easier, if you are running lots of different sites.

What information was useful for you on the d8upgrade report you got from us?

omong_kosong: The report gives me a fast and clear overview about the module status in regard to possible upgrades. The color-coded notification is helpful.

How did the site submission process go for you?

omong_kosong: The process went like a charm and may be qualified as easy to use. The only problem I faced was the login process. Neither the username nor the e-mail address were accepted during the process though asked for in the appropriate login checkboxes. You can help yourself with a password request and a hybrid login if available. But maybe you folks could have a check on this problem.

We modified our login process based on this feedback. You can now sign up with a username and your e-mail address, then log in with the username and password, or you can choose to sign in with your Google+ account. If you choose this option, just click on the Google+ icon in the left sidebar to log in.

I learned about the service through a Drupal newsletter. I am not sure if I would have noted it via other services. Maybe it would be a good idea to post that services on other platforms. You are asking users to tweet about the service and post about it on facebook. Maybe it would also be a good idea to issue "press releases" to IT-magazines and ask website admins to spread the word.

Thank you for these ideas, we are continuously thinking about ways to promote the d8upgrade service.

If you would get the chance to change one thing on d8upgrade what would you change?

omong_kosong: I had some trouble with posting my module list. As it is noted on the upgrade website, the process shall be smoothened. But it took some work to get things done and post the module list the right way. But then I was really overwhelmed by the helpfulness of the people running the website. They approached me with good advice on how to do things. Wow. Nothing you would expect in the world of the web in the first place. Often you get harsh messages, mostly advising you to RTFM. Not so with the guys from the d8 upgrade service. That was formidable and I am therefore more than happy to give something back to the community with my little comments on those few questions asked.

We are glad you had a good experience communicating with our support team. We keep providing as much help as we can to make the process easier for our users.

Anything else you’d like to share with us and the community?

omong_kosong: Well, I guess most of it is said already. Just one more word. Those guys give us Drupal users a great service. We all should strive to assist them with their great work and make our product Drupal the best CMS around.

Jun 01 2015
Jun 01

We started our free d8upgrade notification service to keep site owners informed about the status of their modules so that they can see when they are ready to upgrade to Drupal 8. We are happy to announce that we’ve finished the first batch of upgrade reports.


Excerpt from a site upgrade report

Several of the people who have submitted their sites already received their reports, but now we finished the first round for all of them. Although it took a while to scan the sites and create the initial reports, the process will be much faster from now on. We are planning to provide site upgrade reports on a monthly basis and might increase the frequency once Drupal 8 is released.

To check your site upgrade status, all you have to do is copy and paste your module list and we will take care of the rest. The reports contain a list of your modules along with the latest release information color-coded for a clear overview.

We are planning to get in touch with the maintainers of the most popular modules. The more people submit their site's modules, the more interesting it will be for module maintainers to send us updates about their plans. So any way you can help us promote d8upgrade is much appreciated!

Could you do us a favour? Could you help us promote the service with a tweet or a message on Facebook?

We will also reach out to agencies and Drupal shops and ask them to join our 2for1 campaign, that strives to support the development of Drupal 8 by offering customers a two dollar discount on their future Drupal 8 project for every dollar they contribute to Drupal 8 initiatives (up to a 10 % total discount).

We strive to continuously improve our service, so if you have any comments or suggestions, please let us know!

Apr 16 2015
Apr 16

This is the story of how I built a first prototype for LinkForward, a web application for power networkers, and how I built it in record time using Drupal (12 hours), without a single line of custom code.

Today, MVP (Minimum Viable Product) is a part of every startup founder’s vocabulary. But almost 10 years after Steve Blank first published his ideas on customer development in "The Four Steps to the Epiphany”, I’ve seen myself and countless other startup enthusiasts over-invest in the development of their first prototype for web applications. When I talk with founders about their startup they will often say they are building an MVP and then immediately go on listing a long list of complex expensive features they believe their product will not be viable without (instant messaging, video chat, you name it). The worst is, I’ve done the same several times now. In the beginning of the year, when I got the idea for LinkForward, I decided to do things differently.

The goal

LinkForward is a web application that makes it easier to build and maintain a high quality professional network. Its goal is to make it easy for 2nd degree connections - that don’t know each other but that have someone in common - to introduce each other to their own network. I started building it as a side project because I felt that there is a serious need for a tool that makes it easier to be a power networker. Since mid 2014 I have been working much more consciously on my network. Even though I had returned to Belgium 5 years ago, I realized that I wasn’t really connected into the IT scene at home. Along the way I started feeling that there is an important gap in my networking toolset.

I’ve now got over 2k connections, and many of them I no longer remember. I would love to have a tool that helps me make stronger, more meaningful connections. That helps me build a vibrant, active network with those people that will help me make a bigger impact on the world. Just before New Year, I had joined a friend and her co-founder to advise them about their startup. To implement their idea they wanted to build an AI that would be able to make recommendations to people. In that discussion I suggested them to think of using a network of people as an alternative for expensive and complex AI, this is how I got the idea for LinkForward.

What is wrong with Linkedin

I really love Linkedin, I was never really good at organising business cards, and Linkedin gave me a much better alternative to keep track of my professional network. While Linkedin is great for retaining business connections, it’s inadequate if you want to actively foster your relationships. I see 3 big problems:

  1. In the name of greater engagement Linkedin is actively seducing their users to waste time on empty/shallow interactions: Linkedin’s news feed is saturated with listicles and other sensational but forgettable content. I don’t think that a single click “like XYZ’s new profile picture” helps you build a meaningful relationship in a professional context.
  2. All the incentives are geared towards indiscriminately accepting connections. Recruiters and sales people, the premium users on Linkedin need to build as many connections as possible to increase the number of people they can reach. As a result on a weekly basis I get connection requests from accounts that are obviously fake but that have already accumulated over a hundred connections.
  3. Behaviour reinforcement loops in the system were primarily built for new users (there are some positive improvements on this area recently, but the mass market nature of the system means it is not primarily geared towards power users).

Pretotyping

A little while back I read Pretotype it a free book by Alberto Savoia. While his ideas are very similar to the Minimal Viable Product that Steve Blank popularised, this book helped me to finally get it. You see, when Steve Blank introduced MVP he left it really open what viable meant exactly. And there are a lot of interpretations that range from:

  • Landing page: A page that explains a product before it is built and that lets people sign up to receive notifications about the product.
  • Crowdfunding campaign: Using a platform like Kickstarter or Indiegogo to raise funds from your future customers.
  • Mockup prototype: A prototype built in a mockup tool, be it Balsamiq, Powerpoint, or something similar.
  • Concierge MVP: a manual service, where the founder offers a service that mimicks the product as you envision it to be working in the future.
  • Mechanical turk MVPs (or also “Wizard of Oz MVP”): On the outside it looks as if everything is running automatically, people interact through a web interface with your product, but on the inside you are doing the most expensive interactions by hand.
  • Minimum Loveable Product: a counter action against MVPs that were too minimal. Often focussing on design, to make a product more “loveable”.

The problem is that most startup founders go from demand validation (e.g. talking with potential customers, landing pages, mockups or crowdfunding campaigns) straight to their Minimum Loveable Product. In the process a lot of founders invest loads of time and money in demand that was only validated as an idea. And this is where a lot of people fail badly: they get initial traction and start investing in an idea that people think is great, but there is a really big difference between something that sounds great and something that will actually work.

That is why I’ve learned the hard way that you should build a pretotype before you build your version one. And your pretotype should be modelling the engine of your startup, the key interaction that you believe will be bringing value to your customers.

If the engine doesn’t work you can add all the whistles and bells you want, they will not make your startup a success. On the other hand if you can get that basic interaction going, even if it’s only for a small group of people, and not very effectively, you can start experimenting how you can get from there to a product that people will not just like but actually love.

Pretotyping LinkForward with Drupal

So how do you build your startup’s engine? For LinkForward the initial idea was that introduction requests would be the engine of the community platform: When a user signs up, they can ask an introduction request from the community. When another user in the network knows someone who would be a good introduction, they can flag that person and receive an email they can forward to the person in question.

First iteration (8h) - from idea to pretotype

I built the first iteration for LinkForward on two consecutive evenings, in about 8 hours of work. To do so I used Drupal Gardens, a hosted Drupal solution built by Acquia. With it, it’s really easy to get a Drupal site started without any developer or system administrator knowledge.

It had been a while since I last built a Drupal site, and my Drush knowledge had become a bit rusty (read I forgot everything). I could have asked my colleagues to set up a site for me, or have brushed up my command line knowledge, but I wanted to show that it is possible to make a prototype in Drupal without any real coding skills. The only thing I asked help with was the setup of the nameserver so that I could have a real domain and not just a subdomain.

Setting up the introduction request was really easy: I just added the fields for it to the user profile. The most of my time I spent on tweaking the View that was going to display one user’s ask at a time and the 2 flags I created to let a user interact with the system. Since I didn’t want to involve a themer, I had to do some trial and error positioning the fields using an HTML table (ok maybe this is a bit more hardcore Drupal skill, but I personally don’t consider HTML to be code, since you are manipulating the output directly).

Go live (4h) - inviting the first users

After these 2 evenings I spent another 4 hours doing some extra polishing. I’m not sure anymore when I wrote the content for the home page and the about page, but after approximately 12 hours of work, I went onto Skype and pinged about 70 friends, mostly from the Drupal community, who I thought the product might be relevant for. That is how I got the first 7 users.

Several friends suggested me to check out BNI (Business Network International), "the world's largest referral organization" founded in 1985 by Dr. Ivan Misner. I found a chapter in Ghent that was having their weekly meetup on Wednesday at 7am, I called them up and was able to attend the next morning. From the approximately 30 people that morning 12 signed up for an account.

The 3rd group I recruited from was the attendees at BootCircle, the monthly meetup I organize for startup founders in Ghent and Leuven. I also handed out flyers (black&white hand torn copy machine quality) at a few other meetup events I attended, that got me another 12 participants. Including a few colleagues that brought the total to about 40 participants. Meetup.com, a site that facilitates the organisation of local "user group” style events is a really powerful tool for setting up ad hoc meetings where you can get feedback quickly from people in your target audience.

Engagement experiments

With this initial group of people, I could already see that the engine was not yet working: most of the people that signed up didn’t return, even after they had been prompted. The engagement rate was also rather low. That is why I took the site off of Drupal Gardens so that I could do a few experiments with gamification modules.

I used the user points module in Drupal to create introduction points: every time you flagged that you wanted to introduce somebody, a point was awarded to you and subtracted from the person you indicated you could make an introduction for, a typical gamification scheme. Initially I used flag actions for this from the flag module. But I bumped into some of the limitations of the flag module, and in the end I decided to power up.

This is why I installed the rules module. Rules is Drupal’s blackbelt toolbox, it allows you to model complex business logic without a single line of code. Sending emails, adding or removing points, but also publishing and unpublishing nodes, there are few things you can’t do out of the box with Rules.

I wanted to encourage my users to invite their friends to the platform. Setting this up is trivial with the invite module if you are fine with a simple email invitation system. It’s also possible to extend the module with other types of invitations, but those don’t come out of the box. The invite module works together nicely with the user point module, so that you can award points when a member invites a new user to the platform.

At some point I thought I might be able to improve engagement if I would offer offline introduction groups a virtual place to facilitate the introductions they made. That is why I installed the organic groups module that allows you to set up sections of your site where users get different permissions depending on their group membership.

To make it easier to create a new account and to login, I added Hybridauth a module that lets you integrate with a host of social login services. With it, you can create a new account on the site with a single click of a button, copying your information from the social network service.

Francis, a founder friend who I am masterminding with about LinkForward, suggested that we might increase engagement if a customer could add more than one introduction request. I moved the request fields into their own content type and refactored the views and flagging system to enable it.

All of these changes happened on the live site in configuration only. So not in a code driven way, using features, the way we normally build sites for our customers. I would strongly recommend against this kind of live fine tuning for established business platforms where customers have a fitness and continuity expectation. For a startup however, this kind of agility and live tweaking makes Drupal an extremely powerful platform that allows you to learn at minimal cost in really short time frames. With proper warning, I believe this could even be used in an enterprise environment for a beta pilot.

Summary: Why Drupal is a great tool for modeling startups

  • It is clickable, you don’t need to write code
  • Easy to build a complex content model and to visualize it in rich data display that you can build with views
  • It has the building blocks to create viral loops with modules like the Invite module
  • You can use it to create gamification processes with the User points module
  • Social login is easy to set up with Hybrid Auth, and allows you to easily recognise spam accounts
  • You can model complex business logic with the Rules module
  • User interaction processes can be built with the Flag module

Conclusion - More refactoring ahead

Linkforward is not yet working the way I want it to - its engine is still sputtering - but through the experiments I’ve done in the past months I’ve learned a lot of valuable lessons. Crucially those lessons were really affordable: I estimate I’ve spent about 80-100 hours of my own time, my assistant has spent about 10 hours sending emails as part of one of the experiments, I had about 10 hours of technical support from our developers to set up the site and to install the modules.

Next for Linkforward I’ve got a big overhaul in mind and I’m still considering if I want to throw away the current site, copy over the content and rebuild, or if I want to continue working with the existing site base. There is a minor image upload bug that I don’t want to waste developer time on, so I might as well just restart for the new prototype. I can copy over a lot of the configuration work, so that wouldn’t be such a big deal.

I’m really excited about this project for another reason: It demonstrated that it is possible to provide a lot of value for very little developer time. That is why with Pronovix we’ve decided to launch a service based on this principle: dripdojo.com. It is a service specifically designed for intrapreneurs or entrepreneurs who would like to take a hands on role in the design of their innovation project. The goal is to help them validate the engine of their idea before larger investments are made.

We’ve also been actively using our experiences when advising our startup customers, nobody can afford investing in features that people won’t use, especially not when you are trying to launch your startup...

Pages

About Drupal Sun

Drupal Sun is an Evolving Web project. It allows you to:

  • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
  • Facet based on tags, author, or feed
  • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
  • View the entire article text inline, or in the context of the site where it was created

See the blog post at Evolving Web

Evolving Web