Highest / Lowest Testing with Multiple Symfony Versions

Parent Feed: 

Symfony 4.0 stable has been released, and, while packed with many new and powerful features, still maintains many of the APIs provided in earlier versions. Just before the stable release, many participants at #SymfonyConHackday2017 submitted pull requests in projects all over the web, encouraging them to adopt Symfony 4 support. In many cases, these pull requests consisted of nothing more than the addition of a “|^4” in the version constraints of all of the Symfony dependencies listed in the composer.json file of the selected project, with no code changes needed. The fact that this works is quite a testament to the commitment to backwards compatibility shown by the Symfony team. However, adding Symfony 4 as an allowed version also has testing implications. Ideally, the project would run its tests both on Symfony 4, while continuing to test other supported versions of Symfony at the same time.

In this blog post, we’ll look at testing techniques for projects that need to test two different major versions of a dependency; after that, we will examine how to do the same thing when you need to test three major versions. Finally, we’ll present some generalized scripts that make dependency testing easier for two, three or more different combinations of dependant versions.

Sometimes, the best thing to do is to bump up to the next major version number, adopt Symfony 4, and leave support for earlier versions, if needed, on the older branches. That way, the tests on each branch will cover the Symfony version applicable for that branch. This is a good solution for PHP projects that are building applications. For projects that are libraries in use by other projects, though, it is better to provide a level of continuity between different versions of your dependencies on a single branch. This sort of flexibility can greatly relieve the frustration caused by the “dependency hell” that can arise when different projects cannot be used together because they have strict requirements on their own dependencies that clash. In the case of Libraries that use Symfony components, the best thing to do is to provide Symfony 4 support in the current active branch, and make a new branch that supports only Symfony 4 and later, when new features from the new release are used.

Current / Lowest / Highest Testing

A technique called lowest / current / highest testing is commonly used for projects that support two different major versions of their dependencies. In this testing scheme, you would ensure that Symfony 4 components appear in your composer.lock file. If you Symfony version constraint in your composer.json file was "^3.4|^4", then you could run composer update --prefer-lowest on Travis to bring in the Symfony 3 components.

The portion of your .travis.yml file to support this might look something like the following:

matrix:
  include:
    - php: 7.2
      env: 'HIGHEST_LOWEST="update"'
    - php: 7.1
    - php: 7.0.11
      env: 'HIGHEST_LOWEST="update --prefer-lowest"'

install:
  - 'composer -n ${HIGHEST_LOWEST-install} --prefer-dist'

In this example, we use the expression ${HIGHEST_LOWEST-install} to determine whether we are running a current, lowest or highest test; this simplifies our .travis.yml file by removing a few lines of conditionals. In the bash shell, the expression ${VARIABLE-default} will evaluate to the contents of $VARIABLE if it has been set, and will otherwise return the literal value "default". Therefore, if the HIGHEST_LOWEST environment variable is not set, the composer command shown above will run composer -n install --prefer-dist. This will install the dependencies recorded in our lock file. To run the lowest test, we simply define HIGHEST_LOWEST to be update --prefer-lowest, which will select the lowest version allowed in our composer.json file.

Highest/lowest testing with just two sets of dependencies is easy to set up and takes very little overhead; there is really no reason why you should not do it. Even for projects that support but a single major version of each their dependencies still benefit from highest/lowest testing, as these tests will catch problems the otherwise might accidentally creep into the code base. For example, if one of the project’s dependencies inadvertently introduces a bug that breaks backwards compatibility mid-release, or if an API not available in the lowest-advertised version of a dependency is used, that fact should be flagged by a failing test.

Supporting only two major versions of a dependency is sufficient in many instances. Symfony 2 is no longer supported, so maintaining tests for it is not strictly necessary. In some cases, though, you may wish to continue supporting older packages. If a project has traditionally supported both Symfony 2 and Symfony 3, then support for Symfony 2 should probably be maintained until the next major version of the project. I have seen projects that drop support for obsolete versions of PHP or dependencies without creating a major version increase, but doing this can have a cascading effect on other projects, and should therefore be avoided. There are also some niche use cases for supporting older dependency versions. For example, Drush 8 continues to support obsolete versions of Drupal 8, which still depend on Symfony 2, to prevent problems for people who need to update an old website.

Extending to Test Three Versions

If you are in a position to support three major versions of a dependency in a project all in the same branch, then highest/lowest testing is still possible, but it gets a little more complicated. In the case of Symfony, what we will do is ensure that our lock file contains Symfony 3 components, and use the highest test to cover Symfony 4, and the lowest test to cover Symfony 2. Because Symfony 4 requires a minimum PHP version of 7.1, we can keep our Composer dependencies constrained to Symfony 3 by setting the PHP platform version to 7.0 or lower. We’ll use PHP 5.6, to keep other dependencies at a reasonable baseline.

"require": {
    "php": ">=5.6.0",
    "symfony/console": "^2.8|^3|^4",
    "symfony/finder": "^2.5|^3|^4"
},
"config": {
    "platform": {
        "php": "5.6"
    }
}

There are a couple of implications to doing this that will impact our highest/lowest testing, though. For one thing, the platform PHP version constraint that we added will interfere with the composer update command’s ability to update our dependencies all the way to Symfony 4. We can remove it prior to updating via composer config --unset platform.php. This alone is not enough, though.

The .travis.yml tests then look like the following example:

matrix:
  include:
    - php: 7.2
      env: 'HIGHEST_LOWEST="update"'
    - php: 7.1
    - php: 7.0.11
    - php: 5.6
      env: 'HIGHEST_LOWEST="update --prefer-lowest"'

install:
  - |
    if [ -n "$HIGHEST_LOWEST" ] ; then
      composer config --unset platform.php
    fi
  - 'composer -n ${HIGHEST_LOWEST-install} --prefer-dist'

This isn’t a terrible solution, but it does add a little complexity to the test scripts, and poses some additional questions.

  • The test dependencies that are installed are being selected by sideeffects of the constraints of the project dependencies themselves. If the dependencies of our dependencies change, will our tests still be covering all of the dependencies we expect them to?
  • What if you want to test the highest configuration locally? This involves modifying the local copy of your composer.json and composer.lock files, which introduces the risk these might accidentally be committed.
  • What if you need to make other modifications to the composer.json in some test scenarios--for example, setting the minimum-stability to dev to test the latest HEAD of master for some dependencies?
  • What if you want to do current/highest testing on both Symfony 3.4 and Symfony 4 at the same time?

If you want to do current/highest testing for more than one set of dependency versions, then there is no alternative but to commit multiple composer.lock files. If each composer.lock has an associated composer.json file, that also solves the problem of managing different configuration settings for different test scenarios. The issue of testing these different scenarios is then simplified to to the matter of selecting the specific lock file for the test. There are two ways to do this in Composer:

  • Use the COMPOSER environment variable to specify the composer.json file to use. The composer lock file will be named similarly.
  • Use the --working-dir option to stipulate the directory where the composer.json and composer.lock file are located.

Using either of these techniques, it would be easy to keep multiple composer.json files, and install the right one with a single-line install: step in .travis.yml using environment variables from the test matrix. However, we really do not want to have to modify multiple composer.json files every time we make any change to our project’s main composer.json file. Also, having to remember to run composer update on multiple sets of Composer dependencies is an added step that we also could really do without. Fortunately, these steps can be easily automated using a Composer post-update-cmd script handler. It wouldn’t take too many lines of code to do this manually in a project’s composer.json and .travis.yml files, but we will make things even more streamlined by using the composer-test-scenarios project, as explained in the next section.

Using Composer Test Scenarios

You can add the composer-test-scenarios project to your composer.json file via:

composer require --dev greg-1-anderson/composer-test-scenarios:^1

Copy the scripts section from the example composer.json file from composer-test-scenarios. It contains some recommended steps to use for testing your project with Phpunit. Customize these steps as desired, and then modify the post-update-cmd handler to define the test scenarios you would like to test. Here is the example test scenarios defined in the example file:

"post-update-cmd": [
    "create-scenario symfony2 'symfony/console:^2.8' --platform-php '5.4' --no-lockfile",
    "create-scenario symfony3 'symfony/console:^3.0' --platform-php '5.6'",
    "create-scenario symfony4 'symfony/console:^4.0'"
]

These commands create “test scenarios” named symfony2, symfony3 and symfony4, respectively. As you can see, additional composer requirements are used to control the dependencies that will be selected for each scenario, which is an improvement over what we were doing before. There are also additional options for setting configuration values such as the platform PHP version. The --no-lockfile option may be used for test scenarios that are only used in lowest/highest scenarios, as only “current” tests need a composer.lock file. Once you have defined your scenarios, you no longer need to worry about maintaining the derived composer.json and composer.lock files, as they will be created for you on every run of composer update. The generated files are written into a directory called scenarios; commit the contents of this directory along with your composer.lock file.

The install step in our .travis.yml can now be done in a single line again, with the HIGHEST_LOWEST environment variable being defined the same way they were in the first example:

install:
  - 'composer scenario "${SCENARIO}" "${HIGHEST_LOWEST-install}"'

The composer scenario command will run the scenario step in your composer.json file that you copied in on the previous section, above. This command will run composer install or composer update on the appropriate composer.json file generated by the post-update-cmd, or, if SCENARIO is not defined, then the project’s primary composer.json file will be installed.

In conclusion, the composer-test-scenarios project allows current / highest / lowest testing to be setup with minimal effort, and gives more control over managing the different test scenarios without complicating the test scripts. If you have a project that would benefit from highest / lowest testing, give it a try. Making more use of flexible dependency version constraints in commonly-used PHP libraries reduce “dependency hell” problems, and testing these variations will make them easier to maintain. Being fastidious in these practices will make it easier for everyone to use and adopt improved libraries more quickly and with less effort.


You may also like: 

Topics Development, Drupal Planet, Drupal
Author: 
Original Post: 

About Drupal Sun

Drupal Sun is an Evolving Web project. It allows you to:

  • Do full-text search on all the articles in Drupal Planet (thanks to Apache Solr)
  • Facet based on tags, author, or feed
  • Flip through articles quickly (with j/k or arrow keys) to find what you're interested in
  • View the entire article text inline, or in the context of the site where it was created

See the blog post at Evolving Web

Evolving Web