Micro-benchmarking php streams includes part 2

In part 1 I've reported the results of a micro-benchmark designed to compare the performance of plain php includes vs includes via streams from a sqlite database. In this post I extend the test cases with two more no-includes same work, to serve as a base case and includes from a mysql database, the code is the same as in sqlite case, the connector differs. You can find the code attached at the end previous post.

Overall, having in mind the random factors like netbook load (CPU and IO), the differences between the different test cases are insignificant on the test machine - Acer Aspire One netbook.

Please let me know if you decide to run these benchmarks. I will be especially interested to see what are the differences.

These results are encouraging enough to merit developing a more substantial test. I'm interested in benchmarking drupal. What scenarios would you suggest?

The benchmark

The benchmarking code is this small haskell script, which uses the criterion library to gather, process the statistics and to draw nice plots below.

import Criterion.Main (defaultMain, bench, bgroup)
import System.Cmd (system)

main = defaultMain [
    bgroup "php includes" [
               bench "none" $ system "./clean1.php"
            ,  bench "standard/clean" $ system "./clean.php.txt"
            ,  bench "standard/mixed" $ system "./non-stream1.php"
            ,  bench "streams/sqlite" $ system "./stream1.php.txt"
            ,  bench "streams/mysql" $ system "./stream_mysql.php"
            ]
   ]

I've run it with 200 samples

./bench -tpng:552x368 -kpng:552x368  -s 200

The benchmark results

Base case

The base case uses no include files, but simulates the work by looping 20 times through a print statement.

benchmarking php includes/none
collecting 200 samples, 2 iterations each, in estimated 21.72279 s
bootstrapping with 100000 resamples
mean: 70.03678 ms, lb 60.15143 ms, ub 94.78743 ms, ci 0.950
std dev: 113.6834 ms, lb 63.42229 ms, ub 189.0330 ms, ci 0.950
found 6 outliers among 200 samples (3.0%)
  4 (2.0%) high severe
variance introduced by outliers: 16.000%
variance is moderately inflated by outliers

The outliers are probably the result of the combination of an underpowered system and a number of other applications running at thetime of the test.

Clean includes

This test case includes 20 php files from the current directory.

benchmarking php includes/standard/clean
collecting 200 samples, 2 iterations each, in estimated 21.10023 s
bootstrapping with 100000 resamples
mean: 60.50757 ms, lb 58.28822 ms, ub 65.57988 ms, ci 0.950
std dev: 22.93855 ms, lb 9.368661 ms, ub 40.26889 ms, ci 0.950
found 28 outliers among 200 samples (14.0%)
  13 (6.5%) high mild
  15 (7.5%) high severe
variance introduced by outliers: 1.000%
variance is unaffected by outliers

The difference with the base case should be insignificant, but is probably affected by random system load. It does hint that the differences displayed previously are not significant.

Mixed includes

This test case does the same work as the previous one, but includes the stub code for the database includes.

benchmarking php includes/standard/mixed
collecting 200 samples, 1 iterations each, in estimated 134.8954 s
bootstrapping with 100000 resamples
mean: 70.91094 ms, lb 61.86253 ms, ub 101.4884 ms, ci 0.950
std dev: 106.7305 ms, lb 31.32543 ms, ub 240.4033 ms, ci 0.950
found 20 outliers among 200 samples (10.0%)
  11 (5.5%) high mild
  9 (4.5%) high severe
variance introduced by outliers: 14.000%
variance is moderately inflated by outliers

Again, since the variance is moderately affected by the outliers we should take this result with a pinch of salt.

Sqlite streams

This test includes 20 scripts from an sqlite3 database.

benchmarking php includes/streams/sqlite
collecting 200 samples, 1 iterations each, in estimated 13.31139 s
bootstrapping with 100000 resamples
mean: 79.69618 ms, lb 73.51617 ms, ub 98.44982 ms, ci 0.950
std dev: 69.63256 ms, lb 16.12973 ms, ub 150.9165 ms, ci 0.950
found 28 outliers among 200 samples (14.0%)
  14 (7.0%) high mild
  14 (7.0%) high severe
variance introduced by outliers: 6.000%
variance is slightly inflated by outliers

MySQL streams

This test includes 20 scripts from a local mysql database

benchmarking php includes/streams/mysql
collecting 200 samples, 1 iterations each, in estimated 13.17959 s
bootstrapping with 100000 resamples
mean: 70.95483 ms, lb 69.74397 ms, ub 72.55464 ms, ci 0.950
std dev: 10.04377 ms, lb 8.195368 ms, ub 12.52959 ms, ci 0.950
found 19 outliers among 200 samples (9.5%)
  11 (5.5%) high mild
  8 (4.0%) high severe
variance introduced by outliers: 0.500%
variance is unaffected by outliers

The difference with sqlite3 for this benchmark is insignificant.

Conclusion

It becomes clearer that the performance difference is relatively insignificant, even in such micro-benchmarks designed to highlight it, by being unfair to the streams version. If you add a significant amount of code and actually do something with it, like a drupal site would, the difference won't be noticeable. Still this hypothesis needs testing.