Sebastian Bergmann

This is a follow-up to "The Cost of Test Isolation (and other PHPUnit Features)".

Since the previous posting, I have added a backup/restore mechanism for static attributes of classes to PHPUnit. This is yet another feature of PHPUnit that makes the testing of code that uses global state (which includes, but is not limited to, global and superglobal variables as well as static attributes of classes) easier.

The following profiling graph illustrates the performance penalty incurred by this feature on the test execution.

The following profiling graph was generated for an execution of PHPUnit with the --no-syntax-check --no-globals-backup --no-static-backup switches that disable these features (and also the syntac check of test code).

My recommendation is: disable the backup/restore mechanism of globals and static attributes of classes. Not only because this greatly reduces the time required to run the tests. It also exposes tests, and thus the tested code, that interacts with global state. If, and only if, this interaction with global state cannot be avoided you can select the tests for which the backup/restore mechanism(s) should be enabled in a fine-grained way.

If you are interested in the problems associated with code that interacts with global state, especially with regard to testing it, join me for my "Untestable Code session at the php|tek 2009 conference in Chicago, IL, US in May.

Some of PHPUnit features come with the cost of a performance penality. This posting explores the effect of the --no-syntax-check, $backupGlobals = FALSE;, and --coverage-html options.

Syntax Check: disabled, $GLOBALS Backup: disabled, Code Coverage: disabled

By default, PHPUnit performs a syntax check on the test source files before it loads them. This behaviour can be disabled using the --no-syntax-check option.

Also by default, PHPUnit runs your tests in a way where even changes to global and super-global variables (such as $GLOBALS) do not affect other tests. The backup and restore operations for the global and super-global variables that ensure this behaviour can be disabled by declaring $backupGlobals = FALSE; in a test case class. This was done for the benchmark below.

sb@ubuntu Money % time phpunit --no-syntax-check MoneyTest
PHPUnit 3.3.6-dev by Sebastian Bergmann.

......................

Time: 0 seconds

OK (22 tests, 34 assertions)
phpunit --no-syntax-check MoneyTest
0.16s user 0.08s system 100% cpu 0.235 total

Syntax Check: disabled, $GLOBALS Backup: enabled, Code Coverage: disabled

The following benchmark clearly shows the cost of the calls to the serialize() and unserialize() functions that are performed during the backup and restore operations for the global and super-global variables.

sb@ubuntu Money % time phpunit --no-syntax-check MoneyTest
PHPUnit 3.3.6-dev by Sebastian Bergmann.

......................

Time: 0 seconds

OK (22 tests, 34 assertions)
phpunit --no-syntax-check MoneyTest
0.31s user 0.14s system 93% cpu 0.483 total

Syntax Check: enabled, $GLOBALS Backup: enabled, Code Coverage: disabled

Although the syntax check of test source files involves the creation of a new PHP process, the slowdown does not look too bad according to the following benchmark.

sb@ubuntu Money % time phpunit MoneyTest
PHPUnit 3.3.6-dev by Sebastian Bergmann.

......................

Time: 0 seconds

OK (22 tests, 34 assertions)
phpunit MoneyTest
0.31s user 0.16s system 95% cpu 0.487 total

Syntax Check: enabled, $GLOBALS Backup: enabled, Code Coverage: enabled

The performance penality that comes with code coverage data collection and report generation that we see in the final benchmark has already been discussed previously on this blog here and here.

sb@ubuntu Money % time phpunit --coverage-html /tmp/coverage MoneyTest
PHPUnit 3.3.6-dev by Sebastian Bergmann.

......................

Time: 0 seconds

OK (22 tests, 34 assertions)
Generating code coverage report, this may take a moment.
phpunit --coverage-html /tmp/coverage MoneyTest
0.95s user 0.26s system 95% cpu 1.275 total

Now that collecting code coverage information is not slow anymore (as of Xdebug 2.0.1), PHPUnit's report generation code (PHPUnit_Util_Report_*) has become a bottleneck.



The image above shows the profile of PHPUnit's report generation code at revision 1473.



The image above shows the profile of PHPUnit's report generation code at revision 1475, after I made the report a code coverage report only. The test result aspect of the previous report was not very usefull, made the report generation unneccassary complex and expensive, and should be handled by CruiseControl, for instance, anyways.



The image above shows the profile of PHPUnit's report generation code at revision 1486, after I eliminated the PHPUnit_Util_Array::sortRecursively() method calls.



The image above shows the profile of PHPUnit's report generation code at revision 1488, after I eliminated the PHPUnit_Util_Report_Node_File::tokenToColor() method calls.



The image above shows the profile of PHPUnit's report generation code at revision 1533 with disabled syntax highlighting.



The image above shows the profile of PHPUnit's report generation code at revision 1535 with the new statistics details.



The image above shows the profile of PHPUnit's report generation code at revision 1586 after applying a set of performance patches by Hubert Roksor.



The image above shows the profile of PHPUnit's report generation code at revision 1603 after applying another set of performance patches by Hubert Roksor.