Towards Better Code Coverage Metrics in the PHP World

Remember that I promised to blog about the requirements planning for more sophisticated support of code coverage metrics in PHP tools that I did with Derick earlier this year? While we met in London in February and Montreal in March to discuss this, both of us were too busy to communicate what we want to do yet.

This blog posting will hopefully shine a light on what it is we are planning.

Current Situation

Code Coverage is a common metric used in software testing. PHPUnit supports this valuable software metric by leveraging the PHP_CodeCoverage component which in turn makes use of Xdebug, an extension to the PHP interpreter. The support for code coverage could, however, be improved even further.

Xdebug currently only supports what is usually referred to as Line Coverage. This software metric measures whether each executable line was executed.

Based on the line coverage information provided by Xdebug, PHP_CodeCoverage also calculates the Function / Method Coverage software metric that measures whether each function or method has been invoked. This software metric is actually implemented in a much stricter way in PHP_CodeCoverage because it only considers a function or method as covered when all of its executable lines are executed at least once during test execution.

The Future

In the future it would be nice to make available more sophisticated code coverage metrics such as Statement Coverage, Branch Coverage, Call Coverage, and Path Coverage to the users of PHPUnit, PHP_CodeCoverage, and Xdebug.

Statement Coverage measures whether each statement is executed (a line of code can contain more than one statement). A mapping from statement to the start and end column in the corresponding line of code is desirable but would require changing PHP's compiler. It is, however, possible to extend Xdebug to report the number executable and executed statements for a line of code.

Branch Coverage measures whether the boolean expression of each control structure evaluated to both TRUE and FALSE during the execution of the test suite. This should be relatively easy to implement based on the information provided by Statement Coverage.

Call Coverage measures whether each function or method call is executed and should be possible to implement in userland (in PHP_CodeCoverage) based on static code analysis and statement coverage.

Path Coverage measures whether each of the possible paths in each function or method has been followed. A path is a unique sequence of branches from the entry of the function or methods to its exit. It should be possible to implement this in userland (in PHP_CodeCoverage) based on static code analysis and branch coverage.

As an alternative to Path Coverage it might make sense to implement Linear Code Sequence and Jump (LCSAJ) Coverage instead. This variation of Path Coverage does not require a flow graph and also avoids its exponential difficulty. It should be possible to implement in userland (in PHP_CodeCoverage) based on static code analysis and branch coverage.

The development of these features will take a lot of effort. It will be a challenge to find time in our schedules to work to tackle these features. But I felt like sharing what we are thinking about with you, the users of our tools, as soon as possible.