Isolated (and Parallel) Test Execution in PHPUnit 4

One of the more popular feature requests for PHPUnit is the ability to optionally execute each test using a separate PHP process. As of yesterday, the parallel_test_execution branch has a prototype implementation that, currently unconditionally, uses a fresh PHP interpreter for each test. The advantages of this include full test isolation and the fact that a test can now cause a PHP fatal error or even a segmentation fault of the PHP interpreter without interrupting the test execution. The fact that a fresh PHP interpreter is used for each test also results in reduced memory usage, especially for larger test suites. However, the test execution takes longer due to startup and shutdown costs of the PHP processes. This may be remedied (and hopefully the test execution performance will increase) by the (soon to be implemented) ability to run multiple tests at the same time by spawning for than one PHP process at once. Using a separate PHP process for each test does not come only with a performance problem (see above). The tests from a test suite such as the one of the eZ Components currently does not work with the parallel_test_execution branch as the "bootstrapping" that is performed by the test suite (registering an __autoload() function in the case of the eZ Components) is not (yet, at least) propagated to the child processes. Maybe the solution for this would be yet another set of setUp() and tearDown() methods that are when the child PHP process starts up and ends, respectively. As for controlling and configuring the process separation, I am currently thinking of a global configuration flag that can have three settings:

  1. Per-Test (default) decision of whether or not to run the test in a separate PHP process
  2. Always use separate PHP processes at all to run the tests
  3. Do not use separate PHP processes at all to run the tests
The @isolated annotation can then be used on
  • a test method to configure on a per-test basis whether or not a single test should run in a separate PHP process
  • a test case class to configure on a per-test basis whether or not all the tests of the test case class should run in one separate PHP process
In addition, a configuration flag on a TestSuite object may be used to toggle running the tests aggregated in that test suite object in one separate PHP process. Once the PHP process isolation is implemented, I will look into running the isolated parts of a test suite in parallel to speed things up. At this point I am asking the PHPUnit community for feedback on the above plans. Do they make sense? Am I missing something?