Isolated (and Parallel) Test Execution in PHPUnit 4
One of the more popular feature requests for PHPUnit is the ability to optionally execute each test using a separate PHP process.
As of yesterday, the
The fact that a fresh PHP interpreter is used for each test also results in reduced memory usage, especially for larger test suites. However, the test execution takes longer due to startup and shutdown costs of the PHP processes. This may be remedied (and hopefully the test execution performance will increase) by the (soon to be implemented) ability to run multiple tests at the same time by spawning for than one PHP process at once.
Using a separate PHP process for each test does not come only with a performance problem (see above). The tests from a test suite such as the one of the eZ Components currently does not work with the
As for controlling and configuring the process separation, I am currently thinking of a global configuration flag that can have three settings:
Once the PHP process isolation is implemented, I will look into running the isolated parts of a test suite in parallel to speed things up.
At this point I am asking the PHPUnit community for feedback on the above plans. Do they make sense? Am I missing something?
As of yesterday, the
parallel_test_execution branch has a prototype implementation that, currently unconditionally, uses a fresh PHP interpreter for each test. The advantages of this include full test isolation and the fact that a test can now cause a PHP fatal error or even a segmentation fault of the PHP interpreter without interrupting the test execution.The fact that a fresh PHP interpreter is used for each test also results in reduced memory usage, especially for larger test suites. However, the test execution takes longer due to startup and shutdown costs of the PHP processes. This may be remedied (and hopefully the test execution performance will increase) by the (soon to be implemented) ability to run multiple tests at the same time by spawning for than one PHP process at once.
Using a separate PHP process for each test does not come only with a performance problem (see above). The tests from a test suite such as the one of the eZ Components currently does not work with the
parallel_test_execution branch as the "bootstrapping" that is performed by the test suite (registering an __autoload() function in the case of the eZ Components) is not (yet, at least) propagated to the child processes. Maybe the solution for this would be yet another set of setUp() and tearDown() methods that are when the child PHP process starts up and ends, respectively.As for controlling and configuring the process separation, I am currently thinking of a global configuration flag that can have three settings:
- Per-Test (default) decision of whether or not to run the test in a separate PHP process
- Always use separate PHP processes at all to run the tests
- Do not use separate PHP processes at all to run the tests
@isolated annotation can then be used on - a test method to configure on a per-test basis whether or not a single test should run in a separate PHP process
- a test case class to configure on a per-test basis whether or not all the tests of the test case class should run in one separate PHP process
TestSuite object may be used to toggle running the tests aggregated in that test suite object in one separate PHP process.Once the PHP process isolation is implemented, I will look into running the isolated parts of a test suite in parallel to speed things up.
At this point I am asking the PHPUnit community for feedback on the above plans. Do they make sense? Am I missing something?
19/12/2007 at 11:16 Permalink
Reply
19/12/2007 at 11:38 Permalink
The benefits of being able to run the full test suite which doesn't fall over with Fatal Errors or Segfaults, is really worth it - especially on larger projects. Your Continuous Integration overview will then give you a full status of the test suites.
Having multiple PHP processes should go towards (or even totally?) negate the costs of the extra start up time and having extra configurations means that its flexible to mean peoples needs.
As to the downside of not automatically being able to bootstrap the child processes environment I don't think its a huge problem. Having a convention for the bootstrap process be it for isolated or normal tests isn't a bad idea and would mean that you could turn on parallel testing at any point in the future.
As I say it sounds excellent.
Reply
19/12/2007 at 15:34 Permalink
Reply
19/12/2007 at 15:39 Permalink
Reply
20/12/2007 at 02:45 Permalink
Reply
20/12/2007 at 16:08 Permalink
one extra thing to consider: tests that use databases. if database-affecting tests are executing in parallel, and each are trying to put the same database into a known state, couldn't they be stepping on each other?
Reply
20/12/2007 at 16:14 Permalink
Reply
22/12/2007 at 14:41 Permalink
Another question which comes in my mind is to use runkit's sandboxing functionality which should be cheeper than a huge number of forks but I do not know how this sandboxes behave on fatal errors or segfaults.
Reply
04/01/2008 at 13:24 Permalink
Reply