| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Given the constraints that the PHPUnit test classes need to be run
in their original order, that we might have incomplete information
about the test duration, and that the test durations vary significantly
(from 0 seconds to many tens of seconds), we want to make best efforts
to create split_groups with similar durations (and in so doing,
minimise the duration of the split_group with the longest runtime).
The existing algorithm takes the number of tests and the total
duration of tests and tries to create an even split by either duration
or test count (assuming that the zero-duration tests actually take a
similar, non-zero time to execute). This algorithm can potentially
place two long-running tests into the same group - the group might be
close to its duration limit, and the next test may be a long-running
test. In such a case, the group can end up significantly bigger than
desired.
Change the group-splitting algorithm to chunk the tests into up to
200 chunks, and assign the chunks to split_groups using a dynamic
programming algorithm that uses backtracking to find an optimal
allocation of chunks to split_groups.
Bug: T389189
Change-Id: I61ae44f3fca4a890947ea1499a99690a4fcc2bb8
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The PHPUnit parallel test implementation has support for balancing
buckets by duration when timing information is provided for the
test classes.
Update the `phpunit:prepare-parallel:...` action to download the
latest phpunit results cache file and process it as part of the
test suite bucket / `split_group` creation process. If the file
is empty or cannot be download, proceed as before - simply put
the same number of test classes in each bucket.
Bug: T378797
Bug: T384927
Change-Id: Idd167ef8b1d448c236713b1683f768abb6e12655
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Changes introduced in I550b11f81c15eba01359c30e5c59db0715afd4cd for
T379764 resulted in the log files for the phpunit split groups
being generated with invalid names (`phpunit_output_1_1.log` instead
of `phpunit_output_1_database.log`). This meant the
PhpUnitConsoleLogOutputProcessor was no longer able to find the log
files to present a summary at the end of a failed run.
This patch fixes the bug, and also goes some way to refactoring the
code to make it more testable and debuggable.
Bug: T378481
Change-Id: I0c345ea56529e6768f80a00440e9a13aa05ad069
|
| |\ |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Currently if a PHPUnit `dataProvider` has an error, or if there is
a syntax error in a PHPUnit test case, the
`composer:prepare-parallel:extensions` command used to setup parallel
test runs exits with a non-descriptive error message:
In PhpUnitXmlManager.php line 87:
Could not find file for class PHPUnit\Framework\ErrorTestCase
Improve the error-handling so that a meaninful human-readable error
message is produced, and set the exit-status for the failed run to
something other than 1 (the exit status for a generic failure).
Bug: T379764
Change-Id: I9a5b5e986827b9aac9c84454553fecd922ba1f67
|
| |/
|
|
|
|
|
|
|
|
|
| |
Validate the @covers annotation to be valid on CI runs.
This is done for mosts tests via MediaWikiCoversValidator in base class
MediaWikiIntegrationTestCase or MediaWikiUnitTestCase
Only omitted when @coversNothing is used.
This is already used in other library-related test cases.
Change-Id: Ib31db02b4c623e80049a5f4645c77824244b6d6d
|
| |\ |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
The introduction of parallel testing has made it harder to review
the console output of CI runs containing failures as the failure
information is distributed throughout the log output and not in a
single block at the end of the run.
Parse and collect the PHPUnit tests output from parallel runs, and
dump collected errors at the end of the run in a single block in
the case that there are failures.
Bug: T378481
Change-Id: Iafe4604c1a73e812362b234c676cd07c422417c7
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
The round-robin allocation of tests to `split_groups` resulted in
`split_groups` containing tests from a large number of different
extensions, and the composition of `split_groups` changing
significantly depending on which extensions are currently loaded.
Instead, we now add tests to buckets in their alphabetical order so
that tests are more or less grouped by extension. At the same time
we try to preserve the time balance of the buckets.
Bug: T378478
Change-Id: I1da2848b041a2b6a879c64f3a3ee8dd9471fd06e
|
| |/
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The parallel test suite group builder (`TestSuiteBuilder`) currently
adds tests to the split groups in the order that the test classes
are encountered in the output of phpunit's `--list-tests-xml`. This
ordering is unspecified. This may lead to the split groups being
composed differently in different environments.
Sort the test classes by filename before adding them to the split
groups so that we have a deterministic ordering and composition of
splits.
Bug: T375851
Change-Id: I6e573028dbcfb19125c40ed2e25259e49fbf59a0
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
In T361190 and Quibble 1.9.0, we introduced parallel execution of
PHPUnit tests to speed up the CI jobs. The existing implementation
is purely Python/Quibble, and cannot directly be used by developers
locally. With this patch, we re-implement the parallel test
execution already implemented in CI as a composer task so that the
parallel tests can be run locally.
The `phpunit:parallel:extensions` command expects to be run after
`phpunit:prepare-parallel:extensions`, and expects to find 8 test
suites with the names `split_group_X` (for X in 0 through 7) in the
PHPUnit configuration file. 8 here is currently a hard-coded number
that corresponds to the number of parallel test executions we need
to saturate the CPU of a 4-core developer machine, and experimentally
leads to a good speed-up versus the serial execution.
When this command runs, it forks to launch 8 parallel processes,
each running one of the `split_group_X` suites. The parent process
waits for the children to complete, buffers the output, collects the
exit statuses, then dumps the buffered output and exits with a
non-zero status if any of the child processes failed (i.e. if there
were test failures).
We introduce `phpunit:prepare-parallel:default` as a complement to
`phpunit:prepare-parallel:extensions`, and the two commands
`phpunit:parallel:database` and `phpunit:parallel:databaseless`.
This creates four possible combinations - two different test suites,
and two different test groups. This is a similar setup to that which
we have in CI - the Database and non-Database tests are run in
separate groups, and some jobs use the `extensions` suite while
others just use the default suite.
The `phpunit:parallel:...` commands will fail with a helpful message
if no `split_group_`s are found in the active PHPUnit configuration.
To help test whether the split test runs are really running all the
tests in the suite, we generate and store the PHPUnit results cache
file. Comparing the results cache files from linear versus parallel
runs should tell us if all the tests have been executed.
Bug: T365976
Change-Id: If106802f08edd5d4c841bb7970c69b88ab3bb39b
|
|
|
In T361190 and Quibble 1.9.0, we introduced parallel execution of
PHPUnit tests to speed up the CI jobs. The existing implementation
is purely Python/Quibble, and cannot directly be used by developers
locally. With this patch, we re-implement the test splitting logic
already implemented in CI as a composer task so that the parallel
tests can be run locally.
There are a couple of different approaches to running PHPUnit tests
in parallel. The different approaches have been discussed at length
in T50217. Ideally, we would just install the `paratest` extension
and use that to parallelise the execution. Unfortunately we have
complex test suites (specifically Parser tests and the Scribunto
test suite) that dynamically create tests as they run, which makes
it hard for `paratest` to work out which tests will run.
To overcome this limitation, we use the `phpunit --list-tests`
function to create a list of test classes that would be included in
the execution of the test suite, then scan the filesystem for
classes named in the `tests-list.xml` output. The classes we find
are then collected into smaller groups (`split_group_X`) which we
can run in parallel in separate processes.
We split into 7-8 groups here, as that experimentally leads to an
even spread of the tests and consumes 100% of all cores on a 4-core
processor.
Because `ParserIntegrationTest.php` is a single test class that
generates thousands of integration tests, we put that in its own
bucket rather than allocating it round-robin to one of the split
buckets. This again helps to keep the buckets roughly the same size.
The current implementation only supports splitting the `extensions`
test suite. We need to do some more development and testing to
support splitting other suites.
The new composer command `phpunit:prepare-parallel:extensions` will
generate a `phpunit.xml` file with the same contents as
`phpunit.xml.dist`, but with the split-group suites added. The
result of running all of the split groups should be the same as the
result of running the whole test suite.
Bug: T365976
Change-Id: I2d841ab236c5367961603bb526319053551bec2e
|
