1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
|
This folder contains the web platform tests and the
code required to integrate them with Servo.
To learn how to write tests, go [here](http://web-platform-tests.org/writing-tests/index.html).
Contents
========
In particular, this folder contains:
* `config.ini`: some configuration for the web-platform-tests.
* `include.ini`: the subset of web-platform-tests we currently run.
* `web-platform-tests`: copy of the web-platform-tests.
* `metadata`: expected failures for the web-platform-tests we run.
* `mozilla`: web-platform-tests that cannot be upstreamed.
Running the tests
=================
The simplest way to run the web-platform-tests in Servo is `./mach
test-wpt` in the root directory. This will run the subset of
JavaScript tests defined in `include.ini` and log the output to
stdout.
A subset of tests may be run by providing positional arguments to the
mach command, either as filesystem paths or as test urls e.g.
./mach test-wpt tests/wpt/web-platform-tests/dom/historical.html
to run the dom/historical.html test, or
./mach test-wpt dom
to run all the DOM tests.
There are also a large number of command line options accepted by the
test harness; these are documented by running with `--help`.
Running all tests
------------------------------
Running all the WPT tests with debug mode results in a lot of timeout.
If one wants to run all the tests,
build with `mach build -r`
and
test with `mach test-wpt --release`
Running the tests with an external WPT server
---------------------------------------------
Normally wptrunner starts its own WPT server, but occasionally you
might want to run multiple instances of `mach test-wpt`, such as when
debugging one test while running the full suite in the background, or
when running a single test many times in parallel (--processes only
works across different tests).
This would lead to a “Failed to start HTTP server” errors, because you
can only run one WPT server at a time. To fix this:
1. Follow the steps in [**Running the tests manually**](#running-the-tests-manually)
2. Add a `break` to [start_servers in serve.py](https://github.com/servo/servo/blob/ce92b7bfbd5855aac18cb4f8a8ec59048041712e/tests/wpt/web-platform-tests/tools/serve/serve.py#L745-L783) as follows:
```
--- a/tests/wpt/web-platform-tests/tools/serve/serve.py
+++ b/tests/wpt/web-platform-tests/tools/serve/serve.py
@@ -746,6 +746,7 @@ def start_servers(logger, host, ports, paths, routes, bind_address, config,
mp_context, log_handlers, **kwargs):
servers = defaultdict(list)
for scheme, ports in ports.items():
+ break
assert len(ports) == {"http": 2, "https": 2}.get(scheme, 1)
# If trying to start HTTP/2.0 server, check compatibility
```
3. Run `mach test-wpt` as many times as needed
If you get unexpected TIMEOUT in testharness tests, then the custom
testharnessreport.js may have been installed incorrectly (see
[**Running the tests manually**](#running-the-tests-manually)
for more details).
Running the tests manually
--------------------------
(See also [the relevant section of the upstream README][upstream-running].)
It can be useful to run a test without the interference of the test runner, for
example when using a debugger such as `gdb`. To do this, we need to start the
WPT server manually, which requires some extra configuration.
To do this, first add the following to the system's hosts file:
127.0.0.1 www.web-platform.test
127.0.0.1 www1.web-platform.test
127.0.0.1 www2.web-platform.test
127.0.0.1 web-platform.test
127.0.0.1 xn--n8j6ds53lwwkrqhv28a.web-platform.test
127.0.0.1 xn--lve-6lad.web-platform.test
Navigate to `tests/wpt/web-platform-tests` for the remainder of this section.
Normally wptrunner [installs Servo’s version of testharnessreport.js][environment],
but when starting the WPT server manually, we get the default version, which
won’t report test results correctly. To fix this:
1. Create a directory `local-resources`
2. Copy `tools/wptrunner/wptrunner/testharnessreport-servo.js` to `local-resources/testharnessreport.js`
3. Edit `local-resources/testharnessreport.js` to substitute the variables as follows:
* `%(output)d`
* → `1` if you want to play with the test interactively (≈ pause-after-test)
* → `0` if you don’t care about that (though it’s also ok to use `1` always)
* `%(debug)s` → `true`
4. Create a `./config.json` as follows (see `tools/wave/config.default.json` for defaults):
```
{"aliases": [{
"url-path": "/resources/testharnessreport.js",
"local-dir": "local-resources"
}]}
```
[environment]: https://github.com/servo/servo/blob/ce92b7bfbd5855aac18cb4f8a8ec59048041712e/tests/wpt/web-platform-tests/tools/wptrunner/wptrunner/environment.py#L231-L237
Then start the server with `./wpt serve`. To check if `testharnessreport.js` was installed correctly:
* The standard output of `curl http://web-platform.test:8000/resources/testharnessreport.js`
should look like [testharnessreport-servo.js], not like [the default testharnessreport.js]
* The standard output of `target/release/servo http://web-platform.test:8000/css/css-pseudo/highlight-pseudos-computed.html`
(or any [testharness test]) should contain lines starting with:
* `TEST START`
* `TEST STEP`
* `TEST DONE`
* `ALERT: RESULT:`
[testharnessreport-servo.js]: https://github.com/servo/servo/blob/ce92b7bfbd5855aac18cb4f8a8ec59048041712e/tests/wpt/web-platform-tests/tools/wptrunner/wptrunner/testharnessreport-servo.js
[the default testharnessreport.js]: https://github.com/servo/servo/blob/ce92b7bfbd5855aac18cb4f8a8ec59048041712e/tests/wpt/web-platform-tests/resources/testharnessreport.js
[testharness test]: http://web-platform-tests.org/writing-tests/testharness.html
To prevent browser SSL warnings when running HTTPS tests locally,
you will need to run Servo with `--certificate-path resources/cert-wpt-only`.
[upstream-running]: https://github.com/w3c/web-platform-tests#running-the-tests
Running the tests in Firefox
----------------------------
When working with tests, you may want to compare Servo's result with Firefox.
You can supply `--product firefox` along with the path to a Firefox binary (as
well as few more odds and ends) to run tests in Firefox from your Servo
checkout:
GECKO="$HOME/projects/mozilla/gecko"
GECKO_BINS="$GECKO/obj-firefox-release-artifact/dist/Nightly.app/Contents/MacOS"
./mach test-wpt dom --product firefox --binary $GECKO_BINS/firefox --certutil-binary $GECKO_BINS/certutil --prefs-root $GECKO/testing/profiles
Updating test expectations
==========================
When fixing a bug that causes the result of a test to change, the expected
results for that test need to be changed. This can be done manually, by editing
the `.ini` file under the `metadata` folder that corresponds to the test. In
this case, remove the references to tests whose expectation is now `PASS`, and
remove `.ini` files that no longer contain any expectations.
When a larger number of changes is required, this process can be automated.
This first requires saving the raw, unformatted log from a test run, for
example by running `./mach test-wpt --log-raw /tmp/servo.log`. Once the
log is saved, run from the root directory:
./mach update-wpt /tmp/servo.log
Writing new tests
=================
The simplest way to create a new test is to use the following command:
./mach create-wpt tests/wpt/path/to/new/test.html
This will create test.html in the appropriate directory using the WPT
template for JavaScript tests. Tests are written using [testharness.js](https://github.com/w3c/testharness.js/). Documentation can be found [here](http://testthewebforward.org/docs/testharness-library.html).
To create a new reference test instead, use the following:
./mach create-wpt --reftest tests/wpt/path/to/new/reftest.html --reference tests/wpt/path/to/reference.html
`reference.html` will be created if it does not exist, and `reftest.html`
will be created using the WPT reftest template. To know more about reftests, check [this](http://web-platform-tests.org/writing-tests/reftests.html).
These new tests can then be run in the following manner like any other WPT test:
./mach test-wpt tests/wpt/path/to/new/test.html
./mach test-wpt tests/wpt/path/to/new/reftest.html
Editing tests
=============
web-platform-tests may be edited in-place and the changes committed to
the servo tree. These changes will be upstreamed when the tests are
next synced.
Updating the upstream tests
===========================
In order to update the tests from upstream use the same mach update
commands. e.g. to update the web-platform-tests:
./mach update-wpt --sync
./mach test-wpt --log-raw=update.log
./mach update-wpt update.log
This should create two commits in your servo repository with the
updated tests and updated metadata.
Servo-specific tests
====================
The `mozilla` directory contains tests that cannot be upstreamed for some
reason (e.g. because they depend on Servo-specific APIs), as well as some
legacy tests that should be upstreamed at some point. When run they are
mounted on the server under `/_mozilla/`.
Analyzing reftest results
=========================
Reftest results can be analyzed from a raw log file. To generate this run
with the `--log-raw` option e.g.
./mach test-wpt --log-raw wpt.log
This file can then be fed into the
[reftest analyzer](https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer-structured.xhtml)
which will show all failing tests (not just those with unexpected results).
Note that this ingests logs in a different format to [original version of the
tool](https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml)
written for gecko reftests.
The reftest analyzer allows pixel-level comparison of the test and reference
screenshots. Tests that both fail and have an unexpected result are marked
with a `!`.
Updating the WPT manifest
=========================
MANIFEST.json can be regenerated automatically with the mach command `update-manifest` e.g.
./mach update-manifest
This is equivalent to running
./mach test-wpt --manifest-update SKIP_TESTS
|
