Now the test-runner will check the jerry (server) return code.
Fixed do_eval_at TC crash if jerry was build with --debug option.
JerryScript-DCO-1.0-Signed-off-by: bence gabor kis kisbg@inf.u-szeged.hu
The --test262-object option should be passed independently,
otherwise os.path.dirname(args.engine) in tools\runners\run-test-suite-test262.py
makes no sense
JerryScript-DCO-1.0-Signed-off-by: Yonggang Luo luoyonggang@gmail.com
This makes have option to running test262-es2015 and test262-esnext with debug version
of jerryscript.
On OSX, the $RUNNER -q --jerry-tests are in dead loop, disable it first.
Increase TEST262_CASE_TIMEOUT to 180, so that --build-debug can pass the tests
without timeout
An side effect of this are
```
<test id="built-ins/decodeURIComponent/S15.1.3.2_A2.5_T1.js"><reason></reason></test>
<test id="built-ins/decodeURI/S15.1.3.1_A2.5_T1.js"><reason></reason></test>
```
passed
JerryScript-DCO-1.0-Signed-off-by: Yonggang Luo luoyonggang@gmail.com
When running tests262 cases, the jerry program may exit with error code other than 1,
that's means a crash or JERRY_ASSERT triggered and should be resolved, so dump the test
name and stderr/stdout for tracking those crash in CI.
related issue: #4463
JerryScript-DCO-1.0-Signed-off-by: Yonggang Luo luoyonggang@gmail.com
get_platform_cmd_prefix implement multiple times in multiple function.
Also add function get_python_cmd_prefix, call to python script always
add python command so that on windows, the python script doesn't have
executable permission also can be called.
JerryScript-DCO-1.0-Signed-off-by: Yonggang Luo luoyonggang@gmail.com
Since ES2018 iterator's next method is called once during the prologue of iteration,
rather than during each step. The test is incorrect and stuck in an infinite loop.
https://github.com/tc39/test262/pull/1248 fixed the test and it passes on test262-esnext.
Removing test/language/statements/for-of/iterator-next-reference.js from test262-es2015
is necessary, because it won't pass ever and only slow down testing.
JerryScript-DCO-1.0-Signed-off-by: Csaba Osztrogonác csaba.osztrogonac@h-lab.eu
This patch disables automatic detection of module code, and instead
requires the user to explicitly specify whether to parse a source
as a module or as a script.
To achieve this the jerry_parse API function now takes a new option
which signals that the source should be parsed as a module.
JerryScript-DCO-1.0-Signed-off-by: Dániel Bátyai daniel.batyai@h-lab.eu
Use the platform independent python threading.Timer instead of the unix only timeout tool.
The timeout error are like the following things:
```
C:\Users\lygstate>timeout -version
Error: The specified timeout (/T) value is invalid. The valid range is from -1 to 99999 seconds.
C:\Users\lygstate>timeout 0 python
Error: invalid syntax. The default option does not allow more than '1' times.
Type "TIMEOUT /?" to learn how to use it.
```
JerryScript-DCO-1.0-Signed-off-by: Yonggang Luo luoyonggang@gmail.com
Less operator is not supported between int and None in Python 3. Fix
that Python 2 legacy in test262-harness.py.
JerryScript-DCO-1.0-Signed-off-by: Yonggang Luo luoyonggang@gmail.com
built-ins/RegExp/CharacterClassEscapes tests need more memory than 512Kb.
Additionally increase the timeout to make all tests pass on GitHub CI.
JerryScript-DCO-1.0-Signed-off-by: Csaba Osztrogonác csaba.osztrogonac@h-lab.eu
This PR aims to avoid serializing the exclude_list
before each test_case.
test-262-esnext execution takes only ~2 mins (8-thread).
JerryScript-DCO-1.0-Signed-off-by: Adam Kallai kadam@inf.u-szeged.hu
Changes:
- Add new option to run-tests.py: --test262-test-list to run selected tests only
- Fix exclude list updater accordingly
- Run ESNext tests on GitHub CI in two batches to decrease runtime
JerryScript-DCO-1.0-Signed-off-by: Csaba Osztrogonác csaba.osztrogonac@h-lab.eu
Changes:
- Progress meter is precise and less verbose
- Omit list of failures, test262.report file contains it
JerryScript-DCO-1.0-Signed-off-by: Csaba Osztrogonác csaba.osztrogonac@h-lab.eu
Changes:
- Don't dump each test output to stdout
- Upload test outputs as artifact to GitHub
JerryScript-DCO-1.0-Signed-off-by: Csaba Osztrogonác csaba.osztrogonac@h-lab.eu
Changes:
- Imported and unified test262 test harness for ES2015 and ESNext
- Simplified runner scripts accordingly
- Run tests on CI to be able detect regressions and progressions too
JerryScript-DCO-1.0-Signed-off-by: Csaba Osztrogonác csaba.osztrogonac@h-lab.eu
text parameter of Popen introduced in Python 3.7,
we should use the equivalent universal_newlines.
JerryScript-DCO-1.0-Signed-off-by: Csaba Osztrogonác oszi@inf.u-szeged.hu
Changes:
- Add options to the test runner to be able to update exclude list
- Update the exclude list
- Make Travis CI signal if the exclude list isn't up-to-date
JerryScript-DCO-1.0-Signed-off-by: Csaba Osztrogonác oszi@inf.u-szeged.hu
The output of the test262 runner changed between ES5.1 and ES6,
our script should respect it to signal regressions in return code too.
JerryScript-DCO-1.0-Signed-off-by: Csaba Osztrogonác oszi@inf.u-szeged.hu
With this option you can call a function after the user script and promises have ran, to be able to do assertions that are executed just before the process would exit.
JerryScript-DCO-1.0-Signed-off-by: Daniel Balla dballa@inf.u-szeged.hu
Changes:
- Crashing jerry / jerry-snapshot is always "FAIL",
only the 1 return code is accepted as expected fail.
- "FAIL (XPASS)" string is only used if the test marked
as expected fail, but the return code is 0.
- Simple "FAIL" string is used if the test marked as expected
fail, but the return code is not 0 or 1. (crash)
JerryScript-DCO-1.0-Signed-off-by: Csaba Osztrogonác oszi@inf.u-szeged.hu
Changes:
- Use list comprehension instead of filter in get_tests
to ensure returning list instead of iterable object
- Fix unicode decoding issues with Popen call
JerryScript-DCO-1.0-Signed-off-by: Csaba Osztrogonác oszi@inf.u-szeged.hu
Changes:
* Bash based runners/run-test-suite.sh runner replaced with a python runner
* Common util functions moved to util.py
* Fixed EOL issues in tests directory
JerryScript-DCO-1.0-Signed-off-by: Csaba Osztrogonác oszi@inf.u-szeged.hu
Changes:
- Implemented jerry_port_get_local_time_zone_adjustment on Windows
- Implemented jerry_port_get_current_time on Windows
- Run test262 tests on Windows in PST/PDT timezone
JerryScript-DCO-1.0-Signed-off-by: Csaba Osztrogonác oszi@inf.u-szeged.hu
subprocess.Popen needs universal_newlines=True parameter
to open the file in text mode instead of binary mode.
With this fix readline() returns str instead of bytes.
JerryScript-DCO-1.0-Signed-off-by: Csaba Osztrogonác oszi@inf.u-szeged.hu
Changes:
* Bash based unittest runner replaced with a python runner
* Typo fixed in doctest cmake build system (python executable)
* run-tests.py prints error message if build fails
JerryScript-DCO-1.0-Signed-off-by: Csaba Osztrogonác oszi@inf.u-szeged.hu
The test runner used `&>>` to channel stderr to stdout and append
all that to a file. However, that's Bash 4 syntax, which is not
available everywhere, e.g., on MacOS. Bash pre-4 syntax requires
the classic `>>file 2>&1` notation.
This commit rewrites the test runner to use the portable syntax.
Moreover, it also rewrites `&>` to `>file 2>&1` for the sake of
consistency.
JerryScript-DCO-1.0-Signed-off-by: Akos Kiss akiss@inf.u-szeged.hu
The previous jerry_port interface did not allow timezones to be handled correctly,
even if the host system was up to the task. This PR changes the jerry_port interface
to allow a completely correct implementation to exist, and provides one (for Linux/BSD
systems) in default-date.c.
Fixes#1661
JerryScript-DCO-1.0-Signed-off-by: crazy2be crazy1be@gmail.com
Previously, these would run for several minutes without printing
any progress. Now, at least print the number of tests executed.
JerryScript-DCO-1.0-Signed-off-by: crazy2be crazy1be@gmail.com
The only such test was a legacy of the long removed compact
profile, which is also removed.
JerryScript-DCO-1.0-Signed-off-by: Akos Kiss akiss@inf.u-szeged.hu
With this feature the use can restart the actual debug session
(similar to the multiple source context reset) within a client.
JerryScript-DCO-1.0-Signed-off-by: Imre Kiss kissi.szeged@partner.samsung.com
- Add `--install` option to `tools/build.py`.
- Make use of `--install` in `tools/run-tests.py` by testing the
installed the executables instead of those in the build tree.
Related changes:
- Collect unit test binaries in the `tests` subdir of the build
tree instead of `bin`.
- The `ls`-based collection of the unit test binaries had some
shortcomings hitherto unrevealed (it didn't filter for files so
it could potentially "collect" dictionaries, too), which has now
been replaced with a more stable `find`-based solution.
JerryScript-DCO-1.0-Signed-off-by: Akos Kiss akiss@inf.u-szeged.hu
If the result of a build is not directly executable on the host
system or needs an execution runtime for any other reason (e.g.,
cross-compiled ARM binaries on Intel, emscripten-generated JS
files, binaries built with Valgrind support) then the current test
runners cannot work with them. This patch makes test runners
execute binaries via a runtime given in the `$RUNTIME` environment
variable (if it is specified).
JerryScript-DCO-1.0-Signed-off-by: Akos Kiss akiss@inf.u-szeged.hu
Also remove eval context support. It provides no practical advantage.
JerryScript-DCO-1.0-Signed-off-by: Zoltan Herczeg zherczeg.u-szeged@partner.samsung.com
In one of the previous PR we have encountered a problem,
where the Travis cuts off the test execution
as the generated log file is too big.
By adding a quiet mode for the test runners, we will
only report the failing tests.
JerryScript-DCO-1.0-Signed-off-by: Peter Gal pgal.u-szeged@partner.samsung.com
The test262 python script is not python 3 compatible
so we force it to be executed with python2.
Additionally print out the reason if the execution
failed.
JerryScript-DCO-1.0-Signed-off-by: Peter Gal pgal.u-szeged@partner.samsung.com
