|
|
**Notice:** This only works with **python version >3.5** and **NOT with python2**. If run with python2 you get an error message stating that you pytest version is prior 5.0.
|
|
|
|
|
|
# The (python) test system explained
|
|
|
The python test system relies on pytest (https://docs.pytest.org/en/stable/), which is a standard for python and per default installed on most our machines.
|
|
|
Also looking at:
|
|
|
```
|
|
|
|
|
|
The python test system relies on pytest (<https://docs.pytest.org/en/stable/>), which is a standard for python and per default installed on most of our machines. Also looking at:
|
|
|
|
|
|
```plaintext
|
|
|
$pytest --help
|
|
|
```
|
|
|
may provide you with some basic understanding.
|
|
|
If the `pytest` command is not available try out:
|
|
|
```
|
|
|
|
|
|
may provide you with some basic understanding. If the `pytest` command is not available try out:
|
|
|
|
|
|
```plaintext
|
|
|
$python3 -m pytest masci-tools
|
|
|
```
|
|
|
instead, maybe pytest is already installed within python and you can use it as a module. If this is the case you have to replace `pytest` in all further commands with this.
|
|
|
If this command fails, you might need to install pytest using:
|
|
|
```
|
|
|
|
|
|
instead, maybe pytest is already installed within python and you can use it as a module. If this is the case you have to replace `pytest` in all further commands with this. If this command fails, you might need to install pytest using:
|
|
|
|
|
|
```plaintext
|
|
|
pip install -U pytest
|
|
|
```
|
|
|
```
|
|
|
|
|
|
# How to run tests:
|
|
|
|
|
|
To run the tests pytest has to know two things: where to find the `conftest.py` file, which is under `<fleur_source_folder>/tests/new_pytest_system` and which build dir to test for.
|
|
|
|
|
|
## Execution from tests source dir
|
|
|
|
|
|
One way is to go to the `tests/new_pytest_system` folder and to run all tests execute:
|
|
|
```
|
|
|
|
|
|
```plaintext
|
|
|
$pytest
|
|
|
```
|
|
|
(default assumed build dir is `<fleur_source_folder>/build`)
|
|
|
pytest automatically discovers all tests in the any sub directory structure of the head `conftest.py` file, by looking for certain identifiers per default, like files and functions starting with `test_` or ending with `_test` (https://docs.pytest.org/en/stable/goodpractices.html#test-discovery) . During execution pytest will print a standard report for the test run.
|
|
|
|
|
|
(default assumed build dir is `<fleur_source_folder>/build`) pytest automatically discovers all tests in any subdirectory structure of the head `conftest.py` file, by looking for certain identifiers per default, like files and functions starting with `test_` or ending with `_test` (<https://docs.pytest.org/en/stable/goodpractices.html#test-discovery>) . During execution pytest will print a standard report for the test run.
|
|
|
|
|
|
## Execution from the build dir
|
|
|
If you execute pytest from any other directory, you have to provide the path to the folder with the head `conftest.py` and a path to the build dir with the fleur executables, either relative to the `contest.py` file or absolute.
|
|
|
For example if you build dir is `<fleur_source_folder>/build.123`
|
|
|
```
|
|
|
|
|
|
If you execute pytest from any other directory, you have to provide the path to the folder with the head `conftest.py` and a path to the build dir with the fleur executables, either relative to the `contest.py` file or absolute. For example if you build dir is `<fleur_source_folder>/build.123`
|
|
|
|
|
|
```plaintext
|
|
|
$pytest ../tests/new_pytest_system --build_dir=../../build.123
|
|
|
```
|
|
|
|
|
|
or
|
|
|
```
|
|
|
|
|
|
```plaintext
|
|
|
$pytest ../tests/new_pytest_system --build_dir=`<fleur_source_folder>/build.123`
|
|
|
```
|
|
|
|
|
|
The build process also generates a shell script in the build folder to perform the right pytest command for the current build directory. The arguments to this script are forwarded to pytest.
|
|
|
```
|
|
|
|
|
|
```plaintext
|
|
|
./run_tests.sh
|
|
|
```
|
|
|
If you need a non-default python executable for the tests, you can specify this for the `run_tests.sh` script by setting the `juDFT_PYTHON` environment variable with the right executable
|
|
|
|
|
|
If you need a non-default python executable for the tests, you can specify this for the `run_tests.sh` script by setting the `juDFT_PYTHON` environment variable with the right executable
|
|
|
|
|
|
## Executing a subset off tests
|
|
|
## Executing a subset of tests
|
|
|
|
|
|
Pytest ways to run a subset of the test suite:
|
|
|
|
|
|
* Select tests to run based on substring matching of test names.
|
|
|
* Select tests groups to run based on the markers applied.
|
|
|
```
|
|
|
|
|
|
```plaintext
|
|
|
pytest -k <substring> -v
|
|
|
pytest -m <markername>
|
|
|
```
|
|
|
|
|
|
example markers are:
|
|
|
```
|
|
|
|
|
|
```plaintext
|
|
|
bulk,film,xml,collinear,soc,lo,ldau,non-collinear,spinspiral,forces,hybrid,slow,fast...
|
|
|
```
|
|
|
You can register markers in the `conftest.py` file. Also pytest will print a warning if a it does not understand a certain marker.
|
|
|
|
|
|
To not execute all tests but instead stop at first or xth failure, execute:
|
|
|
```
|
|
|
|
|
|
You can register markers in the `conftest.py` file. Also, pytest will print a warning if it does not understand a certain marker.
|
|
|
|
|
|
To not execute all tests but instead stop at first or x-th failure, execute:
|
|
|
|
|
|
```plaintext
|
|
|
$pytest -x # stop after first failure
|
|
|
$pytest --maxfail=2 # stop after two failures
|
|
|
```
|
|
|
|
|
|
Per default pytest times all tests but only displays the total runtime in the report.
|
|
|
adding the `–duration=N` option will print the times of the `N` slowest tests during the run.
|
|
|
Per default pytest times all tests but only displays the total runtime in the report. adding the `–duration=N` option will print the times of the `N` slowest tests during the run.
|
|
|
|
|
|
To execute all tests in a given (sub)folder (`tests/bar`) or file `tests/bar/test_foo.py` run (will only work from the `tests/new_pytest_system` ):
|
|
|
```
|
|
|
|
|
|
```plaintext
|
|
|
$pytest tests/bar
|
|
|
$pytest tests/bar/test_foo.py
|
|
|
```
|
|
|
|
|
|
# What is added to the build dir:
|
|
|
|
|
|
Under `Testing/` a test session run will create several folders:
|
|
|
|
|
|
- `work`: here we run the tests, i.e execute inpgen and fleur. This folder is cleaned after every test.
|
|
|
- `failed_test_results`: a folder where for all failed tests the content of `work` is preserved of the last test session. This folder is cleaned at the beginning of a test session.
|
|
|
- `parser_testdir`: a folder where all scheduled parser tests go, this is cleaned before each test session. Soon this is cleaned during a session, and failed parser tests are also moved to `failed_test_results`
|
|
|
- (only on CI) `pytest_session.stdout`, `pytest_summmary.out`: On the CI we also write the pytest report and the short summary to files (using tee, check the ci.yml file)
|
|
|
- `failed_test_results`: a folder where for all failed tests the content of `work` is preserved of the last test session. This folder is cleaned at the beginning of a test session. This contains next to all files produced by inpgen/fleur the stdout and stderr and a file `test.log` with some more information about the performed checks and results of those checks
|
|
|
- `parser_testdir`: a folder where all scheduled parser tests go, this is cleaned before each test session. This is cleaned up during a session, and failed parser tests are also moved to `failed_test_results` with the suffix `_parser_test` after the test name.
|
|
|
- (only with `run_tests.sh`) `pytest_session.stdout`: With the prepared script we also write the pytest report to a file using tee
|
|
|
- (only on CI) `pytest_summmary.out`: On the CI the complete summary fo failures is written to a file (check the ci.yml file)
|
|
|
|
|
|
Also cmake writes out a file for pytests with information on the fleur build, to automaticly exclude tests for specific fleur build, i.e specifc libraries like magma, libxc, ...
|
|
|
Also, cmake writes out a file for pytest with information on the fleur build, to automatically exclude tests for specific fleur build, i.e specific libraries like magma, libxc, ...
|
|
|
|
|
|
# How to read the test log:
|
|
|
|
|
|
Pytest writes a log for each test session, which is usually outputed into the terminal. We also pipe (via `| tee <filepath>`) it to the `pytest_session.stdout` file on the CI.
|
|
|
Pytest writes a log for each test session, which is usually output to the terminal. We also pipe (via `| tee <filepath>`) it to the `pytest_session.stdout` file with the prepared script.
|
|
|
|
|
|
A example (shorted) log from a test session run with line numbers may look like this (usually colored):
|
|
|
```
|
|
|
An example (shorted) log from a test session run with line numbers may look like this (usually colored):
|
|
|
|
|
|
```plaintext
|
|
|
0 $ ./run_tests.sh | tee $CI_PROJECT_DIR/build/Testing/pytest_session.stdout
|
|
|
1 ============================= test session starts ==============================
|
|
|
2 platform linux -- Python 3.8.5, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
|
... | ... | @@ -122,22 +142,24 @@ A example (shorted) log from a test session run with line numbers may look like |
|
|
88 FAILED ../test_fleur_mt_outxml_parser[test_CwannXML]
|
|
|
89 ===== 1 failed, 178 passed, 27 skipped, 4 deselected in 1408.04s (0:23:28) =====
|
|
|
```
|
|
|
|
|
|
## Log: Test session header
|
|
|
The log starts with a session header, going in this case from line 1 to 15.
|
|
|
In this header contains default pytest output of versions (line 1), and the output of how many tests pytest has discovered, selected and deselected (line 15).
|
|
|
Further the session header contains information we put there. I.e which fleur executable used (line 4), inpgen executable used (line 5), libraries not linked (line 6), in which folder the tests will run (line 7), where files from failed tests will be copied to (line 8), if parser tests will be run, and for which masci-tools version (line 9), the default mpi command to execute fleur (line 10), at the start of the sessions these folders are cleared and/or created (mentioned on line 11), the rootdir (all other paths below are relative to this) and the pytest configfile (line 12). Line 13 list all markers cmake has written into `pytest_incl.py` within the build dir for compilation related test deselection. Line 14 states if only every x test is run and if the test session has an offset.
|
|
|
|
|
|
The log starts with a session header, going in this case from line 1 to 15. In this header contains default pytest output of versions (line 1), and the output of how many tests pytest has discovered, selected and deselected (line 15). Further the session header contains information we put there. I.e which fleur executable used (line 4), inpgen executable used (line 5), libraries not linked (line 6), in which folder the tests will run (line 7), where files from failed tests will be copied to (line 8), if parser tests will be run, and for which masci-tools version (line 9), the default mpi command to execute fleur (line 10), at the start of the sessions these folders are cleared and/or created (mentioned on line 11), the rootdir (all other paths below are relative to this) and the pytest configfile (line 12). Line 13 list all markers cmake has written into `pytest_incl.py` within the build dir for compilation related test deselection. Line 14 states if only every x test is run and if the test session has an offset.
|
|
|
|
|
|
## Log: Running tests
|
|
|
Then in the lines 16-86 information on the running tests is outputed. Each line contains information on tests from which file are run, followed by a '.' (dot) for each passed test. Skipped tests are marked with an 's', failed tests with and 'F' and tests which had unexpected errors with an 'E'.
|
|
|
The line ends with a procentage, for how far we are into the test session.
|
|
|
|
|
|
Then in the lines 16-86 information on the running tests is outputed. Each line contains information on tests from which file are run, followed by a '.' (dot) for each passed test. Skipped tests are marked with an 's', failed tests with and 'F' and tests which had unexpected errors with an 'E'. The line ends with a procentage, for how far we are into the test session.
|
|
|
|
|
|
## Log: Stacktraces, understanding what failed
|
|
|
After the running tests (not shown in this example) follow long stack traces containing information on were the test failed, and what was recorded in stderr.
|
|
|
This starts after a line containing ```=========== FAILURES ========```.
|
|
|
Further some examples of stack traces explained (also from different test session):
|
|
|
|
|
|
After the running tests (not shown in this example) follow long stack traces containing information on were the test failed, and what was recorded in stderr. This starts after a line containing `=========== FAILURES ========`. Further some examples of stack traces explained (also from different test session):
|
|
|
|
|
|
### Example 1
|
|
|
|
|
|
A full long stackstrace of a parser test failure:
|
|
|
```
|
|
|
|
|
|
```plaintext
|
|
|
______________________________________ test_fleur_mt_outxml_parser[test_CwannXML] ______________________________________
|
|
|
|
|
|
request = <FixtureRequest for <Function test_fleur_mt_outxml_parser[test_CwannXML]>>, fleur_test_name = 'test_CwannXML'
|
... | ... | @@ -212,37 +234,44 @@ WARNING masci_tools.io.parsers.fleur.fleur_outxml_parser:fleur_outxml_parser.py |
|
|
Line 2: Element 'fleurOutput', attribute 'fleurOutputVersion': [facet 'enumeration'] The value '0.35' is not an element of the set {'0.34'}.
|
|
|
...
|
|
|
```
|
|
|
The first lines tells you from which test this is, below is information for which previous run test case this parser test is from (in this case `test_CwannXML`). This information is followed by a print of the complete python code of the test until the line, where the first exception was thrown (indicated by the `>` on the start of the line). This way one sees, what is run, what parsed already.
|
|
|
In the lines following the line starting with the `>` comes the probably most important information often telling us what exactly went wrong. In this case we tested that `assert set(parser_info['parser_warnings']).difference(KNOWN_WARNINGS) == set()`, i.e we throw an `AssertionError` if there are any unsupected parser warnings. The next line shows us what the values on each side of the `==` were. Then at the end of the report any caputered output or captured logging is also printed.
|
|
|
|
|
|
The first lines tells you from which test this is, below is information for which previous run test case this parser test is from (in this case `test_CwannXML`). This information is followed by a print of the complete python code of the test until the line, where the first exception was thrown (indicated by the `>` on the start of the line). This way one sees, what is run, what parsed already. In the lines following the line starting with the `>` comes the probably most important information often telling us what exactly went wrong. In this case we tested that `assert set(parser_info['parser_warnings']).difference(KNOWN_WARNINGS) == set()`, i.e we throw an `AssertionError` if there are any unsupected parser warnings. The next line shows us what the values on each side of the `==` were. Then at the end of the report any caputered output or captured logging is also printed.
|
|
|
|
|
|
### Example 2
|
|
|
|
|
|
Most important lines of a stacktrace of a tests which failed due to a value not beeing as tested for:
|
|
|
```
|
|
|
|
|
|
```plaintext
|
|
|
tenergy = grep_number(res_files['out'], "total energy=", "=")
|
|
|
> assert abs(tenergy - -1270.4886) <= 0.0001
|
|
|
E assert 7.6435286517000804 <= 0.0001
|
|
|
E + where 7.6435286517000804 = abs((-1278.1321286517 - -1270.4886))
|
|
|
/builds/fleur/tests/new_pytest_system/tests/feature_reg/test_Noncollinear_downward_comp.py:28: AssertionError
|
|
|
```
|
|
|
|
|
|
This means we tested for that the total energy is -1270.4886 but instead it turned out to be -1278.1321286517.
|
|
|
|
|
|
### Example 3
|
|
|
|
|
|
Often we grep in files and expect a certain expression to be there, if it is not this will look in a stack trace like that:
|
|
|
```
|
|
|
|
|
|
```plaintext
|
|
|
> assert grep_exists(res_files['out'], "it= 1 is completed")
|
|
|
E AssertionError: assert False
|
|
|
E + where False = <function grep_exists.<locals>._grep_exists at 0x7f3b7aba21f0>('/home/build/Testing/work/out', 'it= 1 is completed')
|
|
|
|
|
|
/tests/new_pytest_system/tests/feature_reg/test_CuBulk.py:26: AssertionError
|
|
|
```
|
|
|
here "it= 1 is completed" is not in the "out" file.
|
|
|
|
|
|
here "it= 1 is completed" is not in the "out" file.
|
|
|
|
|
|
### Example 4
|
|
|
|
|
|
A failure of a fleur execution.
|
|
|
|
|
|
## Log: Test session Summary
|
|
|
At the end of the test session, a short summary of the test session is outputed (line 87-89).
|
|
|
It contains a single line (line 88 in this case) for each failed tests with the error message (truncated to terminal width).
|
|
|
And a line summing up how many tests failed, passed, where skipped, deselected, errored and how long the full session was.
|
|
|
|
|
|
At the end of the test session, a short summary of the test session is outputed (line 87-89). It contains a single line (line 88 in this case) for each failed tests with the error message (truncated to terminal width). And a line summing up how many tests failed, passed, where skipped, deselected, errored and how long the full session was.
|
|
|
|
|
|
# Tests (source) folder layout:
|
|
|
|
... | ... | @@ -251,8 +280,7 @@ It makes sense to organize and group tests already in a directory layout, which |
|
|
- `tests`: here and under the sub-folders go all further test folders, tests are organized through test type and the functionality they test
|
|
|
- `inputfiles`: To separate all tests input files and data files from the test code, we put them in this folder. The folder names do not matter, through it would be nice if the name would tell to which test(s) the folder belongs to. In the future we might have test which have (autogenerated) files which they test against, and we want to keep the input files separated from them.
|
|
|
- `helpers`: Folder where one can place python code which will be in the python path for a pytest run and can therefore be imported for test code.
|
|
|
- `conftest.py`: This is a (are) central pytest file(s), here all fixtures i.e helpers go (what former has partly been taken care of by the 'libtest.py' file or the 'scripts/*') which can be used within the tests, or automatically do something before or after a test run. For more on fixtures read the pytest basics section below or take a look at (https://docs.pytest.org/en/stable/fixture.html#fixture)
|
|
|
|
|
|
- `conftest.py`: This is a (are) central pytest file(s), here all fixtures i.e helpers go (what former has partly been taken care of by the 'libtest.py' file or the 'scripts/\*') which can be used within the tests, or automatically do something before or after a test run. For more on fixtures read the pytest basics section below or take a look at (<https://docs.pytest.org/en/stable/fixture.html#fixture>)
|
|
|
|
|
|
# Implementing new tests:
|
|
|
|
... | ... | @@ -260,59 +288,60 @@ It makes sense to organize and group tests already in a directory layout, which |
|
|
|
|
|
### Decorators:
|
|
|
|
|
|
the stuff with the `@` above the functions are so called decorators, which are executed before the actual function they decorate is executed. So they wrap around the functions. Implementing it this way makes the code easier to read and understand. For example see:
|
|
|
https://pythonbasics.org/decorators/
|
|
|
the stuff with the `@` above the functions are so called decorators, which are executed before the actual function they decorate is executed. So they wrap around the functions. Implementing it this way makes the code easier to read and understand. For example see: <https://pythonbasics.org/decorators/>
|
|
|
|
|
|
## Pytest basics
|
|
|
|
|
|
https://docs.pytest.org/en/stable/
|
|
|
<https://docs.pytest.org/en/stable/>
|
|
|
|
|
|
Fixtures are helper functions, which can be put in the `.py` file of the tests to be available there or in the conftest.py file(s) to be available globally.
|
|
|
We use them for preparation or teardown code, also basicly to make anything available we need throughout the different tests.
|
|
|
Fixtures with scope `session` are executed one a test session level (for example the binaries, because we use the same fleur exe for all tests)
|
|
|
Fixtures with scope `function` can be used within a test, and could with `autouse=True` automatically be executed for every test.
|
|
|
(for more on this look at the `conftest.py`file)
|
|
|
Fixtures are helper functions, which can be put in the `.py` file of the tests to be available there or in the conftest.py file(s) to be available globally. We use them for preparation or teardown code, also basicly to make anything available we need throughout the different tests. Fixtures with scope `session` are executed one a test session level (for example the binaries, because we use the same fleur exe for all tests) Fixtures with scope `function` can be used within a test, and could with `autouse=True` automatically be executed for every test. (for more on this look at the `conftest.py`file)
|
|
|
|
|
|
usefull pytest decorators
|
|
|
```
|
|
|
|
|
|
```plaintext
|
|
|
@pytest.mark.skip() # skips the test (shown in report)
|
|
|
@pytest.mark.skipif() # skips the test under a certain condition
|
|
|
@pytest.mark.parameterize() # creates a test case for each 'parameter'
|
|
|
@pytest.mark.<markername> # mark test if a given label to allow subset execution
|
|
|
```
|
|
|
|
|
|
### Pytest plugins we use:
|
|
|
|
|
|
there are a bunch of them, we use two so far
|
|
|
```
|
|
|
|
|
|
```plaintext
|
|
|
pytest-dependency
|
|
|
```
|
|
|
If you want to add a plugin, please add it its source code, that one does not have to install it to run the tests everywhere.
|
|
|
i.e add it under `tests/new_pytest_system/pytest_plugins`
|
|
|
|
|
|
If you want to add a plugin, please add it its source code, that one does not have to install it to run the tests everywhere. i.e add it under `tests/new_pytest_system/pytest_plugins`
|
|
|
|
|
|
# How to create a new test
|
|
|
|
|
|
As an example and good starting point look at the `tests/feature_reg/test_CuBulk.py` file which contains several tests on a Cu bulk system.
|
|
|
|
|
|
1. Check if the thing you want to test, is not covered by any other test. (Since fleur regression tests are slow). Maybe you can extend a further test, or add a stage, or depend on an other test on which to continue.
|
|
|
1. Create a new folder under the `inputfiles` folder where you put all the input files needed for your test
|
|
|
2. Optional add a new `test_<some_name>.py` or `<some_name>_test.py` file somewhere under the `tests` directory where your test functions/code goes, or add you test function to an already existing file. Usually, one executes all tests within a file, so if you want to execute just your test alone, it needs its own file, or a special marker.
|
|
|
3. If the test function executes without throwing an `exception` pytest will mark it as `PASSED` during execution, Otherwise, the test will fail on the first error (python exception) thrown. Therefore, if you want to test for a simple comparison of something you can use the `assert` statement to do so, which will throw an `AssertionError` if what comes after `assert` is `False`.
|
|
|
examples:
|
|
|
```
|
|
|
1. Check if the thing you want to test, is not covered by any other test. (Since fleur regression tests are slow). Maybe you can extend a further test, or add a stage, or depend on an other test on which to continue.
|
|
|
2. Create a new folder under the `inputfiles` folder where you put all the input files needed for your test
|
|
|
3. Optional add a new `test_<some_name>.py` or `<some_name>_test.py` file somewhere under the `tests` directory where your test functions/code goes, or add you test function to an already existing file. Usually, one executes all tests within a file, so if you want to execute just your test alone, it needs its own file, or a special marker.
|
|
|
4. If the test function executes without throwing an `exception` pytest will mark it as `PASSED` during execution, Otherwise, the test will fail on the first error (python exception) thrown. Therefore, if you want to test for a simple comparison of something you can use the `assert` statement to do so, which will throw an `AssertionError` if what comes after `assert` is `False`. examples:
|
|
|
|
|
|
```plaintext
|
|
|
assert ('out' in res_file_names), 'The out file is missing'
|
|
|
assert fermi == 0.4233
|
|
|
assert abs(fermi - 0.4233) <= 0.001
|
|
|
assert grep_exists(res_files['out'], "it= 1 is completed")
|
|
|
```
|
|
|
|
|
|
You can specify some error message behind the statement like in the first example above. Through this is not needed and sometimes less transparent since pytest per default prints a stack trace for failed tests, where you see which lines where executes and for example what the value of `fermi` from above turned out to be.
|
|
|
|
|
|
Any other exception will due of course, for example this is also fine
|
|
|
```
|
|
|
|
|
|
```plaintext
|
|
|
validate('inp.xml') # this will throw some libxml errors if it does not validate
|
|
|
```
|
|
|
|
|
|
example test function explained:
|
|
|
```
|
|
|
|
|
|
```plaintext
|
|
|
@pytest.mark.bulk # markers to group tests
|
|
|
@pytest.mark.xml # markers to group tests
|
|
|
@pytest.mark.fast # markers to group tests
|
... | ... | @@ -349,5 +378,4 @@ def test_CuBulkXML(execute_fleur, grep_exists, grep_number, stage_for_parser_tes |
|
|
assert abs(dist - 45.8259) <= 0.001
|
|
|
```
|
|
|
|
|
|
|
|
|
There are certain things, which are still missing in the pytest system. Feel free to implement them or to open feature request issues that we can improve. |
|
|
\ No newline at end of file |