New Test infrastructure
Here we collect features we want to have in a new test environment. This list is not ranked.
- No unit-tests only regression tests needed
- It should be easy to write new tests, tests templates for certain things
- developer friendly
- semi-automated way of generating tests
- Should not introduce an external dependency
- code coverage reports would be nice
- it should run on linux, but also mac os. (windows prob not important)
- We need flexibility on the test procedure. Sometimes we have to call Fleur multiple times, sometimes not even once. Depending on the test have different input files because some tests have everything in inp.xml while others use a sym.xml file. Depending on the test we are also interested in different output files.
- The output of the reference run should be placed somewhere in the directory of the test. It may be useful for debugging.
- The test infrastructure should impede the creation of invalid (ill-formed) tests.
- Erros in the test-script should cause the tests to fail
- we want to test different environments, with OMP, MPI, HDF, lib config
- some level of verbosity on what goes wrong. Not just this test failed.
- We should be able to identify whether a test fails because there is no SCALAPACK linked to Fleur. This should issue a special output + the test running in a serialized way
- a trace from fleur in case of an error
- should create a seperate stdout & stderr file