The PEtab test suite is a collection of test models for the PEtab parameter estimation data format. It is intended to be used to verify and quantify PEtab-support by developers of tools for model simulation and parameter estimation.
The PEtab test suite can be downloaded from GitHub via
git clone https://github.com/petab-dev/petab_test_suite
The test suite comes with all necessary files pregenerated.
The PEtab test suite comes with a Python package named petabtests in the homonymous subdirectory. It contains Python functions for generating the tests and evaluating results. For using the test suite, it can be installed via
pip install git+https://github.com/PEtab-dev/petab_test_suite
or, from a local clone, via
pip install petab_test_suite/
For working on the test suite itself, see Development setup.
The petabtests/cases subdirectory contains different test suites
for different PEtab versions and model formats.
Each test suite is a collection of enumerated tests.
Each test consists of a single PEtab problem defined in XXXX/_XXXX.yaml
file and the expected result (chi2 value, log-likelihood, simulation
table reference, and tolerances) in XXXX/_XXXX_solution.yaml.
XXXX/XXX.md contains a short description of the respective test
problem that is not relevant for the execution of the test itself.
To evaluate how a tool performs on a given test problem, three metrics are employed: Simulations, chi2 value and log-likelihood. A tool can be said to cover a test problem if any of those values matches the ground truth values up to some tolerance.
The Python package provides convenience functions for evaluation in petabtests/evaluate.py.
-> Overview of passed test cases for different tools supporting PEtab
Contributions to the PEtab test suite are welcome.
The recommended way to set up a development environment is uv:
git clone https://github.com/petab-dev/petab_test_suite
cd petab_test_suite
uv sync
This installs petabtests in editable mode into .venv, together with the
current development version of
libpetab.
Commands can then be run via uv run, e.g., uv run petabtests_create.
To add a new test case, create a new subdirectory in the respective test suite
directory under petabtests/cases/v{petab_version}/{model_format}/.
The subdirectory name should be a four-digit number, starting with the next
available number.
Most files in the test suite are generated automatically. Those start with an
underscore, except for the README.md files. The only files that need to be
created manually are the XXXX/XXXX.py files that contain the script to
generate the test problem and solution files. Their content should be
self-explanatory. For adding a new test case, copy XXXX/XXXX.py from an
existing test case and adjust it to the new test case.
All remaining files are generated by petabtests_create
(see Development setup).
If existing test cases were modified, run petabtests_clear first to remove
all generated files.
To facilitate debugging simulation issues:
- Try to keep test cases as simple as possible. If multiple features are tested in a single test case, try to also include separate experiments/observables/measurements that test only one feature at a time.
- For non-trivial test cases, explain the expected behavior in the test description.
- Always include measurements for all experiments/observables at time point 0 if possible, as this helps to verify the initial conditions that are a common source of errors.