From 52987e32e7584fcc188aa0eff39971f271bf008d Mon Sep 17 00:00:00 2001 From: Brennan Kinney <5098581+polarathene@users.noreply.github.com> Date: Tue, 3 Jan 2023 21:05:06 +1300 Subject: [PATCH] docs: Revise contributing advice for tests --- docs/content/contributing/general.md | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-) diff --git a/docs/content/contributing/general.md b/docs/content/contributing/general.md index d6bdcb94..c3d681a6 100644 --- a/docs/content/contributing/general.md +++ b/docs/content/contributing/general.md @@ -15,7 +15,7 @@ When refactoring, writing or altering scripts or other files, adhere to these ru To run the test suite, you will need to -1. [Install Docker] +1. [Install Docker][get-docker] 2. Install `jq` (under Ubuntu, use `sudo apt-get -y install jq`) 3. Execute `git submodule update --init --recursive` if you haven't already initialized the git submodules 4. Execute `make clean all` @@ -32,15 +32,20 @@ To run the test suite, you will need to !!! warning "Test Output when Running in Parallel" - When running test in parallel, BATS will run more than one test at any given time. This can result in output not being exactly what you'd expect, i.e. there could be _more_ or _less_ than you'd think. Those writing tests need to take care of this. Always test with `make clean generate-accounts tests/parallel/setX`. + [When running tests in parallel][docs-bats-parallel] (_with `make clean generate-accounts tests/parallel/setX`_), BATS will delay outputting the results until completing all test cases within a file. -??? tip "Running a Specific Test" + This also delays test failures as a result. When troubleshooting parallel set tests, you may prefer to run them serially as advised below. - To run a specific test, use `make build generate-accounts test/`, where `` is the file name of the test (_for more precision use a relative path: `test/test/`_) **excluding** the `.bats` suffix. + When writing tests, ensure that parallel set tests still pass when run in parallel. You need to account for other tests running in parallel that may interfere with your own tests logic. - Example: To run only the tests in `template.bats`, use `make test/template` (or `make test/parallel/set2/template`). +??? tip "Run a Specific Test" -[Install Docker]: https://docs.docker.com/get-docker/ + Run `make build generate-accounts test/`, where `` is the file name of the test **excluding** the `.bats` suffix (_use a relative path if needing to be more specific: `test//`_). + + Multiple test files can be run sequentially with a `,` delimiter between file names: + `make test/tls_letsencrypt,tls_manual` + + **Example:** To run only the tests in `template.bats`, use `make test/template` (_or with relative path: `make test/parallel/set2/template`_). ## Documentation @@ -51,3 +56,6 @@ docker run --rm -it -p 8000:8000 -v "${PWD}:/docs" squidfunk/mkdocs-material ``` This serves the documentation on your local machine on port `8000`. Each change will be hot-reloaded onto the page you view, just edit, save and look at the result. + +[get-docker]: https://docs.docker.com/get-docker/ +[docs-bats-parallel]: https://bats-core.readthedocs.io/en/v1.8.2/usage.html#parallel-execution