Merge pull request #2857 from docker-mailserver/tests/parallelize

## Pull Request

### ci: Run tests in parallel (part 1)

- Tests can now be arbitrarily grouped into sub-directories.
- Some tests are now run in parallel.
- CI will now spawn 4 jobs to run the whole test suite in parallel.



## Individual Commits

### tests(chore): Rename test files to serial and parallel types
- `test_helper.bats` needs more work than this PR provides to be compatible with parallel tests, so must remain as a serial test for now.
- `spam_bounced.bats` had failures as a serial test, but works well converted to a parallel test in a future commit.



### tests(CI): Adjust Makefile & GHA workflow to support new test layout
These updates support running tests that have been relocated into `serial` and `parallel/set*` directories.

- `make tests` now calls the two make targets beneath it. The only difference is that `serial` continues the "1 test at a time" approach used prior to this PR, while the `parallel` target increases the `--jobs` arg to run multiple tests concurrently (_configured by `PARALLEL_JOBS`_).
- The `test/%` target leverages Bash syntax magic to ease running single tests without providing the exact path.
- This syntax also supports providing multiple test names (eg: `make test/clamav,template`) to run.
- `**` (globstar) allows for future improvements that can group multiple test files into sub-directories by their scope (eg: anti-spam, ssl, etc).

---

chore: Add `shopt -s globstar` to other targets
I realized that other targets should have this as well in case it is not set.
It is better to be more explicit here than to have weird errors due to `**` not expanding properly.

---

fix(Makefile):  Add back `.PHONY` targets

I encountered `make` telling me the target was already up-to-date, which of course is nonsense.
I therefore added back the `.PHONY` targets to ensure tests are always run. 

---

docs: Added instructions for running  a single test
See https://github.com/docker-mailserver/docker-mailserver/pull/2857/files#r1008582760



### tests(chore): Use `REPOSITORY_ROOT` export var from Makefile

Allows for using `load` with an absolute path instead of a relative one, which makes it possible to group tests into different directories.

Parallel tests differ slightly, loading the newer `helper/common.bash` and `helper/setup.bash` files instead of the older `test_helper/common.bash` which serial tests continue to use.



### tests(refactor): `common.bash` helper split into two files
The current `test/test_helper/common.bash` was getting large. Setup logic has been extracted out into a new file.

`common.bash` resides in a directory  named `test_helper/`, the `test_` prefix is redundant. 

As an interim solution this provides a new approach for the updated tests, while the "old" tests can use the "old" `common.bash`. Eventually all tests should migrate to the new approach in `helper/` instead of the older `test_helper/`.

The new helper files are located under `test/helper/` (_which drops the `test_` prefix_). The new and updated helpers apply the new naming convention for ENV variables (_such as `CONTAINER_NAME` or `IMAGE_NAME`_).

---

Some refactoring occurred, including new methods like `_run_in_container()` and `_default_teardown()`.

---

I encountered a situation before in which the updated tests would fail because there were collisions of ENV names in the tests (_for example with `CONTAINER_NAME`_).



### tests(refactor): Conversion to parallel tests and use revised helpers
- Introduced `CONTAINER_NAME` and `TEST_NAME_PREFIX` as new vars for better managing test consistency (DRY).
- `CONTAINER_NAME` replaces any repeated container name with the variable. The value will differ slightly as the prior prefix (`mail_`) has been changed to `dms-test-`.
- `TEST_NAME_PREFIX` provides a prefix value for each `@test` description string.

---

chore: Add a reference template for tests

.github/workflows/generic_test.yml

@@ -14,6 +14,9 @@ jobs:
   run-tests:
     name: 'Test'
     runs-on: ubuntu-20.04
+    strategy:
+      matrix:
+        part: [serial, parallel/set1, parallel/set2, parallel/set3]
     steps:
       - name: Checkout
         uses: actions/checkout@v3
@@ -45,6 +48,6 @@ jobs:
           cache-from: type=local,src=/tmp/.buildx-cache
 
       - name: 'Run tests'
-        run: make generate-accounts tests
+        run: make generate-accounts tests/${{ matrix.part }}
         env:
           CI: true

Makefile

@@ -1,8 +1,12 @@
-SHELL             := /bin/bash
+SHELL       := /bin/bash
-.SHELLFLAGS       += -e -u -o pipefail
+.SHELLFLAGS += -e -u -o pipefail
 
-export IMAGE_NAME := mailserver-testing:ci
+PARALLEL_JOBS          ?= 2
-export NAME       ?= $(IMAGE_NAME)
+export REPOSITORY_ROOT := $(CURDIR)
+export IMAGE_NAME      ?= mailserver-testing:ci
+export NAME            ?= $(IMAGE_NAME)
+
+.PHONY: ALWAYS_RUN
 
 # -----------------------------------------------
 # --- Generic Targets ---------------------------
@@ -17,7 +21,7 @@ build:
 		--build-arg VCS_REVISION=$(shell cat VERSION) \
 		.
 
-generate-accounts:
+generate-accounts: ALWAYS_RUN
 	@ cp test/config/templates/postfix-accounts.cf test/config/postfix-accounts.cf
 	@ cp test/config/templates/dovecot-masters.cf test/config/dovecot-masters.cf
 
@@ -33,14 +37,29 @@ clean:
 	-@ while read -r LINE; do [[ $${LINE} =~ test/.+ ]] && sudo rm -rf $${LINE}; done < .gitignore
 
 # -----------------------------------------------
-# --- Tests  & Lints ----------------------------
+# --- Tests  ------------------------------------
 # -----------------------------------------------
 
-tests:
+tests: ALWAYS_RUN
-	@ ./test/bats/bin/bats --timing test/*.bats
+# See https://github.com/docker-mailserver/docker-mailserver/pull/2857#issuecomment-1312724303
+# on why `generate-accounts` is run before each set (TODO/FIXME)
+	@ $(MAKE) generate-accounts tests/serial
+	@ $(MAKE) generate-accounts tests/parallel/set1
+	@ $(MAKE) generate-accounts tests/parallel/set2
+	@ $(MAKE) generate-accounts tests/parallel/set3
 
-test/%:
+tests/serial: ALWAYS_RUN
-	@ ./test/bats/bin/bats --timing $@.bats
+	@ shopt -s globstar ; ./test/bats/bin/bats --timing --jobs 1 test/$@/**.bats
+
+tests/parallel/set%: ALWAYS_RUN
+	@ shopt -s globstar ; ./test/bats/bin/bats --timing --jobs $(PARALLEL_JOBS) test/$@/**.bats
+
+test/%: ALWAYS_RUN
+	@ shopt -s globstar nullglob ; ./test/bats/bin/bats --timing test/tests/**/{$*,}.bats
+
+# -----------------------------------------------
+# --- Lints -------------------------------------
+# -----------------------------------------------
 
 lint: eclint hadolint shellcheck
 

docs/content/contributing/general.md

@@ -24,6 +24,12 @@ To run the test suite, you will need to
 
     We do not support running linting, tests, etc on macOS at this time. Please use a linux VM.
 
+??? tip "Running a Specific Test"
+
+    To run a specific test, use `make build generate-accounts test/<TEST NAME>`, where `<TEST NAME>` is the file name of the test (_for more precision use a relative path: `test/test/<PATH>`_) excluding the `.bats` suffix.
+
+    To run only the tests in `template.bats`, use `make test/template` (or `make test/parallel/set2/template`).
+
 [Install Docker]: https://docs.docker.com/get-docker/
 
 ## Documentation

test/clamav.bats

@@ -1,79 +0,0 @@
-load 'test_helper/common'
-
-TEST_NAME_PREFIX='ClamAV:'
-CONTAINER_NAME='dms-test-clamav'
-RUN_COMMAND=('run' 'docker' 'exec' "${CONTAINER_NAME}")
-
-function setup_file() {
-  local PRIVATE_CONFIG
-  PRIVATE_CONFIG=$(duplicate_config_for_container . "${CONTAINER_NAME}")
-
-  docker run --rm --detach --tty \
-    --name "${CONTAINER_NAME}" \
-    --hostname mail.my-domain.com \
-    --volume "${PRIVATE_CONFIG}:/tmp/docker-mailserver" \
-    --volume "${PWD}/test/test-files:/tmp/docker-mailserver-test:ro" \
-    --env ENABLE_AMAVIS=1 \
-    --env AMAVIS_LOGLEVEL=2 \
-    --env ENABLE_CLAMAV=1 \
-    --env ENABLE_UPDATE_CHECK=0 \
-    --env ENABLE_SPAMASSASSIN=0 \
-    --env ENABLE_FAIL2BAN=0 \
-    --env PERMIT_DOCKER=host \
-    --env CLAMAV_MESSAGE_SIZE_LIMIT=30M \
-    --env LOG_LEVEL=debug \
-    "${IMAGE_NAME}"
-
-  wait_for_finished_setup_in_container "${CONTAINER_NAME}"
-
-  # wait for ClamAV to be fully setup or we will get errors on the log
-  repeat_in_container_until_success_or_timeout 60 "${CONTAINER_NAME}" test -e /var/run/clamav/clamd.ctl
-
-  wait_for_service "${CONTAINER_NAME}" postfix
-  wait_for_smtp_port_in_container "${CONTAINER_NAME}"
-
-  "${RUN_COMMAND[@]}" bash -c "nc 0.0.0.0 25 < /tmp/docker-mailserver-test/email-templates/amavis-virus.txt"
-  assert_success
-
-  wait_for_empty_mail_queue_in_container "${CONTAINER_NAME}"
-}
-
-function teardown_file() {
-  docker rm -f "${CONTAINER_NAME}"
-}
-
-@test "${TEST_NAME_PREFIX} process clamd is running" {
-  "${RUN_COMMAND[@]}" bash -c "ps aux --forest | grep -v grep | grep '/usr/sbin/clamd'"
-  assert_success
-}
-
-@test "${TEST_NAME_PREFIX} log files exist at /var/log/mail directory" {
-  "${RUN_COMMAND[@]}" bash -c "ls -1 /var/log/mail/ | grep -E 'clamav|freshclam|mail.log'| wc -l"
-  assert_success
-  assert_output 3
-}
-
-@test "${TEST_NAME_PREFIX} should be identified by Amavis" {
-  "${RUN_COMMAND[@]}" grep -i 'Found secondary av scanner ClamAV-clamscan' /var/log/mail/mail.log
-  assert_success
-}
-
-@test "${TEST_NAME_PREFIX} freshclam cron is enabled" {
-  "${RUN_COMMAND[@]}" bash -c "grep '/usr/bin/freshclam' -r /etc/cron.d"
-  assert_success
-}
-
-@test "${TEST_NAME_PREFIX} env CLAMAV_MESSAGE_SIZE_LIMIT is set correctly" {
-  "${RUN_COMMAND[@]}" grep -q '^MaxFileSize 30M$' /etc/clamav/clamd.conf
-  assert_success
-}
-
-@test "${TEST_NAME_PREFIX} rejects virus" {
-  "${RUN_COMMAND[@]}" bash -c "grep 'Blocked INFECTED' /var/log/mail/mail.log | grep '<virus@external.tld> -> <user1@localhost.localdomain>'"
-  assert_success
-}
-
-@test "${TEST_NAME_PREFIX} process clamd restarts when killed" {
-  "${RUN_COMMAND[@]}" bash -c "pkill clamd && sleep 10 && ps aux --forest | grep -v grep | grep '/usr/sbin/clamd'"
-  assert_success
-}

test/default_relay_host.bats

@@ -1,28 +0,0 @@
-load 'test_helper/common'
-
-function setup() {
-  local PRIVATE_CONFIG
-  PRIVATE_CONFIG=$(duplicate_config_for_container relay-hosts)
-
-  docker run -d --name mail_with_default_relay \
-    -v "${PRIVATE_CONFIG}":/tmp/docker-mailserver \
-    -v "$(pwd)/test/test-files":/tmp/docker-mailserver-test:ro \
-    -e DEFAULT_RELAY_HOST=default.relay.host.invalid:25 \
-    -e PERMIT_DOCKER=host \
-    -h mail.my-domain.com -t "${NAME}"
-
-    wait_for_finished_setup_in_container mail_with_default_relay
-}
-
-function teardown() {
-  docker rm -f mail_with_default_relay
-}
-
-#
-# default relay host
-#
-
-@test "checking default relay host: default relay host is added to main.cf" {
-  run docker exec mail_with_default_relay /bin/sh -c 'grep -e "^relayhost =" /etc/postfix/main.cf'
-  assert_output 'relayhost = default.relay.host.invalid:25'
-}

test/dovecot_inet_protocol.bats

@@ -1,54 +0,0 @@
-load 'test_helper/common'
-
-function setup_file() {
-  local PRIVATE_CONFIG
-  export ALL IPV4 IPV6
-
-  PRIVATE_CONFIG=$(duplicate_config_for_container . "${IPV4}")
-  ALL="mail_dovecot_all_protocols"
-  IPV4="mail_dovecot_ipv4"
-  IPV6="mail_dovecot_ipv6"
-
-  docker run --rm -d --name "${ALL}" \
-    -v "${PRIVATE_CONFIG}":/tmp/docker-mailserver \
-    -e DOVECOT_INET_PROTOCOLS= \
-    -h mail.my-domain.com \
-    -t "${NAME}"
-
-  docker run --rm -d --name "${IPV4}" \
-    -v "${PRIVATE_CONFIG}":/tmp/docker-mailserver \
-    -e DOVECOT_INET_PROTOCOLS=ipv4 \
-    -h mail.my-domain.com \
-    -t "${NAME}"
-
-  docker run --rm -d --name "${IPV6}" \
-    -v "${PRIVATE_CONFIG}":/tmp/docker-mailserver \
-    -e DOVECOT_INET_PROTOCOLS=ipv6 \
-    -h mail.my-domain.com \
-    -t "${NAME}"
-}
-
-@test 'checking dovecot IP configuration' {
-  wait_for_finished_setup_in_container "${ALL}"
-  run docker exec "${ALL}" grep '^#listen = \*, ::' /etc/dovecot/dovecot.conf
-  assert_success
-  assert_output '#listen = *, ::'
-}
-
-@test 'checking dovecot IPv4 configuration' {
-  wait_for_finished_setup_in_container "${IPV4}"
-  run docker exec "${IPV4}" grep '^listen = \*$' /etc/dovecot/dovecot.conf
-  assert_success
-  assert_output 'listen = *'
-}
-
-@test 'checking dovecot IPv6 configuration' {
-  wait_for_finished_setup_in_container "${IPV6}"
-  run docker exec "${IPV6}" grep '^listen = \[::\]$' /etc/dovecot/dovecot.conf
-  assert_success
-  assert_output 'listen = [::]'
-}
-
-function teardown_file {
-  docker rm -f "${ALL}" "${IPV4}" "${IPV6}"
-}

test/helper-functions.bats

@@ -1,30 +0,0 @@
-load 'test_helper/common'
-
-function setup_file() {
-  local PRIVATE_CONFIG
-  PRIVATE_CONFIG=$(duplicate_config_for_container .)
-
-  docker run -d --name mail_helper_functions \
-    --cap-add=NET_ADMIN \
-    -v "${PRIVATE_CONFIG}":/tmp/docker-mailserver \
-    -v "$(pwd)/test/test-files":/tmp/docker-mailserver-test:ro \
-    -e ENABLE_FETCHMAIL=1 \
-    -h mail.my-domain.com -t "${NAME}"
-
-  wait_for_finished_setup_in_container mail_helper_functions
-}
-
-function teardown_file() {
-    docker rm -f mail_helper_functions
-}
-
-@test "check helper functions (network.sh): _sanitize_ipv4_to_subnet_cidr" {
-    run docker exec mail_helper_functions bash -c "source /usr/local/bin/helpers/index.sh; _sanitize_ipv4_to_subnet_cidr 255.255.255.255/0"
-    assert_output "0.0.0.0/0"
-
-    run docker exec mail_helper_functions bash -c "source /usr/local/bin/helpers/index.sh; _sanitize_ipv4_to_subnet_cidr 192.168.255.14/20"
-    assert_output "192.168.240.0/20"
-
-    run docker exec mail_helper_functions bash -c "source /usr/local/bin/helpers/index.sh; _sanitize_ipv4_to_subnet_cidr 192.168.255.14/32"
-    assert_output "192.168.255.14/32"
-}

test/helper/common.bash

@@ -0,0 +1,229 @@
+#!/bin/bash
+
+function __load_bats_helper() {
+  load "${REPOSITORY_ROOT}/test/test_helper/bats-support/load"
+  load "${REPOSITORY_ROOT}/test/test_helper/bats-assert/load"
+}
+
+__load_bats_helper
+
+# -------------------------------------------------------------------
+
+function _run_in_container() {
+  run docker exec "${CONTAINER_NAME}" "${@}"
+}
+
+function _default_teardown() {
+  docker rm -f "${CONTAINER_NAME}"
+}
+
+# -------------------------------------------------------------------
+
+# @param ${1} timeout
+# @param --fatal-test <command eval string> additional test whose failure aborts immediately
+# @param ... test to run
+function repeat_until_success_or_timeout {
+  local FATAL_FAILURE_TEST_COMMAND
+
+  if [[ "${1}" == "--fatal-test" ]]; then
+    FATAL_FAILURE_TEST_COMMAND="${2}"
+    shift 2
+  fi
+
+  if ! [[ "${1}" =~ ^[0-9]+$ ]]; then
+    echo "First parameter for timeout must be an integer, received \"${1}\""
+    return 1
+  fi
+
+  local TIMEOUT=${1}
+  local STARTTIME=${SECONDS}
+  shift 1
+
+  until "${@}"
+  do
+    if [[ -n ${FATAL_FAILURE_TEST_COMMAND} ]] && ! eval "${FATAL_FAILURE_TEST_COMMAND}"; then
+      echo "\`${FATAL_FAILURE_TEST_COMMAND}\` failed, early aborting repeat_until_success of \`${*}\`" >&2
+      return 1
+    fi
+
+    sleep 1
+
+    if [[ $(( SECONDS - STARTTIME )) -gt ${TIMEOUT} ]]; then
+      echo "Timed out on command: ${*}" >&2
+      return 1
+    fi
+  done
+}
+
+# like repeat_until_success_or_timeout but with wrapping the command to run into `run` for later bats consumption
+# @param ${1} timeout
+# @param ... test command to run
+function run_until_success_or_timeout {
+  if ! [[ ${1} =~ ^[0-9]+$ ]]; then
+    echo "First parameter for timeout must be an integer, received \"${1}\""
+    return 1
+  fi
+
+  local TIMEOUT=${1}
+  local STARTTIME=${SECONDS}
+  shift 1
+
+  until run "${@}" && [[ $status -eq 0 ]]
+  do
+    sleep 1
+
+    if (( SECONDS - STARTTIME > TIMEOUT )); then
+      echo "Timed out on command: ${*}" >&2
+      return 1
+    fi
+  done
+}
+
+# @param ${1} timeout
+# @param ${2} container name
+# @param ... test command for container
+function repeat_in_container_until_success_or_timeout() {
+  local TIMEOUT="${1}"
+  local CONTAINER_NAME="${2}"
+  shift 2
+
+  repeat_until_success_or_timeout --fatal-test "container_is_running ${CONTAINER_NAME}" "${TIMEOUT}" docker exec "${CONTAINER_NAME}" "${@}"
+}
+
+function container_is_running() {
+  [[ "$(docker inspect -f '{{.State.Running}}' "${1}")" == "true" ]]
+}
+
+# @param ${1} port
+# @param ${2} container name
+function wait_for_tcp_port_in_container() {
+  repeat_until_success_or_timeout --fatal-test "container_is_running ${2}" "${TEST_TIMEOUT_IN_SECONDS}" docker exec "${2}" /bin/sh -c "nc -z 0.0.0.0 ${1}"
+}
+
+# @param ${1} name of the postfix container
+function wait_for_smtp_port_in_container() {
+  wait_for_tcp_port_in_container 25 "${1}"
+}
+
+# @param ${1} name of the postfix container
+function wait_for_smtp_port_in_container_to_respond() {
+  local COUNT=0
+  until [[ $(docker exec "${1}" timeout 10 /bin/sh -c "echo QUIT | nc localhost 25") == *"221 2.0.0 Bye"* ]]; do
+    if [[ $COUNT -eq 20 ]]
+    then
+      echo "Unable to receive a valid response from 'nc localhost 25' within 20 seconds"
+      return 1
+    fi
+
+    sleep 1
+    ((COUNT+=1))
+  done
+}
+
+# @param ${1} name of the postfix container
+function wait_for_amavis_port_in_container() {
+  wait_for_tcp_port_in_container 10024 "${1}"
+}
+
+# get the private config path for the given container or test file, if no container name was given
+function private_config_path() {
+  echo "${PWD}/test/duplicate_configs/${1:-$(basename "${BATS_TEST_FILENAME}")}"
+}
+
+function container_has_service_running() {
+  local CONTAINER_NAME="${1}"
+  local SERVICE_NAME="${2}"
+
+  docker exec "${CONTAINER_NAME}" /usr/bin/supervisorctl status "${SERVICE_NAME}" | grep RUNNING >/dev/null
+}
+
+function wait_for_service() {
+  local CONTAINER_NAME="${1}"
+  local SERVICE_NAME="${2}"
+
+  repeat_until_success_or_timeout --fatal-test "container_is_running ${CONTAINER_NAME}" "${TEST_TIMEOUT_IN_SECONDS}" \
+    container_has_service_running "${CONTAINER_NAME}" "${SERVICE_NAME}"
+}
+
+function wait_for_changes_to_be_detected_in_container() {
+  local CONTAINER_NAME="${1}"
+  local TIMEOUT=${TEST_TIMEOUT_IN_SECONDS}
+
+  # shellcheck disable=SC2016
+  repeat_in_container_until_success_or_timeout "${TIMEOUT}" "${CONTAINER_NAME}" bash -c 'source /usr/local/bin/helpers/index.sh; _obtain_hostname_and_domainname; cmp --silent -- <(_monitored_files_checksums) "${CHKSUM_FILE}" >/dev/null'
+}
+
+# Relies on ENV `LOG_LEVEL=debug` or higher
+function wait_until_change_detection_event_completes() {
+  local CONTAINER_NAME="${1}"
+  # Ensure early failure if arg is missing:
+  assert_not_equal "${CONTAINER_NAME}" ""
+
+  # Ensure the container is configured with the required `LOG_LEVEL` ENV:
+  assert_regex \
+    $(docker exec "${CONTAINER_NAME}" env | grep '^LOG_LEVEL=') \
+    '=(debug|trace)$'
+
+  local CHANGE_EVENT_START='Change detected'
+  local CHANGE_EVENT_END='Completed handling of detected change' # debug log
+
+  function __change_event_status() {
+    docker exec "${CONTAINER_NAME}" \
+      grep -oE "${CHANGE_EVENT_START}|${CHANGE_EVENT_END}" /var/log/supervisor/changedetector.log \
+      | tail -1
+  }
+
+  function __is_changedetector_processing() {
+    [[ $(__change_event_status) == "${CHANGE_EVENT_START}" ]]
+  }
+
+  function __is_changedetector_finished() {
+    [[ $(__change_event_status) == "${CHANGE_EVENT_END}" ]]
+  }
+
+  if [[ ! $(__is_changedetector_processing) ]]
+  then
+    # A new change event is expected, wait for it:
+    repeat_until_success_or_timeout 60 __is_changedetector_processing
+  fi
+
+  # Change event is in progress, wait until it finishes:
+  repeat_until_success_or_timeout 60 __is_changedetector_finished
+
+  # NOTE: Although the change event has completed, services like Postfix and Dovecot
+  # may still be in the process of restarting.
+  # You may still want to wait longer if depending on those to be ready.
+}
+
+# An account added to `postfix-accounts.cf` must wait for the `changedetector` service
+# to process the update before Dovecot creates the mail account and associated storage dir:
+function wait_until_account_maildir_exists() {
+  local CONTAINER_NAME=$1
+  local MAIL_ACCOUNT=$2
+
+  local LOCAL_PART="${MAIL_ACCOUNT%@*}"
+  local DOMAIN_PART="${MAIL_ACCOUNT#*@}"
+  local MAIL_ACCOUNT_STORAGE_DIR="/var/mail/${DOMAIN_PART}/${LOCAL_PART}"
+
+  repeat_in_container_until_success_or_timeout 60 "${CONTAINER_NAME}" bash -c "[[ -d ${MAIL_ACCOUNT_STORAGE_DIR} ]]"
+}
+
+function add_mail_account_then_wait_until_ready() {
+  local CONTAINER_NAME=$1
+  local MAIL_ACCOUNT=$2
+  # Password is optional (omit when the password is not needed during the test)
+  local MAIL_PASS="${3:-password_not_relevant_to_test}"
+
+  run docker exec "${CONTAINER_NAME}" setup email add "${MAIL_ACCOUNT}" "${MAIL_PASS}"
+  assert_success
+
+  wait_until_account_maildir_exists "${CONTAINER_NAME}" "${MAIL_ACCOUNT}"
+}
+
+function wait_for_empty_mail_queue_in_container() {
+  local CONTAINER_NAME="${1}"
+  local TIMEOUT=${TEST_TIMEOUT_IN_SECONDS}
+
+  # shellcheck disable=SC2016
+  repeat_in_container_until_success_or_timeout "${TIMEOUT}" "${CONTAINER_NAME}" bash -c '[[ $(mailq) == *"Mail queue is empty"* ]]'
+}

test/helper/setup.bash

@@ -0,0 +1,133 @@
+#!/bin/bash
+
+# -------------------------------------------------------------------
+
+function __initialize_variables() {
+  function __check_if_set() {
+    if [[ ${!1+set} != 'set' ]]
+    then
+      echo "ERROR: (helper/setup.sh) '${1:?No variable name given to __check_if_set}' is not set" >&2
+      exit 1
+    fi
+  }
+
+  local REQUIRED_VARIABLES_FOR_TESTS=(
+    'REPOSITORY_ROOT'
+    'IMAGE_NAME'
+    'CONTAINER_NAME'
+  )
+
+  for VARIABLE in "${REQUIRED_VARIABLES_FOR_TESTS}"
+  do
+    __check_if_set "${VARIABLE}"
+  done
+
+  TEST_TIMEOUT_IN_SECONDS=${TEST_TIMEOUT_IN_SECONDS:-120}
+  NUMBER_OF_LOG_LINES=${NUMBER_OF_LOG_LINES:-10}
+  SETUP_FILE_MARKER="${BATS_TMPDIR:?}/$(basename "${BATS_TEST_FILENAME:?}").setup_file"
+}
+
+# -------------------------------------------------------------------
+
+# @param ${1} relative source in test/config folder
+# @param ${2} (optional) container name, defaults to ${BATS_TEST_FILENAME}
+# @return path to the folder where the config is duplicated
+function duplicate_config_for_container() {
+  local OUTPUT_FOLDER
+  OUTPUT_FOLDER=$(private_config_path "${2}")  || return $?
+
+  rm -rf "${OUTPUT_FOLDER:?}/" || return $? # cleanup
+  mkdir -p "${OUTPUT_FOLDER}" || return $?
+  cp -r "${PWD}/test/config/${1:?}/." "${OUTPUT_FOLDER}" || return $?
+
+  echo "${OUTPUT_FOLDER}"
+}
+
+# TODO: Should also fail early on "docker logs ${1} | egrep '^[  FATAL  ]'"?
+# @param ${1} name of the postfix container
+function wait_for_finished_setup_in_container() {
+  local STATUS=0
+  repeat_until_success_or_timeout --fatal-test "container_is_running ${1}" "${TEST_TIMEOUT_IN_SECONDS}" sh -c "docker logs ${1} | grep 'is up and running'" || STATUS=1
+
+  if [[ ${STATUS} -eq 1 ]]; then
+    echo "Last ${NUMBER_OF_LOG_LINES} lines of container \`${1}\`'s log"
+    docker logs "${1}" | tail -n "${NUMBER_OF_LOG_LINES}"
+  fi
+
+  return ${STATUS}
+}
+
+# Common defaults appropriate for most tests, override vars in each test when necessary.
+# For all tests override in `setup_file()` via an `export` var.
+# For individual test override the var via `local` var instead.
+#
+# For example, if you need an immutable config volume that can't be affected by other tests
+# in the file, then use `local TEST_TMP_CONFIG=$(duplicate_config_for_container . "${UNIQUE_ID_HERE}")`
+function init_with_defaults() {
+  __initialize_variables
+
+  export TEST_TMP_CONFIG
+
+  # In `setup_file()` the default name to use for the currently tested docker container
+  # is `${CONTAINER_NAME}` global defined here. It derives the name from the test filename:
+  # `basename` to ignore absolute dir path and file extension, only extract filename.
+  # In `setup_file()` creates a single copy of the test config folder to use for an entire test file:
+  TEST_TMP_CONFIG=$(duplicate_config_for_container . "${CONTAINER_NAME}")
+
+  # Common complimentary test files, read-only safe to share across containers:
+  export TEST_FILES_CONTAINER_PATH='/tmp/docker-mailserver-test'
+  export TEST_FILES_VOLUME="${REPOSITORY_ROOT}/test/test-files:${TEST_FILES_CONTAINER_PATH}:ro"
+
+  # The config volume cannot be read-only as some data needs to be written at container startup
+  # - two sed failures (unknown lines)
+  # - dovecot-quotas.cf (setup-stack.sh:_setup_dovecot_quotas)
+  # - postfix-aliases.cf (setup-stack.sh:_setup_postfix_aliases)
+  # TODO: Check how many tests need write access. Consider using `docker create` + `docker cp` for easier cleanup.
+  export TEST_CONFIG_VOLUME="${TEST_TMP_CONFIG}:/tmp/docker-mailserver"
+
+  # Default Root CA cert used in TLS tests with `openssl` commands:
+  export TEST_CA_CERT="${TEST_FILES_CONTAINER_PATH}/ssl/example.test/with_ca/ecdsa/ca-cert.ecdsa.pem"
+}
+
+# Using `create` and `start` instead of only `run` allows to modify
+# the container prior to starting it. Otherwise use this combined method.
+# NOTE: Forwards all args to the create method at present.
+function common_container_setup() {
+  common_container_create "${@}"
+  common_container_start
+}
+
+# Common docker setup is centralized here.
+#
+# `X_EXTRA_ARGS` - Optional: Pass an array by it's variable name as a string, it will
+# be used as a reference for appending extra config into the `docker create` below:
+#
+# NOTE: Using array reference for a single input parameter, as this method is still
+# under development while adapting tests to it and requirements it must serve (eg: support base config matrix in CI)
+function common_container_create() {
+  [[ -n ${1} ]] && local -n X_EXTRA_ARGS=${1}
+
+  run docker create \
+    --tty \
+    --name "${CONTAINER_NAME}" \
+    --hostname "${TEST_FQDN:-mail.my-domain.com}" \
+    --volume "${TEST_FILES_VOLUME}" \
+    --volume "${TEST_CONFIG_VOLUME}" \
+    --env ENABLE_AMAVIS=0 \
+    --env ENABLE_CLAMAV=0 \
+    --env ENABLE_UPDATE_CHECK=0 \
+    --env ENABLE_SPAMASSASSIN=0 \
+    --env ENABLE_FAIL2BAN=0 \
+    --env LOG_LEVEL=debug \
+    "${X_EXTRA_ARGS[@]}" \
+    "${IMAGE_NAME}"
+
+  assert_success
+}
+
+function common_container_start() {
+  run docker start "${CONTAINER_NAME}"
+  assert_success
+
+  wait_for_finished_setup_in_container "${CONTAINER_NAME}"
+}

test/mail_spam_bounced.bats

@@ -1,49 +0,0 @@
-load 'test_helper/common'
-# Globals referenced from `test_helper/common`:
-# TEST_NAME
-
-# Can run tests in parallel?: No
-# Shared static container name: TEST_NAME
-
-# Test case
-# ---------
-# When SPAMASSASSIN_SPAM_TO_INBOX=0, spam messages must be bounced (rejected).
-# SPAMASSASSIN_SPAM_TO_INBOX=1 is covered in `mail_spam_junk_folder.bats`.
-# Original test PR: https://github.com/docker-mailserver/docker-mailserver/pull/1485
-
-function teardown() {
-  docker rm -f "${TEST_NAME}"
-}
-
-function setup_file() {
-  init_with_defaults
-}
-
-# Not used
-# function teardown_file() {
-# }
-
-@test "checking amavis: spam message is bounced (rejected)" {
-  # shellcheck disable=SC2034
-  local TEST_DOCKER_ARGS=(
-    --env ENABLE_SPAMASSASSIN=1
-    --env PERMIT_DOCKER=container
-    --env SPAMASSASSIN_SPAM_TO_INBOX=0
-  )
-
-  common_container_setup 'TEST_DOCKER_ARGS'
-
-  _should_bounce_spam
-}
-
-function _should_bounce_spam() {
-  wait_for_smtp_port_in_container_to_respond "${TEST_NAME}"
-
-  # send a spam message
-  run docker exec "${TEST_NAME}" /bin/sh -c "nc 0.0.0.0 25 < /tmp/docker-mailserver-test/email-templates/amavis-spam.txt"
-  assert_success
-
-  # message will be added to a queue with varying delay until amavis receives it
-  run repeat_until_success_or_timeout 60 sh -c "docker logs ${TEST_NAME} | grep 'Blocked SPAM {NoBounceInbound,Quarantined}'"
-  assert_success
-}

test/test_helper/common.bash

@@ -1,7 +1,7 @@
 #!/bin/bash
 
-load 'test_helper/bats-support/load'
+load "${REPOSITORY_ROOT}/test/test_helper/bats-support/load"
-load 'test_helper/bats-assert/load'
+load "${REPOSITORY_ROOT}/test/test_helper/bats-assert/load"
 
 NAME=${NAME:-mailserver-testing:ci}
 

test/test_helper/tls.bash

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-load 'test_helper/common'
+load "${REPOSITORY_ROOT}/test/test_helper/common"
 
 # Helper methods for testing TLS.
 # `_should_*` methods are useful for common high-level functionality.

test/tests/parallel/set1/clamav.bats

@@ -0,0 +1,71 @@
+load "${REPOSITORY_ROOT}/test/helper/setup"
+load "${REPOSITORY_ROOT}/test/helper/common"
+
+TEST_NAME_PREFIX='ClamAV:'
+CONTAINER_NAME='dms-test-clamav'
+
+function setup_file() {
+  init_with_defaults
+
+  # Comment for maintainers about `PERMIT_DOCKER=host`:
+  # https://github.com/docker-mailserver/docker-mailserver/pull/2815/files#r991087509
+  local CUSTOM_SETUP_ARGUMENTS=(
+    --env ENABLE_CLAMAV=1
+    --env ENABLE_AMAVIS=1
+    --env PERMIT_DOCKER=host
+    --env AMAVIS_LOGLEVEL=2
+    --env CLAMAV_MESSAGE_SIZE_LIMIT=30M
+    --env LOG_LEVEL=trace
+  )
+
+  common_container_setup 'CUSTOM_SETUP_ARGUMENTS'
+
+  # wait for ClamAV to be fully setup or we will get errors on the log
+  repeat_in_container_until_success_or_timeout 60 "${CONTAINER_NAME}" test -e /var/run/clamav/clamd.ctl
+
+  wait_for_service "${CONTAINER_NAME}" postfix
+  wait_for_smtp_port_in_container "${CONTAINER_NAME}"
+
+  _run_in_container bash -c "nc 0.0.0.0 25 < /tmp/docker-mailserver-test/email-templates/amavis-virus.txt"
+  assert_success
+
+  wait_for_empty_mail_queue_in_container "${CONTAINER_NAME}"
+}
+
+function teardown_file() { _default_teardown ; }
+
+@test "${TEST_NAME_PREFIX} process clamd is running" {
+  _run_in_container bash -c "ps aux --forest | grep -v grep | grep '/usr/sbin/clamd'"
+  assert_success
+}
+
+@test "${TEST_NAME_PREFIX} log files exist at /var/log/mail directory" {
+  _run_in_container bash -c "ls -1 /var/log/mail/ | grep -E 'clamav|freshclam|mail.log' | wc -l"
+  assert_success
+  assert_output 3
+}
+
+@test "${TEST_NAME_PREFIX} should be identified by Amavis" {
+  _run_in_container grep -i 'Found secondary av scanner ClamAV-clamscan' /var/log/mail/mail.log
+  assert_success
+}
+
+@test "${TEST_NAME_PREFIX} freshclam cron is enabled" {
+  _run_in_container bash -c "grep '/usr/bin/freshclam' -r /etc/cron.d"
+  assert_success
+}
+
+@test "${TEST_NAME_PREFIX} env CLAMAV_MESSAGE_SIZE_LIMIT is set correctly" {
+  _run_in_container grep -q '^MaxFileSize 30M$' /etc/clamav/clamd.conf
+  assert_success
+}
+
+@test "${TEST_NAME_PREFIX} rejects virus" {
+  _run_in_container bash -c "grep 'Blocked INFECTED' /var/log/mail/mail.log | grep '<virus@external.tld> -> <user1@localhost.localdomain>'"
+  assert_success
+}
+
+@test "${TEST_NAME_PREFIX} process clamd restarts when killed" {
+  _run_in_container bash -c "pkill clamd && sleep 10 && ps aux --forest | grep -v grep | grep '/usr/sbin/clamd'"
+  assert_success
+}

test/tests/parallel/set1/default_relay_host.bats

@@ -0,0 +1,23 @@
+load "${REPOSITORY_ROOT}/test/helper/setup"
+load "${REPOSITORY_ROOT}/test/helper/common"
+
+export TEST_NAME_PREFIX='default relay host:'
+export CONTAINER_NAME='dms-test-default_relay_host'
+
+function setup_file() {
+  init_with_defaults
+
+  local CUSTOM_SETUP_ARGUMENTS=(
+    --env DEFAULT_RELAY_HOST=default.relay.host.invalid:25 \
+    --env PERMIT_DOCKER=host \
+  )
+
+  common_container_setup 'CUSTOM_SETUP_ARGUMENTS'
+}
+
+function teardown_file() { _default_teardown ; }
+
+@test "${TEST_NAME_PREFIX} default relay host is added to main.cf" {
+  _run_in_container bash -c 'grep -e "^relayhost =" /etc/postfix/main.cf'
+  assert_output 'relayhost = default.relay.host.invalid:25'
+}

test/tests/parallel/set2/spam_bounced.bats

@@ -0,0 +1,36 @@
+load "${REPOSITORY_ROOT}/test/helper/setup"
+load "${REPOSITORY_ROOT}/test/helper/common"
+
+TEST_NAME_PREFIX='spam (Amavis):'
+CONTAINER_NAME='dms-test-spam_bounced'
+
+function setup_file() {
+  init_with_defaults
+
+  local CUSTOM_SETUP_ARGUMENTS=(
+    --env ENABLE_AMAVIS=1
+    --env ENABLE_SPAMASSASSIN=1
+    --env PERMIT_DOCKER=container
+    --env SPAMASSASSIN_SPAM_TO_INBOX=0
+  )
+
+  common_container_setup 'CUSTOM_SETUP_ARGUMENTS'
+  wait_for_smtp_port_in_container_to_respond "${CONTAINER_NAME}"
+}
+
+function teardown_file() { _default_teardown ; }
+
+# Test case
+# ---------
+# When SPAMASSASSIN_SPAM_TO_INBOX=0, spam messages must be bounced (rejected).
+# SPAMASSASSIN_SPAM_TO_INBOX=1 is covered in `mail_spam_junk_folder.bats`.
+# Original test PR: https://github.com/docker-mailserver/docker-mailserver/pull/1485
+@test "${TEST_NAME_PREFIX} spam message is bounced (rejected)" {
+  # send a spam message
+  _run_in_container /bin/sh -c "nc 0.0.0.0 25 < /tmp/docker-mailserver-test/email-templates/amavis-spam.txt"
+  assert_success
+
+  # message will be added to a queue with varying delay until amavis receives it
+  run repeat_until_success_or_timeout 60 sh -c "docker logs ${CONTAINER_NAME} | grep 'Blocked SPAM {NoBounceInbound,Quarantined}'"
+  assert_success
+}

test/tests/parallel/set2/template.bats

@@ -0,0 +1,39 @@
+# ? load the BATS helper
+load "${REPOSITORY_ROOT}/test/helper/setup"
+load "${REPOSITORY_ROOT}/test/helper/common"
+
+# ? global variable initialization
+# ?   to identify the test easily
+TEST_NAME_PREFIX='template:'
+# ?   must be unique
+CONTAINER_NAME='dms-test-template'
+
+# ? test setup
+
+function setup_file() {
+  # ? optional setup before container is started
+
+  # ? initialize the test helpers
+  init_with_defaults
+
+  # ? add custom arguments supplied to `docker run` here
+  local CUSTOM_SETUP_ARGUMENTS=(
+    --env LOG_LEVEL=trace
+  )
+
+  # ? use a helper to correctly setup the container
+  common_container_setup 'CUSTOM_SETUP_ARGUMENTS'
+
+  # ? optional setup after the container is started
+}
+
+# ? test finalization
+
+function teardown_file() { _default_teardown ; }
+
+# ? actual unit tests
+
+@test "${TEST_NAME_PREFIX} default check" {
+  _run_in_container bash -c "true"
+  assert_success
+}

test/tests/parallel/set3/dovecot_inet_protocol.bats

@@ -0,0 +1,46 @@
+load "${REPOSITORY_ROOT}/test/helper/setup"
+load "${REPOSITORY_ROOT}/test/helper/common"
+
+TEST_NAME_PREFIX='Dovecot protocols:'
+
+@test "${TEST_NAME_PREFIX} dual-stack IP configuration" {
+  local CONTAINER_NAME='dms-test-dovecot_protocols_all'
+  local CUSTOM_SETUP_ARGUMENTS=(--env DOVECOT_INET_PROTOCOLS=)
+
+  init_with_defaults
+  common_container_setup 'CUSTOM_SETUP_ARGUMENTS'
+
+  _run_in_container grep '^#listen = \*, ::' /etc/dovecot/dovecot.conf
+  assert_success
+  assert_output '#listen = *, ::'
+
+  docker rm -f "${CONTAINER_NAME}"
+}
+
+@test "${TEST_NAME_PREFIX} IPv4 configuration" {
+  local CONTAINER_NAME='dms-test-dovecot_protocols_ipv4'
+  local CUSTOM_SETUP_ARGUMENTS=(--env DOVECOT_INET_PROTOCOLS=ipv4)
+
+  init_with_defaults
+  common_container_setup 'CUSTOM_SETUP_ARGUMENTS'
+
+  _run_in_container grep '^listen = \*$' /etc/dovecot/dovecot.conf
+  assert_success
+  assert_output 'listen = *'
+
+  docker rm -f "${CONTAINER_NAME}"
+}
+
+@test "${TEST_NAME_PREFIX} IPv6 configuration" {
+  local CONTAINER_NAME='dms-test-dovecot_protocols_ipv6'
+  local CUSTOM_SETUP_ARGUMENTS=(--env DOVECOT_INET_PROTOCOLS=ipv6)
+
+  init_with_defaults
+  common_container_setup 'CUSTOM_SETUP_ARGUMENTS'
+
+  _run_in_container grep '^listen = \[::\]$' /etc/dovecot/dovecot.conf
+  assert_success
+  assert_output 'listen = [::]'
+
+  docker rm -f "${CONTAINER_NAME}"
+}

test/tests/parallel/set3/helper-functions.bats

@@ -0,0 +1,23 @@
+load "${REPOSITORY_ROOT}/test/helper/setup"
+load "${REPOSITORY_ROOT}/test/helper/common"
+
+TEST_NAME_PREFIX='helper functions inside container:'
+CONTAINER_NAME='dms-test-helper_functions'
+
+function setup_file() {
+  init_with_defaults
+  common_container_setup
+}
+
+function teardown_file() { _default_teardown ; }
+
+@test "${TEST_NAME_PREFIX} _sanitize_ipv4_to_subnet_cidr" {
+  _run_in_container bash -c "source /usr/local/bin/helpers/index.sh; _sanitize_ipv4_to_subnet_cidr 255.255.255.255/0"
+  assert_output "0.0.0.0/0"
+
+  _run_in_container bash -c "source /usr/local/bin/helpers/index.sh; _sanitize_ipv4_to_subnet_cidr 192.168.255.14/20"
+  assert_output "192.168.240.0/20"
+
+  _run_in_container bash -c "source /usr/local/bin/helpers/index.sh; _sanitize_ipv4_to_subnet_cidr 192.168.255.14/32"
+  assert_output "192.168.255.14/32"
+}

test/tests/serial/mail_changedetector.bats

@@ -1,4 +1,4 @@
-load 'test_helper/common'
+load "${REPOSITORY_ROOT}/test/test_helper/common"
 
 # Note if tests fail asserting against `supervisorctl tail changedetector` output,
 # use `supervisorctl tail -<num bytes> changedetector` instead to increase log output.

test/tests/serial/mail_disabled_clamav_spamassassin.bats

@@ -1,4 +1,4 @@
-load 'test_helper/common'
+load "${REPOSITORY_ROOT}/test/test_helper/common"
 
 setup_file() {
   local PRIVATE_CONFIG

test/tests/serial/mail_dnsbl.bats

@@ -1,4 +1,4 @@
-load 'test_helper/common'
+load "${REPOSITORY_ROOT}/test/test_helper/common"
 
 CONTAINER="mail_dnsbl_enabled"
 CONTAINER2="mail_dnsbl_disabled"

test/tests/serial/mail_fail2ban.bats

@@ -1,4 +1,4 @@
-load 'test_helper/common'
+load "${REPOSITORY_ROOT}/test/test_helper/common"
 
 function setup_file() {
   local PRIVATE_CONFIG

test/tests/serial/mail_fetchmail.bats

@@ -1,4 +1,4 @@
-load 'test_helper/common'
+load "${REPOSITORY_ROOT}/test/test_helper/common"
 
 function setup_file() {
   local PRIVATE_CONFIG

test/tests/serial/mail_fetchmail_parallel.bats

@@ -1,4 +1,4 @@
-load 'test_helper/common'
+load "${REPOSITORY_ROOT}/test/test_helper/common"
 
 function setup_file() {
   local PRIVATE_CONFIG

test/tests/serial/mail_hostname.bats

@@ -1,4 +1,4 @@
-load 'test_helper/common'
+load "${REPOSITORY_ROOT}/test/test_helper/common"
 
 
 function setup_file() {

test/tests/serial/mail_lmtp_ip.bats

@@ -1,4 +1,4 @@
-load 'test_helper/common'
+load "${REPOSITORY_ROOT}/test/test_helper/common"
 
 setup_file() {
   local PRIVATE_CONFIG PRIVATE_ETC

test/tests/serial/mail_pop3.bats

@@ -1,4 +1,4 @@
-load 'test_helper/common'
+load "${REPOSITORY_ROOT}/test/test_helper/common"
 
 function setup_file() {
   local PRIVATE_CONFIG

test/tests/serial/mail_postfix_inet.bats

@@ -1,4 +1,4 @@
-load 'test_helper/common'
+load "${REPOSITORY_ROOT}/test/test_helper/common"
 
 # Test case
 # ---------

test/tests/serial/mail_postscreen.bats

@@ -1,4 +1,4 @@
-load 'test_helper/common'
+load "${REPOSITORY_ROOT}/test/test_helper/common"
 
 setup() {
   # Getting mail container IP

test/tests/serial/mail_privacy.bats

@@ -1,4 +1,4 @@
-load 'test_helper/common'
+load "${REPOSITORY_ROOT}/test/test_helper/common"
 
 function setup_file() {
   local PRIVATE_CONFIG

test/tests/serial/mail_quotas_disabled.bats

@@ -1,4 +1,4 @@
-load 'test_helper/common'
+load "${REPOSITORY_ROOT}/test/test_helper/common"
 
 # Test case
 # ---------

test/tests/serial/mail_smtponly.bats

@@ -1,4 +1,4 @@
-load 'test_helper/common'
+load "${REPOSITORY_ROOT}/test/test_helper/common"
 
 function setup_file() {
   docker run --rm -d --name mail_smtponly \

test/tests/serial/mail_spam_junk_folder.bats

@@ -1,4 +1,4 @@
-load 'test_helper/common'
+load "${REPOSITORY_ROOT}/test/test_helper/common"
 
 # Test case
 # ---------

test/tests/serial/mail_special_use_folders.bats

@@ -1,4 +1,4 @@
-load 'test_helper/common'
+load "${REPOSITORY_ROOT}/test/test_helper/common"
 
 setup_file() {
   local PRIVATE_CONFIG

test/tests/serial/mail_ssl_letsencrypt.bats

@@ -1,5 +1,5 @@
-load 'test_helper/common'
+load "${REPOSITORY_ROOT}/test/test_helper/common"
-load 'test_helper/tls'
+load "${REPOSITORY_ROOT}/test/test_helper/tls"
 
 # Globals referenced from `test_helper/common`:
 # TEST_NAME TEST_FQDN TEST_TMP_CONFIG

test/tests/serial/mail_ssl_manual.bats

@@ -1,5 +1,5 @@
 #!/usr/bin/env bats
-load 'test_helper/common'
+load "${REPOSITORY_ROOT}/test/test_helper/common"
 
 function setup_file() {
   # Internal copies made by `start-mailserver.sh`:

test/tests/serial/mail_time.bats

@@ -1,4 +1,4 @@
-load 'test_helper/common'
+load "${REPOSITORY_ROOT}/test/test_helper/common"
 
 setup_file() {
   local PRIVATE_CONFIG

test/tests/serial/mail_tls_dhparams.bats

@@ -1,4 +1,4 @@
-load 'test_helper/common'
+load "${REPOSITORY_ROOT}/test/test_helper/common"
 
 # Test case
 # ---------

test/tests/serial/mail_undef_spam_subject.bats

@@ -1,4 +1,4 @@
-load 'test_helper/common'
+load "${REPOSITORY_ROOT}/test/test_helper/common"
 
 function setup() {
   local PRIVATE_CONFIG

test/tests/serial/mail_with_imap.bats

@@ -1,4 +1,4 @@
-load 'test_helper/common'
+load "${REPOSITORY_ROOT}/test/test_helper/common"
 
 setup_file() {
   local PRIVATE_CONFIG

test/tests/serial/mail_with_ldap.bats

@@ -1,4 +1,4 @@
-load 'test_helper/common'
+load "${REPOSITORY_ROOT}/test/test_helper/common"
 
 function setup_file() {
   pushd test/docker-openldap/ || return 1

test/tests/serial/mail_with_mdbox.bats

@@ -1,4 +1,4 @@
-load 'test_helper/common'
+load "${REPOSITORY_ROOT}/test/test_helper/common"
 
 setup_file() {
   local PRIVATE_CONFIG

test/tests/serial/mail_with_postgrey.bats

@@ -1,4 +1,4 @@
-load 'test_helper/common'
+load "${REPOSITORY_ROOT}/test/test_helper/common"
 
 function setup_file() {
   local PRIVATE_CONFIG

test/tests/serial/mail_with_postgrey_disabled_by_default.bats

@@ -1,4 +1,4 @@
-load 'test_helper/common'
+load "${REPOSITORY_ROOT}/test/test_helper/common"
 
 function setup() {
   local PRIVATE_CONFIG

test/tests/serial/mail_with_relays.bats

@@ -1,4 +1,4 @@
-load 'test_helper/common'
+load "${REPOSITORY_ROOT}/test/test_helper/common"
 
 function setup_file() {
   # We use a temporary config directory since we'll be dynamically editing

test/tests/serial/mail_with_sdbox.bats

@@ -1,4 +1,4 @@
-load 'test_helper/common'
+load "${REPOSITORY_ROOT}/test/test_helper/common"
 
 setup_file() {
   local PRIVATE_CONFIG

test/tests/serial/no_container.bats

@@ -1,4 +1,4 @@
-load 'test_helper/common'
+load "${REPOSITORY_ROOT}/test/test_helper/common"
 
 function setup_file() {
   # Fail early if the test image is already running:

test/tests/serial/open_dkim.bats

@@ -1,4 +1,4 @@
-load 'test_helper/common'
+load "${REPOSITORY_ROOT}/test/test_helper/common"
 
 export IMAGE_NAME CONTAINER_NAME TEST_FILE
 

test/tests/serial/permit_docker.bats

@@ -1,4 +1,4 @@
-load 'test_helper/common'
+load "${REPOSITORY_ROOT}/test/test_helper/common"
 
 NON_DEFAULT_DOCKER_MAIL_NETWORK_NAME=non-default-docker-mail-network
 setup_file() {

test/tests/serial/security_tls_cipherlists.bats

@@ -1,5 +1,5 @@
 #!/usr/bin/env bats
-load 'test_helper/common'
+load "${REPOSITORY_ROOT}/test/test_helper/common"
 # Globals ${BATS_TMPDIR} and ${NAME}
 # `${NAME}` defaults to `mailserver-testing:ci`
 

test/tests/serial/sedfile.bats

@@ -1,4 +1,4 @@
-load 'test_helper/common'
+load "${REPOSITORY_ROOT}/test/test_helper/common"
 
 CONTAINER='sedfile'
 TEST_FILE='/tmp/sedfile-test.txt'

test/tests/serial/setup-cli.bats

@@ -1,4 +1,4 @@
-load 'test_helper/common'
+load "${REPOSITORY_ROOT}/test/test_helper/common"
 
 # Globals referenced from `test_helper/common`:
 # TEST_NAME (should match the filename, minus the bats extension)

test/tests/serial/test_helper.bats

@@ -1,14 +1,14 @@
-load 'test_helper/bats-support/load'
+load "${REPOSITORY_ROOT}/test/test_helper/common"
-load 'test_helper/bats-assert/load'
-load 'test_helper/common'
 
-@test "repeat_until_success_or_timeout returns instantly on success" {
+TEST_NAME_PREFIX='test helper functions:'
+
+@test "${TEST_NAME_PREFIX} repeat_until_success_or_timeout returns instantly on success" {
   SECONDS=0
   repeat_until_success_or_timeout 1 true
   [[ ${SECONDS} -le 1 ]]
 }
 
-@test "repeat_until_success_or_timeout waits for timeout on persistent failure" {
+@test "${TEST_NAME_PREFIX} repeat_until_success_or_timeout waits for timeout on persistent failure" {
   SECONDS=0
   run repeat_until_success_or_timeout 2 false
   [[ ${SECONDS} -ge 2 ]]
@@ -16,7 +16,7 @@ load 'test_helper/common'
   assert_output --partial "Timed out on command"
 }
 
-@test "repeat_until_success_or_timeout aborts immediately on fatal failure" {
+@test "${TEST_NAME_PREFIX} repeat_until_success_or_timeout aborts immediately on fatal failure" {
   SECONDS=0
   run repeat_until_success_or_timeout --fatal-test false 2 false
   [[ ${SECONDS} -le 1 ]]
@@ -24,7 +24,7 @@ load 'test_helper/common'
   assert_output --partial "early aborting"
 }
 
-@test "repeat_until_success_or_timeout expects integer timeout" {
+@test "${TEST_NAME_PREFIX} repeat_until_success_or_timeout expects integer timeout" {
   run repeat_until_success_or_timeout 1 true
   assert_success
 
@@ -35,27 +35,27 @@ load 'test_helper/common'
   assert_failure
 }
 
-@test "run_until_success_or_timeout returns instantly on success" {
+@test "${TEST_NAME_PREFIX} run_until_success_or_timeout returns instantly on success" {
   SECONDS=0
   run_until_success_or_timeout 2 true
   [[ ${SECONDS} -le 1 ]]
   assert_success
 }
 
-@test "run_until_success_or_timeout waits for timeout on persistent failure" {
+@test "${TEST_NAME_PREFIX} run_until_success_or_timeout waits for timeout on persistent failure" {
   SECONDS=0
   ! run_until_success_or_timeout 2 false
   [[ ${SECONDS} -ge 2 ]]
   assert_failure
 }
 
-@test "repeat_in_container_until_success_or_timeout fails immediately for non-running container" {
+@test "${TEST_NAME_PREFIX} repeat_in_container_until_success_or_timeout fails immediately for non-running container" {
   SECONDS=0
   ! repeat_in_container_until_success_or_timeout 10 name-of-non-existing-container true
   [[ ${SECONDS} -le 1 ]]
 }
 
-@test "repeat_in_container_until_success_or_timeout run command in container" {
+@test "${TEST_NAME_PREFIX} repeat_in_container_until_success_or_timeout run command in container" {
   local CONTAINER_NAME
   CONTAINER_NAME=$(docker run --rm -d alpine sleep 100)
   SECONDS=0
@@ -65,7 +65,7 @@ load 'test_helper/common'
   assert_output "${CONTAINER_NAME}"
 }
 
-@test "container_is_running" {
+@test "${TEST_NAME_PREFIX} container_is_running" {
   local CONTAINER_NAME
   CONTAINER_NAME=$(docker run --rm -d alpine sleep 100)
   container_is_running "${CONTAINER_NAME}"
@@ -73,7 +73,7 @@ load 'test_helper/common'
   ! container_is_running "${CONTAINER_NAME}"
 }
 
-@test "wait_for_smtp_port_in_container aborts wait after timeout" {
+@test "${TEST_NAME_PREFIX} wait_for_smtp_port_in_container aborts wait after timeout" {
   local CONTAINER_NAME
   CONTAINER_NAME=$(docker run --rm -d alpine sleep 100)
   SECONDS=0
@@ -84,7 +84,7 @@ load 'test_helper/common'
 }
 
 # NOTE: Test requires external network access available
-@test "wait_for_smtp_port_in_container returns immediately when port found" {
+@test "${TEST_NAME_PREFIX} wait_for_smtp_port_in_container returns immediately when port found" {
   local CONTAINER_NAME
   CONTAINER_NAME=$(docker run --rm -d alpine sh -c "sleep 10")
 
@@ -97,7 +97,7 @@ load 'test_helper/common'
   assert_success
 }
 
-@test "wait_for_finished_setup_in_container" {
+@test "${TEST_NAME_PREFIX} wait_for_finished_setup_in_container" {
   # variable not local to make visible to teardown
   local PRIVATE_CONFIG
   PRIVATE_CONFIG=$(duplicate_config_for_container .)
@@ -119,7 +119,7 @@ load 'test_helper/common'
   [[ ${SECONDS} -gt 0 ]]
 }
 
-@test "duplicate_config_for_container" {
+@test "${TEST_NAME_PREFIX} duplicate_config_for_container" {
   local path
   path=$(duplicate_config_for_container duplicate_config_test)
 
@@ -130,7 +130,7 @@ load 'test_helper/common'
   assert_failure
 }
 
-@test "container_has_service_running/wait_for_service" {
+@test "${TEST_NAME_PREFIX} container_has_service_running/wait_for_service" {
   local PRIVATE_CONFIG
   PRIVATE_CONFIG=$(duplicate_config_for_container .)
 
@@ -158,7 +158,7 @@ load 'test_helper/common'
   assert_failure
 }
 
-@test "wait_for_changes_to_be_detected_in_container fails when timeout is reached" {
+@test "${TEST_NAME_PREFIX} wait_for_changes_to_be_detected_in_container fails when timeout is reached" {
   local PRIVATE_CONFIG
   PRIVATE_CONFIG=$(duplicate_config_for_container .)
 
@@ -184,7 +184,7 @@ load 'test_helper/common'
   ! TEST_TIMEOUT_IN_SECONDS=0 wait_for_changes_to_be_detected_in_container "${CONTAINER_NAME}"
 }
 
-@test "wait_for_changes_to_be_detected_in_container succeeds within timeout" {
+@test "${TEST_NAME_PREFIX} wait_for_changes_to_be_detected_in_container succeeds within timeout" {
   local PRIVATE_CONFIG
   PRIVATE_CONFIG=$(duplicate_config_for_container .)
 
@@ -210,7 +210,7 @@ load 'test_helper/common'
 }
 
 # TODO investigate why this test fails
-@test "wait_for_empty_mail_queue_in_container fails when timeout reached" {
+@test "${TEST_NAME_PREFIX} wait_for_empty_mail_queue_in_container fails when timeout reached" {
   skip 'disabled as it fails randomly: https://github.com/docker-mailserver/docker-mailserver/pull/2177'
 
   local PRIVATE_CONFIG
@@ -242,7 +242,7 @@ load 'test_helper/common'
 }
 
 # TODO investigate why this test fails
-@test "wait_for_empty_mail_queue_in_container succeeds within timeout" {
+@test "${TEST_NAME_PREFIX} wait_for_empty_mail_queue_in_container succeeds within timeout" {
   skip 'disabled as it fails randomly: https://github.com/docker-mailserver/docker-mailserver/pull/2177'
 
   local PRIVATE_CONFIG

test/tests/serial/tests.bats

@@ -1,4 +1,4 @@
-load 'test_helper/common'
+load "${REPOSITORY_ROOT}/test/test_helper/common"
 
 setup_file() {
   local PRIVATE_CONFIG