rspamd: add feature for adjusting options with a file parsed by DMS (#3059)

Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>

README.md

@@ -28,6 +28,7 @@ If you have issues, read the full `README` **and** the [documentation][documenta
 
 - [Postfix](http://www.postfix.org) with SMTP or LDAP auth
 - [Dovecot](https://www.dovecot.org) for SASL, IMAP (or POP3), with LDAP Auth, Sieve and [quotas](https://docker-mailserver.github.io/docker-mailserver/edge/config/user-management/accounts#notes)
+- [Rspamd](https://rspamd.com/)
 - [Amavis](https://www.amavis.org/)
 - [SpamAssassin](http://spamassassin.apache.org/) supporting custom rules
 - [ClamAV](https://www.clamav.net/) with automatic updates

docs/content/config/advanced/optional-config.md

@@ -39,6 +39,7 @@ This is a list of all configuration files and directories which are optional or
 - **dovecot.cf:** replaces `/etc/dovecot/local.conf`. (Docs: [Override Dovecot Defaults][docs-override-dovecot])
 - **dovecot-quotas.cf:** list of custom quotas per mailbox. (Docs: [Accounts][docs-accounts-quota])
 - **user-patches.sh:** this file will be run after all configuration files are set up, but before the postfix, amavis and other daemons are started. (Docs: [FAQ - How to adjust settings with the `user-patches.sh` script][docs-faq-userpatches])
+- **rspamd-commands:** list of simple commands to adjust Rspamd modules in an easy way (Docs: [Rspamd][docs-rspamd-commands])
 
 [docs-accounts-quota]: ../../config/user-management/accounts.md#notes
 [docs-aliases-regex]: ../../config/user-management/aliases.md#configuring-regexp-aliases
@@ -53,4 +54,5 @@ This is a list of all configuration files and directories which are optional or
 [docs-sieve]: ./mail-sieve.md
 [docs-setupsh]: ../../config/setup.sh.md
 [docs-ssl]: ../../config/security/ssl.md
+[docs-rspamd-commands]: ../security/rspamd.md#with-the-help-of-a-custom-file
 [github-commit-setup-stack.sh-L411]: https://github.com/docker-mailserver/docker-mailserver/blob/941e7acdaebe271eaf3d296b36d4d81df4c54b90/target/scripts/startup/setup-stack.sh#L411

docs/content/config/security/rspamd.md

@@ -2,45 +2,105 @@
 title: 'Security | Rspamd'
 ---
 
-!!! warning "Implementation of Rspamd into DMS is WIP!"
+!!! warning "The current state of Rspamd integration into DMS"
+
+    Recent pull requests have stabilized integration of Rspamd to a point that we encourage users to test the feature. We are confident that there are no major bugs in our integration that make using Rspamd infeasible. Please note that there may still be (breaking) changes ahead as integration is still work in progress!
+
+    We expect to stabilize this feature with version `v12.1.0`.
 
 ## About
 
-Rspamd is a "fast, free and open-source spam filtering system". It offers high performance as it is written in C. Visit [their homepage][homepage] for more details.
+Rspamd is a ["fast, free and open-source spam filtering system"][homepage]. DMS integrates Rspamd like any other service. You will need to enable Rspamd (via `ENABLE_RSPAMD=1`) manually as it is disabled by default.
 
-## Integration & Configuration
+We provide a very simple but easy to maintain setup of Rspamd. If you want to have a look at the default configuration files for Rspamd that DMS packs, navigate to [`target/rspamd/` inside the repository][dms-default-configuration]. Please consult the [section "The Default Configuration"](#the-default-configuration) section down below for a written overview.
 
-We provide a very simple but easy to maintain setup of RSpamd. The proxy worker operates in [self-scan mode][proxy-self-scan-mode]. This simplifies the setup as we do not require a normal worker. You can easily change this though by [overriding the configuration by DMS](#providing-overriding-settings).
+If you want to adjust Rspamd's configuration, have a look at the ["Providing Custom Settings & Overriding Settings" section](#providing-custom-settings-overriding-settings) down below.
 
-### Providing & Overriding Settings
+!!! note "AMD64 vs ARM64"
+
+    We are currently doing a best-effort installation of Rspamd for ARM64 (from the Debian backports repository for Debian 11). The current version difference is two minor versions (AMD64 is at version 3.4, ARM64 at 3.2 \[13th Feb 2023\]).
+
+    Maintainers noticed only few differences, some of them with a big impact though. For those running Rspamd on ARM64, we recommend [disabling](#with-the-help-of-a-custom-file) the [DKIM signing module][dkim-signing-module] if you don't use it.
+
+## The Default Configuration
+
+### Mode of Operation
+
+The proxy worker operates in [self-scan mode][proxy-self-scan-mode]. This simplifies the setup as we do not require a normal worker. You can easily change this though by [overriding the configuration by DMS](#providing-custom-settings-overriding-settings).
+
+DMS does not set a default password for the controller worker. You may want to do that yourself. In setup where you already have an authentication provider in front of the Rspamd webpage, you may add `secure_ip = "0.0.0.0/0";` to `worker-controller.inc` to disable password authentication inside Rspamd completely.
+
+### Modules
+
+You can find a list of all Rspamd modules [on their website][modules].
+
+#### Disabled By Default
+
+DMS disables certain modules (clickhouse, elastic, greylist, neural, reputation, spamassassin, url_redirector, metric_exporter) by default. We believe these are not required in a standard setup, and they would otherwise needlessly use system resources.
+
+#### Anti-Virus (ClamAV)
+
+You can choose to enable ClamAV, and Rspamd will then use it to check for viruses. Just set the environment variable `ENABLE_CLAMAV=1`.
+
+#### RBLs (Realtime Blacklists) / DNSBLs (DNS-based Blacklists)
+
+The [RBL module](https://rspamd.com/doc/modules/rbl.html) is enabled by default. As a consequence, Rspamd will perform DNS lookups to a variety of blacklists. Whether an RBL or a DNSBL is queried depends on where the domain name was obtained: RBL servers are queried with IP addresses extracted from message headers, DNSBL server are queried with domains and IP addresses extracted from the message body \[[source][rbl-vs-dnsbl]\].
+
+!!! danger "Rspamd and DNS Block Lists"
+
+    When the RBL module is enabled, Rspamd will do a variety of DNS requests to (amongst other things) DNSBLs. There are a variety of issues involved when using DNSBLs. Rspamd will try to mitigate some of them by properly evaluating all return codes. This evaluation is a best effort though, so if the DNSBL operators change or add return codes, it may take a while for Rspamd to adjust as well.
+
+    If you want to use DNSBLs, **try to use your own DNS resolver** and make sure it is set up correctly, i.e. it should be a non-public & **recursive** resolver. Otherwise, you might not be able ([see this Spamhaus post](https://www.spamhaus.org/faq/section/DNSBL%20Usage#365)) to make use of the block lists.
+
+### Missing in the Current Implementation
+
+We currently lack easy integration for [DKIM signing outgoing mails][dkim-signing-module]. We use OpenDKIM though which works just as well. If you want to use Rspamd for DKIM signing, you need to provide all settings yourself and probably also set the environment variable `ENABLE_OPENDKIM=0`. Rspamd will still check for valid DKIM signatures for incoming mail by default.
+
+## Providing Custom Settings & Overriding Settings
+
+### Manually
 
 DMS brings sane default settings for Rspamd. They are located at `/etc/rspamd/local.d/` inside the container (or `target/rspamd/local.d/` in the repository). If you want to change these settings and / or provide your own settings, you can
 
 1. place files at `/etc/rspamd/override.d/` which will override Rspamd settings and DMS settings
 2. (re-)place files at `/etc/rspamd/local.d/` to override DMS settings and merge them with Rspamd settings
 
-You can find a list of all Rspamd modules [on their website][modules].
+!!! warning "Clashing Overrides"
 
-### DMS' Defaults
+    Note that when also [using the `rspamd-commands` file](#with-the-help-of-a-custom-file), files in `override.d` may be overwritten in case you adjust them manually and with the help of the file.
 
-!!! danger "Rspamd and DNS Block Lists"
+### With the Help of a Custom File
 
-    When using Rspamd, the [RBL module](https://rspamd.com/doc/modules/rbl.html) is enabled by default. As a consequence, Rspamd will do a variety of DNS requests. Amongst other things, Rspamd will query DNS block lists (DNSBLs).
+DMS provides the ability to do simple adjustments to Rspamd modules with the help of a single file. Just place a file called `rspamd-modules.conf` into the directory `docker-data/dms/config/` (which translates to `/tmp/docker-mailserver/` in the container). If this file is present, DMS will evaluate it. The structure is _very_ simple. Each line in the file looks like this:
 
-    There are a variety of issues involved when using DNSBLs. Rspamd will try to mitigate some of them by properly evaluating all return codes. We urge you not to rely on this though.
+```txt
+COMMAND ARGUMENT1 ARGUMENT2 ARGUMENT3
+```
 
-    If you want to use RBLs, **try to use your own DNS resolver** and make sure it is set up correctly, i.e. it should be a non-public & **recursive** resolver. Otherwise, you might not be able ([see this Spamhaus post](https://www.spamhaus.org/faq/section/DNSBL%20Usage#365)) to make use of the block lists.
+where `COMMAND` can be:
 
-You can choose to enable ClamAV, and Rspamd will then use it to check for viruses. Just set the environment variable `ENABLE_CLAMAV=1`.
+1. `disable-module`: disables the module with name `ARGUMENT1`
+2. `enable-module`: explicitly enables the module with name `ARGUMENT1`
+3. `set-option-for-module`: sets the value for option `ARGUMENT2` to `ARGUMENT3` inside module `ARGUMENT1`
+4. `set-option-for-controller`: set the value of option `ARGUMENT1` to `ARGUMENT2` for the controller worker
+5. `set-option-for-proxy`: set the value of option `ARGUMENT1` to `ARGUMENT2` for the proxy worker
+6. `set-common-option`: set the option `ARGUMENT1` that [defines basic Rspamd behaviour][basic-options] to value `ARGUMENT2`
+7. `add-line`: this will add the complete line after `ARGUMENT1` (with all characters) to the file `/etc/rspamd/override.d/<ARGUMENT1>`
 
-DMS disables certain modules (clickhouse, elastic, greylist, neural, reputation, spamassassin, url_redirector, metric_exporter) by default. We believe these are not required in a standard setup, and needlessly use resources. You can re-activate them by replacing `/etc/rspamd/local.d/<MODULE>.conf` or overriding DMS' default with `/etc/rspamd/override.d/<MODULE>.conf`.
+!!! note "File Names & Extensions"
 
-DMS does not set a default password for the controller worker. You may want to do that yourself. In setup where you already have an authentication provider in front of the Rspamd webpage, you may add `secure_ip = "0.0.0.0/0";` to `worker-controller.inc` to disable password authentication inside Rspamd completely.
+    For command 1 - 3, we append the `.conf` suffix to the module name to get the correct file name automatically. For commands 4 - 6, the file name is fixed (you don't even need to provide it). For command 7, you will need to provide the whole file name (including the suffix) yourself!
 
-## Missing in DMS' Current Implementation
+You can also have comments (the line starts with `#`) and blank lines in `rspamd-modules.conf` - they are properly handled and not evaluated.
 
-We currently lack easy integration for DKIM signing outgoing mails. We use OpenDKIM though which works just as well. If you want to use Rspamd for DKIM signing, you need to provide all settings yourself and probably also set the environment `ENABLE_OPENDKIM=0`. Rspamd will still check for valid DKIM signatures for incoming mail by default.
+!!! tip "Adjusting Modules This Way"
+
+    These simple commands are meant to give users the ability to _easily_ alter modules and their options. As a consequence, they are not powerful enough to enable multi-line adjustments. If you need to do something more complex, we advise to do that [manually](#manually)!
 
 [homepage]: https://rspamd.com/
 [modules]: https://rspamd.com/doc/modules/
 [proxy-self-scan-mode]: https://rspamd.com/doc/workers/rspamd_proxy.html#self-scan-mode
+[dms-default-configuration]: https://github.com/docker-mailserver/docker-mailserver/tree/master/target/rspamd
+[rbl-vs-dnsbl]: https://forum.eset.com/topic/25277-dnsbl-vs-rbl-mail-security/?do=findComment&comment=119818
+[dkim-signing-module]: https://rspamd.com/doc/modules/dkim_signing.html
+[basic-options]: https://rspamd.com/doc/configuration/options.html

target/postfix/main.cf

@@ -88,8 +88,6 @@ virtual_alias_maps = texthash:/etc/postfix/virtual
 # Milters used by DKIM
 milter_protocol = 6
 milter_default_action = accept
-dkim_milter = inet:localhost:8891
-dmarc_milter = inet:localhost:8893
 smtpd_milters =
 non_smtpd_milters =
 

target/scripts/build/packages.sh

@@ -132,16 +132,31 @@ function _install_dovecot
 function _install_rspamd
 {
   _log 'trace' 'Adding Rspamd package signatures'
-  curl -sSfL https://rspamd.com/apt-stable/gpg.key | gpg --dearmor >/etc/apt/trusted.gpg.d/rspamd.gpg
+  local DEB_FILE='/etc/apt/sources.list.d/rspamd.list'
+  local RSPAMD_PACKAGE_NAME
 
-  echo "deb [arch=amd64 signed-by=/etc/apt/trusted.gpg.d/rspamd.gpg] http://rspamd.com/apt-stable/ bullseye main" \
+  # We try getting the most recent version of Rspamd for aarch64 (from an official source, which
-    >/etc/apt/sources.list.d/rspamd.list
+  # is the backports repository). The version for aarch64 is 3.2; the most recent version for amd64
-  echo "deb-src [arch=amd64 signed-by=/etc/apt/trusted.gpg.d/rspamd.gpg] http://rspamd.com/apt-stable/ bullseye main" \
+  # that we get with the official PPA is 3.4.
-    >>/etc/apt/sources.list.d/rspamd.list
+  #
+  # Not removing it later is fine as you have to explicitly opt into installing a backports package
+  # which is not something you could be doing by accident.
+  if [[ $(uname --machine) == 'aarch64' ]]
+  then
+    echo '# Official Rspamd PPA does not support aarch64, so we use the Bullseye backports' >"${DEB_FILE}"
+    echo 'deb [arch=arm64] http://deb.debian.org/debian bullseye-backports main' >>"${DEB_FILE}"
+    RSPAMD_PACKAGE_NAME='rspamd/bullseye-backports'
+  else
+    curl -sSfL https://rspamd.com/apt-stable/gpg.key | gpg --dearmor >/etc/apt/trusted.gpg.d/rspamd.gpg
+    local URL='[arch=amd64 signed-by=/etc/apt/trusted.gpg.d/rspamd.gpg] http://rspamd.com/apt-stable/ bullseye main'
+    echo "deb ${URL}" >"${DEB_FILE}"
+    echo "deb-src ${URL}" >>"${DEB_FILE}"
+    RSPAMD_PACKAGE_NAME='rspamd'
+  fi
 
   _log 'debug' 'Installing Rspamd'
   apt-get "${QUIET}" update
-  apt-get "${QUIET}" --no-install-recommends install rspamd redis-server
+  apt-get "${QUIET}" --no-install-recommends install "${RSPAMD_PACKAGE_NAME}" 'redis-server'
 }
 
 function _install_fail2ban

target/scripts/helpers/setup-rspamd.sh

@@ -0,0 +1,182 @@
+#!/bin/bash
+
+# Just a helper to prepend the log messages with `(Rspamd setup)` so
+# users know exactly where the message originated from.
+#
+# @param ${1} = log level
+# @param ${2} = message
+function __rspamd__log { _log "${1:-}" "(Rspamd setup) ${2:-}" ; }
+
+# Run miscellaneous checks against the current configuration so we can
+# properly handle integration into ClamAV, etc.
+#
+# This will also check whether Amavis is enabled and emit a warning as
+# we discourage users from running Amavis & Rspamd at the same time.
+function __rspamd__preflight_checks
+{
+  touch /var/lib/rspamd/stats.ucl
+
+  if [[ ${ENABLE_AMAVIS} -eq 1 ]] || [[ ${ENABLE_SPAMASSASSIN} -eq 1 ]]
+  then
+    __rspamd__log 'warn' 'Running Amavis/SA & Rspamd at the same time is discouraged'
+  fi
+
+  if [[ ${ENABLE_CLAMAV} -eq 1 ]]
+  then
+    __rspamd__log 'debug' 'Enabling ClamAV integration'
+    sedfile -i -E 's|^(enabled).*|\1 = true;|g' /etc/rspamd/local.d/antivirus.conf
+    # RSpamd uses ClamAV's UNIX socket, and to be able to read it, it must be in the same group
+    usermod -a -G clamav _rspamd
+  else
+    __rspamd__log 'debug' 'Rspamd will not use ClamAV (which has not been enabled)'
+  fi
+}
+
+# Adjust Postfix's configuration files. Append Rspamd at the end of
+# `smtpd_milters` in `main.cf`.
+function __rspamd__adjust_postfix_configuration
+{
+  # shellcheck disable=SC2016
+  sed -i -E 's|^(smtpd_milters =.*)|\1 inet:localhost:11332|g' /etc/postfix/main.cf
+}
+
+# Helper for explicitly enabling or disabling a specific module.
+#
+# @param ${1} = module name
+# @param ${2} = `true` when you want to enable the module (default),
+#               `false` when you want to disable the module [OPTIONAL]
+# @param ${3} = whether to use `local` (default) or `override` [OPTIONAL]
+function __rspamd__enable_disable_module
+{
+  local MODULE=${1:?Module name must be provided}
+  local ENABLE_MODULE=${2:-true}
+  local LOCAL_OR_OVERRIDE=${3:-local}
+  local MESSAGE='Enabling'
+
+  if [[ ! ${ENABLE_MODULE} =~ ^(true|false)$ ]]
+  then
+    __rspamd__log 'warn' "__rspamd__enable_disable_module got non-boolean argument for deciding whether module should be enabled or not"
+    return 1
+  fi
+
+  [[ ${ENABLE_MODULE} == true ]] || MESSAGE='Disabling'
+
+  __rspamd__log 'trace' "${MESSAGE} module '${MODULE}'"
+  cat >"/etc/rspamd/${LOCAL_OR_OVERRIDE}.d/${MODULE}.conf" << EOF
+# documentation: https://rspamd.com/doc/modules/${MODULE}.html
+
+enabled = ${ENABLE_MODULE};
+
+EOF
+}
+
+# Disables certain modules by default. This can be overwritten by the user later.
+# We disable the modules listed in `DISABLE_MODULES` as we believe these modules
+# are not commonly used and the average user does not need them. As a consequence,
+# disabling them saves resources.
+function __rspamd__disable_default_modules
+{
+  local DISABLE_MODULES=(
+    clickhouse
+    elastic
+    greylist
+    neural
+    reputation
+    spamassassin
+    url_redirector
+    metric_exporter
+  )
+
+  for MODULE in "${DISABLE_MODULES[@]}"
+  do
+    __rspamd__enable_disable_module "${MODULE}" 'false'
+  done
+}
+
+# Parses `RSPAMD_CUSTOM_COMMANDS_FILE` and executed the directives given by the file.
+# To get a detailed explanation of the commands and how the file works, visit
+# https://docker-mailserver.github.io/docker-mailserver/edge/config/security/rspamd/#with-the-help-of-a-custom-file
+function __rspamd__handle_modules_configuration
+{
+  # Adds an option with a corresponding value to a module, or, in case the option
+  # is already present, overwrites it.
+  #
+  # @param ${1} = file name in /etc/rspamd/override.d/
+  # @param ${2} = module name as it should appear in the log
+  # @patam ${3} = option name in the module
+  # @param ${4} = value of the option
+  #
+  # ## Note
+  #
+  # While this function is currently bound to the scope of `__rspamd__handle_modules_configuration`,
+  # it is written in a versatile way (taking 4 arguments instead of assuming `ARGUMENT2` / `ARGUMENT3`
+  # are set) so that it may be used elsewhere if needed.
+  function __add_or_replace
+  {
+    local MODULE_FILE=${1:?Module file name must be provided}
+    local MODULE_LOG_NAME=${2:?Module log name must be provided}
+    local OPTION=${3:?Option name must be provided}
+    local VALUE=${4:?Value belonging to an option must be provided}
+    # remove possible whitespace at the end (e.g., in case ${ARGUMENT3} is empty)
+    VALUE=${VALUE% }
+
+    local FILE="/etc/rspamd/override.d/${MODULE_FILE}"
+    [[ -f ${FILE} ]] || touch "${FILE}"
+
+    if grep -q -E "${OPTION}.*=.*" "${FILE}"
+    then
+      __rspamd__log 'trace' "Overwriting option '${OPTION}' with value '${VALUE}' for ${MODULE_LOG_NAME}"
+      sed -i -E "s|([[:space:]]*${OPTION}).*|\1 = ${VALUE};|g" "${FILE}"
+    else
+      __rspamd__log 'trace' "Setting option '${OPTION}' for ${MODULE_LOG_NAME} to '${VALUE}'"
+      echo "${OPTION} = ${VALUE};" >>"${FILE}"
+    fi
+  }
+
+  local RSPAMD_CUSTOM_COMMANDS_FILE='/tmp/docker-mailserver/rspamd-modules.conf'
+  if [[ -f "${RSPAMD_CUSTOM_COMMANDS_FILE}" ]]
+  then
+    __rspamd__log 'debug' "Found file 'rspamd-modules.conf' - parsing and applying it"
+
+    while read -r COMMAND ARGUMENT1 ARGUMENT2 ARGUMENT3
+    do
+      case "${COMMAND}" in
+
+        ('disable-module')
+          __rspamd__enable_disable_module "${ARGUMENT1}" 'false' 'override'
+          ;;
+
+        ('enable-module')
+          __rspamd__enable_disable_module "${ARGUMENT1}" 'true' 'override'
+          ;;
+
+        ('set-option-for-module')
+          __add_or_replace "${ARGUMENT1}.conf" "module '${ARGUMENT1}'" "${ARGUMENT2}" "${ARGUMENT3}"
+          ;;
+
+        ('set-option-for-controller')
+          __add_or_replace 'worker-controller.inc' 'controller worker' "${ARGUMENT1}" "${ARGUMENT2} ${ARGUMENT3}"
+          ;;
+
+        ('set-option-for-proxy')
+          __add_or_replace 'worker-proxy.inc' 'proxy worker' "${ARGUMENT1}" "${ARGUMENT2} ${ARGUMENT3}"
+          ;;
+
+        ('set-common-option')
+          __add_or_replace 'options.inc' 'common options' "${ARGUMENT1}" "${ARGUMENT2} ${ARGUMENT3}"
+          ;;
+
+        ('add-line')
+          __rspamd__log 'trace' "Adding complete line to '${ARGUMENT1}'"
+          echo "${ARGUMENT2} ${ARGUMENT3:-}" >>"/etc/rspamd/override.d/${ARGUMENT1}"
+          ;;
+
+        (*)
+          __rspamd__log 'warn' "Command '${COMMAND}' is invalid"
+          continue
+          ;;
+
+      esac
+    done < <(_get_valid_lines_from_file "${RSPAMD_CUSTOM_COMMANDS_FILE}")
+  fi
+}

target/scripts/startup/setup-stack.sh

@@ -99,48 +99,22 @@ function _setup_amavis
 
 function _setup_rspamd
 {
-  _log 'warn' 'Rspamd support is under active development, expect breaking changes at any time'
+  if [[ -f /usr/local/bin/helpers/setup-rspamd.sh ]]
-
-  if [[ ${ENABLE_AMAVIS} -eq 1 ]] || [[ ${ENABLE_SPAMASSASSIN} -eq 1 ]]
   then
-    _log 'warn' 'Running rspamd at the same time as Amavis or SpamAssassin is discouraged'
+    # ShellCheck sources this from two different files, hence we discard the check of external sources.
-  fi
+    # shellcheck source=/dev/null
-
+    source /usr/local/bin/helpers/setup-rspamd.sh
-  if [[ ${ENABLE_CLAMAV} -eq 1 ]]
-  then
-    _log 'debug' 'Rspamd will use ClamAV'
-    sedfile -i -E 's|^(enabled).*|\1 = true;|g' /etc/rspamd/local.d/antivirus.conf
-    # RSpamd uses ClamAV's UNIX socket, and to be able to read it, it must be in the same group
-    usermod -a -G clamav _rspamd
   else
-    _log 'debug' 'Rspamd will not use ClamAV (which has not been enabled)'
+    _shutdown 'error' '(Rspamd setup) Helper functions required for setup were not found'
   fi
 
-  declare -a DISABLE_MODULES
+  _log 'warn' 'Rspamd integration is work in progress - expect (breaking) changes at any time'
-  DISABLE_MODULES=(
+  _log 'debug' 'Enabling Rspamd'
-    clickhouse
-    elastic
-    greylist
-    neural
-    reputation
-    spamassassin
-    url_redirector
-    metric_exporter
-  )
 
-  for MODULE in "${DISABLE_MODULES[@]}"
+  __rspamd__preflight_checks
-  do
+  __rspamd__adjust_postfix_configuration
-    cat >"/etc/rspamd/local.d/${MODULE}.conf" << EOF
+  __rspamd__disable_default_modules
-# documentation: https://rspamd.com/doc/modules/${MODULE}.html
+  __rspamd__handle_modules_configuration
-
-enabled = false;
-
-EOF
-  done
-
-  # shellcheck disable=SC2016
-  sed -i -E 's|^(smtpd_milters =.*)|\1 inet:localhost:11332|g' /etc/postfix/main.cf
-  touch /var/lib/rspamd/stats.ucl
 }
 
 function _setup_dmarc_hostname
@@ -162,9 +136,7 @@ function _setup_postfix_hostname
 function _setup_dovecot_hostname
 {
   _log 'debug' 'Applying hostname to Dovecot'
-  sed -i \
+  sed -i "s|^#hostname =.*$|hostname = '${HOSTNAME}'|g" /etc/dovecot/conf.d/15-lda.conf
-    "s|^#hostname =.*$|hostname = '${HOSTNAME}'|g" \
-    /etc/dovecot/conf.d/15-lda.conf
 }
 
 function _setup_dovecot
@@ -632,6 +604,12 @@ function _setup_SRS
   postconf 'recipient_canonical_classes = envelope_recipient,header_recipient'
 }
 
+# Set up OpenDKIM & OpenDMARC.
+#
+# ## Attention
+#
+# The OpenDKIM milter must come before the OpenDMARC milter in Postfix's#
+# `smtpd_milters` milters options.
 function _setup_dkim_dmarc
 {
   if [[ ${ENABLE_OPENDKIM} -eq 1 ]]
@@ -639,45 +617,45 @@ function _setup_dkim_dmarc
     _log 'debug' 'Setting up DKIM'
 
     mkdir -p /etc/opendkim/keys/
-    touch /etc/opendkim/SigningTable
+    touch /etc/opendkim/{SigningTable,TrustedHosts,KeyTable}
-    touch /etc/opendkim/TrustedHosts
 
     _log 'trace' "Adding OpenDKIM to Postfix's milters"
+    postconf 'dkim_milter = inet:localhost:8891'
     # shellcheck disable=SC2016
-    sed -i -E 's|^(smtpd_milters =.*)|\1 \$dkim_milter|g' /etc/postfix/main.cf
+    sed -i -E                                            \
-    # shellcheck disable=SC2016
+      -e 's|^(smtpd_milters =.*)|\1 \$dkim_milter|g'     \
-    sed -i -E 's|^(non_smtpd_milters =.*)|\1 \$dkim_milter|g' /etc/postfix/main.cf
+      -e 's|^(non_smtpd_milters =.*)|\1 \$dkim_milter|g' \
+      /etc/postfix/main.cf
 
     # check if any keys are available
-    if [[ -e "/tmp/docker-mailserver/opendkim/KeyTable" ]]
+    if [[ -e /tmp/docker-mailserver/opendkim/KeyTable ]]
     then
       cp -a /tmp/docker-mailserver/opendkim/* /etc/opendkim/
-
+      _log 'trace' "DKIM keys added for: $(find /etc/opendkim/keys/ -maxdepth 1 -type f -printf '%f ')"
-      local KEYS
-      KEYS=$(find /etc/opendkim/keys/ -type f -maxdepth 1)
-      _log 'trace' "DKIM keys added for: ${KEYS}"
-      _log 'trace' "Changing permissions on '/etc/opendkim'"
-
       chown -R opendkim:opendkim /etc/opendkim/
       chmod -R 0700 /etc/opendkim/keys/
     else
-      _log 'debug' 'No DKIM key(s) provided - check the documentation on how to get your keys'
+      _log 'debug' 'OpenDKIM enabled but no DKIM key(s) provided'
-      [[ ! -f /etc/opendkim/KeyTable ]] && touch /etc/opendkim/KeyTable
     fi
 
     # setup nameservers parameter from /etc/resolv.conf if not defined
-    if ! grep '^Nameservers' /etc/opendkim.conf
+    if ! grep -q '^Nameservers' /etc/opendkim.conf
     then
-      echo "Nameservers $(grep '^nameserver' /etc/resolv.conf | awk -F " " '{print $2}' | paste -sd ',' -)" >>/etc/opendkim.conf
+      local NAMESERVER_IPS
-
+      NAMESERVER_IPS=$(grep '^nameserver' /etc/resolv.conf | awk -F " " '{print $2}' | paste -sd ',' -)
+      echo "Nameservers ${NAMESERVER_IPS}" >>/etc/opendkim.conf
       _log 'trace' "Nameservers added to '/etc/opendkim.conf'"
     fi
   fi
 
   if [[ ${ENABLE_OPENDMARC} -eq 1 ]]
   then
+    # TODO when disabling SPF is possible, add a check whether DKIM and SPF is disabled
+    #      for DMARC to work, you should have at least one enabled
+    #      (see RFC 7489 https://www.rfc-editor.org/rfc/rfc7489#page-24)
     _log 'trace' "Adding OpenDMARC to Postfix's milters"
-
+    postconf 'dmarc_milter = inet:localhost:8893'
+    # Make sure to append the OpenDMARC milter _after_ the OpenDKIM milter!
     # shellcheck disable=SC2016
     sed -i -E 's|^(smtpd_milters =.*)|\1 \$dmarc_milter|g' /etc/postfix/main.cf
   fi
@@ -754,8 +732,8 @@ function _setup_docker_permit
     _log 'trace' "Adding ${NETWORK_TYPE} (${NETWORK}) to Postfix 'main.cf:mynetworks'"
     _adjust_mtime_for_postfix_maincf
     postconf "$(postconf | grep '^mynetworks =') ${NETWORK}"
-    echo "${NETWORK}" >> /etc/opendmarc/ignore.hosts
+    [[ ${ENABLE_OPENDMARC} -eq 1 ]] && echo "${NETWORK}" >>/etc/opendmarc/ignore.hosts
-    echo "${NETWORK}" >> /etc/opendkim/TrustedHosts
+    [[ ${ENABLE_OPENDKIM} -eq 1 ]] && echo "${NETWORK}" >>/etc/opendkim/TrustedHosts
   }
 
   case "${PERMIT_DOCKER}" in

test/config/rspamd-modules.conf

@@ -0,0 +1,42 @@
+# This is suboptimal, but not strictly required either: we do not
+# need the DKIM signing & RBL functionality (which is most likely
+# not set up anyway or it fails (in case of querying DNSBLs with
+# public resolvers)). Hence, we disable it. It is suboptimal
+# since we are testing the functionality of whether disabling
+# a module works at the same time..
+#
+# When testing on ARM64, this is required at the moment due to
+# strange behavior in Rspamd v3.2 (segmentation faults) when DKIM
+# keys are not provided (notes were added to the docs).
+disable-module dkim_signing
+disable-module rbl
+
+# check whether disabling a module works and whether it
+# really overwrites options that came before
+set-option-for-module testmodule1 someoption somevalue
+disable-module testmodule1
+
+# enabling a module and setting some options for it
+enable-module testmodule2
+set-option-for-module testmodule2 someoption somevalue
+set-option-for-module testmodule2 anotheroption whatAvaLue
+
+# overwriting an option (probably unwanted, we emit a warning)
+set-option-for-module testmodule3 someoption somevalue1
+set-option-for-module testmodule3 someoption somevalue2
+
+# check whether adding a line works even with special characters in it
+add-line testmodule4.something some very long line with "weird $charact"ers
+add-line testmodule4.something and! ano. ther &line
+add-line testmodule4.something # some comment
+
+# check whether spaces in front are handles fine, for nested options
+set-option-for-module testmodule_complicated anOption anotherValue
+
+# check whether controller and proxy options are set/overwritten correctly
+set-option-for-controller someOption someValue42
+set-option-for-proxy abcdefg71 RaNDom
+set-option-for-proxy abcdefg71 RAAAANdooM
+
+# check whether setting basic options works (and whether spaces in values work)
+set-common-option OhMy "PraiseBeLinters !"

test/helper/sending.bash

@@ -54,15 +54,15 @@ function _send_mail_and_get_id() {
   _send_email "${TEMPLATE_FILE}"
   _wait_for_empty_mail_queue_in_container
 
+  # The unique ID Postfix (and other services) use may be different in length
+  # on different systems (e.g. amd64 (11) vs aarch64 (10)). Hence, we use a
+  # range to safely capture it.
   MAIL_ID=$(_exec_in_container tac /var/log/mail.log              \
     | grep -E -m 1 'postfix/smtpd.*: [A-Z0-9]+: client=localhost' \
-    | grep -E -o '[A-Z0-9]{11}')
+    | grep -E -o '[A-Z0-9]{9,12}' || true)
 
-  if [[ -z ${MAIL_ID} ]]
+  run bash -c "-z ${MAIL_ID}"
-  then
+  assert_success 'Could not obtain mail ID - aborting!'
-    echo 'Could not obtain mail ID - aborting!' >&2
-    exit 1
-  fi
 
   echo "${MAIL_ID}"
 }

test/tests/parallel/set1/spam_virus/rspamd.bats

@@ -25,6 +25,7 @@ function setup_file() {
 
   _wait_for_service redis
   _wait_for_service rspamd
+  _wait_for_service clamav
   _wait_for_service postfix
   _wait_for_smtp_port_in_container
 
@@ -33,16 +34,19 @@ function setup_file() {
   export MAIL_ID1=$(_send_mail_and_get_id 'existing-user1')
   export MAIL_ID2=$(_send_mail_and_get_id 'rspamd-spam')
   export MAIL_ID3=$(_send_mail_and_get_id 'rspamd-virus')
+
+  # add a nested option to a module
+  _exec_in_container_bash "echo -e 'complicated {\n    anOption = someValue;\n}' >/etc/rspamd/override.d/testmodule_complicated.conf"
 }
 
 function teardown_file() { _default_teardown ; }
 
 @test "Postfix's main.cf was adjusted" {
-  _run_in_container grep -q 'smtpd_milters = inet:localhost:11332' /etc/postfix/main.cf
+  _run_in_container grep -F 'smtpd_milters = inet:localhost:11332' /etc/postfix/main.cf
   assert_success
 }
 
-@test "logs exist and contains proper content" {
+@test 'logs exist and contains proper content' {
   _service_log_should_contain_string 'rspamd' 'rspamd .* is loading configuration'
   _service_log_should_contain_string 'rspamd' 'lua module clickhouse is disabled in the configuration'
   _service_log_should_contain_string 'rspamd' 'lua module elastic is disabled in the configuration'
@@ -53,14 +57,14 @@ function teardown_file() { _default_teardown ; }
   _service_log_should_contain_string 'rspamd' 'lua module metric_exporter is disabled in the configuration'
 }
 
-@test "normal mail passes fine" {
+@test 'normal mail passes fine' {
   _service_log_should_contain_string 'rspamd' 'F \(no action\)'
 
   _print_mail_log_for_id "${MAIL_ID1}"
   assert_output --partial "stored mail into mailbox 'INBOX'"
 }
 
-@test "detects and rejects spam" {
+@test 'detects and rejects spam' {
   _service_log_should_contain_string 'rspamd' 'S \(reject\)'
   _service_log_should_contain_string 'rspamd' 'reject "Gtube pattern"'
 
@@ -69,7 +73,7 @@ function teardown_file() { _default_teardown ; }
   assert_output --partial '5.7.1 Gtube pattern'
 }
 
-@test "detects and rejects virus" {
+@test 'detects and rejects virus' {
   _service_log_should_contain_string 'rspamd' 'T \(reject\)'
   _service_log_should_contain_string 'rspamd' 'reject "ClamAV FOUND VIRUS "Eicar-Signature"'
 
@@ -78,3 +82,74 @@ function teardown_file() { _default_teardown ; }
   assert_output --partial '5.7.1 ClamAV FOUND VIRUS "Eicar-Signature"'
   refute_output --partial "stored mail into mailbox 'INBOX'"
 }
+
+@test 'custom commands work correctly' {
+  # check `testmodule1` which should be disabled
+  local MODULE_PATH='/etc/rspamd/override.d/testmodule1.conf'
+  _run_in_container_bash "[[ -f ${MODULE_PATH} ]]"
+  assert_success
+  _run_in_container grep -F '# documentation: https://rspamd.com/doc/modules/testmodule1.html' "${MODULE_PATH}"
+  assert_success
+  _run_in_container grep -F 'enabled = false;' "${MODULE_PATH}"
+  assert_success
+  _run_in_container grep -F 'someoption = somevalue;' "${MODULE_PATH}"
+  assert_failure
+
+  # check `testmodule2` which should be enabled and it should have extra options set
+  MODULE_PATH='/etc/rspamd/override.d/testmodule2.conf'
+  _run_in_container_bash "[[ -f ${MODULE_PATH} ]]"
+  assert_success
+  _run_in_container grep -F '# documentation: https://rspamd.com/doc/modules/testmodule2.html' "${MODULE_PATH}"
+  assert_success
+  _run_in_container grep -F 'enabled = true;' "${MODULE_PATH}"
+  assert_success
+  _run_in_container grep -F 'someoption = somevalue;' "${MODULE_PATH}"
+  assert_success
+  _run_in_container grep -F 'anotheroption = whatAvaLue;' "${MODULE_PATH}"
+  assert_success
+
+  # check whether writing the same option twice overwrites the first value in `testmodule3`
+  MODULE_PATH='/etc/rspamd/override.d/testmodule3.conf'
+  _run_in_container grep -F 'someoption = somevalue;' "${MODULE_PATH}"
+  assert_failure
+  _run_in_container grep -F 'someoption = somevalue2;' "${MODULE_PATH}"
+  assert_success
+
+  # check whether adding a single line writes the line properly in `testmodule4.something`
+  MODULE_PATH='/etc/rspamd/override.d/testmodule4.something'
+  _run_in_container_bash "[[ -f ${MODULE_PATH} ]]"
+  assert_success
+  _run_in_container grep -F 'some very long line with "weird $charact"ers' "${MODULE_PATH}"
+  assert_success
+  _run_in_container grep -F 'and! ano. ther &line' "${MODULE_PATH}"
+  assert_success
+  _run_in_container grep -F '# some comment' "${MODULE_PATH}"
+  assert_success
+
+  # check whether spaces in front of options are handles properly in `testmodule_complicated`
+  MODULE_PATH='/etc/rspamd/override.d/testmodule_complicated.conf'
+  _run_in_container_bash "[[ -f ${MODULE_PATH} ]]"
+  assert_success
+  _run_in_container grep -F '    anOption = anotherValue;' "${MODULE_PATH}"
+
+  # check whether controller option was written properly
+  MODULE_PATH='/etc/rspamd/override.d/worker-controller.inc'
+  _run_in_container_bash "[[ -f ${MODULE_PATH} ]]"
+  assert_success
+  _run_in_container grep -F 'someOption = someValue42;' "${MODULE_PATH}"
+  assert_success
+
+  # check whether controller option was written properly
+  MODULE_PATH='/etc/rspamd/override.d/worker-proxy.inc'
+  _run_in_container_bash "[[ -f ${MODULE_PATH} ]]"
+  assert_success
+  _run_in_container grep -F 'abcdefg71 = RAAAANdooM;' "${MODULE_PATH}"
+  assert_success
+
+  # check whether basic options are written properly
+  MODULE_PATH='/etc/rspamd/override.d/options.inc'
+  _run_in_container_bash "[[ -f ${MODULE_PATH} ]]"
+  assert_success
+  _run_in_container grep -F 'OhMy = "PraiseBeLinters !";' "${MODULE_PATH}"
+  assert_success
+}