* chore: Split vhost helper method and use filepath vars
- Helpers `accounts.sh` and `aliases.sh` can move their vhost code into this helper.
- They share duplicate code with `bin/open-dkim` which will also leverage this vhost helper going forward.
* chore: Sync vhost generation logic into helper
- Chunky commit, but mostly copy/paste of logic into a common method.
- `bin/open-dkim` additionally wrapped relevant logic in a function call and revised inline docs.
* chore: Include LDAP vhost support
- Revises notes for LDAP vhost support.
- This now ensures LDAP users get vhost rebuilt to match the startup script for when change detection support is enabled.
- `bin/open-dkim` will additionally be able to support the default `DOMAINNAME` var (set via `helpers/dns.sh`) for LDAP users instead of requiring them to provide one explicitly.
* chore(`bin/open-dkim`): Ensure `DOMAINNAME` is properly set
- This will ensure LDAP users insert the same `DOMAINNAME` value as used during container startup.
- The container itself should panic at startup (during `helpers/dns.sh`) if this isn't configured correctly already, thus it should not introduce any breaking change to users of this utility?
* chore: Set the 2nd value as blank `_`
Line is split by a delimiter such as white-space (or via IFS: `|`), the blank `_` var is to indicate we're not interested in that value, but still leverage how `read -r` works, instead of splitting the var ourselves first thing.
* chore: Remove shellcheck disable lines
No longer applicable with the switch to `_`
* chore(`aliases.sh`): Filepath to local var `DATABASE_VIRTUAL`
* chore(`accounts.sh`): Filepath to local var `DATABASE_ACCOUNTS`
* chore(`accounts.sh`): Filepath to local var `DATABASE_VIRTUAL`
* chore(`accounts.sh`): Filepath to local var `DATABASE_DOVECOT_MASTERS`
* chore(`bin/open-dkim`): Filepaths to local vars (accounts,virtual,vhost)
* chore(`relay.sh`): Filepath to local var `DATABASE_SASL_PASSWD`
* chore: Rename method
Prior PR feedback suggested a better helper method name.
* chore: Normalize filtering config lines as input for iterating
* chore: Remove `_is_comment` helper method
No longer serving a purpose with more appropriate filter method for pre-processing the entire config file.
* fix(listmailuser): Don't parse comments
Avoids passing comments to `dovecot_quota_to_hr()` which fails to handle it and would throws errors.
* chore: Move config filter method to `helpers/utils.sh`
* chore: Extract change-detection method to it's own helper
This doesn't really belong in `helpers/ssl.sh`. Moving to it's own helper script.
* chore: Co-locate related change-detection method from container startup
It seems relevant to migrate the related support during startup for the change detection feature into this helper.
I opted to move the call from `start-mailserver.sh` into the `_setup` call at the end for a more explicit/visible location.
* chore: Move `CHKSUM_FILE` into `helpers/change-detection.sh`
It belongs there, not in `helpers/index.sh`.
* chore: Revise inline documentation
* tests(fix): Ensure correct functionality
Presently `test/test_helper.bats` is using it's own `CHKSUM_FILE` instead of sourcing the var for the filepath.
`test_helper/common.bash` was calling a method to check for changes, but this helper may not correctly detect letsencrypt related changes as these are not ENV rely on, but global vars handled by `helpers/dns.sh`, so that should be run first like it is for `check-for-changes.sh`.
* tests(chore): Use `CHKSUM_FILE` var from helper
* chore: `addmailuser` should use `CHKSUM_FILE` var
* chore: Update `check-for-changes.sh` log message with correct path
Dovecot master accounts can now be configured in DMS via `setup.sh`.
A master account is useful for administration purposes, or to perform mailbox backups of every user account over IMAP.
Upstream Docs: https://doc.dovecot.org/configuration_manual/authentication/master_users/
Co-authored-by: Casper <casperklein@users.noreply.github.com>
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
* chore: Remove `DATABASE` fallback ENV
This was introduced without any mention or need, thus removing until a real use-case requires it.
* chore: Remove `USER_DATABASE` fallback ENV
Likewise, nothing requires this to be customizable.
* chore: Consistently use single quote strings
* first adjustments to use Fail2Ban with nftables
* replace `iptables` -> `nftables` and adjust tests
nftables lists IPs a bit differently , so the order was adjusted for the
tests to be more flexible.
* line correction in mailserver.env
* change from `.conf` -> `.local` and remove redundant config
* revert HEREDOC to `echo`
Co-authored-by: Casper <casperklein@users.noreply.github.com>
* refactored `daemon-stack.sh`
A new method was introduced to uniformaly start daemons and log output
accordingly. The methods for daemon start were renamed (plural ->
singular), therefore the adjustments in `start-mailserver.sh`.
* cleaned Fetchmail setup from `daemon-stack.sh`
Not sure why, but the Fetchmail setup was somehow happening in
`daemon-stack.sh` - this is not supposed to be the case. I relocated the
setup into `setup-stack.sh`, where it belong.
* delete old, unnecessary script in `target/bin/`
These are unused leftovers from the last commit, that relocated the
setup of Fetchmail into `setup.stack.sh`.
* corrected changedetector function name
* Apply suggestions from code review
* adjusted `debug-fetchmail` script
It is absolutely fine to source `setup-stack.sh` because sourcing the
script does not execute a single function (by desing of the script).
This way, we retain functionality.
* praise be ShellCheck
* added `log.sh` to `debug-fetchmail` as a dependency
* final cleanup
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
Co-authored-by: Casper <casperklein@users.noreply.github.com>
* refactored scripts located under `target/bin/`
The scripts under `target/bin/` now use the new log and I replaced some
`""` with `''` on the way. The functionality stays the same, this mostly
style and log.
* corrected fail2ban (script and tests)
* corrected OpenDKIM log output in tests
* reverted (some) changes to `sedfile`
Moreover, a few messages for BATS were streamlined and a regression in
the linting script reverted.
* apple PR feedback
* improve log output from `fail2ban` script
The new output has a single, clear message with the '[ ERROR ] '
prefix, and then output that explains the error afterwards. This is
coherent with the logging style which should be used while providing
more information than just a single line about IPTables not functioning.
* simplified `setquota` script
* consistently named the `__usage` function
Before, scripts located under `target/bin/` were using `usage` or
`__usage`. Now, they're using `__usage` as they should.
* improved `sedfile`
With `sedfile`, we cannot use the helper functions in a nice way because
it is used early in the Dockerfile at a stage where the helper scripts
are not yet copied. The script has been adjusted to be canonical with
all the other scripts under `target/bin/`.
* fixed tests
* removed `__usage` from places where it does not belong
`__usage` is to be used on wrong user input, not on other failures as
well. This was fixed in `delquota` and `setquota`.
* apply PR review feedback
* added new `_log` function
With `_log`, the `_notify` method wa rendered obsolete. `_notify` was
not completely removed due to test failures in `check-for-changes.sh`.
The new `_log` function properly uses log levels such as `trace`,
`debug`, `info`, `warn` and `error`. It provides a cleaner solution
and renders `DMS_DEBUG` obsolete too (as only `_notify` depends on it).
* converted all helper script to new `_log` function
* converted all startup stacks to new `log` function
* `start-mailserver.sh` now uses new `_log` function
* final test and misc small script adjustments
* updated documentation
Split into scoped commits with messages if further details are needed, view those via the associated PR :)
**Commit Summary:**
**`check-for-changes.sh`**
- Prevent `SSL_DOMAIN` silently skipping when value has wildcard prefix `*.` (_at least this was known as a bugfix when originally committed in linked PR_).
- Improved inlined docs for maintainers.
- Additional logging for debugging.
**`helper-functions.sh:_extract_certs_from_acme`**:
- Fail if the input arg (_`$CERT_DOMAIN`, aka the FQDN_) provided for extraction is empty.
- Use `$CERT_DOMAIN` in place of `$HOSTNAME` and `$1` for a consistent value (_previously could mismatch, eg with `SSL_DOMAIN` defined_).
- The conditional is now only for handling extraction failure (_key or cert value is missing from extraction_).
- Log an actual warning or success (debug) based on outcome.
- Don't use `SSL_DOMAIN` with wildcard value for the `mkdir` letsencrypt directory name (_wildcard prefix `*.` is first stripped instead_).
**`acme_extract`** (_new python utility for `acme.json` handling_):
- Extracted out into a python script that can be treated as a utility in the `$PATH` like other helper scripts. It can now be used and optionally tested directly instead of via `helper-functions.sh`.
-Made compatible with Python 3, as Python 2 is EOL and no longer in newer versions of Debian.
"Brief" summary/overview of changes. See the PR discussion or individual commits from the PR for more details.
---
Only applies to the `docs/content/**` content (_and `setup` command_). `target/` and `test/` can be normalized at a later date.
* Normalize to `example.com`
- Domains normalized to `example.com`: `mywebserver.com`, `myserver.tld`, `domain.com`, `domain.tld`, `mydomain.net`, `my-domain.tld`, `my-domain.com`, `example.org`, `whoami.com`.
- Alternative domains normalized to `not-example.com`: `otherdomain.com`, `otherdomain.tld`, `domain2.tld`, `mybackupmx.com`, `whoareyou.org`.
- Email addresses normalized to `admin@example.com` (in `ssl.md`): `foo@bar.com`, `yourcurrentemail@gmail.com`, `email@email.com`, `admin@domain.tld`.
- Email addresses normalized to `external-account@gmail.com`: `bill@gates321boom.com`, `external@gmail.com`, `myemail@gmail.com`, `real-email-address@external-domain.com`.
- **`faq.md`:** A FAQ entry title with `sample.domain.com` changed to `subdomain.example.com`.
- **`mail-fetchmail.md`:** Config examples with FQDNs for `imap`/`pop3` used `example.com` domain for a third-party, changed to `gmail.com` as more familiar third-party/external MTA.
* Normalize config volume path
- Normalizing local config path references to `./docker-data/dms/config/`: `./config/`, `config/`, \``config`\`, `/etc/` (_volume mount src path prefix_).
- Normalize DMS volume paths to `docker-data/dms/mail-{data,state,log}`: `./mail`, `./mail-state` `./data/mail`, `./data/state`, `./data/logs`, `./data/maildata`, `./data/mailstate`, `./data/maillogs`, (_dropped/converted data volumes: `maildata`, `mailstate`_).
- Other docker images also adopt the `docker-data/{service name}/` prefix.
* `ssl.md` - Use `dms/custom-certs` where appropriate.
* Apply normalizations to README and example `docker-compose.yml`
---
Common terms, sometimes interchangeably used or now invalid depending on context: `mail`, `mail container`, `mail server`, `mail-server`, `mailserver`,`docker-mailserver`, `Docker Mailserver`.
Rough transformations applied to most matches (_conditionally, depending on context_):
- 'Docker Mailserver' => '`docker-mailserver`'
- 'mail container' => '`docker-mailserver`' (_optionally retaining ' container'_)
- 'mail server' => 'mail-server' / '`docker-mailserver`'
- 'mail-server' => '`docker-mailserver`'
- 'mailserver' => 'mail-server' / '`docker-mailserver`'
Additionally I checked `docker run` (_plus `exec`, `logs`, etc, sub-commands_) and `docker-compose` commands. Often finding usage of `mail` instead of the expected `mailserver`
Additionally changes `mailserver` hostname in k8s to `mail` to align with other non-k8s examples.
---
* drive-by revisions
Mostly minor revisions or improvements to docs that aren't related to normalization effort.
* changed the locking function to better support multiple servers running at once and sharing the same config
* helper function testing now runs inside of container
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
Decoupling setup process from `setup.sh` script by introducing a setup script _inside_ the container that coordinates the setup process.
**This is not a breaking change**. This way, we do not have to keep track of versions of `setup.sh`.
This change brings the additional benefit for Kubernetes users to be able to make use of `setup` now, without the need for `setup.sh`.
---
* move setup process into container; setup.sh versioning not needed anymore
* add tilde functionality to docs
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
Co-authored-by: Casper <casperklein@users.noreply.github.com>
