Below commit messages are roughly equivalent to what is listed on the PR. The PR provides additional linked resources for reference to support commit message statements.
---
* docs: Add CT log warning
- Added a warning to make users aware that using a public CA like _Let's Encrypt_ will publicly log information that may be somewhat sensitive, or undesirable to have historic records made public which cannot be redacted.
* docs: Revise the manual `certbot` guide
- The `letsencrypt` repo that was linked early in this guide now redirects to the [Certbot repo](https://github.com/certbot/certbot).
- More explicit volume mount instruction for CertBot; the local location was a tad vague.
- Better clarified `/etc/letsencrypt/live` contents structure, as well as FQDN info. Removed the misleading `fqdn:` from `docker-compose.yml` example snippet.
* docs: Revise certbot with Docker guide
- General rewrite of the Docker Certbot section with additional tips (_renewals with automation, and using a alternative CA_).
- Generalized tone and paths in content.
- Update volume mount paths to be consistent with recent normalization effort.
- Moved some instructions into inline-comments for script examples instead.
* docs: Revise Docker with `nginx-proxy` and `acme-companion`
- Break apart into individual steps, indenting content into the step as appropriate.
- Use normalized volume paths (`docker-data/<service>/` prefix).
- `letsencrypt-nginx-proxy-companion` has _changed project name to `acme-companion`_, and _transferred to new maintainers and the `nginx-proxy` organization_. This also affects the DockerHub image references.
- `acme-companion` has _switched from using `simp_le` to `acme.sh`_ for provisioning certificates. This requires mounting an additional volume for persisting provisioner state.
- The dummy container (_webmail_) is no longer `library/nginx`, just [`nginx`](https://hub.docker.com/_/nginx). This container also doesn't appear to be required. I've verified that the ENV can be given to the `mailserver` service container directly. Retained for now.
* docs: Revise Docker Compose with `nginx-proxy` and `acme-companion`
Heavy rewrite of this section. Like the previous commit mentions, this content was outdated. It has been simplified with improved documentation and reference links.
It also looks like there was a mistake in the existing config example as it uses the regular `nginx` image instead of `nginx-proxy`.
- The bulk of the `mailserver` service has been removed, users are advised to have an existing `docker-compose.yml` config for `docker-mailserver` and update only what is relevant to integrate with the cert provisioner.
- `DEBUG` is _false_ by default.
- The `networks:` portion of the example appears to be taken from upstream, _which that has since dropped it_. While we could continue to document this, I consider it more of an advanced config detail that we don't need to touch on in our docs.
- The `htpasswd` volume is unnecessary, only relevant if using _"Basic Authentication"_ to protect access to web service endpoints. `conf.d/` is also not required by default, it can be useful for the `standalone` mode (_documented as a `tip`_). Remaining volumes have inline-comments to document their purpose.
- `volumes_from:` is _not supported in v3 Compose format_, _only v2_ and the Docker CLI. I did not want to advise v2, so I've duplicated the volumes between the two containers instead. Internally `acme-companion` would rely on `volumes_from:` to identify the `nginx-proxy` container, it _provides alternative discovery methods_, the label is outdated and refers the legacy label (_their script logic is the same_); using the ENV `NGINX_PROXY_CONTAINER` seemed most appropriate and has been added.
- Upstream `acme-companion` docs only cover support for v2 Compose format. _There is a note regarding `nginx-proxy`_ having _volumes configured in it's Dockerfile_. Providing a volume for `/etc/nginx/dhparam` is required to avoid creating anonymous volumes each run of `nginx-proxy`. I've used a named data volume here to make it stick out more, it's not desirable and upstream should fix this, then we can drop it.
- I've also opted to only demonstrate the _Two Container (Basic) setup_ that upstream documents. Previously our docs have been showing _`docker-gen` with the Three Container (Advanced) setup_, which allows for not having the Docker API socket attached as a volume to a container exposed to the web. This reduces the security a bit, and I have not mentioned that on our docs. I could caution the reader with a link to upstream about the risk, but I don't think we should maintain the `docker-gen` setup.
* docs(fix): Update anchor links
These mismatched the current section headers they were meant to link to.
* function _defunc removed
* _shutdown is better than just notify in that cases
* PANIC_TYPE 'fail-init' introduced
Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
"Brief" summary/overview of changes. See the PR discussion or individual commits from the PR for more details.
---
Only applies to the `docs/content/**` content (_and `setup` command_). `target/` and `test/` can be normalized at a later date.
* Normalize to `example.com`
- Domains normalized to `example.com`: `mywebserver.com`, `myserver.tld`, `domain.com`, `domain.tld`, `mydomain.net`, `my-domain.tld`, `my-domain.com`, `example.org`, `whoami.com`.
- Alternative domains normalized to `not-example.com`: `otherdomain.com`, `otherdomain.tld`, `domain2.tld`, `mybackupmx.com`, `whoareyou.org`.
- Email addresses normalized to `admin@example.com` (in `ssl.md`): `foo@bar.com`, `yourcurrentemail@gmail.com`, `email@email.com`, `admin@domain.tld`.
- Email addresses normalized to `external-account@gmail.com`: `bill@gates321boom.com`, `external@gmail.com`, `myemail@gmail.com`, `real-email-address@external-domain.com`.
- **`faq.md`:** A FAQ entry title with `sample.domain.com` changed to `subdomain.example.com`.
- **`mail-fetchmail.md`:** Config examples with FQDNs for `imap`/`pop3` used `example.com` domain for a third-party, changed to `gmail.com` as more familiar third-party/external MTA.
* Normalize config volume path
- Normalizing local config path references to `./docker-data/dms/config/`: `./config/`, `config/`, \``config`\`, `/etc/` (_volume mount src path prefix_).
- Normalize DMS volume paths to `docker-data/dms/mail-{data,state,log}`: `./mail`, `./mail-state` `./data/mail`, `./data/state`, `./data/logs`, `./data/maildata`, `./data/mailstate`, `./data/maillogs`, (_dropped/converted data volumes: `maildata`, `mailstate`_).
- Other docker images also adopt the `docker-data/{service name}/` prefix.
* `ssl.md` - Use `dms/custom-certs` where appropriate.
* Apply normalizations to README and example `docker-compose.yml`
---
Common terms, sometimes interchangeably used or now invalid depending on context: `mail`, `mail container`, `mail server`, `mail-server`, `mailserver`,`docker-mailserver`, `Docker Mailserver`.
Rough transformations applied to most matches (_conditionally, depending on context_):
- 'Docker Mailserver' => '`docker-mailserver`'
- 'mail container' => '`docker-mailserver`' (_optionally retaining ' container'_)
- 'mail server' => 'mail-server' / '`docker-mailserver`'
- 'mail-server' => '`docker-mailserver`'
- 'mailserver' => 'mail-server' / '`docker-mailserver`'
Additionally I checked `docker run` (_plus `exec`, `logs`, etc, sub-commands_) and `docker-compose` commands. Often finding usage of `mail` instead of the expected `mailserver`
Additionally changes `mailserver` hostname in k8s to `mail` to align with other non-k8s examples.
---
* drive-by revisions
Mostly minor revisions or improvements to docs that aren't related to normalization effort.
* fix: Spam bounced test copy/paste typo
* tests(docs): Expand inline documentation
Should assist maintainers like myself that are not yet familiar with this functionality, saving some time :)
* Refactor bounced test + Introduce initial container template
DRY'd up the test and extracted a common init pattern for other tests to adopt in future.
The test does not need to run distinct containers at once, so a common name is fine, although the `init_with_defaults()` method could be given an arg to add a suffix: `init_with_defaults "_${BATS_TEST_NUMBER}"` which could be called in `setup()` for tests that can benefit from being run in parallel.
Often it seems the containers only need the bare minimum config such as accounts provided to actually make the container happy to perform a test, so sharing a `:ro` config mount is fine, or in future this could be better addressed.
---
The test would fail if the test cases requiring smtp access ran before postfix was ready (_only a few seconds after setup scripts announce being done_). Added the wait condition for smtp, took a while to track that failure down.
Initial pass for achieving more consistency with docker-compose related configs.
* Set DMS_DEBUG to 0
* align with default docker-compose.yml
Co-authored-by: Casper <casperklein@users.noreply.github.com>
Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
Recent `sedfile` addition moved all scripts section earlier into the Dockerfile so that `sedfile` could be used within the Dockerfile.
However whenever a change is made to scripts which is most of the time for this project, building the Docker image for tests results in all layers after the scripts being invalidated, notably ClamAV, wasting storage of previous instances and increasing build time unnecessarily.
This isn't as noticeable of an issue via the CI as we don't leverage any caching at present there, but for iterating on a local branch and testing, it can be quite the drawback.
- `sedfile` is handled early in the Dockerfile still, while the scripts have been moved as far down as it made sense to.
- `chmod` was split out into it's own RUN command as again it's unnecessary for the rest of it's prior RUN command group to be invalidated.
* chore(refactor): DRY up the `_setup_ssl` method
- `/etc/postfix/ssl` was a bit misleading in usage here. As a maintainer (of my own contribution!) I was confused why only `/etc/postfix/ssl` was referenced and not `/etc/dovecot/ssl`.
- The postfix specific path is unnecessary, dovecot was referencing it via it's config, the same can be done from postfix to a generic DMS specific config location instead.
- This location is defined and created early as `/etc/dms/tls` (with var `DMS_TLS_PATH`). All usage of `/etc/postfix/ssl` has been replaced, making it easier to grok. Several `mkdir` commands related to this have been dropped as a result.
- Likewise, a related `TMP_DMS_TLS_PATH` var provides a reference to the config volume path `/tmp/docker-mailserver` which is used for conditions on presently hard-coded paths.
- Other values that benefit from being DRY have been lifted up into vars. Definitely easier to follow now and makes some further opportunities clearer to tackle in a future refactor.
- `chmod` has been updated where appropriate. Public key/cert is acceptable to have as readable by non-root users (644). The custom type with single fullchain file was not root accessible only, but should as it contains a private key.
- That said, the security benefit can be a bit moot due to source files that were copied remain present, the user would be responsible to ensure similar permissions on their source files.
- I've not touched LetsEncrypt section as I don't have time to investigate into that yet (not familiar with that portion).
---
* chore: Remove mkcert logic and dovecot cert
- No longer serving a purpose.
- Our own TLS startup script handles a variety of cert scenarios, while the dropped code was always generating a self-signed cert and persisting an unused cert regardless with `ONE_DIR=1`.
- To avoid similar issues that DH params had with doveadm validating filepath values in the SSL config, the default dummy values match postfix pointing to "snakeoil" cert. That serves the same purpose as mkcert was covering in the image.
- Bonus, no more hassle with differing mkcert target paths for users replacing our supplied Dovecot with the latest community edition.
---
* Error handling for SSL_TYPE
- Added a panic utility to exit early when SSL_TYPE conditions are misconfigured.
- Some info text had order of key/cert occurrence swapped to be consistent with key then cert.
- Some existing comments moved and rephrased.
- Additional comments added.
- `-f` test for cert files instead of `-e` (true also for directories/devices/symlinks).
- _notify messages lifted out of conditionals so that they always output when the case is hit.
- ~~Empty SSL_TYPE collapsed into catch all panic, while it's contents is now mapped to a new 'disabled' value.~~
---
* Use sedfile + improve sed expressions + update case style
- Uses sedfile when appropriate (file change intentional, not optional match/check).
- sed expressions modified to be DRY and reduce escaping via `-r` flag (acceptable if actual text content contains no `?`,`+`,`()` or `{}` characters, [otherwise they must be escaped](https://www.gnu.org/software/sed/manual/html_node/Extended-regexps.html)).
- sed captures anything matched between the parenthesis`()` and inserts it via `\1` as part of the replacement.
- case statements adopt the `(` prefix, adopting recent shell style for consistency.
---
* Refactor SSL_TYPE=disabled
- Postfix is also disabled now.
- Included heavy inline documentation reference for maintainers.
- Dropped an obsolete postfix config option 'use_tls' on the relayhost function, it was replaced by 'security_level'.
---
* I'm a friggin' sed wizard now
- The `modern` TLS_LEVEL is the default values for the configs they modify. As such, `sedfile` outputs an "Error" which isn't an actual concern, back to regular `sed`.
- I realized that multiple edits for the same file can all be done at once via `-e` (assuming other sed options are the same for each operation), and that `g` suffix is global scope for single line match, not whole file (default as sed iterates through individual lines).
- Some postfix replacements have `smtp` and `smtpd` lines, collapsed into a single `smtpd?` instead now that I know sed better.
---
* tests(fix): Tests that require SSL/TLS to pass
- SSL_TYPE=snakeoil added as temporary workaround.
- nmap tests are being dropped. These were added about 4-5 years ago, I have since made these redundant with the `testssl.sh` tests.
- Additionally the `--link` option is deprecated and IIRC these grades were a bit misleading when I initially used nmap in my own TLS cipher suite update PRs in the past.
- The removed SSL test is already handled in mail_ssl_manual.bats
ldap test:
- Replace `--link` alias option with `--network` and alias assignment.
- Parameterized some values and added the `SSL_TYPE` to resolve the starttls test failure.
privacy test:
- Also needed `SSL_TYPE` to pass the starttls test.
`tests.bats` had another starttls test for imap:
- Workaround for now is to give the main test container `SSL_TYPE=snakeoil`.
---
* Remove the expired lets-encrypt cert
This expired in March 2021. It was originally required when first added back in 2016 as LetsEncrypt was fairly new and not as broadly accepted into OS trust stores.
No longer the case today.
---
* chore: Housekeeping
Not required for this PR branch, little bit of tidying up while working on these two test files.
- privacy test copied over content when extracted from `tests.bats` that isn't relevant.
- ldap test was not as easy to identify the source of DOVECOT_TLS. Added comment to make the prefix connection to `configomat.sh` and `.ext` files more easier to find.
- Additionally converted the two localhost FQDN to vars.
---
* Default SSL_TYPE becomes `''` (aka equivalent to desired `disabled` case)
- This is to prevent other tests from failing by hitting the panic catchall case.
- More ideal would be adjusting tests to default to `disabled`, rather than treating `disabled` as an empty / unset SSL_TYPE value.
---
* Add inline documentation for `dms_panic`
- This could later be better formatted and placed into contributor docs.
Panic with kill (shutdown) not exit (errex):
- `kill 1` from `_shutdown` will send SIGTERM signal to PID 1 (init process).
- `exit 1` within the `start-mailserver.sh` init scripts context, will just exit the initialization script leaving the container running when it shouldn't.
The two previous `_shutdown` methods can benefit from using `dms_panic` wrapper instead to standardize on panic messages.
Although these two config lines have not changed since `debian:buster-slim` image, Dovecot seems to now be affected by it which results in rejecting cipher suites below TLS v1.2.
To continue supporting the `intermediate` TLS_LEVEL, we now need to relax the global config. Dovecot could alternatively be given a modified openssl config to only affect it's interaction with openssl.
Postfix is unaffected and continues to support TLS <1.2 cipher suites when configured to.
This feature was originally introduced by the PR: https://github.com/docker-mailserver/docker-mailserver/pull/1463
- Assign default DH params to use via Dockerfile build instead of copy and update at runtime.
- Parameterized service names and paths.
- Refactor postfix and dovecot dh methods to wrap shared dh logic
- I don't see any value in checking the alternative service for dh params file to copy over, so that's now dropped too.
- Another conditional check is dropped and the default fallback message for existing DH params file is no longer relevant.
- Improved the remaining `_notify` messages. Collapsing the warning into a single logged message also seemed relevant.
- There is no apparent need for special handling with `ONE_DIR=1`. Dropped it.
- Refactor DH params tests
- Combine custom and default DH param tests into single test file
- docs: Add instructions to use custom DH params
There is no official documented support for custom DH parameters. As no guarantee is provided, this is considered an internal change, not a breaking one.
* docs: Improve FAQ entry for `mail-state` folder
- Links to relevant script logic.
- Better list of services data moved to `mail-state`.
* Update docs/content/faq.md
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
* docs(fix): ONE_DIR env default is now `1`
This was missed during the `ONE_DIR` default change in https://github.com/docker-mailserver/docker-mailserver/pull/2148
* fix relative filepath
* fix: use new URI anchor
Co-authored-by: Casper <casperklein@users.noreply.github.com>
Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
* changed the locking function to better support multiple servers running at once and sharing the same config
* helper function testing now runs inside of container
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
Decoupling setup process from `setup.sh` script by introducing a setup script _inside_ the container that coordinates the setup process.
**This is not a breaking change**. This way, we do not have to keep track of versions of `setup.sh`.
This change brings the additional benefit for Kubernetes users to be able to make use of `setup` now, without the need for `setup.sh`.
---
* move setup process into container; setup.sh versioning not needed anymore
* add tilde functionality to docs
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
Co-authored-by: Casper <casperklein@users.noreply.github.com>
Centralize the collection of the HOSTNAME and DOMAINAME so that it's predictable and uniform across the various scripts (using the helper). Ensure it supports the various configurations users can have (both subdomain and without subdomain, override and no override).
---
* using _obtain_hostname_and_domainname helper + covers when not a subdomain
doc: OVERRIDE_HOSTNAME takes priority
* added tests for non-subdomain hostname + further improvements
* moved SRS DOMAINANME tests into hostname test file + Allowing DOMAINNAME ENV to override what would be automatically set
---
Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
* docker_container first, then fall back to docker_image
+ test changes to support
+ test change to wait for smtp port to fix flakey tests since https://github.com/docker-mailserver/docker-mailserver/pull/2104
* quick fix
* Update setup.sh
Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
Co-authored-by: Casper <casperklein@users.noreply.github.com>
