Commit graph

130 commits

Author SHA1 Message Date
Georg Lauterbach 03772f612a
scripts: get all policyd-spf setup in one place (#3263) 2023-04-15 00:40:42 +02:00
Georg Lauterbach 9a284150b2
Rspamd: replace reject_unknown_client_hostname with Rspamd HFILTER_HOSTNAME_UNKNOWN and make it configurable (#3248) 2023-04-11 18:51:23 +02:00
Georg Lauterbach 806d3efef9
Rspamd: add greylisting option & code refactoring (#3206) 2023-04-11 09:16:57 +02:00
Georg Lauterbach 9ee33a81b7
scripts: make policyd-spf configurable (#3246) 2023-04-11 08:52:43 +02:00
Casper f4fe5bf527
Update SA_KILL values; follow up to #3058 (#3204) 2023-03-27 01:59:43 +02:00
Casper 6fa06f4986
Fix: only chmod when there are files (#3203) 2023-03-26 20:30:34 +08:00
Georg Lauterbach e58dd1b95b
Rspamd: more features (#3159) 2023-03-18 23:32:48 +08:00
Casper 2da3e1b022
fix: SRS setup (#3158) 2023-03-12 12:10:45 +01:00
Georg Lauterbach dab70709d9
scripts: improve panic helpers (#3155) 2023-03-06 10:06:50 +01:00
Georg Lauterbach b5fc40eb7a
fix regression introduced in #3153 (#3157) 2023-03-05 22:23:25 +01:00
Casper a8f6fa6181
DRY: Replace path with variable in mail_state.sh (#3153) 2023-03-05 16:59:01 +01:00
Georg Lauterbach 4e9ffbf224
fix(Postfix): special bits for maildrop and public directory (#3149)
* fixed special bits for maildrop and public dir

After changing the group, special bits are lost, but they should be set for the directories `/var/spool/postfix/{maildrop,public}`, otherwise you see the following error:

```
postfix/postdrop[17400]: warning: mail_queue_enter: create file maildrop/729504.17400: Permission denied
```

* fix: Match octal permissions originally provided

Officially Postfix source seems to imply:
- `730` for `maildrop/` (_but has mentioned a sticky bit in the past, set-gid bit only for the postdrop binary involved_)
- `710` for `public/`

Both folders are assigned the same group that `postdrop` belongs to which has the SGID permission for it's executable. SGID special bit on`public/` doesn't seem necessary, but left as-is to match the default from Debian.

---------

Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
2023-03-05 20:57:40 +13:00
Georg Lauterbach 8ec5dbec74
scripts: touchups for v12.0.0 (#3144) 2023-03-04 10:57:42 +01:00
Georg Lauterbach 5e9849d94f
rspamd: rename ENABLE_REDIS & add persistence for Redis (#3143) 2023-03-04 10:45:43 +01:00
Georg Lauterbach f0edcc28d9
config: ensure SASL socket file is not inside a volume mount (#3131) 2023-03-03 23:42:55 +01:00
Georg Lauterbach 5ec6845c96
config/ENV: improve Postfix config for spoof protection (#3127) 2023-03-03 15:55:32 +01:00
Georg Lauterbach 9e2f96448a
scripts: remove PostSRSD wrapper (#3128)
* remove PostSRSD wrapper

The setup is now completely done during _actual_ setup phase. The
wrapper did not even catch signals (SIGINT, etc.), which I think is
strange.

I also added all the ENVs the wrapper relied on (which previously could
have been unset) to the variables script.

* forgot adjusting the `Dockerfile`
2023-03-02 17:44:01 +01:00
Georg Lauterbach f3edcf9cd5
issue warning in case of improper restart (#3129) 2023-03-01 16:33:54 +01:00
Georg Lauterbach 0949f16344
scripts/ENV: make disabling Redis possible (#3132)
* make disabling Redis possible

* add documentation

* Apply suggestions from code review

* Update docs/content/config/environment.md

---------

Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
2023-03-01 16:24:59 +01:00
Georg Lauterbach dfc2b39308
scripts: housekeeping & cleanup setup (2/2) (#3123) 2023-02-28 10:25:23 +01:00
Georg Lauterbach 9ead9a54ac
scripts: follow up of #3115 (feedback) (#3124) 2023-02-27 23:37:35 +01:00
Georg Lauterbach 4b04c3e31c
scripts: housekeeping & cleanup setup (1/2) (#3121) 2023-02-27 20:21:45 +01:00
Georg Lauterbach f35b60042f
scripts: split setup-stack.sh (#3115) 2023-02-26 11:42:14 +01:00
Georg Lauterbach bee9e3627d
rspamd: add feature for adjusting options with a file parsed by DMS (#3059)
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
2023-02-19 12:36:43 +01:00
Georg Lauterbach 40e10d755d
setup: improve Amavis setup routine (#3079)
* improve Amavis setup routine

see https://github.com/docker-mailserver/docker-mailserver/pull/3046#issuecomment-1423718811
2023-02-18 19:22:20 +01:00
Georg Lauterbach 632012aead
add information so Envelope From is properly set (#3081) 2023-02-18 16:21:51 +01:00
Casper 26861ddf92
fix order (#3082) 2023-02-13 00:05:52 +01:00
Georg Lauterbach ac1df91181
chore(Postfix): disable DNSBLs (#3069) 2023-02-11 00:24:09 +01:00
Casper 021a4a3c09
Fix: Make logrotate state persistant (#3077)
Fixes https://github.com/docker-mailserver/docker-mailserver/issues/3075
2023-02-09 22:34:26 +00:00
Brennan Kinney 88767f7cc8
tests(refactor): open_dkim.bats (#3060)
* tests(refactor): Make test cases for opendkim keysizes DRY

- These all do roughly the same logic that can be split into two separate methods.
- `_should_generate_dkim_key()` covers a bit more logic as it can be leveraged to handle other test cases that also perform the same logic.
- The `config/opendkim/` doesn't seem necessary for tests. Only the first few test cases here are testing against it, so we can conditionally make that available. `process_check_restart.bats` also depended on it to run OpenDKIM successfully, but this was due to the `setup-stack.sh` config defaults failing to find an "empty" file forcing `supervisord` to constantly restart the process..
- With this, there we inverse the default opendkim config, so we don't have to mount unique / empty subfolders for each test case, followed by copying over the two extra configs.

* tests(refactor): DRY up more test cases

All the remaining test cases but the last one were refactored here for a clean commit diff. The last test case will be refactored in the following commit.

Plenty of repeated logic spread across these test cases, now condensed into shared methods.

* tests(refactor): Make final test case DRY

* chore: Migrate to new testing helpers

* chore: Revise test case descriptions

* tests(refactor): Improve and simplify assertions

* tests(refactor): Use common container setup instead of `docker run`

- As the majority of test cases are only running `open-dkim` helper, we don't actually have to wait for a full container setup. So an alternative container start is called.
- Also improves assertions a bit more instead of just counting lines.
- Some test cases don't bind mount all of `/tmp/docker-mailserver` contents, thus don't raise permission errors on subsequent test runs.
- Instead of `rm -f` on some config files, have opted to mount them read-only instead, or alternatively mount an anonymous empty volume instead.
- Collapsed the first three test cases into one, thus no `setup_file()` necessary.
- Shift the `_wait_for_finished_setup_in_container()` method into `_common_container_setup()` instead since nothing else is using `_common_container_start()` yet, this allows for avoiding the wait.

* tests(refactor): Collapse dkim key size test cases into single test case

This makes these tests a bit more DRY, and enhances the raised quality issue with these tests. Now not only is the domain checked in the generated DNS dkim record, but we also verify the key size is corrected in the public and private keys via openssl.

* chore: Revise container names

* chore: Swap order of test case 1 and 2

* tests(refactor): Assert generated log output

- `__should_have_tables_trustedhosts_for_domain` shifted in each test case to just after generating the domains keys.
- Asserts `open-dkim` logs instead of just counting them.
- Added checks for domains that should not be present in a test case.
- Additional coverage and notes about the alias from vhost `@localdomain.com`
- Single assert statement with switch statement as all are using common args.

* chore: Minor changes

* tests(refactor):  Share `find` logic in helpers and tests

* tests(fix): Listing file content does not need to match line order

The order printed from local system vs CI differed causing the CI to fail. The order of lines is irrelevant so `--index` is not required.

Additionally correct the prefix of the called method to be only one `_` now that it's a `common.bash` helper method.

* chore: Collapse custom DKIM selector test into custom DKIM domain test

These cover the same test logic for the most part, the first domain could also be testing the custom selector.

`special_use_folders.bats` + `mailbox_format_dbox` can assert lines instead, removing the need for `--partial`.

* Apply suggestions from code review

Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>

* chore: Split switch statement method into wrapper methods

---------

Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
2023-02-10 00:18:06 +13:00
Georg Lauterbach 00b1d88ed7
removal: configomat (submodule) (#3045) 2023-02-05 12:39:05 +01:00
Georg Lauterbach 774a340d76
chore(Amavis): only add configuration to Postfix when enabled (#3046)
* only add Amavis configuration to Postfix when enabled

Since I am running Rspamd nowadays, I noticed there still are ports open
that belong to Amavis. This is because the Amavis configuration is a
fixed part of Postfix's `master.cf`. I changed that. Now, the Amavis
section is added when Amavis really is enabled.

I took the chance and added proper indentation to `master.cf`; hence the
diff is a bit fuzzy. **But**, only the Amavis part was adjusted, the
rest is just styling.
2023-02-03 17:32:07 +01:00
Georg Lauterbach 1a0c2a351a
rspamd: follow-up of #3016 (#3039) 2023-01-30 08:23:58 +01:00
Brennan Kinney 3d8cfc5b5a
chore: Remove wrapper script for fail2ban service (#3032)
* chore: Remove the wrapper script for `fail2ban`

- This does not appear necessary. The server can be run with foreground mode.
- `daemons-stack.sh` removal of the socket can be handled by the fail2ban server when using the `-x` option.

* chore: Remove `touch /var/log/auth.log`

These were both added as supposed fixes in 2016 for the then Ubuntu 2014 base image.

Removing them causes no failures in tests.

* fix: Install optional python packages for `fail2ban`

These have barely any overhead in layer weight. The DNS package may provide some QoL improvements, while the `pyinotify` is a better alternative than polling logs to check for updates.

We have `gamin` package installed but `fail2ban` would complain in the log that it was not able to initialize the module for it. There only appears to be a `python-gamin` dependent on EOL python 2, no longer available from Debian Bullseye.
2023-01-29 12:35:57 +00:00
Georg Lauterbach 555fbb78c4
feature: provide better rspamd suppport (#3016)
* added options to toggle OpenDKIM & OpenDMARC

rspamd can provide DKIM signing and DMARC checking itself, so users
should be able to disable OpenDKIM & OpenDMARC. The default is left at
1, so users have to to opt-in when the want to disable the features.

* misc small enhancements

* adjusted start of rspamd

The order of starting redis + rspamd was reversed (now correct) and
rspamd now starts with the correct user.

* adjusted rspamd core configuration

The main configuration was revised. This includes AV configuration as
well as worker/proxy/controller configuration used to control the main
rspamd processes.

The configuration is not tested extensively, but well enough that I am
confident to go forward with it until we declare rspamd support as
stable.

* update & improve the documentation

* add tests

These are some initial tests which test the most basic functionality.

* tests(refactor): Improve consistency and documentation for test helpers (#3012)

* added `ALWAYS_RUN` target `Makefile` recipies (#3013)

This ensures the recipies are always run.

Co-authored-by: georglauterbach <44545919+georglauterbach@users.noreply.github.com>

* adjusted rspamd test to refactored test helper functions

* improve documentation

* apply suggestions from code review (no. 1 by @polarthene)

Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>

* streamline heredoc (EOM -> EOF)

* adjust rspamd test (remove unnecessary run arguments)

Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
2023-01-25 10:28:59 +01:00
Brennan Kinney cb8e336d25
fix: Ensure state persisted to /var/mail-state retains correct group (#3011)
* fix: RSPAM ENV should only add to array if ENV enabled

* fix: Correctly match ownership for Postfix content

- `/var/lib/postfix` dir and content is `postfix:postfix`, not `postfix:root`.
- `/var/spool/postfix` is `root:root` not `postfix:root` like it's content.
- Add additional comments, including ownership changes by Postfix to `/var/spool/postfix` when process starts / restarts.

* fix: Ensure correct `chown -R` user and groups applied

These were all fine except for clamav not using the correct clamav group. `fetchmail` group is `nogroup` as per the group set by the debian package.

Additionally formatted the `-eq 1 ]]` content to align on the same columns, and added additional comment about the purpose of this `chown -R` usage so that it's clear what bug / breakage it's attempting to prevent / fix.

* refactor: `misc-stack.sh` conditional handling

The last condition doesn't get triggered at all AFAIK.  Nor does it make sense to make a folder path with `mkdir -p` to symlink to when the container does not have anything to copy over?

- If that was for files, the `mkdir -p` approach seems invalid?
- If it was for a directory that could come up later, it should instead be created in advance? None of the current values for `FILES` seem to hit this path.

Removing as it doesn't seem relevant to current support.

Symlinking was done for each case, I've opted to just perform that after the conditional instead.

Additional inline docs added for additional context.

* chore: Move amavis `chown -R` fix into `misc-stack.sh`

This was handled separately for some reason. It belongs with the other services handling this fix in `misc-stack.sh`.

The `-h` option isn't relevant, when paired with `-R` it has no effect.

* fix: Dockerfile should preserve `clamav` ownership with `COPY --link`

The UID and GID were copied over but would not match `clamav` user and group due to numeric ID mismatch between containers. `--chown=clamav` fixes that.

* chore: Workaround `buildx` bug with separate `chown -R`

Avoids increasing the image weight from this change by leveraging `COPY` in the final stage.

* chore: `COPY --link` from a separate stage instead of relying on scratch

The `scratch` approach wasn't great. A single layer invalidation in the previous stage would result in a new 600MB layer to store.

`make build` with this change seems to barely be affected by such if a change came before copying over the linked stage, although with `buildx` and the `docker-container` driver with `--load` it would take much longer to import and seemed to keep adding storage. Possibly because I was testing with a minimal `buildx` command, that wasn't leveraging proper cache options?

* lint: Appease the linting gods

* chore: Align `misc-stack.sh` paths for `chown -R` operations

Review feedback

Co-authored-by: Casper <casperklein@users.noreply.github.com>

* fix: Reduce one extra cache layer copy

No apparent advantage of a `COPY --link` initially in separate stage.

Just `COPY --chown` in the separate stage and `COPY --link` the stage content. 230MB less in build cache used.

* fix: Remove separate ClamAV stage by adding `clamav` user explicitly

Creating the user before the package is installed allows to ensure a fixed numeric ID that we can provide to `--chown` that is compatible with `--link`.

This keeps the build cache minimal for CI, without being anymore complex as a workaround than the separate stage was for the most part.

* chore: Add reference link regarding users to `misc-stack.sh`
2023-01-25 12:53:47 +13:00
Brennan Kinney 133eb9bc2e
tests(refactor): mail_lmtp_ip.bats (#3004)
* chore: Drop ENV `ENABLE_POSTFIX_VIRTUAL_TRANSPORT`

* tests(chore): Remove redundant `dovecot-lmtp` config

None of this is needed. Only relevant change is changing the LMTP service listener for Dovecot and that can be delegated to `user-patches.sh`.

* tests(refactor): Use `user-patches.sh` instead of replacing config file

The only relevant changes in `test/config/dovecot-lmtp` regarding LMTP was:
- `/etc/dovecot/dovecot.conf` (`protocols = imap lmtp`) and `/etc/dovecot/protocols.d/` (`protocols = $protocols lmtp`).
- `conf.d/10-master.conf` only changed the LMTP service listener from a unix socket to TCP on port 24 (_this was the only change required for the test to pass_).

None of those configs are required as:
- `protocols = imap pop3 lmtp` [is the upstream default](https://doc.dovecot.org/settings/core/#core_setting-protocols), no need to add `lmtp`.
- The LMTP service listener is now configured for the test with `user-patches.sh`.

* tests(refactor): `mail_lmtp_ip.bats`

- Converted to new testing conventions and common container helpers.
- `ENABLE_POSTFIX_VIRTUAL_TRANSPORT` was not relevant, dropped.
- Revised test cases, logic remains the same.
- Large custom config used was not documented and doesn't appear to serve any purpose. Simplified by replacing with a single modification with `user-patches.sh`.
- Added some additional comments for context of test and improvements that could be made.

* tests(chore): Adjust comments

The comment from `mail_hostname` provides no valid context, it was likely copied over from `tests.bats` in Oct 2020 by accident.

The email sent is just for testing, nothing relevant to LMTP.

---

Added additional comment for test to reference extra information from.

* tests(chore): Update similar log line matching

Extracts out the match pattern and formatting commands into separate vars (reduces horizontal scrolling), and includes extra docs about what the matched line should be expected to look like.
2023-01-15 18:33:31 +13:00
Brennan Kinney a7e6439a39
fix: Workaround postconf write settling logic (#2998)
* fix: Workaround `postconf` write settle logic

After updating `main.cf`, to avoid an enforced delay from reading the config by postfix tools, we can ensure the modified time is at least 2 seconds in the past as a workaround. This should be ok with our usage AFAIK.

Shaves off 2+ seconds roughly off each container startup, reduces roughly 2+ minutes off tests.

* chore: Only modify `mtime` if less than 2 seconds ago

- Slight improvement by avoiding unnecessary writes with a conditional check on the util method.
- Can more comfortably call this during `postfix reload` in the change detection cycle now.
- Identified other tests that'd benefit from this, created a helper method to call instead of copy/paste.
- The `setup email restrict` command also did a modification and reload. Added util method here too.

* tests(fix): `mail_smtponly.bats` should wait for Postfix

- `postfix reload` fails if the service is not ready yet.
- `service postfix reload` and `/etc/init.d/postfix reload` presumably wait until it is ready? (as these work regardless)

* chore: Review feedback - Move reload method into utilities
2023-01-13 10:10:58 +13:00
Georg Lauterbach 26ac48f34a
feature: provide initial Rspamd support (#2902) 2023-01-05 08:39:00 +01:00
Gabriel Euzet a00cdcdee9
fix regex in quota activation code (#2958) 2023-01-04 18:37:00 +01:00
Georg Lauterbach 0ef8d5ae2b
fix: adjust DNSBL return code interpretation (#2890) 2022-11-07 09:28:47 +01:00
Casper 474eb503ae
Fix: Make fetchmail data persistant (#2851) 2022-10-23 03:12:27 +02:00
Casper a930aeb18a
Remove unusual space from shebang line (#2834) 2022-10-17 10:40:09 +02:00
Casper ac795a5976
fix: run Amavis cron job only when Amavis is enabled (#2831) 2022-10-16 11:52:53 +02:00
Casper efed7d9e44
Remove unnecessary postconf switch '-e' and use single quotes where possible (#2746) 2022-08-29 13:26:44 +02:00
Georg Lauterbach ab55343d8e
scripts: rework environment variables setup (#2716)
* outsourcing env variable setup

This commit contains major parts of the work of refactoring the setup
and usage of environment variables. It outsources the setup into its own
script and provides dedicated functions to be executed at a later point in time.

A **new** env variable was added: `USER_PROVISIONG` which provides a
better way of defining which method / protocol to use when it comes to
setting up users. This way, the `ENABLE_LDAP` variable is deprecated,
but all of this is backwards compatible due to a "compatibility layer", a function provided by the new variables script.

This is not a breaking change. It mostly refators internal scripts. The
only change facing the user-side is the deprecation of `ENABLE_LDAP`. We
can prolong the period of deprecation for this variable as long as we
want, because the new function that ensures backwards compatibility
provides a clean interface for the future.

Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
Co-authored-by: Casper <casperklein@users.noreply.github.com>
2022-08-22 08:31:32 +02:00
Brennan Kinney 26053c22bd
fix: _create_accounts() should run after waiting (#2731)
When waiting on an account to be added to `postfix-accounts.cf`, Dovecot account creation during the startup process had already run.

Startup continued without properly creating the mail account for Dovecot. Methods like `setup email list` (with `ENABLE_QUOTAS=1`) would fail. `changedetector` service was required to be triggered to re-create Dovecot users.

- Wrapped the logic for wait + shutdown into a function call.
- Moved `_create_accounts()` to bottom of the setup function.
2022-08-22 10:44:23 +12:00
Casper 428477a878
Set configomat output to loglevel debug (#2701) 2022-07-28 20:55:21 +02:00
Brennan Kinney e3cc627e18
refactor: Share a common helper (vhost builder) for sourcing domains (#2620)
* chore: Split vhost helper method and use filepath vars

- Helpers `accounts.sh` and `aliases.sh` can move their vhost code into this helper.
- They share duplicate code with `bin/open-dkim` which will also leverage this vhost helper going forward.

* chore: Sync vhost generation logic into helper

- Chunky commit, but mostly copy/paste of logic into a common method.
- `bin/open-dkim` additionally wrapped relevant logic in a function call and revised inline docs.

* chore: Include LDAP vhost support

- Revises notes for LDAP vhost support.
- This now ensures LDAP users get vhost rebuilt to match the startup script for when change detection support is enabled.
- `bin/open-dkim` will additionally be able to support the default `DOMAINNAME` var (set via `helpers/dns.sh`) for LDAP users instead of requiring them to provide one explicitly.

* chore(`bin/open-dkim`): Ensure `DOMAINNAME` is properly set

- This will ensure LDAP users insert the same `DOMAINNAME` value as used during container startup.
- The container itself should panic at startup (during `helpers/dns.sh`) if this isn't configured correctly already, thus it should not introduce any breaking change to users of this utility?

* chore: Set the 2nd value as blank `_`

Line is split by a delimiter such as white-space (or via IFS: `|`), the blank `_` var is to indicate we're not interested in that value, but still leverage how `read -r` works, instead of splitting the var ourselves first thing.

* chore: Remove shellcheck disable lines

No longer applicable with the switch to `_`
2022-06-10 10:57:10 +12:00
Brennan Kinney 0a722276a8
chore: Extract out /var/mail ownership workaround (#2628)
Keep it in sync between the two locations via shared helper method.
2022-06-08 10:09:19 +12:00