Commit graph

50 commits

Author SHA1 Message Date
Georg Lauterbach bee9e3627d
rspamd: add feature for adjusting options with a file parsed by DMS (#3059)
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
2023-02-19 12:36:43 +01:00
Georg Lauterbach 00b1d88ed7
removal: configomat (submodule) (#3045) 2023-02-05 12:39:05 +01:00
Georg Lauterbach 555fbb78c4
feature: provide better rspamd suppport (#3016)
* added options to toggle OpenDKIM & OpenDMARC

rspamd can provide DKIM signing and DMARC checking itself, so users
should be able to disable OpenDKIM & OpenDMARC. The default is left at
1, so users have to to opt-in when the want to disable the features.

* misc small enhancements

* adjusted start of rspamd

The order of starting redis + rspamd was reversed (now correct) and
rspamd now starts with the correct user.

* adjusted rspamd core configuration

The main configuration was revised. This includes AV configuration as
well as worker/proxy/controller configuration used to control the main
rspamd processes.

The configuration is not tested extensively, but well enough that I am
confident to go forward with it until we declare rspamd support as
stable.

* update & improve the documentation

* add tests

These are some initial tests which test the most basic functionality.

* tests(refactor): Improve consistency and documentation for test helpers (#3012)

* added `ALWAYS_RUN` target `Makefile` recipies (#3013)

This ensures the recipies are always run.

Co-authored-by: georglauterbach <44545919+georglauterbach@users.noreply.github.com>

* adjusted rspamd test to refactored test helper functions

* improve documentation

* apply suggestions from code review (no. 1 by @polarthene)

Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>

* streamline heredoc (EOM -> EOF)

* adjust rspamd test (remove unnecessary run arguments)

Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
2023-01-25 10:28:59 +01:00
Brennan Kinney 8d80c6317f
tests(refactor): Adjust mail_changedetector + change detection helpers (#2997)
* tests(refactor): `mail_changedetector.bats` - Leverage DRY methods

`supervisorctl tail` is not the most reliably way to get logs for the latest change detection and has been known to be fragile in the past.

We can instead read the full log for the service directly with `tac` and `sed` to extract all log content since the last change detection.

Common asserts have also been extracted out into separate methods.

* tests(chore): Remove sleep and redundant change event

Container 1 is still blocked at this point from an existing lock and change event.

Make the lock stale immediately and no extra sleep is required when paired with the helper method to wait until the event is processed (which should remove the stale lock).

* tests(refactor): Add more DRY methods

- Simplify the test case so it's easier to grok.
- 2nd test case (blocking) extracts out initial setup into a separate method and merges the later service restart logic which is redundant.
- Additional comments for improved context of what is going on / expected.

* tests(chore): Revise the change detection helper method

- Add explicit counting arg to change detection support.
- Extract revised logic into it's own generic helper method.
- Utilize this for a separate method that monitors for a change event having started, but not waiting for completion.

This allows dropping the 40 sec of remaining `sleep` in `mail_changedetector` test. It was also required due to potentially missing the timing of a change event completing concurrently in a 2nd container that needed to be waited on and then checked.

* tests(chore): Migrate to current test conventions

- Switch to common container setup helpers
- Update container name and change usage to variables instead.
- Adopt the new convention of prefix variable for test cases (revised test case descriptions).

* tests(chore): Remove legacy change detection

This has since been replaced with the new helper watches the `changedetector` service logs directly instead of only detecting a change has occurred via checksum comparison.

No tests use this method anymore, it was originally for `tests.bats`. Thus the tests in `test_helper.bats` are being dropped too. The new helper has test coverage in `changedetector` tests.

* chore: Lock removal should not incur `sleep 5` afterwards

- A new lock should be created by this script after removal. The sleep doesn't help avoid a race condition with lock file creation after removal.
- Reduces test time as a bonus.
- Added some additional comments to test.

* tests(chore): `tls_letsencrypt.bats` leverage improved change detection

- No need to wait on the change detection service anymore during container startup.
- No need to count change events processed either as waiting a fixed duration is no longer relied on.
- This makes the reload count method redundant, dropped.

* tests(chore): Convert `setup-cli.bats` to new test conventions

This test file was already adapted to the original common setup helpers.

- `TEST_NAME` replaced with `CONTAINER_NAME`.
- Prefix var added, test case descriptions drop explicit prefix.
- No other changes.

* tests(chore): Extract out helpers related to change-detection

- New helper file for sharing these helpers to tests.
- Includes the helpful log method from changedetector tests.
- No longer need to maintain duplicate copies of these methods during the test migration. All tests that use them are now importing the separate helper file.
- `tls_letsencrypt.bats` has switched to using the log helper.
- Generic log count helper is removed from `test_helper/common.bash` as any test that needs it in future can adapt to `helper/common.bash`.

* tests(refactor): `tls_letsencrypt.bats` remove `_get_service_logs()`

This helper does not seem useful as moving away from `supervisorctl tail` and no other tests had a need for it.

* tests(chore): Remove common setup methods from `test_helper/common.bash`

No other tests depend on this. Future tests will adopt the revised versions from `helper/setup.bash`.

Additionally updates `helper/setup.bash` comments that are no longer applicable to `TEST_TMP_CONFIG` and `CONTAINER_NAME`.

* chore: Use `|| true` to simplify setting `EXPECTED_COUNT` correctly
2023-01-16 20:39:46 +13:00
Brennan Kinney a7e6439a39
fix: Workaround postconf write settling logic (#2998)
* fix: Workaround `postconf` write settle logic

After updating `main.cf`, to avoid an enforced delay from reading the config by postfix tools, we can ensure the modified time is at least 2 seconds in the past as a workaround. This should be ok with our usage AFAIK.

Shaves off 2+ seconds roughly off each container startup, reduces roughly 2+ minutes off tests.

* chore: Only modify `mtime` if less than 2 seconds ago

- Slight improvement by avoiding unnecessary writes with a conditional check on the util method.
- Can more comfortably call this during `postfix reload` in the change detection cycle now.
- Identified other tests that'd benefit from this, created a helper method to call instead of copy/paste.
- The `setup email restrict` command also did a modification and reload. Added util method here too.

* tests(fix): `mail_smtponly.bats` should wait for Postfix

- `postfix reload` fails if the service is not ready yet.
- `service postfix reload` and `/etc/init.d/postfix reload` presumably wait until it is ready? (as these work regardless)

* chore: Review feedback - Move reload method into utilities
2023-01-13 10:10:58 +13:00
Casper 6ac59ef871
Fix several typos (#2990) 2023-01-10 14:13:50 +01:00
Georg Lauterbach 26ac48f34a
feature: provide initial Rspamd support (#2902) 2023-01-05 08:39:00 +01:00
René Plötz 55f6260bad
fix: Ensure relay host properly handles credentials check (#2965)
A recent change broke the conditional check. Reverting.
2022-12-28 08:50:03 +13:00
Brennan Kinney 4dda5f8b1f
chore: Drop support for deprecated TLS versions (#2945)
* chore: Set `TLS_INTERMEDIATE_SUITE` to only use TLS 1.2 ciphersuites

Removes support of the following cipher suites that are only valid for TLS 1.0 + 1.1:

- `ECDHE-ECDSA-AES128-SHA`
- `ECDHE-RSA-AES128-SHA`
- `ECDHE-ECDSA-AES256-SHA`
- `ECDHE-RSA-AES256-SHA`
- `DHE-RSA-AES128-SHA`
- `DHE-RSA-AES256-SHA`

* chore: Update TLS version min and ignore settings

These are now the same as modern settings.

* fix: Remove min TLS support workaround

No longer required now that outdated TLS versions have been dropped.

* tests: Remove support for TLS 1.0 and 1.1 ciphersuites

* tests: Remove support for TLS 1.0 and 1.1 ciphersuites (Port 25)

The removed SHA1 cipher suites are still supported in TLS 1.2, thus they've been excluded for port 25 via the `SHA1` exclusion pattern in `main.cf`.
2022-12-24 02:30:43 +13:00
Brennan Kinney fe21fe78e2
chore: Remove legacy ENV SASL_PASSWD (#2946)
As per deprecation notice from v11.3 release notes, and a related prior PR; this ENV is to be removed.

It's no longer considered useful, and none of the tests that configured it were actually using it for relaying anything.
2022-12-23 15:30:40 +13:00
Casper 897d84f0f9
scripts: Improve error handling, when parameters are missing (#2854) 2022-10-29 11:04:35 +02:00
Casper a930aeb18a
Remove unusual space from shebang line (#2834) 2022-10-17 10:40:09 +02:00
Georg Lauterbach 4ab23061a1
style: minor adjustments (#2786)
Co-authored-by: Casper <casperklein@users.noreply.github.com>
2022-10-02 13:00:15 +02:00
Casper 1f137f3eca
Only calculate checksums, when there are files to monitor. (#2776) 2022-09-19 10:45:40 +02:00
Georg Lauterbach 39774df85d
scripts: improve helpers/log.sh (#2754)
This PR prepares for other PRs that use the newly introduced helper
functions. The `_log` function itself was adjusted a bit to be shorter
and more concise.
2022-09-03 20:59:56 +00:00
Casper efed7d9e44
Remove unnecessary postconf switch '-e' and use single quotes where possible (#2746) 2022-08-29 13:26:44 +02:00
Georg Lauterbach ab55343d8e
scripts: rework environment variables setup (#2716)
* outsourcing env variable setup

This commit contains major parts of the work of refactoring the setup
and usage of environment variables. It outsources the setup into its own
script and provides dedicated functions to be executed at a later point in time.

A **new** env variable was added: `USER_PROVISIONG` which provides a
better way of defining which method / protocol to use when it comes to
setting up users. This way, the `ENABLE_LDAP` variable is deprecated,
but all of this is backwards compatible due to a "compatibility layer", a function provided by the new variables script.

This is not a breaking change. It mostly refators internal scripts. The
only change facing the user-side is the deprecation of `ENABLE_LDAP`. We
can prolong the period of deprecation for this variable as long as we
want, because the new function that ensures backwards compatibility
provides a clean interface for the future.

Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
Co-authored-by: Casper <casperklein@users.noreply.github.com>
2022-08-22 08:31:32 +02:00
Brennan Kinney 57aeb6db2a
refactor: CLI commands for database management (#2654)
See the associated PR for more detailed commentary on specific changes.

### Commands refactored:
- User (**All:** add / list / update / del + _dovecot-master variants_)
- Quota (**All:** set / del)
- Virtual Alias (**All:** add / list /del)
- Relay (**All:** add-relayhost / add-sasl / exclude-domain)

### Overall changes involve:
- **Fairly common structure:**
  - `_main` method at the top provides an overview of logical steps:
    - After all methods are declared beneath it (_and imported from the new `helpers/database/db.sh`_), the `_main` is called at the bottom of the file.
    - `delmailuser` additionally processes option support for `-y` prior to calling `_main`.
  - `__usage` is now consistent with each of these commands, along with the `help` command.
  - Most logic delegated to new helper scripts. Some duplicate content remains on the basis that it's low-risk to maintenance and avoids less hassle to jump between files to check a single line, usually this is arg validation.
  - Error handling should be more consistent, along with var names (_no more `USER`/`EMAIL`/`FULL_EMAIL` to refer to the same expected value_).
- **Three new management scripts** (in `helpers/database/manage/`) using a common structure for managing changes to their respective "Database" config file.
  - `postfix-accounts.sh` unified not only add and update commands, but also all the dovecot-master versions, a single password call for all 4 of them, with a 5th consumer of the password prompt from the relay command `addsaslpassword`.
  - These scripts delegate actual writes to `helpers/database/db.sh` which provides a common API to support the changes made.
     - This is more verbose/complex vs the current inline operations each command currently has, as it provides generic support instead of slightly different variations being maintained, along with handling some edge cases that existed and would lead to bugs (notably substring matches).
     - Centralizing changes here seems wiser than scattered about. I've tried to make it easy to grok, hopefully it's not worse than the current situation.
     - List operations were kept in their respective commands, `db.sh` is only really managing writes. I didn't see a nice way for removing the code duplication for list commands as the duplication was fairly minimal, especially for `listalias` and `listdovecotmasteruser` which were quite simple in their differences in the loop body.
     - `listmailuser` and `delmailuser` also retain methods exclusive to respective commands, I wasn't sure if there was any benefit to move those, but they were refactored.
2022-07-29 12:10:23 +12:00
Georg Lauterbach 684903f8d2
docs: Revise Kubernetes documentation (#2696) 2022-07-28 10:26:53 +12:00
Thomas VIAL a84b8a130c
fix: Dovecot Quota dummy accounts for aliases should check for existing users with an exact user key lookup (#2640)
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
2022-06-15 12:31:58 +02:00
Brennan Kinney 851ec8cbcd
refactor: Revise check-for-changes.sh (#2615)
- Inline docs for `check-for-changes.sh` have been shuffled around and revised a bit.
- Change processing extracted from the main change detection loop method to their own methods:
   - `_get_changed_files()` - Clarifies what is going on (and how) without having to look it up. To reduce noise in the main logic loop, extracted to a separate method.
   - `_postfix_dovecot_changes()` - The bulk of change processing was moved to this method. I've added conditionals to only run relevant logic.
   - `_ssl_changes()` - Just shifted, no logic changed. `REGEX_NEVER_MATCH` and `ACME_CERT_DIR` vars scope set to `local`.
2022-06-12 11:36:37 +12:00
Brennan Kinney e3cc627e18
refactor: Share a common helper (vhost builder) for sourcing domains (#2620)
* chore: Split vhost helper method and use filepath vars

- Helpers `accounts.sh` and `aliases.sh` can move their vhost code into this helper.
- They share duplicate code with `bin/open-dkim` which will also leverage this vhost helper going forward.

* chore: Sync vhost generation logic into helper

- Chunky commit, but mostly copy/paste of logic into a common method.
- `bin/open-dkim` additionally wrapped relevant logic in a function call and revised inline docs.

* chore: Include LDAP vhost support

- Revises notes for LDAP vhost support.
- This now ensures LDAP users get vhost rebuilt to match the startup script for when change detection support is enabled.
- `bin/open-dkim` will additionally be able to support the default `DOMAINNAME` var (set via `helpers/dns.sh`) for LDAP users instead of requiring them to provide one explicitly.

* chore(`bin/open-dkim`): Ensure `DOMAINNAME` is properly set

- This will ensure LDAP users insert the same `DOMAINNAME` value as used during container startup.
- The container itself should panic at startup (during `helpers/dns.sh`) if this isn't configured correctly already, thus it should not introduce any breaking change to users of this utility?

* chore: Set the 2nd value as blank `_`

Line is split by a delimiter such as white-space (or via IFS: `|`), the blank `_` var is to indicate we're not interested in that value, but still leverage how `read -r` works, instead of splitting the var ourselves first thing.

* chore: Remove shellcheck disable lines

No longer applicable with the switch to `_`
2022-06-10 10:57:10 +12:00
Brennan Kinney c314c9c471
chore(check-for-changes.sh): Drop redundant guards (#2623)
* chore: Remove requirement for `postfix-accounts.cf`

This is an old requirement from when the change detector service was first introduced. It's no longer relevant.

* chore: Do not needlessly create `postfix-aliases.cf`

The config was created regardless to workaround early change detection support. No longer necessary to require the file to exist.

* chore: Drop guards requiring `/tmp/docker-mailserver` to exist

Legacy guards when this was the only location change detection location supported.

There does not appear to be any need for changing into this directory at the start of `check-for-changes.sh` as we use absolute filepaths (originally monitored files were checked with relative paths to this config dir).

* chore: Revise inline docs

* chore: Add change detection monitoring for extra configs

These are also handled at run-time in the current change detection support, so it makes sense to allows these config updates to also trigger change events.
2022-06-09 19:48:07 +12:00
Brennan Kinney 0a722276a8
chore: Extract out /var/mail ownership workaround (#2628)
Keep it in sync between the two locations via shared helper method.
2022-06-08 10:09:19 +12:00
Brennan Kinney 54904aa02c
chore(housekeeping): Normalize how config files filter out unwanted lines (#2619)
* chore(`aliases.sh`): Filepath to local var `DATABASE_VIRTUAL`

* chore(`accounts.sh`): Filepath to local var `DATABASE_ACCOUNTS`

* chore(`accounts.sh`): Filepath to local var `DATABASE_VIRTUAL`

* chore(`accounts.sh`): Filepath to local var `DATABASE_DOVECOT_MASTERS`

* chore(`bin/open-dkim`): Filepaths to local vars (accounts,virtual,vhost)

* chore(`relay.sh`): Filepath to local var `DATABASE_SASL_PASSWD`

* chore: Rename method

Prior PR feedback suggested a better helper method name.

* chore: Normalize filtering config lines as input for iterating

* chore: Remove `_is_comment` helper method

No longer serving a purpose with more appropriate filter method for pre-processing the entire config file.
2022-06-07 01:07:30 +12:00
Brennan Kinney 1f3e672018
fix(listmailuser): Don't parse comments when reading config (#2617)
* fix(listmailuser): Don't parse comments

Avoids passing comments to `dovecot_quota_to_hr()` which fails to handle it and would throws errors.

* chore: Move config filter method to `helpers/utils.sh`
2022-06-06 11:02:52 +12:00
Brennan Kinney 40e2d88482
chore: Merge helpers/sasl.sh into helpers/relay.sh (#2605)
This helper was to support an earlier ENV for SASL auth support. When extracting logic into individual helpers, it was assumed this was separate from relay support, which it appears was not the case.

---

The `SASL_PASSWD` ENV is specified in tests but no longer used. There is no `external-domain.com` relay configured or tested against anywhere in the project.

The ENV was likely used in tests prior to improved relay support that allowed for adding more than a single set of relay credentials.

---

It likewise has no real relevance anywhere else outside of `relay.sh` as it's the only portion of code to operate with it.

It's only relevant for SASL auth as an SMTP client, not the SMTP server (`smtpd`) SASL support that is delegated to Dovecot. Functionality has been completely migrated into `relay.sh` as a result.

Documentation is poor for this ENV, it is unlikely in wide use? Should consider for removal.

---

The ENV has been dependent upon `RELAY_HOST` to actually enable postfix to use `/etc/postfix/sasl_passwd`, thus not likely relevant in existing setups?

---

Migrate `/etc/postfix/sasl_passwd` check from `tests.bats` as it belongs to relay tests.
2022-06-06 10:59:42 +12:00
Brennan Kinney c862e1451d
chore(housekeeping): Create helpers/change-detection.sh (#2610)
* chore: Extract change-detection method to it's own helper

This doesn't really belong in `helpers/ssl.sh`. Moving to it's own helper script.

* chore: Co-locate related change-detection method from container startup

It seems relevant to migrate the related support during startup for the change detection feature into this helper.

I opted to move the call from `start-mailserver.sh` into the `_setup` call at the end for a more explicit/visible location.

* chore: Move `CHKSUM_FILE` into `helpers/change-detection.sh`

It belongs there, not in `helpers/index.sh`.

* chore: Revise inline documentation

* tests(fix): Ensure correct functionality

Presently `test/test_helper.bats` is using it's own  `CHKSUM_FILE` instead of sourcing the var for the filepath.

`test_helper/common.bash` was calling a method to check for changes, but this helper may not correctly detect letsencrypt related changes as these are not ENV rely on, but global vars handled by `helpers/dns.sh`, so that should be run first like it is for `check-for-changes.sh`.

* tests(chore): Use `CHKSUM_FILE` var from helper

* chore: `addmailuser` should use `CHKSUM_FILE` var

* chore: Update `check-for-changes.sh` log message with correct path
2022-06-05 11:59:54 +12:00
Brennan Kinney 598aee166c
refactor: Revised relay.sh helper (#2604)
* chore: Make `_populate_relayhost_map` easier to grok

Changes to `sed` handling that made it quicker to grok, and thus easier for maintainers like myself:
- Switched regex to [extended regex](https://www.gnu.org/software/sed/manual/html_node/Extended-regexps.html).
- Extracted `sed` patterns to be self-descriptive local vars.
- Used a function to reduce noise from intent of loop input (each line as `DOMAIN_PART`).

Input for the loop is filtered through `sort -u` to drop duplicates, reducing iterations.
 
`DOMAIN` loop var renamed to less vague `DOMAIN_PART`. Additional comment in the containing method clarifies what the domain part refers to.

---

`|` regexp syntax needed to be escaped due to switch. Not documented in the earlier link. `-r`/`-E` (ERE) aka extended regexp syntax is [detailed here](https://learnbyexample.github.io/learn_gnused/breere-regular-expressions.html#cheatsheet-and-summary).

* chore: Drop unnecessary postfix parameters

`smtp_tls_note_starttls_offer = yes` - Only adds a log entry to let you know when an unencrypted connection was made, but STARTTLS was offered:
https://www.postfix.org/postconf.5.html#smtp_tls_note_starttls_offer

`smtp_tls_CAfile` is unnecessary. This was added before `smtp_tls_CApath = /etc/ssl/certs` was several months later via a separate PR.

* chore: Move `smtp_` parameters to relevant sections

These have been shifted to relevant logic for now.

---

NOTE: `SASL_PASSWD` previously needed to define `RELAY_HOST` to set `smtp_sasl_password_maps` to enable the `/etc/postfix/sasl_passwd` table. This change now additionally blocks early on in `_relayhost_sasl`. Not likely important due to `RELAY_HOST` logic, user should be using the `RELAY_USER` + `RELAY_PASSWORD` ENV or `postfix-sasl-password.cf` instead.

Especially the sender dependent parameters which are only relevant with user provided configs really.

`SASL_PASSWD` is the oldest ENV for relay support before any other relay feature arrived. It is poorly documented and should not be used.

Potential breakage risk considered acceptable.

* chore: Revise inline docs

Further clarifying current processing logic and adding some additional notes for future work.

* chore: Use a common ENV relay-host getter

The mapping should be in sync between the two configs.

I also wanted to raise awareness of current state of support, which will likely need some refactoring.

This also removes the need for the `RELAY_PORT` fallback method.

The log message was adjusted as configuration is potentially for more than one relay host beyond the currently required ENV config to enable support.

---

NOTE: The ENV `DEFAULT_RELAY_HOST` skips modifying the default transport for an authenticated relay (locked behind `RELAY_HOST` to activate). It presently will only relay mail through a relay host on port 25 instead of delivering directly to the destination. A separate use-case.

* chore: Revise config examples

More verbose example configs with expanded documentation.

Additional doc references for SASL support and cautioning maintainers that may reference popular relay service providers docs. May later be migrated to a "maintainers" section in official docs and link to that.

Brief overview description of what `_populate_relayhost_map` is doing.

* chore: Add notes pertaining to future work

`_populate_relayhost_map` will get some refactoring in future and likely introduce some breaking changes for a future major release.

* chore: Better document relay support inline

This helper now includes a description of it's purpose, links to relevant user docs and supported `setup.sh` commands.

Intent is to keep a maintainer of the feature aware of anything relevant to this feature.
2022-06-05 11:27:15 +12:00
Casper 372e3fe35e
Use color variables instead of escape codes (#2565) 2022-05-10 17:50:33 +02:00
Andrew Cornford 2f3cbfc144
feat: Support for Dovecot master accounts (#2535)
Dovecot master accounts can now be configured in DMS via `setup.sh`.
A master account is useful for administration purposes, or to perform mailbox backups of every user account over IMAP.
Upstream Docs: https://doc.dovecot.org/configuration_manual/authentication/master_users/

Co-authored-by: Casper <casperklein@users.noreply.github.com>
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
2022-05-08 00:28:32 +02:00
Casper 628e902233
Remove unnecessary quotes from command substitutions (#2561) 2022-05-05 10:28:38 +02:00
Casper cbcc3823d3
Fix changedetector restart loop (#2548)
* only restart changedetector, if exit is unexpected.

* prevent supervisord from restarting changedetector on error --> endless loop

* add quotes
2022-04-19 21:09:25 +02:00
Brennan Kinney 1b1877f025
refactor: letsencrypt implicit location discovery (#2525)
* chore: Extract letsencrypt logic into methods

This allows other scripts to share the functionality to discover the correct letsencrypt folder from the 3 possible locations (where specific order is important).

As these methods should now return a string value, the `return 1` after a panic is now dropped.

* chore: Update comments

The todo is resolved with this PR, `_setup_ssl` will be called by both cert conditional statements with purpose for each better documented to maintainers at the start of the logic block.

* refactor: Defer most logic to helper/ssl.sh

The loop is no longer required, extraction is delegated to `_setup_ssl` now.

For the change event prevention, we retrieve the relevant FQDN via the new helper method, beyond that it's just indentation diff.

`check-for-changes.sh` adjusted to allow locally scoped var declarations by wrapping a function. Presently no loop control flow is needed so this seems fine. Made it clear that `CHANGED` is local and `CHKSUM_FILE` is not.

Panic scope doesn't require `SSL_TYPE` for context, it's clearly`letsencrypt`.

* fix: Correctly match wildcard results

Now that the service configs are properly updated, when the services restart they will return a cert with the SAN `DNS:*.example.test`,  which is valid for `mail.example.test`, however the test function did not properly account for this in the regexp query.

Resolved by truncating the left-most DNS label from FQDN and adding a third check to match a returned wildcard DNS result.

Extracted out the common logic to create the regexp query and renamed the methods to communicate more clearly that they check the FQDN is supported, not necessarily explicitly listed by the cert.

* tests(letsencrypt): Enable remaining tests

These will now pass. Adjusted comments accordingly.

Added an additional test on a fake FQDN that should still be valid to a wildcard cert (SNI validation in a proper setup would reject the connection afterwards).

Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
2022-04-18 22:52:50 +12:00
Georg Lauterbach b1594a8b1c
log/scripts: introduce proper log level fallback and env getter function (#2506)
This PR does two small things:

1. The log level, in case it is unset, will now be "calculated" from
   `/etc/dms-settings` and not always default to `info`. This way, we
   can ensure that more often than not, the log level the user chose
   when starting DMS is used everywhere.
2. I noticed that the way I obtained the log level could be used to
   obtain any env variable's log level. I therefore added a function to
   `utils.sh` in case we use it in the future.
2022-04-05 17:10:01 +02:00
Georg Lauterbach c7b16a599c
log: adjust level and message(s) slightly for four messages (#2532)
Co-authored-by: Casper <casperklein@users.noreply.github.com>
2022-04-05 16:21:07 +02:00
Georg Lauterbach 35fb744ffb
scripts: refactored check-for-changes.sh (#2498)
* refactored `check-for-changes.sh`

I refactored `check-for-changes.sh` and used the new log. `_notify` can
therefore be deleted as it is used no more.
I opted to source `/etc/dms-settings` as a whole to
future-proof the script. When the DNS adjustments PRs (that do not exist
by now but will exit in the future) are done, we can then remove
`_obtain_hostname_and_domainname` because we're already writing the
variables to `/etc/dms-settings`. I left instructions in the script in
the form of TODO comments.

Because we now log the date for all messages of the changedetector, we
need to `tail` a bit more log than before.
2022-04-02 19:39:15 +02:00
Georg Lauterbach 7721a48b9b
scripts: refactoring & miscellaneous small changes (#2499)
* `update-check.sh` now uses the new log

* refactored `setup-stack.sh`

The changes are:

1. Replaced `""` wiht `''` where possible (reasoning: Bash is very
   implicit and I'd like to use `''` where possible to indicate no
   variables are expanded here)
2. `> /file` -> `>/file` according to our style guide
3. Some log adjustments for messages where I deemed it appropriate
4. Then, an error message from a Dovecot setup was also prevented (by
   adding a check whether the directory is present before a `: >...`
   command would create a file in this directory).

These are all small, miscellaneous changes that I wanted to combine into
one commit and ultimately one PR because I see no point in opening a PR
for every small change here. I hope this is fine.

* added a small `sleep` to the `_shutdown` function

This ensure the last log message is actually logged before Supervisor
logs the message that it received a SIGTERM. This makes reading the log
easier because now the causal relationship is shown (we are terminating
Supervisor, and not someone else and we're just logging it).

I forgot to replace `""` with `''` in `update-check.sh`, so I included
it here because this is the last commit before PR review.

* re-add exit on successful update (only)

* re-added date information to update-check log messages

* added `_log_with_date` function

The new function will log a message with a proper timestamp. This is all
handled in `log.sh`, we therefore not need to source other files too.

This will be used in the future by `check-for-changes.sh` as well :)

Co-authored-by: Casper <casperklein@users.noreply.github.com>
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
2022-03-26 10:17:08 +01:00
Georg Lauterbach b9dbec3276
scripts: refactored scripts located under target/bin/ (#2500)
* refactored scripts located under `target/bin/`

The scripts under `target/bin/` now use the new log and I replaced some
`""` with `''` on the way. The functionality stays the same, this mostly
style and log.

* corrected fail2ban (script and tests)

* corrected OpenDKIM log output in tests

* reverted (some) changes to `sedfile`

Moreover, a few messages for BATS were streamlined and a regression in
the linting script reverted.

* apple PR feedback

* improve log output from `fail2ban` script

The new output has a single, clear message with the '[  ERROR  ]  '
prefix, and then output that explains the error afterwards. This is
coherent with the logging style which should be used while providing
more information than just a single line about IPTables not functioning.

* simplified `setquota` script

* consistently named the `__usage` function

Before, scripts located under `target/bin/` were using `usage` or
`__usage`. Now, they're using `__usage` as they should.

* improved `sedfile`

With `sedfile`, we cannot use the helper functions in a nice way because
it is used early in the Dockerfile at a stage where the helper scripts
are not yet copied. The script has been adjusted to be canonical with
all the other scripts under `target/bin/`.

* fixed tests

* removed `__usage` from places where it does not belong

`__usage` is to be used on wrong user input, not on other failures as
well. This was fixed in `delquota` and `setquota`.

* apply PR review feedback
2022-03-26 09:30:09 +01:00
Georg Lauterbach b7bf40ac21
renamed function _errex -> _exit_with_error (#2497) 2022-03-21 15:01:07 +01:00
Georg Lauterbach 24031ae365
scripts: new log (#2493)
* added new `_log` function

With `_log`, the `_notify` method wa rendered obsolete. `_notify` was
not completely removed due to test failures in `check-for-changes.sh`.

The new `_log` function properly uses log levels such as `trace`,
`debug`, `info`, `warn` and `error`. It provides a cleaner solution
and renders `DMS_DEBUG` obsolete too (as only `_notify` depends on it).

* converted all helper script to new `_log` function

* converted all startup stacks to new `log` function

* `start-mailserver.sh` now uses new `_log` function

* final test and misc small script adjustments

* updated documentation
2022-03-21 07:07:52 +01:00
NorseGaud 1ab8ea96b2
acme_extract -> acme_extract.py + F_BIN to avoid .py 2022-03-06 22:17:29 -05:00
Casper a8a8c859dc
Rename config examples directory (#2438) 2022-03-02 22:54:14 +00:00
Georg Lauterbach 399284c092
fix: do not add accounts that already exists to account files (#2419)
Co-authored-by: Casper <casperklein@users.noreply.github.com>
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
2022-02-21 13:31:03 +01:00
Georg Lauterbach b61dfe1e24
refactoring: split helper functions into smaller scripts (#2420) 2022-02-21 11:56:57 +01:00
Georg Lauterbach ec8b99335e
Add changedetector functionality for ${SSL_TYPE} == manual (#2404)
Now, setups that use `SSL_TYPE=manual` will profit from the changedetector as well. Certificate changes are picked up and properly propagated.
2022-02-18 11:29:51 +01:00
Georg Lauterbach ce9e86c462
fix: double occurence of /etc/postfix/regexp (#2397) 2022-02-12 09:20:45 +13:00
Georg Lauterbach 99cc9fec2a
Updated ShellCheck to 0.8.0 and Hadolint to 2.8.0 (#2329)
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
2021-12-19 11:56:22 +01:00
Casper a8a39a0a1a
Rename function and adjust comment (#2306) 2021-11-28 17:02:07 +01:00
Brennan Kinney 5254f7c658
fix: check-for-changes.sh should not fall out of sync with shared logic (#2260)
Removes duplicate logic from `check-for-changes.sh` that is used/maintained elsewhere to avoid risk of problems, as this code is already starting to diverge / rot.

---

Previously the change detection support has had code added for rebuilding config upon change detection which is the same as code run during startup scripts. Unfortunately over time this has fallen out of sync. Mostly the startup scripts would get maintenance and the contributor and reviewers may not have been aware of the duplicate code handled by `check-for-changes.sh`.

That code was starting to diverge in addition to some changes in structure (_eg: relay host logic seems interleaved here vs separated out in startup scripts_). I wanted to address this before it risks becoming a much bigger headache.

Rather than bloat `helper-functions.sh` further, I've added a `helpers/` folder extracting relevant common logic between startup scripts and `changedetector`. If you want to follow that process I've kept scoped commits to make those diffs easier. Some minor changes/improvements were added but nothing significant.

---

- chore: Extract relay host logic to new `relay.sh` helper
- chore: Extract `/etc/postfix/sasl_passwd` logic to new `sasl.sh` helper
- chore: Extract `postfix-accounts.cf` logic to new `accounts.sh` helper
- chore: Extract `/etc/aliases` logic to new `aliases.sh` helper
- chore: Extract `/etc/postfix/vhost` logic to new `postfix.sh` helper

- chore: Add inline docs for Postfix configs
> These are possibly more verbose than needed and can be reduced at a later stage.
> They are helpful during this refactor process while investigating that everything is handled correctly.

`accounts.sh`: 
- Add note regarding potential bug for bare domain setups with `/etc/postfix/vhost` and `mydestination` sharing same domain value.

`relay.sh`: 
- Remove the tabs for a single space delimiter, revised associated comment.
- Add PR reference for original `_populate_relayhost_map` implementation which has some useful details.


Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
Co-authored-by: Casper <casperklein@users.noreply.github.com>
2021-11-21 09:33:49 +13:00