ChatMail suite: building a hardened Delta Chat relay on eijo.im

Host: lca-01.kmsp42.com (38.19.201.101) Domain: eijo.im Upstream: deltachat/chatmail Live stats: stats.eijo.im

1. Architecture Overview

eijo.im is a Delta Chat chatmail relay — a purpose-built mail server optimized for Delta Chat’s encrypted messaging protocol. It runs on lca-01 (formerly lis-01, repurposed from EU hub duty) and participates in the FMS WireGuard mesh but does not run pg-router, Caddy, PostgreSQL, or BIRD.

Service Stack

Service	Port(s)	Purpose
Postfix	25, 587, 10025/6	SMTP relay, submission, filtermail reinjection
Dovecot	993, 143	IMAP/S mailbox access
OpenDKIM	(milter socket)	DKIM signing for outbound mail
Nginx	80, 443, 8443, 8484	HTTP/S, ALPN mux, stats reverse proxy, stub
filtermail	10080	Outbound content filter
filtermail-incoming	10081	Inbound content filter
chatmail-metadata	(unix socket)	Account metadata service
doveauth	(unix socket)	Dovecot authentication bridge
iroh-relay	(via nginx 443)	Iroh p2p relay for Delta Chat
mtail	3903	Log-derived mail metrics
Prometheus	9090	Metrics collection
Grafana	3000	Dashboards (proxied at stats.eijo.im)
node-exporter	9100	System metrics
nginx-exporter	9113	Nginx connection/request metrics
postfix-exporter	9154	Mail queue and delivery metrics

Port 443 ALPN Multiplexing

Nginx uses the stream module with ssl_preread to multiplex port 443 by ALPN protocol negotiation:

h2, http/1.1 → upstream 127.0.0.1:8443 (HTTPS: chatmail web UI, stats.eijo.im, iroh-relay)
No ALPN / IMAP → upstream 127.0.0.1:993 (IMAPS via Dovecot)
SMTP → upstream 127.0.0.1:465 (SMTPS via Postfix)

This allows Delta Chat clients to reach IMAP and SMTP over port 443, bypassing restrictive firewalls that only allow HTTPS traffic.

2. Installation

2.1 Initial Deployment (cmdeploy)

ChatMail is installed via cmdeploy, Delta Chat’s deployment tool. This is an out-of-band operation — not managed by Ansible:

# On the control machine:
git clone https://github.com/deltachat/chatmail.git /opt/chatmail
cd /opt/chatmail
pip install -e .
cmdeploy run --chatmail-ini /path/to/chatmail.ini eijo.im

cmdeploy run performs a full installation: installs system packages, configures Postfix, Dovecot, OpenDKIM, Nginx, filtermail, iroh-relay, sets up acmetool for TLS certificates, and starts all services. Subsequent runs are idempotent and apply configuration updates.

2.2 chatmail.ini

The canonical configuration lives in the Ansible repository at registrar-ansible/chatmail.ini and is deployed to /usr/local/lib/chatmaild/chatmail.ini by the chatmail_relay Ansible role.

Key parameters:

[params]
mail_domain = eijo.im

# Rate limiting
max_user_send_per_minute = 60
max_user_send_burst_size = 10

# Storage limits
max_mailbox_size = 500M
max_message_size = 31457280    # 30 MiB

# Retention policy
delete_mails_after = 20        # days
delete_large_after = 7         # days (messages > 200 KiB)
delete_inactive_users_after = 90

# Account constraints (9-char random usernames, 9-char passwords)
username_min_length = 9
username_max_length = 9
password_min_length = 9

# Filtermail ports (internal, loopback only)
filtermail_smtp_port = 10080
postfix_reinject_port = 10025
filtermail_smtp_port_incoming = 10081
postfix_reinject_port_incoming = 10026

# TLS via acmetool
acme_email = admin@registrar.earth

# Metrics
mtail_address = 127.0.0.1

Mail flow: Postfix receives on port 25/587 → filtermail (10080/10081) inspects and re-injects via 10025/10026 → Postfix delivers locally or relays outbound. OpenDKIM signs all outbound mail via a Postfix milter.

3. DNS Configuration

All DNS for eijo.im is served by pg-router on ns-01 and ns-02 via the dns_resolve() PostgreSQL function. Records are stored in the fms database.

Zone Records

Name	Type	Value	TTL
eijo.im.	A	38.19.201.101	300
eijo.im.	MX	10 eijo.im.	300
eijo.im.	NS	ns-01.registrar.earth. / ns-02.registrar.earth.	300
eijo.im.	SOA	ns-01.registrar.earth. admin.registrar.earth. 2026031803 …	300
eijo.im.	TXT	v=spf1 a mx -all	300
www.eijo.im.	A	38.19.201.101	300
stats.eijo.im.	A	38.19.201.101	300
mta-sts.eijo.im.	A	38.19.201.101	300
opendkim._domainkey.eijo.im.	TXT	v=DKIM1; h=sha256; k=rsa; p=MIIBIjAN… (OpenDKIM public key)	300
_dmarc.eijo.im.	TXT	v=DMARC1; p=reject; adkim=s; aspf=s; rua=mailto:admin@registrar.earth	300
_mta-sts.eijo.im.	TXT	v=STSv1; id=20260318	300

Email Authentication Chain

SPF (v=spf1 a mx -all) — only the mail server’s own IP may send for this domain
DKIM (OpenDKIM, selector opendkim, RSA, sha256) — cryptographic signature on every outbound message
DMARC (p=reject; adkim=s; aspf=s) — strict alignment for both DKIM and SPF; reject on failure; aggregate reports to admin@registrar.earth
MTA-STS — enforces TLS for inbound mail delivery; policy served at https://mta-sts.eijo.im/.well-known/mta-sts.txt

4. Nginx Configuration

Nginx serves multiple roles on lca-01:

4.1 Stream Layer (Port 443 ALPN Mux)

stream {
    map $ssl_preread_alpn_protocols $backend {
        ~\bh2\b           web_https;
        ~\bhttp/1.1\b     web_https;
        default            legacy_tls;
    }
    upstream web_https   { server 127.0.0.1:8443; }
    upstream legacy_tls  { server 127.0.0.1:465;  }  # SMTPS / IMAPS fallback

    server {
        listen 443;
        listen [::]:443;
        ssl_preread on;
        proxy_pass $backend;
    }
}

4.2 HTTPS Server Blocks (Port 8443)

Two server blocks listen on 127.0.0.1:8443:

eijo.im — chatmail web UI + iroh-relay + ACME + MTA-STS:

TLS via acmetool (/var/lib/acme/live/eijo.im/)
Serves the chatmail web pages, .well-known/mta-sts.txt, ACME challenges
Proxies iroh-relay WebSocket connections

stats.eijo.im — Grafana reverse proxy:

TLS via certbot (/etc/letsencrypt/live/stats.eijo.im/)
Proxies to Grafana on 127.0.0.1:3000 with WebSocket upgrade support
Headers: X-Real-IP, X-Forwarded-For, X-Forwarded-Proto

4.3 Stub Status (Port 8484)

Internal-only server for prometheus-nginx-exporter:

server {
    listen 127.0.0.1:8484;
    location /nginx_status {
        stub_status;
        allow 127.0.0.1;
        deny all;
    }
}

4.4 TLS Certificates

Domain	Provider	Path	Renewal
eijo.im	acmetool	/var/lib/acme/live/eijo.im/	Automatic (acmetool)
www.eijo.im	acmetool	(SAN on eijo.im cert)	Automatic
mta-sts.eijo.im	acmetool	(SAN on eijo.im cert)	Automatic
stats.eijo.im	certbot	/etc/letsencrypt/live/stats.eijo.im/	certbot renew (cron)

Note: stats.eijo.im uses certbot rather than acmetool because acmetool’s challenge negotiation failed when a 4th SAN was added. certbot was installed as a targeted fallback for this single subdomain.

5. Monitoring Stack

5.1 Prometheus

Config: /etc/prometheus/prometheus.yml

Five scrape jobs, all with alias: "eijo.im" label:

Job	Target	Metrics Source
prometheus	localhost:9090	Prometheus self-monitoring
chatmail-mtail	localhost:3903	mtail — mail delivery, encryption, DKIM, errors
node	localhost:9100	node-exporter + textfile collector
nginx	localhost:9113	nginx-exporter (connections, requests)
postfix	localhost:9154	postfix-exporter (queue, delivery latency)

5.2 Grafana

Config: /etc/grafana/grafana.ini

Binds 127.0.0.1:3000 (not publicly exposed — reverse proxied via Nginx)
Domain: stats.eijo.im, root URL: https://stats.eijo.im/
Anonymous auth enabled with Viewer role (public read-only access)
Sign-up disabled, analytics/reporting disabled

Dashboards:

ChatMail - eijo.im (uid: chatmail-eijo-im, home dashboard)
- 12 panels from ccclxxiii/chatmail-grafana-dashboard
- Mail delivery rate, encryption rate, DKIM signing rate, account creation
- Quota/warning/error rates, cumulative counters
Infrastructure - eijo.im (uid: infra-eijo-im)
- 16 panels: CPU/RAM/disk stat panels, TLS cert expiry gauge
- Fail2ban banned IPs, mail queue depth, IMAP connected users
- CPU over time, memory breakdown (used/buffers/cached/free)
- Public network I/O, WireGuard mesh I/O
- Nginx connections and requests per second
- WireGuard handshake age, fail2ban timeline, mail queue timeline

5.3 Custom Textfile Collector

Script: /usr/local/bin/collect-extra-metrics.sh Cron: /etc/cron.d/prometheus-extra-metrics — runs every minute as root Output: /var/lib/prometheus/node-exporter/extra.prom

Metrics collected:

Metric	Labels	Source
`fail2ban_banned_total`	jail	`fail2ban-client status <jail>`
`fail2ban_bans_total`	jail	`fail2ban-client status <jail>`
`wireguard_last_handshake_seconds`	peer	`wg show wg0 latest-handshakes`
`wireguard_received_bytes_total`	peer	`wg show wg0 transfer`
`wireguard_sent_bytes_total`	peer	`wg show wg0 transfer`
`postfix_queue_depth`	—	`find /var/spool/postfix`
`dovecot_connected_users`	—	`doveadm who`
`tls_cert_expiry_days`	domain	`openssl x509 -enddate`

The script writes to a temp file and atomically moves it to the .prom path to avoid partial reads by node-exporter.

6. Ansible Integration

6.1 Inventory

lca-01 is in the chatmail host group. Its host_vars (host_vars/lca-01.kmsp42.com.yml) define:

Firewall rules for SSH, WireGuard, SMTP (25), SMTPS (465), submission (587), IMAPS (993), IMAP (143), HTTP (80), HTTPS (443)
No Caddy, pg-router, PostgreSQL, BIRD, or backend variables — those services are not present on this node

6.2 chatmail_relay Role

Path: roles/chatmail_relay/

defaults/main.yml

chatmail_domain: eijo.im
chatmail_config_src: "{{ playbook_dir }}/chatmail.ini"
chatmail_config_dest: /usr/local/lib/chatmaild/chatmail.ini
chatmail_repo: https://github.com/deltachat/chatmail.git
chatmail_deploy_dir: /opt/chatmail
chatmail_mtail_enabled: true
chatmail_mtail_address: "127.0.0.1"
chatmail_mtail_port: 3903

tasks/main.yml

Deploy chatmail.ini — copies config from Ansible repo to /usr/local/lib/chatmaild/chatmail.ini; notifies restart handlers for Postfix, Dovecot, filtermail, filtermail-incoming on change.
Ensure mtail is enabled and running — starts and enables the mtail systemd unit when chatmail_mtail_enabled is true.
Verify core services are running — loops over 8 services (postfix, dovecot, opendkim, nginx, filtermail, filtermail-incoming, chatmail-metadata, doveauth) and asserts each is active via systemctl is-active. Tagged [verify].
Verify mail ports — 4 tasks checking that ports 25, 465, 587, and 993 are listening via ss -lntp. Tagged [verify].
Verify mtail metrics endpoint — HTTP GET to http://127.0.0.1:3903/metrics expecting 200. Tagged [verify].

handlers/main.yml

Restart handlers for: postfix, dovecot, opendkim, mtail, nginx, filtermail, filtermail-incoming.

6.3 site.yml Integration

The chatmail play runs after the replica verification block and before the final localhost HTTPS verification:

# ── Chatmail relay (eijo.im) ──
- hosts: chatmail
  become: true
  roles:
    - role: firewall
      tags: [firewall]
    - role: ssh_hardening
      tags: [ssh]
    - role: chatmail_relay
      tags: [chatmail]

This applies the shared firewall and ssh_hardening roles (consistent with all other nodes in the mesh), then the chatmail-specific chatmail_relay role.

Downloads

chatmail.ini — chatmail configuration
tasks/main.yml — task playbook
defaults/main.yml — role defaults
handlers/main.yml — restart handlers

7. systemd Service Hardening

All monitoring services have systemd drop-in override files at /etc/systemd/system/<service>.service.d/hardening.conf. These reduce the attack surface from default exposure scores of ~9.5 to 1.3–1.9.

Common Hardening Directives

Applied to all monitoring services unless noted:

NoNewPrivileges=true
PrivateTmp=true
PrivateDevices=true
ProtectHome=true
ProtectClock=true
ProtectHostname=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectKernelLogs=true
ProtectControlGroups=true
ProtectProc=invisible
RestrictNamespaces=true
RestrictRealtime=true
RestrictSUIDSGID=true
LockPersonality=true
MemoryDenyWriteExecute=true
SystemCallArchitectures=native

Per-Service Notes

Service	ProtectSystem	Special Considerations
prometheus	strict	`ReadWritePaths=/var/lib/prometheus` for TSDB storage
grafana-server	strict	`ReadWritePaths=/var/lib/grafana /var/log/grafana /run/grafana`
prometheus-node-exporter	full	`CapabilityBoundingSet=` (empty — no capabilities needed)
prometheus-nginx-exporter	strict	`ProtectProc=invisible`
prometheus-postfix-exporter	full	Read-only access to Postfix journal logs
mtail	strict	`DynamicUser=true` (no static user), `SupplementaryGroups=systemd-journal`, `ProtectKernelLogs=false` (needs journal access), `UMask=0077`

SystemCallFilter

All services use SystemCallFilter=@system-service which allows the standard set of system calls needed for network services while blocking dangerous calls like @mount, @reboot, @swap, @raw-io, @clock, @debug, @module, @obsolete, and @privileged.

Security Score Improvement

Service	Before	After
prometheus	9.6	1.5
grafana-server	9.6	1.5
prometheus-node-exporter	9.6	1.3
prometheus-nginx-exporter	9.6	1.4
prometheus-postfix-exporter	9.6	1.3
mtail	9.6	1.9

Scores from systemd-analyze security <service>. Lower is better; scores below 2.0 are considered “OK” by systemd’s own rating.

Downloads — chatmail_hardening role

tasks/main.yml — task playbook
defaults/main.yml — service list
handlers/main.yml — daemon-reload and restart handlers
Drop-in configs:

8. Firewall Rules

UFW is managed by the shared firewall Ansible role using rules from host_vars:

Port	Proto	Purpose
22	TCP	SSH (key-only, hardened)
WG	UDP	WireGuard mesh (dynamic port)
25	TCP	SMTP inbound
465	TCP	SMTPS (implicit TLS)
587	TCP	Submission (STARTTLS)
993	TCP	IMAPS
143	TCP	IMAP (STARTTLS)
80	TCP	HTTP (ACME challenges, web UI)
443	TCP	HTTPS (web, iroh-relay, ALPN mux to IMAP/SMTP)

Monitoring services (Prometheus 9090, Grafana 3000, mtail 3903, exporters) bind to 127.0.0.1 only and are not exposed through the firewall. Grafana is accessible publicly only through the Nginx reverse proxy at stats.eijo.im.

9. WireGuard Mesh Participation

lca-01 participates in the FMS WireGuard mesh on fd53::/16 ULA addressing. This provides:

Encrypted backhaul to all other mesh nodes
Reachability for internal metrics scraping (if needed from other nodes)
Consistent SSH access via mesh addresses
WireGuard handshake age and transfer metrics are collected by the textfile collector

The node does not run BIRD (no BGP announcements) since it has no pg-router or resolver services to advertise.

10. Operational Notes

Upgrades

ChatMail upgrades are performed via cmdeploy run from /opt/chatmail, not via Ansible. The Ansible role only manages the configuration file and verifies service health.

cd /opt/chatmail
git pull
cmdeploy run --chatmail-ini /usr/local/lib/chatmaild/chatmail.ini eijo.im

Certificate Renewal

eijo.im, www.eijo.im, mta-sts.eijo.im — acmetool handles renewal automatically via its own timer/cron
stats.eijo.im — certbot handles renewal via systemd timer (certbot.timer)

Log Monitoring

mtail parses Postfix and Dovecot journal logs to produce OpenMetrics counters. These feed the “ChatMail - eijo.im” Grafana dashboard. Key signals to watch:

chatmail_deliveries_total — mail delivery rate (should be non-zero during active use)
chatmail_delivery_errors_total — delivery failures (should be near zero)
chatmail_encrypted_ratio — fraction of encrypted messages (should be near 1.0 for Delta Chat traffic)
chatmail_dkim_signed_total — DKIM signing rate (should match outbound volume)

Health Verification

Run the Ansible verify tags to check all services:

ansible-playbook site.yml --tags verify --limit chatmail

This checks: 8 systemd services active, 4 mail ports listening, mtail metrics endpoint responding.