RFC 9991 failure reporting: Identity-Alignment, DKIM-Canonicalized-Header/-Body, ruf= verification/rate-limiting

DMARCBIS-REMAINING-WORK.md

@@ -0,0 +1,148 @@
+# DMARCbis remaining work
+
+Tracks what's left across RFC 9989 (DMARC core), RFC 9990 (aggregate
+reporting), and RFC 9991 (failure reporting) — the documents that obsolete
+RFC 7489. Source of truth for the overall gap analysis is GitHub issue
+[#371](https://github.com/trusteddomainproject/OpenDMARC/issues/371)
+(`trusteddomainproject/OpenDMARC`); this file exists because that issue
+predates most of the work below and doesn't get rewritten as items close.
+Copies of the RFCs themselves are in the repo root (`rfc7489.txt`,
+`rfc9989.txt`, `rfc9990.txt`, `rfc9991.txt`). `DMARCBIS-WALK-NOTES.txt` and
+`DMARCBIS-EDITOR-EMAIL.txt` cover the DNS Tree Walk design questions
+specifically (all resolved) and are not duplicated here.
+
+## Done
+
+- **RFC 9989 DNS Tree Walk**: walk-mode selection (PSL/RFC7489/RFC9989/AUTO),
+  configurable fallback, secondary alignment walk, `opendmarc-check`
+  comparison tooling. All open questions in `DMARCBIS-WALK-NOTES.txt`
+  resolved. (#430, #431, #433)
+- **RFC 9989 `t=`/`pct=`**: `t=` parsing, fetch accessor, and enforcement
+  step-down (reject->quarantine->none). `pct=` deliberately kept for POLA,
+  with `DMARCbisIgnorePct` for operators who want strict compliance. (#434)
+- **RFC 9990 aggregate reporting**: `np`/`testing`/`discovery_method` in
+  `policy_published`, namespace bumped to `dmarc-2.0`, `<pct>` removed
+  (branch `feat/rfc9990-aggregate-reporting`).
+- **RFC 9991 failure reporting**: `Identity-Alignment` and
+  `DKIM-Domain`/`-Identity`/`-Selector` ARF headers, `ruf=` external
+  destination verification + rate-limiting in `opendmarc-reports
+  --forensic`, `psd=y` excludes `ruf=` (branch
+  `feat/rfc9991-forensic-reporting`). Also restored PR #392's `rua=`
+  destination verification, which had been silently lost, and fixed a
+  NOERROR-vs-NXDOMAIN bug in it found via live testing.
+- **RFC 9991 `DKIM-Canonicalized-Header`/`-Body` ARF fields**: new
+  `ReadCanonicalizedData` option reads `X-DKIM-Canonicalized-Header`/`-Body`
+  staging headers from an upstream OpenDKIM running the new
+  `AddCanonicalizedData` directive (`trusteddomainproject/OpenDKIM` PR
+  #423), correlates them to whichever DKIM signature is blamed for a DMARC
+  failure, and copies the base64 payload into the failure report,
+  re-folded to this codebase's own ARF-body convention. Staging headers
+  are always stripped before final delivery regardless of the setting.
+  See CHANGES-202605.md for the full writeup; cross-project design
+  history preserved below.
+- **RFC 9991 `SPF-DNS` ARF field**: new opt-in `LogSPFDNS` option (only
+  meaningful with `SPFSelfValidate`). Real new code, as anticipated below
+  -- libspf2 has a pluggable "DNS layer" system built exactly for this
+  kind of interception (`SPF_dns_server_t`, chainable via `layer_below`);
+  a new logging layer (`libopendmarc/opendmarc_spf_dns_log.c`) is spliced
+  in above libspf2's own caching layer so it sees every SPF-record lookup
+  regardless of cache hits, including domains reached via
+  `include:`/`redirect=`, without altering evaluation behavior. This
+  closes RFC 9991's SPF-DNS gap entirely; both fields originally listed
+  as blocked in this section are now done. See CHANGES-202605.md for the
+  full writeup.
+
+## Remaining
+
+### RFC 9990 aggregate reporting
+
+- **`pass` disposition value** (S3.1.1.9): `ActionDispositionType` now
+  includes `pass` (message passed DMARC under an *enforcing* policy)
+  alongside `none`/`quarantine`/`reject`. Current code only ever emits the
+  original three. Not touched by the `np`/`testing`/`discovery_method` work.
+- **`policy_test_mode` reason type** (S3.1.6): a `<reason><type>` value a
+  report record should carry when `t=y` caused a policy step-down. Directly
+  adjacent to work already done — `t=` enforcement and `<testing>` in
+  `policy_published` both shipped, but this per-record annotation didn't.
+- **`generator` element** (S3.1.1.3): identifies the report-generating
+  software. Not implemented in `opendmarc-reports`. (Unrelated: the
+  separate `contrib/dmarc-report-totext.pl` *consumer* tool already parses
+  this field from other senders' reports — that's reading, not writing.)
+- **`error` element** (S3.1.1.3/S3.1.5): describes processing errors
+  encountered while evaluating the DMARC Policy Record. Not implemented.
+- **DKIM signature priority + 100-signature cap** (S3.1.3): defines which
+  signatures to include when a message has several (strict-aligned pass
+  first, then relaxed, then others) and caps the list. Not implemented as
+  specified.
+- **Extension mechanism** (S3.2, S5): `<extension>` at file level,
+  namespaced elements at record level. Low priority — only matters if
+  extensions are actually adopted by report consumers.
+- **`rf=`/`ri=` cleanup**: both tags were removed from the DMARC record
+  format by RFC 9989. OpenDMARC still parses them into unused
+  `DMARC_POLICY_T` fields (`rf`, `ri`). Minor; safe to remove.
+
+### RFC 9991 failure reporting
+
+Both RFC 6591 fields originally tracked here (`DKIM-Canonicalized-Header`/
+`-Body` and `SPF-DNS`) have shipped -- see "Done" above. Nothing remaining
+in this section.
+
+### Bugs found along the way
+
+- **`t-verify-authservid-jobid` test-ordering bug, fixed**: found while
+  live-testing the `LogSPFDNS` work (that's what prompted actually
+  getting a real-libspf2 build going, via `--with-spf
+  --with-spf2-include=... --with-spf2-lib=...` -- the environment used
+  for most of this DMARCbis work had been silently building against the
+  built-in fallback SPF evaluator instead, an easy mistake since
+  `--with-spf2` is a *different*, unrecognized flag that `configure`
+  silently ignores rather than erroring on). Root-caused with `ktrace`
+  (confirmed byte-perfect wire delivery of the macro) and `lldb`
+  (breakpoints in both OpenDMARC and, via its exported-but-internal
+  symbols, real libmilter itself): the test sends the `i` (job ID) macro
+  scoped to the MAIL stage, then calls `mt.mailfrom()` without ever
+  having called `mt.helo()` -- so miltertest auto-inserts a filler HELO
+  command at that point, per its documented "fill in skipped steps"
+  behavior. Real libmilter's `st_helo()` handler unconditionally clears
+  any macros already stored for *later* protocol stages
+  (`mi_clr_macros(ctx, CI_HELO+1)`, a correct, by-design safeguard
+  against stale macros surviving from a prior transaction on the same
+  connection) -- which wipes the just-stored MAIL-stage macro before it's
+  ever read back, since it arrived earlier than the (delayed) HELO it's
+  nominally supposed to follow. Not a bug in miltertest, real libmilter,
+  or OpenDMARC's C code -- purely a test-script ordering issue, present
+  identically in every test in this suite that sends the `i` macro this
+  way (all of them), just never noticed elsewhere since no other test
+  asserts on the delivered value. Fixed by adding an explicit
+  `mt.helo()` call before the macro send in every affected test file
+  (all eleven `t-verify-*.lua` files that send the `i` macro), not just
+  the one that was actually failing.
+- **`--with-spf2` now a hard configure error**: the flag that caused the
+  above (silently building against the built-in fallback SPF evaluator
+  instead of real libspf2, since `--with-spf` is the real flag and
+  `--with-spf2` doesn't exist) now fails `configure` immediately with a
+  message pointing at the correct flags, instead of the default
+  autoconf behavior of a warning + silent continue. Verified both that
+  `--with-spf2` now errors and that the legitimate
+  `--with-spf --with-spf2-include=... --with-spf2-lib=...` combination
+  still configures `HAVE_SPF2_H`/`WITH_SPF` correctly.
+- **Four config directives silently rejected at startup**: fixed in the
+  same session (`opendmarc-config.h`'s `dmarcf_config[]` validation table
+  was missing `DMARCbisIgnorePct`, `DMARCbisWalkModeFallback`,
+  `ReadCanonicalizedData`, and `LogSPFDNS`) -- see CHANGES-202605.md. Worth
+  a standing reminder: this table is hand-maintained and separate from
+  every `config_get()` call site, so a new directive silently fails at
+  startup instead of a compile error until someone actually parses a
+  config file containing it, which apparently hadn't happened for three
+  of these four since they were introduced.
+
+### Open decision, not just missing code
+
+- **`!NNNk` RUA size-suffix syntax**: RFC 9990 Appendix C says this syntax
+  is obsolete and receivers (i.e. OpenDMARC, acting as report generator)
+  MUST ignore it. The restored PR #392 code (`check_size_restriction`)
+  actively parses and enforces it, faithfully reproducing pre-regression
+  behavior rather than the RFC 9990 text. Same shape of question as the
+  `pct=` POLA decision: keep honoring it for senders who rely on it, or
+  drop it now that it's back in front of you. Needs an explicit call, not
+  a default.

configure.ac

@@ -360,6 +360,19 @@ AC_ARG_WITH([spf],
 	AC_DEFINE(WITH_SPF, 1, [Define to 1 if you want SPF support.]),
 	[])
 
+dnl There is no "--with-spf2" flag -- SPF support is enabled with
+dnl --with-spf, and libspf2 (as opposed to the built-in fallback
+dnl evaluator) is selected by additionally pointing at it via
+dnl --with-spf2-include/--with-spf2-lib below. Autoconf silently ignores
+dnl unrecognized --with-* options by default (a warning, not an error),
+dnl so this exact, easy typo has previously caused builds to silently
+dnl fall back to the built-in evaluator instead of libspf2. Fail loudly
+dnl instead.
+AC_ARG_WITH([spf2],
+	AS_HELP_STRING([--with-spf2], [(invalid; see --with-spf, --with-spf2-include, --with-spf2-lib)]),
+	[AC_MSG_ERROR([[--with-spf2 is not a valid option. Use --with-spf to enable SPF checking; use --with-spf2-include=path and --with-spf2-lib=path to select libspf2 over the built-in fallback evaluator.]])],
+	[])
+
 AC_ARG_WITH([spf2-include],
 	AS_HELP_STRING([--with-spf2-include], [path to libspf2 includes]),
 	SPF2_INCLUDE="$withval",

contrib/opendmarc-reports.conf.sample

@@ -60,6 +60,17 @@
 # attachment) for later review, e.g. RFC-compliance checking.
 #archive-dir = /var/spool/opendmarc/sent
 
+# --- External destination verification (RFC 9990 S4 / RFC 9991 S5) ---
+# Same file used by the milter's PublicSuffixList; used to recognize
+# same-organization RUA/RUF destinations without a DNS query.
+#public-suffix-list = /etc/opendmarc/public_suffix_list.dat
+
+# --- Forensic report rate limiting (RFC 9991 S2/S8.1) ---
+# Uncomment to cap failure reports per destination address. Unset (no
+# limit) by default.
+#forensic-rate-limit = 10
+#forensic-rate-window = 3600
+
 # --- Miscellaneous ---
 #interval = 86400
 #utc = 1

db/schema.mysql

@@ -64,8 +64,11 @@ CREATE TABLE IF NOT EXISTS requests (
 	aspf TINYINT NOT NULL DEFAULT '0',
 	policy TINYINT NOT NULL DEFAULT '0',
 	spolicy TINYINT NOT NULL DEFAULT '0',
+	npolicy TINYINT NOT NULL DEFAULT '0',
 	pct TINYINT NOT NULL DEFAULT '0',
 	fo TINYINT NOT NULL DEFAULT '0',
+	testing TINYINT NOT NULL DEFAULT '0',
+	discovery_method TINYINT NOT NULL DEFAULT '0',
 	locked TINYINT NOT NULL DEFAULT '0',
 	firstseen TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
 	lastsent TIMESTAMP NULL DEFAULT NULL,
@@ -148,5 +151,16 @@ CREATE TABLE IF NOT EXISTS suppressions (
 	UNIQUE KEY(address)
 );
 
+-- RFC 9991 S2/S8.1: tracks outgoing failure (ruf=) report sends per
+-- recipient address, so opendmarc-reports --forensic can rate-limit them.
+CREATE TABLE IF NOT EXISTS forensic_sent (
+	id INT NOT NULL AUTO_INCREMENT,
+	address VARCHAR(255) NOT NULL,
+	sent_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
+
+	PRIMARY KEY(id),
+	KEY(address, sent_at)
+);
+
 -- CREATE USER 'opendmarc'@'localhost' IDENTIFIED BY 'changeme';
 -- GRANT ALL ON opendmarc.* to 'opendmarc'@'localhost';

db/update-db-schema.mysql

@@ -32,6 +32,13 @@ ALTER TABLE messages MODIFY COLUMN arc_policy TINYINT UNSIGNED NOT NULL DEFAULT
 -- RFC 7489 §8.2: spf_scope distinguishes mfrom vs. helo SPF evaluation.
 ALTER TABLE messages ADD COLUMN spf_scope TINYINT NOT NULL DEFAULT '-1' AFTER spf;
 
+-- RFC 9990 policy_published extensions: npolicy (non-existent-subdomain
+-- policy, the np= tag), testing (t=, RFC 9989 Appendix A.6), and
+-- discovery_method (psl|treewalk, how the record was found).
+ALTER TABLE requests ADD COLUMN npolicy TINYINT NOT NULL DEFAULT '0' AFTER spolicy;
+ALTER TABLE requests ADD COLUMN testing TINYINT NOT NULL DEFAULT '0' AFTER fo;
+ALTER TABLE requests ADD COLUMN discovery_method TINYINT NOT NULL DEFAULT '0' AFTER testing;
+
 -- VERP bounce tracking: suppressions table for persistent report address suppression.
 CREATE TABLE IF NOT EXISTS suppressions (
 	id INT NOT NULL AUTO_INCREMENT,
@@ -42,3 +49,14 @@ CREATE TABLE IF NOT EXISTS suppressions (
 	PRIMARY KEY(id),
 	UNIQUE KEY(address)
 );
+
+-- RFC 9991 S2/S8.1: tracks outgoing failure (ruf=) report sends per
+-- recipient address, so opendmarc-reports --forensic can rate-limit them.
+CREATE TABLE IF NOT EXISTS forensic_sent (
+	id INT NOT NULL AUTO_INCREMENT,
+	address VARCHAR(255) NOT NULL,
+	sent_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
+
+	PRIMARY KEY(id),
+	KEY(address, sent_at)
+);

libopendmarc/Makefile.am

@@ -13,6 +13,7 @@ libopendmarc_la_SOURCES	= dmarc.h \
                           opendmarc_strl.h \
 			  opendmarc_spf.c \
 			  opendmarc_spf_dns.c \
+			  opendmarc_spf_dns_log.c \
                           opendmarc_internal.h
 libopendmarc_la_LDFLAGS = -version-info $(LIBOPENDMARC_VERSION_INFO)
 libopendmarc_la_LIBADD = $(LIBRESOLV)

libopendmarc/dmarc.h.in

@@ -125,7 +125,16 @@ typedef struct {
 #define DMARC_RECORD_PSD_UNSPECIFIED (0)	/* psd= absent                     */
 #define DMARC_RECORD_PSD_N           (1)	/* psd=n: organizational domain    */
 #define DMARC_RECORD_PSD_Y           (2)	/* psd=y: public suffix domain     */
-
+
+/* RFC 9990 policy_published/discovery_method: how the record was found.
+ * UNSPECIFIED covers a direct match at _dmarc.<domain>, where no PSL
+ * lookup or tree walk was needed at all; the RFC only defines "psl" and
+ * "treewalk", so that case is left unreported (discovery_method is
+ * OPTIONAL) rather than forced into one of the two. */
+#define OPENDMARC_DISCOVERY_UNSPECIFIED (0)
+#define OPENDMARC_DISCOVERY_PSL         (1)	/* RFC 7489 org-domain method       */
+#define OPENDMARC_DISCOVERY_TREEWALK    (2)	/* RFC 9989 S 4.10 DNS Tree Walk    */
+
 /*
  * Library one time initialization.
  */
@@ -164,8 +173,14 @@ OPENDMARC_STATUS_T opendmarc_policy_fetch_adkim(DMARC_POLICY_T *pctx, int *adkim
 OPENDMARC_STATUS_T opendmarc_policy_fetch_aspf(DMARC_POLICY_T *pctx, int *aspf);
 OPENDMARC_STATUS_T opendmarc_policy_fetch_p(DMARC_POLICY_T *pctx, int *p);
 OPENDMARC_STATUS_T opendmarc_policy_fetch_sp(DMARC_POLICY_T *pctx, int *sp);
+OPENDMARC_STATUS_T opendmarc_policy_fetch_np(DMARC_POLICY_T *pctx, int *np);
+OPENDMARC_STATUS_T opendmarc_policy_fetch_psd(DMARC_POLICY_T *pctx, int *psd);
 OPENDMARC_STATUS_T opendmarc_policy_fetch_t(DMARC_POLICY_T *pctx, int *t);
+OPENDMARC_STATUS_T opendmarc_policy_fetch_discovery_method(DMARC_POLICY_T *pctx, int *discovery_method);
 OPENDMARC_STATUS_T opendmarc_policy_fetch_fo(DMARC_POLICY_T *pctx, int *fo);
+OPENDMARC_STATUS_T opendmarc_policy_fetch_np(DMARC_POLICY_T *pctx, int *np);
+OPENDMARC_STATUS_T opendmarc_policy_fetch_psd(DMARC_POLICY_T *pctx, int *psd);
+OPENDMARC_STATUS_T opendmarc_policy_fetch_discovery_method(DMARC_POLICY_T *pctx, int *discovery_method);
 u_char **	   opendmarc_policy_fetch_rua(DMARC_POLICY_T *pctx, u_char *list_buf, size_t size_of_buf, int constant);
 u_char **	   opendmarc_policy_fetch_ruf(DMARC_POLICY_T *pctx, u_char *list_buf, size_t size_of_buf, int constant);
 OPENDMARC_STATUS_T opendmarc_policy_fetch_utilized_domain(DMARC_POLICY_T *pctx, u_char *buf, size_t buflen);
@@ -198,8 +213,8 @@ int 		   opendmarc_policy_to_buf(DMARC_POLICY_T *pctx, char *buf, size_t buflen)
 /*
  * SPF Processing
  */
-int     opendmarc_spf_test(char *ip_address, char *mail_from_domain, char *helo_domain, char *spf_record, int soft_fail_as_pass, char *human_readable, size_t human_readable_len, int *use_mailfrom);
-int	opendmarc_spf2_test(char *ip_address, char *mail_from_domain, char *helo_domain, char *spf_record, int softfail_okay_flag, char *human_readable, size_t human_readable_len, int *used_mfrom);
+int     opendmarc_spf_test(char *ip_address, char *mail_from_domain, char *helo_domain, char *spf_record, int soft_fail_as_pass, char *human_readable, size_t human_readable_len, int *use_mailfrom, int want_dns_log, char **spf_dns_lines);
+int	opendmarc_spf2_test(char *ip_address, char *mail_from_domain, char *helo_domain, char *spf_record, int softfail_okay_flag, char *human_readable, size_t human_readable_len, int *used_mfrom, int want_dns_log, char **spf_dns_lines);
 
 #ifdef __cplusplus
 }

libopendmarc/opendmarc_internal.h

@@ -160,6 +160,7 @@ typedef struct dmarc_policy_t {
 	u_char *	from_domain;		/* Input: From: header domain */
 	u_char *	organizational_domain;
 	int		org_domain_from_fallback; /* Non-zero if PSL was absent and label-walk was used */
+	int		discovery_method;	/* RFC 9990: OPENDMARC_DISCOVERY_*, how the record was found */
 
 	/*
 	 * Found in the _dmarc record or supplied to us.
@@ -258,16 +259,20 @@ void opendmarc_policy_library_dns_hook(int *nscountp, struct sockaddr_in *nsaddr
 
 #if HAVE_SPF2_H
 #include "spf.h"
-typedef struct spf_context_struct { 
+#include "spf_dns_resolv.h"
+#include "spf_dns_cache.h"
+typedef struct spf_context_struct {
 	SPF_server_t *		spf_server;
 	SPF_request_t *		spf_request;
 	SPF_response_t *	spf_response;
 	SPF_result_t		spf_result;
 	char    		mailfrom_addr[512];
 	char			mailfrom_domain[256];
 	char    		helo_domain[256];
+	char *			spf_dns_log;	/* RFC 9991 SPF-DNS lines, owned by caller once handed over */
 } SPF_CTX_T;
-int opendmarc_spf2_test(char *ip_address, char *mail_from_domain, char *helo_domain, char *spf_record, int softfail_okay_flag, char *human_readable, size_t human_readable_len, int *used_mfrom);
+int opendmarc_spf2_test(char *ip_address, char *mail_from_domain, char *helo_domain, char *spf_record, int softfail_okay_flag, char *human_readable, size_t human_readable_len, int *used_mfrom, int want_dns_log, char **spf_dns_lines);
+SPF_dns_server_t *opendmarc_spf_dns_log_new(SPF_dns_server_t *layer_below, char **logbuf, int debug);
 
 #else /* not HAVE_SPF2_H */
 
@@ -289,7 +294,7 @@ typedef struct spf_context_struct {
 	char    exp_buf[512];
 	int     did_get_exp;
 } SPF_CTX_T;
-int		opendmarc_spf_test(char *ip_address, char *mail_from_domain, char *helo_domain, char *spf_record, int softfail_okay_flag, char *human_readable, size_t human_readable_len, int *used_mfrom);
+int		opendmarc_spf_test(char *ip_address, char *mail_from_domain, char *helo_domain, char *spf_record, int softfail_okay_flag, char *human_readable, size_t human_readable_len, int *used_mfrom, int want_dns_log, char **spf_dns_lines);
 char ** 	opendmarc_spf_dns_lookup_a(char *domain, char **ary, int *cnt);
 char ** 	opendmarc_spf_dns_lookup_mx(char *domain, char **ary, int *cnt);
 char ** 	opendmarc_spf_dns_lookup_mx_domain(char *domain, char **ary, int *cnt);