1. New Problem - Whee!
At the beginning of this month, Yahoo (or whatever they're called)
and Google tightened down the screws on requiring DMARC and encouraging
DKIM (and/or SPF).
https://blog.google/products/gmail/gmail-security-authentication-spam-prote…https://support.google.com/a/answer/81126?visit_id=638429646339446077-23847…
See links for GMail's alleged implementation, which has numerous parts.
The page _claims_ the Feb. 1 enforcement is just for "bulk senders —
those who send more than 5,000 messages to Gmail addresses in one day",
but word around the Net is that they started spamboxing mail from even
_tiny_ mail sending domains not in compliance.
2. Parts of the Problem
(Gods, I hope my MTA uses TLS for transmitting mail. It's been so long
since I looked at that, I'm not sure. Anyway, that was a hard
requirement for reaching GMail starting 2023-12-31, per above pages.)
Here, verbatim, are the requirements Gmail says it henceforth requires
of all sending domains (with some others for "bulk senders"):
o Set up SPF or DKIM email authentication for your domain.
o Ensure that sending domains or IPs have valid forward and reverse DNS
records, also referred to as PTR records. Learn more
o Use a TLS connection for transmitting email. For steps to set up TLS in
Google Workspace, visit Require a secure connection for email.
o Keep spam rates reported in Postmaster Tools below 0.10% and avoid ever
reaching a spam rate of 0.30% or higher. Learn more about spam rates.
o Format messages according to the Internet Message Format standard (RFC
5322).
o Don’t impersonate Gmail From: headers. Gmail will begin using a DMARC
quarantine enforcement policy, and impersonating Gmail From: headers
might impact your email delivery.
o If you regularly forward email, including using mailing lists or inbound
gateways, add ARC headers to outgoing email. ARC headers indicate the
message was forwarded and identify you as the forwarder. Mailing list
senders should also add a List-id: header, which specifies the mailing
list, to outgoing messages.
(I'm blearily looking up what is meant by "ARC headers", and I have
never previously had cause or interest to look at Google's Postmaster
Tools.)
3. DKIM Considered Harmful
Long rant not indulged here: DKIM is mailing list-hostile (though
mitigations are possible). I'm sitting out DKIM mail-signing, but also
my present host can't do it as equipped anyway. DMARC is a metastandard
atop either or both of SPF and DKIM. By the definition of DMARC, if you
have both (a) a valid DMARC DNS record and (b) _either_ DKIM or SPF (or
both), then you are DMARC-compliant.
4. SPF Is My Jedi Master
linuxmafia.com has had bullet-proof, simple SPF for ages. I like SPF
and consider it well-scoped, well-implemented for preventing others to
believably forge your domain for mail. It's good here, it works. Here:
:r! dig -t txt @ns1.linuxmafia.com linuxmafia.com +short
"v=spf1 ip4:96.95.217.99 -all"
Dirt-simple: It says "If mail arrives from any IP other than
96.95.217.99 purporting to be from linuxmafia.com, please treat it as
forged."
5. Let's Look at a Good DMARC RR
So, obviously I need a value DMARC record as of a week ago. A couple
of years ago, I tried one, and could swear I remember it impairing my
deliverability of mailing list mail (but can't remember details), not
to mention the avalanche of DMARC "report" mails that are the hallmark
of that botched (IMO) Yahoo anti-forgery metastandard.
But, let's look at the RR for mxtoolbox.com. The RR is always hostname
_dmarc , and is a TXT record. In mxtoolbox.com's case, that's a CNAME
pointing to mxtoolbox.com.hosted.dmarc-report.com . Here 'tis:
:r! dig -t txt mxtoolbox.com.hosted.dmarc-report.com +short
"v=DMARC1; p=reject; rua=mailto:634990a7@mxtoolbox.dmarc-report.com,mailto:634990a7@mxtoolbox.dmarc-report.net; ruf=mailto:634990a7@forensics.dmarc-report.com; fo=1; pct=100"
Breaking that down:
FQDN: mxtoolbox.com.hosted.dmarc-report.com.
TTL: 300
class: IN
RR type: TXT
TXT record breakdown (from within doublequotes):
Version: v=DMARC1;
Action to take if the mail fails authentication: p=reject
Report URIs for Aggregate data to (optional): rua=mailto:634990a7@mxtoolbox.dmarc-report.com,mailto:634990a7@mxtoolbox.dmarc-report.net;
Report URIs for Forensics/Failure data to (optional): ruf=mailto:634990a7@forensics.dmarc-report.com;
Forensic Options: fo=1;
Percentage of mail subjected to filtering: pct=100
6. Let's build a _dmarc.linuxmafia.com DMARC RR
So, in consultation with the pages mentioned below, I constructed one
(and these were my resulting notes):
Include v=DMARC1; because DMARCv1 is the only game in town. Note that
this tag must appear first, or the RR is not a valid DMARC record.
Start policy adoption with p=none; (mostly because of mailing list
forwarding).
https://dmarc.org/2017/03/can-i-use-dmarc-if-i-have-only-deployed-spf/
Include sp=none; This is the "p" policy setting, except as applied to
subdomains.
https://mxtoolbox.com/dmarc/details/dmarc-tags/dmarc-sp
Omit aspf=$THING option for now; defaults to "relaxed" SPF alignment
(though I think all mail from linuxmafia.com will originate from that
FQDN as MailFROM and Header From domains).
https://mxtoolbox.com/dmarc/details/dmarc-tags/aspf
Include fo=s; This generates an SPF failure report if the message
failed SPF evaluation, regardless of its alignment. This setting is
used to decide what to send to addresses specified in the RUF tag.
https://mxtoolbox.com/dmarc/details/dmarc-tags/dmarc-failure-reporting-opti…
Include rua=mailto:hostmaster@linuxmafia.com; This says where to send
DMARC Aggregate Feedback reports to. Be warned that this can be a lot
of... stuff.
https://mxtoolbox.com/dmarc/details/dmarc-tags/dmarc-rua
Include ruf=mailto:hostmaster@linuxmafia.com; This says where to send
failure/forensic reports, which are much more detailed than are DMARC
Aggregate reports. Only a few mail senders issue those. Prepare to
revise/lose this if I get swamped with reports.
https://mxtoolbox.com/dmarc/details/dmarc-tags/dmarc-ruf
Omit rf=$THING (report format), because there's only one format
(Authentication Failure Reporting Format = AFRF) so far, and it's
default, making this tag so far pointless.
https://mxtoolbox.com/dmarc/details/dmarc-tags/dmarc-report-format
Include ri=604800; report interval (default 86400 = 1 day), so that
reporting sites send me aggregate reports every 7 days rather than
daily.
https://mxtoolbox.com/dmarc/details/dmarc-tags/dmarc-report-interval
Include pct=100; Means apply the declared policy to mail purporting to
be mine 100% of the time. Since I'm saying "none" for now, this will
matter really only when I shift to p=quarantine; or p=reject; .
https://mxtoolbox.com/dmarc/details/dmarc-tags/dmarc-percentage
Results in:
_dmarc.linuxmafia.com 300 IN TXT "v=DMARC1; p=none; sp=none; rua=mailto:hostmaster@linuxmafia.com; ruf=mailto:hostmaster@linuxmafia.com; fo=s; ri=604800; pct=100"
7. How about Y'all? What are you guys doing about the problem?
Having felt sheepish about the lazy design/implementation of my
domain-checking scripts, I figured I'd start small, by rewriting
/etc/cron.weekly/baycondomain to be iota more sophisticated.
This was a profoundly primitive script that just reported back (via
e-mail) current parent-zone SOA serial numbers of domain baycon.org at
its pair of authoritative nameservers -- so I can vgrep and see if they
disagree, or if one or both of them don't answer. It was a
quick'n'dirty job from 2011, only now revisited.
It's now a _little_ less dumb. Output presentation is meh at best.
Also, its continued hard-coding quantity and FQDNs of the auth.
nameservers is regrettable, and could be eliminated by revising the
thing more, to generalise it. Ditto the continued hardcoded name of the
domain being checked.
Still, as the late Adam Osborne used to say, "Adequacy is sufficient."
----- begin cron script -----
#!/bin/sh
# baycondomain Cron script to sanity-check the BayCon domain's SOA records at
# all of its authoritative nameservers, as a quick and
# dirty way of making sure (1) they're all online and
# (2) they're all serving up the same data (or at least
# data with the same zonefile serial number).
#
# The script queries all nameservers for their current
# SOA value (for baycon.org), and then uses awk to parse
# out of that verbose record just the S/N field, which is
# field #3. The point is that you can visually spot offline
# or aberrant nameservers by their S/Ns being (respectively)
# missing or an out-of-step value.
#
# Written by Rick Moen (rick(a)linuxmafia.com)
# $Id: cron.weekly,v 1.02 2023/09/14 22:04:55 rick
set -o errexit #aka "set -e": exit if any line returns non-true value
set -o nounset #aka "set -u": exit upon finding an uninitialised variable
test -x /usr/bin/mail || exit 0
{
ns1soa=$(dig @NS1.BLUEHOST.COM. baycon.org. soa +short | awk {'print $3'})
ns2soa=$(dig @NS2.BLUEHOST.COM. baycon.org. soa +short | awk {'print $3'})
( [ "${ns1soa:=nonresponding}" = "${ns2soa:=nonresponding}" ] ) \
&& echo "ns1.bluehost.com and ns2.bluehost.com agree on: $ns1soa" \
|| echo "ns1.bluehost.com says $ns1soa, but ns2.bluehost.com is a rebel and says $ns2soa"
} |
/usr/bin/mail -s "Domain baycon.org SOA check" rick(a)linuxmafia.com
----- end cron script -----
----- Forwarded message from root <root(a)linuxmafia.com> -----
Date: Sun, 17 Sep 2023 06:47:01 -0700
From: root <root(a)linuxmafia.com>
To: rick(a)linuxmafia.com
Subject: Domain baycon.org SOA check
ns1.bluehost.com and ns2.bluehost.com agree on: 2023082800
----- End forwarded message -----
Last night, I tidied up my perfunctory weekly cron job that
checks on master & all slave nameserver for my two domains, making the
script itself more terse and easier to read, along with eliminating some
undesired junk report output. The revised Bourne script and its current
output follow, below.
And, if anyone feels like helping improve it more, that'd be cool.
As you'll see, I was lazy in my design & implementation, for which I
make no apology: A good-enough implementation you complete beats a
better one you never quite get to, every time. And yet, it could be a
lot better.
One obvious misfeature is that the script output says (figuratively)
"Here's the five nameservers you ought to see; if any are missing,
something is wrong", but a _better_ script would simply use shell logic
to report, for each of the five, _either_ its returned zonefile S/N _or_
the fact that it doesn't respond. For extra credit, disagreement about
the S/N could be somehow highlighted rather than left for the reader to
vgrep.
Maybe, I dunno, state the master nameserver's S/N first, and then
for each of the slaves say either "$FOO: agrees", "$FOO: no response",
or "$FOO: is a rebel and says $BAR". Or, _even better_, if (as should
be the case routinely) all nameservers are in agreement, say "All
$N nameservers report $BAR", and cite details only in the exceptional case
needing attention, where there are nonresponses or disagreements.
In general, my sense is that one wants to have much terser output when
results are normal and expected: Ideally, the report's substance
should be either "Everything's cool this week" or "Some things are off
this week, details below."
E.g., a well-tuned logcheck report deliberately doesn't tell you
anything about the many things going on that are routine, so that
non-routine and possibly worrying things will stand out better.
I'm also dissatisfied with the results of parsing out "Name Server"
lines from /usr/bin/whois output: Notice the auth. namserver roster
is reported twice, once in all-caps, once in lower case. Why? Because
"whois linuxmafia.com" (or same for unixmercenary.net) _itself_ reports
the output from registrar whois server whois.1api.net twice in a row,
in two stanzas, first without domain stakeholder contacts, the second
time with them. I really have no idea why the latter is the case.
Maybe the registrar whois servers are just unfixably funky: I know from
messing around with Perl script d-check and its predecessor domain-check
that the registrars mess around with whois output at unpredictable
intervals. Maybe I should just be happy that the service exists, at
all: A lot of Internet money people keep wanting to kill public whois,
because of course it's not a profit centre (despite newer Web-mediated
whois that can clot you down with ads).
And last, notice that each of the four reported "dig" output sections
is prefaced by three lines of comments declaring the dig version,
the command line dig is processing, then "(1 server found)", then
"global options: printcmd". I kept playing last night with "+no$FOO"
options, trying to eliminate those from what dig reported, and didn't
find the answer. One perversity "dig" has is that it appears to answer
a bit differently when run at the command line vs. in a shell script.
Of course, I could just pipe the output through a GNU sed filter that
strips out lines starting with ";", but I keep feeling sheepish about
not knowing the magic for tersifying "dig" output more, in scripts.
Oh, and also, my "+no$FOO" experiment gave different results from
queries of the parent-zone NS records vs. queries of in-zone NS records,
as to how much undesired junk output got eliminated. That's why the
current script use different sets of flags for the two "dig" use-cases.
I don't entirely understand why this is the case; I don't think I was
hallucinating, but obviously there are complexities to taming "dig"'s
output verbosity that I haven't yet mastered.
---<begin /etc/cron.weekly/mydomains contents>---
#!/bin/sh
# mydomains Cron script to sanity-check my domains' SOA records at
# all of their authoritative nameservers, as a quick and
# dirty way of making sure (1) they're all online and
# (2) they're all serving up the same data (or at least
# data with the same zonefile serial number).
#
# The script queries all nameservers for their current
# SOA value, and then uses awk to parse out of that
# verbose record just the S/N field, which is field #3.
# The point is that you can visually spot offline or
# aberrant nameservers by their S/Ns being (respectively)
# missing or an out-of-step value.
#
# Written by Rick Moen (rick(a)linuxmafia.com)
# $Id: cron.weekly,v 1.07 2023/09/08 00:23:00 rick
# Copyright (C) Rick Moen, 2011-2023. Do anything you want with this work.
set -o errexit #aka "set -e": exit if any line returns non-true value
set -o nounset #aka "set -u": exit upon finding an uninitialised variable
test -x /usr/bin/mail || exit 0
test -x /usr/bin/whois || exit 0
test -x /usr/bin/awk || exit 0
test -x /bin/grep || exit 0
test -x /usr/bin/dig || exit 0
{
echo "As of 2023-09-08, linuxmafia.com should show five authoritative nameservers:"
echo ""
echo "ns.primate.net. 198.144.194.12, (Aaron T. Porter)"
echo "ns.tx.primate.net. 72.249.38.88 (Aaron T. Porter)"
echo "ns3.linuxmafia.com. 107.204.234.170, aka ns.catwhisker.org (David Wolfskill)"
echo "ns0.sunnyside.com. 192.147.248.10 (Al Whaley)"
echo "ns1.linuxmafia.com. 96.95.217.99 (Rick Moen)"
echo ""
echo "As of 2023-09-08, unixmercenary.net should show five authoritative nameservers:"
echo ""
echo "ns.primate.net. 198.144.194.12, (Aaron T. Porter)"
echo "ns.tx.primate.net. 72.249.38.88 (Aaron T. Porter)"
echo "ns3.linuxmafia.com. 107.204.234.170, aka ns.catwhisker.org (David Wolfskill)"
echo "ns0.sunnyside.com. 192.147.248.10 (Al Whaley)"
echo "ns1.linuxmafia.com. 96.95.217.99 (Rick Moen)"
echo ""
echo "If any is missing from reports below, or produces odd data, something is wrong."
echo ""
echo "Zonefile S/Ns, linuxmafia.com:"
echo ""
for i in $(dig linuxmafia.com. NS +short); do dig @$i linuxmafia.com. soa +short | awk '{ print $3 " on '$i'"}'; done
echo ""
echo "Zonefile S/Ns, unixmercenary.net:"
echo ""
for i in $(dig unixmercenary.net. NS +short); do dig @$i unixmercenary.net. soa +short | awk '{ print $3 " on '$i'"}'; done
echo ""
echo "Authoritative nameservers from whois, linuxmafia.com:"
echo ""
whois linuxmafia.com | grep 'Name Server' | awk -F: '{ print $2 }'
echo ""
echo "Authoritative nameservers from whois, unixmercenary.net:"
echo ""
whois unixmercenary.net | grep 'Name Server' | awk -F: '{ print $2 }'
echo ""
echo "Parent-zone NS records, linuxmafia.com:"
echo ""
dig @$(dig com. NS +short | head -n 1) linuxmafia.com. NS +noall +auth
echo ""
echo "Parent-zone NS records, unixmercenary.net:"
echo ""
dig @$(dig net. NS +short | head -n 1) unixmercenary.net. NS +noall +auth
echo ""
echo "In-domain NS records, linuxmafia.com:"
echo ""
dig @ns1.linuxmafia.com. linuxmafia.com. ns +nocomments +noadd +nocmd +noquestion +noqr +nostats
echo ""
echo "In-domain NS records, unixmercenary.net:"
echo ""
dig @ns1.linuxmafia.com. unixmercenary.net. ns +nocomments +noadd +nocmd +noquestion +noqr +nostats
} |
mail -s "Domains linuxmafia.com and unixmercenary.net SOA check" rick(a)linuxmafia.com
---<end>
----- Forwarded message from root <root(a)linuxmafia.com> -----
Date: Fri, 08 Sep 2023 10:27:59 -0700
From: root <root(a)linuxmafia.com>
To: rick(a)linuxmafia.com
Subject: Domains linuxmafia.com and unixmercenary.net SOA check
As of 2023-09-08, linuxmafia.com should show five authoritative nameservers:
ns.primate.net. 198.144.194.12, (Aaron T. Porter)
ns.tx.primate.net. 72.249.38.88 (Aaron T. Porter)
ns3.linuxmafia.com. 107.204.234.170, aka ns.catwhisker.org (David Wolfskill)
ns0.sunnyside.com. 192.147.248.10 (Al Whaley)
ns1.linuxmafia.com. 96.95.217.99 (Rick Moen)
As of 2023-09-08, unixmercenary.net should show five authoritative nameservers:
ns.primate.net. 198.144.194.12, (Aaron T. Porter)
ns.tx.primate.net. 72.249.38.88 (Aaron T. Porter)
ns3.linuxmafia.com. 107.204.234.170, aka ns.catwhisker.org (David Wolfskill)
ns0.sunnyside.com. 192.147.248.10 (Al Whaley)
ns1.linuxmafia.com. 96.95.217.99 (Rick Moen)
If any is missing from reports below, or produces odd data, something is wrong.
Zonefile S/Ns, linuxmafia.com:
2022041200 on ns3.linuxmafia.com.
2022041200 on ns.primate.net.
2022041200 on ns.tx.primate.net.
2022041200 on ns0.sunnyside.com.
2022041200 on ns1.linuxmafia.com.
Zonefile S/Ns, unixmercenary.net:
2022040401 on ns.primate.net.
2022040401 on ns.tx.primate.net.
2022040401 on ns3.linuxmafia.com.
2022040401 on ns0.sunnyside.com.
2022040401 on ns1.linuxmafia.com.
Authoritative nameservers from whois, linuxmafia.com:
NS.PRIMATE.NETNS.TX.PRIMATE.NETNS0.SUNNYSIDE.COMNS1.LINUXMAFIA.COMNS3.LINUXMAFIA.COMns0.sunnyside.comns1.linuxmafia.com 96.95.217.99
ns.primate.netns.tx.primate.netns3.linuxmafia.com 107.204.234.170
Authoritative nameservers from whois, unixmercenary.net:
NS.PRIMATE.NETNS.TX.PRIMATE.NETNS0.SUNNYSIDE.COMNS1.LINUXMAFIA.COMNS3.LINUXMAFIA.COMns0.sunnyside.comns1.linuxmafia.com 96.95.217.99
ns.primate.netns.tx.primate.netns3.linuxmafia.com 107.204.234.170
Parent-zone NS records, linuxmafia.com:
; <<>> DiG 9.4.2 <<>> @f.gtld-servers.net. linuxmafia.com. NS +noall +auth
; (1 server found)
;; global options: printcmd
linuxmafia.com. 172800 IN NS ns0.sunnyside.com.
linuxmafia.com. 172800 IN NS ns1.linuxmafia.com.
linuxmafia.com. 172800 IN NS ns.primate.net.
linuxmafia.com. 172800 IN NS ns.tx.primate.net.
linuxmafia.com. 172800 IN NS ns3.linuxmafia.com.
Parent-zone NS records, unixmercenary.net:
; <<>> DiG 9.4.2 <<>> @k.gtld-servers.net. unixmercenary.net. NS +noall +auth
; (1 server found)
;; global options: printcmd
unixmercenary.net. 172800 IN NS ns0.sunnyside.com.
unixmercenary.net. 172800 IN NS ns1.linuxmafia.com.
unixmercenary.net. 172800 IN NS ns.primate.net.
unixmercenary.net. 172800 IN NS ns.tx.primate.net.
unixmercenary.net. 172800 IN NS ns3.linuxmafia.com.
In-domain NS records, linuxmafia.com:
; <<>> DiG 9.4.2 <<>> @ns1.linuxmafia.com. linuxmafia.com. ns +nocomments +noadd +nocmd +noquestion +noqr +nostats
; (1 server found)
;; global options: printcmd
linuxmafia.com. 86400 IN NS ns.primate.net.
linuxmafia.com. 86400 IN NS ns0.sunnyside.com.
linuxmafia.com. 86400 IN NS ns.tx.primate.net.
linuxmafia.com. 86400 IN NS ns3.linuxmafia.com.
linuxmafia.com. 86400 IN NS ns1.linuxmafia.com.
In-domain NS records, unixmercenary.net:
; <<>> DiG 9.4.2 <<>> @ns1.linuxmafia.com. unixmercenary.net. ns +nocomments +noadd +nocmd +noquestion +noqr +nostats
; (1 server found)
;; global options: printcmd
unixmercenary.net. 86400 IN NS ns.primate.net.
unixmercenary.net. 86400 IN NS ns1.linuxmafia.com.
unixmercenary.net. 86400 IN NS ns0.sunnyside.com.
unixmercenary.net. 86400 IN NS ns3.linuxmafia.com.
unixmercenary.net. 86400 IN NS ns.tx.primate.net.
----- End forwarded message -----