I wrote:
To clarify, DMARC is an omnibus package, invented by Yahoo, of anti-forgery technologies that incorporates Yahoo's DKIM (DomainKey Identified Mail, a successor to Yahoo's DomainKeys) and Meng Wong's SPF.
Mailing list manager (MLM) package have no problem with SPF because they supply an entirely fresh SMTP envelope on the retransmission to subscribers. In this, they differ from cross-system /etc/aliases and ~/.forward entries, which do _not_ supply a new envelope.
MLM packages have big problems with Yahoo's standards (the DKIM portion of DMARC, and legacy DomainKeys) because MLM's modifications to posting text and headers upon retranmission to subscribers have a strong tendency to break DKIM's (and DomainKeys's) cryptographic attestation about the integrity of the text and headers.
The specific Mailman munging kludge I recommended addresses this problem by dumbing down Mailman's treatment of the internal 'From: ' header where necessary to avoid breaking DKIM (/DomainKeys) attestation. The cost of this is grievous, substitution of the mailing list's originating address for the user's, but this is the least amount of damage that appears able to make Mailman cope with what IMO is a really crappy and MLM-hostile antiforgery design.
I elaborated on this recently on Devuan's main discussion list Dng, because a couple of the regulars got this wrong, lumping in SPF with the MLM-problematic Yahoo anti-forgery methods, and underestimating the problems caused by the latter. The subject arose on Dng because that mailing list's mail was running into rejects on postings sent from, yes, yahoo.com and to a lesser extent GMail, and I had a well-founded suspicion that the problem was DKIM/DMARC. The fellow I address below, Daniel Abrecht, is a fan of DMARC. (Below that, I include other relevant postings.)
Date: Thu, 3 Aug 2017 20:47:46 -0700 From: Rick Moen rick@linuxmafia.com To: dng@lists.dyne.org Subject: Re: [DNG] Excessive bounces
Daniel, again, thank you for your efforts to collect diagnostic data for the Devuan Project, and for your care and precision.
I respect the reasons you're making DMARC work for your domain, and certainly have no argument with you doing so (though I do not come to the same conclusion for my own domain). A few comments that come to mind:
Quoting Daniel Abrecht (dng@danielabrecht.ch)
The SPF checks from the report have currently the result None, which usually indicates that domain does not have an SPF record. I did add ?mx:lists.dyne.org to my SPF record, which should have resulted in an SPF result of Neutral, so either the mailing list needs an SPF record, or google & co. are wrongly reporting a None result instead of a Neutral one.
Let's be careful about what we're speaking about, here. By definition, a posting that transits a mailing list goes through two phases, where to the best of my understanding a different SPF RR is relevant for each.
In the first phase, mail arrives at (say) the lists.dyne.org MTA purporting to be from your domain. If the receiving MTA is checking SPF on arriving mail, then it seeks to validate the envelope header of arrived mail against the published SPF RR of the claimed sender domain as reflected in the envelope. A forged mail at this point would have an envelope claiming it's from danielabrecht.ch, but the IP would fail vetting against your A and MX records (declared as authorised senders in your domain's SPF RR). If it's genuine, otherwise.
The accepted posting gets handed by the receiving MTA to the MLM software. The MLM software now remails, or to put it another way, creates fresh mails (thus, second phase), with an entirely different SMTP envelope reflecting the MLM's host identity. The internal 'From: ' header is (ideally) left intact, the internal 'To: ' header is customised for each subscriber, and some number of other additions and changes get made by the MLM software prior to handing it off to the outbound MTA.
If subscribers' receiving MTAs now attempt, during this second phase, to validate the SMTP envelope, they will do so based on the MLM host's domain, not the original sender's.
When a mail is sent, there are the envelope-to and the envelope-from (which aren't mail headers), but also To and From headers.[...]
Yes, I'm extremely well aware of this. The latter 'From header' is traditionally called the internal 'From:' header to distinguish it from the envelope 'From ' header.
I was on NANAE during the incident in which envelope forgery was invented and demonstrated in the famous revenge-spam attack against Joe Doll of Joe's Cyberpost.
Normally when an SPF check is made, the envelope from address is used. If the mail server doesn't have an SPF header, the SPF result is None and the receiving mail server should accept the mail.
To my knowledge, there is no such thing as an 'SPF header' (except see below). No extra headers get used to implement SPF. It's just a DNS RR that declares what authorised sending MTA hosts exist, and gets used by supporting (receiving) MTAs to vet the envelope sender. (There's also an optional 'Authentication-Results' header that's similar, except for recording border MTAs' results.)
My understanding is that some MTAs add a Received-SPF header to all emails arriving via SMTP that test to any result other than 'fail'. This is added _after_ SPF evaluation to record the results of that check. It's advisory, and merely a place to record the result.
There is no point in getting notified if nothing happens or everything works as expected, but if something didn't work or someone tried to send a mail in my name and failed, I certainly want to know about that.
Certainly your prerogative, but I personally don't care to get notified when someone tries to forge my domain and fails, because IMO that amounts to getting spam about spam.
There's a reason why I've finely tuned logcheck to stop notifying me of pointless and futile attempts to use generic username/password pairs on my sshd, and a thousand other bits of basically meaningless noise that's just part of the routine of having a 24x7 Internet server.
- The recipient can check if the message content was changed
gpg signing alone can do that.
gpg and DKIM have a bit different scope.
Yes, but both can authenticate and attest to the integrity of whatever dataset was signed. And that thus matches 'can check if the message content was changed'. Just crypto-sign the message contents. Anything extraneous inserted into the signed bloc will cause validation failure. Anything outside it will be obviously _not_ attested to, and recipients should accordingly understand that.
You might have more reasonably made the objection 'Ordinary folks won't check PGP attestation.' Maybe not. Personally, I regard this as Not My Problem. If I say something where text integrity really matters, like the body of a CVE, I will PGP-sign it. Readers who care about text integrity will check it, others won't.
Date: Thu, 3 Aug 2017 09:15:17 -0700 From: Rick Moen rick@linuxmafia.com To: dng@lists.dyne.org Subject: Re: [DNG] Excessive bounces
Quoting Simon Hobson (linux@thehobsons.co.uk):
SPF breaks mailing lists and mail forwarders - and this is NOT (IMO) fixable without introducing a wide open front gate for spammers to ride through and completely bypass SPF.
No. it does not break mailing lists. It _does_ break other common types of forwarders unless they adopt SRS-wrapping.
The reason it does not adversely affect mailing lists is that SPF validates only the envelope header: The receving MTA verifies that the delivering MTA's IP address is mentioned in the claimed sending domain's SPF RR (if there is one in that domain's DNS).
Consider the envelope header of your own Dng posting, as received by linuxmafia.com's MTA when it received my subscription's copy. Here's the way your post arrived:
From dng-bounces@lists.dyne.org Thu Aug 03 05: 8:39 2017 Return-path: dng-bounces@lists.dyne.org Envelope-to: rick@linuxmafia.com
So, the envelope sender's domain was dyne.org, not thehobsons.co.uk, and my receiving MTA will perform a DNS check against the former.
:r! dig -t txt dyne.org +short
"google-site-verification=6FghqJroXIvBY8cutq6ouO0RC-a8qynFu6sJR3S-IbA" "v=spf1 mx ip4:178.62.188.7/32 ip4:188.226.191.63/32 ip4:213.127.180.241/32 -all" "google-site-verification=2XoWrMMTQ7jmgcB_76Y_TQSnWDGhR4e-y_KLqoKOK1Q"
:r! dig lists.dyne.org +short 178.62.188.7
And, lo! The envelope sender does validate.
You are probably confusing mailing lists, which provide new envelope headers during forwarding citing the forwarding domain, with other forwarders like /etc/alias entries and ~/.forward files. It's the _latter_ that SPF author Meng Wong invented that goofy Sender Rewriting System. Mailing lists, by contrast, don't have the problem he invented that kludge to fix.
And, again, Simon, my mail domain linuxmafia.com has had an SPF hardfail directive in its DNS since around 2003, and the specifier is extremely narrow:
:r! dig -t txt linuxmafia.com +short "v=spf1 a mx -all"
That says 'If a mail's envelope header claims it's from linuxmafia.com, but the delivering MTA doesn't match either linuxmafia.com's DNS A record or its MX record, please consider it definitively a forgery.' I'm on _many_ mailing lists on many hosts. If my mailing list mail had a deliverability problem caused by hardfailing forgeries of my envelope header, I'd have figured that out, some time over hte past 14 years. It does not happen, because mailing lists work better than /etc/alias entries and ~/.forward files by design.
Earlier posts upthread:
Date: Wed, 2 Aug 2017 10:41:27 -0700 From: Rick Moen rick@linuxmafia.com To: dng@lists.dyne.org Subject: Re: [DNG] Excessive bounces
Quoting Jaromil (jaromil@dyne.org):
I am a bit puzzled about this one, we had some reports of the problem so far, which hasn't occurred before on any other dyne list and is not really reproducible.
what we notice is that our mail server is under some quite heavy load and we are working to move it to a bigger infrastructure by september
however it is highly available already and seems to process everything, so I'm not really sure what is happening... any insight is welcome.
Might be DMARC validation failure. (Gods, do I ever detest that stuff.)
DMARC is proving to be an utter nightmare for mailing lists, in as much as they are mail forwarders, and DMARC was IMO botched in its ability to accomodate the way they work. From memory, and so I'm probably dropping a bunch of detail: Because MLMs such as Mailman (appropriately) change the internal SMTP headers upon retransmitting the poster's mail to subscribers (notably the To: header), it no longer validates against the sender's domain if it is a DMARC-using one with a strict policy. Yahoo and Gmail are examples of sending domains with strict DMARC policies.
There is an (IMO unhappy but least-bad-available) kludge setting in Mailman's admin WebUI to make the MLM compensate for DMARC brain-damage: You go to Privacy Options, Sender Filters, item 'Action to take when anyone posts to the list from a domain with a DMARC Reject/Quarantine Policy' aka dmarc_moderation_action. Change the radio button from Accept (default) to Munge from.
To quote the help text:
from_is_list (general): Replace the From: header address with the list's posting address to mitigate issues stemming from the original From: domain's DMARC or similar policies.
Several protocols now in wide use attempt to ensure that use of the domain in the author's address (ie, in the From: header field) is authorized by that domain. These protocols may be incompatible with common list features such as footers, causing participating email services to bounce list traffic merely because of the address in the From: field. This has resulted in members being unsubscribed despite being perfectly able to receive mail.
The following actions are applied to all list messages when selected here. To apply these actions only to messages where the domain in the From: header is determined to use such a protocol, see the dmarc_moderation_action settings under Privacy options... -> Sender filters.
Settings: [...]
Munge From
This action replaces the poster's address in the From: header with the list's posting address and adds the poster's address to the addresses in the original Reply-To: header.
So, for example, _if_ my sending domain linuxmafia.com had a strong DMARC policy (which it doesn't, because I hate DMARC with a passion), then the 'Munge from' setting would cause my post to Dng to get this 'From: ' header upon retransmission to subscribers:
From: Rick Moen via Dng dng@lists.dyne.org
instead of the normal
From: Rick Moen rick@linuxmafia.com
The reason this helps sidestep DMARC validation is that it's now no longer considered needing validation against linuxmafia.com's (hypothetical) DMARC policy, but rather dyne.org's.
I personally detest this solution because, when I send out my sending address on a mailing list, it is deliberately there so that people can, if necessary, contact me offlist. The kludge complicates this, albeit, if I remember correctly, it tries to compensate for the brain-damage by inserting a Reply-To as well.
It should be noted that the Munge from kludge thus alters -only- the postings of subscribers from DMARC-damaged^H^H^H^W^W^W^Wusing domains, so only _some_ postings will get disfigured in this manner.
Sadly, I recommend opting for this kludge, because otherwise deliverability suffers.
Date: Wed, 2 Aug 2017 10:43:37 -0700 From: Rick Moen rick@linuxmafia.com To: dng@lists.dyne.org Subject: Re: [DNG] Excessive bounces
Quoting Simon Hobson (linux@thehobsons.co.uk):
That's the important thing to look for - and my money is it's related to SPF and/or DMARC.
It won't be SPF.
My domain has a strong SPF policy:
:r! dig -t txt linuxmafia.com +short "v=spf1 a mx -all"
...and no mailing list post from me or my users violates it.
It'll be DMARC. (Long experience says.)
Date: Wed, 2 Aug 2017 10:54:57 -0700 From: Rick Moen rick@linuxmafia.com To: dng@lists.dyne.org Subject: Re: [DNG] Excessive bounces
Oh, follow-up (and this, too, is for Devuan's listadmins' attention in particular):
There is an (IMO unhappy but least-bad-available) kludge setting in Mailman's admin WebUI to make the MLM compensate for DMARC brain-damage: You go to Privacy Options, Sender Filters, item 'Action to take when anyone posts to the list from a domain with a DMARC Reject/Quarantine Policy' aka dmarc_moderation_action. Change the radio button from Accept (default) to Munge from.
I am specifically _not not not_ recommending the similar-looking setting 'Replace the From: header address with the list's posting address to mitigate issues stemming from the original From: domain's DMARC or similar policies' aka from_is_list on General Options. My understanding is that opting for _that_ version of the kludge unconditionally applies it to all postings whether they are from DMARC-encumbered domains or not.
My recommendations:
On Privacy options, Sender filters: dmarc_moderation_action: Munge from dmarc_quarantine_moderation_action): Yes dmarc_none_moderation_action: no
On General options: change nothing.
Date: Wed, 2 Aug 2017 16:59:14 -0700 From: Rick Moen rick@linuxmafia.com To: dng@lists.dyne.org Subject: Re: [DNG] Excessive bounces
Quoting Daniel Abrecht (dng@danielabrecht.ch):
I'm sorry, I'm using DMARC, and I didn't get the DMARC report about the bounced mails, probably because I forgot a DMARC DNS entry for the report receiving mail address. I have changed my DMARC policy from reject to quarantine for now.
It would be excellent if you could provide any DMARC reports you get to the Dng listadmins. Thank you.
Your point is well taken that DMARC and mailing lists can coexist (I've always concurred with that). It's just difficult, and creates adverse consequences. (As background for this, it's useful to know that DMARC is a composite and extension of SPF and DKIM.)
As part of the process, the domain's outgoing mail gets certain headers and body text cryptographically signed and attested to (the DKIM = DomainKeys Identified Mail part of the standard). For such mail to successfully transit a mailing list without breaking validation, the signed text and headers must be completely unchanged. This is a very difficult constraint for MLM software to meet, as occasionally something gets inserted or changed in a header or elsewhere during normal MLM processing, and in particular the To: header by design is supposed to be set upon posting retransmission to the address of each subscriber.
To the best of my recollection (and I'm presently busy and cannot double-check all of this), some subset of the full SMTP headers are included in the DKIM attestation. I can't remember which, nor whether the DKIM-issuing operator can decide which. I vaguely recall that the extra headers MLMs intentionally add, the MLM footer, the MLM modification to the Subject header (like adding [DNG]), and more are all somewhat problematic for DKIM validation.
There are a maddeningly large and diverse number of ways to deal with the problem, and one can spend a lot of time reading about it. E.g.: https://dmarc.org/supplemental/mailman-project-mlm-dmarc-reqs.html http://www.spamresource.com/2014/04/run-email-discussion-list-heres-how-to.h... https://dmarc.org/wiki/FAQ#I_operate_a_mailing_list_and_I_want_to_interopera...
Just a point:
I use DMARC and believe it to be necessary because it allows me to:
- Make sure nobody can use my E-Mail address to impersonate me or send
spam
SPF alone _can_ do exactly that without also needing DKIM/DMARC. (So, sufficient is correct, but necessary is not quite correct.)
- I will be notified if anyone attempts to do so
SPF alone can prevent it from being possible, hence you don't need to be notified. (This of course assumes that receiving domains check SPF for received mail. Not all do, but more do than check DMARC.)
- The recipient can check if the message content was changed
gpg signing alone can do that.
If your SMTP message content is being changed, though, you actually have a lot bigger problems.
- Provide an SPF record. This mailing list doesn't seam to have one
The mailing list isn't an orignator. It's the originating domains that ought (to the extent they wish to do so) to have SPF records.
- Don't change anything from the message below the DKIM headers, add
the other headers before the DKIM signature instead.
To the best of my recollection (I could be misremembering), this is easier said than done.
Anyway, thank you for your substantive help to the Devuan Project.
----- End forwarded message -----