[BALUG-Admin] BALUG-Talk and SPF/DKIM

[Rick Moen]

I wrote:

> To clarify, DMARC is an omnibus package, invented by Yahoo, of
> anti-forgery technologies that incorporates Yahoo's DKIM (DomainKey
> Identified Mail, a successor to Yahoo's DomainKeys) and Meng Wong's SPF.
> 
> Mailing list manager (MLM) package have no problem with SPF because 
> they supply an entirely fresh SMTP envelope on the retransmission to
> subscribers.  In this, they differ from cross-system /etc/aliases and
> ~/.forward entries, which do _not_ supply a new envelope.
> 
> MLM packages have big problems with Yahoo's standards (the DKIM portion
> of DMARC, and legacy DomainKeys) because MLM's modifications to posting
> text and headers upon retranmission to subscribers have a strong tendency 
> to break DKIM's (and DomainKeys's) cryptographic attestation about the
> integrity of the text and headers.
> 
> The specific Mailman munging kludge I recommended addresses this problem
> by dumbing down Mailman's treatment of the internal 'From: ' header
> where necessary to avoid breaking DKIM (/DomainKeys) attestation.  The
> cost of this is grievous, substitution of the mailing list's originating
> address for the user's, but this is the least amount of damage that
> appears able to make Mailman cope with what IMO is a really crappy and
> MLM-hostile antiforgery design.

I elaborated on this recently on Devuan's main discussion list Dng,
because a couple of the regulars got this wrong, lumping in SPF with the
MLM-problematic Yahoo anti-forgery methods, and underestimating the
problems caused by the latter.  The subject arose on Dng because that
mailing list's mail was running into rejects on postings sent from, yes,
yahoo.com and to a lesser extent GMail, and I had a well-founded
suspicion that the problem was DKIM/DMARC.  The fellow I address below,
Daniel Abrecht, is a fan of DMARC.  (Below that, I include other
relevant postings.)




 Date: Thu, 3 Aug 2017 20:47:46 -0700
 From: Rick Moen <rick@linuxmafia.com>
 To: dng@lists.dyne.org
 Subject: Re: [DNG] Excessive bounces

Daniel, again, thank you for your efforts to collect diagnostic data for
the Devuan Project, and for your care and precision.

I respect the reasons you're making DMARC work for your domain, and
certainly have no argument with you doing so (though I do not come to
the same conclusion for my own domain).  A few comments that come
to mind:


Quoting Daniel Abrecht (dng@danielabrecht.ch)

> The SPF checks from the report have currently the result None, which
> usually indicates that domain does not have an SPF record. I did add
> ?mx:lists.dyne.org to my SPF record, which should have resulted in an
> SPF result of Neutral, so either the mailing list needs an SPF record,
> or google & co. are wrongly reporting a None result instead of a
> Neutral one.

Let's be careful about what we're speaking about, here.  By definition,
a posting that transits a mailing list goes through two phases, where
to the best of my understanding a different SPF RR is relevant for each.

In the first phase, mail arrives at (say) the lists.dyne.org MTA
purporting to be from your domain.  If the receiving MTA is checking SPF
on arriving mail, then it seeks to validate the envelope header of
arrived mail against the published SPF RR of the claimed sender domain
as reflected in the envelope.  A forged mail at this point would have an
envelope claiming it's from danielabrecht.ch, but the IP would fail
vetting against your A and MX records (declared as authorised senders in
your domain's SPF RR).  If it's genuine, otherwise.

The accepted posting gets handed by the receiving MTA to the MLM
software.  The MLM software now remails, or to put it another way,
creates fresh mails (thus, second phase), with an entirely different
SMTP envelope reflecting the MLM's host identity.  The internal 'From: '
header is (ideally) left intact, the internal 'To: ' header is
customised for each subscriber, and some number of other additions and
changes get made by the MLM software prior to handing it off to the
outbound MTA.

If subscribers' receiving MTAs now attempt, during this second phase, to
validate the SMTP envelope, they will do so based on the MLM host's
domain, not the original sender's.


> When a mail is sent, there are the envelope-to and the envelope-from
> (which aren't mail headers), but also To and From headers.[...]

Yes, I'm extremely well aware of this.  The latter 'From header' is
traditionally called the internal 'From:' header to distinguish it from
the envelope 'From ' header.

I was on NANAE during the incident in which envelope forgery was
invented and demonstrated in the famous revenge-spam attack against Joe
Doll of Joe's Cyberpost.

> Normally when an SPF check is made, the envelope from address is used.
> If the mail server doesn't have an SPF header, the SPF result is None
> and the receiving mail server should accept the mail.

To my knowledge, there is no such thing as an 'SPF header' (except see
below).  No extra headers get used to implement SPF.  It's just a DNS RR
that declares what authorised sending MTA hosts exist, and gets used by
supporting (receiving) MTAs to vet the envelope sender.  (There's also
an optional 'Authentication-Results' header that's similar, except for
recording border MTAs' results.)

My understanding is that some MTAs add a Received-SPF header to
all emails arriving via SMTP that test to any result other than 'fail'.
This is added _after_ SPF evaluation to record the results of that
check.  It's advisory, and merely a place to record the result.


> There is no point in getting notified if nothing happens or everything
> works as expected, but if something didn't work or someone tried to
> send a mail in my name and failed, I certainly want to know about
> that.

Certainly your prerogative, but I personally don't care to get notified
when someone tries to forge my domain and fails, because IMO that
amounts to getting spam about spam.

There's a reason why I've finely tuned logcheck to stop notifying me of
pointless and futile attempts to use generic username/password pairs on
my sshd, and a thousand other bits of basically meaningless noise that's
just part of the routine of having a 24x7 Internet server.


> >> 3) The recipient can check if the message content was changed
> >
> > gpg signing alone can do that.
>
> gpg and DKIM have a bit different scope.

Yes, but both can authenticate and attest to the integrity of whatever
dataset was signed.  And that thus matches 'can check if the message
content was changed'.  Just crypto-sign the message contents.  Anything
extraneous inserted into the signed bloc will cause validation failure.
Anything outside it will be obviously _not_ attested to, and recipients
should accordingly understand that.

You might have more reasonably made the objection 'Ordinary folks won't
check PGP attestation.'  Maybe not.  Personally, I regard this as Not My
Problem.  If I say something where text integrity really matters, like
the body of a CVE, I will PGP-sign it.  Readers who care about text
integrity will check it, others won't.




 Date: Thu, 3 Aug 2017 09:15:17 -0700
 From: Rick Moen <rick@linuxmafia.com>
 To: dng@lists.dyne.org
 Subject: Re: [DNG] Excessive bounces

Quoting Simon Hobson (linux@thehobsons.co.uk):

> SPF breaks mailing lists and mail forwarders - and this is NOT (IMO)
> fixable without introducing a wide open front gate for spammers to
> ride through and completely bypass SPF.

No. it does not break mailing lists.  It _does_ break other common types
of forwarders unless they adopt SRS-wrapping.

The reason it does not adversely affect mailing lists is that SPF
validates only the envelope header:  The receving MTA verifies that the
delivering MTA's IP address is mentioned in the claimed sending domain's
SPF RR (if there is one in that domain's DNS).

Consider the envelope header of your own Dng posting, as received by
linuxmafia.com's MTA when it received my subscription's copy.  Here's
the way your post arrived:

  From dng-bounces@lists.dyne.org Thu Aug 03 05: 8:39 2017
  Return-path: <dng-bounces@lists.dyne.org>
  Envelope-to: rick@linuxmafia.com

So, the envelope sender's domain was dyne.org, not thehobsons.co.uk, and
my receiving MTA will perform a DNS check against the former.

:r! dig -t txt dyne.org +short

"google-site-verification=6FghqJroXIvBY8cutq6ouO0RC-a8qynFu6sJR3S-IbA"
"v=spf1 mx ip4:178.62.188.7/32 ip4:188.226.191.63/32 ip4:213.127.180.241/32 -all"
"google-site-verification=2XoWrMMTQ7jmgcB_76Y_TQSnWDGhR4e-y_KLqoKOK1Q"


:r! dig lists.dyne.org +short
178.62.188.7

And, lo!  The envelope sender does validate.

You are probably confusing mailing lists, which provide new envelope
headers during forwarding citing the forwarding domain, with other
forwarders like /etc/alias entries and ~/.forward files.  It's the
_latter_ that SPF author Meng Wong invented that goofy Sender Rewriting
System.  Mailing lists, by contrast, don't have the problem he invented
that kludge to fix.


And, again, Simon, my mail domain linuxmafia.com has had an SPF hardfail
directive in its DNS since around 2003, and the specifier is extremely
narrow:

:r! dig -t txt linuxmafia.com +short
"v=spf1 a mx -all"

That says 'If a mail's envelope header claims it's from linuxmafia.com,
but the delivering MTA doesn't match either linuxmafia.com's DNS A
record or its MX record, please consider it definitively a forgery.'
I'm on _many_ mailing lists on many hosts.  If my mailing list mail had
a deliverability problem caused by hardfailing forgeries of my envelope
header, I'd have figured that out, some time over hte past 14 years.  It
does not happen, because mailing lists work better than /etc/alias
entries and ~/.forward files by design.




Earlier posts upthread:



 Date: Wed, 2 Aug 2017 10:41:27 -0700
 From: Rick Moen <rick@linuxmafia.com>
 To: dng@lists.dyne.org
 Subject: Re: [DNG] Excessive bounces

Quoting Jaromil (jaromil@dyne.org):

> I am a bit puzzled about this one, we had some reports of the problem
> so far, which hasn't occurred before on any other dyne list and is not
> really reproducible.
>
> what we notice is that our mail server is under some quite heavy load
> and we are working to move it to a bigger infrastructure by september
>
> however it is highly available already and seems to process
> everything, so I'm not really sure what is happening... any insight is
> welcome.

Might be DMARC validation failure.  (Gods, do I ever detest that stuff.)

DMARC is proving to be an utter nightmare for mailing lists, in as much
as they are mail forwarders, and DMARC was IMO botched in its ability to
accomodate the way they work.  From memory, and so I'm probably dropping
a bunch of detail:  Because MLMs such as Mailman (appropriately) change
the internal SMTP headers upon retransmitting the poster's mail to
subscribers (notably the To: header), it no longer validates against the
sender's domain if it is a DMARC-using one with a strict policy.  Yahoo
and Gmail are examples of sending domains with strict DMARC policies.

There is an (IMO unhappy but least-bad-available) kludge setting in
Mailman's admin WebUI to make the MLM compensate for DMARC brain-damage:
You go to Privacy Options, Sender Filters, item 'Action to take when
anyone posts to the list from a domain with a DMARC Reject/Quarantine
Policy' aka dmarc_moderation_action.  Change the radio button from
Accept (default) to Munge from.

To quote the help text:

  from_is_list (general): Replace the From: header address with the
  list's posting address to mitigate issues stemming from the original
  From: domain's DMARC or similar policies.

  Several protocols now in wide use attempt to ensure that use of the
  domain in the author's address (ie, in the From: header field) is
  authorized by that domain. These protocols may be incompatible with
  common list features such as footers, causing participating email
  services to bounce list traffic merely because of the address in the
  From: field. This has resulted in members being unsubscribed despite
  being perfectly able to receive mail.

  The following actions are applied to all list messages when selected
  here. To apply these actions only to messages where the domain in the
  From: header is determined to use such a protocol, see the
  dmarc_moderation_action settings under Privacy options... -> Sender
  filters.

  Settings:
[...]

  Munge From


  This action replaces the poster's address in the From: header with the
  list's posting address and adds the poster's address to the addresses in
  the original Reply-To: header.

So, for example, _if_ my sending domain linuxmafia.com had a strong
DMARC policy (which it doesn't, because I hate DMARC with a passion),
then the 'Munge from' setting would cause my post to Dng to get this
'From: ' header upon retransmission to subscribers:

  From: Rick Moen via Dng <dng@lists.dyne.org>

instead of the normal

  From: Rick Moen <rick@linuxmafia.com>

The reason this helps sidestep DMARC validation is that it's now no longer
considered needing validation against linuxmafia.com's (hypothetical)
DMARC policy, but rather dyne.org's.


I personally detest this solution because, when I send out my sending
address on a mailing list, it is deliberately there so that people can,
if necessary, contact me offlist.  The kludge complicates this, albeit,
if I remember correctly, it tries to compensate for the brain-damage by
inserting a Reply-To as well.

It should be noted that the Munge from kludge thus alters -only- the
postings of subscribers from DMARC-damaged^H^H^H^W^W^W^Wusing domains,
so only _some_ postings will get disfigured in this manner.

Sadly, I recommend opting for this kludge, because otherwise
deliverability suffers.



 Date: Wed, 2 Aug 2017 10:43:37 -0700
 From: Rick Moen <rick@linuxmafia.com>
 To: dng@lists.dyne.org
 Subject: Re: [DNG] Excessive bounces

Quoting Simon Hobson (linux@thehobsons.co.uk):

> That's the important thing to look for - and my money is it's related
> to SPF and/or DMARC.

It won't be SPF.

My domain has a strong SPF policy:

:r! dig -t txt linuxmafia.com +short
"v=spf1 a mx -all"

...and no mailing list post from me or my users violates it.

It'll be DMARC.  (Long experience says.)



 Date: Wed, 2 Aug 2017 10:54:57 -0700
 From: Rick Moen <rick@linuxmafia.com>
 To: dng@lists.dyne.org
 Subject: Re: [DNG] Excessive bounces

Oh, follow-up (and this, too, is for Devuan's listadmins' attention in
particular):

> There is an (IMO unhappy but least-bad-available) kludge setting in
> Mailman's admin WebUI to make the MLM compensate for DMARC brain-damage:
> You go to Privacy Options, Sender Filters, item 'Action to take when
> anyone posts to the list from a domain with a DMARC Reject/Quarantine
> Policy' aka dmarc_moderation_action.  Change the radio button from
> Accept (default) to Munge from.

I am specifically _not not not_ recommending the similar-looking setting
'Replace the From: header address with the list's posting address to
mitigate issues stemming from the original From: domain's DMARC or
similar policies' aka from_is_list on General Options.  My understanding
is that opting for _that_ version of the kludge unconditionally applies
it to all postings whether they are from DMARC-encumbered domains or
not.

My recommendations:

On Privacy options, Sender filters:
dmarc_moderation_action:  Munge from
dmarc_quarantine_moderation_action):  Yes
dmarc_none_moderation_action:  no

On General options:
change nothing.



 Date: Wed, 2 Aug 2017 16:59:14 -0700
 From: Rick Moen <rick@linuxmafia.com>
 To: dng@lists.dyne.org
 Subject: Re: [DNG] Excessive bounces

Quoting Daniel Abrecht (dng@danielabrecht.ch):

> I'm sorry, I'm using DMARC, and I didn't get the DMARC report about the
> bounced mails, probably because I forgot a DMARC DNS entry for the
> report receiving mail address. I have changed my DMARC policy from
> reject to quarantine for now.

It would be excellent if you could provide any DMARC reports you get to
the Dng listadmins.  Thank you.

Your point is well taken that DMARC and mailing lists can coexist (I've
always concurred with that).  It's just difficult, and creates adverse
consequences.  (As background for this, it's useful to know that DMARC
is a composite and extension of SPF and DKIM.)

As part of the process, the domain's outgoing mail gets certain headers
and body text cryptographically signed and attested to (the DKIM =
DomainKeys Identified Mail part of the standard).  For such mail to
successfully transit a mailing list without breaking validation, the
signed text and headers must be completely unchanged.  This is a very
difficult constraint for MLM software to meet, as occasionally something
gets inserted or changed in a header or elsewhere during normal MLM
processing, and in particular the To: header by design is supposed to
be set upon posting retransmission to the address of each subscriber.

To the best of my recollection (and I'm presently busy and cannot
double-check all of this), some subset of the full SMTP headers are
included in the DKIM attestation.  I can't remember which, nor whether
the DKIM-issuing operator can decide which.  I vaguely recall that the
extra headers MLMs intentionally add, the MLM footer, the MLM
modification to the Subject header (like adding [DNG]), and more are all
somewhat problematic for DKIM validation.

There are a maddeningly large and diverse number of ways to deal with
the problem, and one can spend a lot of time reading about it.  E.g.:
https://dmarc.org/supplemental/mailman-project-mlm-dmarc-reqs.html
http://www.spamresource.com/2014/04/run-email-discussion-list-heres-how-to.html
https://dmarc.org/wiki/FAQ#I_operate_a_mailing_list_and_I_want_to_interoperate_with_DMARC.2C_what_should_I_do.3F


Just a point:

> I use DMARC and believe it to be necessary because it allows me to:
>  1) Make sure nobody can use my E-Mail address to impersonate me or send
> spam

SPF alone _can_ do exactly that without also needing DKIM/DMARC.  (So,
sufficient is correct, but necessary is not quite correct.)

>  2) I will be notified if anyone attempts to do so

SPF alone can prevent it from being possible, hence you don't need to be
notified.  (This of course assumes that receiving domains check SPF for
received mail.  Not all do, but more do than check DMARC.)

>  3) The recipient can check if the message content was changed

gpg signing alone can do that.

If your SMTP message content is being changed, though, you actually have
a lot bigger problems.

>  1) Provide an SPF record. This mailing list doesn't seam to have one

The mailing list isn't an orignator.  It's the originating domains that
ought (to the extent they wish to do so) to have SPF records.


>  2) Don't change anything from the message below the DKIM headers, add
> the other headers before the DKIM signature instead.

To the best of my recollection (I could be misremembering), this is
easier said than done.

Anyway, thank you for your substantive help to the Devuan Project.


----- End forwarded message -----