August 2017 - BALUG-Admin - lists.balug.org

Config settings (was: pre-announce / "soft" open) BALUG-Test list)
by Rick Moen 21 Sep '17

21 Sep '17

1. On https://temp.balug.org/cgi-bin/mailman/admin/balug-test/general , added myself to lisadmin roster. Changed max_days_to_hold from 0 to 5. Rationale: Makes held spam expire out with reasonable speed, while still allowing for listadmin reaction time. (One of my guiding principles of Mailman administration is that if you find yourself trying to outwork mailbots, you're doing it wrong. Along those lines, if you're repeatedly visiting the admin queue to clear it even though you know from the held-mail summaries that it's 100% spam, then you're trying to outwork mailbots.) I personally lead towards 3 rather than 5, sufficient to notice something on a holiday weekend. (On 'conspire', I actually have this as '2'.) Corrected subject_prefix from '[Balug-test]' to '[balug-test]'. Noted without particular comment (except 'OK with me'): You've seemingly toggled send_reminders (monthly password reminder) to 'no'. Last I saw, Mailman installation default is 'yes'. Changed 'respond_to_post_requests' aka 'Send mail to poster when their posting is held for approval?' to 'no'. 2. On https://temp.balug.org/cgi-bin/mailman/admin/balug-test/privacy/sender : Changed 'dmarc_moderation_action' from 'Accept' to 'Munge From'. This is a huge topic where mailing list are collateral damage in the war against SMTP forgery, and where listadmins have only sucky options to choose among. We can discuss this further. (The committee of jackbooted fascists who designed DMARC either hated mailing lists or simply didn't give a damn about them.)

2 29

DreamHost.com - yes, others (besides BALUG) also notice issues
by Michael Paoli 14 Sep '17

14 Sep '17

Yes, we're certainly not the only ones to notice DreamHost.com having issues, e.g.: https://catless.ncl.ac.uk/Risks/30/36#subj22 Just sayin'.

2 3

Status/tracking - DreamHost.com exodus (not there yet, but getting significantly closer)
by Michael Paoli 28 Aug '17

28 Aug '17

Status/tracking - DreamHost.com exodus FYI, added a wiki-page to track at least high-level steps & status: https://www.wiki.balug.org/wiki/doku.php?id=balug:dreamhost_exodus I'm expecting to be fully off DreamHost.com by end of this month, at least if all goes sufficiently and reasonably well. Mostly just takes some available time for me to crank through the remaining steps (and one step - or set of steps, for DreamHost.com to do). Highly optimistically I'd say we'd be done by mid-month ... but more realistically, within 7 to 10 days ... so ... that'd be by 2017-08-22. Let's see if we can make/beat that target! :-)

2 4

Re: [BALUG-Admin] BALUG-Talk and SPF/DKIM
by Rick Moen 21 Aug '17

21 Aug '17

[adding balug-admin] Quoting Glen Martin (glen(a)glen-martin.com) > I expect the alterations are the insertion of a list footer in the > message text. I expect similar insertion in the Subject header to be > problematic as well, though this DMARC didn't complain of that. My understanding is that the sending domain's DKIM policy (portion of DMARC) declares which headers and text portions are attested by cryptographic signing. Failing the DKIM aspect of DMARC means the forwarder host (in this case the MLM host) has altered or inserted into one of the signed areas of the message, and not stripped the DKIM headers. (Some listadmins strip such headers as a way of avoiding the perception of DMARC failure upon retransmission. I'm not sure this is wise. Seems like there might be adverse consequences.) > Here's a header from one of my own messages to the list, having come > back to me through the list and evaluated using my own inbound > testers (opendmarc and opendkim). The two "Authentication-results:" > headers are the problem. For help understanding the headers, I'll > point out that my primary domain on this MX is locutory.org, hence > that name in the headers. Downthread in that offlist conversation with Michael, I notice you said: However, glen-martin.com publishes authorized sending hosts (in SPF), and temp.balug.org (or any other balug.org) isn't on that list. Strike 1. To the best of my understanding, MLM retransmission of messages to subscribers is _not_ going to violate any sender's SPF policy, because the retransmitted message to each subscriber bears a new, different envelope citing the MLM's domain. Hence, on that retransmission, the relevant SPF record that would be consulted upon arrival at the subscriber's MTA is not the poster's domain but rather the MLM's (balug.org's). The example test mail to balug-talk whose full headers you sent to Michael had, as received by you as subscriber: Received-SPF: pass (glen-martin.com: 50.196.148.122 is authorized to use 'glen(a)glen-martin.com' in 'mfrom' identity (mechanism 'ip4:50.196.148.122' matched) (glen-martin.com: 50.196.148.122 is authorized to use 'glen(a)glen-martin.com' in 'mfrom' identity (mechanism 'ip4:50.196.148.122' matched))) If I'm reading that right, temp.balug.org's (IP 198.144.194.238's) MTA software inserted that when it received your original posting, and temp.balug.org considered all to be well at that point, as 50.196.148.122 was one of the declared authorised senders in your domain's SPF RR. temp.balug.org then turned around and retransmitted a fresh copy to each balug-talk subscriber including, of course, you. It bore a totally new SMTP envelope, for which the envelope sender was: Return-Path: <balug-talk-bounces(a)temp.balug.org> So, your receiving MTA, getting the subscriber copy, would have sought to vet transmitting MTA IP 198.144.194.238 (temp.balug.org) against the SPF RR not of _your_ domain (for that copy), but rather of domain balug.org. People get really confused about SPF and mailing list, I notice, forgetting that the MLM's forward is a separate message with a different SMTP envelope (from a different domain). Let's look at the balug.org SPF RR: :r! dig -t txt balug.org +short [null return value] Well, Michael, time to put an SPF RR in the balug.org DNS. This one works for my domain: :r! dig -t txt linuxmafia.com +short "v=spf1 a mx -all" Any questions, please just ask. It says 'use version 1 of SPF protocol, consider the received mail legitimate if it arrives from an IP corresponding with the linuxmafia.com A or MX RR, and please hardfail as a forgery any message purporting to come from linuxmafia.com arriving from any _other_ IP.' > Authentication-Results: mail2.locutory.org (amavisd-new); > dkim=fail (1024-bit key) reason="fail (body has been altered)" > header.d=glen-martin.com Well, could you please compare the message body as you sent it with the message body as you received it in the retransmitted subscriber copy, and tell us what you signed that got changed? What I'm saying is that it's unclear on this end what is the scope of your domain's DKIM cryptographic scrutiny. OK, the 'body has been altered', but what specifically does that mean? Is your domain's DKIM policy so twitchy that Mailman merely adding a Mailman footer breaks it? Does Mailman adding List-Id, List-Subscribe, List-Help, and List-Unsubscribe headers break it? You define and control that DKIM policy for your domain, so perhaps you can tell us. > Balug is also inserting headers/footers into Subject and Body, so the DKIM > checksum fails (message is modified). Strike 2. You cannot expect Mailman mailing lists to _not_ add additional SMTP headers, add a footer, and insert a Subject header tag (like '[BALUG-Talk]'), so it seems to me your DKIM policy ought _not_ to be set to fail if such things are changed by standards-compliant forwarders such as MLMs. Those are things that mailing lists _must_ do in some cases, and traditionally do for excellent reasons (like adding a footer) in others. If you expect mailing lists to not do normal mailing list things, then I submit that IMO your domain's DKIM policy is broken and urgently needs revision.

2 10

Re: [BALUG-Admin] list settings? ...
by Michael Paoli 18 Aug '17

18 Aug '17

[and ... back on list ...] > From: "Rick Moen" <rick(a)linuxmafia.com> > Date: Thu, 17 Aug 2017 06:07:40 -0700 > Quoting Michael Paoli (Michael.Paoli(a)cal.berkeley.edu) > >> Rick, >> >> Please feel most graciously invited to, at your convenience (no >> rush), add yourself as listadmin to BALUG's BALUG-{Announce,Talk} >> list, if/presuming you still wish to do so. > > On the temp.balug.org host, I had already added myself as listadmin to: > balug-admin > balug-test > > I've just added myself to: > BALUG-Talk > BALUG-Announce Much thanks! :-) > 1. How do you feel about lowercasing the real_name and subject_prefix > settings for those? I propose yes. Since BALUG is acronym, rather than name, I prefer to keep it with uppercase for that portion ... and the portion right after the hyphen (-), and particularly in that context, probably makes reasonable sense / better fit, with an initial cap there ... and so it is, also has been that way a long time. And yes, certainly debatable ... visibility vs. annoyance, how to (not) handle an acronym, etc. > 2. Just a note: The merger of the old mbox contents from Dreamhost and > (potentially) other sources of backfill will create a one-time > opportunity to select, if we wish, a different archive_volume_frequency > setting than Monthly (Archiving Options page). BALUG has always used > Monthly, but only because that's the Mailman default rather than as a > conscious choice. > > For announce and test mailing lists, I recommend Yearly. > > For active discussion mailing lists like balug-talk, it's more arguable. > Sometimes Quarterly is nice, sometimes Monthly. > > balug-admin is sufficiently low-traffic that I suggest Yearly. > > Under normal circumstances, there's a strong disincentive against > changing, in that you end up assigning new URLs to existing archived > postings, which breaks offsite links to the significant ones (if any > ;-> ). But if you're merging stuff in, you're going to break most or > all of the existing message URLs anyway. Yes, good considerations. Unfortunately the logic runs contrary to some of the list data. Notably BALUG-Announce and BALUG-Admin (I'll have to review the data again, though, this is off-the-top-of-my-head) have been the least screwed up by DreamHost.com - notably we may have *all* the messages from those lists (or nearly so), and hence it would be good to preserve backwards compatibility on the URLs. I do already have the redirects in place for that - so as long as the sequence numbers line up, the old URLs will redirect to the same messages ... at least after reinjecting those messages, and after lists.balug.org. has been moved over to the new host location. In the meantime, for temp.balug.org, robots.txt tells the search engines to *not* index the archives (less they get confused as numbers will shift relative to what's presently on temp.balug.org) however messages will be injected before lists.balug.org. is on this host where the active lists are now ... and the robots.txt for lists.balug.org on new host location, very much permits indexing by search engines of all the archives. And BALUG-Talk is the one DreamHost.com most thoroughly screwed up. Do have much of that (at least in non-ideal form, but at least have it) to be reinjected, and should soon have raw of the more recent of that ... so, since BALUG-Talk is the one where the numbers and alignment of messages are most likely to shift, as I/we work to try and get as much of the missing messages reinjected, that would be top candidate for potentially changing archive frequency. But it's highest volume, so ought probably stay at monthly anyway. And the other lists, lower volume, so by that quarter or yearly would be good, but as those list (at least I think?) are mostly not missing messages, changing that would break all the (individual message) linkage. So, alas volume vs. archive frequency vs. clean and not-so-clean reinjections and archive fixes to be done, are relatively anti-correlated. :-/ So ... taken all together, I'm relatively inclined to go and remain with monthly. Now ... BALUG-Test on the other hand :-) - that one is a prime candidate to change. I'd guestimate it's activity will be fairish bursts of activity, but otherwise mostly a lot of nothing or nearly so. So, quarterly, or even yearly, probably makes perfectly fine sense there. Should decide before lists.balug.org. moves over. But other than that I think either sounds fine for BALUG-Test. What say ye? Preference/recommendation on BALUG-Test? I'm totally open on that one. And, e.g., on the redirects: Presently have, e.g.: $ curl -I http://lists.balug.org/pipermail/balug-admin-balug.org/2016-April/003013.ht… HTTP/1.1 200 OK Date: Fri, 18 Aug 2017 11:47:02 GMT Server: Apache Last-Modified: Mon, 18 Apr 2016 16:37:20 GMT ETag: "10f5-530c4f972ce72" Accept-Ranges: bytes Content-Length: 4341 Vary: Accept-Encoding Content-Type: text/html Will later have for lists.balug.org (but already in place on temp.balug.org) $ curl -I https://temp.balug.org/pipermail/balug-admin-balug.org/2016-April/003013.ht… HTTP/1.1 301 Moved Permanently Date: Fri, 18 Aug 2017 11:48:28 GMT Server: Apache/2.4.10 (Debian) Location: https://temp.balug.org/pipermail/balug-admin/2016-April/003013.html Content-Type: text/html; charset=iso-8859-1 There's no there there yet, on the redirect target ... but will be after reinjection of archives, and it will be so on lists.balug.org after that's moved over. There's somewhat different scheme on the paths for Debian, than is set up on DreamHost.com. I did also note the redirect bits on: https://www.wiki.balug.org/wiki/doku.php?id=balug:mail_and_lists (look for the strings redirect and/or rewrite - case insensitive, or much of that is currently close to the bottom of that web page) And to keep search engines reasonably happy (and optimize some other bits too), generally better to have *one* canonical location for the (same) content, and (permanent, e.g. 301) redirect to that location, rather than have the exact same content at multiple paths and/or hostnames (search engines tend to "dilute" and down-rank such, at least some fair bit, whereas content at one unique location tends to, in general rank better ... various exceptions, but that's at least approximately the general case).

2 2

Re: [BALUG-Admin] DKIM
by Michael Paoli 18 Aug '17

18 Aug '17

> From: "Rick Moen" <rick(a)linuxmafia.com> > Subject: Re: BALUG-Talk and SPF/DKIM > Date: Thu, 17 Aug 2017 06:31:48 -0700 > To the best of my recollection (and I'm presently busy and cannot > double-check all of this), some subset of the full SMTP headers are > included in the DKIM attestation. I can't remember which, nor whether > the DKIM-issuing operator can decide which. I vaguely recall that the > extra headers MLMs intentionally add, the MLM footer, the MLM > modification to the Subject header (like adding [DNG]), and more are all > somewhat problematic for DKIM validation. Been a while since I looked at it, but as I seem to recall, with DKIM the sender (e.g. MTA) can specify and use within DKIM, exactly which header(s) are included in DKIM - and any headers not specified as included with DKIM are ignored as far as DKIM is concerned. I forget exactly how the body works with DKIM - whether it must be included, or is optional as to whether or not it's included. Anyway, DKIM can be not that horrible - and even useful/beneficial - *if* it's reasonably used. And, it can also be an impossible nightmare if it's used quite improperly. I don't think there's anything in DKIM that prevents one from, e.g. misconfiguring an MTA to DKIM sign headers that ought never ever be signed. At least that's what I seem to recall from earlier. I also recall some handy tool(s) on Linux to (manually) check DKIM on a given, e.g. file or stdin of a full mail message with headers 'n all. Don't recall what tool I used for that though ... let's see if this might help my memory ... $ apropos dkim dkimproxy-sign (1) - computes a DKIM signature for an email message dkimproxy-verify (1) - insert here a description Mail::DKIM (3pm) - Signs/verifies Internet mail with DKIM/DomainKey signa... Mail::DKIM::Algorithm::Base (3pm) - base class for DKIM "algorithms" Mail::DKIM::AuthorDomainPolicy (3pm) - represents an Author Domain Signing Pr... Mail::DKIM::Canonicalization::Base (3pm) - base class for canonicalization me... Mail::DKIM::Canonicalization::DkimCommon (3pm) - common canonicalization methods Mail::DKIM::DkimPolicy (3pm) - represents a DKIM Sender Signing Practices record Mail::DKIM::DkPolicy (3pm) - represents a DomainKeys Sender Signing Policy re... Mail::DKIM::DkSignature (3pm) - represents a DomainKeys-Signature header Mail::DKIM::DNS (3pm) - performs DNS queries for Mail::DKIM Mail::DKIM::Policy (3pm) - abstract base class for originator "signing" policies Mail::DKIM::PrivateKey (3pm) - a private key loaded in memory for DKIM signing Mail::DKIM::Signature (3pm) - represents a DKIM-Signature header Mail::DKIM::Signer (3pm) - generates a DKIM signature for a message Mail::DKIM::SignerPolicy (3pm) - determines signing parameters for a message Mail::DKIM::TextWrap (3pm) - text wrapping module written for use with DKIM Mail::DKIM::Verifier (3pm) - verifies a DKIM-signed message Hmmmm... not sure what I might've used before ... it's been several years or more. Mail::DKIM::Verifier looks probable, or perhaps dkimproxy-verify, but I don't specifically recall.

1 0

Re: [BALUG-Admin] (forw) Mail delivery failed: returning message to sender
by Michael Paoli 18 Aug '17

18 Aug '17

Egad, ... and I just tripped over same rule. Anyway, found and commented that one out. REs I grok, but not Python. At some point I'm inclined to review some of these rules more closely ... notably also determining exactly where they're hitting a false positive match (e.g. line matched). And ... see if the rule might be reasonably "fixable", or ... if it's just too ill conceived for that to be reasonably feasible. At least until then ... that rule is disabled. > From: "Rick Moen" <rick(a)linuxmafia.com> > Subject: (forw) Mail delivery failed: returning message to sender > Date: Thu, 17 Aug 2017 06:41:08 -0700 > Another overaggressive EximConfig rule. > "^.*(ex|former|late).{0,30}(ch(ie|ei)f|comm?ander|defence|dictator|head|leader|minister|president|rebel|ruler).{0,25}(ya|lib(erian?)|nigeria|republic).*\\\$" > > ----- Forwarded message from Mail Delivery System > <Mailer-Daemon(a)linuxmafia.com> ----- > > Date: Thu, 17 Aug 2017 06:32:19 -0700 > From: Mail Delivery System <Mailer-Daemon(a)linuxmafia.com> > To: rick(a)linuxmafia.com > Subject: Mail delivery failed: returning message to sender > > This message was created automatically by mail delivery software. > > A message that you sent could not be delivered to one or more of its > recipients. This is a permanent error. The following address(es) failed: > > balug-admin(a)temp.balug.org > SMTP error from remote mail server after end of data: > host mx.temp.balug.org [198.144.194.238]: 550-Rejected message body text: > Scam text in body: > 550-Former chief/commander/dictator/head of > 550-state/leader/minister/president/rebel/ruler of Libya > 550-. > 550-[EximConfig-2.5-temp.balug.org-Body-Reject]

1 0

Re: [BALUG-Admin] adding 'verified-27440-' _did_ work
by Michael Paoli 18 Aug '17

18 Aug '17

Cool, that's good to know. I thought it didn't when I'd earlier tried it? Or maybe "it depends" ... e.g. "spam" rule/content vs. "prohibited" content (prohibited attachment type, or, egad, a URL with a prohibited ending extension). In any case, I've still work to do on rule tweaking to reduce some false positives. :-/ Oh well, with the "aggressive" rules, figured I'd hit at least some of 'em sooner or later. > From: "Rick Moen" <rick(a)linuxmafia.com> > Subject: Re: Mail delivery failed: returning message to sender > Date: Thu, 17 Aug 2017 06:43:55 -0700 > Just checked, and adding 'verified-27440-' _did_ work.

1 0

Re: [BALUG-Admin] BALUG-Talk and SPF/DKIM
by Rick Moen 18 Aug '17

18 Aug '17

[adding balug-admin:] Quoting Glen Martin (glen(a)glen-martin.com) > Yeah, much agree on your points under "2.". Apologies for the rantiness portions. I do try to have substantive technical content as well. > huh. I'll have to review the configs, I can't remember this option > to conditionally munge. I did it quite a while ago, so may be my > memory at fault. It's definitely there. Help text for that specificMailman WebUI item (Privacy Options, Sender filters, dmarc_moderation_action) says: dmarc_moderation_action (privacy): Action to take when anyone posts to the list from a domain with a DMARC Reject/Quarantine Policy. o Munge From -- applies the 'from_is_list Munge From' [link] transformation to these messages. o Wrap Message -- applies the 'from_is_list Wrap Message' transformation [link] to these messages. o Reject -- this automatically rejects the message by sending a bounce notice to the post's author. The text of the bounce notice can be configured by you. o Discard -- this simply discards the message, with no notice sent to the post's author. This setting takes precedence over the 'from_is_list' [link] setting if the message is From: an affected domain and the setting is other than Accept. Following the first link ('from_is_list Munge From') shows this help text: from_is_list (general): Replace the From: header address with the list's posting address to mitigate issues stemming from the original From: domain's DMARC or similar policies. Several protocols now in wide use attempt to ensure that use of the domain in the author's address (ie, in the From: header field) is authorized by that domain. These protocols may be incompatible with common list features such as footers, causing participating email services to bounce list traffic merely because of the address in the From: field. This has resulted in members being unsubscribed despite being perfectly able to receive mail. The following actions are applied to all list messages when selected here. To apply these actions only to messages where the domain in the From: header is determined to use such a protocol, see the dmarc_moderation_action [link] settings under Privacy options... -> Sender filters. Settings: o No Do nothing special. This is appropriate for anonymous lists. It is appropriate for dedicated announcement lists, unless the From: address of authorized posters might be in a domain with a DMARC or similar policy. It is also appropriate if you choose to use dmarc_moderation_action other than Accept for this list. o Munge From This action replaces the poster's address in the From: header with the list's posting address and adds the poster's address to the addresses in the original Reply-To: header. o Wrap Message Just wrap the message in an outer message with the From: header containing the list's posting address and with the original From: address added to the addresses in the original Reply-To: header and with Content-Type: message/rfc822. This is effectively a one message MIME format digest. The transformations for anonymous_list are applied before any of these actions. It is not useful to apply actions other than No to an anonymous list, and if you do so, the result may be surprising. The Reply-To: header munging actions below interact with these actions as follows: first_strip_reply_to = Yes will remove all the incoming Reply-To: addresses but will still add the poster's address to Reply-To: for all three settings of reply_goes_to_list which respectively will result in just the poster's address, the poster's address and the list posting address or the poster's address and the explicit reply_to_address in the outgoing Reply-To: header. If first_strip_reply_to = No the poster's address in the original From: header, if not already included in the Reply-To:, will be added to any existing Reply-To: address(es). These actions, whether selected here or via dmarc_moderation_action [link] do not apply to messages in digests or archives or sent to usenet via the Mail<->News gateways. If dmarc_moderation_action [link] applies to this message with an action other than Accept, that action rather than this is applied

1 0

Re: [BALUG-Admin] BALUG-Talk and SPF/DKIM
by Rick Moen 17 Aug '17

17 Aug '17

I wrote: > To clarify, DMARC is an omnibus package, invented by Yahoo, of > anti-forgery technologies that incorporates Yahoo's DKIM (DomainKey > Identified Mail, a successor to Yahoo's DomainKeys) and Meng Wong's SPF. > > Mailing list manager (MLM) package have no problem with SPF because > they supply an entirely fresh SMTP envelope on the retransmission to > subscribers. In this, they differ from cross-system /etc/aliases and > ~/.forward entries, which do _not_ supply a new envelope. > > MLM packages have big problems with Yahoo's standards (the DKIM portion > of DMARC, and legacy DomainKeys) because MLM's modifications to posting > text and headers upon retranmission to subscribers have a strong tendency > to break DKIM's (and DomainKeys's) cryptographic attestation about the > integrity of the text and headers. > > The specific Mailman munging kludge I recommended addresses this problem > by dumbing down Mailman's treatment of the internal 'From: ' header > where necessary to avoid breaking DKIM (/DomainKeys) attestation. The > cost of this is grievous, substitution of the mailing list's originating > address for the user's, but this is the least amount of damage that > appears able to make Mailman cope with what IMO is a really crappy and > MLM-hostile antiforgery design. I elaborated on this recently on Devuan's main discussion list Dng, because a couple of the regulars got this wrong, lumping in SPF with the MLM-problematic Yahoo anti-forgery methods, and underestimating the problems caused by the latter. The subject arose on Dng because that mailing list's mail was running into rejects on postings sent from, yes, yahoo.com and to a lesser extent GMail, and I had a well-founded suspicion that the problem was DKIM/DMARC. The fellow I address below, Daniel Abrecht, is a fan of DMARC. (Below that, I include other relevant postings.) Date: Thu, 3 Aug 2017 20:47:46 -0700 From: Rick Moen <rick(a)linuxmafia.com> To: dng(a)lists.dyne.org Subject: Re: [DNG] Excessive bounces Daniel, again, thank you for your efforts to collect diagnostic data for the Devuan Project, and for your care and precision. I respect the reasons you're making DMARC work for your domain, and certainly have no argument with you doing so (though I do not come to the same conclusion for my own domain). A few comments that come to mind: Quoting Daniel Abrecht (dng(a)danielabrecht.ch) > The SPF checks from the report have currently the result None, which > usually indicates that domain does not have an SPF record. I did add > ?mx:lists.dyne.org to my SPF record, which should have resulted in an > SPF result of Neutral, so either the mailing list needs an SPF record, > or google & co. are wrongly reporting a None result instead of a > Neutral one. Let's be careful about what we're speaking about, here. By definition, a posting that transits a mailing list goes through two phases, where to the best of my understanding a different SPF RR is relevant for each. In the first phase, mail arrives at (say) the lists.dyne.org MTA purporting to be from your domain. If the receiving MTA is checking SPF on arriving mail, then it seeks to validate the envelope header of arrived mail against the published SPF RR of the claimed sender domain as reflected in the envelope. A forged mail at this point would have an envelope claiming it's from danielabrecht.ch, but the IP would fail vetting against your A and MX records (declared as authorised senders in your domain's SPF RR). If it's genuine, otherwise. The accepted posting gets handed by the receiving MTA to the MLM software. The MLM software now remails, or to put it another way, creates fresh mails (thus, second phase), with an entirely different SMTP envelope reflecting the MLM's host identity. The internal 'From: ' header is (ideally) left intact, the internal 'To: ' header is customised for each subscriber, and some number of other additions and changes get made by the MLM software prior to handing it off to the outbound MTA. If subscribers' receiving MTAs now attempt, during this second phase, to validate the SMTP envelope, they will do so based on the MLM host's domain, not the original sender's. > When a mail is sent, there are the envelope-to and the envelope-from > (which aren't mail headers), but also To and From headers.[...] Yes, I'm extremely well aware of this. The latter 'From header' is traditionally called the internal 'From:' header to distinguish it from the envelope 'From ' header. I was on NANAE during the incident in which envelope forgery was invented and demonstrated in the famous revenge-spam attack against Joe Doll of Joe's Cyberpost. > Normally when an SPF check is made, the envelope from address is used. > If the mail server doesn't have an SPF header, the SPF result is None > and the receiving mail server should accept the mail. To my knowledge, there is no such thing as an 'SPF header' (except see below). No extra headers get used to implement SPF. It's just a DNS RR that declares what authorised sending MTA hosts exist, and gets used by supporting (receiving) MTAs to vet the envelope sender. (There's also an optional 'Authentication-Results' header that's similar, except for recording border MTAs' results.) My understanding is that some MTAs add a Received-SPF header to all emails arriving via SMTP that test to any result other than 'fail'. This is added _after_ SPF evaluation to record the results of that check. It's advisory, and merely a place to record the result. > There is no point in getting notified if nothing happens or everything > works as expected, but if something didn't work or someone tried to > send a mail in my name and failed, I certainly want to know about > that. Certainly your prerogative, but I personally don't care to get notified when someone tries to forge my domain and fails, because IMO that amounts to getting spam about spam. There's a reason why I've finely tuned logcheck to stop notifying me of pointless and futile attempts to use generic username/password pairs on my sshd, and a thousand other bits of basically meaningless noise that's just part of the routine of having a 24x7 Internet server. > >> 3) The recipient can check if the message content was changed > > > > gpg signing alone can do that. > > gpg and DKIM have a bit different scope. Yes, but both can authenticate and attest to the integrity of whatever dataset was signed. And that thus matches 'can check if the message content was changed'. Just crypto-sign the message contents. Anything extraneous inserted into the signed bloc will cause validation failure. Anything outside it will be obviously _not_ attested to, and recipients should accordingly understand that. You might have more reasonably made the objection 'Ordinary folks won't check PGP attestation.' Maybe not. Personally, I regard this as Not My Problem. If I say something where text integrity really matters, like the body of a CVE, I will PGP-sign it. Readers who care about text integrity will check it, others won't. Date: Thu, 3 Aug 2017 09:15:17 -0700 From: Rick Moen <rick(a)linuxmafia.com> To: dng(a)lists.dyne.org Subject: Re: [DNG] Excessive bounces Quoting Simon Hobson (linux(a)thehobsons.co.uk) > SPF breaks mailing lists and mail forwarders - and this is NOT (IMO) > fixable without introducing a wide open front gate for spammers to > ride through and completely bypass SPF. No. it does not break mailing lists. It _does_ break other common types of forwarders unless they adopt SRS-wrapping. The reason it does not adversely affect mailing lists is that SPF validates only the envelope header: The receving MTA verifies that the delivering MTA's IP address is mentioned in the claimed sending domain's SPF RR (if there is one in that domain's DNS). Consider the envelope header of your own Dng posting, as received by linuxmafia.com's MTA when it received my subscription's copy. Here's the way your post arrived: From dng-bounces(a)lists.dyne.org Thu Aug 03 05: 8:39 2017 Return-path: <dng-bounces(a)lists.dyne.org> Envelope-to: rick(a)linuxmafia.com So, the envelope sender's domain was dyne.org, not thehobsons.co.uk, and my receiving MTA will perform a DNS check against the former. :r! dig -t txt dyne.org +short "google-site-verification=6FghqJroXIvBY8cutq6ouO0RC-a8qynFu6sJR3S-IbA" "v=spf1 mx ip4:178.62.188.7/32 ip4:188.226.191.63/32 ip4:213.127.180.241/32 -all" "google-site-verification=2XoWrMMTQ7jmgcB_76Y_TQSnWDGhR4e-y_KLqoKOK1Q" :r! dig lists.dyne.org +short 178.62.188.7 And, lo! The envelope sender does validate. You are probably confusing mailing lists, which provide new envelope headers during forwarding citing the forwarding domain, with other forwarders like /etc/alias entries and ~/.forward files. It's the _latter_ that SPF author Meng Wong invented that goofy Sender Rewriting System. Mailing lists, by contrast, don't have the problem he invented that kludge to fix. And, again, Simon, my mail domain linuxmafia.com has had an SPF hardfail directive in its DNS since around 2003, and the specifier is extremely narrow: :r! dig -t txt linuxmafia.com +short "v=spf1 a mx -all" That says 'If a mail's envelope header claims it's from linuxmafia.com, but the delivering MTA doesn't match either linuxmafia.com's DNS A record or its MX record, please consider it definitively a forgery.' I'm on _many_ mailing lists on many hosts. If my mailing list mail had a deliverability problem caused by hardfailing forgeries of my envelope header, I'd have figured that out, some time over hte past 14 years. It does not happen, because mailing lists work better than /etc/alias entries and ~/.forward files by design. Earlier posts upthread: Date: Wed, 2 Aug 2017 10:41:27 -0700 From: Rick Moen <rick(a)linuxmafia.com> To: dng(a)lists.dyne.org Subject: Re: [DNG] Excessive bounces Quoting Jaromil (jaromil(a)dyne.org) > I am a bit puzzled about this one, we had some reports of the problem > so far, which hasn't occurred before on any other dyne list and is not > really reproducible. > > what we notice is that our mail server is under some quite heavy load > and we are working to move it to a bigger infrastructure by september > > however it is highly available already and seems to process > everything, so I'm not really sure what is happening... any insight is > welcome. Might be DMARC validation failure. (Gods, do I ever detest that stuff.) DMARC is proving to be an utter nightmare for mailing lists, in as much as they are mail forwarders, and DMARC was IMO botched in its ability to accomodate the way they work. From memory, and so I'm probably dropping a bunch of detail: Because MLMs such as Mailman (appropriately) change the internal SMTP headers upon retransmitting the poster's mail to subscribers (notably the To: header), it no longer validates against the sender's domain if it is a DMARC-using one with a strict policy. Yahoo and Gmail are examples of sending domains with strict DMARC policies. There is an (IMO unhappy but least-bad-available) kludge setting in Mailman's admin WebUI to make the MLM compensate for DMARC brain-damage: You go to Privacy Options, Sender Filters, item 'Action to take when anyone posts to the list from a domain with a DMARC Reject/Quarantine Policy' aka dmarc_moderation_action. Change the radio button from Accept (default) to Munge from. To quote the help text: from_is_list (general): Replace the From: header address with the list's posting address to mitigate issues stemming from the original From: domain's DMARC or similar policies. Several protocols now in wide use attempt to ensure that use of the domain in the author's address (ie, in the From: header field) is authorized by that domain. These protocols may be incompatible with common list features such as footers, causing participating email services to bounce list traffic merely because of the address in the From: field. This has resulted in members being unsubscribed despite being perfectly able to receive mail. The following actions are applied to all list messages when selected here. To apply these actions only to messages where the domain in the From: header is determined to use such a protocol, see the dmarc_moderation_action settings under Privacy options... -> Sender filters. Settings: [...] Munge From This action replaces the poster's address in the From: header with the list's posting address and adds the poster's address to the addresses in the original Reply-To: header. So, for example, _if_ my sending domain linuxmafia.com had a strong DMARC policy (which it doesn't, because I hate DMARC with a passion), then the 'Munge from' setting would cause my post to Dng to get this 'From: ' header upon retransmission to subscribers: From: Rick Moen via Dng <dng(a)lists.dyne.org> instead of the normal From: Rick Moen <rick(a)linuxmafia.com> The reason this helps sidestep DMARC validation is that it's now no longer considered needing validation against linuxmafia.com's (hypothetical) DMARC policy, but rather dyne.org's. I personally detest this solution because, when I send out my sending address on a mailing list, it is deliberately there so that people can, if necessary, contact me offlist. The kludge complicates this, albeit, if I remember correctly, it tries to compensate for the brain-damage by inserting a Reply-To as well. It should be noted that the Munge from kludge thus alters -only- the postings of subscribers from DMARC-damaged^H^H^H^W^W^W^Wusing domains, so only _some_ postings will get disfigured in this manner. Sadly, I recommend opting for this kludge, because otherwise deliverability suffers. Date: Wed, 2 Aug 2017 10:43:37 -0700 From: Rick Moen <rick(a)linuxmafia.com> To: dng(a)lists.dyne.org Subject: Re: [DNG] Excessive bounces Quoting Simon Hobson (linux(a)thehobsons.co.uk) > That's the important thing to look for - and my money is it's related > to SPF and/or DMARC. It won't be SPF. My domain has a strong SPF policy: :r! dig -t txt linuxmafia.com +short "v=spf1 a mx -all" ...and no mailing list post from me or my users violates it. It'll be DMARC. (Long experience says.) Date: Wed, 2 Aug 2017 10:54:57 -0700 From: Rick Moen <rick(a)linuxmafia.com> To: dng(a)lists.dyne.org Subject: Re: [DNG] Excessive bounces Oh, follow-up (and this, too, is for Devuan's listadmins' attention in particular): > There is an (IMO unhappy but least-bad-available) kludge setting in > Mailman's admin WebUI to make the MLM compensate for DMARC brain-damage: > You go to Privacy Options, Sender Filters, item 'Action to take when > anyone posts to the list from a domain with a DMARC Reject/Quarantine > Policy' aka dmarc_moderation_action. Change the radio button from > Accept (default) to Munge from. I am specifically _not not not_ recommending the similar-looking setting 'Replace the From: header address with the list's posting address to mitigate issues stemming from the original From: domain's DMARC or similar policies' aka from_is_list on General Options. My understanding is that opting for _that_ version of the kludge unconditionally applies it to all postings whether they are from DMARC-encumbered domains or not. My recommendations: On Privacy options, Sender filters: dmarc_moderation_action: Munge from dmarc_quarantine_moderation_action): Yes dmarc_none_moderation_action: no On General options: change nothing. Date: Wed, 2 Aug 2017 16:59:14 -0700 From: Rick Moen <rick(a)linuxmafia.com> To: dng(a)lists.dyne.org Subject: Re: [DNG] Excessive bounces Quoting Daniel Abrecht (dng(a)danielabrecht.ch) > I'm sorry, I'm using DMARC, and I didn't get the DMARC report about the > bounced mails, probably because I forgot a DMARC DNS entry for the > report receiving mail address. I have changed my DMARC policy from > reject to quarantine for now. It would be excellent if you could provide any DMARC reports you get to the Dng listadmins. Thank you. Your point is well taken that DMARC and mailing lists can coexist (I've always concurred with that). It's just difficult, and creates adverse consequences. (As background for this, it's useful to know that DMARC is a composite and extension of SPF and DKIM.) As part of the process, the domain's outgoing mail gets certain headers and body text cryptographically signed and attested to (the DKIM = DomainKeys Identified Mail part of the standard). For such mail to successfully transit a mailing list without breaking validation, the signed text and headers must be completely unchanged. This is a very difficult constraint for MLM software to meet, as occasionally something gets inserted or changed in a header or elsewhere during normal MLM processing, and in particular the To: header by design is supposed to be set upon posting retransmission to the address of each subscriber. To the best of my recollection (and I'm presently busy and cannot double-check all of this), some subset of the full SMTP headers are included in the DKIM attestation. I can't remember which, nor whether the DKIM-issuing operator can decide which. I vaguely recall that the extra headers MLMs intentionally add, the MLM footer, the MLM modification to the Subject header (like adding [DNG]), and more are all somewhat problematic for DKIM validation. There are a maddeningly large and diverse number of ways to deal with the problem, and one can spend a lot of time reading about it. E.g.: https://dmarc.org/supplemental/mailman-project-mlm-dmarc-reqs.html http://www.spamresource.com/2014/04/run-email-discussion-list-heres-how-to.… https://dmarc.org/wiki/FAQ#I_operate_a_mailing_list_and_I_want_to_interoper… Just a point: > I use DMARC and believe it to be necessary because it allows me to: > 1) Make sure nobody can use my E-Mail address to impersonate me or send > spam SPF alone _can_ do exactly that without also needing DKIM/DMARC. (So, sufficient is correct, but necessary is not quite correct.) > 2) I will be notified if anyone attempts to do so SPF alone can prevent it from being possible, hence you don't need to be notified. (This of course assumes that receiving domains check SPF for received mail. Not all do, but more do than check DMARC.) > 3) The recipient can check if the message content was changed gpg signing alone can do that. If your SMTP message content is being changed, though, you actually have a lot bigger problems. > 1) Provide an SPF record. This mailing list doesn't seam to have one The mailing list isn't an orignator. It's the originating domains that ought (to the extent they wish to do so) to have SPF records. > 2) Don't change anything from the message below the DKIM headers, add > the other headers before the DKIM signature instead. To the best of my recollection (I could be misremembering), this is easier said than done. Anyway, thank you for your substantive help to the Devuan Project. ----- End forwarded message -----

1 0