+balug-admin(a)lists.balug.org - don't send/cc/bcc to that email if
you're not on that list
Oh, and most of the timezones are UTC, unless indicated otherwise.
So ... first we look at full headers of the received. To make that
simpler, I first unfolded (any header lines starting with a space I
joined to line above), then removed all lines except those with a date
or timestamp or the like and also retaining those that might be
relatively usefully informative, and dropped the rest. Then for
readability I (re)folded those more than 72 characters wide, with a
space indent where folded. And I preserved ordering. That leaves us
with:
Received: by 2002:aa7:da18:0:b0:568:550c:e550 with SMTP id
r24csp203374eds; Thu, 14 Mar 2024 00:51:42 -0700 (PDT)
X-Received: by 2002:a05:6808:1285:b0:3c2:18fe:f810 with SMTP id
a5-20020a056808128500b003c218fef810mr1187019oiw.27.1710402702247; Thu,
14 Mar 2024 00:51:42 -0700 (PDT)
Return-Path: <balug-announce-bounces(a)lists.balug.org>
Received: from balug-sf-lug-v2.balug.org (balug.org.
[2001:470:1f05:19e::2]) by mx.google.com with ESMTPS id
e17-20020aca1311000000b003c365f41a1fsi65493oii.37.2024.03.14.00.51.41
for <michael.paoli(a)cal.berkeley.edu> (version=TLS1_3
cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 14 Mar 2024 00:51:42
-0700 (PDT)
Received: from localhost ([127.0.0.1] helo=balug.org) by
balug-sf-lug-v2.balug.org with esmtp (Exim 4.92) (envelope-from
<balug-announce-bounces(a)lists.balug.org>) id 1rkfsB-0002wb-L7; Thu, 14
Mar 2024 07:51:31 +0000
Received: from shell1.rawbw.com ([198.144.192.42]) by
balug-sf-lug-v2.balug.org with esmtp (Exim 4.92) (envelope-from
<Michael.Paoli(a)cal.berkeley.edu>) id 1rcPqS-0002KU-Ko for
balug-announce(a)lists.balug.org; Tue, 20 Feb 2024 13:07:36 +0000
Received: from webmail.rawbw.com (mail0.rawbw.com [198.144.192.41]) by
shell1.rawbw.com (8.15.1/8.15.1) with ESMTP id 41KD7ZwB068218; Tue, 20
Feb 2024 05:07:35 -0800 (PST) (envelope-from
Michael.Paoli(a)cal.berkeley.edu)
Date: Tue, 20 Feb 2024 05:07:35 -0800
To: BALUG-Announce <balug-announce(a)lists.balug.org>
Message-ID: <94cc17d1e831608786fb59c3a4eb70a9(a)cal.berkeley.edu>
X-Sender: Michael.Paoli(a)cal.berkeley.edu
Received-SPF: none client-ip=198.144.192.42;
envelope-from=Michael.Paoli(a)cal.berkeley.edu; helo=shell1.rawbw.com
Subject: [BALUG-Announce] BALUG: meeting TODAY!: Tu 2024-02-20 & other
BALUG News
From: Michael Paoli via BALUG-Announce <balug-announce(a)lists.balug.org>
Reply-To: Michael Paoli <Michael.Paoli(a)cal.berkeley.edu>
Errors-To: balug-announce-bounces(a)lists.balug.org
Sender: BALUG-Announce <balug-announce-bounces(a)lists.balug.org>
So, we see the big time gap here:
Received: from localhost ([127.0.0.1] helo=balug.org) by
balug-sf-lug-v2.balug.org with esmtp (Exim 4.92) (envelope-from
<balug-announce-bounces(a)lists.balug.org>) id 1rkfsB-0002wb-L7; Thu, 14
Mar 2024 07:51:31 +0000
Received: from shell1.rawbw.com ([198.144.192.42]) by
balug-sf-lug-v2.balug.org with esmtp (Exim 4.92) (envelope-from
<Michael.Paoli(a)cal.berkeley.edu>) id 1rcPqS-0002KU-Ko for
balug-announce(a)lists.balug.org; Tue, 20 Feb 2024 13:07:36 +0000
Those Received: headers are read from bottom up in order of processing,
newer Received: headers are added atop any existing. That's not only
useful for tracing route of mail, but these also generally include
timestamps.
So, I sent it from one of my ISP's email to the list, and that was
received, then it took a long time for the next hop - which was entirely
local on the same host - so something went wrong there between MTA
and/or mailman. If I convert those timestamps to ISO
format, have:
2024-02-20T13:07:36+0000
2024-03-14T07:51:31+0000
$ who -r
run-level 5 Mar 14 06:51
So, between that and a presumed exponential back-off on retries on the
sending MTA, that would presumably account for when the MTA received
that, presuming for the period before that there was some reason it
couldn't or wouldn't. And looking at receiving MTA logs, no shortage
of logs between those timestamps, so ... what happened? We do have:
Message-ID: <94cc17d1e831608786fb59c3a4eb70a9(a)cal.berkeley.edu>
and that should be unique, so can be quite useful for, e.g. searching
in mail logs. What (if anything) can we find for that in the time range
in question?
MTA exim4 ... I don't have mainlog files going back that far, but I do
have rejectlog files going back that far. In chronological order by
mtime for files covering range:
2024-02-20T13:07:36+0000
2024-03-14T07:51:31+0000
have files matching these patterns:
rejectlog.{5{9,8,7,6,5,4,3,2,1,0},4{9,8,7,6,5,4,3,2,1,0},3{9,8,7,6,5,4,3,2,1,0},2{9,8,7}}.gz
paniclog
rejectlog.{2{6,5,4,3,2,1,0},1{9,8,7,6,5,4,3,2,1,0}}.gz
mainlog.10.gz
eval echo \{rejectlog,mainlog\}{.{9,8,7,6,5,4,3,2,1}.gz,}
checking for that Message-ID, have:
(
Message_ID='94cc17d1e831608786fb59c3a4eb70a9(a)cal.berkeley.edu'
eval set -- \
rejectlog.{5{9,8,7,6,5,4,3,2,1,0},4{9,8,7,6,5,4,3,2,1,0},3{9,8,7,6,5,4,3,2,1,0},2{9,8,7}}.gz
\
paniclog \
rejectlog.{2{6,5,4,3,2,1,0},1{9,8,7,6,5,4,3,2,1,0}}.gz \
mainlog.10.gz \
\{rejectlog,mainlog\}{.{9,8,7,6,5,4,3,2}.gz,.1,}
for f
do
case "$f" in
*.gz)
gzip -d < "$f" |
grep -F -q -e "$Message_ID" &&
{
echo "$f"
gzip -d < "$f" | grep -F -e "$Message_ID"
}
;;
*)
grep -F -l -e "$Message_ID" "$f" &&
grep -F -e "$Message_ID" "$f"
;;
esac
done
)
mainlog.5.gz
2024-03-14 07:51:31 1rkfsB-0002wb-L7 <=
balug-announce-bounces(a)lists.balug.org H=localhost (balug.org)
[127.0.0.1] P=esmtp S=7761
id=94cc17d1e831608786fb59c3a4eb70a9(a)cal.berkeley.edu
2024-03-14 07:51:31 1rkfsB-0002wb-Me <=
balug-announce-bounces(a)lists.balug.org H=localhost (balug.org)
[127.0.0.1] P=esmtp S=7761
id=94cc17d1e831608786fb59c3a4eb70a9(a)cal.berkeley.edu
2024-03-14 07:51:31 1rkfsB-0002wb-TW <=
balug-announce-bounces(a)lists.balug.org H=localhost (balug.org)
[127.0.0.1] P=esmtp S=7761
id=94cc17d1e831608786fb59c3a4eb70a9(a)cal.berkeley.edu
2024-03-14 07:51:31 1rkfsB-0002wb-UG <=
balug-announce-bounces(a)lists.balug.org H=localhost (balug.org)
[127.0.0.1] P=esmtp S=7761
id=94cc17d1e831608786fb59c3a4eb70a9(a)cal.berkeley.edu
2024-03-14 07:51:41 1rkfsB-0002wb-L7 => <REDACTED>
...
So, basically it comes in (apparently goes through multiple times in
processing), and then lots of sends out very shortly after that.
Unfortunately don't have the earlier mainlog files, so not sure what may
have happened then - at least as far as MTA goes.
Let's see about mailman. For log files in the relevant time frame, and
ignoring those which are compressed empty files (yeah, I should have
logrotate not compress empty files), these are the files in
chronological order (by mtime):
vette.4.gz
mischief.1
error.3.gz
error.2.gz
security
qrunner.5.gz
smtp-failure.5.gz
locks.5.gz
smtp.5.gz
qrunner.4.gz
bounce.1
subscribe
And going through those files, nothing relevant in the applicable
time frame.
So ... thus far best guess is something that was essentially somewhere
along that chain failed or wedged or died or crashed or the like,
didn't particularly log anything indicating the problem in the
meantime, and things went back to "normal" and got unjammed after host
rebooted.
Oh, mailman did also add it to the archives ... can we see when, that
may provide a clue.
>From Michael.Paoli(a)cal.berkeley.edu Tue Feb 20 13:07:36 2024
Received: from shell1.rawbw.com ([198.144.192.42]) by
balug-sf-lug-v2.balug.org with esmtp (Exim 4.92) (envelope-from
<Michael.Paoli(a)cal.berkeley.edu>) id 1rcPqS-0002KU-Ko for
balug-announce(a)lists.balug.org; Tue, 20 Feb 2024 13:07:36 +0000
Received: from webmail.rawbw.com (mail0.rawbw.com [198.144.192.41]) by
shell1.rawbw.com (8.15.1/8.15.1) with ESMTP id 41KD7ZwB068218; Tue, 20
Feb 2024 05:07:35 -0800 (PST) (envelope-from
Michael.Paoli(a)cal.berkeley.edu)
From: Michael Paoli <Michael.Paoli(a)cal.berkeley.edu>
To: BALUG-Announce <balug-announce(a)lists.balug.org>
Message-ID: <94cc17d1e831608786fb59c3a4eb70a9(a)cal.berkeley.edu>
X-Sender: Michael.Paoli(a)cal.berkeley.edu
Subject: [BALUG-Announce] BALUG: meeting TODAY!: Tu 2024-02-20 & other
BALUG News
X-BeenThere: balug-announce(a)lists.balug.org
Date: Tue, 20 Feb 2024 13:07:36 -0000
X-Original-Date: Tue, 20 Feb 2024 05:07:35 -0800
X-List-Received-Date: Tue, 20 Feb 2024 13:07:36 -0000
At least what shows in the archive (complete mbox file), from the full
headers, looks like it got it in timely manner. What about where
it processes those for showing on web?
# pwd -P && ls -ontr $(find * ! -mtime +30 -type f -print | fgrep Feb)
/var/lib/mailman/archives/private/balug-announce
-rw-rw-r-- 1 38 5689 Mar 14 06:51 2024-February.txt
-rw-rw-r-- 1 38 1729 Mar 14 06:51 2024-February/date.html
-rw-rw-r-- 1 38 1723 Mar 14 06:51 2024-February/subject.html
-rw-rw-r-- 1 38 1725 Mar 14 06:51 2024-February/author.html
-rw-rw-r-- 1 38 1752 Mar 14 06:51 2024-February/thread.html
-rw-rw-r-- 1 38 8276 Mar 14 06:51 2024-February/000340.html
-rw-rw---- 1 38 136 Mar 14 06:51 database/2024-February-thread
-rw-rw---- 1 38 183 Mar 14 06:51 database/2024-February-subject
-rw-rw---- 1 38 131 Mar 14 06:51 database/2024-February-date
-rw-rw---- 1 38 150 Mar 14 06:51 database/2024-February-author
-rw-rw---- 1 38 1259 Mar 14 06:51 database/2024-February-article
-rw-rw-r-- 1 38 2687 Mar 15 03:27 2024-February.txt.gz
#
Yeah, it would seem to implicate a problem within mailman.
Looks like our sequence went about like this:
2024-02-20T13:07:36+0000 email generated on sending MTA,
looks like it made it to receiving MTA and into mailman,
notably mailman show it in the mbox full archive with
corresponding timestamps, but then looks like it stuck at some point
after that, notably not even the archiving was fully processed,
as the web archive files didn't catch up until
2024-03-14T06:51+0000
we have a reboot
$ who -r
run-level 5 Mar 14 06:51
And the mailman web archive files catch up. Then, an hour later:
2024-03-14T07:51:31+0000
mail to list recipients is actually sent out (perhaps some retry logic
caused the hour's delay at that point).
Anyway, that's my best guestimation based on the evidence.
---------- Forwarded message ---------
From: Michael Paoli <michael.paoli(a)cal.berkeley.edu>
Date: Thu, Mar 14, 2024 at 8:26 PM
Subject: Re: [BALUG-Announce] BALUG: meeting TODAY!: Tu 2024-02-20 &
other BALUG News
To: aaronco36 <aaronco36(a)sdf.org>, <balug-admin(a)lists.balug.org>
Cc: Rick Moen <rick(a)linuxmafia.com>
+balug-admin(a)lists.balug.org - don't send/cc/bcc to that email if
you're not on that list
Yeah, ... I noticed that too - haven't dug down yet to figure out what happened,
but some list mail went out (or went out again?) very late - like weeks late.
And like I say, haven't dug down to the bottom of it to figure out what happened
yet. "Other than that" didn't notice anything unusual. But ... maybe
some queue
got stuck or some demon that was supposed to be running wasn't, and then
perhaps with some reboot or update or the like, somehow things got unjammed
and/or reprocessed? Still guessing a bit at this point - will need to review
logs and some full headers and other details to figure out what happened.
Anyway, maybe I get to it sometime this weekend(ish) or so. Shouldn't have
happened, but there isn't an "undo" for mail sent, so 'bout best can do with it
is figure out (hopefully) how it happened, and hopefully prevent a
repeat of that.
Possible some queue got stuck or daemon down or crashed or stuck, and I didn't
catch it earlier, and things just got inordinately delayed ... that'd
at least be my first
guess, though there are certainly other possibilities.
And it wasn't just that one BALUG-Announce mail, but a moderate spattering of
mailman and/or exim4 mail that was impacted - all came through at once.
I think I saw about 20 emails or so (most of which were for list admin
or the like)
all come through at or right around the same time.
Anyway, probably answers to be found in the full headers and/or other
log data and such.
On Thu, Mar 14, 2024 at 7:58 PM aaronco36 <aaronco36(a)sdf.org> wrote:
>
> Hello Rick and Michael,
>
> > Quoting aaronco36 (aaronco36(a)sdf.org):
> >
> >> ????
> >> Now ~12:15 PDT (UTC-07:00) on Thursday 2024-03-14.
> >>
> >> aaronco36(at)SDF.org
> >
> > I'm not sure what you're talking about Aaron. The most recent
> > balug-announce posting (this one, from Michael P.:)
> > https://lists.balug.org/pipermail/balug-announce/2024-February/000340.html)
> > really did go out on Tue Feb 20 13:07:36 UTC 2024, not today.
> >
> >
>
> Hmmmm, mystery to me as well :-\
> Perhaps very delayed transmission having something to do with the top pair
> of "Received:" sections dated this morning, below ??
>
> -A
>
> Contents of the Full Header of today's message:
> ~~~~~~~~~~~~~~~~ quoting in full ~~~~~~~~~~~~~~~~~~~~~
>
> Return-Path: <balug-announce-bounces(a)lists.balug.org>
> Received: from balug-sf-lug-v2.balug.org (balug.org [96.86.170.229])
> by mx.sdf.org (8.16.1/8.14.3) with ESMTPS id 42E8VA6p012184
> (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256 bits) verified
> NO)
> for <aaronco36(a)sdf.org>; Thu, 14 Mar 2024 08:31:10 GMT
> Received: from localhost ([127.0.0.1] helo=balug.org)
> by balug-sf-lug-v2.balug.org with esmtp (Exim 4.92)
> (envelope-from <balug-announce-bounces(a)lists.balug.org>)
> id 1rkfsB-0002wb-Me; Thu, 14 Mar 2024 07:51:31 +0000
> Received: from shell1.rawbw.com ([198.144.192.42])
> by balug-sf-lug-v2.balug.org with esmtp (Exim 4.92)
> (envelope-from <Michael.Paoli(a)cal.berkeley.edu>) id 1rcPqS-0002KU-Ko
> for balug-announce(a)lists.balug.org; Tue, 20 Feb 2024 13:07:36 +0000
> Received: from webmail.rawbw.com (mail0.rawbw.com [198.144.192.41])
> by shell1.rawbw.com (8.15.1/8.15.1) with ESMTP id 41KD7ZwB068218;
> Tue, 20 Feb 2024 05:07:35 -0800 (PST)
> (envelope-from Michael.Paoli(a)cal.berkeley.edu)
> X-Authentication-Warning: shell1.rawbw.com: Host mail0.rawbw.com
> [198.144.192.41] claimed to be webmail.rawbw.com
> MIME-Version: 1.0
> Date: Tue, 20 Feb 2024 05:07:35 -0800
> To: BALUG-Announce <balug-announce(a)lists.balug.org>
> Message-ID: <94cc17d1e831608786fb59c3a4eb70a9(a)cal.berkeley.edu>
> X-Sender: Michael.Paoli(a)cal.berkeley.edu
> Received-SPF: none client-ip=198.144.192.42;
> envelope-from=Michael.Paoli(a)cal.berkeley.edu; helo=shell1.rawbw.com
> Subject: [BALUG-Announce] BALUG: meeting TODAY!: Tu 2024-02-20 & other BALUG
> News
> X-BeenThere: balug-announce(a)lists.balug.org
> X-Mailman-Version: 2.1.29
> Precedence: list
> List-Id: Announcements for and by BALUG <balug-announce.lists.balug.org>
> List-Unsubscribe:
> <https://lists.balug.org/cgi-bin/mailman/options/balug-announce>,
> <mailto:balug-announce-request@lists.balug.org?subject=unsubscribe>
> List-Archive: <https://lists.balug.org/pipermail/balug-announce/>
> List-Post: <mailto:balug-announce@lists.balug.org>
> List-Help: <mailto:balug-announce-request@lists.balug.org?subject=help>
> List-Subscribe:
> <https://lists.balug.org/cgi-bin/mailman/listinfo/balug-announce>,
> <mailto:balug-announce-request@lists.balug.org?subject=subscribe>
> From: Michael Paoli via BALUG-Announce <balug-announce(a)lists.balug.org>
> Reply-To: Michael Paoli <Michael.Paoli(a)cal.berkeley.edu>
> Content-Transfer-Encoding: 7bit
> Content-Type: text/plain; charset="us-ascii"; Format="flowed"
> Errors-To: balug-announce-bounces(a)lists.balug.org
> Sender: "BALUG-Announce" <balug-announce-bounces(a)lists.balug.org>
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
+balug-admin(a)lists.balug.org - don't send/cc/bcc to that email if
you're not on that list
Yeah, ... I noticed that too - haven't dug down yet to figure out what happened,
but some list mail went out (or went out again?) very late - like weeks late.
And like I say, haven't dug down to the bottom of it to figure out what happened
yet. "Other than that" didn't notice anything unusual. But ... maybe
some queue
got stuck or some demon that was supposed to be running wasn't, and then
perhaps with some reboot or update or the like, somehow things got unjammed
and/or reprocessed? Still guessing a bit at this point - will need to review
logs and some full headers and other details to figure out what happened.
Anyway, maybe I get to it sometime this weekend(ish) or so. Shouldn't have
happened, but there isn't an "undo" for mail sent, so 'bout best can do with it
is figure out (hopefully) how it happened, and hopefully prevent a
repeat of that.
Possible some queue got stuck or daemon down or crashed or stuck, and I didn't
catch it earlier, and things just got inordinately delayed ... that'd
at least be my first
guess, though there are certainly other possibilities.
And it wasn't just that one BALUG-Announce mail, but a moderate spattering of
mailman and/or exim4 mail that was impacted - all came through at once.
I think I saw about 20 emails or so (most of which were for list admin
or the like)
all come through at or right around the same time.
Anyway, probably answers to be found in the full headers and/or other
log data and such.
On Thu, Mar 14, 2024 at 7:58 PM aaronco36 <aaronco36(a)sdf.org> wrote:
>
> Hello Rick and Michael,
>
> > Quoting aaronco36 (aaronco36(a)sdf.org):
> >
> >> ????
> >> Now ~12:15 PDT (UTC-07:00) on Thursday 2024-03-14.
> >>
> >> aaronco36(at)SDF.org
> >
> > I'm not sure what you're talking about Aaron. The most recent
> > balug-announce posting (this one, from Michael P.:)
> > https://lists.balug.org/pipermail/balug-announce/2024-February/000340.html)
> > really did go out on Tue Feb 20 13:07:36 UTC 2024, not today.
> >
> >
>
> Hmmmm, mystery to me as well :-\
> Perhaps very delayed transmission having something to do with the top pair
> of "Received:" sections dated this morning, below ??
>
> -A
>
> Contents of the Full Header of today's message:
> ~~~~~~~~~~~~~~~~ quoting in full ~~~~~~~~~~~~~~~~~~~~~
>
> Return-Path: <balug-announce-bounces(a)lists.balug.org>
> Received: from balug-sf-lug-v2.balug.org (balug.org [96.86.170.229])
> by mx.sdf.org (8.16.1/8.14.3) with ESMTPS id 42E8VA6p012184
> (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256 bits) verified
> NO)
> for <aaronco36(a)sdf.org>; Thu, 14 Mar 2024 08:31:10 GMT
> Received: from localhost ([127.0.0.1] helo=balug.org)
> by balug-sf-lug-v2.balug.org with esmtp (Exim 4.92)
> (envelope-from <balug-announce-bounces(a)lists.balug.org>)
> id 1rkfsB-0002wb-Me; Thu, 14 Mar 2024 07:51:31 +0000
> Received: from shell1.rawbw.com ([198.144.192.42])
> by balug-sf-lug-v2.balug.org with esmtp (Exim 4.92)
> (envelope-from <Michael.Paoli(a)cal.berkeley.edu>) id 1rcPqS-0002KU-Ko
> for balug-announce(a)lists.balug.org; Tue, 20 Feb 2024 13:07:36 +0000
> Received: from webmail.rawbw.com (mail0.rawbw.com [198.144.192.41])
> by shell1.rawbw.com (8.15.1/8.15.1) with ESMTP id 41KD7ZwB068218;
> Tue, 20 Feb 2024 05:07:35 -0800 (PST)
> (envelope-from Michael.Paoli(a)cal.berkeley.edu)
> X-Authentication-Warning: shell1.rawbw.com: Host mail0.rawbw.com
> [198.144.192.41] claimed to be webmail.rawbw.com
> MIME-Version: 1.0
> Date: Tue, 20 Feb 2024 05:07:35 -0800
> To: BALUG-Announce <balug-announce(a)lists.balug.org>
> Message-ID: <94cc17d1e831608786fb59c3a4eb70a9(a)cal.berkeley.edu>
> X-Sender: Michael.Paoli(a)cal.berkeley.edu
> Received-SPF: none client-ip=198.144.192.42;
> envelope-from=Michael.Paoli(a)cal.berkeley.edu; helo=shell1.rawbw.com
> Subject: [BALUG-Announce] BALUG: meeting TODAY!: Tu 2024-02-20 & other BALUG
> News
> X-BeenThere: balug-announce(a)lists.balug.org
> X-Mailman-Version: 2.1.29
> Precedence: list
> List-Id: Announcements for and by BALUG <balug-announce.lists.balug.org>
> List-Unsubscribe:
> <https://lists.balug.org/cgi-bin/mailman/options/balug-announce>,
> <mailto:balug-announce-request@lists.balug.org?subject=unsubscribe>
> List-Archive: <https://lists.balug.org/pipermail/balug-announce/>
> List-Post: <mailto:balug-announce@lists.balug.org>
> List-Help: <mailto:balug-announce-request@lists.balug.org?subject=help>
> List-Subscribe:
> <https://lists.balug.org/cgi-bin/mailman/listinfo/balug-announce>,
> <mailto:balug-announce-request@lists.balug.org?subject=subscribe>
> From: Michael Paoli via BALUG-Announce <balug-announce(a)lists.balug.org>
> Reply-To: Michael Paoli <Michael.Paoli(a)cal.berkeley.edu>
> Content-Transfer-Encoding: 7bit
> Content-Type: text/plain; charset="us-ascii"; Format="flowed"
> Errors-To: balug-announce-bounces(a)lists.balug.org
> Sender: "BALUG-Announce" <balug-announce-bounces(a)lists.balug.org>
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
1. New Problem - Whee!
At the beginning of this month, Yahoo (or whatever they're called)
and Google tightened down the screws on requiring DMARC and encouraging
DKIM (and/or SPF).
https://blog.google/products/gmail/gmail-security-authentication-spam-prote…https://support.google.com/a/answer/81126?visit_id=638429646339446077-23847…
See links for GMail's alleged implementation, which has numerous parts.
The page _claims_ the Feb. 1 enforcement is just for "bulk senders —
those who send more than 5,000 messages to Gmail addresses in one day",
but word around the Net is that they started spamboxing mail from even
_tiny_ mail sending domains not in compliance.
2. Parts of the Problem
(Gods, I hope my MTA uses TLS for transmitting mail. It's been so long
since I looked at that, I'm not sure. Anyway, that was a hard
requirement for reaching GMail starting 2023-12-31, per above pages.)
Here, verbatim, are the requirements Gmail says it henceforth requires
of all sending domains (with some others for "bulk senders"):
o Set up SPF or DKIM email authentication for your domain.
o Ensure that sending domains or IPs have valid forward and reverse DNS
records, also referred to as PTR records. Learn more
o Use a TLS connection for transmitting email. For steps to set up TLS in
Google Workspace, visit Require a secure connection for email.
o Keep spam rates reported in Postmaster Tools below 0.10% and avoid ever
reaching a spam rate of 0.30% or higher. Learn more about spam rates.
o Format messages according to the Internet Message Format standard (RFC
5322).
o Don’t impersonate Gmail From: headers. Gmail will begin using a DMARC
quarantine enforcement policy, and impersonating Gmail From: headers
might impact your email delivery.
o If you regularly forward email, including using mailing lists or inbound
gateways, add ARC headers to outgoing email. ARC headers indicate the
message was forwarded and identify you as the forwarder. Mailing list
senders should also add a List-id: header, which specifies the mailing
list, to outgoing messages.
(I'm blearily looking up what is meant by "ARC headers", and I have
never previously had cause or interest to look at Google's Postmaster
Tools.)
3. DKIM Considered Harmful
Long rant not indulged here: DKIM is mailing list-hostile (though
mitigations are possible). I'm sitting out DKIM mail-signing, but also
my present host can't do it as equipped anyway. DMARC is a metastandard
atop either or both of SPF and DKIM. By the definition of DMARC, if you
have both (a) a valid DMARC DNS record and (b) _either_ DKIM or SPF (or
both), then you are DMARC-compliant.
4. SPF Is My Jedi Master
linuxmafia.com has had bullet-proof, simple SPF for ages. I like SPF
and consider it well-scoped, well-implemented for preventing others to
believably forge your domain for mail. It's good here, it works. Here:
:r! dig -t txt @ns1.linuxmafia.com linuxmafia.com +short
"v=spf1 ip4:96.95.217.99 -all"
Dirt-simple: It says "If mail arrives from any IP other than
96.95.217.99 purporting to be from linuxmafia.com, please treat it as
forged."
5. Let's Look at a Good DMARC RR
So, obviously I need a value DMARC record as of a week ago. A couple
of years ago, I tried one, and could swear I remember it impairing my
deliverability of mailing list mail (but can't remember details), not
to mention the avalanche of DMARC "report" mails that are the hallmark
of that botched (IMO) Yahoo anti-forgery metastandard.
But, let's look at the RR for mxtoolbox.com. The RR is always hostname
_dmarc , and is a TXT record. In mxtoolbox.com's case, that's a CNAME
pointing to mxtoolbox.com.hosted.dmarc-report.com . Here 'tis:
:r! dig -t txt mxtoolbox.com.hosted.dmarc-report.com +short
"v=DMARC1; p=reject; rua=mailto:634990a7@mxtoolbox.dmarc-report.com,mailto:634990a7@mxtoolbox.dmarc-report.net; ruf=mailto:634990a7@forensics.dmarc-report.com; fo=1; pct=100"
Breaking that down:
FQDN: mxtoolbox.com.hosted.dmarc-report.com.
TTL: 300
class: IN
RR type: TXT
TXT record breakdown (from within doublequotes):
Version: v=DMARC1;
Action to take if the mail fails authentication: p=reject
Report URIs for Aggregate data to (optional): rua=mailto:634990a7@mxtoolbox.dmarc-report.com,mailto:634990a7@mxtoolbox.dmarc-report.net;
Report URIs for Forensics/Failure data to (optional): ruf=mailto:634990a7@forensics.dmarc-report.com;
Forensic Options: fo=1;
Percentage of mail subjected to filtering: pct=100
6. Let's build a _dmarc.linuxmafia.com DMARC RR
So, in consultation with the pages mentioned below, I constructed one
(and these were my resulting notes):
Include v=DMARC1; because DMARCv1 is the only game in town. Note that
this tag must appear first, or the RR is not a valid DMARC record.
Start policy adoption with p=none; (mostly because of mailing list
forwarding).
https://dmarc.org/2017/03/can-i-use-dmarc-if-i-have-only-deployed-spf/
Include sp=none; This is the "p" policy setting, except as applied to
subdomains.
https://mxtoolbox.com/dmarc/details/dmarc-tags/dmarc-sp
Omit aspf=$THING option for now; defaults to "relaxed" SPF alignment
(though I think all mail from linuxmafia.com will originate from that
FQDN as MailFROM and Header From domains).
https://mxtoolbox.com/dmarc/details/dmarc-tags/aspf
Include fo=s; This generates an SPF failure report if the message
failed SPF evaluation, regardless of its alignment. This setting is
used to decide what to send to addresses specified in the RUF tag.
https://mxtoolbox.com/dmarc/details/dmarc-tags/dmarc-failure-reporting-opti…
Include rua=mailto:hostmaster@linuxmafia.com; This says where to send
DMARC Aggregate Feedback reports to. Be warned that this can be a lot
of... stuff.
https://mxtoolbox.com/dmarc/details/dmarc-tags/dmarc-rua
Include ruf=mailto:hostmaster@linuxmafia.com; This says where to send
failure/forensic reports, which are much more detailed than are DMARC
Aggregate reports. Only a few mail senders issue those. Prepare to
revise/lose this if I get swamped with reports.
https://mxtoolbox.com/dmarc/details/dmarc-tags/dmarc-ruf
Omit rf=$THING (report format), because there's only one format
(Authentication Failure Reporting Format = AFRF) so far, and it's
default, making this tag so far pointless.
https://mxtoolbox.com/dmarc/details/dmarc-tags/dmarc-report-format
Include ri=604800; report interval (default 86400 = 1 day), so that
reporting sites send me aggregate reports every 7 days rather than
daily.
https://mxtoolbox.com/dmarc/details/dmarc-tags/dmarc-report-interval
Include pct=100; Means apply the declared policy to mail purporting to
be mine 100% of the time. Since I'm saying "none" for now, this will
matter really only when I shift to p=quarantine; or p=reject; .
https://mxtoolbox.com/dmarc/details/dmarc-tags/dmarc-percentage
Results in:
_dmarc.linuxmafia.com 300 IN TXT "v=DMARC1; p=none; sp=none; rua=mailto:hostmaster@linuxmafia.com; ruf=mailto:hostmaster@linuxmafia.com; fo=s; ri=604800; pct=100"
7. How about Y'all? What are you guys doing about the problem?
Having felt sheepish about the lazy design/implementation of my
domain-checking scripts, I figured I'd start small, by rewriting
/etc/cron.weekly/baycondomain to be iota more sophisticated.
This was a profoundly primitive script that just reported back (via
e-mail) current parent-zone SOA serial numbers of domain baycon.org at
its pair of authoritative nameservers -- so I can vgrep and see if they
disagree, or if one or both of them don't answer. It was a
quick'n'dirty job from 2011, only now revisited.
It's now a _little_ less dumb. Output presentation is meh at best.
Also, its continued hard-coding quantity and FQDNs of the auth.
nameservers is regrettable, and could be eliminated by revising the
thing more, to generalise it. Ditto the continued hardcoded name of the
domain being checked.
Still, as the late Adam Osborne used to say, "Adequacy is sufficient."
----- begin cron script -----
#!/bin/sh
# baycondomain Cron script to sanity-check the BayCon domain's SOA records at
# all of its authoritative nameservers, as a quick and
# dirty way of making sure (1) they're all online and
# (2) they're all serving up the same data (or at least
# data with the same zonefile serial number).
#
# The script queries all nameservers for their current
# SOA value (for baycon.org), and then uses awk to parse
# out of that verbose record just the S/N field, which is
# field #3. The point is that you can visually spot offline
# or aberrant nameservers by their S/Ns being (respectively)
# missing or an out-of-step value.
#
# Written by Rick Moen (rick(a)linuxmafia.com)
# $Id: cron.weekly,v 1.02 2023/09/14 22:04:55 rick
set -o errexit #aka "set -e": exit if any line returns non-true value
set -o nounset #aka "set -u": exit upon finding an uninitialised variable
test -x /usr/bin/mail || exit 0
{
ns1soa=$(dig @NS1.BLUEHOST.COM. baycon.org. soa +short | awk {'print $3'})
ns2soa=$(dig @NS2.BLUEHOST.COM. baycon.org. soa +short | awk {'print $3'})
( [ "${ns1soa:=nonresponding}" = "${ns2soa:=nonresponding}" ] ) \
&& echo "ns1.bluehost.com and ns2.bluehost.com agree on: $ns1soa" \
|| echo "ns1.bluehost.com says $ns1soa, but ns2.bluehost.com is a rebel and says $ns2soa"
} |
/usr/bin/mail -s "Domain baycon.org SOA check" rick(a)linuxmafia.com
----- end cron script -----
----- Forwarded message from root <root(a)linuxmafia.com> -----
Date: Sun, 17 Sep 2023 06:47:01 -0700
From: root <root(a)linuxmafia.com>
To: rick(a)linuxmafia.com
Subject: Domain baycon.org SOA check
ns1.bluehost.com and ns2.bluehost.com agree on: 2023082800
----- End forwarded message -----