So ... in reviewing /var/log/exim4/rejectlog* files, I discovered mailman-loop alias to be missing 8-O ... anyway, added that 2017-09-09. I guess I somehow missed that earlier step (probably *somewhere* in the mailman setup documentation ... apparently also for Debian, Debian doesn't have an automagic way of adding email aliases - probably mostly due to the variety of MTAs Debian supports).
At least for now it's aliased to postmaster (which gets delivered locally) - probably at least "good enough" for now.
Looks like mailman-loop is mostly used for mailman administrivia emailings - e.g. password reminder mailings - and not used for, e.g. typical/general list postings.
Hmmmm... but optimal bounce processing for mailman (and with exim4 MTA)? I do recall also mailman has some option(s) that use different mechanisms on the envelope FROM ... notably so bounces - even after initial SMTP - may be better correlated to and processed for, the targeted address ... might also help with other bits of backscatter (e.g. somewhere on the Announce list, there's apparently a subscribed email address that generates an annoying backscatter ... and there isn't enough data in the received backscatter to identify the subscribed email address that's generating the backscatter). So ... there might be some more general advantages to tweaking of such sending configuration ... but I recall too it has one notable disadvantage ... since envelope FROM is adjusted per-recipient, each mail is sent as a separate SMTP transaction ... which is somewhat inefficient. Without setting such an option, bunches of same email to same SMTP server with multiple distinct recipients are sent as a single SMTP transaction. Anyway, subscriber volume isn't so huge as to make a major difference in that efficiency ... so ... may be worth at least considering? I think also, if I recall correctly, it's overall configuration, not per-list, ... so ... testing first on lower subscriber count lists would mostly be a matter of timing - at least if I recall that detail correctly.
/var/log/exim4/rejectlog* ... default retention appears to be about 10 days ... maybe I ought bump it up to something in the 15 to 65 day range. I probably ought review it more frequently, but ... if/when I don't, might be good to have some more data to look over there - lest I miss something that may be of interest/significance. Yeah, ... I'll adjust that ... # fgrep -l rejectlog /etc/logrotate.d/*exim* /etc/logrotate.d/exim4-base # ex /etc/logrotate.d/exim4-base /etc/logrotate.d/exim4-base: unmodified: line 9 :1,$p /var/log/exim4/mainlog /var/log/exim4/rejectlog { daily missingok rotate 10 compress delaycompress notifempty create 640 Debian-exim adm } :1,$t$ 9 lines added } :?rotate rotate 10 :s/10/65/p rotate 65 :?rejectlog /var/log/exim4/mainlog /var/log/exim4/rejectlog { :s/^[^ ]* //p /var/log/exim4/rejectlog { :?rejectlog /var/log/exim4/mainlog /var/log/exim4/rejectlog { :s/[^ ]*rejectlog //p /var/log/exim4/mainlog { :1,$p /var/log/exim4/mainlog { daily missingok rotate 10 compress delaycompress notifempty create 640 Debian-exim adm } /var/log/exim4/rejectlog { daily missingok rotate 65 compress delaycompress notifempty create 640 Debian-exim adm } :w /etc/logrotate.d/exim4-base: 18 lines, 244 characters :q # Okay, ... that bit is done.
Anyway, rejectlog data still generally continuing to look good. Did catch and correct the missing mailman-loop issue, otherwise generally not seeing false-positive rejects (there remains the blank body and such issues - but we've got work-arounds to mostly address that - at least until it's properly fixed "some day").
Quoting Michael Paoli (Michael.Paoli@cal.berkeley.edu):
So ... in reviewing /var/log/exim4/rejectlog* files, I discovered mailman-loop alias to be missing 8-O ... anyway, added that 2017-09-09. I guess I somehow missed that earlier step (probably *somewhere* in the mailman setup documentation ... apparently also for Debian, Debian doesn't have an automagic way of adding email aliases - probably mostly due to the variety of MTAs Debian supports).
I'm pretty sure you're mistaken. It's handled by an automatically created entry in /var/lib/mailman/data/aliases , a file that Mailman manages for you.
At least for now it's aliased to postmaster (which gets delivered locally) - probably at least "good enough" for now.
I _think_ you should let Mailman do it as per above.
Here's a post by Mailman chief coder Mark Sapiro re: mailman-loop address: https://mail.python.org/pipermail/mailman-announce/2008-July/000117.html
Looks like mailman-loop is mostly used for mailman administrivia emailings - e.g. password reminder mailings - and not used for, e.g. typical/general list postings.
It's for _bounces_ of password reminder mailings. The special sending address is used just in case those messages also bounce, to prevent formation of a bounce loop. You really shouldn't fool with it.
(e.g. somewhere on the Announce list, there's apparently a subscribed email address that generates an annoying backscatter ... and there isn't enough data in the received backscatter to identify the subscribed email address that's generating the backscatter).
VERP to the rescue. Enable it, at least long enough to identify the problem child. (Long term, might leave it off, as there are disadvantages. See first link about that.)
https://wiki.list.org/DOC/So%20what%20is%20this%20VERP%20stuff http://www.gnu.org/software/mailman/mailman-install/node27.html
Thanks! :-) That rather well anticipated two sets of questions I'd not yet gotten around to asking. Notably ...
Mailman bounce processing. So ... Mailman has per-list bounce processing - and aliases that go to pipes for those, e.g: balug-test-bounces: "|/var/lib/mailman/mail/mailman bounces balug-test" and I'm presuming those just automagically "do the right thing"... but it would also be prudent of me to reasonably check/confirm that too, at some point. Mailman also ... for various administrivia bits, etc., also uses its "default" list - normally mailman - that also has bounce processing: mailman-bounces: "|/var/lib/mailman/mail/mailman bounces mailman" but that, like the other list, may "just" be for stuff that's actually sent to that list. Then there's mailman-loop - bounce/return address of last resort (or so I seem to recall reading) - I'm presuming that's used for, e.g. stuff like password reminders. And it may get bounces there for some email addresses the others don't get ... e.g. I'm guestimating if a subscriber's options are sent to "don't mail", they get no list postings, but they do (may?) still get monthly password reminders - presuming they have that option set on ... in which case I'm guestimating mailman-loop would see bounces, but the other bounce processes, with the subscriber's option set to not mail postings - wouldn't be seeing bounces on those others. So ... *also* well covering - and appropriately, mailman-loop is fairly important. I'll have to review that further soon ... as it's probably not yet right. :-/ Also, Debian+exmi4 - by default doesn't allow aliases to go to pipes or files. Now, I did earlier discover that mailman-loop itself was bouncing 8-O (that was apparent in the exim4 logs - it was rejecting email coming in to that address) ... did (at least partially) "fix" that ... but alas, it's not (yet) being handed off to Mailman bounce processing ... whereas it ought to be. # fgrep loop /var/lib/mailman/data/aliases # The ultimate loop stopper address mailman-loop: /var/lib/mailman/data/owner-bounces.mbox # ls -l /var/lib/mailman/data/owner-bounces.mbox ls: cannot access /var/lib/mailman/data/owner-bounces.mbox: No such file or directory # ls -ld /var/lib/mailman/data/ drwxrwsr-x 2 root list 4096 Sep 22 04:07 /var/lib/mailman/data/ # So ... I do see the alias in that file, but would appear it's either not using that alias file, or it's not enabled to write to file - in either case, incoming email to mailman-loop was bouncing and present config isn't sufficient for that alias in that file to have the alias write out to that file. So, ... like I say, will need to review/fix ... and soon. Thanks for mentioning it, etc. - covered some of my questions, and I wasn't aware of the /var/lib/mailman/data/aliases file.
VERP :-) ... yeah, I'd forgotten what it was called, but recalled reading about it in Mailman configuration stuff. And yes, advantages/disadvantages ... but can be quite good for better processing of bounces and backscatter. So, yeah, may be better on balance, to enable it, than not ... or maybe at least occasionally/periodically enable it. Ah, but should probably first get mailman-loop working as it ought.
From: "Rick Moen" rick@linuxmafia.com Subject: Re: [BALUG-Admin] mailman-loop ... Date: Thu, 21 Sep 2017 19:15:37 -0700
Quoting Michael Paoli (Michael.Paoli@cal.berkeley.edu):
So ... in reviewing /var/log/exim4/rejectlog* files, I discovered mailman-loop alias to be missing 8-O ... anyway, added that 2017-09-09. I guess I somehow missed that earlier step (probably *somewhere* in the mailman setup documentation ... apparently also for Debian, Debian doesn't have an automagic way of adding email aliases - probably mostly due to the variety of MTAs Debian supports).
I'm pretty sure you're mistaken. It's handled by an automatically created entry in /var/lib/mailman/data/aliases , a file that Mailman manages for you.
At least for now it's aliased to postmaster (which gets delivered locally) - probably at least "good enough" for now.
I _think_ you should let Mailman do it as per above.
Here's a post by Mailman chief coder Mark Sapiro re: mailman-loop address: https://mail.python.org/pipermail/mailman-announce/2008-July/000117.html
Looks like mailman-loop is mostly used for mailman administrivia emailings - e.g. password reminder mailings - and not used for, e.g. typical/general list postings.
It's for _bounces_ of password reminder mailings. The special sending address is used just in case those messages also bounce, to prevent formation of a bounce loop. You really shouldn't fool with it.
(e.g. somewhere on the Announce list, there's apparently a subscribed email address that generates an annoying backscatter ... and there isn't enough data in the received backscatter to identify the subscribed email address that's generating the backscatter).
VERP to the rescue. Enable it, at least long enough to identify the problem child. (Long term, might leave it off, as there are disadvantages. See first link about that.)
https://wiki.list.org/DOC/So%20what%20is%20this%20VERP%20stuff http://www.gnu.org/software/mailman/mailman-install/node27.html
Quoting Michael Paoli (Michael.Paoli@cal.berkeley.edu):
Mailman bounce processing. So ... Mailman has per-list bounce processing - and aliases that go to pipes for those, e.g: balug-test-bounces: "|/var/lib/mailman/mail/mailman bounces balug-test" and I'm presuming those just automagically "do the right thing"...
[...]
soon ... as it's probably not yet right. :-/ Also, Debian+exmi4 - by default doesn't allow aliases to go to pipes or files.
To pipes, yes. I don't know what you mean by not to files, in that destination mbox files are files, and /etc/aliases entries to those definitely work. The disabling of processing of pipes in /etc/aliases is/was a security improvement, as there have been just far too many security meltdowns causes by exploits of shell processing in exactly that area.
In recent mail, I'd assumed that, nonetheless, Exim4 _does_ correctly process the pipes used in /var/lib/mailman/data/aliases that the /var/lib/mailman/bin/newlist command adds per-list stanzas to -- but I'm just now remembering that /var/lib/mailman/data/aliases is now vestigial if you add the recommended Mailman plumbing to Exim4 transports and routers. (Disclaimer: This reflects not careful study but rather yet another super-quick look around.)
My surmise today is that the Exim4 additions to routers and transports bypasses the traditional aliases mechanism entirely and makes Exim4 parse the MAILMAN_HOME/lists/$local_part/config.pck (Python 'pickled' format) files for per-list data, instead, probably using the $MAILMAN_HOME/mail/mailman tools, if my overly-quick parsing of Exim4's conffiles is correct.
In which case, sorry about the digression onto a vestigial file. That's the problem with having run this stuff since dinosaur days: You keep wanting to maintain stuff that's actually not used any more, but in many cases is confusingly still around anyway.
(FWIW, I'd also noticed that /var/lib/mailman/data/owner-bounces.mbox is non-existent on my system, but had overhastily written that fact off as merely indicating that it was getting cleaned out or something.)
So, ... like I say, will need to review/fix ... and soon. Thanks for mentioning it, etc. - covered some of my questions, and I wasn't aware of the /var/lib/mailman/data/aliases file.
Probably vestigial, on second thought. My current guess.
Moving on...
Given your empirical observation that inbound mail to mailman-loop was bouncing, that raises the subsidiary question of how it _ought_ to be directed. Sending it to root/postmaster via /etc/aliases of course ensures that it goes somewhere, but is it where it _should_ go? I'm really not sure. Maybe you should ask the Mailman and/or Exim experts.
VERP :-) ... yeah, I'd forgotten what it was called, but recalled reading about it in Mailman configuration stuff. And yes, advantages/disadvantages ... but can be quite good for better processing of bounces and backscatter. So, yeah, may be better on balance, to enable it, than not ... or maybe at least occasionally/periodically enable it. Ah, but should probably first get mailman-loop working as it ought.
VERP will let you find the problem child. Enable it for that, and you can disable it after you track that subscriber down.
I wrote:
Given your empirical observation that inbound mail to mailman-loop was bouncing, that raises the subsidiary question of how it _ought_ to be directed. Sending it to root/postmaster via /etc/aliases of course ensures that it goes somewhere, but is it where it _should_ go? I'm really not sure. Maybe you should ask the Mailman and/or Exim experts.
Recommendation here is an /etc/aliases entry to an mbox file... https://mail.python.org/pipermail/mailman-users/2010-May/069502.html ... _but_ poster seconds your comments that Debian's default Exim configuration doesn't allow delivery to files in /etc/aliases -- which I guess on reflection is true. I don't make entries deliver to _files_, but rather to other users, which is to say, processing the mail back through the mail chain and its LDA (procmail).
So, commenter Roger Lynn says that the real solution is to add the correct Exim4 router + transport to /var/lib/mailman/bin/postfix-to-mailman.py for handling mail-loop, and he opened a bug requesting someone do that.
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=562700
Except -- haha! -- in the end Roger Lynn said 'Screw it' and just did mail-loop: root ...in /etc/aliases.