I wrote:

...

Send mail to poster when their posting is held for approval? (Edit respond_to_post_requests) ... debatable....

Again, glad to discuss my reasoning. (It's late, and I've had too little sleep, or I'd say more here. But I've pretty firmly come down on the 'no' side after much pondering.)

As with many issues in mailing list administation, this issue partly involves the spam issue, and partly involves user psychology. I'm sure you're acutely aware of what a blight on the Internet backscatter spam is. https://en.wikipedia.org/wiki/Backscatter_(email) This is an issue in a _huge_ variety of traditional SMTP infrastructure that is guilty of sending back autoresponses to claimed SMTP senders that are more often than not forged, making the infrastructure complicit in the spam problem. E.g., any unpatched instance of DJB's qmail 1.03 (latest) is a horrifically bad netizen because Dan's qmail-smtpd process _first_ accepts incoming mail and only then decides whether to further process it for local users / MDAs / LDAs or whether to generate a non-delivery report (NDR) and outbound-deliver it to the (usually forged) claimed sender. And -- my point -- Mailman sending 'Post by non-member to a members-only list' held-message notices will in a large percentage of cases be textbook backscatter spam. This will be true unless and until your accepting MTA is close to perfect in spam-rejection in its receiving stage. I hate spam, which means I also hate being guilty of backscatter spam. Which means 'Send mail to poster when their posting is held for approval' is a terrible idea, and the correct setting is 'no'. Separately from that, as I said, is the user psychology bit. But first I'll talk about the big picture, on my views concerning ethical mailing list administrative practices. My view is that ethics requires the listadmin to be responsive to _legitimate_ subscribers and aspiring subscribers (but not, say, to spammers), and to be transparent within certain limits and expectations. So, for example, a posting from a legitmate member should never just be discarded, because that is a failure of responsiveness to the subscriber. Just 'disappearing' an attempted post, e.g., by letting it expire out of queue or hitting the Discard button is, IMO, scummy behaviour. If it will not be approved, it should be explicitly rejected with a nice, concise notice about why. Given that one is doing that, and doing it in a timely manner, is there a functional advantage to subscribers or the listadmin to also autosending an instant 'Your message has been held for moderator attention' notice? Not in my view, though I understand that some subscribers want maximal information instantly. Against that, there are the disadvantages: Sometimes, with luck very rarely, you end up having good reason to suspect a current subscriber of inclination to commit misbehaviour in front of the mailing list as a large audience. So, you might want to quietly set that member's Moderated flag for a day or two, just so you can review what that particular hothead is saying before it goes out. If you're on the ball, the resulting admin-queue delay is so tiny that the subscriber probably won't even notice. Probably, after the first or second post without misbehaviour, you'll hit the checkbox on the Approve dialogue to clear the subscriber's Moderated flag again -- because your confidence has been restored and because a smart listadmin is a lazy listadmin. (You really don't _want_ to vet anyone's posts.) Doing all of the above with the 'Send mail to poster when their posting is held for approval' feature off means it's all transparent to the subscriber and doesn't insult anyone or get anyone's back up. There are probably other examples of this usefulness to _not_ having notices go out every time something lands in the admin queue, but I'm not remembering them at the moment. The larger point is; It's useful, to a degree that IMO massively outweighs some subscribers' desire to know instantly everything that's going on (especially if the listadmin acts on needed tasks in a timely fashion). As an aside, I also strongly favour _not_ manually approving messages arriving from non-subscribed addresses. Instead, reject with a nice explanation that, in 2017, you have to post to mailing lists from your subscription address, but that people with multiple addresses they might wish to use for posting can still do so by subscribing _all_ their addresses and setting all but one's subscriptions to 'nomail'. And also, you should almost never use the 'List of non-member addresses whose postings should be automatically accepted' feature (Privacy Options, Sender Filters), or its equivalent 'Add [$ADDRESS] to one of these sender filters: ' checkbox on the mail queue page. Why? Because any address in that roster gets prospectively exempted from _all_ posting rules, e.g., maximum message size for that mailing list, and unconditionally permitted to post. (The roster is thus a lurking menace to listadmins new to Mailman.)

Quoting Michael Paoli (Michael.Paoli@cal.berkeley.edu):

mailman list - I don't fully grok why, but ... with the Debian install of mailman, it instructs one to create a list named mailman, which I dutifully did so. I'm guestimating removing such a list may be a *bad idea* - so I've no intention of doing so unless I knew quite certainly there was no specific downside on that (including even any specific ways Debian might happen to use such list). However, seems a bit odd - and potentially problematic(/annoying) to have it shown among the "advertised" lists, so, I changed the setting on that list to: Advertise this list when people ask what lists are on this machine? No That prevents it from showing in: https://temp.balug.org/cgi-bin/mailman/listinfo I also changed setting to: What steps are required for subscription? Confirm and approve Thinkin' folks really ought not be subscribed to that list, and adding approval requirement may help prevent unintended subscriptions to that list.

I actually was meaning to talk to you about this matter. You'll notice http://linuxmafia.com/mailman/listinfo and http://lists.svlug.org/lists/listinfo both have mailing list 'Mailman' set as 'Advertise this list when people ask what lists are on this machine?' = no. I don't think there's actually any functionality advantage to requiring listadmin approval, because nobody can actually do anything as a member of that list. The downside of your setting 'Confirm and approve', there, is that _if_ some numbnuts attempts to subscribe, that request will land in the admin queue, and you and I will continue to be pestered about whether to approve it or not. Unlike held _mail_, held subscribe requests never expire out of queue: There is no 'discard held subscribe requests after N days' configuration item. So, I really do think setting 'Confirm and approve' on mailing list Mailman is an actively bad idea, and think you should revert that. You're certainly right that any _advertised_ Mailman list will get subscriptions, no matter how nonsensical that idea is for a particular mailing list. Consider, for example http://lists.svlug.org/lists/listinfo/speakers . As it says on the listinfo page, that is an explicitly read-only archive of a _formerly_ active mailing list, so it's flat crazy to try to subscribe to it. Some background: There was a period around 2005 when J. Paul Reed was SVLUG President and Bill Ward was VP. They'd gotten the catastrophically bad advice that they should distrust all of the active volunteers, which is pretty much the way they conducted themselves after election. One day (March 28, 2005), I was astonished to find that the mailing lists Web-Team, Speakers, and one other whose name eludes me (I think it was lsec, a mailing list about Linux security), all of the working mailing lists of the volunteer teams, had been summarily deleted without discussion including (apparently) all back-postings archives going back five years. In place of the three team working mailing lists, a new one, privately archived and requiring permission of Reed or Ward to join, got created called 'Volunteers'. Inquiring with them about what was going on revealed that they wanted to 'put everyone working on anything SVLUG-related together' (Reed's phrase), so they decided to blow away all of the existing forums and create the new hyper-controlled private one to take their place. As head of the Web Team, I asked to please get a copy of the Web-team mailing list archive, as it had material I needed. It quickly became apparent that Ward had no idea how to retrieve it: He had no knowledge of Mailman, but had just blowed away the mailing list configurations using the admin WebUI's delete control. I figured out how to get the cumulative archive, on my own (and for some years hosted it publicly on linuxmafia.com's Mailman instance). During Reed and Ward's administration, nobody but them had the right to do essentially any work on the mailing list server. Shortly after they left office, I converted Volunteers to a public mailing list open to all subscribers, and resurrected Web-team and Speakers (undoing most of the damage Ward had done). Speakers seemed unlikely to have future traffic because there was no longer a Speaker Coordinator, but some of its content seemed useful, so I edited http://lists.svlug.org/lists/listinfo/speakers so it no longer had a Web form to join the list. (I _do_ have that vestigial mailing list with 'What steps are required for subscription?' set to 'Require approval', just in case some numbnuts tried to join it via the e-mail command interface. Nobody has tried.)

I'm guestimating primary reason such list is created, is to have standard localpart addresses that can be used for mailman requests,

http://www.gnu.org/software/mailman/mailman-install/site-list.html says: [The site-wide mailing list, by default named 'Mailman'] is the one that password reminders will appear to come from, and it is required for proper Mailman operation. Anyway, every GNU Mailman installation has one. (To repeat, I concur about making it unadvertised.)

blank body (with command(s) in Subject: header) This one isn't so trivial to "fix" for mailman, while leaving the appropriate bits in place for the generally desired anti-spam.

If I read correctly what you say further on, it's actually some set of overly paranoid Eximconfig rules that are preventing this mail, not Mailman. I might be able to help with this later, if you like, but I suspect your detective skills on this one are at least as good as mine if not better. Speaking of Eximconfig, here's mail I sent to the author, detailing a fix that I'm pretty sure you will also want to make: Date: Wed, 30 Nov 2005 22:55:15 -0800

From rick@linuxmafia.com Wed Nov 30 22: 5:15 2005 From: Rick Moen <rick@linuxmafia.com> To: Jonathan Boggis Subject: Re: Mailing list posts, and EximConfig

Hi, Jonathan. Responding to your mail from last February: Quoting Jonathan Boggis:

Thanks for your comments.

The problem is with SPF and forgery checks being performed on the header From: address. These help reject the forgeries generated by spammers and viruses, but have the downside of potentially blocking messages sent via mailing lists.

The easiest way to address this in the short term is simply to whitelist the mailing list's domain in accept/sender_domain (I.e: Add lists.svlug.org to this.) This will prevent the SPF and forgery checks on mail sent by this mailing list. However, it will also disable spam checks, so if spam is sent to the mailing list, it won't be trapped by SA.

In the long term, SPF and forgery checks probably need to be a little bit more configurable, i.e: So it's possible to specify whether to do SPF and forgery checks on the envelope sender, header From: and SPF on return path (Not presently done in EximConfig, but should be.)

I found that the best remedy was to disable "spf_from_acl" in /etc/exim4/eximconfig/config/spf.conf . This bit: #spf_from_acl: # # # Check header From: # warn set acl_m8 = ${address:$h_From:} # deny !acl = spf_check # warn message = Received-SPF-From: $acl_m8 ($acl_m7) # accept I've left the envelope-sender check, etc., intact: spf_rcpt_acl: # Check envelope sender warn set acl_m8 = $sender_address deny !acl = spf_check warn message = Received-SPF: $acl_m8 ($acl_m7) accept spf_check: warn set acl_m2 = ${readsocket{/tmp/spfd}\ {ip=$sender_host_address\n\ helo=${if def:sender_helo_name\ {$sender_helo_name}{NOHELO}}\ \nsender=$acl_m8\n\n}{20s}{\n}{socket failure}} # Defer on socket error defer condition = ${if eq{$acl_m2}{socket failure}{yes}{no}} message = Cannot connect to spfd # Prepare answer and get results warn set acl_m2 = ${sg{$acl_m2}{\N=(.*)\n\N}{=\"\$1\" }} set acl_m8 = ${extract{result}{$acl_m2}{$value}{unknown}} set acl_m7 = ${extract{header_comment}{$acl_m2}{$value}{}} # Check for fail deny condition = ${if eq{$acl_m8}{fail}{yes}{no}} message = ${extract{smtp_comment}{$acl_m2}{$value}{}} log_message = Not authorized by SPF accept Unless I'm missing something, that really _is_ the only logical solution, since SPF by its creator's definition and intent _is_ intended to validate "From " and _not_ "From:". I really would urge that you consider disabling "spf_from_acl" in post-2.2 versions. -- Cheers, "Due to circumstances beyond our control, we regret to Rick Moen inform you that circumstances are beyond our control." rick@linuxmafia.com --Paul Benoit

Quoting Michael Paoli (Michael.Paoli@cal.berkeley.edu):

I (over?-)tweaked the mailman list page - it's got just about nothing on it except bit to unsubscribe or edit options. Could alternatively put it mostly to default, and just strip out the bits about subscribe and post ... but practically, don't know if it makes all that much difference. Folks(/bots) could still subscribe via email.

It actually doesn't make any difference, especially if you make it unadvertised.

Also, the "not advertised" - seems to work or mostly work. The only (maybe?) exception I see is if mail is sent to the mailman list with the command list - it does include itself in the list - but if one's emailing it anyway, maybe kind'a pointless for it to not show itself?

To make sure we're clear on what's being discussed, here, an unadvertised list is one not displayed on the sitewide listinfo page. E.g., linuxmafia.com's sitewise listinfo page is: http://linuxmafia.com/mailman/listinfo That page lists eight mailing lists but not these two: http://linuxmafia.com/mailman/listinfo/Mailman (and one other that is non-public) I just e-mailed a test query of the command 'lists' to mailman-request@linuxmafia.com: It sent back a roster of the public mailing lists plus mailing list Mailman (which it included even though I set its unadvertised flag). The non-public other list was omitted. It's possible that this behaviour was changed in a later Mailman release. For better or worse, hardly anyone uses the e-mail command interface; it's quite obscure compared to the Web one.

I think that's "quite close enough" for now, and "good enough" to go forward - with (pending) those adjustments to the text that gets sent.

Looks good.

So, bit of searching (find(1)) and such ... looks like those relevant texts are configurable under: /etc/mailman/en/ - at least for list installed/configured to only use English (I'm not planning to use nor translate nor translate changes to other languages). And looks like they start out, at least initially (and thus far on this host/installation) with same set of files in each (at least same names and contents - looks like mtimes probably also all match). So ... likely just need to suitably edit the applicable ones under /etc/mailman/en/ and probably give mailman a restart, do a bit more testing to confirm, etc., and that should cover that. That should then more than suffice for starting to announce/publicize the "Test" list. Next up is then looking at subscriber options - looks like there are two python database formats mailman uses ... may be able to manipulate user options through that (especially in bluk) to match (transferring) per-user member options. Don't have access to those database files on DreamHost.com, but do have access to the data via admin web pages ... so - may be sufficiently feasible to web scrape those out'a there. Saving approximately 897 list members the annoyance of their options not being preserved ... that's worth doing if it's reasonably feasible. E.g., theoretically, presuming well tested first, might be something like: stop exim4 change old location list to effectively read-only (emergency moderation flag?) web scrape members & options bulk subscribe transferees stop mailman edit member options in database(s) start exim4 start mailman

From: "Michael Paoli" <Michael.Paoli@cal.berkeley.edu> Subject: Re: [BALUG-Admin] BALUG lists ... mailman list(?!), blank body with command in Subject: header Date: Sun, 16 Jul 2017 07:32:45 -0700

Anyway, pretty darn close to officially "announcing" the BALUG-Test list, as it seems to be working mostly pretty dang well (and better than DreamHost.com!). I think I just need tweak some of the mailman text - at least for now - so folks don't - at least based upon the text sent from this mailman installation - expect that command in Subject: header with blank body will work (heck, they can make it as most simple work-around by including "end" in the body). I think that's "quite close enough" for now, and "good enough" to go forward - with (pending) those adjustments to the text that gets sent.

current set of stats from /var/log/exim4/rejectlog* files Mostly looking quite good, covering period: 2017-08-04T06:41:39-0700--2017-08-15T06:15:48-0700 2017-08-15T06:15:48-0700 2017-08-04T06:41:39-0700 10 23:34:09 $ echo '1870/(10*24+23+34/60+09/3600)' | bc -l 7.09491183528675268667 So ... rejecting(+delaying) about 7 spams per hour. And most all legitimate rejections, and thus far not seeing any general mass spam to have made it through ... at least yet. total: 1883 rejected (or greylisted 20/1883) sub-totals: 1870 legitimately rejected (or delayed with greylisting) 13 false positive - but only 1 original, 12 more to diagnose issue summary analysis: 902 auth_server_login authenticator failed 733 HELO *.* (invalid) 99 SMTP protocol synchronization error 66 too many nonmail commands 20 greylisted 14 Too many connections within a 5 minute period (Maximum of 40 allowed.) 12 URL ending in .pm (false positives and work to isolate and correct) 12 Sender ... @temp.balug.org attempting to use forged local or relay 12 Bounced undeliverable notification rejected 9 Relaying from 2 Scam text in body 1 no valid sender in any header line (bad MTA or configuration thereof) 1 Too many consecutive failed deliveries from sender (I DoSed myself)

From: "Michael Paoli" <Michael.Paoli@cal.berkeley.edu> To: balug-admin@lists.balug.org Subject: Re: [BALUG-Admin] BALUG lists ... work-around text: blank body with command in Subject: header, & anti-spam Date: Wed, 19 Jul 2017 23:44:21 -0700

And, anti-spam looking quite good. Quite few false-positives (at least after some earlier configuration adjustments). Whole lot 'o crud spam being rejected ... Looking at logs, 5,384 rejections (but note that some of those are soft-fails, e.g. greylisting) in 10d 10h 34m 3s, that gives us: about 21.5 rejections/hr., or about a rejection every 2.8 minutes. Don't think I've seen a single actual spam email make it through ... yet.

So, thus far, pretty dang good results on the anti-spam.

5384 2017-07-19 23:13:57 2017-07-09 12:39:54 10 10:34:03 5384/(24*3600*10+3600*10+34*60+3)*3600 21.48722400151655740800 1/(5384/(24*3600*10+3600*10+34*60+3)*60) 2.79235698365527488942

Oh, forgot to mention why you'd want those config.db dump files. Reportedly because they are in a guaranteed regular and parseable format for moving config data between Mailman instances even if there's a version skew between them. More at: https://serverfault.com/questions/194450/how-to-transfer-config-of-one-mailm...

Quoting Michael Paoli (Michael.Paoli@cal.berkeley.edu):

Yes, well, from what I see of the files and such, looks like originally the files in /etc/mailman/en/ match those in /usr/share/mailman/en/ And looks like the bits editable in web GUI default to (after a restart of mailman) using those in /usr/share/mailman/en/ [...]

You know, I really should have included a disclaimer of 'I deal with these file locations so seldom that I really cannot consistently remember them, and please reconfirm anything I say about what files to edit for what purpos.' It doesn't help that Mailman uses symplinks for a number of things criss-crossing between /var/lib/mailman locations and /usr/share/mailman (I think) ones.

And per FHS, etc., the above typically wouldn't be editable (/usr filesystem may be mounted read-only....

Well, it's hardly the only offender. I discovered this when I did the research that produced http://linuxmafia.com/faq/Admin/linuxmafia.com-backup.html , and [re-]discovered that Apache httpd's cgi-bin directory is in /usr/lib/cgi-bin/ I likewise keep /usr normally mounted ro, and likewise use the apt hooks to autoremount it for package operations -- but live with the fact that I need to also manually remount /usr to fiddle with cgi-bin scripts. SpamAssassin's rulesets and scoring settings in /usr/share/spamassassin are another example (and I guess I should add that directory to my backup list). I'll admit to getting a bit disenchanted with many of the more-recent additions to FHS (specifically /media and /srv). I've read the rationales, and I don't agree that /media is justified in addition to /mnt. I am now grudgingly reconsidering my reflexive opposition to /srv, because there's a good case that CGI-BIN scripts should be there rather than anywhere in /usr, including the '/usr may be ro' one. Therefore, SpamAssassin rulesets and scoring settings arguably are better there, too. Any time someone says 'Oh, let's solve this problem by putting it in a new directory off /', I get cranky and conservative-minded, though. Which is why I think /sys is regrettable -- even though I've heard how we haplessly fell into the need for that on Linux systems. (Essentially, the kernel people made such a dog's breakfast out of /dev that they had to make a new tree to start over. Nice going, guys.) I actually wanted to consult you on /boot being a separate filesystem. I've done that since forever, but am acutely aware that the original rationale no longer applies, that of old BIOSes being unable to branch from int 13h to disc cylinder locations 1024 and over. So, is there a 2017 reason for still having /boot separate? I can think of scenarios, like you have most of the system on md mirrored partitions but want your bootloader to be able to run from a pair of ext2 ones on both drives, _or_ you have most of the system encrypted. Are there reasons I'm missing?

I suppose too, I ought set up some user in group (or with access to group) list, so that many of the list related commands can be run fine with less-than-root privileges - a not otherwise privileged user but also in group list would seem optimal - as group list seems to have the permissions to do the needed, but would generally lack permissions to muck with stuff it ought not (e.g. files owned by list by not writable by group list). At least on Debian, of what I've seen so far, looks like all the normally needed actions to manage mailman lists, are controlled/accessed via the list group.

Makes sense. FWIW, when I need to muck about with Mailman's files I _generally_ just su to list. Most particularly, it's really important to be user 'list' if you regenerate a pipermail archive using /var/lib/mailman/bin/arch . Otherwise, if you make the mistake of doing it as root, the ownership of the generated Web archive is wrong and you'll get 403 errors until you figure that out and fix it. One ends up running /var/lib/mailman/bin/arch in one of the following situations: 1. Something needs to be redacted, either spam or (say) someone's private telephone number. If you must do this, you start by editing cumulative mbox file /var/lib/mailman/archives/private/example.mbox/example.mbox (say, using mutt). Do this as user 'list'. Ideally, you will not delete an entire message except one at the very end of the mbox, because doing so causes the pipermail Web archive entries to get renumbered URLs. All other things being equal, we want content's URLs to remain constant so people can permalink it. Once you have edited the mbox, you do this and cross your fingers: cd /var/lib/mailman/ bin/arch --wipe -q example archives/private/example.mbox/example.mbox A really vexing problem sometimes arises: /var/lib/mailman/bin/arch evinces broken behaviour if it encounters the string 'From ' flush-left in a message it's parsing. It mistakenly thinks that's an envelope 'From ' delimiting the beginning of a new message. The fix is to, yes, go back to the mbox file, find the damned line, prefix a '>' to that flush-left word (and do likewise with any other such cases), then re-run the arch command, to try again. The syptom of the problem is: You look in the first and last months of the Web archive using your Web browser, and any fragmentary messages will stand out. 2. Combining two mbox files. This is really exactly the same as case 1, except here you have to give up on keeping archive URLs of individual messages constant. For example, a couple of years ago, I re-found the primordial mbox file for the main SVLUG mailing list, in the days it ran on majordomo before there was such a thing as GNU Mailman. I experimentally cat'ed that to the existing cumulative Mailman mbox (keeping backup copies), re-ran arch, dealt with the irksome 'From ' problem a few times, and reached success -- whereby SVLUG now has a Mailman archive older than Mailman is (I think).

Ooooh, yes! I'd be highly interested in such CLI tools/programs. Any particular pointers on what/where to look for such? (Debian packages or names or name pattern, or term to search on?) That will likely come in highly handy pretty darn soon. I also notice Mailman comes with some stuff to dump such, haven't yet checked/seen/noticed if it has something to edit or quite selectively load such (e.g. changes some of a user's options, but leave others untouched - e.g. won't have password to reload). I've got nice web scraping bit (had a wee bit more time this A.M.), so now I can grab *all* the member data *except* for their password (and the relatively trivial bit that Mailman also saves their email as subscriber enters/submits it - which might contain uppercase letters - but Mailman canonicalizes it to lowercase and mostly or entirely uses that - though it does also save the original form). At least I think that's *all* the data - I'll double-check that there's nothing else Mailman saves beyond that (at least of relevance/significance) that's not obtainable from the individual subscriber settings web pages. That then also frees us up from trying to get that data from DreamHost.com (and them possibly mucking it up or saying "no" anyway). Also, the data suck from the web pages is quite "fast enough" - completes typically in under 7 minutes - probably mostly limited by bandwidth and the number and size of pages. (800++ members total). I've a bit more massaging of the data to do, so it will better align to format to be handed off to go into format Mailman wants for the Python pickle (.pck) - and that would presumably be via some handy CLI tools/programs (assuming I can find and use such) - or "worse case" I might have to learn to write some Python and Python pickle code to handle it. In any case, getting closer to being able to transfer almost all that user preference data.

From: "Rick Moen" <rick@linuxmafia.com> Subject: Re: [BALUG-Admin] BALUG lists ... blank body with command in Subject: header, ... subscriber options Date: Wed, 19 Jul 2017 23:55:03 -0700

There are a lot of really nice command-line tools available for site admins to use in dumping and editing config.db and the various pck databases, not limited to those shipped with Mailman. But of course you don't have site admin at Dreamhost. So, yeah, the most you're going to get out of Dreamhost would be for you (or maybe Michael Hubbard) to file a trouble ticket asking for the files. Here's a page about how to do migration optimally:

https://debian-administration.org/article/567/Migrating_mailman_lists

Quoting Michael Paoli (Michael.Paoli@cal.berkeley.edu):

And turns out I can also access and save the case-preserved email bits. Turns out it only explicitly shows that on the pages of members that entered their email address using one or more uppercase characters within.

Yeah, it's true that Mailman's regular display of a mailing list's roster wrongly lower-cases everything in subscribers' e-mail addresses, and that's objectively wrong. I've filed a bug. https://bugs.launchpad.net/mailman/+bug/1707447 (Mailman 3.x mainenance continues at the main project, but 2.1.x has been handed off to Canonical.) Once migration off Dreamhost is complete and you have _site_ administration abilities (use of the $MAILMAN_HOME/bin/* utilities), thankfully you'll be free of these problems. "/var/lib/mailman/bin/list_members -f <listname>' reports everything correctly. Having only listadmin access (as with us at Dreamhost) is a serious handicap.

(And other random bits, like email instead of e-mail, using . to separate portions of phone number rather than -).

You won't be confiscating my hyphens and parentheses from me, any time soon. ;-> -- Rick Moen e-mail: rick@linuxmafia.com cellular: (650) 283-7902 FWIW, 'email' is a loanword from French meaning 'enamaled'. The hyphen stresses a semantic distinction, not to mention the pronunciation one of EE-mayul as opposed to the loanword's pronunciation eh-MAYH. And yes, most English speakers have never heard of that loanword, and it's fair to call it an archaic term, at this point. You're probably one of the lucky 10,000 to hear about it today. https://xkcd.com/1053/

Quoting Michael Paoli (Michael.Paoli@cal.berkeley.edu):

Ooooh, yes! I'd be highly interested in such CLI tools/programs. Any particular pointers on what/where to look for such? (Debian packages or names or name pattern, or term to search on?)

Well, I don't remember, exactly. Your better source would be asking Marc Merlin <marc@merlins.org> . But, that having been said, I _used_ to know something about that, and grabbed a bunch of those that Marc collected on the SVLUG mailing list server. Thus, they are here on linuxmafia.com as part of my safety backup, ergo accessible to you: /usr/local/src/rickstuff/svlug/var-local-scr/scr $ ls -al total 244 drwxr-sr-x 2 root root 4096 Jul 31 2004 . drwxr-sr-x 3 root rick 4096 May 28 2015 .. -rwxr-xr-x 1 root root 3152 Aug 25 2000 addsize -rwxr-xr-x 1 root root 165 Feb 20 2002 base64dec -rwxr-xr-x 1 root root 165 Feb 20 2002 base64enc -rwxr-xr-x 1 root root 1406 Jul 27 1998 bz2grep -rwxr-xr-x 1 root root 301 Jul 27 1998 bzcat -rwxr-xr-x 1 root root 1399 Jul 27 1998 bzgrep -rwxr-xr-x 1 root root 737 Oct 20 1998 catdb -rwxr-xr-x 1 root root 2814 May 23 1998 cdda2mp3 -rwxr-xr-x 1 root root 559 Sep 24 2001 checkaliases -rwxr-xr-x 1 root root 201 May 23 1998 clean -rwxr-xr-x 1 root root 1085 Jul 28 2000 cleanfile -rwxr-xr-x 1 root root 364 Jan 8 1999 cleanss -rwxr-xr-x 1 root root 605 Jul 27 1998 color_test -rwxr-xr-x 1 root root 332 Jul 27 1998 color_test2 -rwxr-xr-x 1 root root 211 Sep 27 1999 copyaudiocd -rwxr-xr-x 1 root root 2594 Nov 12 2001 copysafe -rwxr-xr-x 1 root root 230 May 23 1998 count -rwxr-xr-x 1 root root 541 Feb 14 2000 disp -rwxr-xr-x 1 root root 206 May 23 1998 dup2 -r-xr-xr-x 1 root root 884 Jul 31 2004 evim -rwxr-xr-x 1 root root 37200 Jun 6 2001 expn -rwxr-xr-x 1 root root 926 May 23 1998 findcust -rwxr-xr-x 1 root root 889 May 23 1998 findcustc -rwxr-xr-x 1 root root 1010 May 23 1998 findman -rwxr-xr-x 1 root root 1148 Oct 18 1998 findstr -rwxr-xr-x 1 root root 434 Jun 1 1999 gendnstables -rwxr-xr-x 1 root root 1013 Jul 27 1998 kall -rwxr-xr-x 1 root root 1063 Mar 14 2000 killanims -rwxr-xr-x 1 root root 37 Jan 24 2000 listmp3tags -rwxr-xr-x 1 root root 925 May 23 1998 lock -rwxr-xr-x 1 root root 843 May 23 1998 mem -rwxr-xr-x 1 root root 2432 May 23 1998 multidecode -rwxr-xr-x 1 root root 174 Jul 27 1998 playau -rwxr-xr-x 1 root root 59 Jul 4 1999 qpdecode -rwxr-xr-x 1 root root 99 Jul 4 1999 qpencode -rwxr-xr-x 1 root root 389 Jun 23 2000 recompress -rwxr-xr-x 1 root root 1017 Apr 13 1999 redocheckconfig -rwxr-xr-x 1 root root 588 May 23 1998 ren -rwxr-xr-x 1 root root 341 Jun 13 1999 rename -rwxr-xr-x 1 root root 3252 Aug 7 2001 rescanscsi -rwxr-xr-x 1 root root 687 Feb 13 1999 reset -rwxr-xr-x 1 root root 371 Jul 27 1998 ro -rwxr-xr-x 1 root root 179 Jul 27 1998 rot13 -rwxr-xr-x 1 root root 372 Jul 27 1998 rw -rwxr-xr-x 1 root root 478 Jul 28 2000 savephoto -rwxr-xr-x 1 root root 1627 Jul 28 2000 setmp3tag -rwxr-xr-x 1 root root 5356 Jan 21 2000 ssh-ppp -rwxr-xr-x 1 root root 446 May 30 2000 symbolsearch -rwxr-xr-x 1 root root 352 May 23 1998 termreset [rick@linuxmafia] /usr/local/src/rickstuff/svlug/var-local-scr/scr In fact, tell you what: [rick@linuxmafia] /usr/local/src/rickstuff/svlug/var-local-scr/scr $ cd .. [rick@linuxmafia] /usr/local/src/rickstuff/svlug/var-local-scr $ tar czf /tmp/scr.tar.gz scr [rick@linuxmafia] /usr/local/src/rickstuff/svlug/var-local-scr $ Attached.

I've got nice web scraping bit (had a wee bit more time this A.M.), so now I can grab *all* the member data *except* for their password (and the relatively trivial bit that Mailman also saves their email as subscriber enters/submits it - which might contain uppercase letters - but Mailman canonicalizes it to lowercase and mostly or entirely uses that - though it does also save the original form). At least I think that's *all* the data - I'll double-check that there's nothing else Mailman saves beyond that (at least of relevance/significance) that's not obtainable from the individual subscriber settings web pages. That then also frees us up from trying to get that data from DreamHost.com (and them possibly mucking it up or saying "no" anyway).

More than good enough. Nobody cares much about their Mailman per-subscription passwords, fortunately. If they bitch, just tell 'em they can change them to suit. It's the usual thing: Optimise for the common case.

Not sure why I didn't think of it earlier (perhaps it wasn't quite as obvious earlier), but another approach which may be highly feasible ... use WWW::Mechanize to load options! So, ... presuming it's fully workable (list admin can load? - need to test that), if that bit is doable - notably list admin can load/edit member's options on the individual members' options page, then that ought be fairly straight forward (moderate extension of what I've already coded to gather the options) - would also have key verification advantage - can (reload and) inspect target page after to ensure options match as expected (programmatically* of course). But I notice list admin can't make *some* member changes (e.g. disallows list admin to change member's password ... at least one Mailman version I looked at) - but setting that isn't an issue/concern - it's all the other options. Anyway, in theory (presuming the relevant tests work), could go quite something like this: "disable" old list (set "emergency" moderation, update description regarding migration - effectively leave old location read-only) scrape source data stop exim (so that member settings will be suitably adjusted before they're notified of subscription having been migrated) subscribe members - preserving email case (as user subscribed**) tweak user's options (using the scraped data, and also verifying results), restart exim *dang, I have to resort to american-english-huge before programmatically passes the spell(1) check. **I also notice Mailman (at least one version I checked) doesn't let user's change their email address to another version of same email address but only with changes in case - it complains that they're already subscribed/member - or something to that effect. Now, I could see requiring email verification if case of localpart changes (as they *could* be distinct addresses/recipients), but seems Mailman ought handle that same way it does for user making any other change to their email address. Anyway, presuming it works, may well get me to the needed load functionality more quickly than the Python pickle (.pck) route or the like - plus has the verification advantage.

From: "Michael Paoli" <Michael.Paoli@cal.berkeley.edu> Subject: Re: lot of really nice command-line tools to use in dumping and editing various pck databases Date: Sun, 30 Jul 2017 11:55:20 -0700

Thanks, slick, noted.

I also noticed there's package (mailman-api)for a REST API for Mailman. I also found a very promising mmclient (which uses Mailman's REST API), but alas, don't see a corresponding package by Debian.

Also thought of something else highly practical ... but, topic drift, will post that separately.

And yes, tested, for the target Mailman, list admin can alter the relevant member's options just fine via the web GUI ... so ... Have what should be sure-fire method now (and eliminates unknown bit about how long it would take for me to figure out method via Python pickle (.pck) or similar CLI interface method) - and it will also include verification. So, with that, I'm guestimating I'll likely do the first *for real* list migration sometime today or tomorrow (after writing additional relevant code, and quite full testing, etc.). Don't think I'll have all the lists migrated by end-of-month (I think that was somewhat earlier target timeline), but I'm thinking all the lists would be migrated* within 5 to 10 days from now. *well, will take a bit longer to get mbox archives from DreamHost.com - presuming we can get 'em to cough 'em up. Some other fairly smallish and relatively easy bits to cover after that to fully extricate ourselves from DreamHost.com ... but getting rather close now. Can almost taste it. :-) Yes, freedom is good!

From: "Michael Paoli" <Michael.Paoli@cal.berkeley.edu> Subject: List migrations and preserving options as feasible - another approach ... WWW::Mechanize to load! Date: Sun, 30 Jul 2017 12:22:18 -0700

Not sure why I didn't think of it earlier (perhaps it wasn't quite as obvious earlier), but another approach which may be highly feasible ... use WWW::Mechanize to load options! So, ... presuming it's fully workable (list admin can load? - need to test that), if that bit is doable - notably list admin can load/edit member's options on the individual members' options page, then that ought be fairly straight forward (moderate extension of what I've already coded to gather the options) - would also have key verification advantage - can (reload and) inspect target page after to ensure options match as expected (programmatically* of course). But I notice list admin can't make *some* member changes (e.g. disallows list admin to change member's password ... at least one Mailman version I looked at) - but setting that isn't an issue/concern - it's all the other options. Anyway, in theory (presuming the relevant tests work), could go quite something like this: "disable" old list (set "emergency" moderation, update description regarding migration - effectively leave old location read-only) scrape source data stop exim (so that member settings will be suitably adjusted before they're notified of subscription having been migrated) subscribe members - preserving email case (as user subscribed**) tweak user's options (using the scraped data, and also verifying results), restart exim

*dang, I have to resort to american-english-huge before programmatically passes the spell(1) check.

**I also notice Mailman (at least one version I checked) doesn't let user's change their email address to another version of same email address but only with changes in case - it complains that they're already subscribed/member - or something to that effect. Now, I could see requiring email verification if case of localpart changes (as they *could* be distinct addresses/recipients), but seems Mailman ought handle that same way it does for user making any other change to their email address.

Anyway, presuming it works, may well get me to the needed load functionality more quickly than the Python pickle (.pck) route or the like - plus has the verification advantage.

...

From: "Michael Paoli" <Michael.Paoli@cal.berkeley.edu> Subject: Re: lot of really nice command-line tools to use in dumping and editing various pck databases Date: Sun, 30 Jul 2017 11:55:20 -0700

...

Thanks, slick, noted.

I also noticed there's package (mailman-api)for a REST API for Mailman. I also found a very promising mmclient (which uses Mailman's REST API), but alas, don't see a corresponding package by Debian.

Also thought of something else highly practical ... but, topic drift, will post that separately.

Rick, Hope you have a fabulous time. And of course much thanks for all the assistance, advice, etc. Don't think I'll require any "real-time" (or even nearly so) assistance ... so should all be good. Certainly don't worry in any case. :-)

From: "Rick Moen" <rick@linuxmafia.com> Subject: Re: [BALUG-Admin] List migrations and preserving options as feasible - another approach ... WWW::Mechanize to load! Date: Sun, 30 Jul 2017 19:23:02 -0700

Quoting Michael Paoli (Michael.Paoli@cal.berkeley.edu):

...

So, with that, I'm guestimating I'll likely do the first *for real* list migration sometime today or tomorrow (after writing additional relevant code, and quite full testing, etc.).

Don't think I'll have all the lists migrated by end-of-month (I think that was somewhat earlier target timeline), but I'm thinking all the lists would be migrated* within 5 to 10 days from now.

You should be told: My wife and I are flying on Sunday, Aug. 6th via Copenhagen to Stockholm, then taking a ferry to Helsinki, being there for Worldcon 75, and then taking a slightly different route back to arrive home Tuesday, August 15th.

I will have my computer with me, but may have intermittent access to bandwidth in the usual way of such things, and also during Worldcon 75 will be primarily busy with Worldcon 75.

That having been said, you may not actually need my real-time or short-deadline participation in anything at all, given that IIRC I already have secondary DNS set up and usable the moment the master starts feeding it and the secondary DNS is made authoritative.

I'm no longer desperately short on sleep, but unfortunately I'm still time-constrained, so I'll have to abbreviate this more than I'd prefer. *I* wrote:

I didn't change that (what you refer to as public name of the list) -- only the Subject header 'tag'. GNU Mailman's defaults evince a fetish for initial capitalisation that I find silly and am at some pains to correct.

That fetish for initial capitalisation includes General Options item 'The public name of this list (make case-changes only)' aka config item real_name. E.g., when I first created the conspire@linuxmafia.com mailing list, GNU Mailman defaulted its 'public name' to Conspire -- which I later corrected in General Options to conspire. I have nothing against _mindful_, deliberate use of capital letters such as your idea of using 'BALUG-Test'. I just dislike the AOL-user-like mindless initial capital being used automatically. You probably know what I mean about the AOL mindset: You find that johnsmith@aol.com keeps insisting that that form of his address is incorrect and that you should re-edit it to 'Johnsmith@aol.com'. *I* wrote:

There's a great deal of dumb online arguing about Mailman's _traditional_ default of mailing each subscriber his/her subscription password on the first of each month. After much pondering, I think on balance it's A Good Thing for several reasons I won't get into unless you wish me to.

The dumbest objection is 'mailing passwords in plaintext is bad security', which misses Mailman pointedly advising users to never use a significant password for this role, and the fact that subscriber passwords otherwise can't do significant mischief. Benefits of automailing each subscriber his/her subscriber password: 1. Reminds the user. Let's face it, subscribers don't keep (or cannot refind) the welcome message, and somehow the information never sinks in that they can have a reminder sent via the listinfo page. 2. Helps periodically revalidate the subscribed address, and let the listadmin know about subscribed addresses that have become permanently bad but haven't yet raised bounce score enough to disable delivery or unsubscribe. For example, I run mailing list linuxcare-alumni for ex-employees of that long-vanished San Francisco firm. Because of bounced password-reminder mails, I get early notice of when longtime members are about to fall off because they've stopped using their longtime mail address but failed to update the subscription with their new one. I can then ask the assembled 'Hey, Andrew Scott's ascott@tuatha.org subscription has become invalid because he isn't accepting SMTP any more at that domain. Does anyone know his current address?' You wrote:

Replace the From: header address with the list's posting address to mitigate issues stemming from the original From: domain's DMARC or similar policies. (from_is_list) - looks like that's set to: No Do nothing special ... anyway, probably should reasonably do as existing lists - if that's not too broken; in any case, can review later.

DMARC has mailing list admins in a bind, because the readdressing that mailing lists perform upon redelivery to subscribers makes the 'From:' header of postings form DMARC-using domains appear to be forged (because it breaks DMARC validation). As mentioned, I did _not_ leave that set to 'Do nothing special'. I have it set to "Munge from'. This is on Privacy Options, Sender Options. I'm guessing you mistakenly though I was referring to a confusingly similar configuration item on General Options that (AFAIK) does _unconditional_ munging of all postings irrespective of whether they are from DMARC domains. I do _not_ recommend the latter, as it spreads the damage from DMARC across _all_ postings, not just those that require it. You wrote:

Send mail to poster when their posting is held for approval? (Edit respond_to_post_requests) ... debatable, not critical,

Explaining my view on this is going to require some background and exposition, so I'll return to it later.