Date: Mon, 18 Sep 2017 01:39:58 -0700 From: "Michael Paoli" Michael.Paoli@cal.berkeley.edu To: BALUG-Test balug-test@temp.balug.org Subject: [BALUG-Test] Oooh, ... archive tests ... :-)
So ... since some while (week(s)?) back, I changed the archiving of BALUG-Test from (default of) monthly, to yearly, that's all fine and good *except* ... archiving looks a bit funky ... notably changing that setting doesn't retroactively change what's already been archived, so, e.g. on https://temp.balug.org/pipermail/balug-test/ we see something that looks approximatley like: Archive ... August 2017: ... July 2017: ... 2017: ...
... ...
Am wondering what the TL;DR of all this is, i.e., the summarized outcome and current status of all Michael P's efforts here? -A
aaronco36@sdf.org SDF Public Access UNIX System - http://sdf.org
Aaron, I seem to no longer have Michael's upthread posting around, so I'm keying off your post to respond to him.
Michael Paoli wrote:
I changed the archiving of BALUG-Test from (default of) monthly, to yearly, that's all fine and good *except* ... archiving looks a bit funky ... notably changing that setting doesn't retroactively change what's already been archived, so, e.g. on https://temp.balug.org/pipermail/balug-test/ we see something that looks approximatley like: Archive ... August 2017: ... July 2017: ... 2017: ...
Fix that using /var/lib/mailman/bin/arch to re-gen the archive, like:
$ su - # su - list $ cd /var/lib/mailman $ bin/arch -q --wipe balug-test archives/private/balug-test.mbox/balug-test.mbox $ exit # exit $
You don't want to be the root user, or you'll end up having to later retroactively chown to list:list the generated archves. (I've made that mistake a time or three. Maybe you just figured that out. ;-> )
/var/lib/mailman/bin/arch will set permissions right. Just don't run it as the wrong user.
(The 'arch' in $PATH is of course something else entirely.)
From: "Rick Moen" rick@linuxmafia.com Subject: Re: [BALUG-Test] Oooh, ... archive tests ... :-) Date: Wed, 20 Sep 2017 14:35:12 -0700
Aaron, I seem to no longer have Michael's upthread posting around, so I'm keying off your post to respond to him.
Michael Paoli wrote:
I changed the archiving of BALUG-Test from (default of) monthly, to yearly, that's all fine and good *except* ... archiving looks a bit funky ... notably changing that setting doesn't retroactively change what's already been archived, so, e.g. on https://temp.balug.org/pipermail/balug-test/ we see something that looks approximatley like: Archive ... August 2017: ... July 2017: ... 2017: ...
Fix that using /var/lib/mailman/bin/arch to re-gen the archive, like:
$ su - # su - list $ cd /var/lib/mailman $ bin/arch -q --wipe balug-test archives/private/balug-test.mbox/balug-test.mbox $ exit # exit $
You don't want to be the root user, or you'll end up having to later retroactively chown to list:list the generated archves. (I've made that mistake a time or three. Maybe you just figured that out. ;-> )
/var/lib/mailman/bin/arch will set permissions right. Just don't run it as the wrong user.
(The 'arch' in $PATH is of course something else entirely.)
Been there, done that. ;-) Though the bit about it being applied (done) to the other lists was rather left as implied by that posting, but yes, also done.
references/excerpts:
https://temp.balug.org/pipermail/balug-test/2017/000014.html
From: "Michael Paoli" Michael.Paoli@cal.berkeley.edu To: BALUG-Test balug-test@temp.balug.org Subject: Oooh, ... archive tests ... :-) Date: Mon, 18 Sep 2017 01:39:58 -0700
So ... since some while (week(s)?) back, I changed the archiving of BALUG-Test from (default of) monthly, to yearly, that's all fine and good *except* ... archiving looks a bit funky ... notably changing that setting doesn't retroactively change what's already been archived, so, e.g. on https://temp.balug.org/pipermail/balug-test/ we see something that looks approximatley like: Archive ... August 2017: ... July 2017: ... 2017: ... Which does look kind'a odd (after all, those months are also within the year 2017 - and it also may not be immedately visually apparent as to whether or not the 2017 archive does or doesn't include July 2017 and August 2017. Also, archive generally lists most recent at top, so, having the most recent (year) data at the bottom is also rather odd in its presentation.
So ... let's reload those archives (I'm overdue for some testing of that anyway) ...
$ hostname && id && ls -ld /var/lib/mailman/archives/private/balug-test.mbox/balug-test.mbox /var/lib/mailman/archives/private/balug-test.mbox/ /var/lib/mailman/archives/private/ && umask balug-sf-lug-v2.balug.org uid=38(list) gid=38(list) groups=38(list) drwxrws--- 12 www-data list 4096 Aug 6 03:11 /var/lib/mailman/archives/private/ drwxrwsr-x 2 root list 4096 Jul 10 01:13 /var/lib/mailman/archives/private/balug-test.mbox/ -rw-rw-r-- 1 list list 63407 Aug 19 23:53 /var/lib/mailman/archives/private/balug-test.mbox/balug-test.mbox 0077 $ //I'd not altered permissions or used the mailman arch command to (re)load an //archive ... that's the "Debian way" they're set up - at least on //old-stable. Note also that that differs somewhat from what the //mailman check_perms says they ought be ... but regardless it works, //and I'm guestimating for Debian is (moderately?) preferable settings //on those. $ (umask 0114 && /usr/lib/mailman/bin/arch --wipe balug-test) ... And ... https://temp.balug.org/pipermail/balug-test/ that looks *much* better, ... but ... looks can be deceiving, web page looks fine, but web accessible archive bits thereof ... nope - looks like it only has what was already in the "year" archive, but lacks the earlier messages. Ah, ... silly browser (and/or what mailman may have earlier told it regarding caching) ... a reload of the page, and then all looks fine. And ... permissions? Let's see if what resulted looks good/sane in that regard, most notably expecting 'em to match similar archives - or at least be highly compatible (most notably relevant access via the group list).
$ (cd /var/lib/mailman/archives/ && find * -follow -type f ! -mmin +60 -exec ls -ld {} ;) -rw-rw-r-- 1 list list 1143 Sep 18 01:07 private/balug-test/index.html -rw-rw-r-- 1 list list 31152 Sep 18 01:07 private/balug-test/2017.txt -rw-rw-r-- 1 list list 5036 Sep 18 01:07 private/balug-test/2017/000003.html -rw-rw-r-- 1 list list 6457 Sep 18 01:07 private/balug-test/2017/000005.html -rw-rw-r-- 1 list list 2185 Sep 18 01:07 private/balug-test/2017/000000.html -rw-rw-r-- 1 list list 3804 Sep 18 01:07 private/balug-test/2017/000007.html lrwxrwxrwx 1 list list 11 Sep 18 01:07 private/balug-test/2017/index.html -> thread.html -rw-rw-r-- 1 list list 3449 Sep 18 01:07 private/balug-test/2017/date.html -rw-rw-r-- 1 list list 7136 Sep 18 01:07 private/balug-test/2017/000001.html -rw-rw-r-- 1 list list 2522 Sep 18 01:07 private/balug-test/2017/000008.html -rw-rw-r-- 1 list list 2740 Sep 18 01:07 private/balug-test/2017/000010.html -rw-rw-r-- 1 list list 8441 Sep 18 01:07 private/balug-test/2017/000002.html -rw-rw-r-- 1 list list 8244 Sep 18 01:07 private/balug-test/2017/000004.html -rw-rw-r-- 1 list list 2553 Sep 18 01:07 private/balug-test/2017/000009.html -rw-rw-r-- 1 list list 7865 Sep 18 01:07 private/balug-test/2017/000006.html -rw-rw-r-- 1 list list 3443 Sep 18 01:07 private/balug-test/2017/subject.html -rw-rw-r-- 1 list list 2416 Sep 18 01:07 private/balug-test/2017/000011.html -rw-rw-r-- 1 list list 3390 Sep 18 01:07 private/balug-test/2017/000012.html -rw-rw-r-- 1 list list 3994 Sep 18 01:07 private/balug-test/2017/thread.html -rw-rw-r-- 1 list list 3075 Sep 18 01:07 private/balug-test/2017/000013.html -rw-rw-r-- 1 list list 3445 Sep 18 01:07 private/balug-test/2017/author.html -rw-rw---- 1 list list 1942 Sep 18 01:07 private/balug-test/database/2017-thread -rw-rw---- 1 list list 19497 Sep 18 01:07 private/balug-test/database/2017-article -rw-rw---- 1 list list 2455 Sep 18 01:07 private/balug-test/database/2017-subject -rw-rw---- 1 list list 1756 Sep 18 01:07 private/balug-test/database/2017-date -rw-rw---- 1 list list 2014 Sep 18 01:07 private/balug-test/database/2017-author -rw-rw---- 1 list list 847 Sep 18 01:07 private/balug-test/pipermail.pck -rw-rw-r-- 1 list list 1143 Sep 18 01:07 public/balug-test/index.html -rw-rw-r-- 1 list list 31152 Sep 18 01:07 public/balug-test/2017.txt -rw-rw-r-- 1 list list 5036 Sep 18 01:07 public/balug-test/2017/000003.html -rw-rw-r-- 1 list list 6457 Sep 18 01:07 public/balug-test/2017/000005.html -rw-rw-r-- 1 list list 2185 Sep 18 01:07 public/balug-test/2017/000000.html -rw-rw-r-- 1 list list 3804 Sep 18 01:07 public/balug-test/2017/000007.html lrwxrwxrwx 1 list list 11 Sep 18 01:07 public/balug-test/2017/index.html -> thread.html -rw-rw-r-- 1 list list 3449 Sep 18 01:07 public/balug-test/2017/date.html -rw-rw-r-- 1 list list 7136 Sep 18 01:07 public/balug-test/2017/000001.html -rw-rw-r-- 1 list list 2522 Sep 18 01:07 public/balug-test/2017/000008.html -rw-rw-r-- 1 list list 2740 Sep 18 01:07 public/balug-test/2017/000010.html -rw-rw-r-- 1 list list 8441 Sep 18 01:07 public/balug-test/2017/000002.html -rw-rw-r-- 1 list list 8244 Sep 18 01:07 public/balug-test/2017/000004.html -rw-rw-r-- 1 list list 2553 Sep 18 01:07 public/balug-test/2017/000009.html -rw-rw-r-- 1 list list 7865 Sep 18 01:07 public/balug-test/2017/000006.html -rw-rw-r-- 1 list list 3443 Sep 18 01:07 public/balug-test/2017/subject.html -rw-rw-r-- 1 list list 2416 Sep 18 01:07 public/balug-test/2017/000011.html -rw-rw-r-- 1 list list 3390 Sep 18 01:07 public/balug-test/2017/000012.html -rw-rw-r-- 1 list list 3994 Sep 18 01:07 public/balug-test/2017/thread.html -rw-rw-r-- 1 list list 3075 Sep 18 01:07 public/balug-test/2017/000013.html -rw-rw-r-- 1 list list 3445 Sep 18 01:07 public/balug-test/2017/author.html -rw-rw---- 1 list list 1942 Sep 18 01:07 public/balug-test/database/2017-thread -rw-rw---- 1 list list 19497 Sep 18 01:07 public/balug-test/database/2017-article -rw-rw---- 1 list list 2455 Sep 18 01:07 public/balug-test/database/2017-subject -rw-rw---- 1 list list 1756 Sep 18 01:07 public/balug-test/database/2017-date -rw-rw---- 1 list list 2014 Sep 18 01:07 public/balug-test/database/2017-author -rw-rw---- 1 list list 847 Sep 18 01:07 public/balug-test/pipermail.pck $ //That would, at least at quick glance, appear quite reasonable, and //seems to function okay, etc. //Is it consistent with the other archives (notably in //permissions/ownerships)? $ (cd /var/lib/mailman/archives/ && find */balug-announce -follow -name database -type d -prune -o -type f -exec ls -ld {} ;) -rw-rw-r-- 1 list list 1204 Aug 14 01:10 private/balug-announce/index.html -rw-rw-r-- 1 list list 7800 Aug 14 01:10 private/balug-announce/2017-August.txt -rw-rw-r-- 1 list list 3552 Aug 14 03:27 private/balug-announce/2017-August.txt.gz -rw-rw---- 1 list list 856 Aug 14 01:10 private/balug-announce/pipermail.pck -rw-rw-r-- 1 list list 10651 Aug 14 01:10 private/balug-announce/2017-August/000000.html lrwxrwxrwx 1 list list 11 Aug 14 01:10 private/balug-announce/2017-August/index.html -> thread.html -rw-rw-r-- 1 list list 1751 Aug 14 01:10 private/balug-announce/2017-August/date.html -rw-rw-r-- 1 list list 1745 Aug 14 01:10 private/balug-announce/2017-August/subject.html -rw-rw-r-- 1 list list 1772 Aug 14 01:10 private/balug-announce/2017-August/thread.html -rw-rw-r-- 1 list list 1747 Aug 14 01:10 private/balug-announce/2017-August/author.html -rw-rw-r-- 1 list list 1204 Aug 14 01:10 public/balug-announce/index.html -rw-rw-r-- 1 list list 7800 Aug 14 01:10 public/balug-announce/2017-August.txt -rw-rw-r-- 1 list list 3552 Aug 14 03:27 public/balug-announce/2017-August.txt.gz -rw-rw---- 1 list list 856 Aug 14 01:10 public/balug-announce/pipermail.pck -rw-rw-r-- 1 list list 10651 Aug 14 01:10 public/balug-announce/2017-August/000000.html lrwxrwxrwx 1 list list 11 Aug 14 01:10 public/balug-announce/2017-August/index.html -> thread.html -rw-rw-r-- 1 list list 1751 Aug 14 01:10 public/balug-announce/2017-August/date.html -rw-rw-r-- 1 list list 1745 Aug 14 01:10 public/balug-announce/2017-August/subject.html -rw-rw-r-- 1 list list 1772 Aug 14 01:10 public/balug-announce/2017-August/thread.html -rw-rw-r-- 1 list list 1747 Aug 14 01:10 public/balug-announce/2017-August/author.html $ //I'd say that all looks quite excellently consistent. Hopefully our //umask value didn't even matter to the arch command, and it simply "did //the right thing" anyway ... but we set a prudent value - in case arch //didn't set it or explicity set permissions. Maybe I'll (further) test //that at some point ... but that's lower on the priority queue. //And ... let's also test delete/add of a message to make sure that //works as we hope and expect ... //and that all tests good, no surprises there. //So ... onto merging into the archives the older raw mbox //bits obtained from DreamHost.com.
Quoting aaronco36 (aaronco36@SDF.ORG):
Am wondering what the TL;DR of all this is, i.e., the summarized outcome and current status of all Michael P's efforts here?
I think he means he successfully used balug-test as a test case (pun semi-intended) for import of mail to the mailing list's mbox and generation of the archive. This was to gain confidence that doing likewise to the main mailing lists has a good plan.
For balug-talk in particular (maybe some others), Michael has in mind to attempt to assemble a more-complete history, adding posting salvaged from other sources such as subscribers' saved mail, and merging that into the mailing list's mbox as a basis for running $MAILMAN_HOME/bin/arch to make a new pipermail archive. Seems worthwhile, but doubtless it'll take some time, and he might have to bodge headers on some of the saved mail to make it work. (You ask subscribers whether they have old postings in mbox format, and in my experience they send you the damnedest random formats, disregarding the qualifier about 'mbox' because they didn't understand it and weren't smart enough to ask. I experienced this when sf-lug mailing list subscribers supplied backfill material for that archive. A lot of what I got was not at all what I asked.)
Sufficent scripting-fu can, in my experience, reassemble an mbox that $MAILMAN_HOME/bin/arch will parse correctly. Just make sure that you keep a safety copy of the most recent mbox _verified_ to parse right, so you don't burn your bridges, only your spare time.
I believe backfilling the archives is the only task remaining. (Dreamhost "oops"-discarded quite a lot, over the years.)
I suppose it depends what one means by "all this" and "outcome" Let's see, in the beginning(?), there was the Big Bang, ... and eventually we have the heat death of the universe.
In the meantime, among other things,
Once upon a time, all of BALUG.org stuff was hosted on DreamHost.com hosting.
In more recent years, fair bit of BALUG.org stuff has been hosted elsewhere, however everything for (among others) the domains themselves of balug.org, www.balug.org, and lists.balug.org, notably including the primary and most of the production BALUG web pages, and BALUG's lists, remained hosted on DreamHost.com hosting.
Anyway, that's no longer the case. Absolutely nothing of BALUG.org is hosted on DreamHost.com hosting any more. DreamHost.com freedom (from) day arrived to BALUG.org early this past Monday. Since then we are - and shall remain, 100% free of DreamHost. Yippie! ... (and *finally*).
Oh, and yes, this was all accomplished with relatively minimal pain/annoyance to the well over 800 list subscribers BALUG has. Except for passwords, all user settings/preferences were preserved and moved over, and there's use of transitional domain temp.balug.org, which will again settle to lists.balug.org ... but not (fully) there quite yet.
And yes, we have our archives ... at least what was on DreamHost.com that they'd not irretrievably lost earlier ... plus of course the newer postings.
But .. this doesn't seem all that much like a test post? ;-)
From: aaronco36 aaronco36@SDF.ORG Subject: Re: [BALUG-Test] Oooh, ... archive tests ... :-) Date: Wed, 20 Sep 2017 20:41:13 +0000 (UTC)
Date: Mon, 18 Sep 2017 01:39:58 -0700 From: "Michael Paoli" Michael.Paoli@cal.berkeley.edu To: BALUG-Test balug-test@temp.balug.org Subject: [BALUG-Test] Oooh, ... archive tests ... :-)
So ... since some while (week(s)?) back, I changed the archiving of BALUG-Test from (default of) monthly, to yearly, that's all fine and good *except* ... archiving looks a bit funky ... notably changing that setting doesn't retroactively change what's already been archived, so, e.g. on https://temp.balug.org/pipermail/balug-test/ we see something that looks approximatley like: Archive ... August 2017: ... July 2017: ... 2017: ...
... ...
Am wondering what the TL;DR of all this is, i.e., the summarized outcome and current status of all Michael P's efforts here?