So, thought some folks might find this rather/quite interesting. After
doing Debian 12 --> 13 upgrades (major version upgrade) on the BALUG VM,
ran into some significant issues. 8-O Most notably regarding mailman3
and related. So, began working to correct that. And, some
methodologies and technologies:
As feasible, made lots of use of half-splitting, see, e.g.:
https://en.wikipedia.org/wiki/Troubleshooting#Half-splitting
Also made lots of use of Virtual Machine (VM) technology,
most notably creating and working from bases of pre-upgrade
snapshot, and also post-upgrade snapshot, allowing much testing
work, figuring out what upgrade bit(s) were causing breakage,
and how to selectively downgrade and/or otherwise fix prod(uction),
while generally minimizing breakage/interruptions to prod.
And writing/thinking of such - snapshots. Utilized both LVM
for that, and also heavily utilized and leveraged ZFS for that too.
LVM has some good useful snapshot capabilities (prod BALUG utilizes LVM,
but not ZFS), but they're relatively limited (basically create snapshot,
snapshot can be made/used read-write (rw)), snapshots can be grown to
increase capacity (notably for differences) if/as needed, only one
snapshot of Logical Volume (LV) can exist at a time, snapshots can be
removed, that's about it. ZFS is way more capable when it comes to
snapshots, in fact ZFS is pretty dang powerful tool (has both volume
management capabilities, plus snapshot capabilities, and many more
capabilities and features), but ZFS is also a huge complex animal, and
very different than more traditional *nix filesystems and even more
common volume management thereupon (and RAID, etc.). Anyway, ZFS
snapshots are always and only read-only (ro). However, if/when one
wants rw from that, with ZFS, one can clone a snapshot, and a clone is
or can be made rw. ZFS can also have relatively arbitrary number of
snapshots from a dataset (effectively volume), and they can be made very
quickly and conveniently, and there's also much flexibility on their
naming, for ease in identification. ZFS also well tracks information
about their origin and creation time, so that can also be quite useful
and informative. ZFS also has a rollback capability, that about
instantly allows one to revert a dataset back to an earlier snapshot -
also super handy.
So, I well used all of those in working to well resolve at least
post-upgrade issues that were encountered/discovered on the prod
BALUG VM. What follows is my notes as I worked through that (and I just
bit ago augmented it slightly to help clarify some bits of context).
At this point, it's still bit of a work-in-progress, but the most
notable major issues have all been corrected on (prod) BALUG (VM) at
this point in time. Anyway, thought at least some might find it an
interesting and/or informative read. Oh, also not (explicitly) covered
in the below, certain parts of the APT configuration. Most notably,
with typical major version upgrade on Debian (and probably likewise for
most APT based systems), one updates sources.list(5) configuration from
the old (e.g. Debian 12 "bookworm") to the new (e.g. Debian 13 "trixie").
That was initially done with prod, but later added both, and (covered
below, at least in part) subsequently added APT pinning to generally
prefer the newer (13 "trixie") over the older (12 "bookworm"), but
also later some additional pinning for specific packages (covered in the
below). And in case it may come up, stable in Debian always refers to
the current major release (presently 13 "trixie"), oldstable the one
before that (presently 12 "bookworm"), and oldoldstable the one before
that.
So, by way of example, what follows below is my (now slightly augmented)
notes I've kept while working through this troubleshooting and working
to resolve these issues.
Debian 12 --> 13 upgraded started a bit after 2026-01-28T03:50:34Z
After 12 --> 13 upgrade, mailman3 web stuff, etc. not working
on test VM (balug1213, based on pre-upgrade snapshot)
upgrading mailman3-web breaks it,
downgrading corrects
downgrading doesn't fix prod
on test VM
12 --> 13 upgrade (no dist-upgrade), and with these held:
mailman3
mailman3-doc
mailman3-full
mailman3-web
python3-django-hyperkitty
python3-django-mailman3
python3-mailman-hyperkitty
python3-robot-detection
that still causes it to break
on test VM
doing just select upgrade packages, some set at atime,
was able to do many (hundreds) with it still working,
but exim4-config broke it, or did after reboot, but
exim4-config drags in hunge number of dependencies, see:
test-VM.progressive.status.just.past.breakage
after checking further, it broke, as reboots showed.
first broken:
test-VM.first.broken.dpkg_-l
test-VM.progressive.status.first.broken
last working before broken:
test-VM.last.working.dpkg_-l
test-VM.progressive.status.last.working
Would appear one of these package upgrades broke it, but only after reboot:
python3-fs python3-gunicorn python3-h2 python3-httplib2
python3-importlib-resources python3-josepy python3-lazr.config
python3-mimeparse python3-mistune python3-more-itertools
python3-mpmath python3-olefile python3-ply python3-pygments
python3-pyudev python3-requests python3-requests-oauthlib
python3-rfc3986 python3-soupsieve
at least one of these breaks it after reboot:
python3-mistune python3-more-itertools python3-mpmath python3-olefile
at least one of these breaks it after reboot:
python3-mistune python3-more-itertools
upgrading this packages breaks it after reboot:
python3-mistune
upgrading one of these packages breaks it after reboot:
python3-django-appconf python3-django-compressor
python3-django-filters python3-django-gravatar2 python3-uritemplate
python3-wcwidth python3-whoosh python3-zipp
upgrading one of these packages breaks it after reboot:
python3-django-appconf python3-django-compressor
python3-django-filters python3-django-haystack
python3-django-picklefield
upgrading this packages breaks it after reboot:
python3-django-compressor
upgrading this set and dependencies + installing apache2 breaks it
after reboot:
apache2-doc aptitude-doc-en bind9-doc bsdextrautils clamav-doc
clamav-docs dpkg-dev
and a major diversion, I/O error while doing:
(cd balug13 && umask 022 && apt-get install apache2-doc apache2
apt-file autoconf automake awstats build-essential debhelper
dh-autoreconf dh-strip-nondeterminism dnsvi dpkg-dev grepmail analog
dlint equivs initscripts insserv postgresql-client
postgresql-client-common sa-exim spamassassin spamd spf-tools-perl
update-inetd libalgorithm-c3-perl libalgorithm-diff-perl
libalgorithm-diff-xs-perl libalgorithm-merge-perl libarchive-cpio-perl
libauthen-ntlm-perl libauthen-sasl-perl libb-hooks-endofscope-perl
libbsd-resource-perl libbytes-random-secure-perl libcapture-tiny-perl
libcgi-fast-perl libcgi-formbuilder-perl libcgi-pm-perl
libclass-c3-perl libclass-c3-xs-perl libclass-inspector-perl
libclass-load-perl libclass-load-xs-perl
libclass-method-modifiers-perl libclass-tiny-perl
libclass-xsaccessor-perl libclone-choose-perl libcommon-sense-perl
libconfig-tiny-perl libconvert-binhex-perl libcrypt-random-seed-perl
libcrypt-ssleay-perl libdata-dump-perl libdata-validate-domain-perl
libdbd-sqlite3-perl libdbix-simple-perl libdevel-caller-perl
libdevel-globaldestruction-perl libdevel-lexalias-perl
libdevel-overloadinfo-perl libdevel-partialdump-perl
libdevel-stacktrace-perl libdevel-symdump-perl
libdigest-bubblebabble-perl libdist-checkconflicts-perl
libemail-abstract-perl libemail-address-xs-perl
libemail-date-format-perl libemail-messageid-perl
libemail-mime-contenttype-perl libemail-mime-encodings-perl
libemail-mime-perl libemail-sender-perl libemail-simple-perl
libencode-locale-perl libeval-closure-perl libfcgi-perl
libfile-basedir-perl libfile-copy-recursive-perl
libfile-desktopentry-perl libfile-fcntllock-perl
libfile-find-rule-perl libfile-listing-perl libfile-mimeinfo-perl
libfile-sharedir-perl libfile-slurper-perl libfile-which-perl
libfont-afm-perl libgeo-ipfree-perl)
on test VM
Major rollback/reset of test VM (because of ZFS issue)
now with some snapshots from total pre-upgrade image from balug
through to cleaned up, backed up, and staged for upgrades,
and thus far working:
poola/balug1213@2026-01-30T20:22:44Z.balug.debian12.pre-upgrade.cleaned.up.backed.up.staged.for.upgrades
Let's do some package holds based upon earlier breakage,
putting hold on these packages:
mailman3 mailman3-doc mailman3-full mailman3-web
python3-django-hyperkitty python3-django-mailman3
python3-mailman-hyperkitty python3-robot-detection exim4-config
python3-mistune python3-django-compressor
and also holding below set, because
upgrading this set and dependencies + installing apache2 breaks it
after reboot:
apache2 apache2-doc aptitude-doc-en bind9-doc bsdextrautils
clamav-doc clamav-docs dpkg-dev
and saved: dpkg_-l.2026-01-30T20:52:59Z
and now upgrade, and not broken yet, take fresh snapshot:
poola/balug1213@2026-01-30T21:44:00Z.balug.debian12.many.packages.upgraded.not.broken.yet
apt-mark unhold aptitude-doc-en bind9-doc clamav-doc clamav-docs dpkg-dev
apt-get upgrade, still not broken after reboot, take fresh snapshot:
poola/balug1213@2026-01-30T22:05:22Z.balug.debian12.many.packages.upgraded.not.broken.yet
presently have these held:
apache2 apache2-doc bsdextrautils exim4-config mailman3 mailman3-doc
mailman3-full mailman3-web python3-django-compressor
python3-django-hyperkitty python3-django-mailman3
python3-mailman-hyperkitty python3-mistune python3-robot-detection
apt-mark unhold apache2-doc
apt-get upgrade
apt-mark unhold bsdextrautils
apt-get upgrade, still not broken after reboot, take fresh snapshot:
poola/balug1213@2026-01-30T22:58:27Z.balug.debian12.many.packages.upgraded.not.broken.yet
apt-mark unhold mailman3-doc exim4-config
apt-get upgrade, still not broken after reboot, take fresh snapshot:
poola/balug1213@2026-01-31T08:38:45Z.balug.debian12.many.packages.upgraded.not.broken.yet
apt-mark unhold apache2
apt-get install apache2 (brings in whole ton of additional/upgraded)
and after that Django still appears working but
https://lists.balug.org/mailman3/postorius/lists/?all-lists
fails showing:
Something went wrong
Mailman REST API not available. Please start Mailman core.
systemctl status mailman3 output includes:
Jan 31 09:26:57 balug1213 mailman3[1034]: import nntplib
Jan 31 09:26:57 balug1213 mailman3[1034]: ModuleNotFoundError: No
module named 'nntplib'
And this despite:
/etc/mailman3/mailman.cfg
(still) having:
[runner.nntp]
start: no
Did a case insensitive search for nttp under /etc, and that was all
that was found of potential relevance.
Searches on-line seem to imply
mailman 3 doesn't support (Debian 13's) python 3.13
(which lacks nntplib)
until mailman3 3.3.11 (Debian 13 has mailman3 3.3.10-2)
apt-get install python3-standard-nntplib
systemctl start mailman3
and after reboot, that was sufficient to fix it, yay!
time for another snapshot:
poola/balug1213@2026-01-31T10:04:03Z.balug.debian12.many.packages.upgraded.not.broken.yet
apt-get upgrade (upgrades lots more packages)
And then thoroughly broken, rollback to:
poola/balug1213@2026-01-31T10:04:03Z.balug.debian12.many.packages.upgraded.not.broken.yet
apt-get upgrade wants to upgrade these:
adduser bc clamav clamav-base clamav-daemon clamav-freshclam
clamdscan coreutils dconf-gsettings-backend dconf-service dpkg ftp-ssl
galera-4 gawk gettext gnuplot-data gnuplot-nox groff groff-base
gtk-update-icon-cache intel-media-va-driver ipmitool iputils-ping
jigdo-file kmod libblosc1 libcairo-gobject2 libcairo2 libcolord2
libcolorhug2 libcwidget-dev libcwidget4 libdconf1 libde265-0
libdjvulibre21 libfreetype6 libfsverity0 libgd3 libgdk-pixbuf-2.0-0
libgdk-pixbuf2.0-bin libgraphene-1.0-0 libgssapi-krb5-2
libgstreamer1.0-0 libgudev-1.0-0 libgusb2 libharfbuzz0b libidn2-0
libigdgmm12 libjack0 libjson-glib-1.0-0 libk5crypto3 libkmod2
libkrb5-3 libkrb5support0 liblqr-1-0 libpam-modules libpam-modules-bin
libpam-runtime libpango-1.0-0 libpangocairo-1.0-0 libpangoft2-1.0-0
libpangoxft-1.0-0 libpolkit-agent-1-0 libpolkit-gobject-1-0 librsvg2-2
librsvg2-common librtmp1 librttopo1 libsqlite3-0 libwacom-bin
libwacom-common libwacom9 libxmu6 libz3-4 lynx man-db ntpsec
ntpsec-ntpdate ntpsec-ntpdig nvi perl-openssl-defaults pinentry-curses
procps python3-alembic python3-anyio python3-bs4
python3-charset-normalizer python3-ntp re2c rsync rsyslog sane-airscan
shared-mime-info socat spamc sqlite3 sudo uwsgi-core
uwsgi-plugin-python3 vsftpd wget x11-apps x11-session-utils
x11-xkb-utils x11-xserver-utils xterm
Let's add some more holds:
apt-mark hold python3-alembic python3-anyio python3-bs4
python3-charset-normalizer python3-ntp sqlite3 uwsgi-core
uwsgi-plugin-python3
apt-get upgrade, that worked, and snapshoted:
poola/balug1213@2026-01-31T11:23:04Z.balug.debian12.many.packages.upgraded.not.broken.yet
apt-mark unhold python3-bs4 python3-ntp
apt-get upgrade
apt-mark unhold python3-alembic python3-anyio
apt-get upgrade, that worked, and snapshoted:
poola/balug1213@2026-01-31T11:33:23Z.balug.debian12.many.packages.upgraded.not.broken.yet
apt-mark unhold python3-charset-normalizer
apt-get upgrade
apt-mark unhold mailman3
apt-get install mailman3
Configure database for mailman3 with dbconfig-common? [yes/no] no
Add the HyperKitty configuration to mailman.cfg? [yes/no] no
that worked, and snapshoted:
poola/balug1213@2026-01-31T13:25:11Z.balug.debian12.many.packages.upgraded.not.broken.yet
apt-mark unhold sqlite3
apt-get upgrade, that worked, and snapshoted:
poola/balug1213@2026-01-31T13:34:29Z.balug.debian12.many.packages.upgraded.not.broken.yet
apt-mark unhold uwsgi-core uwsgi-plugin-python3
apt-get upgrade, broken, rollback to:
poola/balug1213@2026-01-31T13:34:29Z.balug.debian12.many.packages.upgraded.not.broken.yet
apt-mark unhold mailman3-web
apt-get upgrade (broken)
import cgi
ModuleNotFoundError: No module named 'cgi'
apt-get install python3-legacy-cgi (still broken)
apt-mark unhold $(apt-mark showhold) && apt-get upgrade (still broken)
apt-get full-upgrade (ran out of space on /usr before it completed)
dpkg --configure -a
apt-get full-upgrade
apt-get update
apt-get full-upgrade
And, somewhat surprising, the web pages seem more-or-less functional,
but their formatting is all messed up, snapshot:
poola/balug1213@2026-02-01T00:59:17Z.balug.debian12.upgraded.to.13.web.seems.functional.but.formatting.all.messed.up
apt-mark manual ntpsec-ntpdate ntpsec-ntpdig
apt-get autoremove
apt-get remove p7zip
apt-get remove p7zip-full
can't remove transitional package python3-tz because of reverse dependencies
updated /etc/letsencrypt contents
rm /etc/apache2/mods-available/ssl.conf.dpkg-old
/etc/ca-certificates.conf.dpkg-old /etc/default/grub.ucf-dist
/etc/dokuwiki/local.php.ucf-dist /etc/mailman3/mailman.cfg.ucf-dist
apt-get install apt-file
apt-file update
purged a bunch of removed packages (to clear out their configurations)
apt-get purge 7zip
web functionality - more than semi-broken - hyperkitty archives also broken,
I should've more thoroughly checked earlier.
rollback to:
poola/balug1213@2026-01-31T13:34:29Z.balug.debian12.many.packages.upgraded.not.broken.yet
postorius, hyperkitty, django appear working
updated /etc/letsencrypt contents
apt-mark unhold $(apt-mark showhold)
grow /usr /var /home so each <=85% full
apt-get update
apt-get upgrade
all broken: "Internal Server Error": postorius, hyperkitty, django
rollback to:
poola/balug1213@2026-01-31T13:34:29Z.balug.debian12.many.packages.upgraded.not.broken.yet
postorius, hyperkitty, django appear working
updated /etc/letsencrypt contents
apt-get update, snapshot:
poola/balug1213@2026-02-01T05:08:05Z.balug.debian12.many.packages.upgraded.not.broken.yet
apt-mark unhold python3-django-hyperkitty
apt-get upgrade
upgrading python3-django-hyperkitty breaks it:
500 Internal Server Error postorius, hyperkitty, django
rollback to:
poola/balug1213@2026-02-01T05:08:05Z.balug.debian12.many.packages.upgraded.not.broken.yet
apt-mark unhold mailman3-full
apt-get upgrade, so far so good, snapshot:
poola/balug1213@2026-02-01T05:36:56Z.balug.debian12.many.packages.upgraded.not.broken.yet
apt-mark unhold python3-robot-detection
apt-get upgrade, so far so good
apt-mark unhold python3-django-mailman3
apt-mark unhold uwsgi-core uwsgi-plugin-python3
apt-get upgrade
all broken: "Internal Server Error": postorius, hyperkitty, django
rolled back to:
poola/balug1213@2026-02-01T05:36:56Z.balug.debian12.many.packages.upgraded.not.broken.yet
apt-mark unhold python3-django-mailman3 python3-robot-detection
apt-get upgrade, so far so good
apt-mark unhold python3-mailman-hyperkitty
apt-get upgrade, so far so good, snapshot:
poola/balug1213@2026-02-01T06:07:43Z.balug.debian12.many.packages.upgraded.not.broken.yet
Have these packages held:
mailman3-web python3-django-compressor python3-django-hyperkitty
python3-mistune uwsgi-core uwsgi-plugin-python3
apt-get full-upgrade
all broken: "Internal Server Error": postorius, hyperkitty, django
rollback:
poola/balug1213@2026-02-01T06:07:43Z.balug.debian12.many.packages.upgraded.not.broken.yet
So, to try and do full-upgrade more step-wise without breaking things,
will attempt hold on many packages it would otherwise upgrade and/or remove.
# echo $(apt-mark showhold)
mailman3-web python3-django-compressor python3-django-hyperkitty
python3-mistune uwsgi-core uwsgi-plugin-python3
# apt-mark hold $( (cd / && umask 022 && apt-get -s full-upgrade)
2>&1 | sed -ne '/^The following packages will be REMOVED:$/{:r;n;/^
/{p;br}};/^The following packages will be upgraded:$/{:u;n;/^
/{p;bu}};')
That holds all that would be upgraded/removed, apt-get -s
full-upgrade shows nothing to do
apt-mark unhold $(apt-mark showhold | grep '^lib')
apt-get -s full-upgrade shows:
30 upgraded, 50 newly installed, 28 to remove and 163 not upgraded
let's do: apt-get full-upgrade, so far so good, snapshot:
poola/balug1213@2026-02-02T03:14:14Z.balug.debian12.many.packages.upgraded.not.broken.yet
apt-mark unhold $(apt-mark showhold | grep -E -e '^(bind9|linux-image)')
apt-get full-upgrade, so far so good, snapshot:
poola/balug1213@2026-02-02T03:33:16Z.balug.debian12.many.packages.upgraded.not.broken.yet
apt-get purge linux-image-6.1.0-42-amd64
apt-mark unhold $(apt-mark showhold | grep -E -e
'^(gnupg|gpg|grub|reiserfsprogs)')
apt-get full-upgrade, so far so good, snapshot:
poola/balug1213@2026-02-02T03:55:56Z.balug.debian12.many.packages.upgraded.not.broken.yet
added [www.]berkeleylug.com domains to /etc/hosts, wiki & wordpress
login okay, added to checks
apt-mark unhold $(apt-mark showhold | grep -E -e '^(dokuwiki|php|wordpress)')
apt-get full-upgrade, so far so good, snapshot:
poola/balug1213@2026-02-02T04:36:57Z.balug.debian12.many.packages.upgraded.not.broken.yet
apt-mark unhold $(apt-mark showhold | grep -E -e
'^(file|firefox|fonts|g\+\+|gcc|binutils|cpp|e2fsprogs)')
apt-get full-upgrade, so far so good, snapshot:
poola/balug1213@2026-02-02T04:53:57Z.balug.debian12.many.packages.upgraded.not.broken.yet
apt-mark unhold $(apt-mark showhold | grep -E -v -e '^(python3|uwsgi)')
apt-get full-upgrade, mailman3-web configuration broken, rollback:
poola/balug1213@2026-02-02T04:53:57Z.balug.debian12.many.packages.upgraded.not.broken.yet
VM balug13 (based off of current snapshot of prod VM balug):
added [www.]berkeleylug.com domains to /etc/hosts, & snapshot:
poola/balug13@2026-02-02T05:47:19Z.balug.13.well.after.upgrade.plus.reconfigured.for.non-conflicting.network.use
VM balug1213:
apt-mark unhold $(apt-mark showhold | grep -E -v -e
'^(exim4|mailman3|python3|uwsgi)')
apt-get full-upgrade, so far so good,
"only" these packages held:
# echo $(apt-mark showhold) | fold -s -w 70
exim4-base exim4-daemon-heavy mailman3-web python3-acme python3-arrow
python3-certbot python3-coreapi python3-cryptography
python3-django-allauth python3-django-compressor
python3-django-hyperkitty python3-dnspython python3-elasticsearch
python3-jsonschema python3-markdown-it python3-mistune python3-openid
python3-openssl python3-passlib python3-pyinotify python3-pythran
python3-readme-renderer python3-tz python3-wheel-whl uwsgi-core
uwsgi-plugin-python3
#
z="$(Z)" && dpkg -l | xz -9 > balug1213.dpkg_-l."$z".xz
snapshot:
poola/balug1213@2026-02-02T13:13:58Z.balug.debian12.many.packages.upgraded.not.broken.yet
VM balug13:
apt proxy configuration
apt-get update
apt-get --allow-downgrades install certbot=2.1.0-4
exim4-base=4.96-15+deb12u7 exim4-daemon-heavy=4.96-15+deb12u7
mailman3-web=0+20200530-2.1 python3-acme=2.1.0-1 python3-arrow=1.2.3-1
python3-certbot=2.1.0-4 python3-coreapi=2.3.3-6
python3-cryptography=38.0.4-3+deb12u1
python3-django=3:3.2.19-1+deb12u1 python3-django-allauth=0.51.0-1
python3-django-compressor=4.0-1 python3-django-extensions=3.2.1-2
python3-django-hyperkitty=1.3.7-1 python3-django-mailman3=1.3.9-1
python3-django-postorius=1.3.8-3
python3-djangorestframework=3.14.0-2+deb12u1 python3-dnspython=2.3.0-1
python3-elasticsearch=7.17.6-1 python3-jsonschema=4.10.3-1
python3-markdown-it=2.1.0-5 python3-mistune=2.0.4-1
python3-openid=3.2.0-2 python3-openssl=23.0.0-1
python3-passlib=1.7.4-3 python3-pyinotify=0.9.6-2
python3-pythran=0.11.0+ds-7 python3-readme-renderer=37.3-2
python3-wheel-whl=0.38.4-2 uwsgi-core=2.0.21-5.1
uwsgi-plugin-python3=2.0.21-5.1
Yay! All web interface stuff appears to be working!, snapshot:
poola/balug13@2026-02-02T13:49:40Z.balug.13.well.after.upgrade.plus.reconfigured.for.non-conflicting.network.use.appears.to.be.working
Now attempt likewise to production balug VM:
VM balug (production!):
apt-get update
apt-get --allow-downgrades install certbot=2.1.0-4
exim4-base=4.96-15+deb12u7 exim4-daemon-heavy=4.96-15+deb12u7
mailman3-web=0+20200530-2.1 python3-acme=2.1.0-1 python3-arrow=1.2.3-1
python3-certbot=2.1.0-4 python3-coreapi=2.3.3-6
python3-cryptography=38.0.4-3+deb12u1
python3-django=3:3.2.19-1+deb12u1 python3-django-allauth=0.51.0-1
python3-django-compressor=4.0-1 python3-django-extensions=3.2.1-2
python3-django-hyperkitty=1.3.7-1 python3-django-mailman3=1.3.9-1
python3-django-postorius=1.3.8-3
python3-djangorestframework=3.14.0-2+deb12u1 python3-dnspython=2.3.0-1
python3-elasticsearch=7.17.6-1 python3-jsonschema=4.10.3-1
python3-markdown-it=2.1.0-5 python3-mistune=2.0.4-1
python3-openid=3.2.0-2 python3-openssl=23.0.0-1
python3-passlib=1.7.4-3 python3-pyinotify=0.9.6-2
python3-pythran=0.11.0+ds-7 python3-readme-renderer=37.3-2
python3-wheel-whl=0.38.4-2 uwsgi-core=2.0.21-5.1
uwsgi-plugin-python3=2.0.21-5.1
Drats! Not working, mailman3 related web pages giving:
Server error
An error occurred while processing your request.
compared all files of type ordinary file beneath /etc between
balug13 and balug, no unexpected differences
tried some URLs with curl, seemed to work!
cleared site cookies, then it works!
Could that be what it was all along? No, not entirely, that wouldn't
explain the other issues earlier (e.g. authenticated but formatting
all messed up).
Issue with exim4, fails with mailman3 but seems to be otherwise working:
R=lman3_router T=mailman3_transport defer (-32): failed to lookup IP
address for localhost
Fixed by adding line:
gethostbyname
to file:
/etc/exim4/conf.d/transport/55_mm3_transport
between the lines:
allow_localhost
hosts = localhost
and restart of exim4, process restarted at:
2026-02-09T20:53:03Z
After that, all the mailman3 web and mail interface appears working again.
VM balug13:
tried apt-get update and apt-get full-upgrade per routine security/bug fixes,
notably also included
[SECURITY] [DSA 6136-1] python-django security update
and that then seriously broke thing(s), notably including at least
mailman3-web
zfs rollback poola/balug13@2026-02-02T13:49:40Z.balug.13.well.after.upgrade.plus.reconfigured.for.non-conflicting.network.use.appears.to.be.working
Both this VM and prod VM have no packages held,
should probably adjust that to make things less fragile as warranted and
appropriate, most notably so, e.g. routine:
apt-get update && apt-get full-upgrade
doesn't break things.
apt-get autopurge
Let's put a hold on many of the packages it wants to upgrade:
echo $(apt-mark showhold)
exim4-base exim4-daemon-heavy mailman3-web postgresql-client-17
python3-arrow python3-coreapi python3-django python3-django-allauth
python3-django-compressor python3-django-extensions
python3-django-hyperkitty python3-django-mailman3
python3-django-postorius python3-djangorestframework python3-dnspython
python3-elasticsearch python3-jsonschema python3-markdown-it
python3-mistune python3-openid python3-passlib python3-pyinotify
python3-pythran python3-readme-renderer python3-wheel-whl uwsgi-core
uwsgi-plugin-python3
apt-get full-upgrade, still good, snapshot:
poola/balug13@2026-02-16T04:28:04Z.balug.13.well.after.upgrade.plus.reconfigured.for.non-conflicting.network.use.appears.to.be.working
apt-mark unhold postgresql-client-17
apt-get full-upgrade, still good, snapshot:
poola/balug13@2026-02-16T04:44:10Z.balug.13.well.after.upgrade.plus.reconfigured.for.non-conflicting.network.use.appears.to.be.working
purged no longer needed kernel, snapshot:
poola/balug13@2026-02-16T04:56:30Z.balug.13.well.after.upgrade.plus.reconfigured.for.non-conflicting.network.use.appears.to.be.working
VM balug (production!):
safety sake / accident prevention, let's likewise apply holds:
apt-mark hold exim4-base exim4-daemon-heavy mailman3-web
python3-arrow python3-coreapi python3-django python3-django-allauth
python3-django-compressor python3-django-extensions
python3-django-hyperkitty python3-django-mailman3
python3-django-postorius python3-djangorestframework python3-dnspython
python3-elasticsearch python3-jsonschema python3-markdown-it
python3-mistune python3-openid python3-passlib python3-pyinotify
python3-pythran python3-readme-renderer python3-wheel-whl uwsgi-core
uwsgi-plugin-python3
so we can generally do:
apt-get update && apt-get full-upgrade
without breaking things.
Probably better to pin those to bookworm(+security(& udpates)) &
unhold, so those held back can follow (bookworm) security and bug
fixes, but holds will do for now.
VM balug13:
apt-mark unhold python3-arrow python3-coreapi python3-dnspython
python3-mistune python3-openid python3-passlib python3-pyinotify
python3-readme-renderer python3-wheel-whl
apt-get full-upgrade, 500 Internal Server Error on all the mailman3
web stuff, rollback:
poola/balug13@2026-02-16T04:56:30Z.balug.13.well.after.upgrade.plus.reconfigured.for.non-conflicting.network.use.appears.to.be.working
apt-mark unhold python3-arrow
apt-get full-upgrade, ok, snapshot:
poola/balug13@2026-02-16T05:45:37Z.balug.13.well.after.upgrade.plus.reconfigured.for.non-conflicting.network.use.appears.to.be.working
apt-mark unhold python3-coreapi
apt-get full-upgrade, ok, snapshot:
poola/balug13@2026-02-16T05:52:42Z.balug.13.well.after.upgrade.plus.reconfigured.for.non-conflicting.network.use.appears.to.be.working
apt-mark unhold python3-dnspython
apt-get full-upgrade, ok, snapshot:
poola/balug13@2026-02-16T06:01:34Z.balug.13.well.after.upgrade.plus.reconfigured.for.non-conflicting.network.use.appears.to.be.working
apt-mark unhold python3-elasticsearch
apt-get full-upgrade, ok, snapshot:
poola/balug13@2026-02-16T06:08:39Z.balug.13.well.after.upgrade.plus.reconfigured.for.non-conflicting.network.use.appears.to.be.working
apt-mark unhold python3-readme-renderer python3-passlib python3-pyinotify
apt-get full-upgrade, ok, snapshot:
poola/balug13@2026-02-16T06:21:49Z.balug.13.well.after.upgrade.plus.reconfigured.for.non-conflicting.network.use.appears.to.be.working
apt-mark unhold python3-markdown-it python3-openid
apt-get full-upgrade, ok, snapshot:
poola/balug13@2026-02-16T06:28:09Z.balug.13.well.after.upgrade.plus.reconfigured.for.non-conflicting.network.use.appears.to.be.working
apt-mark unhold python3-mistune python3-jsonschema
apt-get full-upgrade, 500 Internal Server Error on all the mailman3
web stuff, rollback:
poola/balug13@2026-02-16T06:28:09Z.balug.13.well.after.upgrade.plus.reconfigured.for.non-conflicting.network.use.appears.to.be.working
apt-mark unhold python3-pythran
apt-get full-upgrade, ok, snapshot:
poola/balug13@2026-02-16T06:43:39Z.balug.13.well.after.upgrade.plus.reconfigured.for.non-conflicting.network.use.appears.to.be.working
apt-mark unhold python3-wheel-whl
apt-get full-upgrade, ok, snapshot:
poola/balug13@2026-02-16T06:51:37Z.balug.13.well.after.upgrade.plus.reconfigured.for.non-conflicting.network.use.appears.to.be.working
apt-mark unhold python3-mistune
apt-get full-upgrade, 500 Internal Server Error on all the mailman3
web stuff, rollback:
poola/balug13@2026-02-16T06:51:37Z.balug.13.well.after.upgrade.plus.reconfigured.for.non-conflicting.network.use.appears.to.be.working
apt-mark unhold python3-jsonschema
apt-get full-upgrade, ok, snapshot:
poola/balug13@2026-02-16T07:04:15Z.OK
updated/added apt pinning:
# more /etc/apt/pref*.d/* | cat
::::::::::::::
/etc/apt/preferences.d/97suites
::::::::::::::
Explanation: lower priority for bookworm
Package: *
Pin: release n=bookworm
Pin-Priority: 300
Package: *
Pin: release n=bookworm-updates
Pin-Priority: 300
Package: *
Pin: release n=bookworm-security
Pin-Priority: 300
::::::::::::::
/etc/apt/preferences.d/98packages
::::::::::::::
Explanation: higher priority for certain packages from bookworm
Package: exim4-base exim4-daemon-heavy mailman3-web python3-django
python3-django-allauth python3-django-compressor
python3-django-extensions python3-django-hyperkitty
python3-django-mailman3 python3-django-postorius
python3-djangorestframework python3-mistune uwsgi-core
uwsgi-plugin-python3
Pin: release n=bookworm
Pin-Priority: 995
Package: exim4-base exim4-daemon-heavy mailman3-web python3-django
python3-django-allauth python3-django-compressor
python3-django-extensions python3-django-hyperkitty
python3-django-mailman3 python3-django-postorius
python3-djangorestframework python3-mistune uwsgi-core
uwsgi-plugin-python3
Pin: release n=bookworm-updates
Pin-Priority: 995
Package: exim4-base exim4-daemon-heavy mailman3-web python3-django
python3-django-allauth python3-django-compressor
python3-django-extensions python3-django-hyperkitty
python3-django-mailman3 python3-django-postorius
python3-djangorestframework python3-mistune uwsgi-core
uwsgi-plugin-python3
Pin: release n=bookworm-security
Pin-Priority: 995
#
released all package holds:
apt-mark unhold $(apt-mark showhold)
apt-get update && apt-get full-upgrade, ok, snapshot:
poola/balug13@2026-02-16T08:37:22Z.OK
VM balug (production!):
post Debian 12 --> 13 upgrade issue:
IPv6 addresses mostly not properly initializing at boot,
booting to single user / maintenance mode, then ipdown -a, ifup -a
brings them up.
VM balug13:
reconfigure network to more closely resemble prod
solution / work-around to IPv6 issue, get rid of systemd, use sysvinit,
ok, snapshot:
poola/balug13@2026-02-16T17:43:18Z.OK
VM balug (production!):
apt-get --allow-remove-essential install sysvinit-core systemd-sysv-
apt-get purge openipmi radvd sane-utils
apt-get autopurge
and now IPv6 configured properly upon boot
also noticed (I?, or possibly upgrades?) apparently disabled rsync
server long ago,
TODO should properly reconfigure and enable that (notably so others can
efficiently rsync public list archives)
TODO also still, should figure out (first on test VM)
minimizing packages held back by pinning (without breaking prod), and
better / more "permanent" fix for that (main support for Debian 12
ends 2026-06-10, after that, drops to LTS through 2028-06-30 (then
ELTS through 2023-06-30))
TODO despite exim4's configuration to only send on specific IP addresses,
it will still sometimes use others (e.g. if network routing permits,
such as when desired sending IP address that's needed isn't
available), find a fix for that (or implement work-around - e.g.
firewall it away from doing so, if need be)
---------- Forwarded message ---------
From: Fixit Clinic <fixitclinic(a)gmail.com>
Date: Sun, Feb 15, 2026 at 9:39 PM
Subject: Post about Linux ExChromebook Installfest in Hayward this Saturday?
Hi Michael:
Can you please spread the word through your Linux groups about the
event below this Saturday?
=======
This Saturday Feb 21: Linux ExChromebook Installfest in Hayward
StrayCap Multispace a new not-for-profit Makerspace in Hayward
(https://straycap2.wixsite.com/mysite/home), they’ve got hundreds of
expired school system Chromebooks to convert to Linux laptops. (See
https://bit.ly/cbsave for background.)
So let’s meet this Saturday Feb 21 at StrayCap noon-3PM; bring any
laptop disassembly tools and USB drives with distros and we’ll work
together to convert and reconfigure the Chromebooks while asking
questions like:
-How can we best re-purpose these in our communities to improve
digital literacy, digital sovereignty, digital inclusion, digital
equity?
-Depending on those use cases: which distro(s) work best?
-How much can we automate the installation and post-installation process?
(You’re welcome to take some home to experiment with.)
Hope you see you Saturday, -Peter
Peter Mui @ Fixit Clinic
www.fixitclinic.org – Neighbor-to-neighbor repair fostering thriving,
resilient communities
+1 408 647 5790
fixitclinic(a)gmail.com
========