So, thought some folks might find this rather/quite interesting. After doing Debian 12 --> 13 upgrades (major version upgrade) on the BALUG VM, ran into some significant issues. 8-O Most notably regarding mailman3 and related. So, began working to correct that. And, some methodologies and technologies: As feasible, made lots of use of half-splitting, see, e.g.: https://en.wikipedia.org/wiki/Troubleshooting#Half-splitting Also made lots of use of Virtual Machine (VM) technology, most notably creating and working from bases of pre-upgrade snapshot, and also post-upgrade snapshot, allowing much testing work, figuring out what upgrade bit(s) were causing breakage, and how to selectively downgrade and/or otherwise fix prod(uction), while generally minimizing breakage/interruptions to prod. And writing/thinking of such - snapshots. Utilized both LVM for that, and also heavily utilized and leveraged ZFS for that too. LVM has some good useful snapshot capabilities (prod BALUG utilizes LVM, but not ZFS), but they're relatively limited (basically create snapshot, snapshot can be made/used read-write (rw)), snapshots can be grown to increase capacity (notably for differences) if/as needed, only one snapshot of Logical Volume (LV) can exist at a time, snapshots can be removed, that's about it. ZFS is way more capable when it comes to snapshots, in fact ZFS is pretty dang powerful tool (has both volume management capabilities, plus snapshot capabilities, and many more capabilities and features), but ZFS is also a huge complex animal, and very different than more traditional *nix filesystems and even more common volume management thereupon (and RAID, etc.). Anyway, ZFS snapshots are always and only read-only (ro). However, if/when one wants rw from that, with ZFS, one can clone a snapshot, and a clone is or can be made rw. ZFS can also have relatively arbitrary number of snapshots from a dataset (effectively volume), and they can be made very quickly and conveniently, and there's also much flexibility on their naming, for ease in identification. ZFS also well tracks information about their origin and creation time, so that can also be quite useful and informative. ZFS also has a rollback capability, that about instantly allows one to revert a dataset back to an earlier snapshot - also super handy.
So, I well used all of those in working to well resolve at least post-upgrade issues that were encountered/discovered on the prod BALUG VM. What follows is my notes as I worked through that (and I just bit ago augmented it slightly to help clarify some bits of context). At this point, it's still bit of a work-in-progress, but the most notable major issues have all been corrected on (prod) BALUG (VM) at this point in time. Anyway, thought at least some might find it an interesting and/or informative read. Oh, also not (explicitly) covered in the below, certain parts of the APT configuration. Most notably, with typical major version upgrade on Debian (and probably likewise for most APT based systems), one updates sources.list(5) configuration from the old (e.g. Debian 12 "bookworm") to the new (e.g. Debian 13 "trixie"). That was initially done with prod, but later added both, and (covered below, at least in part) subsequently added APT pinning to generally prefer the newer (13 "trixie") over the older (12 "bookworm"), but also later some additional pinning for specific packages (covered in the below). And in case it may come up, stable in Debian always refers to the current major release (presently 13 "trixie"), oldstable the one before that (presently 12 "bookworm"), and oldoldstable the one before that.
So, by way of example, what follows below is my (now slightly augmented) notes I've kept while working through this troubleshooting and working to resolve these issues.
Debian 12 --> 13 upgraded started a bit after 2026-01-28T03:50:34Z After 12 --> 13 upgrade, mailman3 web stuff, etc. not working on test VM (balug1213, based on pre-upgrade snapshot) upgrading mailman3-web breaks it, downgrading corrects downgrading doesn't fix prod on test VM 12 --> 13 upgrade (no dist-upgrade), and with these held: mailman3 mailman3-doc mailman3-full mailman3-web python3-django-hyperkitty python3-django-mailman3 python3-mailman-hyperkitty python3-robot-detection that still causes it to break on test VM doing just select upgrade packages, some set at atime, was able to do many (hundreds) with it still working, but exim4-config broke it, or did after reboot, but exim4-config drags in hunge number of dependencies, see: test-VM.progressive.status.just.past.breakage after checking further, it broke, as reboots showed. first broken: test-VM.first.broken.dpkg_-l test-VM.progressive.status.first.broken last working before broken: test-VM.last.working.dpkg_-l test-VM.progressive.status.last.working Would appear one of these package upgrades broke it, but only after reboot: python3-fs python3-gunicorn python3-h2 python3-httplib2 python3-importlib-resources python3-josepy python3-lazr.config python3-mimeparse python3-mistune python3-more-itertools python3-mpmath python3-olefile python3-ply python3-pygments python3-pyudev python3-requests python3-requests-oauthlib python3-rfc3986 python3-soupsieve at least one of these breaks it after reboot: python3-mistune python3-more-itertools python3-mpmath python3-olefile at least one of these breaks it after reboot: python3-mistune python3-more-itertools upgrading this packages breaks it after reboot: python3-mistune upgrading one of these packages breaks it after reboot: python3-django-appconf python3-django-compressor python3-django-filters python3-django-gravatar2 python3-uritemplate python3-wcwidth python3-whoosh python3-zipp upgrading one of these packages breaks it after reboot: python3-django-appconf python3-django-compressor python3-django-filters python3-django-haystack python3-django-picklefield upgrading this packages breaks it after reboot: python3-django-compressor upgrading this set and dependencies + installing apache2 breaks it after reboot: apache2-doc aptitude-doc-en bind9-doc bsdextrautils clamav-doc clamav-docs dpkg-dev and a major diversion, I/O error while doing: (cd balug13 && umask 022 && apt-get install apache2-doc apache2 apt-file autoconf automake awstats build-essential debhelper dh-autoreconf dh-strip-nondeterminism dnsvi dpkg-dev grepmail analog dlint equivs initscripts insserv postgresql-client postgresql-client-common sa-exim spamassassin spamd spf-tools-perl update-inetd libalgorithm-c3-perl libalgorithm-diff-perl libalgorithm-diff-xs-perl libalgorithm-merge-perl libarchive-cpio-perl libauthen-ntlm-perl libauthen-sasl-perl libb-hooks-endofscope-perl libbsd-resource-perl libbytes-random-secure-perl libcapture-tiny-perl libcgi-fast-perl libcgi-formbuilder-perl libcgi-pm-perl libclass-c3-perl libclass-c3-xs-perl libclass-inspector-perl libclass-load-perl libclass-load-xs-perl libclass-method-modifiers-perl libclass-tiny-perl libclass-xsaccessor-perl libclone-choose-perl libcommon-sense-perl libconfig-tiny-perl libconvert-binhex-perl libcrypt-random-seed-perl libcrypt-ssleay-perl libdata-dump-perl libdata-validate-domain-perl libdbd-sqlite3-perl libdbix-simple-perl libdevel-caller-perl libdevel-globaldestruction-perl libdevel-lexalias-perl libdevel-overloadinfo-perl libdevel-partialdump-perl libdevel-stacktrace-perl libdevel-symdump-perl libdigest-bubblebabble-perl libdist-checkconflicts-perl libemail-abstract-perl libemail-address-xs-perl libemail-date-format-perl libemail-messageid-perl libemail-mime-contenttype-perl libemail-mime-encodings-perl libemail-mime-perl libemail-sender-perl libemail-simple-perl libencode-locale-perl libeval-closure-perl libfcgi-perl libfile-basedir-perl libfile-copy-recursive-perl libfile-desktopentry-perl libfile-fcntllock-perl libfile-find-rule-perl libfile-listing-perl libfile-mimeinfo-perl libfile-sharedir-perl libfile-slurper-perl libfile-which-perl libfont-afm-perl libgeo-ipfree-perl) on test VM Major rollback/reset of test VM (because of ZFS issue) now with some snapshots from total pre-upgrade image from balug through to cleaned up, backed up, and staged for upgrades, and thus far working: poola/balug1213@2026-01-30T20:22:44Z.balug.debian12.pre-upgrade.cleaned.up.backed.up.staged.for.upgrades Let's do some package holds based upon earlier breakage, putting hold on these packages: mailman3 mailman3-doc mailman3-full mailman3-web python3-django-hyperkitty python3-django-mailman3 python3-mailman-hyperkitty python3-robot-detection exim4-config python3-mistune python3-django-compressor and also holding below set, because upgrading this set and dependencies + installing apache2 breaks it after reboot: apache2 apache2-doc aptitude-doc-en bind9-doc bsdextrautils clamav-doc clamav-docs dpkg-dev and saved: dpkg_-l.2026-01-30T20:52:59Z and now upgrade, and not broken yet, take fresh snapshot: poola/balug1213@2026-01-30T21:44:00Z.balug.debian12.many.packages.upgraded.not.broken.yet apt-mark unhold aptitude-doc-en bind9-doc clamav-doc clamav-docs dpkg-dev apt-get upgrade, still not broken after reboot, take fresh snapshot: poola/balug1213@2026-01-30T22:05:22Z.balug.debian12.many.packages.upgraded.not.broken.yet presently have these held: apache2 apache2-doc bsdextrautils exim4-config mailman3 mailman3-doc mailman3-full mailman3-web python3-django-compressor python3-django-hyperkitty python3-django-mailman3 python3-mailman-hyperkitty python3-mistune python3-robot-detection apt-mark unhold apache2-doc apt-get upgrade apt-mark unhold bsdextrautils apt-get upgrade, still not broken after reboot, take fresh snapshot: poola/balug1213@2026-01-30T22:58:27Z.balug.debian12.many.packages.upgraded.not.broken.yet apt-mark unhold mailman3-doc exim4-config apt-get upgrade, still not broken after reboot, take fresh snapshot: poola/balug1213@2026-01-31T08:38:45Z.balug.debian12.many.packages.upgraded.not.broken.yet apt-mark unhold apache2 apt-get install apache2 (brings in whole ton of additional/upgraded) and after that Django still appears working but https://lists.balug.org/mailman3/postorius/lists/?all-lists fails showing: Something went wrong Mailman REST API not available. Please start Mailman core. systemctl status mailman3 output includes: Jan 31 09:26:57 balug1213 mailman3[1034]: import nntplib Jan 31 09:26:57 balug1213 mailman3[1034]: ModuleNotFoundError: No module named 'nntplib' And this despite: /etc/mailman3/mailman.cfg (still) having: [runner.nntp] start: no Did a case insensitive search for nttp under /etc, and that was all that was found of potential relevance. Searches on-line seem to imply mailman 3 doesn't support (Debian 13's) python 3.13 (which lacks nntplib) until mailman3 3.3.11 (Debian 13 has mailman3 3.3.10-2) apt-get install python3-standard-nntplib systemctl start mailman3 and after reboot, that was sufficient to fix it, yay! time for another snapshot: poola/balug1213@2026-01-31T10:04:03Z.balug.debian12.many.packages.upgraded.not.broken.yet apt-get upgrade (upgrades lots more packages) And then thoroughly broken, rollback to: poola/balug1213@2026-01-31T10:04:03Z.balug.debian12.many.packages.upgraded.not.broken.yet apt-get upgrade wants to upgrade these: adduser bc clamav clamav-base clamav-daemon clamav-freshclam clamdscan coreutils dconf-gsettings-backend dconf-service dpkg ftp-ssl galera-4 gawk gettext gnuplot-data gnuplot-nox groff groff-base gtk-update-icon-cache intel-media-va-driver ipmitool iputils-ping jigdo-file kmod libblosc1 libcairo-gobject2 libcairo2 libcolord2 libcolorhug2 libcwidget-dev libcwidget4 libdconf1 libde265-0 libdjvulibre21 libfreetype6 libfsverity0 libgd3 libgdk-pixbuf-2.0-0 libgdk-pixbuf2.0-bin libgraphene-1.0-0 libgssapi-krb5-2 libgstreamer1.0-0 libgudev-1.0-0 libgusb2 libharfbuzz0b libidn2-0 libigdgmm12 libjack0 libjson-glib-1.0-0 libk5crypto3 libkmod2 libkrb5-3 libkrb5support0 liblqr-1-0 libpam-modules libpam-modules-bin libpam-runtime libpango-1.0-0 libpangocairo-1.0-0 libpangoft2-1.0-0 libpangoxft-1.0-0 libpolkit-agent-1-0 libpolkit-gobject-1-0 librsvg2-2 librsvg2-common librtmp1 librttopo1 libsqlite3-0 libwacom-bin libwacom-common libwacom9 libxmu6 libz3-4 lynx man-db ntpsec ntpsec-ntpdate ntpsec-ntpdig nvi perl-openssl-defaults pinentry-curses procps python3-alembic python3-anyio python3-bs4 python3-charset-normalizer python3-ntp re2c rsync rsyslog sane-airscan shared-mime-info socat spamc sqlite3 sudo uwsgi-core uwsgi-plugin-python3 vsftpd wget x11-apps x11-session-utils x11-xkb-utils x11-xserver-utils xterm Let's add some more holds: apt-mark hold python3-alembic python3-anyio python3-bs4 python3-charset-normalizer python3-ntp sqlite3 uwsgi-core uwsgi-plugin-python3 apt-get upgrade, that worked, and snapshoted: poola/balug1213@2026-01-31T11:23:04Z.balug.debian12.many.packages.upgraded.not.broken.yet apt-mark unhold python3-bs4 python3-ntp apt-get upgrade apt-mark unhold python3-alembic python3-anyio apt-get upgrade, that worked, and snapshoted: poola/balug1213@2026-01-31T11:33:23Z.balug.debian12.many.packages.upgraded.not.broken.yet apt-mark unhold python3-charset-normalizer apt-get upgrade apt-mark unhold mailman3 apt-get install mailman3 Configure database for mailman3 with dbconfig-common? [yes/no] no Add the HyperKitty configuration to mailman.cfg? [yes/no] no that worked, and snapshoted: poola/balug1213@2026-01-31T13:25:11Z.balug.debian12.many.packages.upgraded.not.broken.yet apt-mark unhold sqlite3 apt-get upgrade, that worked, and snapshoted: poola/balug1213@2026-01-31T13:34:29Z.balug.debian12.many.packages.upgraded.not.broken.yet apt-mark unhold uwsgi-core uwsgi-plugin-python3 apt-get upgrade, broken, rollback to: poola/balug1213@2026-01-31T13:34:29Z.balug.debian12.many.packages.upgraded.not.broken.yet apt-mark unhold mailman3-web apt-get upgrade (broken) import cgi ModuleNotFoundError: No module named 'cgi' apt-get install python3-legacy-cgi (still broken) apt-mark unhold $(apt-mark showhold) && apt-get upgrade (still broken) apt-get full-upgrade (ran out of space on /usr before it completed) dpkg --configure -a apt-get full-upgrade apt-get update apt-get full-upgrade And, somewhat surprising, the web pages seem more-or-less functional, but their formatting is all messed up, snapshot: poola/balug1213@2026-02-01T00:59:17Z.balug.debian12.upgraded.to.13.web.seems.functional.but.formatting.all.messed.up apt-mark manual ntpsec-ntpdate ntpsec-ntpdig apt-get autoremove apt-get remove p7zip apt-get remove p7zip-full can't remove transitional package python3-tz because of reverse dependencies updated /etc/letsencrypt contents rm /etc/apache2/mods-available/ssl.conf.dpkg-old /etc/ca-certificates.conf.dpkg-old /etc/default/grub.ucf-dist /etc/dokuwiki/local.php.ucf-dist /etc/mailman3/mailman.cfg.ucf-dist apt-get install apt-file apt-file update purged a bunch of removed packages (to clear out their configurations) apt-get purge 7zip web functionality - more than semi-broken - hyperkitty archives also broken, I should've more thoroughly checked earlier. rollback to: poola/balug1213@2026-01-31T13:34:29Z.balug.debian12.many.packages.upgraded.not.broken.yet postorius, hyperkitty, django appear working updated /etc/letsencrypt contents apt-mark unhold $(apt-mark showhold) grow /usr /var /home so each <=85% full apt-get update apt-get upgrade all broken: "Internal Server Error": postorius, hyperkitty, django rollback to: poola/balug1213@2026-01-31T13:34:29Z.balug.debian12.many.packages.upgraded.not.broken.yet postorius, hyperkitty, django appear working updated /etc/letsencrypt contents apt-get update, snapshot: poola/balug1213@2026-02-01T05:08:05Z.balug.debian12.many.packages.upgraded.not.broken.yet apt-mark unhold python3-django-hyperkitty apt-get upgrade upgrading python3-django-hyperkitty breaks it: 500 Internal Server Error postorius, hyperkitty, django rollback to: poola/balug1213@2026-02-01T05:08:05Z.balug.debian12.many.packages.upgraded.not.broken.yet apt-mark unhold mailman3-full apt-get upgrade, so far so good, snapshot: poola/balug1213@2026-02-01T05:36:56Z.balug.debian12.many.packages.upgraded.not.broken.yet apt-mark unhold python3-robot-detection apt-get upgrade, so far so good apt-mark unhold python3-django-mailman3 apt-mark unhold uwsgi-core uwsgi-plugin-python3 apt-get upgrade all broken: "Internal Server Error": postorius, hyperkitty, django rolled back to: poola/balug1213@2026-02-01T05:36:56Z.balug.debian12.many.packages.upgraded.not.broken.yet apt-mark unhold python3-django-mailman3 python3-robot-detection apt-get upgrade, so far so good apt-mark unhold python3-mailman-hyperkitty apt-get upgrade, so far so good, snapshot: poola/balug1213@2026-02-01T06:07:43Z.balug.debian12.many.packages.upgraded.not.broken.yet Have these packages held: mailman3-web python3-django-compressor python3-django-hyperkitty python3-mistune uwsgi-core uwsgi-plugin-python3 apt-get full-upgrade all broken: "Internal Server Error": postorius, hyperkitty, django rollback: poola/balug1213@2026-02-01T06:07:43Z.balug.debian12.many.packages.upgraded.not.broken.yet So, to try and do full-upgrade more step-wise without breaking things, will attempt hold on many packages it would otherwise upgrade and/or remove. # echo $(apt-mark showhold) mailman3-web python3-django-compressor python3-django-hyperkitty python3-mistune uwsgi-core uwsgi-plugin-python3 # apt-mark hold $( (cd / && umask 022 && apt-get -s full-upgrade) 2>&1 | sed -ne '/^The following packages will be REMOVED:$/{:r;n;/^ /{p;br}};/^The following packages will be upgraded:$/{:u;n;/^ /{p;bu}};') That holds all that would be upgraded/removed, apt-get -s full-upgrade shows nothing to do apt-mark unhold $(apt-mark showhold | grep '^lib') apt-get -s full-upgrade shows: 30 upgraded, 50 newly installed, 28 to remove and 163 not upgraded let's do: apt-get full-upgrade, so far so good, snapshot: poola/balug1213@2026-02-02T03:14:14Z.balug.debian12.many.packages.upgraded.not.broken.yet apt-mark unhold $(apt-mark showhold | grep -E -e '^(bind9|linux-image)') apt-get full-upgrade, so far so good, snapshot: poola/balug1213@2026-02-02T03:33:16Z.balug.debian12.many.packages.upgraded.not.broken.yet apt-get purge linux-image-6.1.0-42-amd64 apt-mark unhold $(apt-mark showhold | grep -E -e '^(gnupg|gpg|grub|reiserfsprogs)') apt-get full-upgrade, so far so good, snapshot: poola/balug1213@2026-02-02T03:55:56Z.balug.debian12.many.packages.upgraded.not.broken.yet added [www.]berkeleylug.com domains to /etc/hosts, wiki & wordpress login okay, added to checks apt-mark unhold $(apt-mark showhold | grep -E -e '^(dokuwiki|php|wordpress)') apt-get full-upgrade, so far so good, snapshot: poola/balug1213@2026-02-02T04:36:57Z.balug.debian12.many.packages.upgraded.not.broken.yet apt-mark unhold $(apt-mark showhold | grep -E -e '^(file|firefox|fonts|g++|gcc|binutils|cpp|e2fsprogs)') apt-get full-upgrade, so far so good, snapshot: poola/balug1213@2026-02-02T04:53:57Z.balug.debian12.many.packages.upgraded.not.broken.yet apt-mark unhold $(apt-mark showhold | grep -E -v -e '^(python3|uwsgi)') apt-get full-upgrade, mailman3-web configuration broken, rollback: poola/balug1213@2026-02-02T04:53:57Z.balug.debian12.many.packages.upgraded.not.broken.yet VM balug13 (based off of current snapshot of prod VM balug): added [www.]berkeleylug.com domains to /etc/hosts, & snapshot: poola/balug13@2026-02-02T05:47:19Z.balug.13.well.after.upgrade.plus.reconfigured.for.non-conflicting.network.use VM balug1213: apt-mark unhold $(apt-mark showhold | grep -E -v -e '^(exim4|mailman3|python3|uwsgi)') apt-get full-upgrade, so far so good, "only" these packages held: # echo $(apt-mark showhold) | fold -s -w 70 exim4-base exim4-daemon-heavy mailman3-web python3-acme python3-arrow python3-certbot python3-coreapi python3-cryptography python3-django-allauth python3-django-compressor python3-django-hyperkitty python3-dnspython python3-elasticsearch python3-jsonschema python3-markdown-it python3-mistune python3-openid python3-openssl python3-passlib python3-pyinotify python3-pythran python3-readme-renderer python3-tz python3-wheel-whl uwsgi-core uwsgi-plugin-python3 # z="$(Z)" && dpkg -l | xz -9 > balug1213.dpkg_-l."$z".xz snapshot: poola/balug1213@2026-02-02T13:13:58Z.balug.debian12.many.packages.upgraded.not.broken.yet VM balug13: apt proxy configuration apt-get update apt-get --allow-downgrades install certbot=2.1.0-4 exim4-base=4.96-15+deb12u7 exim4-daemon-heavy=4.96-15+deb12u7 mailman3-web=0+20200530-2.1 python3-acme=2.1.0-1 python3-arrow=1.2.3-1 python3-certbot=2.1.0-4 python3-coreapi=2.3.3-6 python3-cryptography=38.0.4-3+deb12u1 python3-django=3:3.2.19-1+deb12u1 python3-django-allauth=0.51.0-1 python3-django-compressor=4.0-1 python3-django-extensions=3.2.1-2 python3-django-hyperkitty=1.3.7-1 python3-django-mailman3=1.3.9-1 python3-django-postorius=1.3.8-3 python3-djangorestframework=3.14.0-2+deb12u1 python3-dnspython=2.3.0-1 python3-elasticsearch=7.17.6-1 python3-jsonschema=4.10.3-1 python3-markdown-it=2.1.0-5 python3-mistune=2.0.4-1 python3-openid=3.2.0-2 python3-openssl=23.0.0-1 python3-passlib=1.7.4-3 python3-pyinotify=0.9.6-2 python3-pythran=0.11.0+ds-7 python3-readme-renderer=37.3-2 python3-wheel-whl=0.38.4-2 uwsgi-core=2.0.21-5.1 uwsgi-plugin-python3=2.0.21-5.1 Yay! All web interface stuff appears to be working!, snapshot: poola/balug13@2026-02-02T13:49:40Z.balug.13.well.after.upgrade.plus.reconfigured.for.non-conflicting.network.use.appears.to.be.working Now attempt likewise to production balug VM: VM balug (production!): apt-get update apt-get --allow-downgrades install certbot=2.1.0-4 exim4-base=4.96-15+deb12u7 exim4-daemon-heavy=4.96-15+deb12u7 mailman3-web=0+20200530-2.1 python3-acme=2.1.0-1 python3-arrow=1.2.3-1 python3-certbot=2.1.0-4 python3-coreapi=2.3.3-6 python3-cryptography=38.0.4-3+deb12u1 python3-django=3:3.2.19-1+deb12u1 python3-django-allauth=0.51.0-1 python3-django-compressor=4.0-1 python3-django-extensions=3.2.1-2 python3-django-hyperkitty=1.3.7-1 python3-django-mailman3=1.3.9-1 python3-django-postorius=1.3.8-3 python3-djangorestframework=3.14.0-2+deb12u1 python3-dnspython=2.3.0-1 python3-elasticsearch=7.17.6-1 python3-jsonschema=4.10.3-1 python3-markdown-it=2.1.0-5 python3-mistune=2.0.4-1 python3-openid=3.2.0-2 python3-openssl=23.0.0-1 python3-passlib=1.7.4-3 python3-pyinotify=0.9.6-2 python3-pythran=0.11.0+ds-7 python3-readme-renderer=37.3-2 python3-wheel-whl=0.38.4-2 uwsgi-core=2.0.21-5.1 uwsgi-plugin-python3=2.0.21-5.1 Drats! Not working, mailman3 related web pages giving: Server error An error occurred while processing your request. compared all files of type ordinary file beneath /etc between balug13 and balug, no unexpected differences tried some URLs with curl, seemed to work! cleared site cookies, then it works! Could that be what it was all along? No, not entirely, that wouldn't explain the other issues earlier (e.g. authenticated but formatting all messed up). Issue with exim4, fails with mailman3 but seems to be otherwise working: R=lman3_router T=mailman3_transport defer (-32): failed to lookup IP address for localhost Fixed by adding line: gethostbyname to file: /etc/exim4/conf.d/transport/55_mm3_transport between the lines: allow_localhost hosts = localhost and restart of exim4, process restarted at: 2026-02-09T20:53:03Z After that, all the mailman3 web and mail interface appears working again. VM balug13: tried apt-get update and apt-get full-upgrade per routine security/bug fixes, notably also included [SECURITY] [DSA 6136-1] python-django security update and that then seriously broke thing(s), notably including at least mailman3-web zfs rollback poola/balug13@2026-02-02T13:49:40Z.balug.13.well.after.upgrade.plus.reconfigured.for.non-conflicting.network.use.appears.to.be.working Both this VM and prod VM have no packages held, should probably adjust that to make things less fragile as warranted and appropriate, most notably so, e.g. routine: apt-get update && apt-get full-upgrade doesn't break things. apt-get autopurge Let's put a hold on many of the packages it wants to upgrade: echo $(apt-mark showhold) exim4-base exim4-daemon-heavy mailman3-web postgresql-client-17 python3-arrow python3-coreapi python3-django python3-django-allauth python3-django-compressor python3-django-extensions python3-django-hyperkitty python3-django-mailman3 python3-django-postorius python3-djangorestframework python3-dnspython python3-elasticsearch python3-jsonschema python3-markdown-it python3-mistune python3-openid python3-passlib python3-pyinotify python3-pythran python3-readme-renderer python3-wheel-whl uwsgi-core uwsgi-plugin-python3 apt-get full-upgrade, still good, snapshot: poola/balug13@2026-02-16T04:28:04Z.balug.13.well.after.upgrade.plus.reconfigured.for.non-conflicting.network.use.appears.to.be.working apt-mark unhold postgresql-client-17 apt-get full-upgrade, still good, snapshot: poola/balug13@2026-02-16T04:44:10Z.balug.13.well.after.upgrade.plus.reconfigured.for.non-conflicting.network.use.appears.to.be.working purged no longer needed kernel, snapshot: poola/balug13@2026-02-16T04:56:30Z.balug.13.well.after.upgrade.plus.reconfigured.for.non-conflicting.network.use.appears.to.be.working VM balug (production!): safety sake / accident prevention, let's likewise apply holds: apt-mark hold exim4-base exim4-daemon-heavy mailman3-web python3-arrow python3-coreapi python3-django python3-django-allauth python3-django-compressor python3-django-extensions python3-django-hyperkitty python3-django-mailman3 python3-django-postorius python3-djangorestframework python3-dnspython python3-elasticsearch python3-jsonschema python3-markdown-it python3-mistune python3-openid python3-passlib python3-pyinotify python3-pythran python3-readme-renderer python3-wheel-whl uwsgi-core uwsgi-plugin-python3 so we can generally do: apt-get update && apt-get full-upgrade without breaking things. Probably better to pin those to bookworm(+security(& udpates)) & unhold, so those held back can follow (bookworm) security and bug fixes, but holds will do for now. VM balug13: apt-mark unhold python3-arrow python3-coreapi python3-dnspython python3-mistune python3-openid python3-passlib python3-pyinotify python3-readme-renderer python3-wheel-whl apt-get full-upgrade, 500 Internal Server Error on all the mailman3 web stuff, rollback: poola/balug13@2026-02-16T04:56:30Z.balug.13.well.after.upgrade.plus.reconfigured.for.non-conflicting.network.use.appears.to.be.working apt-mark unhold python3-arrow apt-get full-upgrade, ok, snapshot: poola/balug13@2026-02-16T05:45:37Z.balug.13.well.after.upgrade.plus.reconfigured.for.non-conflicting.network.use.appears.to.be.working apt-mark unhold python3-coreapi apt-get full-upgrade, ok, snapshot: poola/balug13@2026-02-16T05:52:42Z.balug.13.well.after.upgrade.plus.reconfigured.for.non-conflicting.network.use.appears.to.be.working apt-mark unhold python3-dnspython apt-get full-upgrade, ok, snapshot: poola/balug13@2026-02-16T06:01:34Z.balug.13.well.after.upgrade.plus.reconfigured.for.non-conflicting.network.use.appears.to.be.working apt-mark unhold python3-elasticsearch apt-get full-upgrade, ok, snapshot: poola/balug13@2026-02-16T06:08:39Z.balug.13.well.after.upgrade.plus.reconfigured.for.non-conflicting.network.use.appears.to.be.working apt-mark unhold python3-readme-renderer python3-passlib python3-pyinotify apt-get full-upgrade, ok, snapshot: poola/balug13@2026-02-16T06:21:49Z.balug.13.well.after.upgrade.plus.reconfigured.for.non-conflicting.network.use.appears.to.be.working apt-mark unhold python3-markdown-it python3-openid apt-get full-upgrade, ok, snapshot: poola/balug13@2026-02-16T06:28:09Z.balug.13.well.after.upgrade.plus.reconfigured.for.non-conflicting.network.use.appears.to.be.working apt-mark unhold python3-mistune python3-jsonschema apt-get full-upgrade, 500 Internal Server Error on all the mailman3 web stuff, rollback: poola/balug13@2026-02-16T06:28:09Z.balug.13.well.after.upgrade.plus.reconfigured.for.non-conflicting.network.use.appears.to.be.working apt-mark unhold python3-pythran apt-get full-upgrade, ok, snapshot: poola/balug13@2026-02-16T06:43:39Z.balug.13.well.after.upgrade.plus.reconfigured.for.non-conflicting.network.use.appears.to.be.working apt-mark unhold python3-wheel-whl apt-get full-upgrade, ok, snapshot: poola/balug13@2026-02-16T06:51:37Z.balug.13.well.after.upgrade.plus.reconfigured.for.non-conflicting.network.use.appears.to.be.working apt-mark unhold python3-mistune apt-get full-upgrade, 500 Internal Server Error on all the mailman3 web stuff, rollback: poola/balug13@2026-02-16T06:51:37Z.balug.13.well.after.upgrade.plus.reconfigured.for.non-conflicting.network.use.appears.to.be.working apt-mark unhold python3-jsonschema apt-get full-upgrade, ok, snapshot: poola/balug13@2026-02-16T07:04:15Z.OK updated/added apt pinning: # more /etc/apt/pref*.d/* | cat :::::::::::::: /etc/apt/preferences.d/97suites :::::::::::::: Explanation: lower priority for bookworm Package: * Pin: release n=bookworm Pin-Priority: 300
Package: * Pin: release n=bookworm-updates Pin-Priority: 300
Package: * Pin: release n=bookworm-security Pin-Priority: 300 :::::::::::::: /etc/apt/preferences.d/98packages :::::::::::::: Explanation: higher priority for certain packages from bookworm Package: exim4-base exim4-daemon-heavy mailman3-web python3-django python3-django-allauth python3-django-compressor python3-django-extensions python3-django-hyperkitty python3-django-mailman3 python3-django-postorius python3-djangorestframework python3-mistune uwsgi-core uwsgi-plugin-python3 Pin: release n=bookworm Pin-Priority: 995
Package: exim4-base exim4-daemon-heavy mailman3-web python3-django python3-django-allauth python3-django-compressor python3-django-extensions python3-django-hyperkitty python3-django-mailman3 python3-django-postorius python3-djangorestframework python3-mistune uwsgi-core uwsgi-plugin-python3 Pin: release n=bookworm-updates Pin-Priority: 995
Package: exim4-base exim4-daemon-heavy mailman3-web python3-django python3-django-allauth python3-django-compressor python3-django-extensions python3-django-hyperkitty python3-django-mailman3 python3-django-postorius python3-djangorestframework python3-mistune uwsgi-core uwsgi-plugin-python3 Pin: release n=bookworm-security Pin-Priority: 995 # released all package holds: apt-mark unhold $(apt-mark showhold) apt-get update && apt-get full-upgrade, ok, snapshot: poola/balug13@2026-02-16T08:37:22Z.OK VM balug (production!): post Debian 12 --> 13 upgrade issue: IPv6 addresses mostly not properly initializing at boot, booting to single user / maintenance mode, then ipdown -a, ifup -a brings them up. VM balug13: reconfigure network to more closely resemble prod solution / work-around to IPv6 issue, get rid of systemd, use sysvinit, ok, snapshot: poola/balug13@2026-02-16T17:43:18Z.OK VM balug (production!): apt-get --allow-remove-essential install sysvinit-core systemd-sysv- apt-get purge openipmi radvd sane-utils apt-get autopurge and now IPv6 configured properly upon boot also noticed (I?, or possibly upgrades?) apparently disabled rsync server long ago, TODO should properly reconfigure and enable that (notably so others can efficiently rsync public list archives) TODO also still, should figure out (first on test VM) minimizing packages held back by pinning (without breaking prod), and better / more "permanent" fix for that (main support for Debian 12 ends 2026-06-10, after that, drops to LTS through 2028-06-30 (then ELTS through 2023-06-30)) TODO despite exim4's configuration to only send on specific IP addresses, it will still sometimes use others (e.g. if network routing permits, such as when desired sending IP address that's needed isn't available), find a fix for that (or implement work-around - e.g. firewall it away from doing so, if need be)