So, modest update. Yes, it's very much issue on server (thus far doesn't appear to be issue of Comcast Business' doing). Also, have tried it with server on VM, and haven't been able to replicate the issue there. So, possible it might just be some funkiness in configuration I have to work out - there are some other bits too, e.g. systemd doesn't quite cleanly start it - it takes a while, then complains that the startup attempt failed with timeout ... but upon checking it's actually properly up and running. Also have non-trivial differences between my actual production setup and what I'd tested on VM, e.g. production is using actual delegated zones, also has Dynamic DNS (DDNS), also configured using chroot, etc. So, not trivial to do apples-to-apples comparison, though did at least compare with exact same versions of Debian and Debian's BIND. But even if I find what makes the difference configuration-wise or the like, doesn't mean it's necessarily a reasonably feasible fix. E.g. the bug might only show up with chroot or DDNS or with actual domains (where client is a secondary). Anyway, haven't given up on it, but not super high on the priorities to nail that one ... and might not be particularly feasibly fixable at current anyway. And yeah, if you saw/see notifies from one of my IPs for zone example.com - yeah, that'd be from some of the VM testing (from both stable, and possibly also unstable+experimental).
And if you're curious and/or bored and interested in more, I was earlier testing on zone tmp.mpaoli.net. I later switched to instead using dnssec-test.mpaoli.net Some interesting test/history bits of those may be seen on https://dnsviz.net/ And for some bit I was even using tmp2.mpaoli.net, but don't know that I got as far as checking any of that zone on https://dnsviz.net/ And yes, https://wiki.debian.org/DNSSEC%20Howto%20for%20BIND%209.9+ is even more nicely updated to current and quite thoroughly tested at this point. And keys are still being rolled on mpaoli.net (most notably KSK, but also some ZSK), so for bit longer on will see additional cruft there (because TTLs and to not break thing), but the remaining cleanup will happen on that relatively soon (most notably once last relevant TTLs have passed). And yes, one can see key steps along that process via: https://dnsviz.net/d/mpaoli.net/dnssec/ See, e.g.: https://dnsviz.net/d/mpaoli.net/ZmORYA/dnssec/ for bit more interesting along the way. And alas, if I'd gotten it fully optimal along the way (notably with lifetime policy in dnssec-policy statement), I could've avoided creating and having used and in place a whole additional ZSK that wasn't even needed and is in most all regards quite redundant. In any case, all will be completed with cleanup fairly soon on that. Yeah, rotating KSKs is non-trivial, but advances in both RFCs and BIND are continuing to make that easier.
Anyway, I agree, it ought not be doing the excessive notify - annoying that. But, no great harm, and I may or may not mange to get that issue fixed anytime soon.
On Sat, Jun 8, 2024 at 4:31 PM Michael Paoli michael.paoli@berkeley.edu wrote:
So, haven't 100% gotten to the bottom of the excess NOTIFY, but it is apparently present in ISC BIND 9.18.24 and some other versions, and Debian bind9 1:9.18.24-1 which is the current version for the current stable Debian 12 bookworm. Also thinking that bug, for Debian stable, probably doesn't make it to >=important priority (probably just priority normal), so probably won't get fixed in the current stable. Thus far, this is what I've run across that seems most informative about it: https://gitlab.isc.org/isc-projects/bind9/-/issues/3242 So, workarounds? I'm not going to be upgrading that host from stable for quite some time (most likely year(s)), and other alternatives to fixing it on the server side is probably more additional overhead than I'm likely to want to bother with - so probably not going there. On the client side, can reconfigure logging to not log those bits, or possibly ignore them (e.g. by filter) in whatever notifications/reporting one is doing. So, yes, it is definitely coming from the nameserver itself, and all indications are it's a bug. I did see some chatter somewhere about it possibly being a but only if client doesn't respond in some way(s)? But that's not what I see in my testing on clients that do respond as expected, so I don't think that's it at all - at least in my case.
Oh, as for possibly adjusting the logging on [ns1.]linuxmafia.com, I notice it's got bind9 version 1:9.7.1.dfsg.P2-2 installed, and yes, also has bind9-doc version 1:9.4.2-4, but that's different version. However, it does have, in /var/cache/apt/archives/ bind9-doc_9.7.1.dfsg.P2-2_all.deb So, that would be the corresponding documentation package, which could be installed, or the files simply extracted from that and consulted as may be desired. Anyway, within that, around: doc/bind9-doc/arm/Bv9ARM.ch06.html#id2575303 It has information about configuring logging, what the defaults are and the category notify in logging configuration that one can use to adjust what one does/doesn't want done regarding logging of notify activity.
And, along the way, I've also rather well updated: Debian wiki: DNSSEC Howto for BIND 9.9+: https://wiki.debian.org/DNSSEC%20Howto%20for%20BIND%209.9+ And also well whipping mpaoli.net (and nameserver in general) configuration into shape for the current Debian stable.
If I find anything else particularly noteworthy about the notify situation, I'll update.
On Tue, Jun 4, 2024 at 4:21 PM Michael Paoli michael.paoli@berkeley.edu wrote:
There's two different issues (well, three, but 3rd is minor), not to be confused:
The second - unrelated - issue. I've got excess notifies being sent from my server(s). Now, not at all unusual that sometimes there may be bursts of activity and updates and such, at least some of that is to be expected. What's quite unexpected and ought not be happening, but is, is I've got server sending frequent (e.g. like every 90 seconds-ish - or so) notifies when basic zone DNS data hasn't changed - same SOA serial, etc. So, something's still amis with that which I need to correct (doesn't really break any functionalities ... but rather wasteful all those excess notifies being sent, also tends to cause needless traffic on both sending side, and receiving servers checking and finding they've got nothing to do because they're already current). And, as I'd earlier mentioned, I suspect from some major upgrades I did a few weeks ago ... well, that was off-list, so, let me include fair bit more of it here (but not the whole thing, it gets very long, especially with all the earlier referenced):
---------- Forwarded message --------- From: Michael Paoli michael.paoli@berkeley.edu Date: Mon, Jun 3, 2024 at 11:19 PM Subject: Re: [BALUG-Admin] (forw) linuxmafia.com 2024-06-03 15:02 System Events To: Al aw009@sunnyside.com Cc: Rick Moen rick@linuxmafia.com
Yeah, ... something's still not quite right there. Did upgrade on that host/server a couple weekends ago Debian: buster 10 --> bullseye 11 --> bookworm 12 $ TZ=GMT0 stat -c '%z %Z %n' /etc/debian_version 2024-05-14 07:37:15.000000000 +0000 1715672235 /etc/debian_version $ Some of ye olde(er) DNS configuration options, while still (marginally?) functional, are quite deprecated, and I'm getting warnings on that and one other minor issue ... anyway, need to do some cleanup/reconfig with DNS server on that host. Should all be fine after that. Functional in the meantime, but ... not yet optimal ... clearly.
On Mon, Jun 3, 2024 at 10:43 PM Al aw009@sunnyside.com wrote:
I'm probably not reading those logs right but part of it looked like mpaoli was being signed every minute? Only partial logs, not enough to be sure. 7:28, 7:29. Just a thought...
On June 3, 2024 20:15:06 Michael Paoli via BALUG-Admin balug-admin@lists.balug.org wrote:
Well, under some circumstances, I'd expect a relative flury, or "bursts" of notify activity. E.g. I do TLS(/"SSL") certs via LetsEncrypt.org (LE), using DNS verification. Some/many of those certs will have multiple Subject Alternative Name (SAN) domains, and there may be fair number of such certs. So, adding the relevant records for verification, and then removing them after having obtained the issued cert(s) - that can be a fair burst of activity. But those generally wouldn't persist over a long period of time. E.g. every 90 seconds over many hours or days or more ... then something else is going on. So ... let me dig (pun not quite intended but apt) a bit more and peek what's currently goin' on there ...
...
----- End forwarded message -----
I suspect it has to do with some changes in how DNSSEC is configured, most notably a configuration syntax I'm using is (now) quite deprecated, and it may be thus defaulting to some rather undesirable behavior(s) which may include triggering the excess notifies. Anyway, want to test it carefully and not break things, that's why I'm not willy-nilly throwing things at it. Goals being, to not only fix it, but if feasible without causing further breakage along the way, and also not disabling DNSSEC at any point (shouldn't need to) ... and while I'm at it, also reviewing and checking/updating earlier documentation (much of which I wrote): Debian wiki: DNSSEC Howto for BIND 9.9+: https://wiki.debian.org/DNSSEC%20Howto%20for%20BIND%209.9+ And, along the way, I've also got temporary zone and subdomain I'm fiddling with: tmp.mpaoli.net. to test things, etc. Anyway, it's still a work in progress to get that all squared away (get issue fixed, not further break things, and update relevant documentation). Anyway, as far as I'm aware, the excess notifies being sent don't have anything to do with the Comcast issues.
On Tue, Jun 4, 2024 at 3:12 PM Rick Moen rick@linuxmafia.com wrote:
More of same, involving back-to-back NOTIFY traffic about mpaoli.net, allegedly originating from master nameserver 96.86.170.226 . Michael, I greatly doubt your nameserver is doing this. mpaoli.net's zonefile is pretty unchanging. I suspect Comcast middleman fsckery. Thoughts?
----- Forwarded message from logcheck system account logcheck@linuxmafia.com -----
Date: Tue, 04 Jun 2024 02:02:01 -0700 From: logcheck system account logcheck@linuxmafia.com To: root@linuxmafia.com Subject: linuxmafia.com 2024-06-04 02:02 System Events
System Events
Jun 4 01:47:15 linuxmafia named[15622]: zone mpaoli.net/IN: Transfer started. Jun 4 01:47:15 linuxmafia named[15622]: transfer of 'mpaoli.net/IN' from 96.86.170.226#53: connected using 96.95.217.99#52012 Jun 4 01:47:15 linuxmafia named[15622]: zone mpaoli.net/IN: transferred serial 1717468800 Jun 4 01:47:15 linuxmafia named[15622]: transfer of 'mpaoli.net/IN' from 96.86.170.226#53: Transfer completed: 1 messages, 6 records, 546 bytes, 0.064 secs (8531 bytes/sec) Jun 4 02:00:09 linuxmafia named[15622]: zone mpaoli.net/IN: Transfer started. Jun 4 02:00:09 linuxmafia named[15622]: transfer of 'mpaoli.net/IN' from 96.86.170.226#53: connected using 96.95.217.99#57758 Jun 4 02:00:09 linuxmafia named[15622]: zone mpaoli.net/IN: transferred serial 1717491608 Jun 4 02:00:09 linuxmafia named[15622]: transfer of 'mpaoli.net/IN' from 96.86.170.226#53: Transfer completed: 1 messages, 15 records, 1266 bytes, 0.072 secs (17583 bytes/sec)