[BALUG-Admin] excess NOTIFY (about ever 90s) ISC BIND 9.18.24 ...

Mon Jun 10 07:49:38 UTC 2024

So, modest update.
Yes, it's very much issue on server (thus far doesn't appear to be
issue of Comcast Business' doing).
Also, have tried it with server on VM, and haven't been able to
replicate the issue there.
So, possible it might just be some funkiness in configuration I have
to work out - there
are some other bits too, e.g. systemd doesn't quite cleanly start it -
it takes a while, then
complains that the startup attempt failed with timeout ... but upon
checking it's actually
properly up and running.  Also have non-trivial differences between my actual
production setup and what I'd tested on VM, e.g. production is using
actual delegated
zones, also has Dynamic DNS (DDNS), also configured using chroot, etc.  So, not
trivial to do apples-to-apples comparison, though did at least compare
with exact same
versions of Debian and Debian's BIND.  But even if I find what makes
the difference
configuration-wise or the like, doesn't mean it's necessarily a
reasonably feasible fix.
E.g. the bug might only show up with chroot or DDNS or with actual
domains (where
client is a secondary).  Anyway, haven't given up on it, but not super
high on the
priorities to nail that one ... and might not be particularly feasibly
fixable at current
anyway.
And yeah, if you saw/see notifies from one of my IPs for zone
example.com - yeah,
that'd be from some of the VM testing (from both stable, and possibly
also unstable+experimental).

And if you're curious and/or bored and interested in more, I was earlier testing
on zone tmp.mpaoli.net.  I later switched to instead using
dnssec-test.mpaoli.net
Some interesting test/history bits of those may be seen on
https://dnsviz.net/
And for some bit I was even using tmp2.mpaoli.net, but don't know that
I got as far
as checking any of that zone on https://dnsviz.net/
And yes,
https://wiki.debian.org/DNSSEC%20Howto%20for%20BIND%209.9+
is even more nicely updated to current and quite thoroughly tested at
this point.
And keys are still being rolled on mpaoli.net (most notably KSK, but
also some ZSK),
so for bit longer on will see additional cruft there (because TTLs and
to not break thing),
but the remaining cleanup will happen on that relatively soon (most
notably once last relevant TTLs
have passed).  And yes, one can see key steps along that process via:
https://dnsviz.net/d/mpaoli.net/dnssec/
See, e.g.:
https://dnsviz.net/d/mpaoli.net/ZmORYA/dnssec/
for bit more interesting along the way.
And alas, if I'd gotten it fully optimal along the way (notably
with lifetime policy in dnssec-policy statement), I could've avoided
creating and having used and in place a whole additional ZSK that
wasn't even needed and is in most all regards quite redundant.
In any case, all will be completed with cleanup fairly soon on that.
Yeah, rotating KSKs is non-trivial, but advances in both RFCs and
BIND are continuing to make that easier.

Anyway, I agree, it ought not be doing the excessive notify - annoying that.
But, no great harm, and I may or may not mange to get that issue fixed
anytime soon.

On Sat, Jun 8, 2024 at 4:31 PM Michael Paoli <michael.paoli@berkeley.edu> wrote:
>
> So, haven't 100% gotten to the bottom of the excess NOTIFY,
> but it is apparently present in ISC BIND 9.18.24 and some other versions,
> and Debian bind9 1:9.18.24-1 which is the current version for the current stable
> Debian 12 bookworm.  Also thinking that bug, for Debian stable, probably
> doesn't make it to >=important priority (probably just priority normal),
> so probably won't get fixed in the current stable.
> Thus far, this is what I've run across that seems most informative about it:
> https://gitlab.isc.org/isc-projects/bind9/-/issues/3242
> So, workarounds?  I'm not going to be upgrading that host from stable
> for quite some time (most likely year(s)), and other alternatives to fixing it
> on the server side is probably more additional overhead than I'm likely to
> want to bother with - so probably not going there.
> On the client side, can reconfigure logging to not log those bits,
> or possibly ignore them (e.g. by filter) in whatever
> notifications/reporting one is doing.
> So, yes, it is definitely coming from the nameserver itself, and all indications
> are it's a bug.  I did see some chatter somewhere about it possibly being a
> but only if client doesn't respond in some way(s)?  But that's not what I see
> in my testing on clients that do respond as expected, so I don't think that's
> it at all - at least in my case.
>
> Oh, as for possibly adjusting the logging on [ns1.]linuxmafia.com,
> I notice it's got
> bind9 version 1:9.7.1.dfsg.P2-2
> installed, and yes, also has
> bind9-doc version 1:9.4.2-4,
> but that's different version.
> However, it does have, in /var/cache/apt/archives/
> bind9-doc_9.7.1.dfsg.P2-2_all.deb
> So, that would be the corresponding documentation package,
> which could be installed, or the files simply extracted from that and
> consulted as may be desired.  Anyway, within that, around:
> doc/bind9-doc/arm/Bv9ARM.ch06.html#id2575303
> It has information about configuring logging, what the defaults are
> and the category
> notify
> in logging configuration that one can use to adjust what one
> does/doesn't want done regarding logging of
> notify
> activity.
>
> And, along the way, I've also rather well updated:
> Debian wiki: DNSSEC Howto for BIND 9.9+:
> https://wiki.debian.org/DNSSEC%20Howto%20for%20BIND%209.9+
> And also well whipping mpaoli.net (and nameserver in general) configuration
> into shape for the current Debian stable.
>
> If I find anything else particularly noteworthy about the notify situation,
> I'll update.
>
> On Tue, Jun 4, 2024 at 4:21 PM Michael Paoli <michael.paoli@berkeley.edu> wrote:
> >
> > There's two different issues (well, three, but 3rd is minor), not to
> > be confused:
> >
> > The second - unrelated - issue.  I've got excess notifies being sent
> > from my server(s).
> > Now, not at all unusual that sometimes there may be bursts of activity
> > and updates and such,
> > at least some of that is to be expected.
> > What's quite unexpected and ought not be happening, but is,
> > is I've got server sending frequent (e.g. like every 90 seconds-ish -
> > or so) notifies
> > when basic zone DNS data hasn't changed - same SOA serial, etc.
> > So, something's still amis with that which I need to correct (doesn't
> > really break any
> > functionalities ... but rather wasteful all those excess notifies being sent,
> > also tends to cause needless traffic on both sending side,
> > and receiving servers checking and finding they've got nothing
> > to do because they're already current).
> > And, as I'd earlier mentioned, I suspect from some major upgrades I did
> > a few weeks ago ... well, that was off-list, so, let me include fair
> > bit more of it here
> > (but not the whole thing, it gets very long, especially with all the
> > earlier referenced):
> > -----
> > ---------- Forwarded message ---------
> > From: Michael Paoli <michael.paoli@berkeley.edu>
> > Date: Mon, Jun 3, 2024 at 11:19 PM
> > Subject: Re: [BALUG-Admin] (forw) linuxmafia.com 2024-06-03 15:02 System Events
> > To: Al <aw009@sunnyside.com>
> > Cc: Rick Moen <rick@linuxmafia.com>
> >
> > Yeah, ... something's still not quite right there.
> > Did upgrade on that host/server a couple weekends ago
> > Debian: buster 10 --> bullseye 11 --> bookworm 12
> > $ TZ=GMT0 stat -c '%z %Z %n' /etc/debian_version
> > 2024-05-14 07:37:15.000000000 +0000 1715672235 /etc/debian_version
> > $
> > Some of ye olde(er) DNS configuration options, while still
> > (marginally?) functional, are quite deprecated, and I'm getting
> > warnings on that and one other minor issue ... anyway, need to do
> > some cleanup/reconfig with DNS server on that host.
> > Should all be fine after that.  Functional in the meantime, but ...
> > not yet optimal ... clearly.
> >
> > On Mon, Jun 3, 2024 at 10:43 PM Al <aw009@sunnyside.com> wrote:
> > >
> > > I'm probably not reading those logs right but part of it looked like mpaoli was being signed every minute?  Only partial logs, not enough to be sure.  7:28, 7:29.
> > > Just a thought...
> > >
> > > On June 3, 2024 20:15:06 Michael Paoli via BALUG-Admin <balug-admin@lists.balug.org> wrote:
> > >
> > >> Well, under some circumstances, I'd expect a relative flury, or
> > >> "bursts" of notify
> > >> activity.  E.g. I do TLS(/"SSL") certs via LetsEncrypt.org (LE), using DNS
> > >> verification.  Some/many of those certs will have multiple
> > >> Subject Alternative Name (SAN) domains, and there may be fair number
> > >> of such certs.  So, adding the relevant records for verification, and then
> > >> removing them after having obtained the issued cert(s) - that can be a
> > >> fair burst of activity.  But those generally wouldn't persist over a long
> > >> period of time.  E.g. every 90 seconds over many hours or days
> > >> or more ... then something else is going on.  So ... let me dig
> > >> (pun not quite intended but apt) a bit more and peek what's currently
> > >> goin' on there ...
> > ...
> > ----- End forwarded message -----
> > -----
> > I suspect it has to do with some changes in how DNSSEC is configured,
> > most notably a configuration syntax I'm using is (now) quite deprecated,
> > and it may be thus defaulting to some rather undesirable behavior(s)
> > which may include triggering the excess notifies.
> > Anyway, want to test it carefully and not break things, that's why I'm not
> > willy-nilly throwing things at it.  Goals being, to not only fix it, but if
> > feasible without causing further breakage along the way, and also not
> > disabling DNSSEC at any point (shouldn't need to) ... and while I'm at
> > it, also reviewing and checking/updating earlier documentation (much of which
> > I wrote):
> > Debian wiki: DNSSEC Howto for BIND 9.9+:
> > https://wiki.debian.org/DNSSEC%20Howto%20for%20BIND%209.9+
> > And, along the way, I've also got temporary zone and subdomain I'm
> > fiddling with: tmp.mpaoli.net. to test things, etc.  Anyway, it's still a work
> > in progress to get that all squared away (get issue fixed, not further
> > break things,
> > and update relevant documentation).
> > Anyway, as far as I'm aware, the excess notifies being sent don't have
> > anything to do with the Comcast issues.
> >
> > On Tue, Jun 4, 2024 at 3:12 PM Rick Moen <rick@linuxmafia.com> wrote:
> > >
> > > More of same, involving back-to-back NOTIFY traffic about mpaoli.net,
> > > allegedly originating from master nameserver 96.86.170.226 .  Michael, I
> > > greatly doubt your nameserver is doing this.  mpaoli.net's zonefile is
> > > pretty unchanging.  I suspect Comcast middleman fsckery.  Thoughts?
> > >
> > > ----- Forwarded message from logcheck system account <logcheck@linuxmafia.com> -----
> > >
> > > Date: Tue, 04 Jun 2024 02:02:01 -0700
> > > From: logcheck system account <logcheck@linuxmafia.com>
> > > To: root@linuxmafia.com
> > > Subject: linuxmafia.com 2024-06-04 02:02 System Events
> > >
> > > System Events
> > > =-=-=-=-=-=-=
> > > Jun  4 01:47:15 linuxmafia named[15622]: zone mpaoli.net/IN: Transfer started.
> > > Jun  4 01:47:15 linuxmafia named[15622]: transfer of 'mpaoli.net/IN' from 96.86.170.226#53: connected using 96.95.217.99#52012
> > > Jun  4 01:47:15 linuxmafia named[15622]: zone mpaoli.net/IN: transferred serial 1717468800
> > > Jun  4 01:47:15 linuxmafia named[15622]: transfer of 'mpaoli.net/IN' from 96.86.170.226#53: Transfer completed: 1 messages, 6 records, 546 bytes, 0.064 secs (8531 bytes/sec)
> > > Jun  4 02:00:09 linuxmafia named[15622]: zone mpaoli.net/IN: Transfer started.
> > > Jun  4 02:00:09 linuxmafia named[15622]: transfer of 'mpaoli.net/IN' from 96.86.170.226#53: connected using 96.95.217.99#57758
> > > Jun  4 02:00:09 linuxmafia named[15622]: zone mpaoli.net/IN: transferred serial 1717491608
> > > Jun  4 02:00:09 linuxmafia named[15622]: transfer of 'mpaoli.net/IN' from 96.86.170.226#53: Transfer completed: 1 messages, 15 records, 1266 bytes, 0.072 secs (17583 bytes/sec)