So, haven't 100% gotten to the bottom of the excess NOTIFY,
but it is apparently present in ISC BIND 9.18.24 and some other versions,
and Debian bind9 1:9.18.24-1 which is the current version for the current stable
Debian 12 bookworm. Also thinking that bug, for Debian stable, probably
doesn't make it to >=important priority (probably just priority normal),
so probably won't get fixed in the current stable.
Thus far, this is what I've run across that seems most informative about it:
https://gitlab.isc.org/isc-projects/bind9/-/issues/3242
So, workarounds? I'm not going to be upgrading that host from stable
for quite some time (most likely year(s)), and other alternatives to fixing it
on the server side is probably more additional overhead than I'm likely to
want to bother with - so probably not going there.
On the client side, can reconfigure logging to not log those bits,
or possibly ignore them (e.g. by filter) in whatever
notifications/reporting one is doing.
So, yes, it is definitely coming from the nameserver itself, and all indications
are it's a bug. I did see some chatter somewhere about it possibly being a
but only if client doesn't respond in some way(s)? But that's not what I see
in my testing on clients that do respond as expected, so I don't think that's
it at all - at least in my case.
Oh, as for possibly adjusting the logging on [ns1.]linuxmafia.com,
I notice it's got
bind9 version 1:9.7.1.dfsg.P2-2
installed, and yes, also has
bind9-doc version 1:9.4.2-4,
but that's different version.
However, it does have, in /var/cache/apt/archives/
bind9-doc_9.7.1.dfsg.P2-2_all.deb
So, that would be the corresponding documentation package,
which could be installed, or the files simply extracted from that and
consulted as may be desired. Anyway, within that, around:
doc/bind9-doc/arm/Bv9ARM.ch06.html#id2575303
It has information about configuring logging, what the defaults are
and the category
notify
in logging configuration that one can use to adjust what one
does/doesn't want done regarding logging of
notify
activity.
And, along the way, I've also rather well updated:
Debian wiki: DNSSEC Howto for BIND 9.9+:
https://wiki.debian.org/DNSSEC%20Howto%20for%20BIND%209.9+
And also well whipping mpaoli.net (and nameserver in general) configuration
into shape for the current Debian stable.
If I find anything else particularly noteworthy about the notify situation,
I'll update.
On Tue, Jun 4, 2024 at 4:21 PM Michael Paoli <michael.paoli(a)berkeley.edu> wrote:
>
> There's two different issues (well, three, but 3rd is minor), not to
> be confused:
>
> The second - unrelated - issue. I've got excess notifies being sent
> from my server(s).
> Now, not at all unusual that sometimes there may be bursts of activity
> and updates and such,
> at least some of that is to be expected.
> What's quite unexpected and ought not be happening, but is,
> is I've got server sending frequent (e.g. like every 90 seconds-ish -
> or so) notifies
> when basic zone DNS data hasn't changed - same SOA serial, etc.
> So, something's still amis with that which I need to correct (doesn't
> really break any
> functionalities ... but rather wasteful all those excess notifies being sent,
> also tends to cause needless traffic on both sending side,
> and receiving servers checking and finding they've got nothing
> to do because they're already current).
> And, as I'd earlier mentioned, I suspect from some major upgrades I did
> a few weeks ago ... well, that was off-list, so, let me include fair
> bit more of it here
> (but not the whole thing, it gets very long, especially with all the
> earlier referenced):
> -----
> ---------- Forwarded message ---------
> From: Michael Paoli <michael.paoli(a)berkeley.edu>
> Date: Mon, Jun 3, 2024 at 11:19 PM
> Subject: Re: [BALUG-Admin] (forw) linuxmafia.com 2024-06-03 15:02 System Events
> To: Al <aw009(a)sunnyside.com>
> Cc: Rick Moen <rick(a)linuxmafia.com>
>
> Yeah, ... something's still not quite right there.
> Did upgrade on that host/server a couple weekends ago
> Debian: buster 10 --> bullseye 11 --> bookworm 12
> $ TZ=GMT0 stat -c '%z %Z %n' /etc/debian_version
> 2024-05-14 07:37:15.000000000 +0000 1715672235 /etc/debian_version
> $
> Some of ye olde(er) DNS configuration options, while still
> (marginally?) functional, are quite deprecated, and I'm getting
> warnings on that and one other minor issue ... anyway, need to do
> some cleanup/reconfig with DNS server on that host.
> Should all be fine after that. Functional in the meantime, but ...
> not yet optimal ... clearly.
>
> On Mon, Jun 3, 2024 at 10:43 PM Al <aw009(a)sunnyside.com> wrote:
> >
> > I'm probably not reading those logs right but part of it looked like mpaoli was being signed every minute? Only partial logs, not enough to be sure. 7:28, 7:29.
> > Just a thought...
> >
> > On June 3, 2024 20:15:06 Michael Paoli via BALUG-Admin <balug-admin(a)lists.balug.org> wrote:
> >
> >> Well, under some circumstances, I'd expect a relative flury, or
> >> "bursts" of notify
> >> activity. E.g. I do TLS(/"SSL") certs via LetsEncrypt.org (LE), using DNS
> >> verification. Some/many of those certs will have multiple
> >> Subject Alternative Name (SAN) domains, and there may be fair number
> >> of such certs. So, adding the relevant records for verification, and then
> >> removing them after having obtained the issued cert(s) - that can be a
> >> fair burst of activity. But those generally wouldn't persist over a long
> >> period of time. E.g. every 90 seconds over many hours or days
> >> or more ... then something else is going on. So ... let me dig
> >> (pun not quite intended but apt) a bit more and peek what's currently
> >> goin' on there ...
> ...
> ----- End forwarded message -----
> -----
> I suspect it has to do with some changes in how DNSSEC is configured,
> most notably a configuration syntax I'm using is (now) quite deprecated,
> and it may be thus defaulting to some rather undesirable behavior(s)
> which may include triggering the excess notifies.
> Anyway, want to test it carefully and not break things, that's why I'm not
> willy-nilly throwing things at it. Goals being, to not only fix it, but if
> feasible without causing further breakage along the way, and also not
> disabling DNSSEC at any point (shouldn't need to) ... and while I'm at
> it, also reviewing and checking/updating earlier documentation (much of which
> I wrote):
> Debian wiki: DNSSEC Howto for BIND 9.9+:
> https://wiki.debian.org/DNSSEC%20Howto%20for%20BIND%209.9+
> And, along the way, I've also got temporary zone and subdomain I'm
> fiddling with: tmp.mpaoli.net. to test things, etc. Anyway, it's still a work
> in progress to get that all squared away (get issue fixed, not further
> break things,
> and update relevant documentation).
> Anyway, as far as I'm aware, the excess notifies being sent don't have
> anything to do with the Comcast issues.
>
> On Tue, Jun 4, 2024 at 3:12 PM Rick Moen <rick(a)linuxmafia.com> wrote:
> >
> > More of same, involving back-to-back NOTIFY traffic about mpaoli.net,
> > allegedly originating from master nameserver 96.86.170.226 . Michael, I
> > greatly doubt your nameserver is doing this. mpaoli.net's zonefile is
> > pretty unchanging. I suspect Comcast middleman fsckery. Thoughts?
> >
> > ----- Forwarded message from logcheck system account <logcheck(a)linuxmafia.com> -----
> >
> > Date: Tue, 04 Jun 2024 02:02:01 -0700
> > From: logcheck system account <logcheck(a)linuxmafia.com>
> > To: root(a)linuxmafia.com
> > Subject: linuxmafia.com 2024-06-04 02:02 System Events
> >
> > System Events
> > =-=-=-=-=-=-=
> > Jun 4 01:47:15 linuxmafia named[15622]: zone mpaoli.net/IN: Transfer started.
> > Jun 4 01:47:15 linuxmafia named[15622]: transfer of 'mpaoli.net/IN' from 96.86.170.226#53: connected using 96.95.217.99#52012
> > Jun 4 01:47:15 linuxmafia named[15622]: zone mpaoli.net/IN: transferred serial 1717468800
> > Jun 4 01:47:15 linuxmafia named[15622]: transfer of 'mpaoli.net/IN' from 96.86.170.226#53: Transfer completed: 1 messages, 6 records, 546 bytes, 0.064 secs (8531 bytes/sec)
> > Jun 4 02:00:09 linuxmafia named[15622]: zone mpaoli.net/IN: Transfer started.
> > Jun 4 02:00:09 linuxmafia named[15622]: transfer of 'mpaoli.net/IN' from 96.86.170.226#53: connected using 96.95.217.99#57758
> > Jun 4 02:00:09 linuxmafia named[15622]: zone mpaoli.net/IN: transferred serial 1717491608
> > Jun 4 02:00:09 linuxmafia named[15622]: transfer of 'mpaoli.net/IN' from 96.86.170.226#53: Transfer completed: 1 messages, 15 records, 1266 bytes, 0.072 secs (17583 bytes/sec)
So ... wee bit of reference introduction,
then my tests/findings/recommendations/etc.,
then more of the earlier referenced bits.
> From: "Michael Paoli" <Michael.Paoli(a)cal.berkeley.edu>
> Subject: Re: [BALUG-Admin] DNS slaves for BALUG? :-) ... IPv6 issue
> somewhere between master and slaves?
> Date: Sat, 20 Feb 2016 18:52:14 -0800
> No problem, I'll take a look, and let you know what I find.
>
> Quite likely it's lack of IPv6 or some other IPv6 issue
> on slave end(s), but I'll do some further checking to see if I can
> isolate it to that (at least for ns1.linuxmafia.com. - I'm presuming
> same/similar for ns1.svlug.org.), and see whether or not
> zone can be pulled via IPv6 - and beyond the IPv6 addresses
> where I've done so on my home networks (all of which happen to
> be via same ISP for IPv6 - so would be good to check from some
> other(s) - I think I may have a location or two (or three or more?) that
> I may be able to check that from).
>
> Thanks, I'll let you know.
So ... most of the testing details first.
Then past that section,
recommendations for the very specific slaves in
question - based upon those test results.
Then there's bit more information/comments after that.
And then a wee bit more additional testing.
$ dig +short ns1.linuxmafia.com. A ns1.linuxmafia.com. AAAA
198.144.195.186
$ ssh -ax linuxmafia.com. 'umask 077 && hostname &&
> PATH=/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin:"$PATH" \
> && { { >>/dev/null 2>&1 type ip && ip -4 a s; ip -6 a s; ip -6 r s; \
> } || { ifconfig -a; netstat -nr; } }'
linuxmafia.com
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
inet 127.0.0.1/8 scope host lo
3: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
state UP qlen 1000
inet 198.144.195.186/29 brd 198.144.195.191 scope global eth2
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
3: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qlen 1000
inet6 fe80::220:edff:fe13:ba89/64 scope link
valid_lft forever preferred_lft forever
fe80::/64 dev eth2 proto kernel metric 256 mtu 1500 advmss 1440 hoplimit 0
$
So ... [ns1.]linuxmafia.com. has an IPv6 stack and IPv6 link-local
apparently configured, but no IPv6 route for anything beyond link-local
and localhost.
Let's see if I can confirm that ...
$ ssh -ax linuxmafia.com. 'umask 077 && hostname && \
> PATH=/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin:"$PATH" \
> && ping6 -n -c 5 2001:470:1f04:19e::2'
linuxmafia.com
connect: Network is unreachable
$
Yes, can't get to 2001:470:1f04:19e::2 from
[ns1.]linuxmafia.com.
198.144.195.186
::1/128
fe80::/64
without some type of route to 2001:470:1f04:19e::2 from that host.
the:
connect: Network is unreachable
probably indicates we're getting an ICMP network unreachable,
probably from the host itself, as it has no available route to get to
2001:470:1f04:19e::2
The diagnostic bit(s):
failure trying master 2001:470:1f04:19e::2#53 (source ::#0): operation
canceled
would be, I'd guess, the nameserver's software's way of indicating it
can't get to there or is otherwise failing - the "(source ::#0)" may be
some kind of hint as to source IP that it's trying from or that it's
trying that from IPv6, and knows not how to get there, or fails from
that source IP, or IPv6 in general.
Let me see if I can pull the zone via IPv6 to elsewhere ...
$ ip -6 a s | fgrep inet6; ip -6 r s | fgrep via
inet6 ::1/128 scope host
inet6 fe80::221a:6ff:fe03:c207/64 scope link
inet6 2001:470:66:76f::2/64 scope global
inet6 fe80::c690:c2eb/64 scope link
inet6 fe80::fc54:ff:fe13:5199/64 scope link
default via 2001:470:66:76f::1 dev he-ipv6 metric 1024
$ dig @2001:470:1f04:19e::2 -t AXFR \
> e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa. +norecurse +noall +nocmd \
> +nocomments +answer | grep '^[ ]*[^ ;]'
e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa. 7200 IN SOA ns1.balug.org.
postmaster.balug.org. 1456011000 10800 3600 1209600 3600
e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa. 7200 IN NS ns0.mpaoli.net.
e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa. 7200 IN NS ns1.balug.org.
e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa. 7200 IN NS ns1.svlug.org.
e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa. 7200 IN NS ns1.linuxmafia.com.
2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa. 7200
IN PTR balug.org.
e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa. 7200 IN SOA ns1.balug.org.
postmaster.balug.org. 1456011000 10800 3600 1209600 3600
$
Yes, I'm able to pull the zone (master is specifically configured:
allow-transfer { any; };
for at least the [L]UG zone(s) - and that includes
e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa.
but the client IPv6 I pulled from is from same ISP/provider as the
server pulled from. Do I have another IPv6 independent of that ISP
that I can check from?
$ ssh -ax m-net.arbornet.org. hostname
ssh: connect to host m-net.arbornet.org. port 22: No route to host
$
Hmmmm...
$ (cd ~/tmp && stat -c '%y %n' .m-net.arbornet.org* && cat
> .m-net.arbornet.org.lastlogin)
2016-02-10 00:05:03.690977780 -0800 .m-net.arbornet.org.lastlogin
michaelp pts/7 Feb 10 03:05 (198.144.194.235)
Connection to m-net.arbornet.org. closed.
$
I've got a crontab job:
$ crontab -l | fgrep m-net.arbornet.org | grep -v '^#'
5 8 * * * TZ=PST8PDT export TZ; set -e; trap 'trap - 0; exit 0' 0;
exec </dev/null >>/dev/null 2>&1; cd tmp;
lastlogin=.m-net.arbornet.org.lastlogin
lastoutput=.m-net.arbornet.org.lastoutput; set +e; tmp=`find
"$lastlogin" -prune ! -mtime +15 -type f -print`; [ x"$tmp" =
x"$lastlogin" ] || { </dev/null >"$lastoutput" 2>&1 ssh -a -n -t -t -x
-o BatchMode=yes m-net.arbornet.org. 'umask 077 && exec who am I' &&
mv -f "$lastoutput" "$lastlogin"; }
$
that keeps my account there from going away (I think it has to be logged
into once every 30 days to not get removed). My crontab job, daily,
makes a login attempt if its last successful login was more than 15
days ago.
Looks like it worked rather recently ... but not working at the moment.
So, another ISP's IPv6 ...
ignore some of the despicable operating system bits below ...
but at least with Cygwin it's almost bearable ... testing from a
work connection (hey, fair's fair, often use home connections from work
to do work to, e.g. test various accessibility from The Internet, etc.)
...
$ IPCONFIG /ALL | fgrep IPv6\ Address | fgrep -v Link-local
IPv6 Address. . . . . . . . . . . :
2600:1010:b161:5e74:7c7a:2cff:fe94:2fa1(Preferred)
$ dig @2001:470:1f04:19e::2 -t AXFR \
> e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa. +norecurse +noall +nocmd \
> +nocomments +answer | grep '^[ ]*[^ ;]'
/usr/src/ports/bind/bind-9.10.2-2.P2.x86_64/src/bind-9.10.2-P2/lib/isc/unix/socket.c:2867: setsockopt(20, IPV6_RECVTCLASS) failed: Invalid
argument
e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa. 7200 IN SOA ns1.balug.org.
postmaster.balug.org. 1456011000 10800 3600 1209600 3600
e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa. 7200 IN NS ns0.mpaoli.net.
e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa. 7200 IN NS ns1.balug.org.
e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa. 7200 IN NS ns1.svlug.org.
e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa. 7200 IN NS ns1.linuxmafia.com.
2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa. 7200
IN PTR balug.org.
e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa. 7200 IN SOA ns1.balug.org.
postmaster.balug.org. 1456011000 10800 3600 1209600 3600
$
Okay, enough of that, ... set *that* operating system aside ...
now back to a *real* operating system :-> ...
And to look at ISPs to see/show they're different, at least to the
extent can check with whois(1), we have ...
First the master:
$ 2>&1 whois -H 2001:470:1f04:19e::2 |
> egrep -i '^(netname|nethandle|netrange|cidr|organization)'
NetRange: 2001:470:: - 2001:470:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF
CIDR: 2001:470::/32
NetName: HURRICANE-IPV6
NetHandle: NET6-2001-470-1
Organization: Hurricane Electric, Inc. (HURC)
$
Then the first global routable IPv6 I tried from (expecting same ISP in
whois):
$ 2>&1 whois -H 2001:470:66:76f::2 |
> egrep -i '^(netname|nethandle|netrange|cidr|organization)'
NetRange: 2001:470:: - 2001:470:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF
CIDR: 2001:470::/32
NetName: HURRICANE-IPV6
NetHandle: NET6-2001-470-1
Organization: Hurricane Electric, Inc. (HURC)
$
No surprises there - it's also part of same NetRange / CIDR, as we can
see in even the first set of whois data.
And last client I tried, I'm hoping/expecting different ISP (is a
big telecom company after all) ...
$ 2>&1 whois -H 2600:1010:b161:5e74:7c7a:2cff:fe94:2fa1 |
> egrep -i '^(netname|nethandle|netrange|cidr|organization)'
NetRange: 2600:1000:: - 2600:1017:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF
CIDR: 2600:1010::/29, 2600:1000::/28
NetName: WIRELESSDATANETWORK
NetHandle: NET6-2600-1000-1
Organization: Cellco Partnership DBA Verizon Wireless (CLLC)
$
And yes, an ISP, that as far as I'm aware, is quite independent of the
other ISP shown for the two other IPv6 addresses.
So ... recommendations ...
I'm not sure what DNS slave server software is in use
but for the masters, if the IPv6 addresses are listed after IPv4 (which
may already be the case) and it still complains, my recommendation would
be to comment out or otherwise disable the IPv6 master address(es) until
such time as the slave hosts have some type of routable Internet IPv6
connectivity.
$ dig +short ns1.svlug.org. A ns1.svlug.org. AAAA
64.62.190.98
$
I'd guess issue is same for ns1.svlug.org. - I don't see any IPv6
addresses there, but I don't have access to the host, as far as I'm
aware, so am not in position to test more fully from that slave host.
So, if same or quite similar applies to that slave host, then same
recommendation would apply.
I'm also guestimating, the DNS slave server software likely checks
*all* masters - I'd expect that - probably starts with SOA, and then
grabs "highest" (per the DNS RFCs) SOA serial number zone successfully
found from master it successfully did SOA query on. It may or may not
have some fallback to try AXFR if SOA fails, or if SOA succeeds but AXFR
fails, there's probably some retry/failover algorithm it uses among the
master(s) configured for that slave for that zone. So, it probably at
least *tries* all configured masters, and depending upon the software,
and possible configuration and logging levels, it may complain for any
it fails to get the data from, or I'd think at least most minimally, if
it failed with all masters - but I'm guessing it would more likely
complain if it failed with *any* of the masters, as one may generally
want to at least minimally have that logged, for possible
investigation/action (expected event/maintenance, or known outage, or
... some other issue/problem)?
Miscellaneous comments :-) :
IPv6 - random John and Jane Doe consumer might not yet know or care (and
may never know, or particularly care), but The Internet is (slowly)
going the way of IPv6. IPv4 won't disappear anytime soon. However,
with the exhaustion of IPv4 addresses, increasingly clients are not only
dual-stack (or have some funky way to still be able to access IPv4),
but increasingly there will be clients much prefer IPv6 address and/or
do or will only have IPv6 access. IPv6 adoption has also been growing
at a relatively fast rate. If I recall correctly, last year IPv6 growth
was up something like 10%. I remember even about 4 years ago, looking
at DNS traffic of a major provider, AAAA queries were up to nearly 2% of
volume of queries, relative to A+AAAA - that was up about 300% compared
to only about a year or two earlier. Not sure about similar current stats,
but in general, any such clients that are IPv6 only, those would
typically be lost traffic/customers, or they may be stuck with some less
preferred means of IPv4 access. We're well past the time where at least
most all major providers should also be offering IPv6 connectivity (e.g.
web sites, etc.). The big players in ISPs space all (or nearly all?) do
IPv6, some of the smaller ISPs are still getting there (and even some of
the colos aren't fully there yet). But in any case, IPv6 is coming,
and one should prepare and get ready for it (but not an extreme rush for
everyone - if you're domestic only web site with only domestic users,
may not be much of a rush to use IPv6 ... but dang, it certainly is also
nice to not be squeezed with very limited number of IPv4 addresses to
work with).
Oh, if one doesn't have IPv6 available from one's ISP, there are IPv6
tunnel brokers, so one can use IPv6 over IPv4 - and thus reach the IPv6
Internet. E.g. Hurricane Electric provides such - for free (at least as
in beer). They also have a lot of excellent IPv6 training materials
(though a bit dated now, it's still mostly quite applicable).
Anyway, I do have such IPv6 tunnels from Hurricane Electric (alas, my
home ISP isn't yet offering native IPv6 ... I did also notice my work
cellular, didn't have IPv6 only a few months or so ago, but a month or
two after that, had added IPv6). Anyway, have an IPv6 tunnel for myself.
Also have one for BALUG (I did a separate account for BALUG, so it's
quite independent of my personal account).
From Hurricane Electric:
https://ipv6.he.net/certification/https://tunnelbroker.net/
They also offer some free DNS services ... but they may be slightly
funky in how they behave, and also somewhat limited, in at least some
regards:
https://dns.he.net/
But those do also include IPv6 nameservers.
Also, in the ssh examples above, I didn't include bothering to show use
of ssh-add - generally used in advance to decrypt private key(s) and
then temporarily (for specified length of time) hold the decrypted
private key(s) in RAM - best to never have the keys in the clear written
to non-volatile storage (e.g. disk or SSD or similar). The ssh-add
program also takes care to not let private keys be written to swap - but
I also have my swap encrypted, and likewise most of my drive storage.
And in the above regular expressions where there's character class
([...]) showing whitespace within, the whitespace is generally a space
and a tab (my email client doesn't conveniently allow me to literally
pass that along - and even if I did, clients displaying such may not
preserve such in display, and in any case it wouldn't generally be
visually obvious). Some RE contents allow \t for tab, but those
I showed above generally don't support that.
And retrying earlier test:
$ ssh -ax m-net.arbornet.org. hostname
m-net.arbornet.org
$
That works now, so trying further with that ...
$ ssh -ax m-net.arbornet.org. 'hostname &&
> PATH="/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin:$PATH" \
> && { if >>/dev/null 2>&1 type ip; then ip -6 a s; ip -6 r s
> else ifconfig -a; netstat -nr; fi; }'
m-net.arbornet.org
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
ether 00:50:56:8b:48:cf
inet 162.202.67.157 netmask 0xffffffe0 broadcast 162.202.67.159
inet6 fe80::250:56ff:fe8b:48cf%em0 prefixlen 64 scopeid 0x1
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
em1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
ether 00:0c:29:34:2e:d3
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: no carrier
plip0: flags=8810<POINTOPOINT,SIMPLEX,MULTICAST> metric 0 mtu 1500
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=3<RXCSUM,TXCSUM>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
inet 127.0.0.1 netmask 0xff000000
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ipfw0: flags=8801<UP,SIMPLEX,MULTICAST> metric 0 mtu 65536
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
Routing tables
Internet:
Destination Gateway Flags Refs Use Netif Expire
default 162.202.67.129 UGS 0 4573 em0
127.0.0.1 link#4 UH 0 47170 lo0
162.202.67.128/27 link#1 U 0 127 em0
162.202.67.157 link#1 UHS 0 0 lo0
Internet6:
Destination Gateway Flags
Netif Expire
::/96 ::1 UGRS
lo0
::1 ::1 UH
lo0
::ffff:0.0.0.0/96 ::1 UGRS
lo0
fe80::/10 ::1 UGRS
lo0
fe80::%em0/64 link#1 U
em0
fe80::250:56ff:fe8b:48cf%em0 link#1 UHS
lo0
fe80::%lo0/64 link#4 U
lo0
fe80::1%lo0 link#4 UHS
lo0
ff01::%em0/32 fe80::250:56ff:fe8b:48cf%em0 U
em0
ff01::%lo0/32 ::1 U
lo0
ff02::/16 ::1 UGRS
lo0
ff02::%em0/32 fe80::250:56ff:fe8b:48cf%em0 U
em0
ff02::%lo0/32 ::1 U
lo0
$
I don't think I see anything there that's globally routable IPv6
ff0x::114 Used for experiments
... yeah, I was wondering what ff01::... is/was ... wasn't able to
identify that. Well, let's try wee bit:
$ ssh -ax m-net.arbornet.org. 'hostname &&
> PATH="/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin:$PATH" \
> && >>/dev/null 2>&1 type ping6 && ping6 -n -c 5 2001:470:1f04:19e::2'
m-net.arbornet.org
ping6: UDP connect: No route to host
$
Yep, it doesn't have a route to get there, so can't test further from
there.
>> From: "Rick Moen" <rick(a)linuxmafia.com>
>> Subject: Re: [BALUG-Admin] DNS slaves for BALUG? :-)
>> Date: Sat, 20 Feb 2016 18:33:09 -0800
>
>> I wrote:
>>
>>> Quoting Michael Paoli (Michael.Paoli(a)cal.berkeley.edu):
>>>
>>>> If you could please, and would be willing,
>>>> could you cover DNS slave services for BALUG,
>>>> notably these zones:
>>>> e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa
>>>> balug.org
>>>> master(s) (all for each of the above):
>>>> 198.144.194.238
>>>> 2001:470:1f04:19e::2
>>>
>>> FYI, ns1.linuxmafia.com is repeatedly logging this:
>>>
>>> Feb 20 17:50:45 linuxmafia named[17291]: zone
>>> e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa/IN: refresh: failure
>>> trying master 2001:470:1f04:19e::2#53 (source ::#0): operation
>>> canceled
>>>
>>> I'm unsure why the AXFR/IXFR to the IPv6 address fails, and currently
>>> lack the time and patience to pursue that matter. For now, I'm going to
>>> disable that master IP on my BIND9 conffile.
>>
>> Before I do that, Michael, would you like to any some checking on the
>> _master_ nameserver end? The most probable explanation is some deficiency
>> in IPv6 support on ns1.linuxmafia.com, and that would be my default
>> assumption.
>>
>> FYI, both zones are affected:
>>
>> linuxmafia:/etc/bind# zgrep 'failure trying master' /var/log/*
>> /var/log/daemon.log:Feb 20 12:55:59 linuxmafia named[17291]: zone
>> balug.org/IN: refresh: failure trying master
>> 2001:470:1f04:19e::2#53 (source ::#0): operation canceled
>> /var/log/daemon.log:Feb 20 13:39:16 linuxmafia named[17291]: zone
>> e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa/IN: refresh: failure
>> trying master 2001:470:1f04:19e::2#53 (source ::#0): operation
>> canceled
>> /var/log/daemon.log:Feb 20 17:50:45 linuxmafia named[17291]: zone
>> e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa/IN: refresh: failure
>> trying master 2001:470:1f04:19e::2#53 (source ::#0): operation
>> canceled
>> /var/log/syslog:Feb 20 12:55:59 linuxmafia named[17291]: zone
>> balug.org/IN: refresh: failure trying master
>> 2001:470:1f04:19e::2#53 (source ::#0): operation canceled
>> /var/log/syslog:Feb 20 13:39:16 linuxmafia named[17291]: zone
>> e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa/IN: refresh: failure
>> trying master 2001:470:1f04:19e::2#53 (source ::#0): operation
>> canceled
>> /var/log/syslog:Feb 20 17:50:45 linuxmafia named[17291]: zone
>> e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa/IN: refresh: failure
>> trying master 2001:470:1f04:19e::2#53 (source ::#0): operation
>> canceled
>> linuxmafia:/etc/bind#
So, modest update.
Yes, it's very much issue on server (thus far doesn't appear to be
issue of Comcast Business' doing).
Also, have tried it with server on VM, and haven't been able to
replicate the issue there.
So, possible it might just be some funkiness in configuration I have
to work out - there
are some other bits too, e.g. systemd doesn't quite cleanly start it -
it takes a while, then
complains that the startup attempt failed with timeout ... but upon
checking it's actually
properly up and running. Also have non-trivial differences between my actual
production setup and what I'd tested on VM, e.g. production is using
actual delegated
zones, also has Dynamic DNS (DDNS), also configured using chroot, etc. So, not
trivial to do apples-to-apples comparison, though did at least compare
with exact same
versions of Debian and Debian's BIND. But even if I find what makes
the difference
configuration-wise or the like, doesn't mean it's necessarily a
reasonably feasible fix.
E.g. the bug might only show up with chroot or DDNS or with actual
domains (where
client is a secondary). Anyway, haven't given up on it, but not super
high on the
priorities to nail that one ... and might not be particularly feasibly
fixable at current
anyway.
And yeah, if you saw/see notifies from one of my IPs for zone
example.com - yeah,
that'd be from some of the VM testing (from both stable, and possibly
also unstable+experimental).
And if you're curious and/or bored and interested in more, I was earlier testing
on zone tmp.mpaoli.net. I later switched to instead using
dnssec-test.mpaoli.net
Some interesting test/history bits of those may be seen on
https://dnsviz.net/
And for some bit I was even using tmp2.mpaoli.net, but don't know that
I got as far
as checking any of that zone on https://dnsviz.net/
And yes,
https://wiki.debian.org/DNSSEC%20Howto%20for%20BIND%209.9+
is even more nicely updated to current and quite thoroughly tested at
this point.
And keys are still being rolled on mpaoli.net (most notably KSK, but
also some ZSK),
so for bit longer on will see additional cruft there (because TTLs and
to not break thing),
but the remaining cleanup will happen on that relatively soon (most
notably once last relevant TTLs
have passed). And yes, one can see key steps along that process via:
https://dnsviz.net/d/mpaoli.net/dnssec/
See, e.g.:
https://dnsviz.net/d/mpaoli.net/ZmORYA/dnssec/
for bit more interesting along the way.
And alas, if I'd gotten it fully optimal along the way (notably
with lifetime policy in dnssec-policy statement), I could've avoided
creating and having used and in place a whole additional ZSK that
wasn't even needed and is in most all regards quite redundant.
In any case, all will be completed with cleanup fairly soon on that.
Yeah, rotating KSKs is non-trivial, but advances in both RFCs and
BIND are continuing to make that easier.
Anyway, I agree, it ought not be doing the excessive notify - annoying that.
But, no great harm, and I may or may not mange to get that issue fixed
anytime soon.
On Sat, Jun 8, 2024 at 4:31 PM Michael Paoli <michael.paoli(a)berkeley.edu> wrote:
>
> So, haven't 100% gotten to the bottom of the excess NOTIFY,
> but it is apparently present in ISC BIND 9.18.24 and some other versions,
> and Debian bind9 1:9.18.24-1 which is the current version for the current stable
> Debian 12 bookworm. Also thinking that bug, for Debian stable, probably
> doesn't make it to >=important priority (probably just priority normal),
> so probably won't get fixed in the current stable.
> Thus far, this is what I've run across that seems most informative about it:
> https://gitlab.isc.org/isc-projects/bind9/-/issues/3242
> So, workarounds? I'm not going to be upgrading that host from stable
> for quite some time (most likely year(s)), and other alternatives to fixing it
> on the server side is probably more additional overhead than I'm likely to
> want to bother with - so probably not going there.
> On the client side, can reconfigure logging to not log those bits,
> or possibly ignore them (e.g. by filter) in whatever
> notifications/reporting one is doing.
> So, yes, it is definitely coming from the nameserver itself, and all indications
> are it's a bug. I did see some chatter somewhere about it possibly being a
> but only if client doesn't respond in some way(s)? But that's not what I see
> in my testing on clients that do respond as expected, so I don't think that's
> it at all - at least in my case.
>
> Oh, as for possibly adjusting the logging on [ns1.]linuxmafia.com,
> I notice it's got
> bind9 version 1:9.7.1.dfsg.P2-2
> installed, and yes, also has
> bind9-doc version 1:9.4.2-4,
> but that's different version.
> However, it does have, in /var/cache/apt/archives/
> bind9-doc_9.7.1.dfsg.P2-2_all.deb
> So, that would be the corresponding documentation package,
> which could be installed, or the files simply extracted from that and
> consulted as may be desired. Anyway, within that, around:
> doc/bind9-doc/arm/Bv9ARM.ch06.html#id2575303
> It has information about configuring logging, what the defaults are
> and the category
> notify
> in logging configuration that one can use to adjust what one
> does/doesn't want done regarding logging of
> notify
> activity.
>
> And, along the way, I've also rather well updated:
> Debian wiki: DNSSEC Howto for BIND 9.9+:
> https://wiki.debian.org/DNSSEC%20Howto%20for%20BIND%209.9+
> And also well whipping mpaoli.net (and nameserver in general) configuration
> into shape for the current Debian stable.
>
> If I find anything else particularly noteworthy about the notify situation,
> I'll update.
>
> On Tue, Jun 4, 2024 at 4:21 PM Michael Paoli <michael.paoli(a)berkeley.edu> wrote:
> >
> > There's two different issues (well, three, but 3rd is minor), not to
> > be confused:
> >
> > The second - unrelated - issue. I've got excess notifies being sent
> > from my server(s).
> > Now, not at all unusual that sometimes there may be bursts of activity
> > and updates and such,
> > at least some of that is to be expected.
> > What's quite unexpected and ought not be happening, but is,
> > is I've got server sending frequent (e.g. like every 90 seconds-ish -
> > or so) notifies
> > when basic zone DNS data hasn't changed - same SOA serial, etc.
> > So, something's still amis with that which I need to correct (doesn't
> > really break any
> > functionalities ... but rather wasteful all those excess notifies being sent,
> > also tends to cause needless traffic on both sending side,
> > and receiving servers checking and finding they've got nothing
> > to do because they're already current).
> > And, as I'd earlier mentioned, I suspect from some major upgrades I did
> > a few weeks ago ... well, that was off-list, so, let me include fair
> > bit more of it here
> > (but not the whole thing, it gets very long, especially with all the
> > earlier referenced):
> > -----
> > ---------- Forwarded message ---------
> > From: Michael Paoli <michael.paoli(a)berkeley.edu>
> > Date: Mon, Jun 3, 2024 at 11:19 PM
> > Subject: Re: [BALUG-Admin] (forw) linuxmafia.com 2024-06-03 15:02 System Events
> > To: Al <aw009(a)sunnyside.com>
> > Cc: Rick Moen <rick(a)linuxmafia.com>
> >
> > Yeah, ... something's still not quite right there.
> > Did upgrade on that host/server a couple weekends ago
> > Debian: buster 10 --> bullseye 11 --> bookworm 12
> > $ TZ=GMT0 stat -c '%z %Z %n' /etc/debian_version
> > 2024-05-14 07:37:15.000000000 +0000 1715672235 /etc/debian_version
> > $
> > Some of ye olde(er) DNS configuration options, while still
> > (marginally?) functional, are quite deprecated, and I'm getting
> > warnings on that and one other minor issue ... anyway, need to do
> > some cleanup/reconfig with DNS server on that host.
> > Should all be fine after that. Functional in the meantime, but ...
> > not yet optimal ... clearly.
> >
> > On Mon, Jun 3, 2024 at 10:43 PM Al <aw009(a)sunnyside.com> wrote:
> > >
> > > I'm probably not reading those logs right but part of it looked like mpaoli was being signed every minute? Only partial logs, not enough to be sure. 7:28, 7:29.
> > > Just a thought...
> > >
> > > On June 3, 2024 20:15:06 Michael Paoli via BALUG-Admin <balug-admin(a)lists.balug.org> wrote:
> > >
> > >> Well, under some circumstances, I'd expect a relative flury, or
> > >> "bursts" of notify
> > >> activity. E.g. I do TLS(/"SSL") certs via LetsEncrypt.org (LE), using DNS
> > >> verification. Some/many of those certs will have multiple
> > >> Subject Alternative Name (SAN) domains, and there may be fair number
> > >> of such certs. So, adding the relevant records for verification, and then
> > >> removing them after having obtained the issued cert(s) - that can be a
> > >> fair burst of activity. But those generally wouldn't persist over a long
> > >> period of time. E.g. every 90 seconds over many hours or days
> > >> or more ... then something else is going on. So ... let me dig
> > >> (pun not quite intended but apt) a bit more and peek what's currently
> > >> goin' on there ...
> > ...
> > ----- End forwarded message -----
> > -----
> > I suspect it has to do with some changes in how DNSSEC is configured,
> > most notably a configuration syntax I'm using is (now) quite deprecated,
> > and it may be thus defaulting to some rather undesirable behavior(s)
> > which may include triggering the excess notifies.
> > Anyway, want to test it carefully and not break things, that's why I'm not
> > willy-nilly throwing things at it. Goals being, to not only fix it, but if
> > feasible without causing further breakage along the way, and also not
> > disabling DNSSEC at any point (shouldn't need to) ... and while I'm at
> > it, also reviewing and checking/updating earlier documentation (much of which
> > I wrote):
> > Debian wiki: DNSSEC Howto for BIND 9.9+:
> > https://wiki.debian.org/DNSSEC%20Howto%20for%20BIND%209.9+
> > And, along the way, I've also got temporary zone and subdomain I'm
> > fiddling with: tmp.mpaoli.net. to test things, etc. Anyway, it's still a work
> > in progress to get that all squared away (get issue fixed, not further
> > break things,
> > and update relevant documentation).
> > Anyway, as far as I'm aware, the excess notifies being sent don't have
> > anything to do with the Comcast issues.
> >
> > On Tue, Jun 4, 2024 at 3:12 PM Rick Moen <rick(a)linuxmafia.com> wrote:
> > >
> > > More of same, involving back-to-back NOTIFY traffic about mpaoli.net,
> > > allegedly originating from master nameserver 96.86.170.226 . Michael, I
> > > greatly doubt your nameserver is doing this. mpaoli.net's zonefile is
> > > pretty unchanging. I suspect Comcast middleman fsckery. Thoughts?
> > >
> > > ----- Forwarded message from logcheck system account <logcheck(a)linuxmafia.com> -----
> > >
> > > Date: Tue, 04 Jun 2024 02:02:01 -0700
> > > From: logcheck system account <logcheck(a)linuxmafia.com>
> > > To: root(a)linuxmafia.com
> > > Subject: linuxmafia.com 2024-06-04 02:02 System Events
> > >
> > > System Events
> > > =-=-=-=-=-=-=
> > > Jun 4 01:47:15 linuxmafia named[15622]: zone mpaoli.net/IN: Transfer started.
> > > Jun 4 01:47:15 linuxmafia named[15622]: transfer of 'mpaoli.net/IN' from 96.86.170.226#53: connected using 96.95.217.99#52012
> > > Jun 4 01:47:15 linuxmafia named[15622]: zone mpaoli.net/IN: transferred serial 1717468800
> > > Jun 4 01:47:15 linuxmafia named[15622]: transfer of 'mpaoli.net/IN' from 96.86.170.226#53: Transfer completed: 1 messages, 6 records, 546 bytes, 0.064 secs (8531 bytes/sec)
> > > Jun 4 02:00:09 linuxmafia named[15622]: zone mpaoli.net/IN: Transfer started.
> > > Jun 4 02:00:09 linuxmafia named[15622]: transfer of 'mpaoli.net/IN' from 96.86.170.226#53: connected using 96.95.217.99#57758
> > > Jun 4 02:00:09 linuxmafia named[15622]: zone mpaoli.net/IN: transferred serial 1717491608
> > > Jun 4 02:00:09 linuxmafia named[15622]: transfer of 'mpaoli.net/IN' from 96.86.170.226#53: Transfer completed: 1 messages, 15 records, 1266 bytes, 0.072 secs (17583 bytes/sec)