[BALUG-Admin] (forw) linuxmafia.com 2024-06-03 15:02 System Events

Tue Jun 4 03:14:20 UTC 2024

Well, under some circumstances, I'd expect a relative flury, or
"bursts" of notify
activity.  E.g. I do TLS(/"SSL") certs via LetsEncrypt.org (LE), using DNS
verification.  Some/many of those certs will have multiple
Subject Alternative Name (SAN) domains, and there may be fair number
of such certs.  So, adding the relevant records for verification, and then
removing them after having obtained the issued cert(s) - that can be a
fair burst of activity.  But those generally wouldn't persist over a long
period of time.  E.g. every 90 seconds over many hours or days
or more ... then something else is going on.  So ... let me dig
(pun not quite intended but apt) a bit more and peek what's currently
goin' on there ...

https://www.mpaoli.net/~michael/bin/DNS_CK
$ DNS_CK mpaoli.net
FQDN: mpaoli.net.:
authority:
mpaoli.net. 172800 IN NS ns0.balug.org.
mpaoli.net. 172800 IN NS ns0.mpaoli.net.
mpaoli.net. 172800 IN NS ns1.linuxmafia.com.
mpaoli.net. 172800 IN NS nsx.sunnyside.com.
mpaoli.net. 172800 IN NS nsy.sunnysidex.com.
mpaoli.net. 172800 IN SOA ns0.mpaoli.net.
Michael\.Paoli.cal.berkeley.edu. 1717399740 10800 3600 3600000 86400
@96.86.170.229 (ns0.balug.org.)
mpaoli.net. 172800 IN SOA ns0.mpaoli.net.
Michael\.Paoli.cal.berkeley.edu. 1717399740 10800 3600 3600000 86400
@2001:470:1f05:19e::2 (ns0.balug.org.)
mpaoli.net. 172800 IN SOA ns0.mpaoli.net.
Michael\.Paoli.cal.berkeley.edu. 1717399740 10800 3600 3600000 86400
@96.86.170.226 (ns0.mpaoli.net.)
mpaoli.net. 172800 IN SOA ns0.mpaoli.net.
Michael\.Paoli.cal.berkeley.edu. 1717399740 10800 3600 3600000 86400
@2001:470:67:76f::2 (ns0.mpaoli.net.)
mpaoli.net. 172800 IN SOA ns0.mpaoli.net.
Michael\.Paoli.cal.berkeley.edu. 1717399740 10800 3600 3600000 86400
@96.95.217.99 (ns1.linuxmafia.com.)
mpaoli.net. 172800 IN SOA ns0.mpaoli.net.
Michael\.Paoli.cal.berkeley.edu. 1717399740 10800 3600 3600000 86400
@50.242.105.52 (nsx.sunnyside.com.)
;; communications error to 2603:3024:180d:f100:50:242:105:34#53: timed
out ;; communications error to 2603:3024:180d:f100:50:242:105:34#53:
timed out ;; communications error to
2603:3024:180d:f100:50:242:105:34#53: timed out ;; no servers could be
reached @2603:3024:180d:f100:50:242:105:34 (nsx.sunnyside.com.)
mpaoli.net. 172800 IN SOA ns0.mpaoli.net.
Michael\.Paoli.cal.berkeley.edu. 1717399740 10800 3600 3600000 86400
@50.18.139.240 (nsy.sunnysidex.com.)
mpaoli.net. 172800 IN SOA ns0.mpaoli.net.
Michael\.Paoli.cal.berkeley.edu. 1717399740 10800 3600 3600000 86400
@2600:1f1c:528:c500:5e0b:8a37:6598:356c (nsy.sunnysidex.com.)
$

Okay, so there's also (probably) unrelated issue with:
2603:3024:180d:f100:50:242:105:34#53 nsx.sunnyside.com.

96.95.217.99 (ns1.linuxmafia.com.) shows current SERIAL,
so seems it's been updated relatively lately ... using
Dynamic DNS (DDNS) with UNIX epoch based time for
SOA SERIAL so ... that was likely updated roughly around ...
$ date -Iseconds -d @1717399740
2024-06-03T00:29:00-07:00
$
Since I've also got automatic zone signing with DNSSEC,
the SOA SERIAL - between unsigned (internal) and signed (what's actually
served up) may and typically will differ a bit, and the thus the timings may not
so precisely align to UNIX epoch (logs show, in many cases, both the
signed and unsigned SERIAL - most notably when freshly signing or
the like).
Anyway, that's not all that long ago - don't think I manually changed something
then, but may have been automatic or scheduled (e.g. at(1)), e.g. clearing
out some temporary goop I earlier added, or may have been automatic
occasional rolling of DNSSEC related data or the like.
Let's see, out of curiosity, peeking at my logs ...
2024-06-03T07:28:55.951311+00:00 tigger named[385668]: zone
mpaoli.net/IN (signed): reconfiguring zone keys
2024-06-03T07:28:55.953811+00:00 tigger named[385668]: zone
mpaoli.net/IN (signed): next key event: 03-Jun-2024 08:28:55.946
2024-06-03T07:29:00.976632+00:00 tigger named[385668]: zone
mpaoli.net/IN (signed): sending notifies (serial 1717399740)
Okay, ... that much, at least in-and-of itself would seem to make sense ...
But checking further, I'm seeing way too many notifies being sent,
And without good clear reason as to why.
{
gzip -d < syslog.7.gz
gzip -d < syslog.6.gz
gzip -d < syslog.4.gz
gzip -d < syslog.3.gz
gzip -d < syslog.2.gz
cat < syslog.1
cat < syslog
} |
grep -a ' tigger named\[[0-9]*\]: zone mpaoli\.net\/IN (signed):
sending notifies (serial 1717399740)' |
sed -ne '1p;$p;$='
2024-06-03T07:29:00.976632+00:00 tigger named[385668]: zone
mpaoli.net/IN (signed): sending notifies (serial 1717399740)
2024-06-04T02:06:33.456563+00:00 tigger named[423849]: zone
mpaoli.net/IN (signed): sending notifies (serial 1717399740)
740
So ... 740 notifies sent within that timespan shown above.
That's far too many for that zone in that range of time,
notably same SOA SERIAL, so the data itself hasn't even changed.
Anyway, that's issue on my end to find and fix.

So, I wouldn't expect ongoing NOTIFY every 90s ... but well, do have far
too many going, on, for reason(s) to be determined (and for me to fix).
I did also do major OS (Debian: buster 10 --> bullseye 11 --> bookworm 12)
upgrade on one of the DNS servers (that very one) not too long ago
(bit over two weeks ago:
$ TZ=GMT0 stat -c '%z %Z %n' /etc/debian_version
2024-05-14 07:37:15.000000000 +0000 1715672235 /etc/debian_version
$
) ... so, maybe that caused something?  Who knows.
Anyway, I'll look into and fix that.

And checking any possibly relevant SOA serial mismatches on ns1.linuxmafia.com
for "my" domains ...
$ DNS_CK
...
no SOA SERIAL discrepancies found for ns1.linuxmafia.com - all lined
up as expected and current.

> Jun  3 14:48:39 linuxmafia named[15622]: client 73.189.65.18#16833: received notify for zone 'mpaoli.net'
But yeah, that wouldn't be from me.
96.86.170.226 is the only IP I definitely should be sending NOTIFY
from for mpaoli.net,
though there might possibly be some additional (e.g. other
authoritatives, IPv6, ...),
but certainly not 73.189.65.18 - not my IP at all.  Sounds like some
drain damaged Comcast Business SecurityEdge, or other, DNS fckery - at
least that'd be my first guess (and especially given their track record).

So ... let me check for bit ... everything I've got going out to
96.95.217.99:53 UDP
and everything
96.95.217.99:53 UDP has coming in that looks like DNS NOTIFY ...
So ... on my end of things, and point where I can snag any relevant
traffic, I do see apparently (too) periodic NOTIFY:
# tcpdump -i eno1 -n -s 0 -v udp and host 96.95.217.99 and port 53
tcpdump: listening on eno1, link-type EN10MB (Ethernet), snapshot
length 262144 bytes
02:30:41.945105 IP (tos 0x0, ttl 64, id 64173, offset 0, flags [none],
proto UDP (17), length 126)
    96.86.170.226.51682 > 96.95.217.99.53: 62396 notify [b2&3=0x2400]
[1a] SOA? mpaoli.net. (98)
02:30:41.975644 IP (tos 0x0, ttl 54, id 56888, offset 0, flags [none],
proto UDP (17), length 56)
    96.95.217.99.53 > 96.86.170.226.51682: 62396 notify*- 0/0/0 (28)
02:32:12.436952 IP (tos 0x0, ttl 64, id 1569, offset 0, flags [none],
proto UDP (17), length 126)
    96.86.170.226.23199 > 96.95.217.99.53: 48406 notify [b2&3=0x2400]
[1a] SOA? mpaoli.net. (98)
02:32:12.468027 IP (tos 0x0, ttl 54, id 56894, offset 0, flags [none],
proto UDP (17), length 56)
    96.95.217.99.53 > 96.86.170.226.23199: 48406 notify*- 0/0/0 (28)
^C
4 packets captured
4 packets received by filter
0 packets dropped by kernel
#
Yeah, definitely too frequently - something for
me to fix on my end.

And, let's see what we can see of that at/around the ns1.linuxmafia.com. end:
This was done on guido, on physical interface connecting to hardwired network,
There is (or should be) no NAT/SNAT involved on these IPs, and in promiscuous
mode there (default), should see ns1.linuxmafia.com's traffic fine
(and bit easier
and more capable to see it on guido with the much more recent OS and
software there):
# tcpdump -i enp6s0 -n -s 0 -v udp and host 96.95.217.99 and port 53 |
grep --line-buffered -B 1 -F -i notify
tcpdump: listening on enp6s0, link-type EN10MB (Ethernet), snapshot
length 262144 bytes
02:36:44.479125 IP (tos 0x0, ttl 54, id 25954, offset 0, flags [none],
proto UDP (17), length 126)
    96.86.170.226.45546 > 96.95.217.99.53: 34616 notify [b2&3=0x2400]
[1a] SOA? mpaoli.net. (98)
02:36:44.480772 IP (tos 0x0, ttl 64, id 56916, offset 0, flags [none],
proto UDP (17), length 56)
    96.95.217.99.53 > 96.86.170.226.45546: 34616 notify*- 0/0/0 (28)
--
02:38:14.967803 IP (tos 0x0, ttl 54, id 36016, offset 0, flags [none],
proto UDP (17), length 126)
    96.86.170.226.63880 > 96.95.217.99.53: 31947 notify [b2&3=0x2400]
[1a] SOA? mpaoli.net. (98)
02:38:14.969472 IP (tos 0x0, ttl 64, id 56922, offset 0, flags [none],
proto UDP (17), length 56)
    96.95.217.99.53 > 96.86.170.226.63880: 31947 notify*- 0/0/0 (28)
^C534 packets captured
537 packets received by filter
0 packets dropped by kernel

#
So ... not seeing surprises with that traffic (other than notifies too
frequently sent on my side).

As for 73.189.65.18 ... well, let's see ...
# tcpdump -i enp6s0 -n -s 0 -v \( port 53 or port 5353 \) and host 73.189.65.18
tcpdump: listening on enp6s0, link-type EN10MB (Ethernet), snapshot
length 262144 bytes
^C
0 packets captured
0 packets received by filter
0 packets dropped by kernel
#
Waited fair while and ... really not seeing anything there currently.
Trying bit more generally ...
# tcpdump -i enp6s0 -n -s 0 -v port 5353
... yeah, that captured too much (irrelevant IPs and various false
positives) ...
# tcpdump -i enp6s0 -n -s 0 -v port 5353 and net 96.80.0.0/12
meanwhile on [ns1.]linuxmafia.com.:
$ dig -p 5353 +nomultiline +notcp +ignore @96.86.170.229 +noall
+answer sflug.com. SOA
;; connection timed out; no servers could be reached
$
And our capture then has:
# tcpdump -i enp6s0 -n -s 0 -v port 5353 and net 96.80.0.0/12
tcpdump: listening on enp6s0, link-type EN10MB (Ethernet), snapshot
length 262144 bytes
03:09:16.695708 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto
UDP (17), length 55)
    96.95.217.99.42136 > 96.86.170.229.5353: 62654+ SOA (QM)? sflug.com. (27)
03:09:21.696282 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto
UDP (17), length 55)
    96.95.217.99.42136 > 96.86.170.229.5353: 62654+ SOA (QM)? sflug.com. (27)
03:09:26.696984 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto
UDP (17), length 55)
    96.95.217.99.42136 > 96.86.170.229.5353: 62654+ SOA (QM)? sflug.com. (27)
So ... client sends, server receives and responds, client never
receives response.
Other Internet clients don't have this issue.
And if same client does that query using port 53 or protocol TCP, then
issue isn't seen.

As for 73.189.65.18, that may take more digging/persistence/investigation to
find/know what's up with that.

On Mon, Jun 3, 2024 at 3:29 PM Rick Moen <rick@linuxmafia.com> wrote:
>
> OK, those fsckers are _definitely_ screwing with my DNS, again.
>
> See that "73.189.65.18" IP?  Well...
>
> $ dig -x 73.189.65.18 +short
> c-73-189-65-18.hsd1.ca.comcast.net.
> $
>
> I'm betting this also somehow accounts for my mameserver mysteriously
> getting NOTIFYs for domain mpaoli.net every 90 seconds.
>
>
> ----- Forwarded message from logcheck system account <logcheck@linuxmafia.com> -----
>
> Date: Mon, 03 Jun 2024 15:02:02 -0700
> From: logcheck system account <logcheck@linuxmafia.com>
> To: root@linuxmafia.com
> Subject: linuxmafia.com 2024-06-03 15:02 System Events
>
> System Events
> =-=-=-=-=-=-=
> Jun  3 14:48:39 linuxmafia named[15622]: client 73.189.65.18#16833: received notify for zone 'mpaoli.net'
> Jun  3 14:48:39 linuxmafia named[15622]: zone mpaoli.net/IN: refused notify from non-master: 73.189.65.18#16833
> Jun  3 14:48:44 linuxmafia named[15622]: client 73.189.65.18#16833: received notify for zone 'mpaoli.net'
> Jun  3 14:48:44 linuxmafia named[15622]: zone mpaoli.net/IN: refused notify from non-master: 73.189.65.18#16833
>
> ----- End forwarded message -----