Okay, fixed[1]. I'll dig[2] into it later to investigate how issue did or likely occurred.
Thanks Rick for catching that!
footnotes/references/excerpts: 1. For certain definitions of "fixed" - serial numbers corrected, didn't check/validate anything else in particular - just "bumped" (updated) them and "pushed" (notified) 'em out, and rechecked 'till they all appeared out there okay on master(s) & slaves. $ DNS_SOA_CK balug.org sf-lug.org FQDN=balug.org. authority: balug.org. 86400 IN NS ns1.balug.org. balug.org. 86400 IN NS ns1.linuxmafia.com. balug.org. 86400 IN NS ns1.svlug.org. balug.org. 86400 IN NS puck.nether.net. balug.org. IN SOA ns1.balug.org. hostmaster.balug.org. 1559249859 9000 1800 1814400 86400 @198.144.194.238 (ns1.balug.org.) balug.org. IN SOA ns1.balug.org. hostmaster.balug.org. 1559249859 9000 1800 1814400 86400 @2001:470:1f04:19e::2 (ns1.balug.org.) balug.org. IN SOA ns1.balug.org. hostmaster.balug.org. 1559249859 9000 1800 1814400 86400 @198.144.195.186 (ns1.linuxmafia.com.) balug.org. IN SOA ns1.balug.org. hostmaster.balug.org. 1559249859 9000 1800 1814400 86400 @64.62.190.98 (ns1.svlug.org.) balug.org. IN SOA ns1.balug.org. hostmaster.balug.org. 1559249859 9000 1800 1814400 86400 @2600:3c01::f03c:91ff:fe96:e78e (ns1.svlug.org.) balug.org. IN SOA ns1.balug.org. hostmaster.balug.org. 1559249859 9000 1800 1814400 86400 @204.42.254.5 (puck.nether.net.) balug.org. IN SOA ns1.balug.org. hostmaster.balug.org. 1559249859 9000 1800 1814400 86400 @2001:418:3f4::5 (puck.nether.net.) FQDN=sf-lug.org. authority: sf-lug.org. 86400 IN NS ns.primate.net. sf-lug.org. 86400 IN NS ns1.linuxmafia.com. sf-lug.org. 86400 IN NS ns1.sf-lug.org. sf-lug.org. 86400 IN NS ns1.svlug.org. sf-lug.org. IN SOA ns1.sf-lug.org. jim.well.com. 1559249874 10800 3600 1209600 86400 @198.144.194.12 (ns.primate.net.) sf-lug.org. IN SOA ns1.sf-lug.org. jim.well.com. 1559249874 10800 3600 1209600 86400 @2001:470:1f04:51a::2 (ns.primate.net.) sf-lug.org. IN SOA ns1.sf-lug.org. jim.well.com. 1559249874 10800 3600 1209600 86400 @198.144.195.186 (ns1.linuxmafia.com.) sf-lug.org. IN SOA ns1.sf-lug.org. jim.well.com. 1559249874 10800 3600 1209600 86400 @198.144.194.238 (ns1.sf-lug.org.) sf-lug.org. IN SOA ns1.sf-lug.org. jim.well.com. 1559249874 10800 3600 1209600 86400 @2001:470:1f04:19e::2 (ns1.sf-lug.org.) sf-lug.org. IN SOA ns1.sf-lug.org. jim.well.com. 1559249874 10800 3600 1209600 86400 @64.62.190.98 (ns1.svlug.org.) sf-lug.org. IN SOA ns1.sf-lug.org. jim.well.com. 1559249874 10800 3600 1209600 86400 @2600:3c01::f03c:91ff:fe96:e78e (ns1.svlug.org.) $ 2. pun or the like retroactively intended. ;-) 3. unreferenced footnote. My DNS_SOA_CK program essentially grabs the "upstream" delegating NS records (at least one instance of 'em), gets the A and AAAA records of all the delegated nameservers, then gets the SOA records from each of those, and displays that data, for the domain(s) specified - or a default set if none specified. (I wrote it semi-recently - I got tired of doing it semi-manually on a semi-frequent basis; very handy for, among other things, also checking that master(s) and slaves are caught up when going through letsencrypt.org wildcard cert validation request via DNS verification; also very handy to see that the delegated nameservers are responding and with the expected data (or at least expected zone S/N)).
From: "Michael Paoli" Michael.Paoli@cal.berkeley.edu Subject: Re: [BALUG-Admin] S/N bobble -- waiting for Michael P. on this 8-O Date: Thu, 30 May 2019 13:50:37 -0700
From: "Rick Moen" rick@linuxmafia.com Subject: [BALUG-Admin] S/N bobble -- waiting for Michael P. on this Date: Thu, 30 May 2019 11:20:26 -0700
Magic 8-ball (i.e., logcheck on ns1.linuxmafia.com) says:
System Events
May 30 10:05:36 linuxmafia named[11750]: zone balug.org/IN: serial number (1558725628) received from master 198.144.194.238#53 < ours (1558799284) May 30 10:30:43 linuxmafia named[11750]: zone sf-lug.org/IN: serial number (1558622463) received from master 198.144.194.238#53 < ours (1558799278) May 30 10:32:42 linuxmafia named[11750]: zone balug.org/IN: serial number (1558725628) received from master 198.144.194.238#53 < ours (1558799284) May 30 10:57:43 linuxmafia named[11750]: zone balug.org/IN: serial number (1558725628) received from master 198.144.194.238#53 < ours (1558799284)
Er?
Michael, O Great Oracle of the DNS master, before I go expunging the local cached zone on ns1.linuxmafia.com so as to converge in the master, any thoughts or desire to act on your end? Normally, I would expect the current situation to be _strenuously avoided_ by never taking S/Ns in a retrograde direction on a zone's DNS master, so I infer that investigation may be in order (or at least brief discussion).
8-O Opps, ... that should'a never happened. I'll investigate & correct. Shouldn't require any explicit slave action.
I wonder if maybe VM came up that shouldn't have, or ??? Anyway, will check into it and correct (might be busy mostly with other stuff 'till about this evening or so, but expect I'll have it rectified by/around then ... might then take wee bit for slaves to follow along & get themselves corrected - but likely pretty fast on that and without explicit slave action needed).