So ... wee bit of reference introduction, then my tests/findings/recommendations/etc., then more of the earlier referenced bits.
From: "Michael Paoli" Michael.Paoli@cal.berkeley.edu Subject: Re: [BALUG-Admin] DNS slaves for BALUG? :-) ... IPv6 issue somewhere between master and slaves? Date: Sat, 20 Feb 2016 18:52:14 -0800
No problem, I'll take a look, and let you know what I find.
Quite likely it's lack of IPv6 or some other IPv6 issue on slave end(s), but I'll do some further checking to see if I can isolate it to that (at least for ns1.linuxmafia.com. - I'm presuming same/similar for ns1.svlug.org.), and see whether or not zone can be pulled via IPv6 - and beyond the IPv6 addresses where I've done so on my home networks (all of which happen to be via same ISP for IPv6 - so would be good to check from some other(s) - I think I may have a location or two (or three or more?) that I may be able to check that from).
Thanks, I'll let you know.
So ... most of the testing details first. Then past that section, recommendations for the very specific slaves in question - based upon those test results. Then there's bit more information/comments after that. And then a wee bit more additional testing.
$ dig +short ns1.linuxmafia.com. A ns1.linuxmafia.com. AAAA 198.144.195.186 $ ssh -ax linuxmafia.com. 'umask 077 && hostname &&
PATH=/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin:"$PATH" \ && { { >>/dev/null 2>&1 type ip && ip -4 a s; ip -6 a s; ip -6 r s; \ } || { ifconfig -a; netstat -nr; } }'
linuxmafia.com 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN inet 127.0.0.1/8 scope host lo 3: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 inet 198.144.195.186/29 brd 198.144.195.191 scope global eth2 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 inet6 ::1/128 scope host valid_lft forever preferred_lft forever 3: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qlen 1000 inet6 fe80::220:edff:fe13:ba89/64 scope link valid_lft forever preferred_lft forever fe80::/64 dev eth2 proto kernel metric 256 mtu 1500 advmss 1440 hoplimit 0 $
So ... [ns1.]linuxmafia.com. has an IPv6 stack and IPv6 link-local apparently configured, but no IPv6 route for anything beyond link-local and localhost. Let's see if I can confirm that ...
$ ssh -ax linuxmafia.com. 'umask 077 && hostname && \
PATH=/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin:"$PATH" \ && ping6 -n -c 5 2001:470:1f04:19e::2'
linuxmafia.com connect: Network is unreachable $
Yes, can't get to 2001:470:1f04:19e::2 from [ns1.]linuxmafia.com. 198.144.195.186 ::1/128 fe80::/64 without some type of route to 2001:470:1f04:19e::2 from that host. the: connect: Network is unreachable probably indicates we're getting an ICMP network unreachable, probably from the host itself, as it has no available route to get to 2001:470:1f04:19e::2 The diagnostic bit(s): failure trying master 2001:470:1f04:19e::2#53 (source ::#0): operation canceled would be, I'd guess, the nameserver's software's way of indicating it can't get to there or is otherwise failing - the "(source ::#0)" may be some kind of hint as to source IP that it's trying from or that it's trying that from IPv6, and knows not how to get there, or fails from that source IP, or IPv6 in general. Let me see if I can pull the zone via IPv6 to elsewhere ...
$ ip -6 a s | fgrep inet6; ip -6 r s | fgrep via inet6 ::1/128 scope host inet6 fe80::221a:6ff:fe03:c207/64 scope link inet6 2001:470:66:76f::2/64 scope global inet6 fe80::c690:c2eb/64 scope link inet6 fe80::fc54:ff:fe13:5199/64 scope link default via 2001:470:66:76f::1 dev he-ipv6 metric 1024 $ dig @2001:470:1f04:19e::2 -t AXFR \
e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa. +norecurse +noall +nocmd \ +nocomments +answer | grep '^[ ]*[^ ;]'
e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa. 7200 IN SOA ns1.balug.org. postmaster.balug.org. 1456011000 10800 3600 1209600 3600 e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa. 7200 IN NS ns0.mpaoli.net. e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa. 7200 IN NS ns1.balug.org. e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa. 7200 IN NS ns1.svlug.org. e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa. 7200 IN NS ns1.linuxmafia.com. 2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa. 7200 IN PTR balug.org. e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa. 7200 IN SOA ns1.balug.org. postmaster.balug.org. 1456011000 10800 3600 1209600 3600 $
Yes, I'm able to pull the zone (master is specifically configured: allow-transfer { any; }; for at least the [L]UG zone(s) - and that includes e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa. but the client IPv6 I pulled from is from same ISP/provider as the server pulled from. Do I have another IPv6 independent of that ISP that I can check from?
$ ssh -ax m-net.arbornet.org. hostname ssh: connect to host m-net.arbornet.org. port 22: No route to host $ Hmmmm... $ (cd ~/tmp && stat -c '%y %n' .m-net.arbornet.org* && cat
.m-net.arbornet.org.lastlogin)
2016-02-10 00:05:03.690977780 -0800 .m-net.arbornet.org.lastlogin michaelp pts/7 Feb 10 03:05 (198.144.194.235) Connection to m-net.arbornet.org. closed. $ I've got a crontab job:
$ crontab -l | fgrep m-net.arbornet.org | grep -v '^#' 5 8 * * * TZ=PST8PDT export TZ; set -e; trap 'trap - 0; exit 0' 0; exec </dev/null >>/dev/null 2>&1; cd tmp; lastlogin=.m-net.arbornet.org.lastlogin lastoutput=.m-net.arbornet.org.lastoutput; set +e; tmp=`find "$lastlogin" -prune ! -mtime +15 -type f -print`; [ x"$tmp" = x"$lastlogin" ] || { </dev/null >"$lastoutput" 2>&1 ssh -a -n -t -t -x -o BatchMode=yes m-net.arbornet.org. 'umask 077 && exec who am I' && mv -f "$lastoutput" "$lastlogin"; } $
that keeps my account there from going away (I think it has to be logged into once every 30 days to not get removed). My crontab job, daily, makes a login attempt if its last successful login was more than 15 days ago. Looks like it worked rather recently ... but not working at the moment.
So, another ISP's IPv6 ... ignore some of the despicable operating system bits below ... but at least with Cygwin it's almost bearable ... testing from a work connection (hey, fair's fair, often use home connections from work to do work to, e.g. test various accessibility from The Internet, etc.) ...
$ IPCONFIG /ALL | fgrep IPv6\ Address | fgrep -v Link-local IPv6 Address. . . . . . . . . . . : 2600:1010:b161:5e74:7c7a:2cff:fe94:2fa1(Preferred) $ dig @2001:470:1f04:19e::2 -t AXFR \
e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa. +norecurse +noall +nocmd \ +nocomments +answer | grep '^[ ]*[^ ;]'
/usr/src/ports/bind/bind-9.10.2-2.P2.x86_64/src/bind-9.10.2-P2/lib/isc/unix/socket.c:2867: setsockopt(20, IPV6_RECVTCLASS) failed: Invalid argument e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa. 7200 IN SOA ns1.balug.org. postmaster.balug.org. 1456011000 10800 3600 1209600 3600 e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa. 7200 IN NS ns0.mpaoli.net. e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa. 7200 IN NS ns1.balug.org. e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa. 7200 IN NS ns1.svlug.org. e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa. 7200 IN NS ns1.linuxmafia.com. 2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa. 7200 IN PTR balug.org. e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa. 7200 IN SOA ns1.balug.org. postmaster.balug.org. 1456011000 10800 3600 1209600 3600 $
Okay, enough of that, ... set *that* operating system aside ... now back to a *real* operating system :-> ... And to look at ISPs to see/show they're different, at least to the extent can check with whois(1), we have ... First the master:
$ 2>&1 whois -H 2001:470:1f04:19e::2 |
egrep -i '^(netname|nethandle|netrange|cidr|organization)'
NetRange: 2001:470:: - 2001:470:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF CIDR: 2001:470::/32 NetName: HURRICANE-IPV6 NetHandle: NET6-2001-470-1 Organization: Hurricane Electric, Inc. (HURC) $
Then the first global routable IPv6 I tried from (expecting same ISP in whois):
$ 2>&1 whois -H 2001:470:66:76f::2 |
egrep -i '^(netname|nethandle|netrange|cidr|organization)'
NetRange: 2001:470:: - 2001:470:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF CIDR: 2001:470::/32 NetName: HURRICANE-IPV6 NetHandle: NET6-2001-470-1 Organization: Hurricane Electric, Inc. (HURC) $
No surprises there - it's also part of same NetRange / CIDR, as we can see in even the first set of whois data. And last client I tried, I'm hoping/expecting different ISP (is a big telecom company after all) ...
$ 2>&1 whois -H 2600:1010:b161:5e74:7c7a:2cff:fe94:2fa1 |
egrep -i '^(netname|nethandle|netrange|cidr|organization)'
NetRange: 2600:1000:: - 2600:1017:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF CIDR: 2600:1010::/29, 2600:1000::/28 NetName: WIRELESSDATANETWORK NetHandle: NET6-2600-1000-1 Organization: Cellco Partnership DBA Verizon Wireless (CLLC) $
And yes, an ISP, that as far as I'm aware, is quite independent of the other ISP shown for the two other IPv6 addresses.
So ... recommendations ... I'm not sure what DNS slave server software is in use but for the masters, if the IPv6 addresses are listed after IPv4 (which may already be the case) and it still complains, my recommendation would be to comment out or otherwise disable the IPv6 master address(es) until such time as the slave hosts have some type of routable Internet IPv6 connectivity.
$ dig +short ns1.svlug.org. A ns1.svlug.org. AAAA 64.62.190.98 $
I'd guess issue is same for ns1.svlug.org. - I don't see any IPv6 addresses there, but I don't have access to the host, as far as I'm aware, so am not in position to test more fully from that slave host. So, if same or quite similar applies to that slave host, then same recommendation would apply.
I'm also guestimating, the DNS slave server software likely checks *all* masters - I'd expect that - probably starts with SOA, and then grabs "highest" (per the DNS RFCs) SOA serial number zone successfully found from master it successfully did SOA query on. It may or may not have some fallback to try AXFR if SOA fails, or if SOA succeeds but AXFR fails, there's probably some retry/failover algorithm it uses among the master(s) configured for that slave for that zone. So, it probably at least *tries* all configured masters, and depending upon the software, and possible configuration and logging levels, it may complain for any it fails to get the data from, or I'd think at least most minimally, if it failed with all masters - but I'm guessing it would more likely complain if it failed with *any* of the masters, as one may generally want to at least minimally have that logged, for possible investigation/action (expected event/maintenance, or known outage, or ... some other issue/problem)?
Miscellaneous comments :-) : IPv6 - random John and Jane Doe consumer might not yet know or care (and may never know, or particularly care), but The Internet is (slowly) going the way of IPv6. IPv4 won't disappear anytime soon. However, with the exhaustion of IPv4 addresses, increasingly clients are not only dual-stack (or have some funky way to still be able to access IPv4), but increasingly there will be clients much prefer IPv6 address and/or do or will only have IPv6 access. IPv6 adoption has also been growing at a relatively fast rate. If I recall correctly, last year IPv6 growth was up something like 10%. I remember even about 4 years ago, looking at DNS traffic of a major provider, AAAA queries were up to nearly 2% of volume of queries, relative to A+AAAA - that was up about 300% compared to only about a year or two earlier. Not sure about similar current stats, but in general, any such clients that are IPv6 only, those would typically be lost traffic/customers, or they may be stuck with some less preferred means of IPv4 access. We're well past the time where at least most all major providers should also be offering IPv6 connectivity (e.g. web sites, etc.). The big players in ISPs space all (or nearly all?) do IPv6, some of the smaller ISPs are still getting there (and even some of the colos aren't fully there yet). But in any case, IPv6 is coming, and one should prepare and get ready for it (but not an extreme rush for everyone - if you're domestic only web site with only domestic users, may not be much of a rush to use IPv6 ... but dang, it certainly is also nice to not be squeezed with very limited number of IPv4 addresses to work with).
Oh, if one doesn't have IPv6 available from one's ISP, there are IPv6 tunnel brokers, so one can use IPv6 over IPv4 - and thus reach the IPv6 Internet. E.g. Hurricane Electric provides such - for free (at least as in beer). They also have a lot of excellent IPv6 training materials (though a bit dated now, it's still mostly quite applicable). Anyway, I do have such IPv6 tunnels from Hurricane Electric (alas, my home ISP isn't yet offering native IPv6 ... I did also notice my work cellular, didn't have IPv6 only a few months or so ago, but a month or two after that, had added IPv6). Anyway, have an IPv6 tunnel for myself. Also have one for BALUG (I did a separate account for BALUG, so it's quite independent of my personal account). From Hurricane Electric: https://ipv6.he.net/certification/ https://tunnelbroker.net/ They also offer some free DNS services ... but they may be slightly funky in how they behave, and also somewhat limited, in at least some regards: https://dns.he.net/ But those do also include IPv6 nameservers.
Also, in the ssh examples above, I didn't include bothering to show use of ssh-add - generally used in advance to decrypt private key(s) and then temporarily (for specified length of time) hold the decrypted private key(s) in RAM - best to never have the keys in the clear written to non-volatile storage (e.g. disk or SSD or similar). The ssh-add program also takes care to not let private keys be written to swap - but I also have my swap encrypted, and likewise most of my drive storage.
And in the above regular expressions where there's character class ([...]) showing whitespace within, the whitespace is generally a space and a tab (my email client doesn't conveniently allow me to literally pass that along - and even if I did, clients displaying such may not preserve such in display, and in any case it wouldn't generally be visually obvious). Some RE contents allow \t for tab, but those I showed above generally don't support that.
And retrying earlier test: $ ssh -ax m-net.arbornet.org. hostname m-net.arbornet.org $
That works now, so trying further with that ...
$ ssh -ax m-net.arbornet.org. 'hostname &&
PATH="/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin:$PATH" \ && { if >>/dev/null 2>&1 type ip; then ip -6 a s; ip -6 r s else ifconfig -a; netstat -nr; fi; }'
m-net.arbornet.org em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM> ether 00:50:56:8b:48:cf inet 162.202.67.157 netmask 0xffffffe0 broadcast 162.202.67.159 inet6 fe80::250:56ff:fe8b:48cf%em0 prefixlen 64 scopeid 0x1 nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> media: Ethernet autoselect (1000baseT <full-duplex>) status: active em1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM> ether 00:0c:29:34:2e:d3 nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> media: Ethernet autoselect status: no carrier plip0: flags=8810<POINTOPOINT,SIMPLEX,MULTICAST> metric 0 mtu 1500 nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=3<RXCSUM,TXCSUM> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4 inet 127.0.0.1 netmask 0xff000000 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> ipfw0: flags=8801<UP,SIMPLEX,MULTICAST> metric 0 mtu 65536 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> Routing tables
Internet: Destination Gateway Flags Refs Use Netif Expire default 162.202.67.129 UGS 0 4573 em0 127.0.0.1 link#4 UH 0 47170 lo0 162.202.67.128/27 link#1 U 0 127 em0 162.202.67.157 link#1 UHS 0 0 lo0
Internet6: Destination Gateway Flags Netif Expire ::/96 ::1 UGRS lo0 ::1 ::1 UH lo0 ::ffff:0.0.0.0/96 ::1 UGRS lo0 fe80::/10 ::1 UGRS lo0 fe80::%em0/64 link#1 U em0 fe80::250:56ff:fe8b:48cf%em0 link#1 UHS lo0 fe80::%lo0/64 link#4 U lo0 fe80::1%lo0 link#4 UHS lo0 ff01::%em0/32 fe80::250:56ff:fe8b:48cf%em0 U em0 ff01::%lo0/32 ::1 U lo0 ff02::/16 ::1 UGRS lo0 ff02::%em0/32 fe80::250:56ff:fe8b:48cf%em0 U em0 ff02::%lo0/32 ::1 U lo0 $
I don't think I see anything there that's globally routable IPv6 ff0x::114 Used for experiments ... yeah, I was wondering what ff01::... is/was ... wasn't able to identify that. Well, let's try wee bit:
$ ssh -ax m-net.arbornet.org. 'hostname &&
PATH="/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin:$PATH" \ && >>/dev/null 2>&1 type ping6 && ping6 -n -c 5 2001:470:1f04:19e::2'
m-net.arbornet.org ping6: UDP connect: No route to host $
Yep, it doesn't have a route to get there, so can't test further from there.
From: "Rick Moen" rick@linuxmafia.com Subject: Re: [BALUG-Admin] DNS slaves for BALUG? :-) Date: Sat, 20 Feb 2016 18:33:09 -0800
I wrote:
Quoting Michael Paoli (Michael.Paoli@cal.berkeley.edu):
If you could please, and would be willing, could you cover DNS slave services for BALUG, notably these zones: e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa balug.org master(s) (all for each of the above): 198.144.194.238 2001:470:1f04:19e::2
FYI, ns1.linuxmafia.com is repeatedly logging this:
Feb 20 17:50:45 linuxmafia named[17291]: zone e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa/IN: refresh: failure trying master 2001:470:1f04:19e::2#53 (source ::#0): operation canceled
I'm unsure why the AXFR/IXFR to the IPv6 address fails, and currently lack the time and patience to pursue that matter. For now, I'm going to disable that master IP on my BIND9 conffile.
Before I do that, Michael, would you like to any some checking on the _master_ nameserver end? The most probable explanation is some deficiency in IPv6 support on ns1.linuxmafia.com, and that would be my default assumption.
FYI, both zones are affected:
linuxmafia:/etc/bind# zgrep 'failure trying master' /var/log/* /var/log/daemon.log:Feb 20 12:55:59 linuxmafia named[17291]: zone balug.org/IN: refresh: failure trying master 2001:470:1f04:19e::2#53 (source ::#0): operation canceled /var/log/daemon.log:Feb 20 13:39:16 linuxmafia named[17291]: zone e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa/IN: refresh: failure trying master 2001:470:1f04:19e::2#53 (source ::#0): operation canceled /var/log/daemon.log:Feb 20 17:50:45 linuxmafia named[17291]: zone e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa/IN: refresh: failure trying master 2001:470:1f04:19e::2#53 (source ::#0): operation canceled /var/log/syslog:Feb 20 12:55:59 linuxmafia named[17291]: zone balug.org/IN: refresh: failure trying master 2001:470:1f04:19e::2#53 (source ::#0): operation canceled /var/log/syslog:Feb 20 13:39:16 linuxmafia named[17291]: zone e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa/IN: refresh: failure trying master 2001:470:1f04:19e::2#53 (source ::#0): operation canceled /var/log/syslog:Feb 20 17:50:45 linuxmafia named[17291]: zone e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa/IN: refresh: failure trying master 2001:470:1f04:19e::2#53 (source ::#0): operation canceled linuxmafia:/etc/bind#