Search results for "BALUG: Lists, stats, etc." - BALUG-Admin

BALUG: Lists, stats, etc.

by Michael Paoli

dates, lists, number of members: YYYY-MM-DD announce talk admin 2024-08-01 497 232 32 2024-07-01 497 232 32 2024-06-01 498 231 33 2024-05-01 499 231 33 2024-04-01 500 230 33 2024-03-01 498 230 33 2024-02-01 499 229 33 2024-01-01 497 227 31 2023-12-01 497 227 32 2023-11-01 496 249 32 2023-10-01 496 248 32 2023-09-01 497 248 32 2023-08-01 498 248 32 2023-07-01 500 248 32 2023-06-01 501 249 32 2023-05-01 501 249 32 2023-04-01 501 249 32 2023-03-01 501 249 32 2023-02-01 501 249 32 2023-01-01 500 249 32 2022-12-01 520 249 32 2022-11-01 564 249 32 2022-10-01 589 249 32 2022-09-01 589 249 33 2022-08-01 616 249 33 2022-07-01 834 260 35 2022-06-01 832 259 35 2022-05-01 832 259 35 2022-04-01 832 259 35 2022-03-01 833 259 35 2022-02-01 833 259 35 2022-01-01 833 258 35 2021-12-01 832 257 35 2021-11-01 831 257 35 2021-10-01 832 257 35 2021-09-01 832 257 35 2021-08-01 832 257 35 2021-07-01 832 257 35 2021-06-01 832 256 35 2021-05-01 833 257 35 2021-04-01 832 259 35 2021-03-01 833 259 35 2021-02-01 833 259 35 2021-01-01 833 259 35 2020-12-01 832 259 35 2020-11-01 833 260 35 2020-10-01 834 260 35 2020-09-01 835 260 35 2020-08-01 834 259 34 2020-07-01 834 266 34 2020-06-01 838 266 34 2020-05-01 839 267 34 2020-04-01 840 270 34 2020-03-01 840 271 34 2020-02-01 841 271 34 2020-01-01 841 271 34 2019-12-01 841 271 34 2019-11-01 840 271 34 2019-10-01 842 271 34 2019-09-01 844 272 34 2019-08-01 847 272 34 2019-07-01 852 272 36 2019-06-01 854 273 37 2019-05-01 856 272 38 2019-04-01 853 273 38 2019-03-01 851 273 38 2019-02-01 853 273 38 2019-01-01 856 274 38 2018-12-01 858 274 38 2018-11-01 861 274 39 2018-10-02 862 274 39 2018-09-01 864 274 38 2018-07-28 866 278 38 2018-06-08 870 282 38 2018-04-28 875 282 38 2018-02-17 880 283 36 2017-11-04 883 284 36 2017-08-19 897 293 37 2017-07-01 897 297 38 2017-05-27 897 297 38 2017-05-16 897 296 38 2017-04-01 896 297 38 2017-03-30 885 297 38 2016-05-03 886 291 39 2016-04-03 887 290 39 2016-03-05 888 291 45 2016-01-05 887 295 45 2015-12-05 887 296 45 2015-11-07 887 301 45 2015-10-06 888 301 45 2015-09-05 887 301 45 2015-08-04 888 301 45 2015-07-03 888 300 45 2015-06-05 888 301 46 2015-05-02 885 298 45 2015-04-04 884 297 44 2015-03-07 886 298 44 2015-02-06 885 303 44 2015-01-07 885 304 45 2014-12-06 886 304 45 2014-11-01 884 302 45 2014-10-04 881 303 46 2014-09-06 878 309 44 2014-06-13 875 307 44 2014-05-02 874 306 44 2014-04-04 876 306 44 2014-03-01 876 306 43 2014-01-11 872 305 42 2013-10-05 854 293 40 2013-08-31 855 295 41 2013-07-20 852 297 40 2013-04-10 838 286 41 2013-03-03 841 289 41 2013-02-02 843 291 41 2012-12-15 845 310 41 2012-03-23 818 310 39 2012-02-01 813 307 40 2011-04-28 768 291 37 2011-02-27 762 288 36 2010-05-02 730 289 34 2010-04-01 727 291 33 2010-03-16 727 293 33 2009-12-16 716 290 33 2009-09-09 700 288 32 2009-03-27 677 287 29 2009-01-10 662 290 30 2008-08-17 635 290 29 2008-03-03 571 247 28 2008-01-02 337 260 28 2007-12-28 336 261 28 2007-09-03 264 258 27 2007-07-24 287 256 27 2007-06-17 280 249 27 references: http://www.balug.org/#Lists

1 year

BALUG: Lists, stats, etc.

by Michael Paoli

dates, lists, number of members: YYYY-MM-DD announce talk admin 2025-08-01 459 235 31 2025-07-01 559 466 148 2025-06-01 556 447 146 2025-05-01 566 433 146 2024-08-01 497 232 32 2024-07-01 497 232 32 2024-06-01 498 231 33 2024-05-01 499 231 33 2024-04-01 500 230 33 2024-03-01 498 230 33 2024-02-01 499 229 33 2024-01-01 497 227 31 2023-12-01 497 227 32 2023-11-01 496 249 32 2023-10-01 496 248 32 2023-09-01 497 248 32 2023-08-01 498 248 32 2023-07-01 500 248 32 2023-06-01 501 249 32 2023-05-01 501 249 32 2023-04-01 501 249 32 2023-03-01 501 249 32 2023-02-01 501 249 32 2023-01-01 500 249 32 2022-12-01 520 249 32 2022-11-01 564 249 32 2022-10-01 589 249 32 2022-09-01 589 249 33 2022-08-01 616 249 33 2022-07-01 834 260 35 2022-06-01 832 259 35 2022-05-01 832 259 35 2022-04-01 832 259 35 2022-03-01 833 259 35 2022-02-01 833 259 35 2022-01-01 833 258 35 2021-12-01 832 257 35 2021-11-01 831 257 35 2021-10-01 832 257 35 2021-09-01 832 257 35 2021-08-01 832 257 35 2021-07-01 832 257 35 2021-06-01 832 256 35 2021-05-01 833 257 35 2021-04-01 832 259 35 2021-03-01 833 259 35 2021-02-01 833 259 35 2021-01-01 833 259 35 2020-12-01 832 259 35 2020-11-01 833 260 35 2020-10-01 834 260 35 2020-09-01 835 260 35 2020-08-01 834 259 34 2020-07-01 834 266 34 2020-06-01 838 266 34 2020-05-01 839 267 34 2020-04-01 840 270 34 2020-03-01 840 271 34 2020-02-01 841 271 34 2020-01-01 841 271 34 2019-12-01 841 271 34 2019-11-01 840 271 34 2019-10-01 842 271 34 2019-09-01 844 272 34 2019-08-01 847 272 34 2019-07-01 852 272 36 2019-06-01 854 273 37 2019-05-01 856 272 38 2019-04-01 853 273 38 2019-03-01 851 273 38 2019-02-01 853 273 38 2019-01-01 856 274 38 2018-12-01 858 274 38 2018-11-01 861 274 39 2018-10-02 862 274 39 2018-09-01 864 274 38 2018-07-28 866 278 38 2018-06-08 870 282 38 2018-04-28 875 282 38 2018-02-17 880 283 36 2017-11-04 883 284 36 2017-08-19 897 293 37 2017-07-01 897 297 38 2017-05-27 897 297 38 2017-05-16 897 296 38 2017-04-01 896 297 38 2017-03-30 885 297 38 2016-05-03 886 291 39 2016-04-03 887 290 39 2016-03-05 888 291 45 2016-01-05 887 295 45 2015-12-05 887 296 45 2015-11-07 887 301 45 2015-10-06 888 301 45 2015-09-05 887 301 45 2015-08-04 888 301 45 2015-07-03 888 300 45 2015-06-05 888 301 46 2015-05-02 885 298 45 2015-04-04 884 297 44 2015-03-07 886 298 44 2015-02-06 885 303 44 2015-01-07 885 304 45 2014-12-06 886 304 45 2014-11-01 884 302 45 2014-10-04 881 303 46 2014-09-06 878 309 44 2014-06-13 875 307 44 2014-05-02 874 306 44 2014-04-04 876 306 44 2014-03-01 876 306 43 2014-01-11 872 305 42 2013-10-05 854 293 40 2013-08-31 855 295 41 2013-07-20 852 297 40 2013-04-10 838 286 41 2013-03-03 841 289 41 2013-02-02 843 291 41 2012-12-15 845 310 41 2012-03-23 818 310 39 2012-02-01 813 307 40 2011-04-28 768 291 37 2011-02-27 762 288 36 2010-05-02 730 289 34 2010-04-01 727 291 33 2010-03-16 727 293 33 2009-12-16 716 290 33 2009-09-09 700 288 32 2009-03-27 677 287 29 2009-01-10 662 290 30 2008-08-17 635 290 29 2008-03-03 571 247 28 2008-01-02 337 260 28 2007-12-28 336 261 28 2007-09-03 264 258 27 2007-07-24 287 256 27 2007-06-17 280 249 27 references: http://www.balug.org/#Lists

1 month, 1 week

BALUG: Lists, stats, etc.

by Michael Paoli

dates, lists, number of members: YYYY-MM-DD announce talk admin 2025-09-01 459 235 31 2025-08-01 459 235 31 2025-07-01 559 466 148 2025-06-01 556 447 146 2025-05-01 566 433 146 2024-08-01 497 232 32 2024-07-01 497 232 32 2024-06-01 498 231 33 2024-05-01 499 231 33 2024-04-01 500 230 33 2024-03-01 498 230 33 2024-02-01 499 229 33 2024-01-01 497 227 31 2023-12-01 497 227 32 2023-11-01 496 249 32 2023-10-01 496 248 32 2023-09-01 497 248 32 2023-08-01 498 248 32 2023-07-01 500 248 32 2023-06-01 501 249 32 2023-05-01 501 249 32 2023-04-01 501 249 32 2023-03-01 501 249 32 2023-02-01 501 249 32 2023-01-01 500 249 32 2022-12-01 520 249 32 2022-11-01 564 249 32 2022-10-01 589 249 32 2022-09-01 589 249 33 2022-08-01 616 249 33 2022-07-01 834 260 35 2022-06-01 832 259 35 2022-05-01 832 259 35 2022-04-01 832 259 35 2022-03-01 833 259 35 2022-02-01 833 259 35 2022-01-01 833 258 35 2021-12-01 832 257 35 2021-11-01 831 257 35 2021-10-01 832 257 35 2021-09-01 832 257 35 2021-08-01 832 257 35 2021-07-01 832 257 35 2021-06-01 832 256 35 2021-05-01 833 257 35 2021-04-01 832 259 35 2021-03-01 833 259 35 2021-02-01 833 259 35 2021-01-01 833 259 35 2020-12-01 832 259 35 2020-11-01 833 260 35 2020-10-01 834 260 35 2020-09-01 835 260 35 2020-08-01 834 259 34 2020-07-01 834 266 34 2020-06-01 838 266 34 2020-05-01 839 267 34 2020-04-01 840 270 34 2020-03-01 840 271 34 2020-02-01 841 271 34 2020-01-01 841 271 34 2019-12-01 841 271 34 2019-11-01 840 271 34 2019-10-01 842 271 34 2019-09-01 844 272 34 2019-08-01 847 272 34 2019-07-01 852 272 36 2019-06-01 854 273 37 2019-05-01 856 272 38 2019-04-01 853 273 38 2019-03-01 851 273 38 2019-02-01 853 273 38 2019-01-01 856 274 38 2018-12-01 858 274 38 2018-11-01 861 274 39 2018-10-02 862 274 39 2018-09-01 864 274 38 2018-07-28 866 278 38 2018-06-08 870 282 38 2018-04-28 875 282 38 2018-02-17 880 283 36 2017-11-04 883 284 36 2017-08-19 897 293 37 2017-07-01 897 297 38 2017-05-27 897 297 38 2017-05-16 897 296 38 2017-04-01 896 297 38 2017-03-30 885 297 38 2016-05-03 886 291 39 2016-04-03 887 290 39 2016-03-05 888 291 45 2016-01-05 887 295 45 2015-12-05 887 296 45 2015-11-07 887 301 45 2015-10-06 888 301 45 2015-09-05 887 301 45 2015-08-04 888 301 45 2015-07-03 888 300 45 2015-06-05 888 301 46 2015-05-02 885 298 45 2015-04-04 884 297 44 2015-03-07 886 298 44 2015-02-06 885 303 44 2015-01-07 885 304 45 2014-12-06 886 304 45 2014-11-01 884 302 45 2014-10-04 881 303 46 2014-09-06 878 309 44 2014-06-13 875 307 44 2014-05-02 874 306 44 2014-04-04 876 306 44 2014-03-01 876 306 43 2014-01-11 872 305 42 2013-10-05 854 293 40 2013-08-31 855 295 41 2013-07-20 852 297 40 2013-04-10 838 286 41 2013-03-03 841 289 41 2013-02-02 843 291 41 2012-12-15 845 310 41 2012-03-23 818 310 39 2012-02-01 813 307 40 2011-04-28 768 291 37 2011-02-27 762 288 36 2010-05-02 730 289 34 2010-04-01 727 291 33 2010-03-16 727 293 33 2009-12-16 716 290 33 2009-09-09 700 288 32 2009-03-27 677 287 29 2009-01-10 662 290 30 2008-08-17 635 290 29 2008-03-03 571 247 28 2008-01-02 337 260 28 2007-12-28 336 261 28 2007-09-03 264 258 27 2007-07-24 287 256 27 2007-06-17 280 249 27 references: http://www.balug.org/#Lists

1 week, 5 days

Re: [BALUG-Admin] excess NOTIFY (about ever 90s) ISC BIND 9.18.24 ...

by Michael Paoli

So, haven't 100% gotten to the bottom of the excess NOTIFY, but it is apparently present in ISC BIND 9.18.24 and some other versions, and Debian bind9 1:9.18.24-1 which is the current version for the current stable Debian 12 bookworm. Also thinking that bug, for Debian stable, probably doesn't make it to >=important priority (probably just priority normal), so probably won't get fixed in the current stable. Thus far, this is what I've run across that seems most informative about it: https://gitlab.isc.org/isc-projects/bind9/-/issues/3242 So, workarounds? I'm not going to be upgrading that host from stable for quite some time (most likely year(s)), and other alternatives to fixing it on the server side is probably more additional overhead than I'm likely to want to bother with - so probably not going there. On the client side, can reconfigure logging to not log those bits, or possibly ignore them (e.g. by filter) in whatever notifications/reporting one is doing. So, yes, it is definitely coming from the nameserver itself, and all indications are it's a bug. I did see some chatter somewhere about it possibly being a but only if client doesn't respond in some way(s)? But that's not what I see in my testing on clients that do respond as expected, so I don't think that's it at all - at least in my case. Oh, as for possibly adjusting the logging on [ns1.]linuxmafia.com, I notice it's got bind9 version 1:9.7.1.dfsg.P2-2 installed, and yes, also has bind9-doc version 1:9.4.2-4, but that's different version. However, it does have, in /var/cache/apt/archives/ bind9-doc_9.7.1.dfsg.P2-2_all.deb So, that would be the corresponding documentation package, which could be installed, or the files simply extracted from that and consulted as may be desired. Anyway, within that, around: doc/bind9-doc/arm/Bv9ARM.ch06.html#id2575303 It has information about configuring logging, what the defaults are and the category notify in logging configuration that one can use to adjust what one does/doesn't want done regarding logging of notify activity. And, along the way, I've also rather well updated: Debian wiki: DNSSEC Howto for BIND 9.9+: https://wiki.debian.org/DNSSEC%20Howto%20for%20BIND%209.9+ And also well whipping mpaoli.net (and nameserver in general) configuration into shape for the current Debian stable. If I find anything else particularly noteworthy about the notify situation, I'll update. On Tue, Jun 4, 2024 at 4:21 PM Michael Paoli <michael.paoli(a)berkeley.edu> wrote: > > There's two different issues (well, three, but 3rd is minor), not to > be confused: > > The second - unrelated - issue. I've got excess notifies being sent > from my server(s). > Now, not at all unusual that sometimes there may be bursts of activity > and updates and such, > at least some of that is to be expected. > What's quite unexpected and ought not be happening, but is, > is I've got server sending frequent (e.g. like every 90 seconds-ish - > or so) notifies > when basic zone DNS data hasn't changed - same SOA serial, etc. > So, something's still amis with that which I need to correct (doesn't > really break any > functionalities ... but rather wasteful all those excess notifies being sent, > also tends to cause needless traffic on both sending side, > and receiving servers checking and finding they've got nothing > to do because they're already current). > And, as I'd earlier mentioned, I suspect from some major upgrades I did > a few weeks ago ... well, that was off-list, so, let me include fair > bit more of it here > (but not the whole thing, it gets very long, especially with all the > earlier referenced): > ----- > ---------- Forwarded message --------- > From: Michael Paoli <michael.paoli(a)berkeley.edu> > Date: Mon, Jun 3, 2024 at 11:19 PM > Subject: Re: [BALUG-Admin] (forw) linuxmafia.com 2024-06-03 15:02 System Events > To: Al <aw009(a)sunnyside.com> > Cc: Rick Moen <rick(a)linuxmafia.com> > > Yeah, ... something's still not quite right there. > Did upgrade on that host/server a couple weekends ago > Debian: buster 10 --> bullseye 11 --> bookworm 12 > $ TZ=GMT0 stat -c '%z %Z %n' /etc/debian_version > 2024-05-14 07:37:15.000000000 +0000 1715672235 /etc/debian_version > $ > Some of ye olde(er) DNS configuration options, while still > (marginally?) functional, are quite deprecated, and I'm getting > warnings on that and one other minor issue ... anyway, need to do > some cleanup/reconfig with DNS server on that host. > Should all be fine after that. Functional in the meantime, but ... > not yet optimal ... clearly. > > On Mon, Jun 3, 2024 at 10:43 PM Al <aw009(a)sunnyside.com> wrote: > > > > I'm probably not reading those logs right but part of it looked like mpaoli was being signed every minute? Only partial logs, not enough to be sure. 7:28, 7:29. > > Just a thought... > > > > On June 3, 2024 20:15:06 Michael Paoli via BALUG-Admin <balug-admin(a)lists.balug.org> wrote: > > > >> Well, under some circumstances, I'd expect a relative flury, or > >> "bursts" of notify > >> activity. E.g. I do TLS(/"SSL") certs via LetsEncrypt.org (LE), using DNS > >> verification. Some/many of those certs will have multiple > >> Subject Alternative Name (SAN) domains, and there may be fair number > >> of such certs. So, adding the relevant records for verification, and then > >> removing them after having obtained the issued cert(s) - that can be a > >> fair burst of activity. But those generally wouldn't persist over a long > >> period of time. E.g. every 90 seconds over many hours or days > >> or more ... then something else is going on. So ... let me dig > >> (pun not quite intended but apt) a bit more and peek what's currently > >> goin' on there ... > ... > ----- End forwarded message ----- > ----- > I suspect it has to do with some changes in how DNSSEC is configured, > most notably a configuration syntax I'm using is (now) quite deprecated, > and it may be thus defaulting to some rather undesirable behavior(s) > which may include triggering the excess notifies. > Anyway, want to test it carefully and not break things, that's why I'm not > willy-nilly throwing things at it. Goals being, to not only fix it, but if > feasible without causing further breakage along the way, and also not > disabling DNSSEC at any point (shouldn't need to) ... and while I'm at > it, also reviewing and checking/updating earlier documentation (much of which > I wrote): > Debian wiki: DNSSEC Howto for BIND 9.9+: > https://wiki.debian.org/DNSSEC%20Howto%20for%20BIND%209.9+ > And, along the way, I've also got temporary zone and subdomain I'm > fiddling with: tmp.mpaoli.net. to test things, etc. Anyway, it's still a work > in progress to get that all squared away (get issue fixed, not further > break things, > and update relevant documentation). > Anyway, as far as I'm aware, the excess notifies being sent don't have > anything to do with the Comcast issues. > > On Tue, Jun 4, 2024 at 3:12 PM Rick Moen <rick(a)linuxmafia.com> wrote: > > > > More of same, involving back-to-back NOTIFY traffic about mpaoli.net, > > allegedly originating from master nameserver 96.86.170.226 . Michael, I > > greatly doubt your nameserver is doing this. mpaoli.net's zonefile is > > pretty unchanging. I suspect Comcast middleman fsckery. Thoughts? > > > > ----- Forwarded message from logcheck system account <logcheck(a)linuxmafia.com> ----- > > > > Date: Tue, 04 Jun 2024 02:02:01 -0700 > > From: logcheck system account <logcheck(a)linuxmafia.com> > > To: root(a)linuxmafia.com > > Subject: linuxmafia.com 2024-06-04 02:02 System Events > > > > System Events > > =-=-=-=-=-=-= > > Jun 4 01:47:15 linuxmafia named[15622]: zone mpaoli.net/IN: Transfer started. > > Jun 4 01:47:15 linuxmafia named[15622]: transfer of 'mpaoli.net/IN' from 96.86.170.226#53: connected using 96.95.217.99#52012 > > Jun 4 01:47:15 linuxmafia named[15622]: zone mpaoli.net/IN: transferred serial 1717468800 > > Jun 4 01:47:15 linuxmafia named[15622]: transfer of 'mpaoli.net/IN' from 96.86.170.226#53: Transfer completed: 1 messages, 6 records, 546 bytes, 0.064 secs (8531 bytes/sec) > > Jun 4 02:00:09 linuxmafia named[15622]: zone mpaoli.net/IN: Transfer started. > > Jun 4 02:00:09 linuxmafia named[15622]: transfer of 'mpaoli.net/IN' from 96.86.170.226#53: connected using 96.95.217.99#57758 > > Jun 4 02:00:09 linuxmafia named[15622]: zone mpaoli.net/IN: transferred serial 1717491608 > > Jun 4 02:00:09 linuxmafia named[15622]: transfer of 'mpaoli.net/IN' from 96.86.170.226#53: Transfer completed: 1 messages, 15 records, 1266 bytes, 0.072 secs (17583 bytes/sec)

1 year, 3 months

Re: [BALUG-Admin] DNS slaves for BALUG? :-) ... IPv6 issue somewhere between master and slaves?

by Michael Paoli

So ... wee bit of reference introduction, then my tests/findings/recommendations/etc., then more of the earlier referenced bits. > From: "Michael Paoli" <Michael.Paoli(a)cal.berkeley.edu> > Subject: Re: [BALUG-Admin] DNS slaves for BALUG? :-) ... IPv6 issue > somewhere between master and slaves? > Date: Sat, 20 Feb 2016 18:52:14 -0800 > No problem, I'll take a look, and let you know what I find. > > Quite likely it's lack of IPv6 or some other IPv6 issue > on slave end(s), but I'll do some further checking to see if I can > isolate it to that (at least for ns1.linuxmafia.com. - I'm presuming > same/similar for ns1.svlug.org.), and see whether or not > zone can be pulled via IPv6 - and beyond the IPv6 addresses > where I've done so on my home networks (all of which happen to > be via same ISP for IPv6 - so would be good to check from some > other(s) - I think I may have a location or two (or three or more?) that > I may be able to check that from). > > Thanks, I'll let you know. So ... most of the testing details first. Then past that section, recommendations for the very specific slaves in question - based upon those test results. Then there's bit more information/comments after that. And then a wee bit more additional testing. $ dig +short ns1.linuxmafia.com. A ns1.linuxmafia.com. AAAA 198.144.195.186 $ ssh -ax linuxmafia.com. 'umask 077 && hostname && > PATH=/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin:"$PATH" \ > && { { >>/dev/null 2>&1 type ip && ip -4 a s; ip -6 a s; ip -6 r s; \ > } || { ifconfig -a; netstat -nr; } }' linuxmafia.com 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN inet 127.0.0.1/8 scope host lo 3: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 inet 198.144.195.186/29 brd 198.144.195.191 scope global eth2 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 inet6 ::1/128 scope host valid_lft forever preferred_lft forever 3: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qlen 1000 inet6 fe80::220:edff:fe13:ba89/64 scope link valid_lft forever preferred_lft forever fe80::/64 dev eth2 proto kernel metric 256 mtu 1500 advmss 1440 hoplimit 0 $ So ... [ns1.]linuxmafia.com. has an IPv6 stack and IPv6 link-local apparently configured, but no IPv6 route for anything beyond link-local and localhost. Let's see if I can confirm that ... $ ssh -ax linuxmafia.com. 'umask 077 && hostname && \ > PATH=/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin:"$PATH" \ > && ping6 -n -c 5 2001:470:1f04:19e::2' linuxmafia.com connect: Network is unreachable $ Yes, can't get to 2001:470:1f04:19e::2 from [ns1.]linuxmafia.com. 198.144.195.186 ::1/128 fe80::/64 without some type of route to 2001:470:1f04:19e::2 from that host. the: connect: Network is unreachable probably indicates we're getting an ICMP network unreachable, probably from the host itself, as it has no available route to get to 2001:470:1f04:19e::2 The diagnostic bit(s): failure trying master 2001:470:1f04:19e::2#53 (source ::#0): operation canceled would be, I'd guess, the nameserver's software's way of indicating it can't get to there or is otherwise failing - the "(source ::#0)" may be some kind of hint as to source IP that it's trying from or that it's trying that from IPv6, and knows not how to get there, or fails from that source IP, or IPv6 in general. Let me see if I can pull the zone via IPv6 to elsewhere ... $ ip -6 a s | fgrep inet6; ip -6 r s | fgrep via inet6 ::1/128 scope host inet6 fe80::221a:6ff:fe03:c207/64 scope link inet6 2001:470:66:76f::2/64 scope global inet6 fe80::c690:c2eb/64 scope link inet6 fe80::fc54:ff:fe13:5199/64 scope link default via 2001:470:66:76f::1 dev he-ipv6 metric 1024 $ dig @2001:470:1f04:19e::2 -t AXFR \ > e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa. +norecurse +noall +nocmd \ > +nocomments +answer | grep '^[ ]*[^ ;]' e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa. 7200 IN SOA ns1.balug.org. postmaster.balug.org. 1456011000 10800 3600 1209600 3600 e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa. 7200 IN NS ns0.mpaoli.net. e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa. 7200 IN NS ns1.balug.org. e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa. 7200 IN NS ns1.svlug.org. e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa. 7200 IN NS ns1.linuxmafia.com. 2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa. 7200 IN PTR balug.org. e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa. 7200 IN SOA ns1.balug.org. postmaster.balug.org. 1456011000 10800 3600 1209600 3600 $ Yes, I'm able to pull the zone (master is specifically configured: allow-transfer { any; }; for at least the [L]UG zone(s) - and that includes e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa. but the client IPv6 I pulled from is from same ISP/provider as the server pulled from. Do I have another IPv6 independent of that ISP that I can check from? $ ssh -ax m-net.arbornet.org. hostname ssh: connect to host m-net.arbornet.org. port 22: No route to host $ Hmmmm... $ (cd ~/tmp && stat -c '%y %n' .m-net.arbornet.org* && cat > .m-net.arbornet.org.lastlogin) 2016-02-10 00:05:03.690977780 -0800 .m-net.arbornet.org.lastlogin michaelp pts/7 Feb 10 03:05 (198.144.194.235) Connection to m-net.arbornet.org. closed. $ I've got a crontab job: $ crontab -l | fgrep m-net.arbornet.org | grep -v '^#' 5 8 * * * TZ=PST8PDT export TZ; set -e; trap 'trap - 0; exit 0' 0; exec </dev/null >>/dev/null 2>&1; cd tmp; lastlogin=.m-net.arbornet.org.lastlogin lastoutput=.m-net.arbornet.org.lastoutput; set +e; tmp=`find "$lastlogin" -prune ! -mtime +15 -type f -print`; [ x"$tmp" = x"$lastlogin" ] || { </dev/null >"$lastoutput" 2>&1 ssh -a -n -t -t -x -o BatchMode=yes m-net.arbornet.org. 'umask 077 && exec who am I' && mv -f "$lastoutput" "$lastlogin"; } $ that keeps my account there from going away (I think it has to be logged into once every 30 days to not get removed). My crontab job, daily, makes a login attempt if its last successful login was more than 15 days ago. Looks like it worked rather recently ... but not working at the moment. So, another ISP's IPv6 ... ignore some of the despicable operating system bits below ... but at least with Cygwin it's almost bearable ... testing from a work connection (hey, fair's fair, often use home connections from work to do work to, e.g. test various accessibility from The Internet, etc.) ... $ IPCONFIG /ALL | fgrep IPv6\ Address | fgrep -v Link-local IPv6 Address. . . . . . . . . . . : 2600:1010:b161:5e74:7c7a:2cff:fe94:2fa1(Preferred) $ dig @2001:470:1f04:19e::2 -t AXFR \ > e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa. +norecurse +noall +nocmd \ > +nocomments +answer | grep '^[ ]*[^ ;]' /usr/src/ports/bind/bind-9.10.2-2.P2.x86_64/src/bind-9.10.2-P2/lib/isc/unix/socket.c:2867: setsockopt(20, IPV6_RECVTCLASS) failed: Invalid argument e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa. 7200 IN SOA ns1.balug.org. postmaster.balug.org. 1456011000 10800 3600 1209600 3600 e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa. 7200 IN NS ns0.mpaoli.net. e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa. 7200 IN NS ns1.balug.org. e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa. 7200 IN NS ns1.svlug.org. e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa. 7200 IN NS ns1.linuxmafia.com. 2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa. 7200 IN PTR balug.org. e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa. 7200 IN SOA ns1.balug.org. postmaster.balug.org. 1456011000 10800 3600 1209600 3600 $ Okay, enough of that, ... set *that* operating system aside ... now back to a *real* operating system :-> ... And to look at ISPs to see/show they're different, at least to the extent can check with whois(1), we have ... First the master: $ 2>&1 whois -H 2001:470:1f04:19e::2 | > egrep -i '^(netname|nethandle|netrange|cidr|organization)' NetRange: 2001:470:: - 2001:470:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF CIDR: 2001:470::/32 NetName: HURRICANE-IPV6 NetHandle: NET6-2001-470-1 Organization: Hurricane Electric, Inc. (HURC) $ Then the first global routable IPv6 I tried from (expecting same ISP in whois): $ 2>&1 whois -H 2001:470:66:76f::2 | > egrep -i '^(netname|nethandle|netrange|cidr|organization)' NetRange: 2001:470:: - 2001:470:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF CIDR: 2001:470::/32 NetName: HURRICANE-IPV6 NetHandle: NET6-2001-470-1 Organization: Hurricane Electric, Inc. (HURC) $ No surprises there - it's also part of same NetRange / CIDR, as we can see in even the first set of whois data. And last client I tried, I'm hoping/expecting different ISP (is a big telecom company after all) ... $ 2>&1 whois -H 2600:1010:b161:5e74:7c7a:2cff:fe94:2fa1 | > egrep -i '^(netname|nethandle|netrange|cidr|organization)' NetRange: 2600:1000:: - 2600:1017:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF CIDR: 2600:1010::/29, 2600:1000::/28 NetName: WIRELESSDATANETWORK NetHandle: NET6-2600-1000-1 Organization: Cellco Partnership DBA Verizon Wireless (CLLC) $ And yes, an ISP, that as far as I'm aware, is quite independent of the other ISP shown for the two other IPv6 addresses. So ... recommendations ... I'm not sure what DNS slave server software is in use but for the masters, if the IPv6 addresses are listed after IPv4 (which may already be the case) and it still complains, my recommendation would be to comment out or otherwise disable the IPv6 master address(es) until such time as the slave hosts have some type of routable Internet IPv6 connectivity. $ dig +short ns1.svlug.org. A ns1.svlug.org. AAAA 64.62.190.98 $ I'd guess issue is same for ns1.svlug.org. - I don't see any IPv6 addresses there, but I don't have access to the host, as far as I'm aware, so am not in position to test more fully from that slave host. So, if same or quite similar applies to that slave host, then same recommendation would apply. I'm also guestimating, the DNS slave server software likely checks *all* masters - I'd expect that - probably starts with SOA, and then grabs "highest" (per the DNS RFCs) SOA serial number zone successfully found from master it successfully did SOA query on. It may or may not have some fallback to try AXFR if SOA fails, or if SOA succeeds but AXFR fails, there's probably some retry/failover algorithm it uses among the master(s) configured for that slave for that zone. So, it probably at least *tries* all configured masters, and depending upon the software, and possible configuration and logging levels, it may complain for any it fails to get the data from, or I'd think at least most minimally, if it failed with all masters - but I'm guessing it would more likely complain if it failed with *any* of the masters, as one may generally want to at least minimally have that logged, for possible investigation/action (expected event/maintenance, or known outage, or ... some other issue/problem)? Miscellaneous comments :-) : IPv6 - random John and Jane Doe consumer might not yet know or care (and may never know, or particularly care), but The Internet is (slowly) going the way of IPv6. IPv4 won't disappear anytime soon. However, with the exhaustion of IPv4 addresses, increasingly clients are not only dual-stack (or have some funky way to still be able to access IPv4), but increasingly there will be clients much prefer IPv6 address and/or do or will only have IPv6 access. IPv6 adoption has also been growing at a relatively fast rate. If I recall correctly, last year IPv6 growth was up something like 10%. I remember even about 4 years ago, looking at DNS traffic of a major provider, AAAA queries were up to nearly 2% of volume of queries, relative to A+AAAA - that was up about 300% compared to only about a year or two earlier. Not sure about similar current stats, but in general, any such clients that are IPv6 only, those would typically be lost traffic/customers, or they may be stuck with some less preferred means of IPv4 access. We're well past the time where at least most all major providers should also be offering IPv6 connectivity (e.g. web sites, etc.). The big players in ISPs space all (or nearly all?) do IPv6, some of the smaller ISPs are still getting there (and even some of the colos aren't fully there yet). But in any case, IPv6 is coming, and one should prepare and get ready for it (but not an extreme rush for everyone - if you're domestic only web site with only domestic users, may not be much of a rush to use IPv6 ... but dang, it certainly is also nice to not be squeezed with very limited number of IPv4 addresses to work with). Oh, if one doesn't have IPv6 available from one's ISP, there are IPv6 tunnel brokers, so one can use IPv6 over IPv4 - and thus reach the IPv6 Internet. E.g. Hurricane Electric provides such - for free (at least as in beer). They also have a lot of excellent IPv6 training materials (though a bit dated now, it's still mostly quite applicable). Anyway, I do have such IPv6 tunnels from Hurricane Electric (alas, my home ISP isn't yet offering native IPv6 ... I did also notice my work cellular, didn't have IPv6 only a few months or so ago, but a month or two after that, had added IPv6). Anyway, have an IPv6 tunnel for myself. Also have one for BALUG (I did a separate account for BALUG, so it's quite independent of my personal account). From Hurricane Electric: https://ipv6.he.net/certification/ https://tunnelbroker.net/ They also offer some free DNS services ... but they may be slightly funky in how they behave, and also somewhat limited, in at least some regards: https://dns.he.net/ But those do also include IPv6 nameservers. Also, in the ssh examples above, I didn't include bothering to show use of ssh-add - generally used in advance to decrypt private key(s) and then temporarily (for specified length of time) hold the decrypted private key(s) in RAM - best to never have the keys in the clear written to non-volatile storage (e.g. disk or SSD or similar). The ssh-add program also takes care to not let private keys be written to swap - but I also have my swap encrypted, and likewise most of my drive storage. And in the above regular expressions where there's character class ([...]) showing whitespace within, the whitespace is generally a space and a tab (my email client doesn't conveniently allow me to literally pass that along - and even if I did, clients displaying such may not preserve such in display, and in any case it wouldn't generally be visually obvious). Some RE contents allow \t for tab, but those I showed above generally don't support that. And retrying earlier test: $ ssh -ax m-net.arbornet.org. hostname m-net.arbornet.org $ That works now, so trying further with that ... $ ssh -ax m-net.arbornet.org. 'hostname && > PATH="/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin:$PATH" \ > && { if >>/dev/null 2>&1 type ip; then ip -6 a s; ip -6 r s > else ifconfig -a; netstat -nr; fi; }' m-net.arbornet.org em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM> ether 00:50:56:8b:48:cf inet 162.202.67.157 netmask 0xffffffe0 broadcast 162.202.67.159 inet6 fe80::250:56ff:fe8b:48cf%em0 prefixlen 64 scopeid 0x1 nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> media: Ethernet autoselect (1000baseT <full-duplex>) status: active em1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM> ether 00:0c:29:34:2e:d3 nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> media: Ethernet autoselect status: no carrier plip0: flags=8810<POINTOPOINT,SIMPLEX,MULTICAST> metric 0 mtu 1500 nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=3<RXCSUM,TXCSUM> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4 inet 127.0.0.1 netmask 0xff000000 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> ipfw0: flags=8801<UP,SIMPLEX,MULTICAST> metric 0 mtu 65536 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> Routing tables Internet: Destination Gateway Flags Refs Use Netif Expire default 162.202.67.129 UGS 0 4573 em0 127.0.0.1 link#4 UH 0 47170 lo0 162.202.67.128/27 link#1 U 0 127 em0 162.202.67.157 link#1 UHS 0 0 lo0 Internet6: Destination Gateway Flags Netif Expire ::/96 ::1 UGRS lo0 ::1 ::1 UH lo0 ::ffff:0.0.0.0/96 ::1 UGRS lo0 fe80::/10 ::1 UGRS lo0 fe80::%em0/64 link#1 U em0 fe80::250:56ff:fe8b:48cf%em0 link#1 UHS lo0 fe80::%lo0/64 link#4 U lo0 fe80::1%lo0 link#4 UHS lo0 ff01::%em0/32 fe80::250:56ff:fe8b:48cf%em0 U em0 ff01::%lo0/32 ::1 U lo0 ff02::/16 ::1 UGRS lo0 ff02::%em0/32 fe80::250:56ff:fe8b:48cf%em0 U em0 ff02::%lo0/32 ::1 U lo0 $ I don't think I see anything there that's globally routable IPv6 ff0x::114 Used for experiments ... yeah, I was wondering what ff01::... is/was ... wasn't able to identify that. Well, let's try wee bit: $ ssh -ax m-net.arbornet.org. 'hostname && > PATH="/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin:$PATH" \ > && >>/dev/null 2>&1 type ping6 && ping6 -n -c 5 2001:470:1f04:19e::2' m-net.arbornet.org ping6: UDP connect: No route to host $ Yep, it doesn't have a route to get there, so can't test further from there. >> From: "Rick Moen" <rick(a)linuxmafia.com> >> Subject: Re: [BALUG-Admin] DNS slaves for BALUG? :-) >> Date: Sat, 20 Feb 2016 18:33:09 -0800 > >> I wrote: >> >>> Quoting Michael Paoli (Michael.Paoli(a)cal.berkeley.edu): >>> >>>> If you could please, and would be willing, >>>> could you cover DNS slave services for BALUG, >>>> notably these zones: >>>> e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa >>>> balug.org >>>> master(s) (all for each of the above): >>>> 198.144.194.238 >>>> 2001:470:1f04:19e::2 >>> >>> FYI, ns1.linuxmafia.com is repeatedly logging this: >>> >>> Feb 20 17:50:45 linuxmafia named[17291]: zone >>> e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa/IN: refresh: failure >>> trying master 2001:470:1f04:19e::2#53 (source ::#0): operation >>> canceled >>> >>> I'm unsure why the AXFR/IXFR to the IPv6 address fails, and currently >>> lack the time and patience to pursue that matter. For now, I'm going to >>> disable that master IP on my BIND9 conffile. >> >> Before I do that, Michael, would you like to any some checking on the >> _master_ nameserver end? The most probable explanation is some deficiency >> in IPv6 support on ns1.linuxmafia.com, and that would be my default >> assumption. >> >> FYI, both zones are affected: >> >> linuxmafia:/etc/bind# zgrep 'failure trying master' /var/log/* >> /var/log/daemon.log:Feb 20 12:55:59 linuxmafia named[17291]: zone >> balug.org/IN: refresh: failure trying master >> 2001:470:1f04:19e::2#53 (source ::#0): operation canceled >> /var/log/daemon.log:Feb 20 13:39:16 linuxmafia named[17291]: zone >> e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa/IN: refresh: failure >> trying master 2001:470:1f04:19e::2#53 (source ::#0): operation >> canceled >> /var/log/daemon.log:Feb 20 17:50:45 linuxmafia named[17291]: zone >> e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa/IN: refresh: failure >> trying master 2001:470:1f04:19e::2#53 (source ::#0): operation >> canceled >> /var/log/syslog:Feb 20 12:55:59 linuxmafia named[17291]: zone >> balug.org/IN: refresh: failure trying master >> 2001:470:1f04:19e::2#53 (source ::#0): operation canceled >> /var/log/syslog:Feb 20 13:39:16 linuxmafia named[17291]: zone >> e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa/IN: refresh: failure >> trying master 2001:470:1f04:19e::2#53 (source ::#0): operation >> canceled >> /var/log/syslog:Feb 20 17:50:45 linuxmafia named[17291]: zone >> e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa/IN: refresh: failure >> trying master 2001:470:1f04:19e::2#53 (source ::#0): operation >> canceled >> linuxmafia:/etc/bind#

9 years, 6 months

Re: [BALUG-Admin] excess NOTIFY (about ever 90s) ISC BIND 9.18.24 ...

by Michael Paoli

So, modest update. Yes, it's very much issue on server (thus far doesn't appear to be issue of Comcast Business' doing). Also, have tried it with server on VM, and haven't been able to replicate the issue there. So, possible it might just be some funkiness in configuration I have to work out - there are some other bits too, e.g. systemd doesn't quite cleanly start it - it takes a while, then complains that the startup attempt failed with timeout ... but upon checking it's actually properly up and running. Also have non-trivial differences between my actual production setup and what I'd tested on VM, e.g. production is using actual delegated zones, also has Dynamic DNS (DDNS), also configured using chroot, etc. So, not trivial to do apples-to-apples comparison, though did at least compare with exact same versions of Debian and Debian's BIND. But even if I find what makes the difference configuration-wise or the like, doesn't mean it's necessarily a reasonably feasible fix. E.g. the bug might only show up with chroot or DDNS or with actual domains (where client is a secondary). Anyway, haven't given up on it, but not super high on the priorities to nail that one ... and might not be particularly feasibly fixable at current anyway. And yeah, if you saw/see notifies from one of my IPs for zone example.com - yeah, that'd be from some of the VM testing (from both stable, and possibly also unstable+experimental). And if you're curious and/or bored and interested in more, I was earlier testing on zone tmp.mpaoli.net. I later switched to instead using dnssec-test.mpaoli.net Some interesting test/history bits of those may be seen on https://dnsviz.net/ And for some bit I was even using tmp2.mpaoli.net, but don't know that I got as far as checking any of that zone on https://dnsviz.net/ And yes, https://wiki.debian.org/DNSSEC%20Howto%20for%20BIND%209.9+ is even more nicely updated to current and quite thoroughly tested at this point. And keys are still being rolled on mpaoli.net (most notably KSK, but also some ZSK), so for bit longer on will see additional cruft there (because TTLs and to not break thing), but the remaining cleanup will happen on that relatively soon (most notably once last relevant TTLs have passed). And yes, one can see key steps along that process via: https://dnsviz.net/d/mpaoli.net/dnssec/ See, e.g.: https://dnsviz.net/d/mpaoli.net/ZmORYA/dnssec/ for bit more interesting along the way. And alas, if I'd gotten it fully optimal along the way (notably with lifetime policy in dnssec-policy statement), I could've avoided creating and having used and in place a whole additional ZSK that wasn't even needed and is in most all regards quite redundant. In any case, all will be completed with cleanup fairly soon on that. Yeah, rotating KSKs is non-trivial, but advances in both RFCs and BIND are continuing to make that easier. Anyway, I agree, it ought not be doing the excessive notify - annoying that. But, no great harm, and I may or may not mange to get that issue fixed anytime soon. On Sat, Jun 8, 2024 at 4:31 PM Michael Paoli <michael.paoli(a)berkeley.edu> wrote: > > So, haven't 100% gotten to the bottom of the excess NOTIFY, > but it is apparently present in ISC BIND 9.18.24 and some other versions, > and Debian bind9 1:9.18.24-1 which is the current version for the current stable > Debian 12 bookworm. Also thinking that bug, for Debian stable, probably > doesn't make it to >=important priority (probably just priority normal), > so probably won't get fixed in the current stable. > Thus far, this is what I've run across that seems most informative about it: > https://gitlab.isc.org/isc-projects/bind9/-/issues/3242 > So, workarounds? I'm not going to be upgrading that host from stable > for quite some time (most likely year(s)), and other alternatives to fixing it > on the server side is probably more additional overhead than I'm likely to > want to bother with - so probably not going there. > On the client side, can reconfigure logging to not log those bits, > or possibly ignore them (e.g. by filter) in whatever > notifications/reporting one is doing. > So, yes, it is definitely coming from the nameserver itself, and all indications > are it's a bug. I did see some chatter somewhere about it possibly being a > but only if client doesn't respond in some way(s)? But that's not what I see > in my testing on clients that do respond as expected, so I don't think that's > it at all - at least in my case. > > Oh, as for possibly adjusting the logging on [ns1.]linuxmafia.com, > I notice it's got > bind9 version 1:9.7.1.dfsg.P2-2 > installed, and yes, also has > bind9-doc version 1:9.4.2-4, > but that's different version. > However, it does have, in /var/cache/apt/archives/ > bind9-doc_9.7.1.dfsg.P2-2_all.deb > So, that would be the corresponding documentation package, > which could be installed, or the files simply extracted from that and > consulted as may be desired. Anyway, within that, around: > doc/bind9-doc/arm/Bv9ARM.ch06.html#id2575303 > It has information about configuring logging, what the defaults are > and the category > notify > in logging configuration that one can use to adjust what one > does/doesn't want done regarding logging of > notify > activity. > > And, along the way, I've also rather well updated: > Debian wiki: DNSSEC Howto for BIND 9.9+: > https://wiki.debian.org/DNSSEC%20Howto%20for%20BIND%209.9+ > And also well whipping mpaoli.net (and nameserver in general) configuration > into shape for the current Debian stable. > > If I find anything else particularly noteworthy about the notify situation, > I'll update. > > On Tue, Jun 4, 2024 at 4:21 PM Michael Paoli <michael.paoli(a)berkeley.edu> wrote: > > > > There's two different issues (well, three, but 3rd is minor), not to > > be confused: > > > > The second - unrelated - issue. I've got excess notifies being sent > > from my server(s). > > Now, not at all unusual that sometimes there may be bursts of activity > > and updates and such, > > at least some of that is to be expected. > > What's quite unexpected and ought not be happening, but is, > > is I've got server sending frequent (e.g. like every 90 seconds-ish - > > or so) notifies > > when basic zone DNS data hasn't changed - same SOA serial, etc. > > So, something's still amis with that which I need to correct (doesn't > > really break any > > functionalities ... but rather wasteful all those excess notifies being sent, > > also tends to cause needless traffic on both sending side, > > and receiving servers checking and finding they've got nothing > > to do because they're already current). > > And, as I'd earlier mentioned, I suspect from some major upgrades I did > > a few weeks ago ... well, that was off-list, so, let me include fair > > bit more of it here > > (but not the whole thing, it gets very long, especially with all the > > earlier referenced): > > ----- > > ---------- Forwarded message --------- > > From: Michael Paoli <michael.paoli(a)berkeley.edu> > > Date: Mon, Jun 3, 2024 at 11:19 PM > > Subject: Re: [BALUG-Admin] (forw) linuxmafia.com 2024-06-03 15:02 System Events > > To: Al <aw009(a)sunnyside.com> > > Cc: Rick Moen <rick(a)linuxmafia.com> > > > > Yeah, ... something's still not quite right there. > > Did upgrade on that host/server a couple weekends ago > > Debian: buster 10 --> bullseye 11 --> bookworm 12 > > $ TZ=GMT0 stat -c '%z %Z %n' /etc/debian_version > > 2024-05-14 07:37:15.000000000 +0000 1715672235 /etc/debian_version > > $ > > Some of ye olde(er) DNS configuration options, while still > > (marginally?) functional, are quite deprecated, and I'm getting > > warnings on that and one other minor issue ... anyway, need to do > > some cleanup/reconfig with DNS server on that host. > > Should all be fine after that. Functional in the meantime, but ... > > not yet optimal ... clearly. > > > > On Mon, Jun 3, 2024 at 10:43 PM Al <aw009(a)sunnyside.com> wrote: > > > > > > I'm probably not reading those logs right but part of it looked like mpaoli was being signed every minute? Only partial logs, not enough to be sure. 7:28, 7:29. > > > Just a thought... > > > > > > On June 3, 2024 20:15:06 Michael Paoli via BALUG-Admin <balug-admin(a)lists.balug.org> wrote: > > > > > >> Well, under some circumstances, I'd expect a relative flury, or > > >> "bursts" of notify > > >> activity. E.g. I do TLS(/"SSL") certs via LetsEncrypt.org (LE), using DNS > > >> verification. Some/many of those certs will have multiple > > >> Subject Alternative Name (SAN) domains, and there may be fair number > > >> of such certs. So, adding the relevant records for verification, and then > > >> removing them after having obtained the issued cert(s) - that can be a > > >> fair burst of activity. But those generally wouldn't persist over a long > > >> period of time. E.g. every 90 seconds over many hours or days > > >> or more ... then something else is going on. So ... let me dig > > >> (pun not quite intended but apt) a bit more and peek what's currently > > >> goin' on there ... > > ... > > ----- End forwarded message ----- > > ----- > > I suspect it has to do with some changes in how DNSSEC is configured, > > most notably a configuration syntax I'm using is (now) quite deprecated, > > and it may be thus defaulting to some rather undesirable behavior(s) > > which may include triggering the excess notifies. > > Anyway, want to test it carefully and not break things, that's why I'm not > > willy-nilly throwing things at it. Goals being, to not only fix it, but if > > feasible without causing further breakage along the way, and also not > > disabling DNSSEC at any point (shouldn't need to) ... and while I'm at > > it, also reviewing and checking/updating earlier documentation (much of which > > I wrote): > > Debian wiki: DNSSEC Howto for BIND 9.9+: > > https://wiki.debian.org/DNSSEC%20Howto%20for%20BIND%209.9+ > > And, along the way, I've also got temporary zone and subdomain I'm > > fiddling with: tmp.mpaoli.net. to test things, etc. Anyway, it's still a work > > in progress to get that all squared away (get issue fixed, not further > > break things, > > and update relevant documentation). > > Anyway, as far as I'm aware, the excess notifies being sent don't have > > anything to do with the Comcast issues. > > > > On Tue, Jun 4, 2024 at 3:12 PM Rick Moen <rick(a)linuxmafia.com> wrote: > > > > > > More of same, involving back-to-back NOTIFY traffic about mpaoli.net, > > > allegedly originating from master nameserver 96.86.170.226 . Michael, I > > > greatly doubt your nameserver is doing this. mpaoli.net's zonefile is > > > pretty unchanging. I suspect Comcast middleman fsckery. Thoughts? > > > > > > ----- Forwarded message from logcheck system account <logcheck(a)linuxmafia.com> ----- > > > > > > Date: Tue, 04 Jun 2024 02:02:01 -0700 > > > From: logcheck system account <logcheck(a)linuxmafia.com> > > > To: root(a)linuxmafia.com > > > Subject: linuxmafia.com 2024-06-04 02:02 System Events > > > > > > System Events > > > =-=-=-=-=-=-= > > > Jun 4 01:47:15 linuxmafia named[15622]: zone mpaoli.net/IN: Transfer started. > > > Jun 4 01:47:15 linuxmafia named[15622]: transfer of 'mpaoli.net/IN' from 96.86.170.226#53: connected using 96.95.217.99#52012 > > > Jun 4 01:47:15 linuxmafia named[15622]: zone mpaoli.net/IN: transferred serial 1717468800 > > > Jun 4 01:47:15 linuxmafia named[15622]: transfer of 'mpaoli.net/IN' from 96.86.170.226#53: Transfer completed: 1 messages, 6 records, 546 bytes, 0.064 secs (8531 bytes/sec) > > > Jun 4 02:00:09 linuxmafia named[15622]: zone mpaoli.net/IN: Transfer started. > > > Jun 4 02:00:09 linuxmafia named[15622]: transfer of 'mpaoli.net/IN' from 96.86.170.226#53: connected using 96.95.217.99#57758 > > > Jun 4 02:00:09 linuxmafia named[15622]: zone mpaoli.net/IN: transferred serial 1717491608 > > > Jun 4 02:00:09 linuxmafia named[15622]: transfer of 'mpaoli.net/IN' from 96.86.170.226#53: Transfer completed: 1 messages, 15 records, 1266 bytes, 0.072 secs (17583 bytes/sec)

1 year, 3 months

balug.org(/sf-lug.{org, com}) host OOM oops Re: (forw) linuxmafia.com 2017-09-30 11:02 System Events

by Michael Paoli

To on-list, because, well, ... why not? :-) And ... BALUG-Admin, (and/or) SF-LUG? Well, it's more so balug.org host also running some SF-LUG services, rather than vice versa, so ... "She's dead, Jim." - Rick Moen (Thanks Rick!) noticed some issues, checked a bit and ... named was no longer running! 8-O Host was still up and otherwise seemed (relatively) healthy, but ... what happened to named? Did some digging ... not dig(1) particularly, but ... checking logs. So ... restarted named ... seemed healthy and okay and such. Started checking logs ... /var/log/daemon.log* Sep 29 02:58:37 balug-sf-lug-v2 systemd[1]: Unit bind9.service entered failed state. That seemed about the last peep about it ... but there were indications of other problems ... notably things apparently being killed off. :-/ /var/log/messages* Uh oh, ... the dreaded OOM PID killer! ... Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.816128] fail2ban-server invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.830751] fail2ban-server cpuset=/ mems_allowed=0 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.832896] CPU: 0 PID: 914 Comm: fail2ban-server Not tainted 3.16.0-4-amd64 #1 Debian 3.16.43-2+deb8u5 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.834810] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.835826] 0000000000000000 ffffffff81514291 ffff88003ca049e0 0000000000000000 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.836197] ffffffff81511e69 0000000000000000 ffffffff810d6f6f 0000000000000000 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.836197] ffffffff81518c4e 0000000000000200 ffffffff81068a53 ffffffff810c44e4 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.836197] Call Trace: Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.836197] [<ffffffff81514291>] ? dump_stack+0x5d/0x78 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.836197] [<ffffffff81511e69>] ? dump_header+0x76/0x1e8 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.836197] [<ffffffff810d6f6f>] ? smp_call_function_single+0x5f/0xa0 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.836197] [<ffffffff81518c4e>] ? mutex_lock+0xe/0x2a Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.836197] [<ffffffff81068a53>] ? put_online_cpus+0x23/0x80 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.836197] [<ffffffff810c44e4>] ? rcu_oom_notify+0xc4/0xe0 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.836197] [<ffffffff8115431c>] ? do_try_to_free_pages+0x4ac/0x520 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.836197] [<ffffffff81142ddd>] ? oom_kill_process+0x21d/0x370 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.836197] [<ffffffff8114299d>] ? find_lock_task_mm+0x3d/0x90 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.836197] [<ffffffff81143543>] ? out_of_memory+0x473/0x4b0 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.836197] [<ffffffff8114940f>] ? __alloc_pages_nodemask+0x9ef/0xb50 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.836197] [<ffffffff8118894d>] ? alloc_pages_current+0x9d/0x150 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.836197] [<ffffffff81141b40>] ? filemap_fault+0x1a0/0x420 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.836197] [<ffffffff81167d0a>] ? __do_fault+0x3a/0xa0 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.836197] [<ffffffff8116a8ce>] ? do_read_fault.isra.54+0x4e/0x300 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.836197] [<ffffffff8116c0fc>] ? handle_mm_fault+0x63c/0x1150 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.836197] [<ffffffff810582c7>] ? __do_page_fault+0x177/0x4f0 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.836197] [<ffffffff8109ea37>] ? put_prev_entity+0x57/0x350 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.836197] [<ffffffff8105299b>] ? kvm_clock_get_cycles+0x1b/0x20 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.836197] [<ffffffff810c9b32>] ? ktime_get_ts+0x42/0xe0 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.836197] [<ffffffff811be157>] ? poll_select_copy_remaining+0xe7/0x140 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.836197] [<ffffffff8151c4d8>] ? async_page_fault+0x28/0x30 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.875236] Mem-Info: Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.876102] Node 0 DMA per-cpu: Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.877119] CPU 0: hi: 0, btch: 1 usd: 0 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.878426] Node 0 DMA32 per-cpu: Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.879490] CPU 0: hi: 186, btch: 31 usd: 0 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.880714] active_anon:113183 inactive_anon:113245 isolated_anon:0 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.880714] active_file:46 inactive_file:58 isolated_file:0 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.880714] unevictable:0 dirty:0 writeback:0 unstable:0 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.880714] free:12241 slab_reclaimable:3169 slab_unreclaimable:5274 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.880714] mapped:2458 shmem:2849 pagetables:5275 bounce:0 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.880714] free_cma:0 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.888200] Node 0 DMA free:4636kB min:700kB low:872kB high:1048kB active_anon:5292kB inactive_anon:5328kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:280kB shmem:280kB slab_reclaimable:104kB slab_unreclaimable:244kB kernel_stack:48kB pagetables:124kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.897611] lowmem_reserve[]: 0 982 982 982 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.899344] Node 0 DMA32 free:44328kB min:44352kB low:55440kB high:66528kB active_anon:447440kB inactive_anon:447652kB active_file:184kB inactive_file:232kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1032184kB managed:1008516kB mlocked:0kB dirty:0kB writeback:0kB mapped:9552kB shmem:11116kB slab_reclaimable:12572kB slab_unreclaimable:20852kB kernel_stack:3008kB pagetables:20976kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:727 all_unreclaimable? yes Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.910174] lowmem_reserve[]: 0 0 0 0 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.911889] Node 0 DMA: 5*4kB (M) 1*8kB (U) 10*16kB (U) 7*32kB (UM) 10*64kB (UM) 2*128kB (UM) 3*256kB (UM) 1*512kB (M) 0*1024kB 1*2048kB (R) 0*4096kB = 4636kB Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.917906] Node 0 DMA32: 738*4kB (UE) 390*8kB (EM) 189*16kB (UE) 117*32kB (UEM) 120*64kB (EM) 60*128kB (UEM) 23*256kB (UEM) 8*512kB (UEM) 2*1024kB (U) 0*2048kB 1*4096kB (R) = 44328kB Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.925589] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.927978] 12189 total pagecache pages Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.928995] 9236 pages in swap cache Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.929978] Swap cache stats: add 1598908, delete 1589672, find 1953234/2271848 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.931903] Free swap = 0kB Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.932854] Total swap = 1048544kB Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.933794] 262044 pages RAM Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.934655] 0 pages HighMem/MovableOnly Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.935688] 5917 pages reserved Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.936598] 0 pages hwpoisoned Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.937500] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.940324] [ 172] 0 172 7217 269 20 43 0 systemd-journal Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.942449] [ 184] 0 184 10356 2 22 217 -1000 systemd-udevd Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.944605] [ 597] 0 597 3015 16 12 768 0 haveged Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.946531] [ 598] 1 598 1571 14 9 13 0 uptimed Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.948669] [ 599] 101 599 42559 4837 67 6402 0 named Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.950662] [ 600] 0 600 64667 70 29 197 0 rsyslogd Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.952633] [ 601] 0 601 4756 5 15 40 0 atd Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.954470] [ 602] 0 602 6476 20 18 47 0 cron Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.957340] [ 603] 0 603 39348 46 43 2237 0 lwresd Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.960249] [ 607] 0 607 13796 29 31 139 -1000 sshd Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.962284] [ 609] 107 609 10563 48 24 69 -900 dbus-daemon Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.964515] [ 617] 0 617 7088 44 19 38 0 systemd-logind Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.966680] [ 671] 105 671 305156 30716 486 181302 0 clamd Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.968711] [ 687] 0 687 15925 289 35 2831 0 spfd Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.970791] [ 699] 103 699 8346 49 22 109 0 ntpd Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.973677] [ 701] 0 701 3724 3 12 36 0 agetty Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.976699] [ 702] 0 702 17950 3 40 132 0 login Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.978631] [ 715] 0 715 5062 3 15 113 0 mysqld_safe Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.980675] [ 858] 17041 858 150162 3794 90 11586 0 mysqld Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.982763] [ 859] 0 859 5536 3 16 51 0 logger Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.984811] [ 878] 38 878 15029 249 32 2049 0 mailmanctl Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.987040] [ 879] 38 879 15637 1826 34 1126 0 python Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.989403] [ 880] 38 880 15700 1885 35 1145 0 python Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.992585] [ 881] 38 881 14998 283 32 2005 0 python Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.994953] [ 882] 38 882 15700 548 34 2470 0 python Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.997219] [ 883] 38 883 15002 286 33 2019 0 python Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188213.999303] [ 884] 38 884 16807 521 36 2524 0 python Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188214.001351] [ 885] 38 885 15690 968 33 1995 0 python Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188214.003496] [ 886] 38 886 15024 250 33 2036 0 python Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188214.005724] [ 912] 0 912 42169 599 87 17903 0 /usr/sbin/spamd Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188214.008183] [ 914] 0 914 49613 670 30 1160 0 fail2ban-server Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188214.010262] [ 997] 0 997 78222 269 114 3705 0 apache2 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188214.011990] [ 1027] 0 1027 43875 10022 91 10191 0 spamd child Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188214.013870] [ 1029] 0 1029 42735 3586 87 15469 0 spamd child Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188214.015655] [ 1039] 113 1039 25706 45 48 473 0 exim4 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188214.017462] [ 1048] 1607 1048 9051 141 22 119 0 systemd Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188214.020096] [ 1049] 1607 1049 12900 14 26 740 0 (sd-pam) Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188214.022931] [ 1051] 0 1051 13451 3 33 115 0 sudo Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188214.025523] [ 1083] 0 1083 13311 1 32 106 0 su Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188214.027987] [ 1084] 0 1084 5085 3 15 142 0 bash Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188214.030570] [ 2215] 0 2215 23309 3 49 234 0 sshd Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188214.033272] [ 2217] 1607 2217 23309 44 47 201 0 sshd Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188214.035822] [ 2218] 1607 2218 5081 3 15 138 0 bash Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188214.038453] [ 2253] 1607 2253 5992 7 18 55 0 screen Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188214.041654] [ 2258] 1607 2258 6066 69 18 106 0 screen Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188214.044778] [ 2273] 1607 2273 5083 82 15 63 0 bash Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188214.047324] [ 2306] 0 2306 13451 3 32 118 0 sudo Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188214.050021] [ 2339] 0 2339 13311 1 31 105 0 su Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188214.051936] [ 2340] 0 2340 5103 73 16 91 0 bash Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188214.053694] [15410] 33 15410 88963 13397 135 3271 0 apache2 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188214.055645] [16284] 33 16284 80083 4647 118 2974 0 apache2 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188214.058999] [19656] 33 19656 80456 4573 117 3142 0 apache2 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188214.062614] [20288] 33 20288 79754 3974 117 3037 0 apache2 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188214.065805] [21280] 33 21280 79333 3244 115 3168 0 apache2 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188214.068966] [21283] 33 21283 85967 10413 129 3078 0 apache2 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188214.072292] [21284] 33 21284 86848 10748 130 3174 0 apache2 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188214.075728] [21306] 33 21306 85528 9358 127 3217 0 apache2 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188214.078872] [21420] 33 21420 86187 9993 128 3206 0 apache2 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188214.082082] [21832] 33 21832 85304 9835 128 3046 0 apache2 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188214.085599] [21937] 33 21937 79546 3702 116 3210 0 apache2 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188214.090313] [21940] 33 21940 85942 9674 128 3325 0 apache2 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188214.093417] [21950] 33 21950 88632 12360 133 3305 0 apache2 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188214.096711] [21951] 33 21951 87235 11223 131 3214 0 apache2 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188214.099971] [21953] 33 21953 87544 10917 131 3330 0 apache2 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188214.103143] [21976] 33 21976 87350 10978 131 3313 0 apache2 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188214.105194] [21984] 33 21984 87404 11163 131 3246 0 apache2 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188214.107390] [21985] 33 21985 86796 10190 129 3419 0 apache2 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188214.110497] [22036] 33 22036 87183 10215 129 3421 0 apache2 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188214.112329] [22039] 33 22039 85816 9314 128 3330 0 apache2 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188214.114268] [22040] 33 22040 78803 2445 113 3337 0 apache2 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188214.115973] [22042] 33 22042 79744 2694 116 3336 0 apache2 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188214.118243] [22043] 33 22043 84175 7485 123 3421 0 apache2 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188214.120124] [22044] 33 22044 83023 6360 121 3416 0 apache2 Sep 29 01:16:05 balug-sf-lug-v2 kernel: [188214.122435] [22052] 33 22052 78230 288 100 3689 0 apache2 clamd is fair bit of memory hog, but knew that, added more RAM to host earlier, and also, it's only one PID. The big RAM suck is Apache ... not per PID, but in total. 26 Apache PIDs, that quickly adds up. /var/log/syslog* And looks like it was around here that named got whacked: Sep 29 02:52:04 balug-sf-lug-v2 systemd[1]: bind9.service: main process exited, code=killed, status=9/KILL reviewed the Apache logs (/var/log/apache2/*) ... looks mostly like some overzealous web crawlers, plus some bad bots, were both simultaneously pounding away quite hard on Apache. Looks like I'd not tuned Apache sufficiently to well withstand such a load, so Apache (mpm_prefork) forked lots of PIDs to handle the requests and attempt to keep up, but alas, beyond the resources reasonably available on the host ... and that's when things then went South rather quickly. So ... would be better to limit how much resource Apache would suck up, rather than allow Apache to consume excessive resources relative to those available. That may result in some web service failures/errors ... but that's better than Apache otherwise negatively impacing services on the host. So ... Apache configuration ... cd /etc/apache2 && ./.unroll < apache2.conf ... # ./.unroll START: IncludeOptional mods-enabled/*.load LoadModule mpm_prefork_module /usr/lib/apache2/modules/mod_mpm_prefork.so ... # ./.unroll START: IncludeOptional mods-enabled/*.conf ... <IfModule mpm_prefork_module> StartServers 5 MinSpareServers 5 MaxSpareServers 10 MaxRequestWorkers 150 MaxConnectionsPerChild 0 </IfModule> From the Apache documentation: "Most important is that MaxRequestWorkers be big enough to handle as many simultaneous requests as you expect to receive, but small enough to assure that there is enough physical RAM for all processes." Oops! So, looks like things were going badly with 26 Apache PIDs (likely 1 "master" (parent) and the rest spawned children). So, likely MaxRequestWorkers should be something at least below 25 - and even 25 would be (bit) too high. That's also a lot of load/work - simultaneously handling up to 25 requests. If something(s) are requesting that much ... some requests can wait - better that than quite negatively impacting the host overall. So ... I'm thinking 20. How does it look at present? 2>&1 fuser /usr/sbin/apache2 /usr/sbin/apache2: 443e 969e 997e 3211e 7433e 13566e 13604e 13605e 13623e 31738e 31749e 11 PIDs, so 20 ought be pretty reasonable for MaxRequestWorkers. That would be about 21 Apache PIDs total, and simultaneously handling up to about 20 requests. # ls -ld /etc/apache2/mods-enabled/mpm_prefork.conf lrwxrwxrwx 1 root root 34 Oct 30 2015 /etc/apache2/mods-enabled/mpm_prefork.conf -> ../mods-available/mpm_prefork.conf # cd /etc/apache2/mods-available # ex mpm_prefork.conf mpm_prefork.conf: unmodified: line 16 :/150 MaxRequestWorkers 150 :s/150/20/p MaxRequestWorkers 20 :w mpm_prefork.conf: 16 lines, 570 characters :q # (cd / && umask 022 && apachectl graceful) # ci -d -u -M -m'adjust MaxRequestWorkers to avoid excess RAM consumption and dreaded kernel OOM PID killer' mpm_prefork.conf RCS/mpm_prefork.conf,v <-- mpm_prefork.conf new revision: 1.2; previous revision: 1.1 done # Could also possibly consider going to a threaded model (if that plays well/safe with other Apache bits and related installed). ... and then a reboot for good measure (notably in case other PIDs got whacked by kernel OOM PID killer that ought be restarted). And then hopefully all is well with the universe again ... at least for now. > From: "Michael Paoli" <Michael.Paoli(a)cal.berkeley.edu> > To: "Rick Moen" <rick(a)linuxmafia.com> > Subject: Re: (forw) linuxmafia.com 2017-09-30 11:02 System Events > Date: Sat, 30 Sep 2017 20:41:23 -0700 > So, ... looks like, at least to a 1st order > approximation/guestimation based upon various log data ... > a combination of some overzealous web crawlers and bad bots > hit hard with the frequency and simultaneity of their requests, > Apache - probably still at or near to default on its configuration > for handling such as for workers/threads, tried to be very highly > accommodating to the excessive requests, starting up lots of > processes/threads ... and ... probably not properly tuned, > beyond reasonable for the available resources ... and > then fairly soon into that things started going seriously South. > So ... probably first bit, ... tweak some Apache settings so it's > not overly generous in providing resources, and excessive in > consumption of resources ... better Apache return a "server busy" > or similar type of error, than consume excessive resources > to the point where it quite negatively impacts the host > (dang Linux memory (mis) management ... if it didn't overcommit, > it could simply tell Apache, "sorry no more memory for you - enough > aleady" ... and generally nothing else would suffer, but ... alas, > if only ...). So, ... anyway, tweak some Apache settings on that, > reboot (to revive any other innocent PIDs that may have been > unceremoniously slaughtered), and ... keep an eye on it and see > how things go. Have bumped into relatively similar before ... > but it's been several years. Issue from many years ago were > bad bots massively excessively registering accounts on the > wiki ... the bots were too stupid to manage to do anything with > the wiki once they registered all those accounts ... but the > registration was so massively parallel it was a DoS attack > on the wiki/host, and it balooned resource consumption so high and > fast, host would lock up solid without leaving much of a trace > as to what happened ... took a bit of sleuthing and adding > some wee bits 'o extra data collection to track down > and nail that one. The work-around was then to change > then wiki so no web based registrations were allowed > anymore ... rare enough folks are added on the wiki, > that such can be handled manually ... that was sufficient > change to work around that issue that several years ago > when that was going on. Anyway, MTA/Mailman/anti-spam etc. > have upped the resource requirements/consumption some fairish > bit (did also give the virtual host more virtual RAM at the time > too) ... but ... probably again time for some more resource > allocation/tuning ... and looks like at present time Apache is > first logical place to adjust that. > >> From: "Michael Paoli" <Michael.Paoli(a)cal.berkeley.edu> >> Subject: Re: (forw) linuxmafia.com 2017-09-30 11:02 System Events >> Date: Sat, 30 Sep 2017 20:07:48 -0700 > >> The bloody OOM PID killer kicked in, and at some point >> in its infinite wisdom (stupidity) thought that SIGKILL >> to named on a nameserver was a "good" <cough, cough> idea. >> I'll have to see if I can isolate what was sucking up too >> much resource. I never did much care for Linux's (or at least >> some/many distributions leaving it enabled in kernel by default) >> overcommitting (notably as crude work-around for folks that >> write crappy programs that request lots of memory which they often >> never need or use) ... which then, when something actually needs >> the memory the kernel "gave" (promised) it, when it cheated >> and overcommitted ... yeah, ... that's when things get very ugly >> very fast - a.k.a. OOM PID killer, ... ugh! >> >> Anyway, more log sleuthing, to see what ate up so much >> resource ... and ... probably due for reboot after the OOM >> kicked in anyway, ... dear knows what else got whacked that >> ought not have been whacked. >> >> >>> From: "Michael Paoli" <Michael.Paoli(a)cal.berkeley.edu> >>> Subject: Re: (forw) linuxmafia.com 2017-09-30 11:02 System Events >>> Date: Sat, 30 Sep 2017 19:36:01 -0700 >> >>> Thanks, ... hopefully it's "all better now". >>> Somehow named wasn't listening 8-O ... and I think that's also >>> why the (still listening) MTA was then, uh, "upset." >>> I'll see what I can find as to why named was down and when it >>> went down ... maybe operator error, maybe ... who knows what. >>> Anyway, I'll see what I can find (and will recheck its general >>> health). >>> >>> Anyway, thanks for bringing it to my attention - I'd not >>> seen that quite yet. >>> >>>> From: "Rick Moen" <rick(a)linuxmafia.com> >>>> Subject: (forw) linuxmafia.com 2017-09-30 11:02 System Events >>>> Date: Sat, 30 Sep 2017 12:00:28 -0700 >>> >>>> Just checking to make sure you're aware of ongoing nameserver downtime. >>>> Also, MTA downtime. >>>> >>>> $ telnet mx.balug.org 25 >>>> Trying 198.144.194.238... >>>> Connected to mx.balug.org. >>>> Escape character is '^]'. >>>> 451 Temporary local problem - please try later >>>> Connection closed by foreign host. >>>> $ >>>> >>>> ----- Forwarded message from logcheck system account >>>> <logcheck(a)linuxmafia.com> ----- >>>> >>>> Date: Sat, 30 Sep 2017 11:02:01 -0700 >>>> From: logcheck system account <logcheck(a)linuxmafia.com> >>>> To: root(a)linuxmafia.com >>>> Subject: linuxmafia.com 2017-09-30 11:02 System Events >>>> >>>> System Events >>>> =-=-=-=-=-=-= >>>> Sep 30 10:20:52 linuxmafia named[32734]: zone sf-lug.org/IN: >>>> refresh: retry limit for master 198.144.194.238#53 exceeded >>>> (source 0.0.0.0#0) >>>> Sep 30 10:20:52 linuxmafia named[32734]: zone sf-lug.org/IN: >>>> Transfer started. >>>> Sep 30 10:20:52 linuxmafia named[32734]: transfer of >>>> 'sf-lug.org/IN' from 198.144.194.238#53: failed to connect: >>>> connection refused >>>> Sep 30 10:20:52 linuxmafia named[32734]: transfer of >>>> 'sf-lug.org/IN' from 198.144.194.238#53: Transfer completed: 0 >>>> messages, 0 records, 0 bytes, 0.062 secs (0 bytes/sec) >>>> Sep 30 10:24:59 linuxmafia named[32734]: zone balug.org/IN: >>>> refresh: retry limit for master 198.144.194.238#53 exceeded >>>> (source 0.0.0.0#0) >>>> Sep 30 10:30:20 linuxmafia named[32734]: zone sf-lug.com/IN: >>>> refresh: retry limit for master 198.144.194.238#53 exceeded >>>> (source 0.0.0.0#0) >>>> Sep 30 10:45:15 linuxmafia named[32734]: zone >>>> e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa/IN: refresh: retry limit >>>> for master 198.144.194.238#53 exceeded (source 0.0.0.0#0) >>>> Sep 30 10:45:15 linuxmafia named[32734]: zone >>>> e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa/IN: Transfer started. >>>> Sep 30 10:45:15 linuxmafia named[32734]: transfer of >>>> 'e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa/IN' from >>>> 198.144.194.238#53: failed to connect: connection refused >>>> Sep 30 10:45:15 linuxmafia named[32734]: transfer of >>>> 'e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa/IN' from >>>> 198.144.194.238#53: Transfer completed: 0 messages, 0 records, 0 >>>> bytes, 0.065 secs (0 bytes/sec) >>>> Sep 30 10:51:37 linuxmafia named[32734]: zone balug.org/IN: >>>> refresh: retry limit for master 198.144.194.238#53 exceeded >>>> (source 0.0.0.0#0) >>>> >>>> >>>> ----- End forwarded message -----

7 years, 11 months

BALUG-Admin search results for query "BALUG: Lists, stats, etc."