BALUG-Admin

Download

balug-admin@lists.balug.org

April 2021

2 participants
2 discussions

/var space issues on balug VM, almost certainly from Internet miscreants & their bots
by Michael Paoli 26 Apr '21

26 Apr '21

/var space issues on balug VM, almost certainly from Internet miscreants & their bots So ... bit more background on this below, and see also wiki page: https://www.wiki.balug.org/wiki/doku.php?id=system:annoyances I'll likely have the more detailed analysis/summary/progress there - including also history of the wiki page. I'll likely post more to BALUG-Admin <balug-admin(a)lists.balug.org> when it's more-or-less resloved ... or other major or possibly significant developments/findings/changes. Anyway, feel free to ask/discuss on BALUG-Admin if you may wish (or toss about ideas/suggestions/questions/whatever). Anyway, I thought the list traffic would be more fitting for BALUG-Admin than BALUG-Talk, so ... pointing that discussion to here ... also linked (notably on wiki page) to the earlier, if one wants to catch up on reading that (much or possibly all of which, might also be captured below). references/excerpts: > From: "Michael Paoli" <Michael.Paoli(a)cal.berkeley.edu> > Subject: -->BALUG-Admin Re: [BALUG-Talk] Possible solution of > blankness or editing outline-only on BALUG DokuWiki? > Date: Fri, 23 Apr 2021 21:40:57 -0700 > I'm going to suggest taking this discussion to the > BALUG-Admin <BALUG-Admin(a)lists.balug.org> > https://lists.balug.org/cgi-bin/mailman/listinfo/balug-admin > list, as, > though the matter does involved Linux somewhat generally, it's much > more specific and relevant to BALUG and the specific inner > workings thereof. So, expect my next update and further communications > on the matter to be on the BALUG-Admin list, and I'll likely start > it off with > latest update(s) and some background references/links on the matter, at least > as of late. > > Also ... > dates, lists, number of members: > YYYY-MM-DD announce talk admin > 2021-04-01 832 259 35 > https://lists.balug.org/pipermail/balug-admin/2021-April/001046.html > I'm guestmating probably not around 259 subscribers are that interested > in these particularly details ... but quite possibly closer to about > 35 are ... or would be. > I may well also put up a wiki page regarding findings/developments ... > or maybe update an older one if such suitable wiki page already exists. > > Anyway, I still have some 'work' to do ... > # date -Iseconds && df -h /var > 2021-04-24T04:39:59+00:00 > Filesystem Size Used Avail Use% Mounted on > /dev/mapper/balug-var 6.4G 5.9G 287M 96% /var > # > >> From: "Michael Paoli" <Michael.Paoli(a)cal.berkeley.edu> >> Subject: Re: [BALUG-Talk] Possible solution of blankness or editing >> outline-only on BALUG DokuWiki? >> Date: Fri, 23 Apr 2021 08:44:22 -0700 > >> Well, not out of the woods yet. >> >> Have done fair bit of compression (even using xz -9) of some >> moderately older data on /var ... and have grown /var several >> times, ... yet it filled up again very recently (or exceedingly >> close - I saw it at 100% again, but that's probably rounded, so >> at least >99.5%). >> >> So, at present, now have: >> # df -h /var >> Filesystem Size Used Avail Use% Mounted on >> /dev/mapper/balug-var 6.4G 5.7G 491M 93% /var >> # >> And very recently (only few days back) that was: >>> df -h /var >>> Filesystem Size Used Avail Use% Mounted on >>> /dev/mapper/balug-var 4.9G 4.5G 190M 97% /var >> Can't just keep growing /var indefinitely. >> Much of the volume is mail and related - logs, queued mail, >> bounces & failures, mail to listadmin and postmaster, etc., and mostly >> lots of other nuisance stuff - fail2ban database takes fair bit of space, >> but trimming what that holds would increase data in other logs - notably >> of lots of nuisance failed login/authentication attempts and the like. >> >> And, even if I caught it before completely full, some services (e.g. >> exim4) will proactively stop processing when the free filesystem >> space runs too low. >> >> So, yeah, I'll need to do some more work on it. May need to analyze why >> it's ballooning more than it was in past - what's the nature of the >> problematic traffic that keeps driving up the storage. >> >> I suspect it's mostly spam (and attempts thereof) and bad bots >> doing/attempting stuff ... essentially all some type(s) of nefarious >> unwanted traffic chewing up resources. >> >> So, in only a couple days or so, it's already nearly 1.5 GiB more data, >> and that's already after having done fair bit of aggressive data >> data compression on some older data (e.g. some of the older mail data >> easily compressed from around 200 of MiB to under a few tens of MB ... >> and I even moved the compressed off of the /var filesystem). >> >> So, ..., yeah, miscreants on 'da Internet ... more work for me to do. :-/ >> >> Would also be nice to get some of that /var space back, and not have to give >> so much of it over to what's basically mostly unintersting crud. >> >> And, have had stuff like this happen before, ... e.g. like bad bots trying >> to register on wiki - and chewing up huge volumes of space ... that's why >> there's no self-registration option anymore ... blame it on abusers on >> 'da Internet. Manual registration only now. Even captcha's weren't really >> enough for most of that ... bad bots still pounded away and repeatedly >> filled up log space. Maybe more fail2ban config would help that >> ... but then >> that also ends up chewing up additional space with its database ... >> >> # du -x /var | sort -bnr | head -n 12 >> 5866812 /var >> 2102592 /var/lib >> 1249060 /var/log >> 1017624 /var/spool >> 1015100 /var/mail >> 932200 /var/spool/exim4 >> 731780 /var/lib/fail2ban >> 616584 /var/lib/clamav >> 531456 /var/log/apache2 >> 505952 /var/log/exim4 >> 491980 /var/spool/exim4/input >> 364892 /var/spool/exim4/msglog >> # >> >>> From: "aaronco36 via BALUG-Talk" <balug-talk(a)lists.balug.org> >>> Subject: Re: [BALUG-Talk] Possible solution of blankness or >>> editing outline-only on BALUG DokuWiki? >>> Date: Thu, 22 Apr 2021 16:21:14 +0000 (UTC) >> >>> Quoting Michael Paoli <Michael.Paoli at cal.berkeley.edu> from [1]: >>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>> Thanks for the report, but I'm unable to reproduce the issue. >>> Anyone else able to reproduce it? Are you/anyone seeing the >>> issue at this time? Anyone else also see it earlier? >>> >>> I've tried the following on: >>> https://www.wiki.balug.org/wiki/doku.php without being able to >>> reproduce the issue: >>> Chromium >>> Chromium incognito mode >>> Firefox >>> Firefox private browsing mode >>> Lynx >>> curl >>> wget >>> Chome on Android from another ISP >>> Chrom icognito mode on Android from another ISP >>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>> >>> Cut-and-pasted a text screen-capture and submitted it as a >>> [BALUG-Test] 'Test of BALUG Wiki pages' posting yesterday morning; >>> see [2]. >>> AFAICT, that "blankness or editing outline-only issue" for BALUG >>> DokuWiki pages [3] and [4] occurred from before 08:00am PDT up >>> through the late morning, and then went away by mid-afternoon >>> yesterday. >>> Probably reloaded both those BALUG DokuWiki pages a half-dozen >>> times or so over the morning time-period using mostly Firefox-ESR >>> standard mode and Firefox-ESR private browsing mode, and maybe >>> once or twice using TOR browser in its [least] Safe mode. >>> >>> Same results for all browsers used to test those pair of pages >>> during that entire morning time-period. >>> >>> >>> Quoting Michael Paoli <Michael.Paoli at cal.berkeley.edu> from [5]: >>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>> That may have done it. >>> Looks like I need to add more space to /var and/or clean some more stuff >>> off of it: >>> df -h /var >>> Filesystem Size Used Avail Use% Mounted on >>> /dev/mapper/balug-var 4.9G 4.5G 190M 97% /var >>> # >>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>> >>> Thanks much, Michael, for effectively diagnosing and resolving the >>> issue! :-) >>> >>> Besides considering the issue arising from the DokuWiki site >>> itself or BALUG's device /dev/mapper/balug-var, was also >>> considering the possibility that the issue might have arisen from >>> my own browser client. >>> >>> Why so? >>> >>> Well, there was no issue whatsoever with successfully loading >>> BALUG DokuWiki pages [3] and [4] throughout Sunday and Monday of >>> this week. >>> What changed on Tuesday afternoon was upgrading Firefox-ESR to a >>> slightly higher, minor version as per 'less >>> /var/log/apt/history.log' : >>>> Start-Date: 2021-04-20 14:10:03 >>>> Commandline: apt upgrade >>>> Upgrade: firefox-esr:amd64 (78.9.0esr-1~deb10u1, 78.10.0esr-1~deb10u1) >>>> End-Date: 2021-04-20 14:10:15 >>> >>> One or more bugs in the upgraded FF 78.10.0esr could have >>> conceivably rendered text and links in the BALUG DokuWiki >>> incorrectly afterwards, as remote as this possibility might seem >>> to be. >>> It's good to know that this wasn't the case at all, and that this >>> latest Firefox-ESR browser was fine after all :-) >>> >>> Thanks again Michael! >>> >>> -Aaron >>> >>> =========================== >>> REFERENCES >>> =========================== >>> [1]https://lists.balug.org/pipermail/balug-talk/2021-April/000220.html >>> [2]https://lists.balug.org/pipermail/balug-test/2021/000048.html >>> [3]https://www.wiki.balug.org/wiki/doku.php >>> [4]https://www.wiki.balug.org/wiki/doku.php?id=balug:offered_wanted_hardware… >>> [5]https://lists.balug.org/pipermail/balug-talk/2021-April/000221.html >>> ===========================

2 4

BALUG: Lists, stats, etc.
by Michael Paoli 01 Apr '21

01 Apr '21

dates, lists, number of members: YYYY-MM-DD announce talk admin 2021-04-01 832 259 35 2021-03-01 833 259 35 2021-02-01 833 259 35 2021-01-01 833 259 35 2020-12-01 832 259 35 2020-11-01 833 260 35 2020-10-01 834 260 35 2020-09-01 835 260 35 2020-08-01 834 259 34 2020-07-01 834 266 34 2020-06-01 838 266 34 2020-05-01 839 267 34 2020-04-01 840 270 34 2020-03-01 840 271 34 2020-02-01 841 271 34 2020-01-01 841 271 34 2019-12-01 841 271 34 2019-11-01 840 271 34 2019-10-01 842 271 34 2019-09-01 844 272 34 2019-08-01 847 272 34 2019-07-01 852 272 36 2019-06-01 854 273 37 2019-05-01 856 272 38 2019-04-01 853 273 38 2019-03-01 851 273 38 2019-02-01 853 273 38 2019-01-01 856 274 38 2018-12-01 858 274 38 2018-11-01 861 274 39 2018-10-02 862 274 39 2018-09-01 864 274 38 2018-07-28 866 278 38 2018-06-08 870 282 38 2018-04-28 875 282 38 2018-02-17 880 283 36 2017-11-04 883 284 36 2017-08-19 897 293 37 2017-07-01 897 297 38 2017-05-27 897 297 38 2017-05-16 897 296 38 2017-04-01 896 297 38 2017-03-30 885 297 38 2016-05-03 886 291 39 2016-04-03 887 290 39 2016-03-05 888 291 45 2016-01-05 887 295 45 2015-12-05 887 296 45 2015-11-07 887 301 45 2015-10-06 888 301 45 2015-09-05 887 301 45 2015-08-04 888 301 45 2015-07-03 888 300 45 2015-06-05 888 301 46 2015-05-02 885 298 45 2015-04-04 884 297 44 2015-03-07 886 298 44 2015-02-06 885 303 44 2015-01-07 885 304 45 2014-12-06 886 304 45 2014-11-01 884 302 45 2014-10-04 881 303 46 2014-09-06 878 309 44 2014-06-13 875 307 44 2014-05-02 874 306 44 2014-04-04 876 306 44 2014-03-01 876 306 43 2014-01-11 872 305 42 2013-10-05 854 293 40 2013-08-31 855 295 41 2013-07-20 852 297 40 2013-04-10 838 286 41 2013-03-03 841 289 41 2013-02-02 843 291 41 2012-12-15 845 310 41 2012-03-23 818 310 39 2012-02-01 813 307 40 2011-04-28 768 291 37 2011-02-27 762 288 36 2010-05-02 730 289 34 2010-04-01 727 291 33 2010-03-16 727 293 33 2009-12-16 716 290 33 2009-09-09 700 288 32 2009-03-27 677 287 29 2009-01-10 662 290 30 2008-08-17 635 290 29 2008-03-03 571 247 28 2008-01-02 337 260 28 2007-12-28 336 261 28 2007-09-03 264 258 27 2007-07-24 287 256 27 2007-06-17 280 249 27 references: http://www.balug.org/#Lists

1 0