So ... much better now. :-) Still more to do, but it's at least "better enough" - things are quite in operation again. I did stop SMTP (exim4) on the balug Virtual Machine (VM) host for a while while dealing with the problem (roughly about 24 hours or so). But at this point, things are back in "normal" operation ... and even better now! (At least for the most part). E.g. ...
And, now with ye olde crufty eximconfig out of the way, which has been doing some of the anti-spam, some drain bamaged anti-spam it had that was causing issues with legitimate mail, is probably quite gone now. E.g., mailed commands to mailman can now be done using just the Subject: header and with an empty body - and that now works fine. Whereas earlier, the eximconfig configuration would mostly reject emails with an empty body as spam. So, I'll also be updating configurations to take out text regarding the workaround for that (include a body with just end in it) - as that's no longer needed at all.
So, yeah, drain bamaged anti-spam ... I recall many moons ago, Rick Moen sent an email which had a http or https url which ended in linuxmafia.com ... probably just https://linuxmafia.com Well, silly rules in eximconfig would look at that and essentially says, oh my gosh, that's a URL to a .COM executable file, must protect those happless Microsoft Windows/DOS users from that, and can't allow that thorough, it's probably some malware binary anyway. Well, anyway, at this point shouldn't be getting tripped up by stuff like that.
Anyway, lots more to do, but the present is "better enough" to (literally) run with it (currently running). And, will make further improvements from that base.
From: "Michael Paoli" Michael.Paoli@cal.berkeley.edu Subject: /var space issues on balug VM, almost certainly from Internet miscreants & their bots Date: Fri, 23 Apr 2021 23:30:52 -0700
/var space issues on balug VM, almost certainly from Internet miscreants & their bots
So ... bit more background on this below, and see also wiki page: https://www.wiki.balug.org/wiki/doku.php?id=system:annoyances
I'll likely have the more detailed analysis/summary/progress there - including also history of the wiki page.
I'll likely post more to BALUG-Admin balug-admin@lists.balug.org when it's more-or-less resloved ... or other major or possibly significant developments/findings/changes.
Anyway, feel free to ask/discuss on BALUG-Admin if you may wish (or toss about ideas/suggestions/questions/whatever). Anyway, I thought the list traffic would be more fitting for BALUG-Admin than BALUG-Talk, so ... pointing that discussion to here ... also linked (notably on wiki page) to the earlier, if one wants to catch up on reading that (much or possibly all of which, might also be captured below).
references/excerpts:
From: "Michael Paoli" Michael.Paoli@cal.berkeley.edu Subject: -->BALUG-Admin Re: [BALUG-Talk] Possible solution of blankness or editing outline-only on BALUG DokuWiki? Date: Fri, 23 Apr 2021 21:40:57 -0700
I'm going to suggest taking this discussion to the BALUG-Admin BALUG-Admin@lists.balug.org https://lists.balug.org/cgi-bin/mailman/listinfo/balug-admin list, as, though the matter does involved Linux somewhat generally, it's much more specific and relevant to BALUG and the specific inner workings thereof. So, expect my next update and further communications on the matter to be on the BALUG-Admin list, and I'll likely start it off with latest update(s) and some background references/links on the matter, at least as of late.
Also ... dates, lists, number of members: YYYY-MM-DD announce talk admin 2021-04-01 832 259 35 https://lists.balug.org/pipermail/balug-admin/2021-April/001046.html I'm guestmating probably not around 259 subscribers are that interested in these particularly details ... but quite possibly closer to about 35 are ... or would be. I may well also put up a wiki page regarding findings/developments ... or maybe update an older one if such suitable wiki page already exists.
Anyway, I still have some 'work' to do ... # date -Iseconds && df -h /var 2021-04-24T04:39:59+00:00 Filesystem Size Used Avail Use% Mounted on /dev/mapper/balug-var 6.4G 5.9G 287M 96% /var #
From: "Michael Paoli" Michael.Paoli@cal.berkeley.edu Subject: Re: [BALUG-Talk] Possible solution of blankness or editing outline-only on BALUG DokuWiki? Date: Fri, 23 Apr 2021 08:44:22 -0700
Well, not out of the woods yet.
Have done fair bit of compression (even using xz -9) of some moderately older data on /var ... and have grown /var several times, ... yet it filled up again very recently (or exceedingly close - I saw it at 100% again, but that's probably rounded, so at least >99.5%).
So, at present, now have: # df -h /var Filesystem Size Used Avail Use% Mounted on /dev/mapper/balug-var 6.4G 5.7G 491M 93% /var # And very recently (only few days back) that was:
df -h /var Filesystem Size Used Avail Use% Mounted on /dev/mapper/balug-var 4.9G 4.5G 190M 97% /var
Can't just keep growing /var indefinitely. Much of the volume is mail and related - logs, queued mail, bounces & failures, mail to listadmin and postmaster, etc., and mostly lots of other nuisance stuff - fail2ban database takes fair bit of space, but trimming what that holds would increase data in other logs - notably of lots of nuisance failed login/authentication attempts and the like.
And, even if I caught it before completely full, some services (e.g. exim4) will proactively stop processing when the free filesystem space runs too low.
So, yeah, I'll need to do some more work on it. May need to analyze why it's ballooning more than it was in past - what's the nature of the problematic traffic that keeps driving up the storage.
I suspect it's mostly spam (and attempts thereof) and bad bots doing/attempting stuff ... essentially all some type(s) of nefarious unwanted traffic chewing up resources.
So, in only a couple days or so, it's already nearly 1.5 GiB more data, and that's already after having done fair bit of aggressive data data compression on some older data (e.g. some of the older mail data easily compressed from around 200 of MiB to under a few tens of MB ... and I even moved the compressed off of the /var filesystem).
So, ..., yeah, miscreants on 'da Internet ... more work for me to do. :-/
Would also be nice to get some of that /var space back, and not have to give so much of it over to what's basically mostly unintersting crud.
And, have had stuff like this happen before, ... e.g. like bad bots trying to register on wiki - and chewing up huge volumes of space ... that's why there's no self-registration option anymore ... blame it on abusers on 'da Internet. Manual registration only now. Even captcha's weren't really enough for most of that ... bad bots still pounded away and repeatedly filled up log space. Maybe more fail2ban config would help that ... but then that also ends up chewing up additional space with its database ...
# du -x /var | sort -bnr | head -n 12 5866812 /var 2102592 /var/lib 1249060 /var/log 1017624 /var/spool 1015100 /var/mail 932200 /var/spool/exim4 731780 /var/lib/fail2ban 616584 /var/lib/clamav 531456 /var/log/apache2 505952 /var/log/exim4 491980 /var/spool/exim4/input 364892 /var/spool/exim4/msglog #
From: "aaronco36 via BALUG-Talk" balug-talk@lists.balug.org Subject: Re: [BALUG-Talk] Possible solution of blankness or editing outline-only on BALUG DokuWiki? Date: Thu, 22 Apr 2021 16:21:14 +0000 (UTC)
Quoting Michael Paoli <Michael.Paoli at cal.berkeley.edu> from [1]:
Thanks for the report, but I'm unable to reproduce the issue. Anyone else able to reproduce it? Are you/anyone seeing the issue at this time? Anyone else also see it earlier? I've tried the following on: https://www.wiki.balug.org/wiki/doku.php without being able to reproduce the issue: Chromium Chromium incognito mode Firefox Firefox private browsing mode Lynx curl wget Chome on Android from another ISP Chrom icognito mode on Android from another ISPCut-and-pasted a text screen-capture and submitted it as a [BALUG-Test] 'Test of BALUG Wiki pages' posting yesterday morning; see [2]. AFAICT, that "blankness or editing outline-only issue" for BALUG DokuWiki pages [3] and [4] occurred from before 08:00am PDT up through the late morning, and then went away by mid-afternoon yesterday. Probably reloaded both those BALUG DokuWiki pages a half-dozen times or so over the morning time-period using mostly Firefox-ESR standard mode and Firefox-ESR private browsing mode, and maybe once or twice using TOR browser in its [least] Safe mode.
Same results for all browsers used to test those pair of pages during that entire morning time-period.
Quoting Michael Paoli <Michael.Paoli at cal.berkeley.edu> from [5]:
That may have done it. Looks like I need to add more space to /var and/or clean some more stuff off of it: df -h /var Filesystem Size Used Avail Use% Mounted on /dev/mapper/balug-var 4.9G 4.5G 190M 97% /var #Thanks much, Michael, for effectively diagnosing and resolving the issue! :-)
Besides considering the issue arising from the DokuWiki site itself or BALUG's device /dev/mapper/balug-var, was also considering the possibility that the issue might have arisen from my own browser client.
Why so?
Well, there was no issue whatsoever with successfully loading BALUG DokuWiki pages [3] and [4] throughout Sunday and Monday of this week. What changed on Tuesday afternoon was upgrading Firefox-ESR to a slightly higher, minor version as per 'less /var/log/apt/history.log' :
Start-Date: 2021-04-20 14:10:03 Commandline: apt upgrade Upgrade: firefox-esr:amd64 (78.9.0esr-1~deb10u1, 78.10.0esr-1~deb10u1) End-Date: 2021-04-20 14:10:15
One or more bugs in the upgraded FF 78.10.0esr could have conceivably rendered text and links in the BALUG DokuWiki incorrectly afterwards, as remote as this possibility might seem to be. It's good to know that this wasn't the case at all, and that this latest Firefox-ESR browser was fine after all :-)
Thanks again Michael!
-Aaron
=========================== REFERENCES =========================== [1]https://lists.balug.org/pipermail/balug-talk/2021-April/000220.html [2]https://lists.balug.org/pipermail/balug-test/2021/000048.html [3]https://www.wiki.balug.org/wiki/doku.php [4]https://www.wiki.balug.org/wiki/doku.php?id=balug:offered_wanted_hardware_et... [5]https://lists.balug.org/pipermail/balug-talk/2021-April/000221.html ===========================