[BALUG-Talk] Possible solution of blankness or editing outline-only on BALUG DokuWiki?

Fri Apr 23 15:44:22 UTC 2021

Well, not out of the woods yet.

Have done fair bit of compression (even using xz -9) of some
moderately older data on /var ... and have grown /var several
times, ... yet it filled up again very recently (or exceedingly
close - I saw it at 100% again, but that's probably rounded, so
at least >99.5%).

So, at present, now have:
# df -h /var
Filesystem             Size  Used Avail Use% Mounted on
/dev/mapper/balug-var  6.4G  5.7G  491M  93% /var
#
And very recently (only few days back) that was:
> df -h /var
> Filesystem             Size  Used Avail Use% Mounted on
> /dev/mapper/balug-var  4.9G  4.5G  190M  97% /var
Can't just keep growing /var indefinitely.
Much of the volume is mail and related - logs, queued mail,
bounces & failures, mail to listadmin and postmaster, etc., and mostly
lots of other nuisance stuff - fail2ban database takes fair bit of space,
but trimming what that holds would increase data in other logs - notably
of lots of nuisance failed login/authentication attempts and the like.

And, even if I caught it before completely full, some services (e.g.
exim4) will proactively stop processing when the free filesystem
space runs too low.

So, yeah, I'll need to do some more work on it.  May need to analyze why
it's ballooning more than it was in past - what's the nature of the
problematic traffic that keeps driving up the storage.

I suspect it's mostly spam (and attempts thereof) and bad bots
doing/attempting stuff ... essentially all some type(s) of nefarious
unwanted traffic chewing up resources.

So, in only a couple days or so, it's already nearly 1.5 GiB more data,
and that's already after having done fair bit of aggressive data
data compression on some older data (e.g. some of the older mail data
easily compressed from around 200 of MiB to under a few tens of MB ...
and I even moved the compressed off of the /var filesystem).

So, ..., yeah, miscreants on 'da Internet ... more work for me to do.  :-/

Would also be nice to get some of that /var space back, and not have to give
so much of it over to what's basically mostly unintersting crud.

And, have had stuff like this happen before, ... e.g. like bad bots trying
to register on wiki - and chewing up huge volumes of space ... that's why
there's no self-registration option anymore ... blame it on abusers on
'da Internet.  Manual registration only now.  Even captcha's weren't really
enough for most of that ... bad bots still pounded away and repeatedly
filled up log space.  Maybe more fail2ban config would help that ... but then
that also ends up chewing up additional space with its database ...

# du -x /var | sort -bnr | head -n 12
5866812 /var
2102592 /var/lib
1249060 /var/log
1017624 /var/spool
1015100 /var/mail
932200  /var/spool/exim4
731780  /var/lib/fail2ban
616584  /var/lib/clamav
531456  /var/log/apache2
505952  /var/log/exim4
491980  /var/spool/exim4/input
364892  /var/spool/exim4/msglog
#

> From: "aaronco36 via BALUG-Talk" <balug-talk@lists.balug.org>
> Subject: Re: [BALUG-Talk] Possible solution of blankness or editing  
> outline-only on BALUG DokuWiki?
> Date: Thu, 22 Apr 2021 16:21:14 +0000 (UTC)

> Quoting Michael Paoli <Michael.Paoli at cal.berkeley.edu> from [1]:
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> Thanks for the report, but I'm unable to reproduce the issue.
> Anyone else able to reproduce it?  Are you/anyone seeing the
> issue at this time?  Anyone else also see it earlier?
>
> I've tried the following on:  
> https://www.wiki.balug.org/wiki/doku.php without being able to  
> reproduce the issue:
> Chromium
> Chromium incognito mode
> Firefox
> Firefox private browsing mode
> Lynx
> curl
> wget
> Chome on Android from another ISP
> Chrom icognito mode on Android from another ISP
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> Cut-and-pasted a text screen-capture and submitted it as a  
> [BALUG-Test] 'Test of BALUG Wiki pages' posting yesterday morning;  
> see [2].
> AFAICT, that "blankness or editing outline-only issue" for BALUG  
> DokuWiki pages [3] and [4] occurred from before 08:00am PDT up  
> through the late morning, and then went away by mid-afternoon  
> yesterday.
> Probably reloaded both those BALUG DokuWiki pages a half-dozen times  
> or so over the morning time-period using mostly Firefox-ESR standard  
> mode and Firefox-ESR private browsing mode, and maybe once or twice  
> using TOR browser in its [least] Safe mode.
>
> Same results for all browsers used to test those pair of pages  
> during that entire morning time-period.
>
>
> Quoting Michael Paoli <Michael.Paoli at cal.berkeley.edu> from [5]:
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> That may have done it.
> Looks like I need to add more space to /var and/or clean some more stuff
> off of it:
> df -h /var
> Filesystem             Size  Used Avail Use% Mounted on
> /dev/mapper/balug-var  4.9G  4.5G  190M  97% /var
> #
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> Thanks much, Michael, for effectively diagnosing and resolving the issue! :-)
>
> Besides considering the issue arising from the DokuWiki site itself  
> or BALUG's device /dev/mapper/balug-var, was also considering the  
> possibility that the issue might have arisen from my own browser  
> client.
>
> Why so?
>
> Well, there was no issue whatsoever with successfully loading BALUG  
> DokuWiki pages [3] and [4] throughout Sunday and Monday of this week.
> What changed on Tuesday afternoon was upgrading Firefox-ESR to a  
> slightly higher, minor version as per 'less  
> /var/log/apt/history.log' :
>> Start-Date: 2021-04-20  14:10:03
>> Commandline: apt upgrade
>> Upgrade: firefox-esr:amd64 (78.9.0esr-1~deb10u1, 78.10.0esr-1~deb10u1)
>> End-Date: 2021-04-20  14:10:15
>
> One or more bugs in the upgraded FF 78.10.0esr could have  
> conceivably rendered text and links in the BALUG DokuWiki  
> incorrectly afterwards, as remote as this possibility might seem to  
> be.
> It's good to know that this wasn't the case at all, and that this  
> latest Firefox-ESR browser was fine after all :-)
>
> Thanks again Michael!
>
> -Aaron
>
> ===========================
> REFERENCES
> ===========================
> [1]https://lists.balug.org/pipermail/balug-talk/2021-April/000220.html
> [2]https://lists.balug.org/pipermail/balug-test/2021/000048.html
> [3]https://www.wiki.balug.org/wiki/doku.php
> [4]https://www.wiki.balug.org/wiki/doku.php?id=balug:offered_wanted_hardware_etc
> [5]https://lists.balug.org/pipermail/balug-talk/2021-April/000221.html
> ===========================
>
> aaronco36@sdf.org