[BALUG-Talk] Better now (email/lists, etc.) (was: Re: -->BALUG-Admin Re: Possible solution of blankness or editing outline-only on BALUG DokuWiki?)

Sun Apr 25 11:37:17 UTC 2021

Did turn email/SMTP (exim4) off for a while while addressing issues,
and that would also stop/delay list traffic.

But now things should generally be much better than they were.  Yea!
More details can(/will) be found on the BALUG-Admin list & archives.
https://lists.balug.org/cgi-bin/mailman/listinfo/balug-admin
balug-admin@lists.balug.org
https://lists.balug.org/pipermail/balug-admin/
There's also quite a bit of detail on wiki page:
https://www.wiki.balug.org/wiki/doku.php?id=system:annoyances
Some of the follow-on improvements and other bits/details may be or go to
other wiki page(s) and/or be covered on the aforementioned BALUG-Admin
list.

> From: "Michael Paoli" <Michael.Paoli@cal.berkeley.edu>
> Subject: -->BALUG-Admin Re: [BALUG-Talk] Possible solution of  
> blankness or editing outline-only on BALUG DokuWiki?
> Date: Fri, 23 Apr 2021 21:40:57 -0700

> I'm going to suggest taking this discussion to the
> BALUG-Admin <BALUG-Admin@lists.balug.org>
> https://lists.balug.org/cgi-bin/mailman/listinfo/balug-admin
> list, as,
> though the matter does involved Linux somewhat generally, it's much
> more specific and relevant to BALUG and the specific inner
> workings thereof.  So, expect my next update and further communications
> on the matter to be on the BALUG-Admin list, and I'll likely start  
> it off with
> latest update(s) and some background references/links on the matter, at least
> as of late.
>
> Also ...
> dates, lists, number of members:
> YYYY-MM-DD announce talk admin
> 2021-04-01      832  259    35
> https://lists.balug.org/pipermail/balug-admin/2021-April/001046.html
> I'm guestmating probably not around 259 subscribers are that interested
> in these particularly details ... but quite possibly closer to about
> 35 are ... or would be.
> I may well also put up a wiki page regarding findings/developments ...
> or maybe update an older one if such suitable wiki page already exists.
>
> Anyway, I still have some 'work' to do ...
> # date -Iseconds && df -h /var
> 2021-04-24T04:39:59+00:00
> Filesystem             Size  Used Avail Use% Mounted on
> /dev/mapper/balug-var  6.4G  5.9G  287M  96% /var
> #
>
>> From: "Michael Paoli" <Michael.Paoli@cal.berkeley.edu>
>> Subject: Re: [BALUG-Talk] Possible solution of blankness or editing  
>> outline-only on BALUG DokuWiki?
>> Date: Fri, 23 Apr 2021 08:44:22 -0700
>
>> Well, not out of the woods yet.
>>
>> Have done fair bit of compression (even using xz -9) of some
>> moderately older data on /var ... and have grown /var several
>> times, ... yet it filled up again very recently (or exceedingly
>> close - I saw it at 100% again, but that's probably rounded, so
>> at least >99.5%).
>>
>> So, at present, now have:
>> # df -h /var
>> Filesystem             Size  Used Avail Use% Mounted on
>> /dev/mapper/balug-var  6.4G  5.7G  491M  93% /var
>> #
>> And very recently (only few days back) that was:
>>> df -h /var
>>> Filesystem             Size  Used Avail Use% Mounted on
>>> /dev/mapper/balug-var  4.9G  4.5G  190M  97% /var
>> Can't just keep growing /var indefinitely.
>> Much of the volume is mail and related - logs, queued mail,
>> bounces & failures, mail to listadmin and postmaster, etc., and mostly
>> lots of other nuisance stuff - fail2ban database takes fair bit of space,
>> but trimming what that holds would increase data in other logs - notably
>> of lots of nuisance failed login/authentication attempts and the like.
>>
>> And, even if I caught it before completely full, some services (e.g.
>> exim4) will proactively stop processing when the free filesystem
>> space runs too low.
>>
>> So, yeah, I'll need to do some more work on it.  May need to analyze why
>> it's ballooning more than it was in past - what's the nature of the
>> problematic traffic that keeps driving up the storage.
>>
>> I suspect it's mostly spam (and attempts thereof) and bad bots
>> doing/attempting stuff ... essentially all some type(s) of nefarious
>> unwanted traffic chewing up resources.
>>
>> So, in only a couple days or so, it's already nearly 1.5 GiB more data,
>> and that's already after having done fair bit of aggressive data
>> data compression on some older data (e.g. some of the older mail data
>> easily compressed from around 200 of MiB to under a few tens of MB ...
>> and I even moved the compressed off of the /var filesystem).
>>
>> So, ..., yeah, miscreants on 'da Internet ... more work for me to do.  :-/
>>
>> Would also be nice to get some of that /var space back, and not have to give
>> so much of it over to what's basically mostly unintersting crud.
>>
>> And, have had stuff like this happen before, ... e.g. like bad bots trying
>> to register on wiki - and chewing up huge volumes of space ... that's why
>> there's no self-registration option anymore ... blame it on abusers on
>> 'da Internet.  Manual registration only now.  Even captcha's weren't really
>> enough for most of that ... bad bots still pounded away and repeatedly
>> filled up log space.  Maybe more fail2ban config would help that  
>> ... but then
>> that also ends up chewing up additional space with its database ...
>>
>> # du -x /var | sort -bnr | head -n 12
>> 5866812 /var
>> 2102592 /var/lib
>> 1249060 /var/log
>> 1017624 /var/spool
>> 1015100 /var/mail
>> 932200  /var/spool/exim4
>> 731780  /var/lib/fail2ban
>> 616584  /var/lib/clamav
>> 531456  /var/log/apache2
>> 505952  /var/log/exim4
>> 491980  /var/spool/exim4/input
>> 364892  /var/spool/exim4/msglog
>> #
>>
>>> From: "aaronco36 via BALUG-Talk" <balug-talk@lists.balug.org>
>>> Subject: Re: [BALUG-Talk] Possible solution of blankness or  
>>> editing outline-only on BALUG DokuWiki?
>>> Date: Thu, 22 Apr 2021 16:21:14 +0000 (UTC)
>>
>>> Quoting Michael Paoli <Michael.Paoli at cal.berkeley.edu> from [1]:
>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>> Thanks for the report, but I'm unable to reproduce the issue.
>>> Anyone else able to reproduce it?  Are you/anyone seeing the
>>> issue at this time?  Anyone else also see it earlier?
>>>
>>> I've tried the following on:  
>>> https://www.wiki.balug.org/wiki/doku.php without being able to  
>>> reproduce the issue:
>>> Chromium
>>> Chromium incognito mode
>>> Firefox
>>> Firefox private browsing mode
>>> Lynx
>>> curl
>>> wget
>>> Chome on Android from another ISP
>>> Chrom icognito mode on Android from another ISP
>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>
>>> Cut-and-pasted a text screen-capture and submitted it as a  
>>> [BALUG-Test] 'Test of BALUG Wiki pages' posting yesterday morning;  
>>> see [2].
>>> AFAICT, that "blankness or editing outline-only issue" for BALUG  
>>> DokuWiki pages [3] and [4] occurred from before 08:00am PDT up  
>>> through the late morning, and then went away by mid-afternoon  
>>> yesterday.
>>> Probably reloaded both those BALUG DokuWiki pages a half-dozen  
>>> times or so over the morning time-period using mostly Firefox-ESR  
>>> standard mode and Firefox-ESR private browsing mode, and maybe  
>>> once or twice using TOR browser in its [least] Safe mode.
>>>
>>> Same results for all browsers used to test those pair of pages  
>>> during that entire morning time-period.
>>>
>>>
>>> Quoting Michael Paoli <Michael.Paoli at cal.berkeley.edu> from [5]:
>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>> That may have done it.
>>> Looks like I need to add more space to /var and/or clean some more stuff
>>> off of it:
>>> df -h /var
>>> Filesystem             Size  Used Avail Use% Mounted on
>>> /dev/mapper/balug-var  4.9G  4.5G  190M  97% /var
>>> #
>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>
>>> Thanks much, Michael, for effectively diagnosing and resolving the  
>>> issue! :-)
>>>
>>> Besides considering the issue arising from the DokuWiki site  
>>> itself or BALUG's device /dev/mapper/balug-var, was also  
>>> considering the possibility that the issue might have arisen from  
>>> my own browser client.
>>>
>>> Why so?
>>>
>>> Well, there was no issue whatsoever with successfully loading  
>>> BALUG DokuWiki pages [3] and [4] throughout Sunday and Monday of  
>>> this week.
>>> What changed on Tuesday afternoon was upgrading Firefox-ESR to a  
>>> slightly higher, minor version as per 'less  
>>> /var/log/apt/history.log' :
>>>> Start-Date: 2021-04-20  14:10:03
>>>> Commandline: apt upgrade
>>>> Upgrade: firefox-esr:amd64 (78.9.0esr-1~deb10u1, 78.10.0esr-1~deb10u1)
>>>> End-Date: 2021-04-20  14:10:15
>>>
>>> One or more bugs in the upgraded FF 78.10.0esr could have  
>>> conceivably rendered text and links in the BALUG DokuWiki  
>>> incorrectly afterwards, as remote as this possibility might seem  
>>> to be.
>>> It's good to know that this wasn't the case at all, and that this  
>>> latest Firefox-ESR browser was fine after all :-)
>>>
>>> Thanks again Michael!
>>>
>>> -Aaron
>>>
>>> ===========================
>>> REFERENCES
>>> ===========================
>>> [1]https://lists.balug.org/pipermail/balug-talk/2021-April/000220.html
>>> [2]https://lists.balug.org/pipermail/balug-test/2021/000048.html
>>> [3]https://www.wiki.balug.org/wiki/doku.php
>>> [4]https://www.wiki.balug.org/wiki/doku.php?id=balug:offered_wanted_hardware_etc
>>> [5]https://lists.balug.org/pipermail/balug-talk/2021-April/000221.html
>>> ===========================