Some web stats by domains, etc.
I was interested, thought others might be curious/interested too, so ...
I've got the log rotation set up so it retains bit over a year's worth of
web server logs
$ awk '{if($1=="rotate" || $1 !~ /^#/ && $1 ~ /ly$/)print}'
/etc/logrotate.d/apache2
weekly
rotate 60
so, fair bit of data available.
This web server runs on the balug Virtual Machine (VM) host,
which covers for not only the BALUG [Linux] User Group ([L]UG),
but others too. Anyway, grabbed the data from the logs, and did a wee
bit of analysis. This is from the {ssl_,}access.log* files.
So, basically analyzed domain and port, here's the information/analysis,
and counts, from the highest levels on down to that:
13945791 Total
That's the total traffic seen, all sites/domains that got logged.
That's approximately:
232430 per week (13945791/60)
33204 per day (13945791/60/7)
1384 per hour (13945791/60/7/24)
23 per minute (13945791/60/7/24/60)
0.38 per second (13945791/60/7/24/60/60)
So about one per 2.6 seconds (1/(13945791/60/7/24/60/60))
by [L]UG or project or whatever and the like. Note also that test
traffic isn't excluded, so, e.g. some domains that aren't in DNS or
so delegated (or not yet or no longer in DNS) also show wee bit 'o
traffic.
9504843 BALUG
3463857 BerkeleyLUG
968724 SF-LUG
8283 digitalwitness.org
51 BAD
33 BUUG
In a bit more detail, by relevant TLD:
9504843 balug.org
3457611 berkeleylug.com
896560 sf-lug.org
40847 sf-lug.com
13698 sflug.org
8283 digitalwitness.org
7458 sflug.net
6246 berkeleylug.org
6084 sflug.com
4077 sf-lug.net
51 bad.debian.net
33 buug.org
Note that for all of SF-LUG's various domains,
traffic drops by >20x once we leave the canonical TLD ([www.]sf-lug.org)
and drop to 1st runner up in SF-LUG traffic, >65x to 2nd runner up, and
> 120x for 3rd runner up.
For BerkeleyLUG, the obscure and almost completely unknown (was never
canonical nor particularly promoted) BerkeleyLUG.org,
The canonical BerkeleyLUG.com has >550x the traffic ...
however BerkeleyLUG.org no longer exists, so it was something >1/550 of
the traffic when it still existed (but even back then I recall it being
relatively negligible portion of traffic).
So, by domain ... secure.balug.org is somewhat surprising. It's probably
mostly from "bad" or not so well behaved bots, poking at the BALUG wiki,
trying to login there, and getting redirected ... or maybe such bots
thinking there's something more "interesting" to go after because it's
got "secure" in the name? Maybe ought phase that one out, not nearly
so relevant anymore. I think the original idea is that one would
redirect to force https, but I believe now all the domains offer https,
and as/where relevant (e.g. wiki login and after doing so with cookie
that has authenticated state) redirection can be done from http to
https with same domain - no need for a separate domain for that.
Also a bit surprising too, is BerkeleyLUG.com being as high as it is.
It's WordPress, and bots - legitimate, or not so, and search engines,
well, WordPress effectively "expands" to a quite huge set of unique
URLs, even if the content of each isn't all that incredibly unique.
E.g. I remember earlier trying to crawl the site when it was still
hosted by WordPress.com - it expanded into a huge amount of content
that wasn't particularly feasible to replicate in-place as-is without
WordPress - especially relatively to the actual full data in/behind
the site - a much more manageable and smaller set of data. So that
may, at least partially, explain the somewhat surprising large number
there.
Likewise, BALUG's wiki - lots of content - especially if one crawls the
archive of all older versions of all pages.
And BALUG's list - every posting can be individually crawled, so, lots of
URLs, and probably keeps search engines / bots relatively busy.
And one other that's slightly surprising ... the mx entries.
Really noting promoting - or even configured for that as web,
other than it existing in DNS. So that's probably mostly "bad" bots
and/or a bit of test traffic.
Also, pi.berkeleylug.com is slightly, but not entirely redundant with
https://berkeleylug.com/Pi.BerkeleyLUG/
Notably pi.berkeleylug.com 302 redirects to the above ...
however also, there's account and sudo and dynamic DNS highly available
to pi.berkelelylug - so if/whenever they / that SIG wants it to go to
somewhere else in DNS, it's readily available for that. But if the
DNS & web traffic hits the BALUG host's web server, it 302 ("temporary")
redirects as noted above.
4467098 secure.balug.org
3408433 berkeleylug.com
2083824 www.balug.org
1692238 www.wiki.balug.org
998636 lists.balug.org
875806 www.sf-lug.org
185888 balug.org
47829 www.berkeleylug.com
42223 www.archive.balug.org
23561 www.sf-lug.com
17286 sf-lug.com
15402 sf-lug.org
14875 old-debian.balug.org
8993 sflug.org
5372 www.digitalwitness.org
5175 wiki.balug.org
5047 berkeleylug.org
4705 www.sflug.org
4295 sflug.com
4052 sflug.net
3841 www.ipv4.sf-lug.org
3406 www.sflug.net
3291 www.new.balug.org
3291 sf-lug.net
2911 digitalwitness.org
2819 www.test.balug.org
2519 www.ipv4.balug.org
2016 www.beta.balug.org
1789 www.sflug.com
1225 www.php.test.balug.org
1199 www.berkeleylug.org
1182 ipv4.sf-lug.org
1096 mx.balug.org
966 ipv4.balug.org
786 www.sf-lug.net
753 www.pi.berkeleylug.com
611 mx.lists.balug.org
596 pi.berkeleylug.com
266 www.ipv6.balug.org
262 www.ipv6.sf-lug.org
77 ipv6.balug.org
67 ipv6.sf-lug.org
51 bad.debian.net
25 www.buug.org
8 buug.org
And, nothing horribly surprising here, given the above. This is
essentially same again, but broken down by by port.
Also, if we total up all of http (:80) and https (:443) we get:
7263155 :80 (http)
6682636 :443 (https)
And by domain and port:
3570578 secure.balug.org:80
2942325 berkeleylug.com:443
1762326 www.balug.org:443
1355057 www.wiki.balug.org:80
896520 secure.balug.org:443
834027 www.sf-lug.org:80
616148 lists.balug.org:443
466108 berkeleylug.com:80
382488 lists.balug.org:80
337181 www.wiki.balug.org:443
321498 www.balug.org:80
140573 balug.org:80
45315 balug.org:443
43816 www.berkeleylug.com:80
41779 www.sf-lug.org:443
29510 www.archive.balug.org:80
22723 www.sf-lug.com:80
15730 sf-lug.com:80
14827 old-debian.balug.org:80
12713 www.archive.balug.org:443
11427 sf-lug.org:80
6618 sflug.org:80
4604 berkeleylug.org:80
4420 www.digitalwitness.org:80
4134 wiki.balug.org:80
4013 www.berkeleylug.com:443
3975 sf-lug.org:443
3548 sflug.com:80
3263 sflug.net:80
3251 www.sflug.org:80
2738 sf-lug.net:80
2485 digitalwitness.org:80
2462 www.test.balug.org:80
2394 www.ipv4.sf-lug.org:80
2375 sflug.org:443
2244 www.new.balug.org:80
1964 www.sflug.net:80
1767 www.beta.balug.org:80
1556 sf-lug.com:443
1454 www.sflug.org:443
1447 www.ipv4.sf-lug.org:443
1442 www.sflug.net:443
1409 www.sflug.com:80
1392 www.ipv4.balug.org:443
1127 www.ipv4.balug.org:80
1047 www.new.balug.org:443
1041 wiki.balug.org:443
975 www.php.test.balug.org:80
952 www.digitalwitness.org:443
879 ipv4.sf-lug.org:80
838 www.sf-lug.com:443
789 sflug.net:443
787 www.berkeleylug.org:80
747 sflug.com:443
736 ipv4.balug.org:80
718 www.pi.berkeleylug.com:80
599 mx.balug.org:80
553 sf-lug.net:443
497 mx.balug.org:443
463 www.sf-lug.net:80
443 berkeleylug.org:443
426 digitalwitness.org:443
412 www.berkeleylug.org:443
381 mx.lists.balug.org:80
380 www.sflug.com:443
376 pi.berkeleylug.com:80
357 www.test.balug.org:443
323 www.sf-lug.net:443
303 ipv4.sf-lug.org:443
250 www.php.test.balug.org:443
249 www.beta.balug.org:443
230 mx.lists.balug.org:443
230 ipv4.balug.org:443
220 pi.berkeleylug.com:443
194 www.ipv6.sf-lug.org:80
173 www.ipv6.balug.org:80
93 www.ipv6.balug.org:443
68 www.ipv6.sf-lug.org:443
67 ipv6.balug.org:443
60 ipv6.sf-lug.org:443
48 old-debian.balug.org:443
45 bad.debian.net:80
35 www.pi.berkeleylug.com:443
18 www.buug.org:80
10 ipv6.balug.org:80
7 www.buug.org:443
7 ipv6.sf-lug.org:80
6 bad.debian.net:443
4 buug.org:80
4 buug.org:443
Anyway, maybe some day I'll get some "real" web reporting in place.
It is on the todo list, but ...
$ wc todo
6156 22307 188878 todo
$
It's also not a FIFO list, nor FILO.
It's mostly a priority interrupt driven stack that gets a lot
of reordering applied to it, and tends to grow much more than it
shrinks. Well, at least I don't have to worry about running
out of stuff to do - have well over lifetime's worth 'o stuff on the
list.
More BALUG and [L]UG specific lists may be found on BALUG's wiki,
though they're not necessarily complete nor current.
----- Forwarded message from Rick Moen <rick(a)linuxmafia.com> -----
Date: Sat, 21 Nov 2020 12:41:28 -0800
From: Rick Moen <rick(a)linuxmafia.com>
To: Al <aw009(a)sunnyside.com>
Cc: Michael Paoli <Michael.Paoli(a)cal.berkeley.edu>
Subject: Re: Fwd: cooperation
I'm going to also send this to balug-admin for collective knowledge.
Quoting Al Whaley (aw009(a)sunnyside.com):
> Rick,
> this one caught my eye... it seems to have gone through mailman on
> linuxmafia for the SFLUG mailing list.
_Sort_ of, but mostly not.
You as a listadmin for the SF-LUG mailing list receive copies from
Mailman of outside mail addressed to the sf-lug-owner(a)linuxmafia.com
alias (note -owner suffix). If the initial MTA defences don't reject
the inbound mail to that role alias address, then it'll get out to the
listadmins, i.e., to you + me + Jim Stockford (the SF-LUG list's
three-person listadmin roster).
If you check full headers, you'll find that the 419 scam mail
originated from Microsoft Hotmail IP address 40.92.51.100. See these
Received headers for that initial information:
From mailman-bounces(a)linuxmafia.com Sat Nov 21 09: 9:52 2020
Return-path: <mailman-bounces(a)linuxmafia.com>
Envelope-to: rick(a)linuxmafia.com
Delivery-date: Sat, 21 Nov 2020 09:09:52 -0800
Received: from localhost ([127.0.0.1] helo=linuxmafia.com)
by linuxmafia.com with esmtp (Exim 4.72)
(envelope-from <mailman-bounces(a)linuxmafia.com>)
id 1kgWOV-0005CC-Vj; Sat, 21 Nov 2020 09:09:52 -0800
Received: from mail-db8eur06olkn2100.outbound.protection.outlook.com
([40.92.51.100]
helo=EUR06-DB8-obe.outbound.protection.outlook.com)
by linuxmafia.com with esmtp (Exim 4.72)
(envelope-from <jmpumjzzjjsxssvvvzzz(a)hotmail.com>)
id 1kgWOQ-0005C3-Vq; Sat, 21 Nov 2020 09:09:50 -0800
Grepping the MTA logs for that IP....
linuxmafia:/var/log/exim4# zgrep 40.92.51.100 mainlog*
mainlog:2020-11-21 09:09:50 1kgWOQ-0005C3-Vq SA: Action: scanned but message isn't spam: score=0.8 required=4.0 (scanned in 3/3 secs | Message-Id: 1kgWOQ-0005C3-Vq). From <jmpumjzzjjsxssvvvzzz(a)hotmail.com> (host=mail-db8eur06olkn2100.outbound.protection.outlook.com
[40.92.51.100]) for sf-lug-owner(a)linuxmafia.com, sf-lug(a)linuxmafia.com mainlog:2020-11-21 09:09:50 1kgWOQ-0005C3-Vq <= jmpumjzzjjsxssvvvzzz(a)hotmail.com H=mail-db8eur06olkn2100.outbound.protection.outlook.com (EUR06-DB8-obe.outbound.protection.outlook.com) [40.92.51.100] P=esmtp S=8365 id=AM6PR08MB424596EC3D85FB4BE09C3DD6CDFE0(a)AM6PR08MB4245.eurprd08.prod.outlook.com
linuxmafia:/var/log/exim4#
...you can see that the SMTP session, i.e., the SMTP envelope
information embodied in the SMTP delivery conversation, must have contained
MAIL FROM: <jmpumjzzjjsxssvvvzzz(a)hotmail.com>
RCPT TO: <sf-lug-owner(a)linuxmafia.com>
RCPT TO: <sf-lug(a)linuxmafia.com>
Measured spamicity (0.8) at receiving MTA level was nowhere near high
enough for my MTA to reject (4.0), because this really _was_ from a
Hotmail user, and the mail's body text was not sufficiently spammy to a
degree that automated MTA-level spamd scanning for garbage adjudged
processed meat, so this 419 scam mail got passed along to the Mailman
MLM. Mailman passed through the copy _back_ to the MTA that was
addressed out to the listadmins, _but_ by contrast lodged the direct
mailing list copy in Mailman's admin queue for sf-lug(a)linuxmafia.com,
whence it will expire out in five days and then automatically get thrown
away.
Why in the MLM admin queue? Two reasons:
1. Since in this case the scam mail's internal SMTP headers were
'implicitly addressed', i.e., there were no To: or Cc: addresses
whatsoever inside the message proper, only SMTP envelope addressees
(outside the message proper), Mailman will never send such mail out to
subscribers, on account of Mailman's built-in filter blocking all
'implicitly addressed' mail. I.e., Mailman requires that valid mail to
subscribers specify on either the internal To: header or the internal CC:
header the mailing list's direct address or an alias the listadmins have
specified inside the admin WebUI. If that header is _not_ present, the
message deterministically lodges in the admin queue, flagged as
'implicitly addressed'.
2. Also, it didn't originate from a subscriber, hence would (for that
reason as well) have been intercepted and held as being from a
non-subscribed address.
> it's not in the archives so I'm assuming it didn't go out to
> everyone, unless you deleted it, and it wasn't marked
> as being sent for moderation...
> what am I missing here?
Naturally, I would love for inbound junkmail filtering to reject even
more than it already does, but, frankly, linuxmafia.com system-level
spam-rejection is already quite good. Diminishing returns and false
positives rapidly become a problem as one tries to make MTA spam
immune response even better.
Where I sit, it seems to me it's a fact of life that being a Mailman
addressee on any [listname]-owner aliases _is_ going to get you
increased MLM-mediated spam because of being a member of that alias, as
additional protections covering MLM subscribers simply don't apply to
that admin-role alias address.
Speaking of that, I believe it's past time that I remove Jim Stockford
(i.e., jim(a)well.com) from the sf-lug-owner(a)linuxmafia.com roster, and
I'm going to do that right now.
It realised I probably needed to get around to that, about a year or so
ago, when he was _publicly_ claiming on the SF-LUG mailing list that
linuxmafia.com is an SMTP spam source, very likely because of having
received some amount of junkmail via the sf-lug-owner alias. (His
public accusation was, however, _unbelievably_ vague, so I couldn't
investigate his claim.)
You, by contrast, did exactly the right thing in asking me rather than
going on a public accusation campaign, so the comparison is night and
day. That distinction having been made, to this day I resent Jim's
(false) public accusation, and the only thing that mitigates it is that,
for unclear personal reasons, Jim has been lately unable to discharge
his responsibilities.
I didn't want to delete him from the roster in haste, since Jim founded
the group, and supposedly solemnly committed to being _the_ listadmin
responsible for it, (but has never actually done the job, over the
entire 15-year period I've given SF-LUG temporary mailing list hosting,
pending them getting their server running again -- which never
happened).
Done. jim(a)well.com is now omitted from the SF-LUG listadmin roster.
(Just for the record: This means that the _sole_ requirement I insisted
on, before being willing to grant SF-LUG temporary mailing list hosting
to help after whatever mysterious failure destroyed SF-LUG's -own-
mailing list server in late 2005 -- the requirement that Jim shoulder
listadmin duties so I don't have my workload increased by SF-LUG's
presence on my server -- has been continuously in default from that day
in December 2005 to the present.)
----- End forwarded message -----
I get emails ... <sigh>
... seems to be a lot 'o these goin' around:
Threat/extortion emails (mostly all bark no or negligible bite).
Seems they're probably targeting those more probable to fall for it,
e.g. smaller businesses and such that aren't particularly tech savvy,
... not that their targeting is particularly accurate, but maybe at
least what they aim for by the content of their email.
Trimmed and redacted for brevity, etc., both of these from same IP:
------------------------------------------------------------------------
Received: from dedi.coronaxy.com ([185.198.58.92]:38380)
by balug-sf-lug-v2.balug.org
for <balug-announce(a)lists.balug.org>; Tue, 27 Oct 2020 04:41:53 +0000
To: balug-announce(a)lists.balug.org
Date: Tue, 27 Oct 2020 04:41:34 +0000
From: Patrick Wilson <patrick-wilson(a)coronaxy.com>
Subject: Your website www.balug.org has been hacked
We are the Cozy Bear and we have chosen www.balug.org as target for
our next DDoS attack.
Your network will be subject to a DDoS attack starting at 2020
November 2nd (Monday).
THIS IS NOT A HOAX, and to prove it right now we will start a small
attack on www.balug.org that will last for 30 minutes.
This means that your website, e-mail and other connected services will
be unavailable for everyone.
We will refrain from attacking your servers for a small fee.
The current fee is $1200(USD) in bitcoins (BTC). The fee will increase
by 1000 USD for each day after deadline that passed without payment.
Please send Bitcoin to the following Bitcoin address (cAsE-SeNsitIve):
[REDACTED]
If you decide not to pay, we will start the attack on the indicated
date and uphold it until you do, there's no counter measure to this,
you will only end up wasting more money trying to find a solution
(Cloudflare, Sucuri, Imperva and similar services are useless, because
we will hit your network directly).
We will completely destroy your reputation and make sure your services
will remain offline until you pay.
We will also download your database and do as much damage as possible.
Once you have paid we won't start the attack and you will never hear
from us again.
------------------------------------------------------------------------
From <elijah-martin(a)coronaxy.com> Tue Oct 27 17:35:38 2020
Received: from dedi.coronaxy.com ([185.198.58.92]:56414)
for <balug-announce-owner(a)lists.balug.org>; Tue, 27 Oct 2020 17:35:38 +0000
To: balug-announce-owner(a)lists.balug.org
Date: Tue, 27 Oct 2020 17:35:16 +0000
From: Elijah Martin <elijah-martin(a)coronaxy.com>
Subject: If www.balug.org is important to you, you must read this
We have hacked your website(s) www.balug.org and extracted your databases.
We will systematically go through a series of steps of totally
damaging your reputation. First your database will be leaked or sold
to the highest bidder which they will use with whatever their
intentions are. Next if there are e-mails found they will be e-mailed
that their information has been sold or leaked and your site
www.balug.org was at fault thusly damaging your reputation and having
angry customers/associates with whatever angry customers/associates
do. Lastly any links that you have indexed in the search engines will
be de-indexed based off of blackhat techniques that we used in the
past to de-index our targets.
How do I stop this?
We are willing to refrain from destroying your site's reputation for a
small fee. The current fee is $1050(USD) in bitcoins (BTC).
Please send the bitcoin to the following Bitcoin address (Copy and
paste as it is case sensitive):
[REDACTED]
If you decide not to pay, we will start the attack at the indicated
date and uphold it until you do, there's no counter measure to this,
you will only end up wasting more money trying to find a solution. We
will completely destroy your reputation amongst google and your
customers.
This is not a hoax, do not reply to this email, don't try to reason or
negotiate, we will not read any replies. Once you have paid we will
stop what we were doing and you will never hear from us again!
------------------------------------------------------------------------
And yes, simple Internet search to confirm, and many such threats being
emailed about.
Maybe if I get bored and have some time some day, I'll put in an
anti-spam filter to specifically reject these upon attempted delivery
during the SMTP phase ... maybe with reject message/diagnostic
including something along the lines of:
Due to overwhelming request volume, we are no longer able to accept
ransomware/blackmail/threat requests for payment via email.
We do however, still accept such requests via USPS mail.
For prompt attention to your request, please send it via USPS to:
<automatically inserted email address they attempted to> c/o
ref: <automatically inserted reference on failed email attempt>
FBI: ATTN: ransomware/blackmail/threat requests for payment
<address>
USA
Thank you for your cooperation on these matters to help ensure prompt
attention by those relevant and responsible for handling your request.