From Michael.Paoli@cal.berkeley.edu Mon Mar 1 13:00:05 2021 From: Michael.Paoli@cal.berkeley.edu (Michael Paoli) Date: Mon, 01 Mar 2021 13:00:05 +0000 Subject: [BALUG-Admin] BALUG: Lists, stats, etc. Message-ID: dates, lists, number of members: YYYY-MM-DD announce talk admin 2021-03-01 833 259 35 2021-02-01 833 259 35 2021-01-01 833 259 35 2020-12-01 832 259 35 2020-11-01 833 260 35 2020-10-01 834 260 35 2020-09-01 835 260 35 2020-08-01 834 259 34 2020-07-01 834 266 34 2020-06-01 838 266 34 2020-05-01 839 267 34 2020-04-01 840 270 34 2020-03-01 840 271 34 2020-02-01 841 271 34 2020-01-01 841 271 34 2019-12-01 841 271 34 2019-11-01 840 271 34 2019-10-01 842 271 34 2019-09-01 844 272 34 2019-08-01 847 272 34 2019-07-01 852 272 36 2019-06-01 854 273 37 2019-05-01 856 272 38 2019-04-01 853 273 38 2019-03-01 851 273 38 2019-02-01 853 273 38 2019-01-01 856 274 38 2018-12-01 858 274 38 2018-11-01 861 274 39 2018-10-02 862 274 39 2018-09-01 864 274 38 2018-07-28 866 278 38 2018-06-08 870 282 38 2018-04-28 875 282 38 2018-02-17 880 283 36 2017-11-04 883 284 36 2017-08-19 897 293 37 2017-07-01 897 297 38 2017-05-27 897 297 38 2017-05-16 897 296 38 2017-04-01 896 297 38 2017-03-30 885 297 38 2016-05-03 886 291 39 2016-04-03 887 290 39 2016-03-05 888 291 45 2016-01-05 887 295 45 2015-12-05 887 296 45 2015-11-07 887 301 45 2015-10-06 888 301 45 2015-09-05 887 301 45 2015-08-04 888 301 45 2015-07-03 888 300 45 2015-06-05 888 301 46 2015-05-02 885 298 45 2015-04-04 884 297 44 2015-03-07 886 298 44 2015-02-06 885 303 44 2015-01-07 885 304 45 2014-12-06 886 304 45 2014-11-01 884 302 45 2014-10-04 881 303 46 2014-09-06 878 309 44 2014-06-13 875 307 44 2014-05-02 874 306 44 2014-04-04 876 306 44 2014-03-01 876 306 43 2014-01-11 872 305 42 2013-10-05 854 293 40 2013-08-31 855 295 41 2013-07-20 852 297 40 2013-04-10 838 286 41 2013-03-03 841 289 41 2013-02-02 843 291 41 2012-12-15 845 310 41 2012-03-23 818 310 39 2012-02-01 813 307 40 2011-04-28 768 291 37 2011-02-27 762 288 36 2010-05-02 730 289 34 2010-04-01 727 291 33 2010-03-16 727 293 33 2009-12-16 716 290 33 2009-09-09 700 288 32 2009-03-27 677 287 29 2009-01-10 662 290 30 2008-08-17 635 290 29 2008-03-03 571 247 28 2008-01-02 337 260 28 2007-12-28 336 261 28 2007-09-03 264 258 27 2007-07-24 287 256 27 2007-06-17 280 249 27 references: http://www.balug.org/#Lists From Michael.Paoli@cal.berkeley.edu Fri Mar 19 04:33:28 2021 From: Michael.Paoli@cal.berkeley.edu (Michael Paoli) Date: Thu, 18 Mar 2021 21:33:28 -0700 Subject: [BALUG-Admin] Some web stats by domains, etc. Message-ID: <20210318213328.355329i622gp4ha8@webmail.rawbw.com> Some web stats by domains, etc. I was interested, thought others might be curious/interested too, so ... I've got the log rotation set up so it retains bit over a year's worth of web server logs $ awk '{if($1=="rotate" || $1 !~ /^#/ && $1 ~ /ly$/)print}' /etc/logrotate.d/apache2 weekly rotate 60 so, fair bit of data available. This web server runs on the balug Virtual Machine (VM) host, which covers for not only the BALUG [Linux] User Group ([L]UG), but others too. Anyway, grabbed the data from the logs, and did a wee bit of analysis. This is from the {ssl_,}access.log* files. So, basically analyzed domain and port, here's the information/analysis, and counts, from the highest levels on down to that: 13945791 Total That's the total traffic seen, all sites/domains that got logged. That's approximately: 232430 per week (13945791/60) 33204 per day (13945791/60/7) 1384 per hour (13945791/60/7/24) 23 per minute (13945791/60/7/24/60) 0.38 per second (13945791/60/7/24/60/60) So about one per 2.6 seconds (1/(13945791/60/7/24/60/60)) by [L]UG or project or whatever and the like. Note also that test traffic isn't excluded, so, e.g. some domains that aren't in DNS or so delegated (or not yet or no longer in DNS) also show wee bit 'o traffic. 9504843 BALUG 3463857 BerkeleyLUG 968724 SF-LUG 8283 digitalwitness.org 51 BAD 33 BUUG In a bit more detail, by relevant TLD: 9504843 balug.org 3457611 berkeleylug.com 896560 sf-lug.org 40847 sf-lug.com 13698 sflug.org 8283 digitalwitness.org 7458 sflug.net 6246 berkeleylug.org 6084 sflug.com 4077 sf-lug.net 51 bad.debian.net 33 buug.org Note that for all of SF-LUG's various domains, traffic drops by >20x once we leave the canonical TLD ([www.]sf-lug.org) and drop to 1st runner up in SF-LUG traffic, >65x to 2nd runner up, and > 120x for 3rd runner up. For BerkeleyLUG, the obscure and almost completely unknown (was never canonical nor particularly promoted) BerkeleyLUG.org, The canonical BerkeleyLUG.com has >550x the traffic ... however BerkeleyLUG.org no longer exists, so it was something >1/550 of the traffic when it still existed (but even back then I recall it being relatively negligible portion of traffic). So, by domain ... secure.balug.org is somewhat surprising. It's probably mostly from "bad" or not so well behaved bots, poking at the BALUG wiki, trying to login there, and getting redirected ... or maybe such bots thinking there's something more "interesting" to go after because it's got "secure" in the name? Maybe ought phase that one out, not nearly so relevant anymore. I think the original idea is that one would redirect to force https, but I believe now all the domains offer https, and as/where relevant (e.g. wiki login and after doing so with cookie that has authenticated state) redirection can be done from http to https with same domain - no need for a separate domain for that. Also a bit surprising too, is BerkeleyLUG.com being as high as it is. It's WordPress, and bots - legitimate, or not so, and search engines, well, WordPress effectively "expands" to a quite huge set of unique URLs, even if the content of each isn't all that incredibly unique. E.g. I remember earlier trying to crawl the site when it was still hosted by WordPress.com - it expanded into a huge amount of content that wasn't particularly feasible to replicate in-place as-is without WordPress - especially relatively to the actual full data in/behind the site - a much more manageable and smaller set of data. So that may, at least partially, explain the somewhat surprising large number there. Likewise, BALUG's wiki - lots of content - especially if one crawls the archive of all older versions of all pages. And BALUG's list - every posting can be individually crawled, so, lots of URLs, and probably keeps search engines / bots relatively busy. And one other that's slightly surprising ... the mx entries. Really noting promoting - or even configured for that as web, other than it existing in DNS. So that's probably mostly "bad" bots and/or a bit of test traffic. Also, pi.berkeleylug.com is slightly, but not entirely redundant with https://berkeleylug.com/Pi.BerkeleyLUG/ Notably pi.berkeleylug.com 302 redirects to the above ... however also, there's account and sudo and dynamic DNS highly available to pi.berkelelylug - so if/whenever they / that SIG wants it to go to somewhere else in DNS, it's readily available for that. But if the DNS & web traffic hits the BALUG host's web server, it 302 ("temporary") redirects as noted above. 4467098 secure.balug.org 3408433 berkeleylug.com 2083824 www.balug.org 1692238 www.wiki.balug.org 998636 lists.balug.org 875806 www.sf-lug.org 185888 balug.org 47829 www.berkeleylug.com 42223 www.archive.balug.org 23561 www.sf-lug.com 17286 sf-lug.com 15402 sf-lug.org 14875 old-debian.balug.org 8993 sflug.org 5372 www.digitalwitness.org 5175 wiki.balug.org 5047 berkeleylug.org 4705 www.sflug.org 4295 sflug.com 4052 sflug.net 3841 www.ipv4.sf-lug.org 3406 www.sflug.net 3291 www.new.balug.org 3291 sf-lug.net 2911 digitalwitness.org 2819 www.test.balug.org 2519 www.ipv4.balug.org 2016 www.beta.balug.org 1789 www.sflug.com 1225 www.php.test.balug.org 1199 www.berkeleylug.org 1182 ipv4.sf-lug.org 1096 mx.balug.org 966 ipv4.balug.org 786 www.sf-lug.net 753 www.pi.berkeleylug.com 611 mx.lists.balug.org 596 pi.berkeleylug.com 266 www.ipv6.balug.org 262 www.ipv6.sf-lug.org 77 ipv6.balug.org 67 ipv6.sf-lug.org 51 bad.debian.net 25 www.buug.org 8 buug.org And, nothing horribly surprising here, given the above. This is essentially same again, but broken down by by port. Also, if we total up all of http (:80) and https (:443) we get: 7263155 :80 (http) 6682636 :443 (https) And by domain and port: 3570578 secure.balug.org:80 2942325 berkeleylug.com:443 1762326 www.balug.org:443 1355057 www.wiki.balug.org:80 896520 secure.balug.org:443 834027 www.sf-lug.org:80 616148 lists.balug.org:443 466108 berkeleylug.com:80 382488 lists.balug.org:80 337181 www.wiki.balug.org:443 321498 www.balug.org:80 140573 balug.org:80 45315 balug.org:443 43816 www.berkeleylug.com:80 41779 www.sf-lug.org:443 29510 www.archive.balug.org:80 22723 www.sf-lug.com:80 15730 sf-lug.com:80 14827 old-debian.balug.org:80 12713 www.archive.balug.org:443 11427 sf-lug.org:80 6618 sflug.org:80 4604 berkeleylug.org:80 4420 www.digitalwitness.org:80 4134 wiki.balug.org:80 4013 www.berkeleylug.com:443 3975 sf-lug.org:443 3548 sflug.com:80 3263 sflug.net:80 3251 www.sflug.org:80 2738 sf-lug.net:80 2485 digitalwitness.org:80 2462 www.test.balug.org:80 2394 www.ipv4.sf-lug.org:80 2375 sflug.org:443 2244 www.new.balug.org:80 1964 www.sflug.net:80 1767 www.beta.balug.org:80 1556 sf-lug.com:443 1454 www.sflug.org:443 1447 www.ipv4.sf-lug.org:443 1442 www.sflug.net:443 1409 www.sflug.com:80 1392 www.ipv4.balug.org:443 1127 www.ipv4.balug.org:80 1047 www.new.balug.org:443 1041 wiki.balug.org:443 975 www.php.test.balug.org:80 952 www.digitalwitness.org:443 879 ipv4.sf-lug.org:80 838 www.sf-lug.com:443 789 sflug.net:443 787 www.berkeleylug.org:80 747 sflug.com:443 736 ipv4.balug.org:80 718 www.pi.berkeleylug.com:80 599 mx.balug.org:80 553 sf-lug.net:443 497 mx.balug.org:443 463 www.sf-lug.net:80 443 berkeleylug.org:443 426 digitalwitness.org:443 412 www.berkeleylug.org:443 381 mx.lists.balug.org:80 380 www.sflug.com:443 376 pi.berkeleylug.com:80 357 www.test.balug.org:443 323 www.sf-lug.net:443 303 ipv4.sf-lug.org:443 250 www.php.test.balug.org:443 249 www.beta.balug.org:443 230 mx.lists.balug.org:443 230 ipv4.balug.org:443 220 pi.berkeleylug.com:443 194 www.ipv6.sf-lug.org:80 173 www.ipv6.balug.org:80 93 www.ipv6.balug.org:443 68 www.ipv6.sf-lug.org:443 67 ipv6.balug.org:443 60 ipv6.sf-lug.org:443 48 old-debian.balug.org:443 45 bad.debian.net:80 35 www.pi.berkeleylug.com:443 18 www.buug.org:80 10 ipv6.balug.org:80 7 www.buug.org:443 7 ipv6.sf-lug.org:80 6 bad.debian.net:443 4 buug.org:80 4 buug.org:443 Anyway, maybe some day I'll get some "real" web reporting in place. It is on the todo list, but ... $ wc todo 6156 22307 188878 todo $ It's also not a FIFO list, nor FILO. It's mostly a priority interrupt driven stack that gets a lot of reordering applied to it, and tends to grow much more than it shrinks. Well, at least I don't have to worry about running out of stuff to do - have well over lifetime's worth 'o stuff on the list. More BALUG and [L]UG specific lists may be found on BALUG's wiki, though they're not necessarily complete nor current.