Greetings! This is an advisory about ns1.linuxmafia.com DNS nameserver
downtime having ended. Root cause: AT&T (_not_ my ISP) sabotaged
my ASDL at their local exchange around 8am Tueday, then took about
2 days and 7 hours to find and fix their problem. All services
are back.
ns1.linuxmafia.com is back to doing auth. nameservice, as
arranged, for the following domains of yours:
balug.org (slave)
sf-lug.com (slave)
sf-lug.org (slave)
Evidence below is via fugly shell script ~/bin/testns that
I just cranked out:
#!/bin/bash
domain=$1
for ns in $(whois $domain | grep "Name Server" | \
awk '{ print $3 }' | tr '\r\n' ' ');
do echo -n $ns 'is '; dig +short @"$ns". $domain. SOA | awk '{print $3}';
done
:r! bin/testns balug.orgNS1.LINUXMAFIA.COM is 1540540908
NS1.SVLUG.ORG is 1540540908
NS1.BALUG.ORG is 1540540908
:r! bin/testns sf-lug.comNS.PRIMATE.NET is 1540541199
NS1.LINUXMAFIA.COM is 1540541199
NS1.SF-LUG.COM is 1540541199
:r! bin/testns sf-lug.orgNS1.LINUXMAFIA.COM is 1540541435
NS.PRIMATE.NET is 1539414019
NS1.SVLUG.ORG is 1540541435
NS1.SF-LUG.ORG is 1540541435
DO NOT REPLY ALL! (*unless* you're a member of **both** lists)
Just an FYI mostly, but temporarily we have a bit of reduced
redundancy on DNS for balug.org/sf-lug.org/sf-lug.com
Expecting this to be relatively temporary (at least once AT&T finally
gets their sh*t together ... at least for a little bit).
Anyway, in case anyone wonders or notices. Hopefully it will
be *all better soon* ... at least **relatively** soon.
Impacts should mostly be pretty minimal - some initial queries might
occasionally take a bit longer (possibly timing out on first
(randomly selected) authoritative nameserver), but should otherwise
generally have almost zero impact (due to TTLs, data will generally
be cached for a while once successfully resolved).
$ (for d in balug.orgsf-lug.orgsf-lug.com; do dig +noall +answer
+nottl "$d". NS | sed -e 's/[ ]\{1,\}/ /g'; done) | fgrep
linuxmafia.combalug.org. IN NS ns1.linuxmafia.com.
sf-lug.org. IN NS ns1.linuxmafia.com.
sf-lug.com. IN NS ns1.linuxmafia.com.
$
----- Forwarded message from rickmoen(a)gmail.com -----
Date: Wed, 24 Oct 2018 01:59:36 -0700
From: "Rick Moen" <rickmoen(a)gmail.com>
Reply-To: rick(a)deirdre.net
Subject: ns1.linuxmafia.com downtime
To: "Michael Paoli" <Michael.Paoli(a)cal.berkeley.edu>
Greetings! This is an advisory about current downtime of my
ns1.linuxmafia.com DNS nameserver, starting about 8am on Tuesday, Oct.
23rd. Near and I and Mike Durkin, proprietor of Raw Bandwidth
Communications ('RBC', my ISP) have been able to determine, AT&T somehow
sabotaged my household ASDL, and thus took my entire household including
the server online. Mike is now trying to get them to fix they screw-up.
Meantime, ns1.linuxmafia.com is _not_ doing auth. nameservice, as
arranged, for the following domains of yours:
balug.org (slave)
sf-lug.com (slave)
sf-lug.org (slave)
I'm advising everyone I'm doing auth. DNS for of the ongoing outage,
so this is your notice. Hope to give better news soon.
----- End forwarded message -----
BALUG VM was down for fair while earlier today.
Has now been up again for over 7 hours now.
Looks like there was an I/O hiccup on the physical host,
which didn't particularly impact the physical hosts, but
was enough of an interruption (delay) that the BALUG VM kernel
paniced.
Did have a 3rd hard drive testing, etc. on the physical host
at the time ... might've hit issues and possibly it did a bus
reset? Who knows for sure. Anyway ...
Went down sometime after:
2018-09-02T01:27:36-07:00
and was brought back up around:
2018-09-02T13:39:30-07:00
Various bits I noted in log:
$ curl -s --range 375155-378925 http://www.archive.balug.org/log.txt
2018-09-02 Michael Paoli
host crashed sometime after:
2018-09-02T01:27:36-07:00
but probably before about:
2018-09-02T01:35:00-07:00
on console, we got:
# [54894.969741] sd 0:0:0:0: [sda] tag#3 ABORT operation started
[54900.078084] sd 0:0:0:0: ABORT operation timed-out.
[54900.080312] sd 0:0:0:0: [sda] tag#2 ABORT operation started
[54905.198438] sd 0:0:0:0: ABORT operation timed-out.
[54905.200517] sd 0:0:0:0: [sda] tag#1 ABORT operation started
[54905.357128] Kernel panic - not syncing: assertion "i &&
sym_get_cam_status(cp->cmd) == DID_SOFT_ERROR" failed: file
"/build/linux-AcJpTp/linux-4.9.110/drivers/scsi/sym53c8xx_2/sym_hipd.c", line
3399
[54905.357128]
[54905.367774] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.9.0-8-amd64
#1 Debian 4.9.110-3+deb9u4
[54905.370776] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[54905.372768] 0000000000000000 ffffffff84f31e54 ffff9e2f75d5a300
ffff9e2f7fc03e50
[54905.375471] ffffffff84d7f6ad 0000000000000020 ffff9e2f7fc03e60
ffff9e2f7fc03df8
[54905.378226] 3ea9db08406f9671 0000000100d04ae4 ffffffffc048a250
ffffffffc0489e80
[54905.380982] Call Trace:
[54905.381867] <IRQ> [54905.382541] [<ffffffff84f31e54>] ?
dump_stack+0x5c/0x78
[54905.384428] [<ffffffff84d7f6ad>] ? panic+0xe4/0x23f
[54905.386164] [<ffffffffc048512e>] ? sym_interrupt+0x1c9e/0x1e80 [sym53c8xx]
[54905.388543] [<ffffffffc03aa010>] ?
usb_hcd_poll_rh_status+0x170/0x170 [usbcore]
[54905.391102] [<ffffffffc03a9fc9>] ?
usb_hcd_poll_rh_status+0x129/0x170 [usbcore]
[54905.393627] [<ffffffffc03aa010>] ?
usb_hcd_poll_rh_status+0x170/0x170 [usbcore]
[54905.396144] [<ffffffff84ce7562>] ? call_timer_fn+0x32/0x120
[54905.398071] [<ffffffffc047ea4b>] ? sym53c8xx_intr+0x3b/0x70 [sym53c8xx]
[54905.400386] [<ffffffff84cd418e>] ? __handle_irq_event_percpu+0x7e/0x1a0
[54905.402673] [<ffffffff84cd42e0>] ? handle_irq_event_percpu+0x30/0x70
[54905.404898] [<ffffffff84cd4359>] ? handle_irq_event+0x39/0x60
[54905.406901] [<ffffffff84cd7870>] ? handle_fasteoi_irq+0xa0/0x170
[54905.409001] [<ffffffff84c27faf>] ? handle_irq+0x1f/0x30
[54905.410834] [<ffffffff852187ee>] ? do_IRQ+0x4e/0xe0
[54905.412528] [<ffffffff85216556>] ? common_interrupt+0x96/0x96
[54905.414523] <EOI> [54905.415216] [<ffffffff852151f0>] ?
__sched_text_end+0x1/0x1
[54905.417231] [<ffffffff852154c2>] ? native_safe_halt+0x2/0x10
[54905.419235] [<ffffffff8521520a>] ? default_idle+0x1a/0xd0
[54905.421137] [<ffffffff84cbc7da>] ? cpu_startup_entry+0x1ca/0x240
[54905.423215] [<ffffffff8593df5e>] ? start_kernel+0x447/0x467
[54905.425186] [<ffffffff8593d120>] ? early_idt_handler_array+0x120/0x120
[54905.427438] [<ffffffff8593d408>] ? x86_64_start_kernel+0x14c/0x170
[54905.429842] Kernel Offset: 0x3c00000 from 0xffffffff81000000
(relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[54905.433484] ---[ end Kernel panic - not syncing: assertion "i &&
sym_get_cam_status(cp->cmd) == DID_SOFT_ERROR" failed: file
"/build/linux-AcJpTp/linux-4.9.110/drivers/scsi/sym53c8xx_2/sym_hipd.c", line
3399
[54905.433484]
... also noted within that same timeframe, on physical host, there
were some storage related events ... but no hard failues seen on that
physical host and no outages or failures or such observed on that
physical host:
Sep 2 01:29:04 vicki smartd[1093]: Device: /dev/sda [SAT], SMART
Usage Attribute: 195 Hardware_ECC_Recovered changed from 63 to 69
Sep 2 01:29:04 vicki smartd[1093]: Device: /dev/sdb [SAT], SMART
Usage Attribute: 190 Airflow_Temperature_Cel changed from 69 to 70
Sep 2 01:29:04 vicki smartd[1093]: Device: /dev/sdb [SAT], SMART
Usage Attribute: 194 Temperature_Celsius changed from 31 to 30
Sep 2 01:29:04 vicki smartd[1093]: Device: /dev/sdb [SAT], SMART
Usage Attribute: 195 Hardware_ECC_Recovered changed from 63 to 66
$
Amusing EximConfig overzealousness.
----- Forwarded message from Mail Delivery System <Mailer-Daemon(a)linuxmafia.com> -----
Date: Fri, 22 Jun 2018 22:11:56 -0700
From: Mail Delivery System <Mailer-Daemon(a)linuxmafia.com>
To: rick(a)linuxmafia.com
Subject: Mail delivery failed: returning message to sender
This message was created automatically by mail delivery software.
A message that you sent could not be delivered to one or more of its
recipients. This is a permanent error. The following address(es) failed:
balug-talk(a)lists.balug.org
SMTP error from remote mail server after end of data:
host mx.lists.balug.org [198.144.194.238]: 550-Rejected message body text:
URL link to prohibited file:
550-http://linuxmafia.com/faq/Admin/linuxmafia.com
550-.
550-[EximConfig-2.5-balug.org-Body-Reject]
550-.
550-.Verify: verified-29120-balug-talk(a)lists.balug.org
550-Contact: postmaster(a)balug.org
550-.
550-Sorry, your message has been rejected because
550-its body text/content is prohibited for the
550-above reason.
550-.
550-We apologise if you have sent a legitimate
550-message and it has been blocked. If this is
550-the case, please re-send adding verified-29120-
550-to the beginning of the E-mail address of each
550-recipient. If you do this, your message will
550-get through these restrictions.
550-.
550-If your message has been incorrectly blocked,
550-please let us know at the above contact address.
550 .
------ This is a copy of the message, including all the headers. ------
Return-path: <rick(a)linuxmafia.com>
Received: from rick by linuxmafia.com with local (Exim 4.72)
(envelope-from <rick(a)linuxmafia.com>)
id 1fWapc-00029x-Cx
for balug-talk(a)lists.balug.org; Fri, 22 Jun 2018 22:11:28 -0700
Date: Fri, 22 Jun 2018 22:11:28 -0700
From: Rick Moen <rick(a)linuxmafia.com>
To: balug-talk(a)lists.balug.org
Subject: Re: [BALUG-Talk] (forw) Re: [License-review] Fwd: [Non-DoD Source]
Resolution on NOSA 2.0
Message-ID: <20180623051128.GD32401(a)linuxmafia.com>
References: <20180622212441.GA32401(a)linuxmafia.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
In-Reply-To: <20180622212441.GA32401(a)linuxmafia.com>
Organization: If you lived here, you'd be $HOME already.
X-Mas: Bah humbug.
X-Clacks-Overhead: GNU Terry Pratchett
User-Agent: Mutt/1.5.20 (2009-06-14)
X-SA-Exim-Connect-IP: <locally generated>
X-SA-Exim-Mail-From: rick(a)linuxmafia.com
X-SA-Exim-Scanned: No (on linuxmafia.com); SAEximRunCond expanded to false
Just because, I'll sharpen some of those points that are relevant to
Tuesday's BALUG discussion.
Speaker Liz Krumbach, after her excellent talk about DC/OS, gave
generously of her time for a Q&A session with attendees sitting around
the table with her at Henry's Hunan.
In passing, Liz mentioned (and I'll paraphrase, here, and apologise in
advance if I misstate anything) that, although her heart is still in the
notion of independent Internet hosting by individuals and community
institutions, the reality is that pretty much everything's moving to
some form of hosted cloud storage, and that even she and her husband now
freely put some fairly sensitive personal data in Google Docs these
days, even though they know all about the drawbacks. (Fair enough.
That's her perception, and her and her husband's choice, and I know Liz
well enough to understand her actions to be thoughtful.)
She also pointed out, again perfectly reasonably, that increasingly most
people don't even host their own files in practically any category,
streaming their video content, streaming their music, and so on. Even
though they _could_ be more independent.
Two BALUG attendees, whom I could name (but will be nice), _then_ chimed in,
going rather far beyond what Liz said (while acting as if they were
agreeing), says that anyone who attempts to run a personal Internet
server, and they said they knew there were some sitting around the
table, were certifiable lunatics, and that it was utterly impractical.
(Hi, I'm Rick Moen, listadmin of most of the Bay Area's Linux User Group
mailing lists and owner/operator of linuxmafia.com / unixmercenary.net,
home of two LUGs, a major community calendar, etc. on old castoff
machines I run literally on fixed IP on home ADSL in my garage. I've
done this sort of thing since the 1980s, when as a staff accountant with
no IT background, I started running my own Internet servers because
nobody told me I couldn't. I just threw things together, tinkered a
little, learned on the fly, and it worked. _Later_, I became a system
administrator as my profession.)
> The Scheme community failed to take basic steps to ensure that there was
> functional, periodically tested failover to independent machine
> resources, and also tested offsite backups.
The core of this is: Do you have a creditable plan of action for each
of the things that might be plausibly be expected to go wrong? Planning
for Internet infrastructure, no different from planning for _any_
organised activity, requires contingency planning. Running a
department, or pretty much anything else? Then, make sure people know
what to do or not do if there's a fire. Or a power outage. Or a
flooded floor in an office building because the sprinklers went off or
the pipes broke upstairs.
In the case of computers and network infrastructure, it's the same
principle, and thus not really different.
Hardware: Moving-rust mass storage, fans, cables, and input devices
wear out most often, from mechanical stress. SSDs wear out too, from a
different type of stress. Do you have a plan when, not if, that
happens?
Software: Upgrades can go tragically bottom-side up if coincidental to a
hardware failure. (Ask me how I know.) Tired sysadmins, or any junior
SA armed with root access, is even more dangerous to systems than a
programmer bearing a screwdriver. Mishaps happen. Do you have a
documented plan for _what_ to back up andexactly how to restore, and
have it somewhere safe? (I do, and archive.org and cache.google.com
back it up for me:
http://linuxmafia.com/faq/Admin/linuxmafia.com-backup.html )
Is your DNS free of single points of failure, and multiple people able
and willing to administer the domain and DNS?
Do you have either failover or a credible plan to bring replacement
hardware / software online if it's important but not urgent? Like:
> Life is imperfect, so I know these things usually amount to 'We didn't
> have time and energy to do more than a half-assed job, so that's what we
> did', but, seriously, consider what the low-hanging fruit would have
> been, just an afternoon's work on the failover problem:
>
> 1. Separate person sets up a *ix machine on static IP in his/her
> (separate) premises. Today, this could be on a junk P4 or PIII.
>
> 2. That person coordinates with the first guy to do daily rsyncs
> of all important data to the failover box.
>
> 3. In the ideal case, the failover box would also be fully configured
> as a hot spare, but this is a fine point. It would probably suffice
> for a few qualified experts to agree that all essential data are being
> captured daily. If production host fails or the guy in charge goes
> crazy, parties able to control DNS flip it, and if necessary there's a
> mad scramble to make the failover host fully functional. The main thing
> that cannot be done without is the data.
>
> 4. Some ongoing monitoring is required, to make sure failover
> replication doesn't silently break.
>
> For extra credit, once every six months, _deliberately_ flip the
> failover switch. Nothing is quite as good at proving ability to
> failover as doing failover.
People look at the rickety old PIII in my garage and think 'Eek, that's
fragile.' Sure it is. But highly replaceable, as witness the fact that
linuxmafia.com has gone through about eight motherboards in a variety of
throwaway, worthless server boxes, and as many sets of hard drives.
It certainly can die. And then I just deploy another (and this time
better) one.
If I wanted to, if I weren't too lazy to do the work, it wouldn't be
difficult to do hot-sparing. Constructing the same server twice is
really no more difficult than doing it once, and then at your leisure
you can set up periodic rsync backup scripts to update the spare box's
data, to slave the spare's database process off the master's so it's
always kept updated, and so on. Resources required: Two throwaway
server machines, two static IP addresses, a little thinking and testing,
part of a weekend. For extra credit, have a hot-spare offsite
somewhere.
For additional extra credit, enroll all instructions for constructing a
spare machine from bare metal (or minimal OS install) into your choice
of configuration management software (Chef, Puppet, Ansible, etc.) to
make spinning one up even faster and easier.
If this is lunacy, it's the lunacy that gets things done. And the point
is: It's not friggin' brain surgery, including the high availability /
failover pieces that give you for free all the _important_ bits of what
the big boys charge money for. The trick is to _know what you're doing_
and verify thta the important parts are covered.
And knowing what you're doing isn't that hard. I figured it out just by
tinkering and paying attention back when I was a _staff accountant_.
No, you won't achieve five nines of reliability, but you won't need to;
you're not running NAQDAQ. You won't be completely immune to DDoSing,
but you won't need to; you're not Cloudfront.net.
What you _can_ do without too much difficulty as an aspiring technical
Linux user -- because this happens to be what Linux is really good at --
is basic, not too fancy, generic Internet services that Linux has done
extremely well for around a quarter century: e-mail, mailing lists,
regular ol' Web sites, ssh, a medley of other things. Making the
contents be easily portable indefinitely comes along for free, which is
definitely not true of more-complex online content and fancier hosted
services. And doing sysadminly assurance of failover, backup/restore,
etc. is not a lot harder.
Of the two guys who called anyone who makes the effort and walks the walk of
open source and autonomous Internet presence a 'lunatic' rely,
one is a San Franciscan who relies on a partimus.org Internet virthost
that he appears to not do anything to keep operating (that host being
not even an autonomous Internet server, but rather just a virtual-hosted
domain on Dreamhost, the specialty WordPress hosting provider terrible
at e-mail/mailing lists that BALUG's admins, mostly Michael Paoli and
partly me, ran a special project just to get BALUG off their hosting).
The other is a Berkeley guy with an ever-changing name who sharecrops
for Google, i.e., using nothing but free-of-charge Google services.
Neither of them has to my knowledge ever even _aspired_ to learn how to
set up and run Internet infrastructure for the Linux Internet community,
yet both a sure that someone who does is a 'lunatic', and that what many
such people have done pretty easily and for peanuts in monetary outlay
is hopelessly impractical.
I see. Right. Puts it in context.
--
Cheers, "A recursive .sig
Rick Moen Can impart wisdom and truth.
rick(a)linuxmafia.com Call proc signature()"
McQ! (4x80) -- WalkingTheWalk on Slashdot
----- End forwarded message -----
Michael, let me recap points I made in our offlist discussion involving
trying to help the guy who insists unsubscribing 'doesn't work'. In the
case of such a user:
1. It's useless merely asking the subscriber 'What steps you're
following', because the subscriber doesn't remember.
Comment: If you're serious, stress the need to _reproduce_ the problem
and take accurate notes.
2. It's useless merely asking the subscriber 'The error diagnostic(s)
(if any) you're getting', because the subscriber doesn't remember.
Comment: As above.
3. It's useless merely asking the subscriber to provide 'FULL EMAIL
HEADERS of an example message', because the subscriber has no idea how.
(Shouting doesn't clarify.)
Comment: The user may not even know what a 'header' is, at all.
If he/she does, the current ones look pretty 'full'; what does
'full' mean? They've probably never seen full headers, and wouldn't
know them from an anaconda. If you're serious, best solution would
be to provide two image files, showing the same messaage with truncated vs.
full headers, and say 'See how the full version has many your mailer
normally hides for simplicity? Notice the ones starting with
Received? Those and more are needed for diagnosis. Dig in your
mailer for a means to reveal them that might be called "Show
original" or "Show source", but might be called something else.'
4. It's pointless to admonish the subscriber to never furnish his/her
subscription password to the listadmins, as the listadmins have full
power to do mischief if they're Bad People, without such passwords.
And telling 'I can't unsubscribe' people that is just a pointless
distraction from their real problem.
Comment: Seriously, users haplessly mailing their subscriptions
around the Internet is very close to harmless to them. Them mailing
those to listdmins is _totally_ harmless. Neither risk is worth
dwelling on, especially in an instructional mail the user is already
going to find both challenging and a little offputting.
On points 1-3, I'm very, very much reminded of hard lessons learned from
CABAL's long history running public installfests. When we started doing
that in 1998, we brightly and optimistically hoped and expected
attendees would download and fill out a simple hardware questionnaire,
e.g., how much RAM, what CPU, how big hard drives.
As other 1990s LUGs around the world also found out, this was a total,
whopping flop. I think it flopped so hard that we deleted the
questionnaire out of sheer disgust, though some other early CABAL materials
survive here: http://linuxmafia.com/pub/linux/cabal/ Note in
particular the CABAL FAQ that Don Marti and I co-wrote: Some of that
excess optimism about attendees being able to pony up information about
their computers survives in parts of the FAQ.
The hard lesson was: They don't know. Which means, if you're serious
about getting that information accurately out of them, you need to
either teach them or furnish an automated software tool to gather it.
The best answer we got (to the questionnaire) was no answer. But some
attendees figured they needed to write down something to make the
organisers happy, so they took wild guesses about, say, how much RAM
their machines had, and tended to be hilariously wrong. (Needless to
say, guesses were worse than lack of answers.)
Why don't they know? Because they never bothered to figure out how
to get those answers, and didn't know where to start. Even though,
say, the POST RAM count whizzed by their eyes every time they booted
(well, until OEMs started hiding the POST process behind pretty
pictures), they lacked the curiosity to poke at the machine and
investigate to answer obvious questions.
But, the subtler problem: Sometimes they _once_ knew, but forgot what
they used to know, and figured a retrospective attempt to guess from dim
and fading memory was good enough. _You and I_ know that getting
reliable information from a computer means producing it
contemporaneously and capturing / writing it down _then_. But amateurs
make the error of trying to just remember this stuff -- _all the time_.
I had been co-author of 'How to Ask Questions the Smart Way' for untold
years when the insight of people totally failing to capture
contemporaneous diagnostic data hit me like a ton of bricks. I had kept
on, for years, trying to convey the right message to readers through
cute, clever word tricks like this 'Missouri' one in the essay:
Describe the problem's symptoms, not your guesses
[...]
Since the preceding point seems to be a tough one for many people to
grasp, here's a phrase to remind you: "All diagnosticians are from
Missouri." That US state's official motto is "Show me" (earned in 1899,
when Congressman Willard D. Vandiver said "I come from a country that
raises corn and cotton and cockleburs and Democrats, and frothy
eloquence neither convinces nor satisfies me. I'm from Missouri.
You've got to show me.") In diagnosticians' case, it's not a matter of
skepticism, but rather a literal, functional need to see whatever is as
close as possible to the same raw evidence that you see, rather than
your surmises and summaries. Show us.
Describe your problem's symptoms in chronological order
[...]
Users, even people familiar with the essay, kept posting their guesses
and not raw diagnostic data. Why? I figured it out: Because the raw
data scrolled off the screen and into fading memory, and the user wasn't
smart enough to go re-acquire it.
If I'd been in the room with that user, I'd have said 'What? Wait, you
_didn't_ go back and reproduce the symptom a second time so you can take
accurate notes? What sort of crazy approach is that? Why the hell not?'
But they don't know -- and, being remote from them across the Internet,
we can't see them screwing it up, and are unaware of them flubbing the
ground-level basics.
My real epiphany involved the least competent poster to the OCLUG
(Orange County LUG) mailing list, a guy who's always posting vague
requests for help and supplies about 70% of the mailing list's traffic.
At one point, he provided a supposed diagnostic message that was
_obviously_ grossly inaccurate for reasons including misspellings, and,
suddenly realising the key problem, I asked 'You typed that from memory,
didn't you?' He was a little cagey, but the answer was yes. I
patiently pointed out that helpers would be attempting to Web-search the
diagnostics, so inaccurate transcription was fatal. 'Oh.' And _why_
did he type it from memory? Because it just never occurred to him to
reproduce the problem (taking accurate notes, this time) _before_ asking
for help.
Remember: They don't 'get' these rock-bottom basics. Just not. If
they did, they wouldn't be in endless trouble. And frankly, anyone who
cannot figure out how to unsubscribe given help in every posting footer
and bountiful help on the Web is self-selected in advance for exactly
that kind of haplessness.
[Bcc: balug-speaker-coordinators(a)balug.org]
Well, taking this on-list because, ... well, why not, ...
> From: "Grant Bowman" <grantbow(a)partimus.org>
> Subject: handling the spam to balug-speaker-coordinators
> Date: Mon, 23 Apr 2018 13:54:32 -0700
> What's the best way to handle the spam to balug-speaker-coordinators?
Well, let's see,
o one could volunteer, get an account on the relevant host, and at least
make suggestions regarding configuration changes to reduce spam, ... but
note too, that for most @balug.org addresses (notably aliases), one would
want a quite low rate of false positives. But regardless I'm sure the
anti-spam filtering could be well improved even with that constraint.
Oooh, you already have an account - that makes much of that that much
easier:
$ hostname; fgrep -i bowman /etc/passwd
balug-sf-lug-v2.balug.org
grantbow:x:14589:14589:Grant Bowman,,,:/home/grantbow:/bin/sh
$
o one could ask/recruit volunteers to assist ... but may be faster and more
efficient (and frankly more likely to get done and/or sooner) if one is
able to do more/much of it oneself. E.g. much more likely something gets
done if one is able to do it oneself, or short of that, hand one with the
access a well tested configuration to fix the issue along with evidence
it's been well tested and addresses the issue - much more likely something
like that gets done, than something more like
"I have an issue, figure it out and fix it for me." ;-)
And yes, there may well even be folks that would be interested in learning
some of the relevant bits - so you might, e.g. ask on list - maybe could
even be bit of team/cooperative effort.
o If I'm not mistaken, most all of the MTA and anti-spam configuration on the
host is world-readable - most notable exception being private key for
TLS/SSL cert for STARTTLS MTA processing. If there's something one needs
to read there but doesn't have access, let me know, and likely that can
be addressed as relevant/appropriate.
Wow, lookie, even have a fair bit of sudo access already (probably more
than needed for looking at MTA/anti-spam configuration bits, but
regardless):
$ sudo sudo -l -U grantbow | sed -ne '/may run/,$p'
User grantbow may run the following commands on balug-sf-lug-v2:
(root) /bin/su - balugwm2
(root) /bin/su - sflug
(root) /bin/su - sflugwww
(sflug) ALL
(root) sudoedit /etc/bind/master/sf-lug.org
(root) sudoedit /etc/bind/master/sf-lug.com
(root) /bin/systemctl reload bind9
$
Also, I've made mention on some list(s) ... don't recall which one(s)
off-the-top-of-my-head (perhaps this one) ... a few items of note. May
possibly consider changing MTA on the host - but it would need to well work
with mailman (which covers the @lists.balug.org addresses). Most of the
anti-spam uses Eximconfig - but though that was a good base start for
anti-spam, it's apparently no longer maintained, so also quite possible
the "best" anti-spam solution(s) for the needs (including fit for the
OS and mailman, and also resources the VM does/doesn't have), may possibly
involve a change in MTA. But given all the interdependencies (mailman,
live production, lists, aliases, etc.), change of MTA would need be well
tested somehow ... and any anti-spam configuration changes should also be
sufficiently well tested - at least to the extent feasible.
Also, a bunch of the current anti-spam is relatively Python based ... but
that may mostly have to do with the current MTA and Eximconfig more than
anything else.
Anyway, certainly feel free to have a look around, etc.
If you want me to implement some specific (reasonably well tested/vetted)
configuration changes, just let me know the specifics (also probably not bad
idea to put it on this list ... more eyeballs may better spot things we
could do better, and public searchable is more opportunities for others to
find and learn from same).
Also, one can expect that the @balug.org email addresses ... notably
especially publicly listed (and mandatory, etc.), such as
balug-speaker-coordinators, postmaster, balug-webmaster, balug-contact,
etc. do get spam, and given the need for low false-positive rate on filtering,
there's likely to always be some moderate flow of spam that make it through.
I'm sure I've made at least some mention of this before.
List admins such as myself likewise deal with a lot of that for the
@lists.balug.org ... but there it's mostly caught and held for processing
as a non-list member post attempt, reviewed by listadmin(s), and discarded
(or simply left to expire based on the generally obvious spam nature
and the ease of doing nothing to have it disposed of).
BALUG & IPv6,
Did finally, yesterday, get around to updating, so BALUG is now
using IPv6 address that also has "reverse" (PTR) data that indicates
balug.org - most notably useful for SMTP and reducing probability of
rejection as "spam". That was the principle motivation for the IPv6
address change.
$ dig +short balug.org. AAAA
2001:470:1f05:19e::2
$ dig +short -x 2001:470:1f05:19e::2
balug.org.
$
That IP address is also the default source IPv6 address
for IPv6 traffic to The Internet from the Virtual Machine (VM).
Many/most publicly accessible services are still (also) listening
on IPv6 wildcard address, but may also change that in future.
DNS NS data hasn't been changed yet (future). But that does
remind me ... DNS IPv6 notify source address - yes, that,
by default, had also changed ... so today I switch that
back at least 'till DNS slaves have been suitably notified,
etc.
SF-LUG - at least that of it on same VM, has its own separate IPv6
address ... however since they're on one single host, for default
outbound IPv6 address, they share the same common IPv6 - which is
primarily for BALUG (it's a BALUG VM also hosting some SF-LUG services,
not vice versa).
> From: "Michael Paoli" <Michael.Paoli(a)cal.berkeley.edu>
> To: SF-LUG <sf-lug(a)linuxmafia.com>
> Subject: SF-LUG & IPv6: Re: IPv6 ... all those addresses! ...
> Mmmmmm... yummy! (+ free t-shirt!)
> Date: Sat, 16 Dec 2017 12:40:45 -0800
> So ... most things SF-LUG, notwithstanding list (Thanks Rick!)
> and DNS nameserver related record bits (future),
> and possibly some DNS TTL bits (up to 4 hours for older cached
> data to expire),
> have mostly had SF-LUG's IPv6 moved to address "just" for
> SF-LUG stuff:
> $ dig +short www.sf-lug.org. AAAA
> 2001:470:1f05:19e::3
> $ dig +short -x 2001:470:1f05:19e::3
> sf-lug.org.
> $
> Well ... those non-list SF-LUG things, are hosted on
> virtual machine which is primarily BALUG, so IPv6
> connections originated from that virtual machine will
> still have a different IPv6 source address (notably that
> of BALUG's), but mostly, other than that, SF-LUG has it's
> "own" IPv6 address. The sf-lug.org domain is canonical,
> and for web, it's www.sf-lug.org, but the IPv6 changes
> likewise apply to the non-canonical domain sf-lug.com.
>
>> From: "Michael Paoli" <Michael.Paoli(a)cal.berkeley.edu>
>> To: SF-LUG <sf-lug(a)linuxmafia.com>
>> Subject: IPv6 ... all those addresses! ... Mmmmmm... yummy! (+
>> free t-shirt!)
>> Date: Wed, 25 Oct 2017 07:19:59 -0700
>
>> $ (for ipv6 in 2001:470:1f04:19e::2 2001:470:1f05:19e::2
>> 2001:470:1f05:19e::3; do echo "$ipv6" $(ssh -6 -aqx "$ipv6"
>> 'hostname') $(dig -x "$ipv6" +short); done)
>> 2001:470:1f04:19e::2 balug-sf-lug-v2.balug.org
>> BALUG-1-pt.tunnel.tserv3.fmt2.ipv6.he.net.
>> 2001:470:1f05:19e::2 balug-sf-lug-v2.balug.orgbalug.org.
>> 2001:470:1f05:19e::3 balug-sf-lug-v2.balug.orgsf-lug.org.
>> $
>>
>> So, yes, IPv6 ... *lots* of addresses! :-) And that's generally a *good*
>> thing. We have been and are seeing with IPv4, where too few ends up. 8-O
>>
>> Anyway, a couple items ...
>>
>> Virtual Machine, SF-LUG, and BALUG. Should have gotten around to it
>> much earlier, but ...
>> So ... excepting SF-LUG list stuff (Thanks Rick!), DNS slaves (Thanks
>> Rick & others) and some other trace bits, the SF-LUG and BALUG stuff
>> essentially all runs on one virtual machine. Now, it would generally be
>> good, if/as feasible, to have those reasonably well separated out ...
>> most notably so they can/could be "peeled apart" easily if or whenever
>> that might be desired or necessary. If I had an (over) abundance of
>> resources (system resources, sysadmin resources, IPs, bandwidth, etc.),
>> I'd probably have them (much) more fully separated out. Though there
>> are also some efficiencies gained by *not* having them totally separated
>> out (though many of those efficiencies could also be gained, even if
>> separated out, through various configuration management and similar
>> tools/infrastructure). Anyway, to the extent reasonable feasible within
>> the one VM, SF-LUG and BALUG are *reasonably* separate. E.g., though
>> running on a common webserver, their distinct configurations are well
>> separated out, and for most intents and purposes - most notably to
>> "outside" (seeing the Webserver from The Internet), they mostly look and
>> act pretty separate and independent. E.g. they even have their own
>> separate SSL/TLS CA certs - even though they share the same IPv4
>> address. And, ... IPv6? :-) Yeah, that's the part I should've gotten
>> around to (much) earlier, but anyway ... no shortage of addresses, so
>> ... just added some, so SF-LUG and BALUG can effectively have their own.
>> And ... why might that matter? Email is a fairly good example. MTA
>> should have "reverse" DNS entry. If one IP is used, there's one entry
>> (or set of entries) for the IP ... which is less than optimal. But with
>> separate IPs, each can have their own "reverse" DNS entry. E.g.:
>> $ dig +short -x 2001:470:1f05:19e::2
>> balug.org.
>> $ dig +short -x 2001:470:1f05:19e::3
>> sf-lug.org.
>> With both of those IPs - in fact all 3 IPv6 IPs shown in the code
>> snippet at the top of this post - on the same (virtual) host.
>> For MTA, the "reverse" DNS should generally match the sending domain, or
>> a parent domain (but not above registrant level). Such can also look
>> more "clean" and proper for, e.g. Web site addresses.
>>
>> And ... Linux, and adding additional IPv6 IPs to a host? That's
>> relatively easy. One could go quite hog wild 8-O with that ... but I
>> don't want to be to excessive/messy with the VM, etc. So ... at present
>> there are 3 Internet routable IPv6 addresses, 2 of which I very recently
>> added. One of them is local host endpoint of IPv6 tunnel (alas, my ISP
>> doesn't yet offer native IPv6 on the particular plan I'm on ... but they
>> have at least started deploying native IPv6). The "reverse" DNS on that
>> one ... uhm, yeah, I can't set/change that - it's set by the tunnel
>> provider. However, from same tunnel provider, also get another routable
>> /64 block where I get to fully control the "reverse" DNS ... and have
>> done so. Hence two more addresses - one for SF-LUG, one for BALUG.
>>
>> Still lots more to do on that - like more DNS data updates,
>> setting/changing (or not) which IP address is default source IP address,
>> due attention to any transitions and possible race conditions, etc. All
>> of which should be done carefully to avoid potential problems. Might
>> even pick different IP addresses, for various reasons. In any case,
>> very nice to have the flexibility and availability ... unlike IPv4 where
>> it's more like ... ya'll just get one and you're gonna have to learn to
>> share ... deal with it.