[BALUG-Admin] High(er) availability for BALUG ancillary sites
Michael Paoli
Michael.Paoli@cal.berkeley.edu
Thu Nov 26 11:57:11 PST 2015
So, ... now also high(er) availability for the BALUG ancillary sites.
Also upgraded "vicki" to Debian GNU/Linux 8.2 (jessie) some days or so
earlier ... but note however that "vicki" is the physical host that
now only the *sometimes* has the balug VM running atop it (much/most
of the time that balug VM is running atop my personal laptop).
Nevertheless (and because of that - and live migration!) ... high(er)
availability!:
----- Forwarded message from Michael.Paoli@cal.berkeley.edu -----
Date: Thu, 26 Nov 2015 11:30:47 -0800
From: "Michael Paoli" <Michael.Paoli@cal.berkeley.edu>
Subject: High(er) availability for SF-LUG site(s) (& some BALUG stuff too).
To: SF-LUG <sf-lug@linuxmafia.com>
High(er) availability for SF-LUG site(s) (& some BALUG stuff too).
This covers the web sites & DNS master (list is hosted separately by
Rick Moen), and this also covers fair bit of BALUG stuff too (everything
*except* BALUG's [www.]balug.org and list stuff - e.g. wiki, archives,
test/beta/staging sites, etc.)
Anyway, more-or-less per earlier plan, I did get so far yesterday, as
doing the first *live* migration of that host. And also *without*
shared storage (which does also work perfectly fine - just takes a wee
bit longer for the actual overall migration, but still when the actual
final switch itself happens for guest VM itself, it's exceedingly fast
(on the order of 10s of milliseconds or so? - I haven't actually timed
that final bit quite yet). So, ... that VM host is no longer "stuck"
just on my personal laptop :-) ... which means it can very much remain
up and Internet accessible - even when my laptop isn't (or, e.g. travels
away from home).
$ wakeonlan 00:30:48:91:97:90
...
Not 100% anticipated, but not a huge surprise, and easy enough to
address - some of the last kinks to be worked out were in allowing the
live migration to be successful. CPU type and flags/capability:
error: unsupported configuration: guest and host CPU are not
compatible: Host CPU does not provide required features: popcnt,
sse4.2, sse4.1
Ah, so laptop CPU wee bit more modern than that on "vicki" - and by
more-or-less default, guest CPU was configured to take advantage of
many/most of those host CPU capabilities. Easy enough to deal with that
- bring the VM guest down, reconfigure the virtual CPU to disable those
3 capabilities, bring VM guest back up again, and repeat the attempt -
made it fine past that error. Next glitch was a bit more puzzling:
error: internal error: unable to execute QEMU command 'migrate': State
blocked by non-migratable device '0000:00:05.0/ich9_ahci
Wee bit 'o search and ... QEMU can't live migrate SATA (at least not
yet safely in version I'm using, and at least by default migration of
such is disabled for safety reasons). Bring host down, take virtual
hard drive off of SATA, turn it into SCSI and attach it to SCSI ... and
... same error? Checked configuration again - nothing attached to
(virtual) SATA bus/controller, but the SATA bus/controller still there,
... next step, remove those, and repeat ... and ... success, all went
fine, no errors:
# virsh migrate --live --persistent --copy-storage-all --verbose balug \
> qemu+ssh://192.168.55.2/system && virsh autostart --disable balug
... and all went fine and dandy. And then, live migrating back:
# virsh migrate --live --undefinesource --copy-storage-all --verbose \
> balug qemu+ssh://tigger.mpaoli.net./system
And that went perfectly fine too, not so much as a glitch to notice on
the guest VM itself (though the storage replication took a while, so
it's not a speedy move from perspective of the physical hosts) ... TCP
connections between guest and Internet, etc., all maintained perfectly
fine across the live migrations.
Wee bit more stuff to do / work on ... e.g. (at least theoretically),
o Turn it into a (nearly) push-button operation (run one relatively
simple script - or pair of scripts - partially drafted, but yet to
polish those off.).
o Investigate/test --copy-storage-inc - if suitable and safe, that may
significantly speed the disk data copy portion of the migration (some
of the storage I have set up is highly optimized for physical storage
space reduction, but consequently has very low write performance
characteristics - which is mostly quite fine, but slows migration
especially back to laptop; read performance, however, is more than
sufficient. E.g. on physical host (laptop) we have:
# ls -hnos balug-sda
4.8G -rw------- 1 114 16G Nov 26 18:41 balug-sda
#
Quite efficient (deduplication + compression) space utilization - but
at cost of write performance (and some CPU burn, particularly on
heavy writing, and some more suck of RAM too) - but that happens to
be the trade-off I want the majority of the time for that storage -
so that's highly acceptable (laptop SSD is "only" about 150 GiB ...
and I've a whole lot of other stuff on it too - I'm fine with LUG VM
taking ~5 GiB of physical storage ... but not gonna give it 16 GiB!).
Where it resides, it also does deduplication across some ISOs that
quite correspond to the installed operating system (and also other
data), so that also aids in reducing total physical storage space
consumed.
).
o I'll also carefully review, and likely adjust/tweak other bits of the
migration options and handling of the VMs after migration - mostly
notably bits regarding undefine or not, and autostart or not - and
where. And of course test it all out more fully. :-)
> From: "Michael Paoli" <Michael.Paoli@cal.berkeley.edu>
> Subject: Re: sf-lug site & hardware
> Date: Tue, 24 Nov 2015 06:32:02 -0800
> Just an FYI update.
>
> So, my (overly optimistic) theoretical timeline - was hoping to have
> the sf-lug site relocated onto the higher availability hardware
> (notably not on VM on my laptop) by around 2015-11-15 or so. Have
> adjusted the target timeline a bit, after some considerations (and also
> being relatively busy with other stuff too). Anyway, one thing I
> didn't fully take into account earlier - fan noise. That system that
> was in the colo - 1U unit, is comparatively noisy (I've gotten a bit
> spoiled mostly not listening to fan noise of such volume - even though
> it uses a fan and airflow design that mostly avoids tiny 1U high-RPM
> fan(s) - it's still noisier than most typical desktop systems - but
> less noisy than many typical 1U servers). So, ... I adjusted my
> (theoretical) plans a bit. With wakeonlan, qemu-kvm live migration,
> and wee bit of infrastructure (which I was mostly planning to do
> anyway), and small bit of scripting, I could arrange to have the VM
> running on the noisier (but higher availability) hardware, mostly only
> when it wouldn't be running on my laptop at home. And with live
> migration, the migration would be effectively "invisible" to the guest
> VM host itself, its state, connections to it and sessions on it, etc.
> Anyway, fair bit closer to having that plan fully implemented. Current
> target timeline for completion, by 2015-11-29, or at least not later
> than 2015-12-13. May be fair bit sooner. I'll update once it's in
> place and fully operational (did get a fair chunk of related
> infrastructure completed yesterday and today).
>
> references/excerpts:
> https://en.wikipedia.org/wiki/Wake-on-LAN
> https://en.wikipedia.org/wiki/Live_migration
>
>> From: "Michael Paoli" <Michael.Paoli@cal.berkeley.edu>
>> Subject: Re: sf-lug site & hardware
>> Date: Thu, 12 Nov 2015 14:01:41 -0800
>
>> FYI, this morning Jim Stockford and I did retrieve the physical server
>> host from the colo, upon which, up until some months back, the sf-lug
>> web site was running. So, that improves the hardware resource
>> situation. I'm guestimating I'll have the sf-lug website again running
>> on VM atop this hardware by sometime this weekend or so - that should
>> improve the availability a fair bit (notably the sf-lug website then
>> won't go down when my personal laptop goes down, offline, or out the
>> door from home).
>>
>> Thanks Jim!
>>
>>
>>> From: "Michael Paoli" <Michael.Paoli@cal.berkeley.edu>
>>> Subject: Re: Have you guys thought about http://www.freelists.org/
>>> (hosted ...)
>>> Date: Wed, 11 Nov 2015 18:26:25 -0800
>>
>>> would be down or that it wasn't (relatively) high availability (at least
>>> compared to virtual machine running on my personal laptop - which does
>>> have the sf-lug site go out when my laptop goes out ... hopefully that
>>> situation will be improved in near future ... waiting on some resources
>>> to be able to do that.)
>>>
>>> references/excerpts:
>>> http://linuxmafia.com/pipermail/sf-lug/2015q4/011454.html
>>> http://linuxmafia.com/pipermail/sf-lug/2015q4/011441.html
>>>
>>>> From: Shane Tzen <shane@faultymonk.org>
>>>> Date: Wed, 11 Nov 2015 15:56:14 -0800
>>>> Subject: Re: [sf-lug] updated/upgraded: SF-LUG - operating system
>>>> presently hosting
>>>> To: Michael Paoli <Michael.Paoli@cal.berkeley.edu>
>>>> Cc: SF-LUG <sf-lug@linuxmafia.com>
>>>>
>>>> Have you guys thought about http://www.freelists.org/about.html ?
>>>>
>>>> Looks like various LUGs are hosted -
>>>> http://www.freelists.org/cat/Linux_and_UNIX
>>>>
>>>>
>>>> On Fri, Oct 30, 2015 at 3:41 AM, Michael Paoli <
>>>> Michael.Paoli@cal.berkeley.edu> wrote:
>>>>
>>>>> It's been updated/upgraded:
>>>>> from: Debian GNU/Linux 7.9 (wheezy)
>>>>> to: Debian GNU/Linux 8.2 (jessie)
>>>>>
>>>>> http://lists.balug.org/pipermail/balug-admin-balug.org/2015-October/002989.html
>>>>>
>>>>> Still definitely *not* high availability though (alas, still sits atop
>>>>> a virtual machine on my *laptop*!).
>>>>>
>>>>> Hopefully in not too horribly distant future (like *real soon*), the
>>>>> physical box the site was earlier running upon will be successfully
>>>>> retrieved - once that happens, some high(er) availability options
>>>>> become possible.
>>>>>
>>>>> Let me know if you notice anything awry (notwithstanding the less than
>>>>> high availability).
>>>>>
>>>>> From: "Michael Paoli" <Michael.Paoli@cal.berkeley.edu>
>>>>>> Subject: It's alive*!: Re: SF-LUG - DNS, web site, ..., etc.
>>>>>> Date: Mon, 24 Aug 2015 03:10:26 -0700
>>>>>>
>>>>>
>>>>> Anyway, have taken the liberty ...
>>>>>> it's alive* ...
>>>>>> the [www.]sf-lug.{org,com}
>>>>>> websites are available again.
----- End forwarded message -----
> From: "Michael Paoli" <Michael.Paoli@cal.berkeley.edu>
> Subject: BALUG ancillary sites UPgraded
> Date: Fri, 30 Oct 2015 03:33:52 -0700
> Upgraded (updated) from: Debian GNU/Linux 7.9 (wheezy)
> to: Debian GNU/Linux 8.2 (jessie)
>
> If you notice anything awry, let me know.
> I did notice the wiki pages do have a rather different look & feel.
>
>> From: "Michael Paoli" <Michael.Paoli@cal.berkeley.edu>
>> Subject: BALUG ancillary sites UP: Re: Of relevance for BALUG
>> ancillary sites
>> Date: Sat, 22 Aug 2015 03:00:16 -0700
>
>> These BALUG sites are UP again.
>> Due note, however, that the new location is *not* high availability,
>> nor high bandwidth.
>>
>> Specifically, at present, the VM hosting those sites has been move onto
>> a laptop (8-O !) ... which does occasionally get out and about (and when
>> it does, those BALUG sites will be temporarily down again), and it's
>> sitting at the end of a DSL line - which also shares bandwidth with
>> other hosts.
>>
>> I'll likely be doing some things in the coming weeks or more to improve
>> the availability (notably move off the present laptop where the VM is
>> currently), and may at some future point also have improved bandwidth -
>> but for now, is what it is.
>>
>> Let me know if anyone notices anything particularly amiss (notwihstanding
>> the inherent limitations and availability limitations noted above).
>>
>>> From: "Michael Paoli" <Michael.Paoli@cal.berkeley.edu>
>>> Subject: BALUG ancillary sites DOWN: Re: Of relevance for BALUG
>>> ancillary sites
>>> Date: Thu, 20 Aug 2015 09:05:43 -0700
>>
>>> The BALUG ancillary sites were still up yesterday morning, but they're
>>> DOWN now. I'm presuming power/network has been removed from the "vicki"
>>> physical host at the colo site. I expect there will be more about this
>>> on the SF-LUG list *real soon now*.
>>> http://linuxmafia.com/mailman/listinfo/sf-lug
>>>
>>> I do have recent backups - having reasonably anticipated this scenario,
>>> I have backups from this as recent as yesterday morning.
>>>
>>>> From: "Michael Paoli" <Michael.Paoli@cal.berkeley.edu>
>>>> Subject: Of relevance for BALUG ancillary sites
>>>> Date: Sat, 25 Jul 2015 01:30:48 -0700
>>>
>>>> Just a bit of background. Some BALUG sites may be impacted soon,
>>>> but *not*
>>>> www.balug.org or balug.org itself.
>>>> See:
>>>> http://lists.balug.org/pipermail/balug-admin-balug.org/2014-July/002233.html
>>>> for *which* sites will (and won't) be impacted
>>>> As to some background and why, have a look at:
>>>> http://linuxmafia.com/pipermail/sf-lug/2015q3/011330.html
>>>> And, for the BALUG specific bits I left out of the above:
>>>>
>>>>> From: "Michael Paoli" <Michael.Paoli@cal.berkeley.edu>
>>>>> Subject: Re: vicki (exodus) & GoGrid/Datapipe
>>>>> Date: Thu, 23 Jul 2015 03:14:08 -0700
>>>>
>>>>> BALUG: what balug has on the virtual machine atop vicki is certainly
>>>>> highly useful and convenient, and would be relative pain to be without,
>>>>> but we could survive without, at least for a bit, and have in fact
>>>>> relocated it (temporarily) elsewhere in past.
>>>>>
>>>>> What BALUG does *not* have atop the (VM on the) vicki host:
>>>>> www.balug.org, and balug.org itself aren't on that host, but other
>>>>> sites/subdomains are, e.g. [www.]{archive,wiki}.balug.org, among others.
>>>>>
>>>>> DNS - only for some subdomains, again not balug.org. itself.
More information about the BALUG-Admin
mailing list