BALUG-Talk

balug-talk@lists.balug.org

150 discussions

ssh materials :-)
by Michael Paoli 21 May '25

21 May '25

So, one of the many things that we discussed at yesterday's BALUG meeting ... ssh Notably some bits about how to well and properly use it, getting used to it, folks getting stuff wrong ... and right. Anway, thought of it bit later after meeting, but I do have materials on ssh which I've repeatedly used in helping train folks on ssh and security thereof, etc. https://www.mpaoli.net/~michael/linux/ssh/ssh.odp Yes, presented before at BALUG and for other trainings/presentations. (hmmm, I should probably also add ssh certs to it, and some other modest updates). Also, originals from the much earlier BALUG presentation: https://www.archive.balug.org/2010/2010-02-16/

1 0

Re: Possible topic(s) for 2025-05-20 BALUG meeting (storage, ...)
by Darrion Burgess 03 May '25

03 May '25

I think it's a great idea, I've been looking at allot of vm work too and that sounds like fruitful research. Especially excited to learn more about filesystems On Sat, May 3, 2025, 10:03 AM Michael Paoli via BALUG-Talk < balug-talk(a)lists.balug.org> wrote: > > > > ---------- Forwarded message ---------- > From: Michael Paoli <michael.paoli(a)berkeley.edu> > To: BALUG-Talk <balug-talk(a)lists.balug.org> > Cc: > Bcc: > Date: Sat, 3 May 2025 10:01:58 -0700 > Subject: [BALUG-Talk] Possible topic(s) for 2025-05-20 BALUG meeting > (storage, ...) > So, was thinking, possible topic(s) for 2025-05-20 BALUG meeting. > I had in mind, potentially something (like/approximating): > Storage (partitioning, ext{2|3}4}, md, LVM, LUKS, ZFS, migrating, > modernizing, growing, shrinking, etc.) > > Probably most notably since I've been doing some more mucking about > with most all of that recently - looking at some drive upgrades > (notably capacity increases), and been doing fair bit of planning and > testing (mostly on VMs) to test out various scenarios, and also > determine how I want to get from where I/we are, to where I/we want to > be (some of the storage layout hasn't been significantly redesigned in > well over a decade). > > Could also talk about Ventoy (also been doing some testing of that > recently - pretty nifty, but alas, doesn't handle all bootable > images), or of course other topic(s) folks might be interested in. > > So, thoughts/suggestions on topic(s) for 2025-05-20 meeting? > > > > ---------- Forwarded message ---------- > From: Michael Paoli via BALUG-Talk <balug-talk(a)lists.balug.org> > To: BALUG-Talk <balug-talk(a)lists.balug.org> > Cc: > Bcc: > Date: Sat, 3 May 2025 10:01:58 -0700 > Subject: [BALUG-Talk] Possible topic(s) for 2025-05-20 BALUG meeting > (storage, ...) > _______________________________________________ > list: BALUG-Talk(a)lists.balug.org help: https://lists.balug.org/help/ > unsubscribe email: balug-talk-unsubscribe(a)lists.balug.org

1 0

Possible topic(s) for 2025-05-20 BALUG meeting (storage, ...)
by Michael Paoli 03 May '25

03 May '25

So, was thinking, possible topic(s) for 2025-05-20 BALUG meeting. I had in mind, potentially something (like/approximating): Storage (partitioning, ext{2|3}4}, md, LVM, LUKS, ZFS, migrating, modernizing, growing, shrinking, etc.) Probably most notably since I've been doing some more mucking about with most all of that recently - looking at some drive upgrades (notably capacity increases), and been doing fair bit of planning and testing (mostly on VMs) to test out various scenarios, and also determine how I want to get from where I/we are, to where I/we want to be (some of the storage layout hasn't been significantly redesigned in well over a decade). Could also talk about Ventoy (also been doing some testing of that recently - pretty nifty, but alas, doesn't handle all bootable images), or of course other topic(s) folks might be interested in. So, thoughts/suggestions on topic(s) for 2025-05-20 meeting?

1 0

Re: Discussion topic(s) for Tuesday's meeting? :-)
by Darrion Burgess 15 Apr '25

15 Apr '25

Sorry for late reply, This sounds like just what I was hoping to learn about To give some context, I am currently trying to learn how to make reproducible OS builds using mkosi <https://github.com/systemd/mkosi> One of the big advantages of the tool is that is helps get closer to the points outlined in Fitting Everything together <https://0pointer.net/blog/fitting-everything-together.html> by Pottering in which he outlines some key goals including: - increasing usage of the TPM to increase security - self-signing keys so we can have operating systems that are both immutable AND hackable (no more corporate signed keys!) - ensuring the validity of the entire stack by using secure boot, encrypted home drives, and sandboxing for user applications to make a more secure environment All of this is with the goal of running particleos <https://github.com/systemd/particleos> eventually but i would settle for just a custom-rolled arch distro that I upgrade in A/B fashion. But one of the fundamental assumptions of all of this is that im self-signing keys and using them for everything from secure-boot, to my home drive if necessary but I have held off from learning these security topics until now! On Thu, Apr 10, 2025 at 10:54 PM Michael Paoli via BALUG-Talk < balug-talk(a)lists.balug.org> wrote: > > > > ---------- Forwarded message ---------- > From: Michael Paoli <michael.paoli(a)berkeley.edu> > To: BALUG-Talk <balug-talk(a)lists.balug.org> > Cc: Darrion Burgess <dargondab9(a)gmail.com> > Bcc: > Date: Thu, 10 Apr 2025 22:52:53 -0700 > Subject: [BALUG-Talk] Discussion topic(s) for Tuesday's meeting? :-) > Thanks all, yes meeting last month was excellent, thanks for making it > so! > > So, I was thinking, topic(s) (not that we need be limited to such) for > meeting this month (soon - Tuesday!) ... and following a bit from last > month's meeting, perhaps something along the lines: > Linux encryption and security hardware. > So, perhaps around LUKS, TPM chip, YubiKey, FIDO2, etc. > More-or-less extension of fair bit that was discussed at last meeting. > And, not too horribly redundant with presentations/talks/topics that > have been done at BALUG before (e.g. done LUKS presentation before, > have at least had security as topic, but don't think we've (much) > covered TPM chip, YubiKey, FIDO2, etc. before, so was thinking to mix it > up and expand it a bit more. > Also, did we have some other question(s)/topic(s) from last meeting that > we didn't quite get around to covering? > > Feel free to let me know your thoughts, and I'll put together at > least something regarding (at least leading, but by no means limited to) > topics for the meeting, and will get that then updated on the web site, > and also use that for items sent to the Announce list before the > meeting. Anyway, I'm hoping/aiming to get that done and at least > initial bits out in the next couple days or so > > > > ---------- Forwarded message ---------- > From: Michael Paoli via BALUG-Talk <balug-talk(a)lists.balug.org> > To: BALUG-Talk <balug-talk(a)lists.balug.org> > Cc: Darrion Burgess <dargondab9(a)gmail.com> > Bcc: > Date: Thu, 10 Apr 2025 22:52:53 -0700 > Subject: [BALUG-Talk] Discussion topic(s) for Tuesday's meeting? :-) > _______________________________________________ > list: BALUG-Talk(a)lists.balug.org help: https://lists.balug.org/help/ > unsubscribe email: balug-talk-unsubscribe(a)lists.balug.org

1 0

Getting started with Linux from Microsoft Windows, macOS, etc. (was: Linux w/in MS-Windows... great chat! extended)
by Michael Paoli 11 Apr '25

11 Apr '25

Well, for starting to dip the toe into Linux, from Microsoft Windows or macOS ... On non-ancient Microsoft Windows, for ease of entry, I'd probably first point to Microsoft's WSL - as such is pretty conveniently available, and tightly integrates with Microsoft Windows, so it makes it quite easy to, e.g. move data back and forth, and generally access resources of one from the other. Been a while since I mucked about with WSL, but one thing I found (quite) annoying with it, though it's quite the Linux(-like) operating environment, it's not "really" a Linux operating system - at least not fully so - and not even a VM. So, if one is rather to quite used to Linux, and using and treating it as a system, and probably even more so administration thereof ... well, yeah, there's a lot of stuff one will bump into that just won't work, because it's not really and fully an installed running Linux operating system, nor even such in a VM. But "other than that", yeah, sure, mostly seems to work quite well, and hard to beat for easily available on (non-ancient) Microsoft Windows and good tight integration. Next I might suggest Oracle's VirtualBox. And, it has its advantages and disadvantages. There is of course the whole Oracle is evil thing: https://www.youtube.com/watch?v=-zRN7XLCRhc&t=1980s etc., but some of the other disadvantages, over the years, Oracle has made it increasingly difficult to do more custom installations of Linux. Still quite possible, but in more recent years, VirtualBox wants to be "smart" and "help", by recognizing many Linux ISOs, and for those it recognizes, essentially to large extent taking over and automating/controlling the installation, making it much more difficult to have access to the more typical means by which Linux would generally be installed from ISO, and accessing its menus and options and installation dialogs and such - and instead VirtualBox just gives one its choices, and hides all those other details from one. Anyway, is still (at least last I checked) possible to do the more customary VM install, quite as if one were simply booting the ISO, and installing from what it offers, and with its menus and dialogs, etc., but VirtualBox just makes it increasingly difficult to manage to do such an install. Anyway, those are the among the most frustrating bits I find with VirtualBox ... though last I actually did an install from it was likely year or more ago - and has been about half a year since I've used VirtualBox - but I've often heavily used it, notably in $work environments, on macOS and/or Microsoft Windows. And, it does have the advantages that it is a full VM, unlike, e.g. WSL (or Cygwin, or Homebrew). It also has good integration with Microsoft Windows / macOS (notably filesystem access, etc.), but not as tightly integrated as WSL. What I may quite suggest though (alas, haven't yet done so myself, but if I were about to do a fresh installation to Microsoft Windows or macOS, probably where I'd start), although significantly more complex to set up, would be Homebrew, and with/from that, qemu/kvm, libvirt and friends, etc. I think that gives a significantly better VM platform, though it's fair bit more fiddly bits to get that all set up and nicely working, and also bit more work to do, e.g. filesystem integration bits. But once all in place, probably well worth it. Yes, one of my work peers had gone that direction, and the results were most excellent. So, though I've not yet done that myself, I'd think the results and capabilities would be very much like where I've done quite similar where the host OS is Linux ... but not with Homebrew in there, but including all the other relevant bits, e.g. qemu/kvm, and libvirt and friends. I've not added tight filesystem integration between VMs and the physical host OS with that though, with or without Homebrew ... - I've not yet had/felt compelling need ... but were I doing it on Microsoft Windows or macOS, or otherwise needed to more directly share lots of files between VM and physical host, then I'd probably take the additional steps to set that up. Other worthy of mention bits ... Dual boot - once upon a time, that was generally what folks were pointed at, and most Linux distros make installing that way pretty easy. The big downsides with that, though, it's more complex than doing, e.g. WSL or VirtualBox to have installed and up and running, and probably more importantly, with dual boot - one only runs one OS at at time. And with that, most folks typically end up spending most all their time just running one, and not doing much with the other ... and most of the time that means not doing much with Linux and often effectively giving up on it relatively early on. But there still are some circumstances where I'd recommend dual boot. E.g. if one is transitioning, notably to entirely switch away from whatever the other OS is, or is on that particular physical host, then dual boot is best way to be sure one knows exactly how it will behave on that particular hardware, get any associated kinks worked out, etc. So, e.g., when I fist ran Linux in 1989, yeah, I was doing dual boot ... Linux first on floppies, then dual boot on the hard drive (and may have still required a boot floppy for the way I was set up), and finally completing my migration from UNIX to Linux. And back then, VM was totally infeasible if not for all intents and purposes impossible, for the hardware and its resources I was using at the time. So, sometimes dual boot is the answer, but these days, most of the time it's not the answer / best fit. There's also Homebrew itself - not Linux, but can provide much of a *nix-like environment on, e.g. Microsoft Windows or MacOS. For X11 on MacOS, I found XQuartz to work highly well. For X11 on Microsoft Windows, been years since I did it, but I found Cygwin/X worked quite well. And, somewhat similar to Homebrew, for Microsoft Windows, there's Cygwin, though been years since I ran Cygwin on Microsoft Windows, so not sure how it's comparing capability-wise and such to Homebrew these days (in fact I've not run Homebrew on Microsoft Windows, but have generally heard quite good things from those that have). From other platforms, e.g. the BSDs, what options are (feasibly) available may be bit different. Of course dual boot is a possibility, some VM technologies may well work, and there may be some other possibilities too - of the various bits mentioned, not sure which may be available to the BSDs. Oh, another worthy of mention - VMware - they may have different possibilities available for various platforms, but licensing and/or cost may be an issue, depending upon environment, what they do/don't currently offer (and notably what for free and under what terms), etc., and what they do offer, may be subject to change (with ownership change in recent years, that's been a bit more bumpy ... but hopefully that's more settling/settled out at this point). Anyway, I haven't run such in some fair number of years now - don't know if they even offer software for running on macOS. Anyway, it has been a good option in past, not sure about most current. And, I may be missing some bits, but I think those are at least most that come up in conversations, etc., regarding various relevant possibilities. On Sun, Mar 23, 2025 at 12:26 PM aaronco thirtysix via BALUG-Talk <balug-talk(a)lists.balug.org> wrote: > > > > > ---------- Forwarded message ---------- > From: aaronco36(a)sdf.org > To: balug-talk(a)lists.balug.org > Cc: > Bcc: > Date: Sun, 23 Mar 2025 19:26:06 -0000 > Subject: [BALUG-Talk] Linux w/in MS-Windows... great chat! extended > Quoted snippet of Darrion Burgess from BALUG-Talk thread > https://lists.balug.org/mailman3/hyperkitty/list/balug-talk@lists.balug.org… > : > > > > to start, the easiest way to ease into is using the windows subsystem > > for linux (WSL) so you can try out different distributions using > > the windows terminal. > > > > Its even sophisticated enough to have a wayland compositor at this > > point so you can even run GUI apps from the linux containers. > > Besides WSL, another good(IMHO) means of more easily running Linux distros > w/in MS-Windows is to use Oracle VirtualBox > Quoting its main site https://www.virtualbox.org/ : > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > VirtualBox is a general-purpose full virtualization > software for x86_64 hardware (with version 7.1 > additionally for macOS/Arm), targeted at laptop, > desktop, server and embedded use. > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > The VirtualBox 7.1.6 platform packages dwonloads for Windows and other OS > hosts can be found at https://www.virtualbox.org/wiki/Downloads > > Am concurrently attending both the in-person and Jit.si Meet virtual > meetups of the Berkeley Linux Users Group https://berkeleylug.com/ -- and > am using VirtualBox for Debian GNU/Linux 12.x 'Bookworm' as hosting OS on > a second laptop. > VBox works great and is more [ful]filling! :-D > > -A > -- > > > > > > ---------- Forwarded message ---------- > From: aaronco thirtysix via BALUG-Talk <balug-talk(a)lists.balug.org> > To: balug-talk(a)lists.balug.org > Cc: > Bcc: > Date: Sun, 23 Mar 2025 19:26:06 -0000 > Subject: [BALUG-Talk] Linux w/in MS-Windows... great chat! extended > _______________________________________________ > list: BALUG-Talk(a)lists.balug.org help: https://lists.balug.org/help/ > unsubscribe email: balug-talk-unsubscribe(a)lists.balug.org

1 0

Discussion topic(s) for Tuesday's meeting? :-)
by Michael Paoli 11 Apr '25

11 Apr '25

Thanks all, yes meeting last month was excellent, thanks for making it so! So, I was thinking, topic(s) (not that we need be limited to such) for meeting this month (soon - Tuesday!) ... and following a bit from last month's meeting, perhaps something along the lines: Linux encryption and security hardware. So, perhaps around LUKS, TPM chip, YubiKey, FIDO2, etc. More-or-less extension of fair bit that was discussed at last meeting. And, not too horribly redundant with presentations/talks/topics that have been done at BALUG before (e.g. done LUKS presentation before, have at least had security as topic, but don't think we've (much) covered TPM chip, YubiKey, FIDO2, etc. before, so was thinking to mix it up and expand it a bit more. Also, did we have some other question(s)/topic(s) from last meeting that we didn't quite get around to covering? Feel free to let me know your thoughts, and I'll put together at least something regarding (at least leading, but by no means limited to) topics for the meeting, and will get that then updated on the web site, and also use that for items sent to the Announce list before the meeting. Anyway, I'm hoping/aiming to get that done and at least initial bits out in the next couple days or so

1 0

Linux w/in MS-Windows... great chat! extended
by aaronco36＠sdf.org 23 Mar '25

23 Mar '25

Quoted snippet of Darrion Burgess from BALUG-Talk thread https://lists.balug.org/mailman3/hyperkitty/list/balug-talk@lists.balug.org… : > > to start, the easiest way to ease into is using the windows subsystem > for linux (WSL) so you can try out different distributions using > the windows terminal. > > Its even sophisticated enough to have a wayland compositor at this > point so you can even run GUI apps from the linux containers. Besides WSL, another good(IMHO) means of more easily running Linux distros w/in MS-Windows is to use Oracle VirtualBox Quoting its main site https://www.virtualbox.org/ : ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ VirtualBox is a general-purpose full virtualization software for x86_64 hardware (with version 7.1 additionally for macOS/Arm), targeted at laptop, desktop, server and embedded use. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The VirtualBox 7.1.6 platform packages dwonloads for Windows and other OS hosts can be found at https://www.virtualbox.org/wiki/Downloads Am concurrently attending both the in-person and Jit.si Meet virtual meetups of the Berkeley Linux Users Group https://berkeleylug.com/ -- and am using VirtualBox for Debian GNU/Linux 12.x 'Bookworm' as hosting OS on a second laptop. VBox works great and is more [ful]filling! :-D -A --

1 0

Thanks for the great chat!
by dargondab9＠gmail.com 22 Mar '25

22 Mar '25

hello Dan and Michael, Thanks again for putting on the meeting. It was wonderful to meet you both and I look forward to having more conversations with you all about the topics that interest you.

3 2

device mapper! (dmsetup, etc.) (bits from BALUG meeting 2024-11-19)
by Michael Paoli 26 Nov '24

26 Nov '24

So, 2024-11-19 BALUG meeting, one of our discussion topics: device mapper - and dmsetup(8) and related. So, device mapper - it is used to create/manage block device, which in turn has a specification of how it blocks are mapped to zero or more other device(s). It operates in units of traditional 512 byte blocks, and can handle a quite wide variety of possible ways of doing mapping. It has at least: cache, clone, dust, crypt, delay, ebs, error, flakey, linear, mirror, multipath, raid, snapshot, striped, thin, zero Some of the bits I mentioned. It's relatively lower level, but many other services and the like leverage the device mapper. E.g. LVM uses device mapper for its lower level configuration, as does cryptestup e.g. for LUKS. Sometimes using device mapper more directly can be quite useful. E.g. want to test I/O issues, it has dust, error, and flakey, so one can set up device that gives errors on I/O based upon various criteria as specified. Can also specify different specific data to be read back - which may be different than that which was written, etc. Also has RAID capabilities - which brings me to example. Some while back, had case to assist someone in coming up with a solution to something they wanted to do. They had a fair sized hardware RAID array, RAID-5 with 4x12TB drives. They wanted to migrate to md raid5, with 4x12TB (new) drives. And they wanted to minimize downtime to the extent feasible. Well, easiest way to do that, would be to effectively layer RAID-1 - at least temporarily atop that - sync - then split that mirror. But LVM or md, etc. RAID-1 not so great for that - as they'd all generally want to write their relevant headers on the devices, etc., and at best non-trivial to write that data, and block status tracking data, etc., somewhere else - if they even support that at all. So, solution for that? Use device mapper. Can then just directly raid1 mirror the blocks of the two devices - the original RAID-5 hardware RAID device, and the newer replacement md raid5 software RAID device. So, first of all, documentation, etc. There's the dmsetup(8) man page. Pretty good, but it leaves a lot to be desired. Notably doesn't well cover a lot of details that are or may be necessary. So next stop - and what it quite refers to - kernel documentation. E.g.: file:///usr/share/doc/linux-doc-6.1/html/admin-guide/device-mapper/ etc. Pretty good ... but not so great on, e.g. more complete examples, etc. and even some relevant information turned out to be quite missing from the kernel documentation (though hey, could probably read the relevant source ... but that generally wouldn't be easiest way). So next, some bits of Internet searching ... and found some good resources, e.g.: https://wiki.gentoo.org/wiki/Device-mapper#Mirror_and_RAID1 So, at least taken together, there was sufficient information. So ... I earlier did a test demo run, to show how it could be done ... but didn't save all my information/notes on that, so let me repeat that. And a bit better this time - including log devices - that way even if, e.g. system were to crash while the sync was in progress, could be cleanly resumed. So, I don't have 8x14TB in spare drives sitting around, so I'll do much smaller with some space on /tmp and some files there and using losetup(8) to create block devices suitable for use. So, below, my comments on lines starting with //: // So, don't have 8x14TB to work with, but on /tmp at present, can // easily spare 64GiB. So will instead do ~24GiB to emulate the old and // 4x8GiB to emulate the new. Since the "old" source is hardware // RAID-5, will just do appropriately sized storage for that, as I don't // have hardware RAID-5 available for that. First let's create the // backing files and loop devices for our first 4 devices to represent // our target drives. # mkdir /tmp/dmr1 && cd /tmp/dmr1 # truncate -s $(expr 8 \* 1024 \* 1024 \* 1024) f{1,2,3,4} && (for n in 1 2 3 4; do losetup -f --show f"$n"; done) /dev/loop2 /dev/loop3 /dev/loop4 /dev/loop5 # // And now create our software md raid5 device: # mdadm --create --level=raid5 --raid-devices=4 /dev/md24 /dev/loop[2-5] mdadm: Defaulting to version 1.2 metadata mdadm: array /dev/md24 started. # // And now let's get its exact size: $ cat /sys/block/md24/size 50276352 $ // That's in 512 byte blocks. As for bytes $ expr 50276352 \* 512 25741492224 $ // So let's create our source device of exactly that size: # truncate -s 25741492224 f0 && losetup -f --show f0 /dev/loop6 # // In our actual case, source would need to be not larger than the // target. If source were larger, we'd need to shrink the (relevant) // data before copying, e.g. reduce the size of the filesystem a bit, // possibly repartition slightly, etc. // And let's confirm our sizes match: $ cmp /sys/block/md24/size /sys/block/loop6/size && echo MATCHED MATCHED $ // And let's create some data on our source device: # mkfs -t ext3 -L 24gr5 -m 0 /dev/loop6 && mount -o nosuid,nodev /dev/loop6 /mnt && { dd if=/dev/urandom of=/mnt/urandom bs=1048576 status=none; < /mnt/urandom sha512sum && umount /mnt; } // ... dd: error writing '/mnt/urandom': No space left on device 6e0487bed425a7bb667d169e001415a4a18c6413ee5be56f032ebac7ea827dae9caee9ab0d0801e1d1b537eabc75cee9842de00a1089fd0afd7e7630752128aa - # // Now lets set up our device mapper device, will do this with metadata // devices to track, so, e.g. if interrupted (e.g. system crash // or abrupt power down, or drives disconnected, etc.) can still safely // resume after (we'll essentially ignore that our backing store is on // the volatile /tmp - this is still just demo after all). // Unfortunately the kernel documentation doesn't say how large these // need to be. Probably relatively small % of the total space, I'll // give it more than ample space on sparse file, then we can look at // actual block usage after. So, devices 50276352 512 byte blocks, // let's say 5% of that - that's ought be much more than enough - but // sparse files, won't much matter. $ echo '50276352*512/20' | bc -l 1287074611.20000000000000000000 $ // And let's round that up to 4KiB boundary: $ echo '1287074611.2/4/1024' | bc -l 314227.20000000000000000000 $ expr 314228 \* 4 \* 1024 1287077888 $ # truncate -s 1287077888 m{0,1} # (for n in 0 1; do losetup -f --show m"$n"; done) /dev/loop7 /dev/loop8 # // Will also look at status right after creation and 30 seconds later, // and will mount right after we create it too: # dmsetup create dmr1 --table '0 50276352 raid raid1 5 0 region_size 32 rebuild 1 2 /dev/loop7 /dev/loop6 /dev/loop8 /dev/md24' && dmsetup status dmr1 && mount -o nosuid,nodev /dev/mapper/dmr1 /mnt && sleep 30 && dmsetup status dmr1 0 50276352 raid raid1 2 Aa 0/50276352 recover 0 0 - 0 50276352 raid raid1 2 Aa 2046848/50276352 recover 0 0 - # // And a while later we have: # dmsetup status dmr1 0 50276352 raid raid1 2 AA 50276352/50276352 idle 0 0 - # // That field of all uppercase "A" characters tells us the RAID-1 // devices are fully synced up. // Let's deconstruct our RAID-1 device and compare the files on the // filesystems - which have now been copied via mirroring. Will also // read and recompute hash of one of the files to also check that still // matches. Also, so they don't conflict on the filesystems, will // change the label and UUID on the "old" one, so the original data of // that remains on the "new" target one. # umount /mnt // check again that we're synced before removal: # dmsetup status dmr1 0 50276352 raid raid1 2 AA 50276352/50276352 idle 0 0 - # dmsetup remove dmr1 # tune2fs -L 24gr5.old -U random /dev/loop6 tune2fs 1.47.0 (5-Feb-2023) # mkdir mnt-old mnt-new # mount -o ro,nosuid,nodev /dev/loop6 mnt-old # mount -o ro,nosuid,nodev /dev/md24 mnt-new # cmp mnt-{old,new}/urandom && echo MATCHED MATCHED # < mnt-new/urandom sha512sum 6e0487bed425a7bb667d169e001415a4a18c6413ee5be56f032ebac7ea827dae9caee9ab0d0801e1d1b537eabc75cee9842de00a1089fd0afd7e7630752128aa - # // And we can see that the files match and the hash matches our earlier. // Let's do it one more time, except this time with the filesystem very // busy while it's mounted and doing the RAID-1 sync. // And this time we mirror from the new, to old, as the new now has // exactly the data we want. # umount mnt-old && umount mnt-new # dmsetup create dmr1 --table '0 50276352 raid raid1 5 0 region_size 32 rebuild 0 2 /dev/loop7 /dev/loop6 /dev/loop8 /dev/md24' && mount -o nosuid,nodev /dev/mapper/dmr1 /mnt && { dd if=/dev/urandom of=/mnt/urandom bs=1048576 status=none; < /mnt/urandom sha512sum; } dd: error writing '/mnt/urandom': No space left on device c818fb535d037b01868252a3f2464cc17fa70b8f4cb21436a0f7d3d9c85b4783ac7e7835f47d55c588287688013b687743886f5640ddfb91ff9e2f8177dd5b38 - # // We check until we see it's synced: # dmsetup status dmr1 0 50276352 raid raid1 2 AA 50276352/50276352 idle 0 0 - # // Then we unmount, and again reconfirm it's synced: # umount /mnt # dmsetup status dmr1 0 50276352 raid raid1 2 AA 50276352/50276352 idle 0 0 - # // Now we again deconstruct the RAID-1, update label and UUID on old, // and mount and compare, and also again compute hash on one of the // files to see that also matches: # dmsetup remove dmr1 # tune2fs -L 24gr5.old -U random /dev/loop6 tune2fs 1.47.0 (5-Feb-2023) # mount -o ro,nosuid,nodev /dev/loop6 mnt-old # mount -o ro,nosuid,nodev /dev/md24 mnt-new # cmp mnt-{old,new}/urandom && echo MATCHED MATCHED # < mnt-old/urandom sha512sum c818fb535d037b01868252a3f2464cc17fa70b8f4cb21436a0f7d3d9c85b4783ac7e7835f47d55c588287688013b687743886f5640ddfb91ff9e2f8177dd5b38 - // So, files again matched, and our hash again matches. Also, checked // how much space was actually used for the meta devices to track RAID-1 // status: # stat -c '%b %n' m[01] 400 m0 400 m1 # // So that's 400 512 byte blocks, 200 KiB. // That size apparently depends upon size of device and region_size, // and does apparently have a limit on total maximum size it will use // for that meta device. So, our original scenario to replicate the RAID-5 from hardware RAID to software (md) RAID while minimizing downtime would go about like this: o create the new target md device, note its precise size o stop all I/O on the old device (e.g. unmount it). o note size of old - must not be larger than new, if necessary shrink and/or repartition, etc. or the like as appropriate. o create dm device as RAID-1 between the old and new, syncing to new o return I/O to service (using the dm device in place of the old device) o after sync has completed (may take hours to days for larger/slower storage), as before, stop all I/O - except now on the dm device instead of the old device. Again check/wait until it's fully synced. o tear down the dm device o Adjust labels, UUIDs, etc. if/as applicable on old to not conflict with new o return I/O to service, using new in place of old o if/as desired, tear down or decommission old Key advantage is relative minimization of downtime - across the hours (or days or more) while the RAID-1 mirror is syncing to duplicate the storage data, no downtime is needed, and the storage can be used per usual. A bit of downtime to shuffle about, notably from old, to dm, then to new, but other than that, things generally remain online and actively available. Additionally, no headers or encapsulation need happen on the old or new storage itself - that's all handled externally, so that keeps it clean and relatively simple.

1 0

Web automation, e.g. AT&T's "Unified Messaging" (voicemail) and downloading all that content.
by Michael Paoli 19 Nov '24

19 Nov '24

Ah, lovely web automation! :-) So, lately had a little mini-project to give myself. AT&T's "Unified Messaging" (voicemail). Wanted to "cut the cord" - bye-bye landline - porting ye olde landline # to mobile. But first, wanted to download all of my content from AT&T's "Unified Messaging" (voicemail). AT&T's "Unified Messaging" (UM/um), in addition to ye olde phone DTMF ("Touch Tone") interface to the voicemail, also has web interface. So, web interface. Essentially works as web GUI interface to email in "INBOX", messages are stored in email, and within an email item, voicemail as .wav attachment, text attachment having transcript as body - which will generally have empty body if it wasn't able to transcribe it. And generally html attachment, an html version of that text attachment. And, "of course", Perl also has the lovely WWW::Mechanize. So ... I got to programming. mitmproxy was also handy to figure out some bits going on within the SSL/TLS communications between client (e.g. web browser) and AT&T server(s). And got the key bits of that finished up this past Sunday. And got 'er all nicely downloaded. $ um.att.com um.att.com: Inbox is empty. Exiting $ That's what it outputs at the end, when there's nothing left to download. It also handles deleting the "email" item (message and related) from the AT&T "INBOX" once it's successfully downloaded. $ cd ~/.um.att.com.d/data $ ls -A1 | sed -e 's/^.*\././' | sort | uniq -c | sort -k 1,1bnr -k 2,2 117 .eml 117 .wav 113 .txt 112 .html $ Very nicely handles it all. .eml is the full raw "original" email as AT&T has it in the "INBOX", .wav files are the raw audio portion thereof, .txt the text transcript (or no file if that part was empty), and .html the html equivalent of that text. Ah, I was wondering about why one less .html than .txt ... peeking further, the .txt has: Message too short for transcription And that original .eml has no html part, and the .wav ... yeah, no words in that audio. Alas, I didn't clean out quite all the junk before downloading everything ... and the slight mismatch makes that bit of junk pretty easy to spot ... likewise grep on the .txt files is rather handy. So, the file names start with ISO date and time, which is derived from the Date: header which is timestamp of when the end of the message was received. Likewise that same time data is used to set the mtime on the files. File names also contain data from Subject: and From: fields, generally identifying caller name/number, or when not (CNID) identified otherwise unknown caller / Identity Withheld, e.g.: ... unknown caller ... Identity withheld <unknown_caller...> https://www.mpaoli.net/~michael/bin/um.att.com Ah, one of these days I need to tweak Apache configuration so it "knows", e.g. that file (and that name and location), can be handled like plain text, not a binary. Yeah, I know there's a "magic" type option that can read the files and make intelligent guess on that, but that's excessive overhead for most cases - so really need to just configure the exceptions ... down to directory or even per-file basis. (On my to-do list ... with thousands of other items yet to be done ... at least maybe when I get around to it). And ... maybe even others might find it handy, or handy starting point. Though this one was done almost / mostly as a one-off/one-shot. Though until the number completes being ported over, very handy to still check if anything has shown up there, and download it if so. It might need some adjustments to handle some other email messages. E.g. the ones from AT&T about the INBOX being nearly full. And looks like I probably won't have need for that (nor example data to match it to and test it on). And I didn't handle the more general email case (which I think UM will also accept and have in "INBOX"), as I only ever used UM for voicemail.

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

BALUG-Talk