[BALUG-Talk] FHS, ro /usr ?, separate (or not) /boot, etc.

[Michael Paoli]

Moving this (mostly) non-Mailman (notwithstanding some relevant context/
examples) "discussion" to BALUG-Talk - as most of it is not
BALUG administrivia (BALUG-Admin) specific, but of more general Linux
interest.  (FHS, filesystems, security, ...)

> From: "Rick Moen" <rick@linuxmafia.com>
> Subject: Re: [BALUG-Admin] BALUG lists ... blank body with command  
> in Subject: header, ...
> Date: Thu, 20 Jul 2017 12:48:35 -0700

> Quoting Michael Paoli (Michael.Paoli@cal.berkeley.edu):
>
>> originally the files in
>> /etc/mailman/en/
>> match those in
>> /usr/share/mailman/en/
>> And looks like the bits editable in web GUI default to
>
> It doesn't help that Mailman uses symplinks for a number of things
> criss-crossing between /var/lib/mailman locations and /usr/share/mailman
> (I think) ones.

Well, as/where "it" does that (or distribution's (e.g. Debian's)
modification/installation/configuration of such does so,
I think it is a semi-elegant "solution" to a somewhat ugly problem.
E.g. per Filesystem Hierarchy Standard (FHS)
http://www.pathname.com/fhs/
https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard
etc., it's permissible for /usr to be mounted read-only (ro) in
"normal" operation (e.g. when not performing administrative actions
such as adding/removing/updating software, but including administrative
actions such as editing of configurations of installed software, routine
changes which the software itself makes, etc.).  However, fair bits of
software don't follow FHS on that, or didn't in past and there's desire
to have backwards compatibility with things being more easily alterable
if not physically, at least logically, in their (former) location(s)
on/under /usr.  So, one quite common way to do that is to create
symbolic links from relevant location(s) on /usr to relevant locations
where things will be editable (and/or contain variable data), e.g.
somewhere under /etc and/or /var (and/or for 3rd party applications,
/opt - though better yet, /opt should also be possibly ro (though I
don't think FHS goes quite as far as to explicitly state such?), and any
otherwise "variable" stuff under /opt ought be symbolic link(s) to elsewhere,
e.g. under /var/opt and/or something similar for /etc (e.g. /etc itself, or
/etc/opt or the like)).

Similar applies to cases where a relatively common binary can be covered
by somewhat different (but similar) programs ... and even both may be
installed.  E.g. update-alternatives(8) and /etc/alternatives.  Though
that may have first come to Debian (not sure which distribution used it
first), I've even see it show up on Fedora and derivatives (Red Hat,
CentOS, ...) - though on Fedora and derivatives thereof, I've seen it used
on far far far fewer software packages compared to Debian ... but at least
that number is greater than zero, and it does exist on Fedora and
derivatives thereof (at least last I checked).  Anyway, what the
alternatives mechanism "solves" is essentially this (notably for those
that may not be familiar).  Let's see we're on a host, and after having
settled the emacs/vi war, everyone wants vi(1) ;-) (okay, theoretical
argument).  But, alas, some want nvi (which is vi on BSD), and some want
vim (which last I was aware, Fedora doesn't even make nvi available!).
What if some want *both*, but, well, vi on standard PATH in /bin/ or
/usr/bin ... well, it's got to provide one of those (or some reasonable
vi-like alternative) ... ought that not be a reasonable matter of system
configuration?  Well, with alternatives, it is, but ... there's FHS.
So, those configurations should be managed ... under /etc.  But, what
if/when the binaries are properly under /usr/bin?  Well, we have, e.g.:
$ readlink /usr/bin/vi
/etc/alternatives/vi
$ readlink /etc/alternatives/vi
/usr/bin/nvi
$
So those on such a lovely system can be treated to beautiful nvi (which is
the vi on BSD systems) by using the command vi (and avoid all the
annoyances of vim
http://www.rawbw.com/~mp/linux/vim/vim_annoyances.txt
), but heaven forbid, should a user on such system actually want
to execute vim (e.g. to redemonstrate its bugs/annoyances), they
merely need type vim
$ type vim
vim is /usr/bin/vim
$ readlink /usr/bin/vim
/etc/alternatives/vim
$ readlink /etc/alternatives/vim
/usr/bin/vim.basic
$ ls -l /usr/bin/vim.basic
-rwxr-xr-x 1 root root 2240936 Mar  9 16:56 /usr/bin/vim.basic
$
And the lovely alternatives mechanism also updates related links, so
e.g.
$ man vi
will give what corresponds to /usr/bin/vi
So ... a pretty darn good solution to a relatively difficult
problem/issue.  Some distributions, however decide thou shalt have
no choice, and, e.g. provide only and exactly one editor for vi,
namely vim and all its annoyances (or on BSD, one gets the lovely
nvi for vi, but if one wants to get vim by invoking command vi,
there isn't mechanism that puts that in place on the PATH and in standard
location for all general users).

>> And per FHS, etc., the above typically wouldn't be editable (/usr
>> filesystem may be mounted read-only....
>
> Well, it's hardly the only offender.  I discovered this when I did the
> research that produced
> http://linuxmafia.com/faq/Admin/linuxmafia.com-backup.html , and
> [re-]discovered that Apache httpd's cgi-bin directory is in
> /usr/lib/cgi-bin/

Egad, uhm, yeah, some software (notably often "upstream") may still have
stuff in/under /usr that ought not be there.  Some distributions will
choose to alter the software such that the offending stuff is no longer
under /usr - which fixes the FHS issue - and if also fixed corresponding
changes are made for all related software for that distribution - is
then mostly all fine and dandy - *except* it still breaks any
backwards compatibility that may be expected - e.g. users might
expect the stuff to at least logically appear to be under /usr,
likewise software (e.g. random home-grown local script) may have been
developed expecting the stuff under /usr - and that script breaks if
it's no longer (at least logically) present under /usr where it was
and where the script expects it.

> I likewise keep /usr normally mounted ro, and likewise use the apt hooks
> to autoremount it for package operations -- but live with the fact that
> I need to also manually remount /usr to fiddle with cgi-bin scripts.
>
> SpamAssassin's rulesets and scoring settings in /usr/share/spamassassin
> are another example (and I guess I should add that directory to my
> backup list).

Yes, many examples of stuff that ought not (at least physically) be
under /usr, and is generally (FHS) fixed on quality distributions
(e.g. Debian) to be FHS compliant, but may provide symbolic links
or such on /usr - to allow for compatibility with things/users that
may still expect it to appear/function as if it's on /usr - even if
it's no longer physically on/under /usr.

> I'll admit to getting a bit disenchanted with many of the more-recent
> additions to FHS (specifically /media and /srv).  I've read the
> rationales, and I don't agree that /media is justified in addition to
> /mnt.  I am now grudgingly reconsidering my reflexive opposition to
> /srv, because there's a good case that CGI-BIN scripts should be there
> rather than anywhere in /usr, including the '/usr may be ro' one.
> Therefore, SpamAssassin rulesets and scoring settings arguably are
> better there, too.

Well, not sure if I've read the latest FHS (I also miss something that was
removed from earlier FHS - and I also continue to use:
/var/local/
it answered the question of where variable content local stuff goes,
which at least partially answered the question:
So, I've got lots and lots of data that needs additional filesystem(s),
where ought those be mounted?  There doesn't seem to be a fully
universal answer for that, and FHS (at least I last read) fails to fully
and unambiguously answer that.  Well, /usr/local (or beneath that) would
be suitable for some stuff, /var/local (at least per earlier FHS) for
some other stuff ... bits of FHS and what I'd guestimate for logical
extensions ... /etc/local for config of local (not distribution or 3rd
party) stuff, /var/opt for variable 3rd party stuff, etc. - anyway, at
least something approximating that - but alas I see many (most) sites
that don't do it that way, and often, egad, create semi-arbitrary
additional mount points right under / <sigh>.)

I think /mnt is "nice", but has always been rather limited - as it's
just a single mostly guaranteed (normally) available mount point - so
mostly useful for a temporary mount location, or when one needs to mount
directly under root filesystem and root filesystem is ro (e.g. recovery
or maintenance scenario) and there may be no other filesystems mounted, but
need a dir to mount something at least temporarily ... oh where oh where? -
that's when /mnt is really your friend.
So, ... /media, /sys, /srv ... I think things could've gone in better
directions.  Adding stuff right under / should be done very
conservatively/minimally - I think there were probably better solutions
for all of those.  And I think /sys was (and is) totally unnecessary ...
great stuff there, but it should'a gone elsewhere - e.g. /proc/sys.

> Any time someone says 'Oh, let's solve this problem by putting it in a
> new directory off /', I get cranky and conservative-minded, though.
> Which is why I think /sys is regrettable -- even though I've heard how
> we haplessly fell into the need for that on Linux systems.
> (Essentially, the kernel people made such a dog's breakfast out of /dev
> that they had to make a new tree to start over.  Nice going, guys.)

Whether it was /dev, or /proc, I think at least between *major changes*
of Linux kernel versions, those were ample opportunities to clean up
and fix (and add new stuff) under /dev and/or /proc - no need to add a
whole separate /sys (or geez, at least don't mount it at /sys - why not
have it just mounted as /dev/sys or /proc/sys, huh?  I mean we have
a perfectly good devpts mounted on /dev/pts - why not likewise put
sysfs at /proc/sys or /dev/sys ?  Ah well.

I've got mixed thoughts on /media ... I think it's at least better
than /cdrom, but perhaps replacing /cdrom with something a bit more
general than /media?  E.g. perhaps combining functionality of /mnt and
/media somewhere/somehow?  I dunno.

But yeah, sometime it feels too much like feature creep, with too much
stuff being added not quite, but almost willy-nilly, to FHS, with
semi-arbitrary new locations.

> I actually wanted to consult you on /boot being a separate filesystem.
> I've done that since forever, but am acutely aware that the original
> rationale no longer applies, that of old BIOSes being unable to branch
> from int 13h to disc cylinder locations 1024 and over.
>
> So, is there a 2017 reason for still having /boot separate?  I can think
> of scenarios, like you have most of the system on md mirrored partitions
> but want your bootloader to be able to run from a pair of ext2 ones on
> both drives, _or_ you have most of the system encrypted.  Are there
> reasons I'm missing?

Well ... :-) I think there definitely still are reasons to have /boot
as a separate filesystem.  Some may not be relevant, or as
relevant/significant as before, but I think many of the
arguments/advantages of separate /boot still apply.
Stuff I'd argue/advocate for separate /boot (I'll skip the
counter-arguments):
o ro/security( / accident reduction) - a separate /boot can be ro,
   without necessitating / be ro
o faster/safer fsck/recovery - for filesystems/data critical to boot and
   getting to at least single user mode, within reason, smaller and
   separate is generally an advantage.  Many often lump *lots* into root
   (/) filesystem and/or it may get rather to quite large, so having a
   separate /boot can be rather to quite advantageous.
o one can be ultra-conservative on the filesystem type for /boot,
   independent of choice of filesystem type for /.  E.g. I think many of
   my /boot filesystems are still ext2 - exceedingly backwards compatible
   with most anything that would deal with them or that I might want to
   have deal with them.  And with /boot kept sufficiently small and
   clean, fsck time for recovery of /boot even if it's just ext2, is
   still pretty darn quick.  Though as I think about it now / nowadays,
   and think what filesystem type is most widely and thoroughly tested
   and used, well supported, maintained, and works with pretty much any
   Linux tools ... might be time by now to advance /boot from ext2 to
   ext3.  For similar I like to not have / be "too big", and like
   separate /usr, /home, and /var.  Same also applies, as feasible, to
   those filesystems too.
o lovely useful tip/"trick" I like and have always done on /boot:
   $ ls -ld /boot/boot
   lrwxrwxrwx 1 root root 1 Dec 29  2011 /boot/boot -> .
   So, bootloader configs and all that, to the paths have a leading
   /boot, or not?  If the path is relative to the root of the filesystem
   used for boot, and that may be /boot filesystem, or may be / (root)
   filesystem, do we need to change having / not having that leading
   /boot in the boot loader configuration bits?  No.  Any reasonably sane
   boot loader deals fine with symbolic links.  by having the symbolic
   link I note above, all boot configurations have the /boot prefix, and
   it doesn't matter if /boot is a separate filesytem or not, or changes
   back and forth - the logical pathnames all remain prefixed with /boot
   and the bootloader handled 'em all fine - period.  End of all that
   silliness of whether or not we have a /boot prefix in there - we
   always do and it always works.
o all kinds of stuff one might possibly want for / (root), but that
   bootloaders at least often can't handle or would prefer the
   bootloaders generally not deal with that complexity (even if the
   (claim they) generally can):
   o encrypted / (root)
   o md / (root)
   o LVM / (root)
   o latest wizbang spiffy/experimental filesystem type for / (root)
   o two or more of the above ;-) (I'm using at least two of the above on
     at least one host :-))
o separate /boot leaves open the possibility to do more "interesting"
   things with / (root) - even in future and if one isn't doing so
   presently and may not have any immediate or short-term plans to do so.
o rather darn easy to get rid of a separate /boot and place it on /
   (root) - presuming suitable filesytem type, not encrypted, and other
   similar requirements, but significantly more difficult to later add a
   separate /boot filesystem - especially if one hasn't saved reasonable
   space at start of drive, it's a single drive system or that's the only
   drive that can be reasonably conveniently booted from; drive space is
   relatively cheap these days - probably good to invest in saving that
   space up front and separate on the drive - even if one isn't doing a
   separate /boot filesystem - in case one wants to change it later to
   separate /boot; to make such transition even easier in future if one
   isn't doing separate /boot, also create a partition there - even if
   not at all using it presently.
o 1024 cylinder limit?  Not much of an issue these days - but might want
   to still consider it for possibly high levels of backwards
   compatibility.  With much larger drives, there's whole partitioning
   format issue and UEFI and all that - but that's a whole 'nother set of
   questions / can of worms.

>> I suppose too, I ought set up some user in group (or with access to group)
>> list, so that many of the list related commands can be run fine with
>> less-than-root privileges - a not otherwise privileged user but also in
>> group list would seem optimal - as group list seems to have the permissions
>> to do the needed, but would generally lack permissions to muck with stuff it
>> ought not (e.g. files owned by list by not writable by group list).
>> At least on Debian, of what I've seen so far, looks like all the normally
>> needed actions to manage mailman lists, are controlled/accessed via the
>> list group.
>
> Makes sense.  FWIW, when I need to muck about with Mailman's files I
> _generally_ just su to list.  Most particularly, it's really important
> to be user 'list' if you regenerate a pipermail archive using
> /var/lib/mailman/bin/arch .  Otherwise, if you make the mistake of doing
> it as root, the ownership of the generated Web archive is wrong and
> you'll get 403 errors until you figure that out and fix it.

Good points.  But I noticed on Debian (presently still oldstable):
$ grep '^list:' /etc/passwd
list:x:38:38:Mailing List Manager:/var/list:/usr/sbin/nologin
$ awk -F: '{if($3==38)print;}' /etc/group
list:x:38:
$
So ... su - list ain't gonna cut it.  8-O.
But I added:
$ expand -t 4 < ~root/bin/golist
#!/bin/sh
cd / &&
exec perl -e '
     ($),$()=(q(38 38),38);
     ($<,$>)=(38,38);
     $ENV{HOME}=q(/var/list);
     exec {bash} (q(/bin/bash));
'
$
:-)
It's a start anyway ... I ought add bits on that, to adjust PATH,
read list's HOME from getpwent or the like for list's entry,
probably even likewise read list's UID and GID similarly.  Like I said,
it's a start.
But also, may want to have two different "list" role accounts,
one using list's UID (and GID) - most notably for stuff that will, e.g.
create new files/directories., and one that uses list's GID, but
unprivileged user, for operations that are essentially read-only.
Ought also make that thing entirely perl - it's not like perl
lacks a chdir function.