[BALUG-Talk] FHS, ro /usr ?, separate (or not) /boot, etc.

[Rick Moen]

Quoting Michael Paoli (Michael.Paoli@cal.berkeley.edu):

> Moving this (mostly) non-Mailman (notwithstanding some relevant context/
> examples) "discussion" to BALUG-Talk - as most of it is not
> BALUG administrivia (BALUG-Admin) specific, but of more general Linux
> interest.  (FHS, filesystems, security, ...)

Good idea.  Thanks.

[Mailman symlinks criss-crossing from var/lib/mailman to
/usr/share/mailman:]

> Well, as/where "it" does that (or distribution's (e.g. Debian's)
> modification/installation/configuration of such does so,
> I think it is a semi-elegant "solution" to a somewhat ugly problem.
> E.g. per Filesystem Hierarchy Standard (FHS)
> http://www.pathname.com/fhs/
> https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard
> etc., it's permissible for /usr to be mounted read-only (ro) in
> "normal" operation (e.g. when not performing administrative actions
> such as adding/removing/updating software, but including administrative
> actions such as editing of configurations of installed software, routine
> changes which the software itself makes, etc.).

Well, sure, it does have that very considerable merit.  It's something
that is merely initially a little bewildering, as you might think all of
Mailman (other than /etc files) lives in /var, and suddenly you turn
around a corner and find yourself in /usr.  

Also, speaking in very general terms, at a couple of jobs involving
large Internet installations, I developed a well-founded horror of
overuse of symlinks, as they can get you in serious trouble if you are
not careful: not just the nuisance of dangling symlinks (you see the
link and assume it points to the thing it always has, but that thing has
been moved or removed) but also circular symlinks.

The Mailman setup doesn't suffer those, of course, but if you've ever
debugged symlink problems, seeing them used abundantly will make your 
skin itch, just by association.

Again speaking very broadly, symlinks are also a reason to be very
careful about command options during mass-file operations, to ensure
that you don't accidentally traverse a symlink onto an unintended
filesystem.  E.g., for rsync, option '-x, --one-file-system       don't
cross filesystem boundaries', and similar ones for other tools including
cp and rm.  (This is also, on the side, yet another argument for keeping
/usr normally read-only -- because you might forget to use those
options.)

> /opt should also be possibly ro

IMO, /opt should be

lrwxrwxrwx   1 root root      14 Mar  4  2008 opt -> /usr/local/opt

I do not agree with the rationale for /opt existing at all.  IMO, it
was a bad decision kowtowing to proprietary software interests.  But,
like most other parts of an FHS implementation I don't approve of, 
it's easily fixed.  /media, I got rid of completely.

(And, just to clarify, I don't mean I like having anything in
/usr/local/opt, and IIRC there's still nothing in it.  I just think 
/opt is a bad idea, and if anything _were_ written there, I'd want it to
not land on the root filesystem.  Honestly, I should probably just
delete the symlink and the stil-empty directory it points to, but those
remain present to catch anything that tries to write there before I can
say 'No you don't.))

The reality of FHS, at least during its initial years under Dan Quinlan, 
was that it was a necessarily political effort to coax all of the Unixes
into moving gradually towards more-rational file locations and
less-wacky directory trees.  E.g., as FHS points out, the sendmail
binary was historically /usr/lib/sendmail, which is nuts since that
ain't no library.  And there were much worse things in the bad old days.
However, as a concidence of FHS being an effort at persuasion by someone
(Quinlan) with no inherent power base and quite the flock of cats to
herd, it has always included quite a number of dodgy compromises 

I'm not going to take the time to find examples, but you occasionally
find strange wording in the FHS giving very grudging blessing to some
subtree about which you think 'No, that's just bloody wrong', and then
you realise Quinlan was avoiding offending indulgers of some wrongheaded
Unix fossil practice that is still too prominent to ignore.

In the 2000s, Rusty Russell and (the late) Chris Yeoh, both former
coworkers of mine, took over the FHS from Quinlan, and it's under their
watch that FHS sprouted some innovations that at least at first (and in
the case of /media, lastingly) got my back up, a bit, and made me say
'I'm not sure I'm OK with that.'

But I certainly applaud the project's goals even if it's under the wing
of the at-best-useless Linux Foundation
(https://landley.net/notes-2010.html#18-07-2010), because I know Rusty
and know he doesn't do things without good reason.

BTW, in case I implied otherwise, I should mention that /sys is
_not_ FHS-mandated, which makes sense when you remember that /sys is a
Linuxism, and FHS aspires to create a standard most Unix at least
vaguely resemble (other than OSX, which inherits a bunch of mutant silly
stuff from NeXTSTEP, ideological 'the newbies hate FHS' efforts like
GoboLinux, guix/nix, the 0install package system -
https://en.wikipedia.org/wiki/Zero_Install, DJB's slashpackage scheme,
and Poettering & Co's latest terrible idea to make our lives miserable -
http://0pointer.net/blog/revisiting-how-we-put-together-linux-systems.html).

[snip discussion of /etc/alternatives, which I agree is cool, and proved
persuasive enough that RHEL/CentOS picked it up from Debian.  I don't
oppose symlinks generally.  I just am wary of their overuse.]

> Egad, uhm, yeah, some software (notably often "upstream") may still have
> stuff in/under /usr that ought not be there. 

Rewriting and recompiling are then in order, say I.  ;-)

> Yes, many examples of stuff that ought not (at least physically) be
> under /usr, and is generally (FHS) fixed on quality distributions
> (e.g. Debian) to be FHS compliant, but may provide symbolic links
> or such on /usr - to allow for compatibility with things/users that
> may still expect it to appear/function as if it's on /usr - even if
> it's no longer physically on/under /usr.

Sadly, the SpamAssassin and cgi-bin trees are physically on /usr, not
just symlinked in.

> Well, not sure if I've read the latest FHS 

3.0, the latest, appeared in 2015.
http://refspecs.linuxfoundation.org/FHS_3.0/fhs-3.0.html

(I also miss something that was
> removed from earlier FHS - and I also continue to use:
> /var/local/
> it answered the question of where variable content local stuff goes,


Yeah, I wonder what Rusty and Chris were thinking there?  Too bad FHS
lacks a changelog.

ISTR that FHS now says to place all variable data that dictates a
program's state into /var/lib, that this data should stick around
after a reboot, and that this data is host-specific.  And it 
says /var/opt should house data of apps installed in /opt.

IMO, having a 'lib' directory that isn't libraries and nothing but is a
mistake, but then much of /var remains a bit of a trainwreck.

> So, I've got lots and lots of data that needs additional filesystem(s),
> where ought those be mounted?

Wherever the nature and role of that data dictates.

> There doesn't seem to be a fully universal answer for that, and FHS
> (at least I last read) fails to fully and unambiguously answer that.

It couldn't possibly do so.  'Collection of data that needs additional
filesystems' is too vague a specifier.

> I think /mnt is "nice", but has always been rather limited - as it's
> just a single mostly guaranteed (normally) available mount point - so
> mostly useful for a temporary mount location, or when one needs to mount
> directly under root filesystem and root filesystem is ro (e.g. recovery
> or maintenance scenario) and there may be no other filesystems mounted, but
> need a dir to mount something at least temporarily ... oh where oh where? -
> that's when /mnt is really your friend.

Here's what I do:  If there's going to be a temporary filesystem, I
mkdir /mnt/$FOO and use that.  The CD-ROM got mounted at /mnt/cdrom, not
frellin' /cdrom nor /media.  Keep all that casual stuff under /mnt.  No
need for a separate tree anywhere else -- particularly not off the
system root..

[/sys:]

> Whether it was /dev, or /proc, I think at least between *major changes*
> of Linux kernel versions, those were ample opportunities to clean up
> and fix (and add new stuff) under /dev and/or /proc - no need to add a
> whole separate /sys (or geez, at least don't mount it at /sys - why not
> have it just mounted as /dev/sys or /proc/sys, huh?  I mean we have
> a perfectly good devpts mounted on /dev/pts - why not likewise put
> sysfs at /proc/sys or /dev/sys ?  Ah well.

They certainly could have put the new stuff within /proc somewhere
rather than punting and creating /sys (as of Linux 2.6 and later).  
It was a bad call.

[separate /boot:]

> Well ... :-) I think there definitely still are reasons to have /boot
> as a separate filesystem.  Some may not be relevant, or as
> relevant/significant as before, but I think many of the
> arguments/advantages of separate /boot still apply.
> Stuff I'd argue/advocate for separate /boot (I'll skip the
> counter-arguments):

Yes, I concur with a bunch of those.  I _really_ like having
system-critical boot stuff be on an isolated, quiescent, ultra-simple
and conservatively chosen filesystem, kept normally read-only.

> o lovely useful tip/"trick" I like and have always done on /boot:
>   $ ls -ld /boot/boot
>   lrwxrwxrwx 1 root root 1 Dec 29  2011 /boot/boot -> .

Eh, yes.  Noted down for use.

A trivial thing I do, but it can save your life (metaphorically) some
times:  All mountpoint directories get this, before one mounts the 
tree that goes there:

# touch NOTHING_IS_MOUNTED_HERE
# chattr +i NOTHING_IS_MOUNTED_HERE

If you ever unexpectedly see that zero-length file, you know... what it
says is so.


> Good points.  But I noticed on Debian (presently still oldstable):
> $ grep '^list:' /etc/passwd
> list:x:38:38:Mailing List Manager:/var/list:/usr/sbin/nologin
> $ awk -F: '{if($3==38)print;}' /etc/group
> list:x:38:
> $

FWIW, I have
list:x:38:38:Mailing List Manager:/var/list:/bin/sh

Can't remember whether the shell is a local tweak on my system or not.
The list account's a little pecular, but I really do su to root and then
list, to do mailing list archive manipulations, and it works fine.