[BALUG-Talk] Servers and security

[Michael Paoli]

> From: "Kim Davalos" <kdavalos@sonic.net>
> To: conspire@linuxmafia.com, BALUG-Talk <balug-talk@lists.balug.org>
> Subject: [BALUG-Talk] Servers and security
> Date: Thu, 29 Mar 2018 07:28:57 -0700

Ah, well, first of all ... don't want to cross-post - notably send
the same email to multiple lists - that tends to cause annoyances
for list admins - folks hit "Reply-all", the replies go to all lists,
many such folks aren't subscribed to all lists, the list admins get
emails about attempted postings to list by a non-subscriber ...
anyway, so, ... don't do that.  :-)

> Curious about what folks due to harden/secure their servers.  
> Specifically I am NOT asking to be told what to do/how to do it.

Well, that's a huge topic in-and-of-itself.

I typically start the analysis from the bottom and work up.  I'll give
at least bit of overview.

Physical security.  With some slight exceptions, if it's not physically
secured, it's not secure.  In general, if one can gain physical access,
one can get at the data ... period ... well, yeah, with some exceptions.
Most notably/relevant - good strong robust encryption (and suitable
protection of (private) keys).  Such encryption will protect exposure
of the data - but may do little to help with availability (if all the
data is encrypted and someone picks up and walks off with the server,
the data may not get exposed ... but it may also quite cease to be
available).

And, "of course", in general, there's a whole lot in terms of
threat model - what are the (more/most) probable attacks, what
are the risks, what's the cost/benefit analysis, what are the
"worst case" scenarios on the downside risks ... and what are such
to varying levels of possible but improbable.  Eventually there's
some acceptable risk ... once one gets low enough on the
probabilities (large invading army of tanks and armed soldiers
points tank artillery and other guns and makes various threats
etc. and demand access ... not a threat model most need be significantly
concerned about).

There are also related areas ... e.g. availability, backup/restore/recovery,
etc.

On physical security, one other bit I'll mention beyond the direct physical,
and (related as counter-measure) encryption, ... tamper-resistant hardware.
Some hardware - e.g. certain integrated circuits (ICs), etc., may be rather
to quite tamper resistant.  Not tamper *proof* - such isn't 100% fool
proof (they keep making more crafty and ingenious fools - that do and try
stuff that nobody ever thought they would), but depending how
built/designed, some types of hardware can be made to - at least minimally,
provide evidence that hardware tampering has occurred, and, to the more
extreme cases, hardware can be made very tamper-resistant to extracting
any of the to-be-protected data from the device/hardware.  E.g. some ICs
are designed to not only not give up their data without proper
authentication/key, but to also effectively self-destruct (notably
wipe or destroy that data) if one tries to access the data through
unauthorized means (e.g. tries too many incorrect keys, or tries to
physically open the device).  Even some PC type hardware provides some
minimal tamper-resistance ... e.g. some will set a BIOS/CMOS flag if
the cover is opened, and will warn about that until that "error" is
flagged as cleared ... and if one wants to usefully use that, the
clearing of that error would generally require one to first use a
BIOS/CMOS password.

So ... 'bout half a step up from physical (and skipping some related
bits between - e.g. social engineering, procedures, ...) ... booting,
how does the system boot ... particularly from what, and how is that
controlled/secured.  Goes back to physical ... and related - e.g. remote
administration and out-of-band access, but ... how does one control
what the system can/can't boot from and when/where/how that can be
changed.  If one can get the system to boot from arbitrary data, one
can then bypass (almost) all security ... notably anything except good
strong encryption - and even that may be bypassed if such boot control
provides a means to get at the keys.

So ... jumping ahead several steps, and presuming Linux (or at least Unix
or similar).  Files and permissions ... basic host based security.
I think nowadays folks tend to far too much ignore this bit and jump
straight to network.  Even if nowadays there's typically helluva lot more
systems and networking and interconnectivity, etc., basic host security
does still matter.  If nothing else, it's a rather to highly key part of
defense-in-depth.  Defense-in-depth is essentially the security equivalent
of not putting all your eggs in one basket.  If one relies entirely upon
one (or a very few) things for security, and that one (or few) thing(s)
breaks or is bypassed ... yeah, defense-in-depth ... multiple layers of
security.  One can find examples/analogies with, e.g. castle.  Many might
mostly just think of the castle walls and heavy door.  But, ... there's
typically much more.  E.g. may start with watch tower portion to see
potential approaching threat ... then there's the moat, right? - with
alligators, or very hungry piranha, or that contains or can be quickly
topped with a highly flammable liquid and lit on fire.  And then there's
the draw bridge and heavy door - maybe an iron/steel gate just in front of
the door too.  There are also layers of defenses inside the castle too,
but, one can read lots more about castles to learn of that ... but
hopefully one gets the general idea.

Anyway, host based security - most notably files and permissions and
ownerships, etc.  With judicious and proper settings thereof, a host
can typically be much more secure.  Notably if someone or something
attempts to do something they shouldn't be able to do, with proper
permissions on files (of any type ... remember too, Linux/Unix, a
directory is "just" a different type of file), it's much more likely
the attempt to do the unauthorized will be blocked.  Appropriate
logging/monitoring also useful - if someone/something attempts
something it's not authorized to do - best to have some log/audit
trail on that - figure out what happened, who done it, what they
did/didn't manage to do, etc.  Such is also useful for someone
having had the access, but exceeded what they were supposed to
do - e.g. also log actions the system considers "authorized".
"Of course" too, cost/benefit analysis ... logging "everything"
can also be a hit on performance and resources.  Oh, ... and
log securely and remotely ... (also) over encrypted channels to
remote system(s).  And ... administrative realms, those that have
access to the systems being logged, should generally *NOT* have access
to those systems that are logging the actions on those systems!
See also the general topics of "separation of duties" - that covers
that rather well, and many related security best practices.
Also relevant for separation of duties, is, e.g.
sysadmins "vs." developers - often there's inherent conflict
of interest.  Nowadays, especially with DevOps, often that distinction
is much more blurred.  But too, often there are countermeasures and
other compensating controls (at least partially) to help, at least
reasonably cover those risks ... or at least cover enough such that
the cost/benefit analysis and risk assessment comes out at least okay.

And ... a bit more on permissions, etc., before I forget to mention.
I typically look at things such as what is/isn't writable by what
users/groups.  What is/isn't readable by what users/groups.
If something must be world writable (e.g. /tmp) is it properly
secured (e.g. sticky bit) ... anything else world writable, and does
it need to be and if so is it properly secured?  What about all the
SUID/SGID stuff?  Is that more-or-less okay, or can we clamp down further
on that?  Are there places folks likely screwed that up?  (Yes, I did
once find big SGID hole in a binary, and responsibly reported it to the
vendor).  Anyway, check, review, (carefully) adjust as/where appropriate,
check/test/retest, etc.  I've yet to see any out-of-the-box Linux/Unix
installation that couldn't be significantly improved on security with
such a reasonably careful review and suitable adjustments thereupon.
Oh, and don't forget about umask, and various configuration bits, etc.
Also, file mount permissions.  How 'bout nosuid on all filesystems except
where it's actually needed (/ and /usr and if present often /opt).
How 'bout read-only (ro) on all filesystems except where need to
normally be mounted read-write (rw).  I mean, other than / and /var
and /home, do you really need rw most of the time?  (extra hint:
Debian (and most/all derivatives / apt based systems, one can configure
such that, e.g. /usr is remounted rw for performing routine software
maintenance tasks, and remounted ro at the completion of such ... sadly,
at least last I checked, yum lacks and such capability to add that to
its configuration.  Anyway, for example, on my Debian systems, I  
generally have /boot and /usr mounted ro, but apt remounts them rw  
when I'm doing upgrades/installation/removal of software - then  
remounts them ro after (at least if it can without a reboot or such).   
There are also various ways
to check, if a filesystem fails to remount ro, what is preventing that,
Debian even has tools to aid with such).  Also, ro gives one a
slight performance boost (but one sacrifices atime data).
Some folks do noexec on, e.g. /tmp ... but I personally think that's a bit
of overkill, as noexec is generally quite trivial to bypass,
and noexec on /tmp (and /var/tmp) has a tendency to break some stuff (e.g.
script/program that will write something there then expect to be able to
execute it there ... such is kind'a common in, e.g., installation
scripts/programs).  Oh, and almost forgot to mention ... nodev.
Unless it contains /dev, mount it nodev.  :-)  And when mounting
semi-arbitrary filesystems (e.g. something on USB/SD flash, some ISO
image, etc.), geez ... nosuid, nodev, rw only if you need/expect to be
altering data there ... also if you need to protect access to the data,
mount it beneath some directory that already locks out those that
shouldn't have access ... e.g. some 500 or 550 permission directory ...
unless you really expect to need/want to make it, at least potentially,
accessible to all.  Paying attention to that can also be important if
the UIDs/GIDs on the mounted filesystem don't match/correspond to those
on the host they're being mounted upon.

SE Linux ... at least historically on Unix/Linux ... superuser
(UID 0 ... almost always "root", and generally referred to as such),
can do essentially anything on the system (well, just about, and short
of cracking strong encryption without access to key).  Among other things,
SE Linux breaks that apart into a set of capabilities.  A commonly used
example: back up everything that's on the host ... okay, a quite typically
needed capability.  But to do that, what does one really need?  Need to
be able to read everything that's to be backed up, and need to write
the backup data to wherever that data is being written.  But ...
does that require ability to alter data on the host, or read memory, or
any of the many things superuser can do?  No, not at all.  Hence, enter
capabilities (not unique to SE Linux, but in the land of Linux, SE Linux
would generally be the mechanism by which such is done).  Anyway,
SE Linux *can* well be used to significantly enhance the security of a
Linux host.  It's also a whole 'nother layer of complexity ... so,
for better *and* worse (and the trade-offs that come with that), there
too is that to be considered.  So, sure SE Linux - may be unwarranted
overkill for many scenarios/situations or may fail the cost/benefit and
similar analysis ... but in others it may be rather to highly
warranted.  And, ... one needn't go crazy with SE Linux, ... many distros
can allow one to start doing some SE Linux without a whole lot 'o
extra complexity and overhead ... but there may be various gottchas if
one isn't playing sufficiently close attention.  And, of course, like
really anything in security, SE Linux is no silver bullet (well, unless
perhaps one is doing werewolf security - in which case silver bullet
may in fact be exactly that).

There's also various tamper detection, ... e.g. tripwire, logging,
etc.  Is one able to detect when something's been change that shouldn't
have been changed? ... and even if most or all other layers of security
fail to otherwise stop or detect some unauthorized change being made?
See also inotify.

Oh, and an important layer (somewhere in/around the stack) ... policy.
If one doesn't have a security policy, one doesn't have (much) security.
Security policy needs really be signed off at the highest applicable
levels, and backed by the appropriate management structure.  If one doesn't
have that, the much of the security is wishful thinking - as (often major)
holes can be poked at it most anywhere at most any time for most any
(lack of) reason whatsoever.  So, ... policy and relevant backing thereof
matters.  And also, properly done, policy also covers how to handle various
types of security incidents.  It's generally best to have that figured out
(or at least mostly figured out) *before* there is a security incident.

And, ... skipping some more layers, ... (closer to) networking, ...
what services is one running?  Are they only local to the host, or are
they exposed to network interfaces?  There's all that and the
analysis/securing of such as appropriate.  And too, there's firewalls,
logging of traffic, etc.  There's also (often (mis?)placed) under
network security, various packet inspection, intrusion
detection/prevention based on network packets ... heck, even various
anti-malware for email could be lumped into that category.

So, ... firewalls, ... there's tons 'o information, etc. about that.  I
often think *too much* emphasis is put there (though firewalls often
can be or are important, if not critical ... but ... mostly they're
only one to a few layers ... albeit typically significant/important ones).
But ... firewalls, I'll mention a few key points.  Often folks(/managers)
over rely upon or are overconfident about firewalls.  E.g. a not-so-clueful
manager may have attitude essentially like "Oh, not a problem, we're
protected by a nice secure firewall".  To which I may often counter-argue
something like, "and ... we have ... over 150,000 authorized users
within that firewall.  Would one take an isolated population that
large - even of mostly quite decent folks - and have *zero* police
force present?  Heck, even Vatican City has the occasional redident
that will murder."  Or, too, as often I and others have described
such: "Hard crunchy outside, soft chewy middle."  Oh, also, with
respect to "firewalls" - I do also quite like host hardening.  Even
*with* firewalls (on and/or off host), host hardening is a key element
of defense-in-depth.  There's a whole lot else that can be said
about firewalls - but most all that would be redundant to what most
everyone else says about firewalls ... so I'll skip all that.

And, yes, ... mentioned social engineering.  The humans are often the
weakest links.  Much of that goes to policy, awareness, enforcement,
etc.

So, yeah, ... (computer/system) security ... a huge topic in-and-of-itself.
I've just barely scratched the surface in what I've described.

> More interested in hearing about different practices and approaches,  
> e.g., firewall management - iptables/nftables vs something like  
> Check Point, limiting installed packages to what is necessary,  
> closing unused ports, access restrictions, etc.

I think one will find both a wide range of approaches (many greatly varied
cost/benefit and threat model environments) ... and also quite a bit of
commonality/overlap too.

> Best would be vigorous debate and disagreement.

:-)  Ah, ... some nice sparring anyway, ... we'll skip the flame wars, right?