From: "Kim Davalos" kdavalos@sonic.net To: conspire@linuxmafia.com, BALUG-Talk balug-talk@lists.balug.org Subject: [BALUG-Talk] Servers and security Date: Thu, 29 Mar 2018 07:28:57 -0700

Ah, well, first of all ... don't want to cross-post - notably send the same email to multiple lists - that tends to cause annoyances for list admins - folks hit "Reply-all", the replies go to all lists, many such folks aren't subscribed to all lists, the list admins get emails about attempted postings to list by a non-subscriber ... anyway, so, ... don't do that. :-)

Curious about what folks due to harden/secure their servers. Specifically I am NOT asking to be told what to do/how to do it.

Well, that's a huge topic in-and-of-itself.

I typically start the analysis from the bottom and work up. I'll give at least bit of overview.

Physical security. With some slight exceptions, if it's not physically secured, it's not secure. In general, if one can gain physical access, one can get at the data ... period ... well, yeah, with some exceptions. Most notably/relevant - good strong robust encryption (and suitable protection of (private) keys). Such encryption will protect exposure of the data - but may do little to help with availability (if all the data is encrypted and someone picks up and walks off with the server, the data may not get exposed ... but it may also quite cease to be available).

And, "of course", in general, there's a whole lot in terms of threat model - what are the (more/most) probable attacks, what are the risks, what's the cost/benefit analysis, what are the "worst case" scenarios on the downside risks ... and what are such to varying levels of possible but improbable. Eventually there's some acceptable risk ... once one gets low enough on the probabilities (large invading army of tanks and armed soldiers points tank artillery and other guns and makes various threats etc. and demand access ... not a threat model most need be significantly concerned about).

There are also related areas ... e.g. availability, backup/restore/recovery, etc.

On physical security, one other bit I'll mention beyond the direct physical, and (related as counter-measure) encryption, ... tamper-resistant hardware. Some hardware - e.g. certain integrated circuits (ICs), etc., may be rather to quite tamper resistant. Not tamper *proof* - such isn't 100% fool proof (they keep making more crafty and ingenious fools - that do and try stuff that nobody ever thought they would), but depending how built/designed, some types of hardware can be made to - at least minimally, provide evidence that hardware tampering has occurred, and, to the more extreme cases, hardware can be made very tamper-resistant to extracting any of the to-be-protected data from the device/hardware. E.g. some ICs are designed to not only not give up their data without proper authentication/key, but to also effectively self-destruct (notably wipe or destroy that data) if one tries to access the data through unauthorized means (e.g. tries too many incorrect keys, or tries to physically open the device). Even some PC type hardware provides some minimal tamper-resistance ... e.g. some will set a BIOS/CMOS flag if the cover is opened, and will warn about that until that "error" is flagged as cleared ... and if one wants to usefully use that, the clearing of that error would generally require one to first use a BIOS/CMOS password.

So ... 'bout half a step up from physical (and skipping some related bits between - e.g. social engineering, procedures, ...) ... booting, how does the system boot ... particularly from what, and how is that controlled/secured. Goes back to physical ... and related - e.g. remote administration and out-of-band access, but ... how does one control what the system can/can't boot from and when/where/how that can be changed. If one can get the system to boot from arbitrary data, one can then bypass (almost) all security ... notably anything except good strong encryption - and even that may be bypassed if such boot control provides a means to get at the keys.

So ... jumping ahead several steps, and presuming Linux (or at least Unix or similar). Files and permissions ... basic host based security. I think nowadays folks tend to far too much ignore this bit and jump straight to network. Even if nowadays there's typically helluva lot more systems and networking and interconnectivity, etc., basic host security does still matter. If nothing else, it's a rather to highly key part of defense-in-depth. Defense-in-depth is essentially the security equivalent of not putting all your eggs in one basket. If one relies entirely upon one (or a very few) things for security, and that one (or few) thing(s) breaks or is bypassed ... yeah, defense-in-depth ... multiple layers of security. One can find examples/analogies with, e.g. castle. Many might mostly just think of the castle walls and heavy door. But, ... there's typically much more. E.g. may start with watch tower portion to see potential approaching threat ... then there's the moat, right? - with alligators, or very hungry piranha, or that contains or can be quickly topped with a highly flammable liquid and lit on fire. And then there's the draw bridge and heavy door - maybe an iron/steel gate just in front of the door too. There are also layers of defenses inside the castle too, but, one can read lots more about castles to learn of that ... but hopefully one gets the general idea.

Anyway, host based security - most notably files and permissions and ownerships, etc. With judicious and proper settings thereof, a host can typically be much more secure. Notably if someone or something attempts to do something they shouldn't be able to do, with proper permissions on files (of any type ... remember too, Linux/Unix, a directory is "just" a different type of file), it's much more likely the attempt to do the unauthorized will be blocked. Appropriate logging/monitoring also useful - if someone/something attempts something it's not authorized to do - best to have some log/audit trail on that - figure out what happened, who done it, what they did/didn't manage to do, etc. Such is also useful for someone having had the access, but exceeded what they were supposed to do - e.g. also log actions the system considers "authorized". "Of course" too, cost/benefit analysis ... logging "everything" can also be a hit on performance and resources. Oh, ... and log securely and remotely ... (also) over encrypted channels to remote system(s). And ... administrative realms, those that have access to the systems being logged, should generally *NOT* have access to those systems that are logging the actions on those systems! See also the general topics of "separation of duties" - that covers that rather well, and many related security best practices. Also relevant for separation of duties, is, e.g. sysadmins "vs." developers - often there's inherent conflict of interest. Nowadays, especially with DevOps, often that distinction is much more blurred. But too, often there are countermeasures and other compensating controls (at least partially) to help, at least reasonably cover those risks ... or at least cover enough such that the cost/benefit analysis and risk assessment comes out at least okay.

And ... a bit more on permissions, etc., before I forget to mention. I typically look at things such as what is/isn't writable by what users/groups. What is/isn't readable by what users/groups. If something must be world writable (e.g. /tmp) is it properly secured (e.g. sticky bit) ... anything else world writable, and does it need to be and if so is it properly secured? What about all the SUID/SGID stuff? Is that more-or-less okay, or can we clamp down further on that? Are there places folks likely screwed that up? (Yes, I did once find big SGID hole in a binary, and responsibly reported it to the vendor). Anyway, check, review, (carefully) adjust as/where appropriate, check/test/retest, etc. I've yet to see any out-of-the-box Linux/Unix installation that couldn't be significantly improved on security with such a reasonably careful review and suitable adjustments thereupon. Oh, and don't forget about umask, and various configuration bits, etc. Also, file mount permissions. How 'bout nosuid on all filesystems except where it's actually needed (/ and /usr and if present often /opt). How 'bout read-only (ro) on all filesystems except where need to normally be mounted read-write (rw). I mean, other than / and /var and /home, do you really need rw most of the time? (extra hint: Debian (and most/all derivatives / apt based systems, one can configure such that, e.g. /usr is remounted rw for performing routine software maintenance tasks, and remounted ro at the completion of such ... sadly, at least last I checked, yum lacks and such capability to add that to its configuration. Anyway, for example, on my Debian systems, I generally have /boot and /usr mounted ro, but apt remounts them rw when I'm doing upgrades/installation/removal of software - then remounts them ro after (at least if it can without a reboot or such). There are also various ways to check, if a filesystem fails to remount ro, what is preventing that, Debian even has tools to aid with such). Also, ro gives one a slight performance boost (but one sacrifices atime data). Some folks do noexec on, e.g. /tmp ... but I personally think that's a bit of overkill, as noexec is generally quite trivial to bypass, and noexec on /tmp (and /var/tmp) has a tendency to break some stuff (e.g. script/program that will write something there then expect to be able to execute it there ... such is kind'a common in, e.g., installation scripts/programs). Oh, and almost forgot to mention ... nodev. Unless it contains /dev, mount it nodev. :-) And when mounting semi-arbitrary filesystems (e.g. something on USB/SD flash, some ISO image, etc.), geez ... nosuid, nodev, rw only if you need/expect to be altering data there ... also if you need to protect access to the data, mount it beneath some directory that already locks out those that shouldn't have access ... e.g. some 500 or 550 permission directory ... unless you really expect to need/want to make it, at least potentially, accessible to all. Paying attention to that can also be important if the UIDs/GIDs on the mounted filesystem don't match/correspond to those on the host they're being mounted upon.

SE Linux ... at least historically on Unix/Linux ... superuser (UID 0 ... almost always "root", and generally referred to as such), can do essentially anything on the system (well, just about, and short of cracking strong encryption without access to key). Among other things, SE Linux breaks that apart into a set of capabilities. A commonly used example: back up everything that's on the host ... okay, a quite typically needed capability. But to do that, what does one really need? Need to be able to read everything that's to be backed up, and need to write the backup data to wherever that data is being written. But ... does that require ability to alter data on the host, or read memory, or any of the many things superuser can do? No, not at all. Hence, enter capabilities (not unique to SE Linux, but in the land of Linux, SE Linux would generally be the mechanism by which such is done). Anyway, SE Linux *can* well be used to significantly enhance the security of a Linux host. It's also a whole 'nother layer of complexity ... so, for better *and* worse (and the trade-offs that come with that), there too is that to be considered. So, sure SE Linux - may be unwarranted overkill for many scenarios/situations or may fail the cost/benefit and similar analysis ... but in others it may be rather to highly warranted. And, ... one needn't go crazy with SE Linux, ... many distros can allow one to start doing some SE Linux without a whole lot 'o extra complexity and overhead ... but there may be various gottchas if one isn't playing sufficiently close attention. And, of course, like really anything in security, SE Linux is no silver bullet (well, unless perhaps one is doing werewolf security - in which case silver bullet may in fact be exactly that).

There's also various tamper detection, ... e.g. tripwire, logging, etc. Is one able to detect when something's been change that shouldn't have been changed? ... and even if most or all other layers of security fail to otherwise stop or detect some unauthorized change being made? See also inotify.

Oh, and an important layer (somewhere in/around the stack) ... policy. If one doesn't have a security policy, one doesn't have (much) security. Security policy needs really be signed off at the highest applicable levels, and backed by the appropriate management structure. If one doesn't have that, the much of the security is wishful thinking - as (often major) holes can be poked at it most anywhere at most any time for most any (lack of) reason whatsoever. So, ... policy and relevant backing thereof matters. And also, properly done, policy also covers how to handle various types of security incidents. It's generally best to have that figured out (or at least mostly figured out) *before* there is a security incident.

And, ... skipping some more layers, ... (closer to) networking, ... what services is one running? Are they only local to the host, or are they exposed to network interfaces? There's all that and the analysis/securing of such as appropriate. And too, there's firewalls, logging of traffic, etc. There's also (often (mis?)placed) under network security, various packet inspection, intrusion detection/prevention based on network packets ... heck, even various anti-malware for email could be lumped into that category.

So, ... firewalls, ... there's tons 'o information, etc. about that. I often think *too much* emphasis is put there (though firewalls often can be or are important, if not critical ... but ... mostly they're only one to a few layers ... albeit typically significant/important ones). But ... firewalls, I'll mention a few key points. Often folks(/managers) over rely upon or are overconfident about firewalls. E.g. a not-so-clueful manager may have attitude essentially like "Oh, not a problem, we're protected by a nice secure firewall". To which I may often counter-argue something like, "and ... we have ... over 150,000 authorized users within that firewall. Would one take an isolated population that large - even of mostly quite decent folks - and have *zero* police force present? Heck, even Vatican City has the occasional redident that will murder." Or, too, as often I and others have described such: "Hard crunchy outside, soft chewy middle." Oh, also, with respect to "firewalls" - I do also quite like host hardening. Even *with* firewalls (on and/or off host), host hardening is a key element of defense-in-depth. There's a whole lot else that can be said about firewalls - but most all that would be redundant to what most everyone else says about firewalls ... so I'll skip all that.

And, yes, ... mentioned social engineering. The humans are often the weakest links. Much of that goes to policy, awareness, enforcement, etc.

So, yeah, ... (computer/system) security ... a huge topic in-and-of-itself. I've just barely scratched the surface in what I've described.

More interested in hearing about different practices and approaches, e.g., firewall management - iptables/nftables vs something like Check Point, limiting installed packages to what is necessary, closing unused ports, access restrictions, etc.

I think one will find both a wide range of approaches (many greatly varied cost/benefit and threat model environments) ... and also quite a bit of commonality/overlap too.

Best would be vigorous debate and disagreement.

:-) Ah, ... some nice sparring anyway, ... we'll skip the flame wars, right?