Quoting Jonathan Jefferies (jjefferies@gracenote.com):
At the risk of "me-to"ism this sounds like a very interesting subject well worth attending.
FWIW, I don't really recognise "malware detection" as a separate thing worthy of study, generally. By contrast, detecting _root compromise_ is worthwhile as a last-ditch measure when/if your primary measures fail, i.e. prevention, damage reduction, defence in depth, hardening.
In particular, file-based IDSes like AIDE, Integrit, Samhain, and Prelude-IDS are what "rpm -qa" wants to be when it grows up and gets serious. (Question for any Scarlet Chapeau guys who still seriously uses that to detect root compromise: Why would /var/lib/rpm/* be reliable on a compromised host?)
I've advised some sysadmins that they should sell use of file-based IDESes to management as "virus scanners for Unix". (Doing the right thing for the wrong reasons is a whole lot better than nothing, eh?)