Hints on how to check your machine for intrusion

The compromise of kernel.org and related machines has made it clear that
some developers, at least, have had their systems penetrated.  As we
seek to secure our infrastructure, it is imperative that nobody falls
victim to the belief that it cannot happen to them.  We all need to
check our systems for intrusions.  Here are some helpful hints as
proposed by a number of developers on how to check to see if your Linux
machine might be infected with something:


0. One way to be sure that your system is not compromised is to simply
   do a clean install; we can all benefit from a new start sometimes.
   Before reinstalling any systems, though, consider following the steps
   below to learn if your system has been hit or not.

1. Install the chkrootkit package from your distro repository and see if it
   reports anything.  If your distro doesn't have the chkroot package,
   download it from:
 http://www.chkrootkit.org/

   Another tool is the ossec-rootcheck tool which can be found at:
 http://www.ossec.net/main/rootcheck

   And another one is the rkhunter program:
    http://www.rootkit.nl/projects/rootkit_hunter.html
   [Note, this tool has the tendancy to give false-positives on some
   Debian boxes, please read /usr/share/doc/rkhunter/README.Debian.gz if
   you run this on a Debian machine]

2. Verify that your package signatures match what your package manager thinks
   they are.

   To do this on a rpm-based system, run the following command:
    rpm --verify --all
   Please read the rpm man page for information on how to interpret the
   output of this command.

   To do this on a Debian based system, run the following bash snippet:
 dpkg -l \*|while read s n rest; do if [ "$s" == "ii" ]; then echo $n;
 fi; done > ~/tmp.txt
 for f in `cat ~/tmp.txt`; do debsums -s -a $f; done
   If you have a source-based system (Gentoo, LFS, etc.) you presumably
   know what you are doing already.

3. Verify that your packages are really signed with the distro's keys.

   Here's a bash snippet that can do this on a rpm based system to
   verify that the packages are signed with any key, not necessarily
   your distro's key.  That exercise is left for the reader:

 for package in `rpm -qa`; do
  sig=`rpm -q --qf '%{SIGPGP:pgpsig}\n' $package`
  if [ -z "$sig" ] ; then
   # check if there is a GPG key, not a PGP one
   sig=`rpm -q --qf '%{SIGGPG:pgpsig}\n' $package`
   if [ -z "$sig" ] ; then
    echo "$package does not have a signature!!!"
   fi
  fi
 done
   Unfortunately there is no known way of verifying this on Debian-based
   systems.

4. To replace a package that you find suspect, uninstall it and install
   it anew from your distro.  For example, if you want to reinstall the
   ssh daemon, you would do:
 $ /etc/init.d/sshd stop
 rpm -e openssh
 zypper install openssh # for openSUSE based systems
 yum install openssh # for Fedora based systems

   Ideally do this from a live cdrom boot, using the 'rpm --root' option
   to point rpm at the correct location.


5. From a liveCD environment, look for traces such as:
   a. Rogue startup scripts in /etc/rc*.d and equivalent directories.
   b. Strange directories in /usr/share that do not belong to a package.
      This can be checked on an rpm system with the following bash snippet:
 for file in `find /usr/share/`; do
  package=`rpm -qf -- ${file} | grep "is not owned"`
  if [ -n "$package" ] ; then
   echo "weird file ${file}, please check this out"
  fi
 done
6. Look for mysterious log messages, such as:
   a. Unexpected logins in wtmp and /var/log/secure*, quite possibly
      from legitimate users from unexpected hosts.
   b. Any program trying to touch /dev/mem.
   c. References to strange (non-text) ssh version strings in
      /var/log/secure*.  These do not necessarily indicate *successful*
      breakins, but they indicate *attempted* breakins which means your
      system or IP address has been targeted.

7. If any of the above steps show possible signs of compromise, you
   should investigate further and identify the actual cause.  If it
   becomes clear that the system has indeed been compromised, you should
   certainly reinstall the system from the beginning, and change your
   credentials on all machines that this machine would have had access
   to, or which you connected to through this machine.  You will need
   to check your other systems carefully, and you should almost
   certainly notify the administrators of other systems to which you
   have access.

Finally, please note that these hints are not guaranteed to turn up
signs of a compromised systems.  There are a lot of attackers out there;
some of them are rather more sophisticated than others.  You should
always be on the alert for any sort of unexpected behavior from the
systems you work with.

----------------------------------------------------------------------------

----------------------------------------------------------------------------

I would like to add here a few controls I ran on firewall and system logs,
that are easy to perform and which report few false positives :

  - check that communications between your local machines are expected ;
    for instance if you have an SSH bouncing machine, it probably receives
    tens of thousands of SSH connection attempts from outside every day,
    but it should never ever attempt to connect to another machine unless
    it's you who are doing it. So checking the firewall logs for SSH
    connections on port 22 from local machines should only report your
    activity (and nothing should happen when you sleep).

  - no SSH log should report failed connection attempts between your
    local machines (you do have your keys and remember your password).
    And if it happens from time to time (eg: user mismatch between
    machines), it should look normal to you. You should never observe
    a connection attempt for a user you're not familiar with (eg: admin).

     $ grep sshd /var/log/messages
     $ grep sshd /var/log/messages | grep 'Invalid user'
  - outgoing connections from your laptop, desktop or anything should
    never happen when you're not there, unless there is a well known
    reason (package updates, browser left open and refreshing ads). All
    unexpected activity should be analysed (eg: connections to port 80
    not coming from a browser should only match one distro mirror).
    This is particularly true for cheap appliances which become more
    and more common and are rarely secured. A NAS or media server, a
    switch, a WiFi router, etc... has no reason to ever connect anywhere
    without you being aware of it (eg: download a firmware update).

  - check for suspicious DNS requests from machines that are normally
    not accessed. A number of services perform DNS requests when
    connected to, in order to log a resolved address. If the machine
    was penetrated and the logs wiped, the DNS requests will probably
    still lie in the firewall logs. While there's nothing suspect from
    a machine that does tens of thousands DNS requests a day, one that
    does 10 might be suspect.

  - check for outgoing SMTP connections. Most machines probably never
    send any mail outside or route them through a specific relay. If
    one machine suddenly tries to send mails directly to the outside,
    it might be someone trying to steal some data (eg: mail ssh keys).

  - check for long holes in logs various service logs. The idea is that
    if a system was penetrated and the guy notices he left a number of
    traces, he will probably have wiped some logs. A simple way to check
    for this is to count the number of events per hour and observe huge
    variations. Eg:

       $ cut -c1-9 < /var/log/syslog |uniq -c
       8490 Oct  1 00
       7712 Oct  1 01
       8316 Oct  1 02
       6743 Oct  1 03
       7428 Oct  1 04
       7041 Oct  1 05
       7762 Oct  1 06
       6562 Oct  1 07
       7137 Oct  1 08
        160 Oct  1 09
    Activity looks normal here. Something like this however would be
    extremely suspect :

       8490 Oct  1 00
        712 Oct  1 01
       6743 Oct  1 03

  - check that you never observe in logs a local address that you
    don't know. For instance, if your reverse proxy is on a DMZ which
    is provided by the same physical switch as your LAN and your switch
    becomes ill and loses all its VLAN configuration, it them becomes
    easy to add an alias to the reverse-proxy to connect directly to
    LAN machines and bypass a firewall (and its logs).

  - it's always a good exercise to check for setuids on all your machines.
    You'll generally discover a number of things you did not even suspect
    existed and will likely want to remove them. For instance, my file
    server had dbus-daemon-launch-helper setuid root. I removed this crap
    as dbus has nothing to do on such a machine. Similarly I don't need
    fdmount to mount floppies. I might not use floppies often, and if I do,
    I know how to use sudo.

       $ find / -user root -perm -4000 -ls

  - last considerations to keep in mind is that machines which receive
    incoming connections from outside should never be able to go out, and
    should be isolated in their own LAN. It's not hard to do at all, and
    it massively limits the ability to bounce between systems and to steal
    information. It also makes firewall logs much more meaningful, provided
    they are stored on a support with limited access, of course :-)
 

Also refer: 

http://www.ossec.net/main/rootcheck

http://www.rootkit.nl/projects/rootkit_hunter.html

http://www.chkrootkit.org/