The compromise of kernel.org and related machines has made it clear that
some developers, at least, have had their systems penetrated. As we
seek to secure our infrastructure, it is imperative that nobody falls
victim to the belief that it cannot happen to them. We all need to
check our systems for intrusions. Here are some helpful hints as
proposed by a number of developers on how to check to see if your Linux
machine might be infected with something:
0. One way to be sure that your system is not compromised is to simply
do a clean install; we can all benefit from a new start sometimes.
Before reinstalling any systems, though, consider following the steps
below to learn if your system has been hit or not.
1. Install the chkrootkit package from your distro repository and see if it
reports anything. If your distro doesn't have the chkroot package,
download it from:
http://www.chkrootkit.org/
Another tool is the ossec-rootcheck tool which can be found at:
http://www.ossec.net/main/rootcheck
And another one is the rkhunter program:
http://www.rootkit.nl/projects/rootkit_hunter.html
[Note, this tool has the tendancy to give false-positives on some
Debian boxes, please read /usr/share/doc/rkhunter/README.Debian.gz if
you run this on a Debian machine]
2. Verify that your package signatures match what your package manager thinks
they are.
To do this on a rpm-based system, run the following command:
rpm --verify --all
Please read the rpm man page for information on how to interpret the
output of this command.
To do this on a Debian based system, run the following bash snippet:
dpkg -l \*|while read s n rest; do if [ "$s" == "ii" ]; then echo $n;
fi; done > ~/tmp.txt
for f in `cat ~/tmp.txt`; do debsums -s -a $f; done
If you have a source-based system (Gentoo, LFS, etc.) you presumably
know what you are doing already.
3. Verify that your packages are really signed with the distro's keys.
Here's a bash snippet that can do this on a rpm based system to
verify that the packages are signed with any key, not necessarily
your distro's key. That exercise is left for the reader:
for package in `rpm -qa`; do
sig=`rpm -q --qf '%{SIGPGP:pgpsig}\n' $package`
if [ -z "$sig" ] ; then
# check if there is a GPG key, not a PGP one
sig=`rpm -q --qf '%{SIGGPG:pgpsig}\n' $package`
if [ -z "$sig" ] ; then
echo "$package does not have a signature!!!"
fi
fi
done
Unfortunately there is no known way of verifying this on Debian-based
systems.
4. To replace a package that you find suspect, uninstall it and install
it anew from your distro. For example, if you want to reinstall the
ssh daemon, you would do:
$ /etc/init.d/sshd stop
rpm -e openssh
zypper install openssh # for openSUSE based systems
yum install openssh # for Fedora based systems
Ideally do this from a live cdrom boot, using the 'rpm --root' option
to point rpm at the correct location.
5. From a liveCD environment, look for traces such as:
a. Rogue startup scripts in /etc/rc*.d and equivalent directories.
b. Strange directories in /usr/share that do not belong to a package.
This can be checked on an rpm system with the following bash snippet:
for file in `find /usr/share/`; do
package=`rpm -qf -- ${file} | grep "is not owned"`
if [ -n "$package" ] ; then
echo "weird file ${file}, please check this out"
fi
done
6. Look for mysterious log messages, such as:
a. Unexpected logins in wtmp and /var/log/secure*, quite possibly
from legitimate users from unexpected hosts.
b. Any program trying to touch /dev/mem.
c. References to strange (non-text) ssh version strings in
/var/log/secure*. These do not necessarily indicate *successful*
breakins, but they indicate *attempted* breakins which means your
system or IP address has been targeted.
7. If any of the above steps show possible signs of compromise, you
should investigate further and identify the actual cause. If it
becomes clear that the system has indeed been compromised, you should
certainly reinstall the system from the beginning, and change your
credentials on all machines that this machine would have had access
to, or which you connected to through this machine. You will need
to check your other systems carefully, and you should almost
certainly notify the administrators of other systems to which you
have access.
Finally, please note that these hints are not guaranteed to turn up
signs of a compromised systems. There are a lot of attackers out there;
some of them are rather more sophisticated than others. You should
always be on the alert for any sort of unexpected behavior from the
systems you work with.
----------------------------------------------------------------------------
----------------------------------------------------------------------------
I would like to add here a few controls I ran on firewall and system logs,
that are easy to perform and which report few false positives :
- check that communications between your local machines are expected ;
for instance if you have an SSH bouncing machine, it probably receives
tens of thousands of SSH connection attempts from outside every day,
but it should never ever attempt to connect to another machine unless
it's you who are doing it. So checking the firewall logs for SSH
connections on port 22 from local machines should only report your
activity (and nothing should happen when you sleep).
- no SSH log should report failed connection attempts between your
local machines (you do have your keys and remember your password).
And if it happens from time to time (eg: user mismatch between
machines), it should look normal to you. You should never observe
a connection attempt for a user you're not familiar with (eg: admin).
$ grep sshd /var/log/messages
$ grep sshd /var/log/messages | grep 'Invalid user'
- outgoing connections from your laptop, desktop or anything should
never happen when you're not there, unless there is a well known
reason (package updates, browser left open and refreshing ads). All
unexpected activity should be analysed (eg: connections to port 80
not coming from a browser should only match one distro mirror).
This is particularly true for cheap appliances which become more
and more common and are rarely secured. A NAS or media server, a
switch, a WiFi router, etc... has no reason to ever connect anywhere
without you being aware of it (eg: download a firmware update).
- check for suspicious DNS requests from machines that are normally
not accessed. A number of services perform DNS requests when
connected to, in order to log a resolved address. If the machine
was penetrated and the logs wiped, the DNS requests will probably
still lie in the firewall logs. While there's nothing suspect from
a machine that does tens of thousands DNS requests a day, one that
does 10 might be suspect.
- check for outgoing SMTP connections. Most machines probably never
send any mail outside or route them through a specific relay. If
one machine suddenly tries to send mails directly to the outside,
it might be someone trying to steal some data (eg: mail ssh keys).
- check for long holes in logs various service logs. The idea is that
if a system was penetrated and the guy notices he left a number of
traces, he will probably have wiped some logs. A simple way to check
for this is to count the number of events per hour and observe huge
variations. Eg:
$ cut -c1-9 < /var/log/syslog |uniq -c
8490 Oct 1 00
7712 Oct 1 01
8316 Oct 1 02
6743 Oct 1 03
7428 Oct 1 04
7041 Oct 1 05
7762 Oct 1 06
6562 Oct 1 07
7137 Oct 1 08
160 Oct 1 09
Activity looks normal here. Something like this however would be
extremely suspect :
8490 Oct 1 00
712 Oct 1 01
6743 Oct 1 03
- check that you never observe in logs a local address that you
don't know. For instance, if your reverse proxy is on a DMZ which
is provided by the same physical switch as your LAN and your switch
becomes ill and loses all its VLAN configuration, it them becomes
easy to add an alias to the reverse-proxy to connect directly to
LAN machines and bypass a firewall (and its logs).
- it's always a good exercise to check for setuids on all your machines.
You'll generally discover a number of things you did not even suspect
existed and will likely want to remove them. For instance, my file
server had dbus-daemon-launch-helper setuid root. I removed this crap
as dbus has nothing to do on such a machine. Similarly I don't need
fdmount to mount floppies. I might not use floppies often, and if I do,
I know how to use sudo.
$ find / -user root -perm -4000 -ls
- last considerations to keep in mind is that machines which receive
incoming connections from outside should never be able to go out, and
should be isolated in their own LAN. It's not hard to do at all, and
it massively limits the ability to bounce between systems and to steal
information. It also makes firewall logs much more meaningful, provided
they are stored on a support with limited access, of course :-)
Also refer:
http://www.ossec.net/main/rootcheck
http://www.rootkit.nl/projects/rootkit_hunter.html
http://www.chkrootkit.org/
No comments:
Post a Comment
tag ur valuable ideas below