Security incident report Oxford Particle Physics April 2003 P.D.Gronbech The start email from a user Hello, It appears that my password at CERN may have been compromised, as a precaution I have changed all my passwords and have forwarded this e-mail to you regarding the problem. Thanks. Regards, a.user. Cc: CERN Computer Security <[email protected]> Subject: URGENT: password change required CERN users, Passwords for the accounts below have been found in an intruder's file on a system at CERN which has been broken into. You must urgently change your password for accounts both at CERN and in your home institutes. Please inform your system manager at your home institute to check your systems for a linux ptrace exploit and possible suckit root installed. Thanks for your rapid collaboration. Denise Heagerty, [email protected] Google search • ptrace exploit http://www.securiteam.com/exploits/5WP0V009FC.html http://www.austin2600.org/mirrors/004560.html • suckit root kit http://www.soohrt.org/stuff/linux/suckit/ http://la-samhna.de/library/lkm.html http://hysteria.sk/sd/f/suckit/ • checkrootkit http://www.chkrootkit.org/ • ettercap http://ettercap.sourceforge.net/ • phrack http://www.phrack.org/ I’ve got 39 Linux boxes!! Don’t panic! • • • • • • • Find out which machines are affected Block in firewall Investigate re-install OS patch to prevent re-infection change passwords Inform others What to do • • • • • Run chkrootkit on each box. Found 8 suspect boxes. Block these boxes completely in the physics firewall One was running in promiscuous mode, so therefore running a network sniffer. So look at this one first. Try using the trick from one of the web pages to see what processes are running. cd /proc for i in `seq 1 33000`;do test –f $i/cmdline && (cat $i/cmdline; echo $i); done found ettercap running from /usr/share/locale/sk/.sk12 This dir was invisible to ls but you could cd into it. was able to kill the ettercap process was able to do the ./sk u to remove it but a reboot found it reinstalled itself. What to do.. • Later advice from cern suggested looking at ls –il /sbin/init /sbin/telinit output should be like this 192481 -rwxr-xr-x 1 root root 192489 lrwxrwxrwx 1 root root 26920 Mar 14 2002 /sbin/init 4 Mar 29 12:35 /sbin/telinit -> init This is a good quick test to try on all nodes. The suckit rootkit replaces the original init and stores the original as initsk12 where sk12 is the name or the directory where the kit is stored. cd /usr/share/locale/sk/.sk12 ./sk u This uninstalls the rootkit if you trust this! mv /sbin/init /sbin/init.hacked cp /sbin/initsk12 /sbin/init crash the system brings the system back up with out the rootkit running. More analysis and archiving can be done before the reinstall Short term fixes add the following line to /etc/rc.d/rc.local sysctl –w kernel.modprobe=/rubbish This prevents kernel modules loading and hence stops the LKM (Linux Kernel Module) type rootkits from getting loaded. An earlier solution was to just add kernel.modprobe=/rubbish to the /etc/sysctl.conf file but this will stop your server serving NFS files as NFS is a kernel module. Alternative fix • build a no ptrace module and load that. • This requires recompiling your kernel. • Several examples available on the web. • http://www.hackinglinuxexposed.com/tools/p/noptrace.c.html • http://slonce.com/kodz/code/Linux/noptrace.c • http://www.cantech.net.au/plug/2001-10/msg00434.html Further analysis • Usually the log of captured passwords would be in the /usr/share/locale/sk/.sk12 directory and the default name for this is .sniffer. • This can be read with horror as you see username password combinations for a whole host of machines. • So far I have not seen any ssh pass phrases so these may be safe • If ettercap was running then its log would be in its sub directory. I found this had not captured anything useful, which is likely to be the case on a switched network and with users mainly using ssh. • Why re install: You don’t know what other backdoors / trojans may have also been installed. Force all users to change passwords. • At approx 10:00am on 4th April I took a copy of the yp passwd file. • I emailed all users that they must change their password on a machine not infected! • At 11:00pm I compared the password hashes and closed all accounts that had not changed their passwords. • Subsequent emails from users requesting that they be let in can be dealt with and they can be told again to change ASAP. This can then be checked. What went wrong • We normally spend most of the time trying to block network based attacks. • Most scans are looking for old versions of ssh, httpd etc that have known exploits. • We normally protect against this by blocking these protocols in the firewall and only allowing say ssh to one machine from outside. • This machine should be patched regularly! • The problem is if a valid username password combination is captured somewhere the attacker can happily ssh into your machine. • Now you have to worry about local root exploits Kernel updates • • • • • Why do we not always update the kernel….. It breaks Open AFS (if installed with the standard rpm) It causes other problems? Don’t have enough time. Why might the latest kernel not work, its meant too!! • • • • • • • • • • We have SMP machines We have scsi disks We have software RAID We have ext3? We have NIS (yp) We have 30 users, and 600 processes We have 2-4 GB ram We use NFS extensively We have Gigabit Ethernet cards We have PBS Inform others • Inform your local CERT if you have one, ours was able to suggest I checked another machine in another part of physics, from seeing connections to it from the same ip addresses. • The suggestion was that it was a Greek dial up account. Have heard since that it might be from Estonia? • Using the HEPSYSMAN mailing list to alert other HEP sites proved very useful. Particularly relevant as our users are most likely to be logging into these systems and or CERN / DESY etc. This also allowed sites to check there systems and discover the hacked systems quicker helping to prevent further spread. • I would suggest and Andrew Sansum agrees with this that it should be compulsory for hep sites to own up about these incidents ASAP. • Email [email protected] about the compromised username password combinations you have found in your log files. Discussion • What schemes do people use for keeping up to date • Which kernels do you use • Red hat stock kernel • Red Hat latest • www.kernel.org compiled from source? • What firewalls do people use • Local system based firewalling ipchains / iptables, could we have an example setup available to share? • tcp wrappers • clean out old accounts • limit access from off site to one host? Bastion host idea. • One time passwords… crypto cards?