Another kit downloaded was clearly intended to scan other computers, in the same manner that the compromised computer had been scanned. One of the utilities within the kit builds a list of Internet hosts which had port 22/tcp open. Another component takes the list of hosts and runs it through a password dictionary, both against the root account with many different passwords — such as, admin, 123, 123123, toor, and password — and also against user accounts, typically with blank passwords or using the compromised account's username. It was the use of this latter utility that was tripping the IDS alerts.

Recovery

To recover the machine, we made a backup of some of the data on it — but no executable content — and the system was re-built from CDs. There was no evidence that the attacker had managed to install a rootkit, but then they may have just been really clever at hiding their tracks. We don't want to take any unnecessary risks. Passwords were audited on the newly installed machine, and the ssh daemon was moved to an alternate port as part of a defense-in-depth policy.

If your workplace has implemented ITIL, you might think that your first responsibility is to get the customer back up and running as soon as possible. However, you need to balance haste with getting enough information to contain and eradicate the threat, rather than just immediately wiping the machine and re-installing. Fortunately, in this case, it was only one machine that was affected, but there were several others on site which could have been compromised in the same manner.

Prevention

After you've sorted out your immediate problems, don't forget that a vital part of incident response is trying to prevent such things occurring in the future.

This compromise could have been prevented in a few ways. First, I prefer to run ssh daemons on an alternate port when accessible to the outside world — just in case someone does decide to set a very poor password. It also cuts down the amount of rubbish to wade through in your log files. If you have users besides yourself on a machine, you can install pam_cracklib or john the ripper to regularly audit the password strength of user accounts.

I also called the other users who were administering their own SSH servers which were accessible from outside the firewall and suggested they take similar precautions.

The IDS had demonstrated that it worked as designed, but at this point you might wish to get it to page you if it ever sees outgoing SSH brute force attempts, so you can respond more quickly to any future incidents of this sort.

Conclusion

We had a false sense of security about Linux machines on our site. None have ever been compromised due to security vulnerabilities within the software, but several have as a result of misconfiguration on the part of the administrators. We were also too busy watching for Windows worms, such as Slammer, Blaster and Welchia rather than looking out for more general types of port-scanning. The IDS was configured to page me in the case of a large amount of port-scanning on ports important to Windows, but not for SSH at that stage.

People also have some misconceptions about SSH, as they do with VPN. Yes, it is a secure way of transmitting data, but that just means the attacker has a secure tunnel to your internal network if you misconfigure it! Using encryption also makes your job of using the IDS harder, as it can't see the contents of the attacker's shell session.

Having a database of all your equipment and your network topology is invaluable when responding to incidents. Together with the IDS alerts and a regular p0f dump of what machines were actually active on the network each day, it gives you a superb view of what is actually happening during an incident. You may want to add additional tools such logging a each machine's OS and patch level on a regular basis, or how many outbound packets your firewall is dropping for each machine. After you've dealt with an incident, try to think what data could have helped you and whether you could collect it as a matter of routine.