For example, p0f is an excellent tool to run on a SPAN port so you can see exactly what hosts are active at what times — the only problem is that the size of the output is huge. An example of dealing with p0f output on a SPAN port is described in Taming p0f by chunk processing STDIN. This data can then be imported into your favorite database.

At this point I booted the computer using a Knoppix-based distribution so I could take a look at the filesystem. If you think there's any chance you're going to end up in court, you need to be a lot more careful than I was. In this case, I correctly guessed that the damage was minimal and the chance of prosecuting was nil. These days I would use the Backtrack bootable CD which is designed specifically for this sort of thing.

Evidence Gathering

Of course, by powering off the computer, I lost any information that was stored in the RAM alone, but I was after a quick analysis in case I had other computers to look into — at this stage it was not clear how widespread the problem could be. One of my favorite forensics texts is Forensic Discovery by Dan Farmer and Wietse Venema which goes into a lot more detail about recovering information from disk and memory. You should at least keep a log book of what you're doing at each stage, and keep an audit trail of each piece of evidence you collect.

Initially, the contents of the /var directory was captured to a laptop via a network cable and the invaluable netcat utility.

server% nc -l -p 31337 > evil.tar

victim% tar cvf /tmp/evil.tar /var ; \
cat /tmp/evil.tar | nc -p 31337 server.ip.addr

server% tar xvf evil.tar

Examination of the /var/log/auth.log revealed password guessing attempts against the openssh daemon. Since the victim machine was attempting to scan other networks for SSH servers, it was reasonable to suppose it had been compromised by a guessed password, and in turn was probing for weak passwords. This topic had also been covered on the invaluable ISC diary not too long before the incident occurred.

The sshd log entries looked like this:

Jun 26 22:31:04 victim sshd[15384]:
  Failed password for root from ::ffff:w.x.y.z port 30937 ssh2
Jun 26 22:31:06 victim sshd[15386]:
  Illegal user network from ::ffff:w.x.y.z
Jun 26 22:31:06 victim sshd[15386]:
  error: Could not get shadow information for NOUSER
Jun 26 22:31:06 victim sshd[15386]: Failed password 
  for illegal user network from ::ffff:w.x.y.z port 30951 ssh2
Jun 26 22:31:08 victim sshd[15388]:
  Illegal user word from ::ffff:w.x.y.z
Jun 26 22:31:08 victim sshd[15388]:
  error: Could not get shadow information for NOUSER
Jun 26 22:31:08 victim sshd[15388]: Failed password 
  for illegal user word from ::ffff:w.x.y.z port 30963 ssh2
Jun 26 22:31:10 victim sshd[15390]: Failed password 
  for root from ::ffff:w.x.y.z port 30980 ssh2
Jun 26 22:31:11 victim sshd[15392]: Failed password 
  for root from ::ffff:w.x.y.z port 30992 ssh2
Jun 26 22:31:13 victim sshd[15394]: Failed password 
  for root from ::ffff:w.x.y.z port 31007 ssh2
Jun 26 22:31:15 victim sshd[15396]: Failed password 
  for root from ::ffff:w.x.y.z port 31021 ssh2
Jun 26 22:31:17 victim sshd[15398]: Failed password 
  for root from ::ffff:w.x.y.z port 31031 ssh2
Jun 26 22:31:19 victim sshd[15400]: Failed password 
  for root from ::ffff:w.x.y.z port 31049 ssh2
Jun 26 22:31:20 victim sshd[15403]: Failed password 
  for root from ::ffff:w.x.y.z port 31062 ssh2
Jun 26 22:31:22 victim sshd[15405]: Failed password 
  for root from ::ffff:w.x.y.z port 31073 ssh2

We can see that the attacker — IP address w.x.y.z — was looking for user accounts with weak passwords, and making obvious guesses at passwords to the root account. In the end, some time over the weekend, they found the account called upload with a password of 'upload' and got a shell on the server. The last command revealed the origin of the successful login:

upload pts/0 Mon Jun 27 07:39 - 07:49 (00:09) evil.example.com.ro
upload pts/1 Sun Jun 26 23:10 - 23:10 (00:00) evil.example.com.ro
upload pts/1 Sun Jun 26 23:01 - 23:09 (00:08) w.x.y.z

Oops.