The Debian team is to be commended for how they handled this incident: quickly, openly, and honestly.
The method? The cracker used keylogging software to sniff the password of a user authorized to log in to one of the servers on Wednesday, 19 November 2003, then logged in and took advantage of a vulnerability in the Linux kernel to escalate to root. After that, it was a short time before the other machines were compromised as well. Further details about the exploit are available in a number of places, including Linux Today and wiggy.net.
Let's cut to the question many readers probably have: if you use Linux, should you be worried?
Well, yes and no. The vulnerability used in the privilege escalation affects all versions of the Linux kernel prior to 2.4.23 (or 2.5.69 if you're running that series of the kernel, or 2.6.0-test6 if you're using the absolute latest and greatest). And that's from all vendors, including Debian, Red Hat, Mandrake, Slackware, and SUSE. However, in order to exploit the vulnerability, the cracker first must have a local account on the machine, with shell access. In other words, the bad guys can't just force their way into any old Linux box, unless they first can login as a user onto that box.
The short answer: the chances of getting bitten are low, but you should still patch your systems and upgrade to the latest kernel as soon as its convenient. A compromised system is not a good thing.
Now, on to the bigger picture. The Debian team is to be commended for how they handled this incident: quickly, openly, and honestly. The exploit was discovered on 20 November; it was publicly announced on 21 November, a little more than 12 hours later. Too many other companies and organizations try to obfuscate and hide when they are the victims of security breaches, and this serves to benefit no one. Debian takes open source and the freedom of free software seriously - perhaps more seriously than any other Linux distro - and that philosophy has been played out publicly over the last couple of weeks. Bravo, Debian.
Further commendations should go to the various organizations and companies responsible for many of the most popular Linux distributions. In most cases, these companies are competitors, but they all worked together to analyze, fix, and advertise fixes to the problem. It's a great example of cooperative behavior working to benefit everyone, including the companies themselves and their customers.
It's not surprising that the problem started at the weakest link, the users who log in to systems. I'm not pointing the finger at the individual who had a keylogger installed on his system, as I don't have any details about how that happened. But once again, security pros should take note that little problems with end users can easily escalate into huge problems with entire organizations and their IT infrastructure.
Once someone has owned a machine, certain tasks are inevitable. Once again, the Debian team did exactly what the readers of SecurityFocus would do as well: image the drives for forensic purposes, deactivate all accounts, passwords, and SSH keys on the machines and request that users change passwords, and then wipe the machines and reinstall from scratch. By publicly detailing these steps, Debian has done a service to newbies in the security field and given them a textbook example of the necessary reactions to a severe attack.
In fact, Debian has used the break-in as what educators like to call a "teachable moment". Recommendations have been made to users and developers to investigate the use of chkrootkit, a program that checks systems for the signs of a rootkit like the one that was installed on Debian's servers. In fact, a full list of tasks for any Debian developer suspecting compromise has been published as well, and it offers excellent advice in very clean language.
Users on debian.org will need to change passwords not only on the four Debian machines, but on any computers they might have accessed from the Debian servers that required a password. This of course should remind everyone to use different passwords on each machine you log in to. Yes, I know it's a complete pain, but now it should be completely obvious why it's a necessity.
There are some more painful lessons to be learned from this incident, however. A report from Debian contains the following paragraph:
Even though this kernel bug was discovered in September by Andrew Morton and already fixed in recent pre-release kernels since October, its security implication wasn't considered that severe. Hence, no security advisories were issued by any vendor. However, after it was discovered to be used as a local root exploit the Common Vulnerabilities and Exposures project has assigned CAN-2003-0961 to this problem. It is fixed in Linux 2.4.23 which was released last weekend and in the Debian advisory DSA 403.
Clearly, the judgment on the severity was off base, and as a result, Debian paid the price. I hope that kernel maintainers will learn from Debian's break-in, and will work harder in the future to update the kernel when needed.
When bad things happen to good operating systems, it's important to fix the problems, learn from them, and move on. The problems that Debian has suffered through over the last couple of weeks will ultimately prove to be a good thing for the Linux community, and for security pros as a whole. I hope Debian never sees another major security incursion, but if it does, I'm confident that it will handle it in as professional and productive a manner as it has this time.