Jose Nazario discusses worms
Federico Biancuzzi,

Federico Biancuzzi interviews Jose Nazario to discuss modern computer worms and the design goals behind them.

Could you introduce yourself?

I'm Jose Nazario. You may know me from endeavors such as Wormblog and InfosecDaily, two sites I created to help myself and others keep abreast of information security knowledge. Wormblog is a project I started in the spring of 2004 as a way to post paper abstracts and events in the worm landscape. InfosecDaily was a system I set up for myself to track blogs, news, and notices in information security. There's hundreds of sites to track, and I wanted a way to look at it all in one glance, so I set up an automatic news creation system. The sites are public, and anyone is free to look at them. Infosecdaily, together with Ejovi Nuwere, has recently undergone a nice transformation to help manage the news flow.

I currently work for Arbor Networks, a network security solutions provider. We help secure enterprises and service providers from a variety of threats, including worms and DDoS attacks. Prior to joining Arbor I contributed to and worked on a few open source projects, including LinuxPPC (many years ago), libnet and OpenBSD, and completed a Ph.D. in biochemistry at Case Western Reserve University where I studied enzyme kinetics and mechanisms.

While studying for my Ph.D., I began to get interested in high performance computing, specifically for large scale dynamics calculations. This required me to learn a lot more about computing than I knew previously (I had always dabbled), and I began to enjoy the computing circles a lot more than the biochemistry ones. After a security incident on a server I ran, I dove head first into that speciality. Along about 1999 or 2000, I participated in a number of forensic investigations and incident response activities. One of the things we found was that a DDoS toolkit was being distributed using a simple, worm like program. Between this and the ADMw0rm, I saw a trend emerging. We looked at some of the evidence in front of us and made some extrapolations and eventually wrote them up as "The Future of Internet Worms", which I presented at the BlackHat briefings in 2001. What's funny is that in the past couple of years we're starting to see these ideas come to fruition with systems like the RBot and NeseBot families.

Worms were particularily fascinating to me because of my studies in enzyme kinetics, or their rates of catalysis. Self propagating network worms are somewhat similar to biological populations, so the concepts and math were familiar to me. It was a natural fit, and I came along at the right time.

Is there any way to take advantage of the worm's speed of spread and use it for something good?

In theory, sure. In reality, I haven't seen it.

When a worm hits, suddenly you're faced with a challenge: an adversary that can be moving very quickly and launching from an increasing number of points. A problem that was once in only one place is suddenly everywhere. It's a challenge to stop the spread of worm. Bruce Ediger has explored this in theory, and he's a big proponent of the idea.

It's tempting to think about fighting fire with fire when a worm hits -- launching a counterworm to stop the worm. The most natural thing to do is to deliver a counterworm with a payload that contains the patch for the security vulnerability exploited by the worm, which would prevent its spread.

However, remember the following things. Even if you knew instantly what vulnerabilities the worm was exploiting and how to prevent its use of that hole, how would you prepare a worm with the patch payload in time to launch it in a meaningful time period? How would you outpace the worm (in about 6 hours, Blaster had reached it's peak propagation speed; SQLSlammer reached that speed in a matter of a few minutes; Witty hit that point in a matter of minutes, too)?

A number of reasonable ethical concerns also come to mind. How would you prevent the worm from propagating beyond your responsibility's border? How would you ensure that the patch really worked and stopped the worm? How would you shut down the worm?

None of these real world concerns are satisfied, in my experience, with counterworms. Too many problems crop up and they're simply never effective at really slowing down the worm. The Nachi or Welchia worm didn't stop the Blaster worm from propagating too quickly.

Bear in mind that you can beat the worm to the punch. You own the network, you own the hosts. You have legitimate, immediate access everywhere. Once you detect the worm, you can employ a number of specific actions on your network to stop the worm's spread, through firewall rules, router ACLs or even VLAN ACLs (VACLs). You can also use agents on the hosts to deploy patches, shut off services, or quarantine or vaccinate the hosts immediately.

This approach remediates the concerns I have about legitimate access, breaking software installations, shutting down the action immediately and controlling the effect.

A lot of people don't install patches, even if they fix security bugs. That is why a big company such as Microsoft is trying to make the process automatic and obligatory. Should Microsoft release patching worms to fix every vulnerable system on the Internet in a matter of minutes?

There's no need. With Windows update tools installed and automaticaly enabled on the end host, Microsoft (or any big software vendor) has that access. AV vendors have been doing this for years, both Apple and Microsoft do this at the OS level, and so on. Some people even have their IDS signatures update automatically.

When you have an agent-based system like that deployed, there's no need to deploy self propagating code (like a worm) to affect change everywhere. You can achieve the same result more quickly with agents. Instead of waiting for the worm to find all hosts, you can hit every host immediately.

I'm not sure it is the same thing. First of all because of bandwidth. When you have 100 (or maybe 500?) million Windows users that want to install a 200MB service pack, how much bandwidth should you have? And how many hours (days? weeks?) do you need to patch them all? A patching-worm could take advantage of a user's bandwidth.

Actually patch distribution is pretty efficient by now. Both Microsoft and Apple, and many of the big AV vendors, use geographically and topologically dispersed centers to distribute patches. So, it's not nearly as crushing as you may think it is from a quick "back of the envelope" calculation.

I don't see how a patching worm would be any more efficient, because bandwidth will be wasted by the worm as it tries to acquire hosts and spread. Like worms themselves, a patching worm would suffer a similar fate.

There's also the issue of time. Downloading a 200MB file means being online and vulnerable for minutes (or hours). What about an attack or a worm in this timeframe?

An efficient patch can be distributed in a matter of a hours to days. With only one exception (the Witty worm), no worm has ever been constructed and deployed that fast. The time frame between a worm's release and the disclosure of the vulnerability that the worm uses is, on average, about 4 weeks.

If Microsoft wants to make patching obligatory, what is the difference between a program running on your computer (the Update agent) and an external patching agent?

As a security administrator, you have more control over an update agent than you do an external agent. It's more aware of the host's configuration and possible conflicts because it can directly query the system's status and applications.

An agent on the host is more succeptible to subversion (i.e., falsely stating a patch or some innoculant has been installed when it hasn't) than an external agent, so I'm not advocating that it be one over the other. However, if you had only one to do I'd go with the agent on the system. While it can be a management nightmare (ie maybe tens of thousands of agents in a large enterprise deployment), it's much safer in the long run than an external agent or a patching worm.

Have you ever seen a worm that closes the hole used to compromise the system? This strategy would stop a patching-worm...

That doesn't matter, because you can find another vector to propagate the 'patch distribution worm' if you're taking that approach. So, even if the worm patches the vulnerability behind it and installs malicious software for the worm or bot author's use (as opposed to your benign patching worm which doesn't leave any unwanted access behind), you can get your patching worm to propagate.

Several early worms patched the system behind them, which had the effect of stopping other attackers from using the same hole to get in. There were still numerous security holes which allowed for other attacks to succeed, though.

However, I'm still dead set against the idea or use of a counterworm for all of the reasons I have outlined above.

Windows XP-SP2 introduced the support for the famous NX bit included in modern CPUs. Here what the vendors say:

Intel's Execute Disable Bit "can help to prevent some classes of viruses and worms that exploit buffer overrun vulnerabilities thus helping to improve the overall security of the system."

AMD's Enhanced Virus Protection "acts as a preventative measure causing the virus to be localized, short-lived, and non-contagious, eventually being flushed from system memory."

I think the NX bit can help in preventing some bugs from being exploited, but it cannot do anything against worms. The point is that a human being exploiting a vulnerability could play with the exploit and the target until he succeeds. A worm is different. Generally it doesn't auto-modify the exploit to infect the target.

So, do you think that software companies should focus on randomizing addresses and offsets instead, like OpenBSD's StackGap does?

I'm a firm proponent of layered security, and this includes low level system protection measures like the NX bit, randomized stack gaps, or the like. Some of them work very well to defeat simple attacks, and in concert they work together to defeat more complicated attackers.

While I like tools like XP SP2's nonexecutable stack, they come at price at times. Some software broke with the SP2 changes, and this is one of the reasons why people were slow to deploy it. Clearly software is a very complicated system, and so all of these interactions can't be anticipated. Some of them will be negative when you change a system, even slightly. However, I think that this approach (nonexecutable stacks) is one of the key pieces of defeating large classes of attacks.

There are a number of attacks that the nonexecutable stack leaves untouched, including format string attacks. Employing various defenses, including randomized memory layouts and gaps between items in the address space, you can start to defeat these sorts of attacks, too.

Now, bear in mind that we've seen worms brute force the offset for a buffer overflow attack in the past, so it can be done. It's slow, but it's effective. If a worm has either a small enough address space to test or a way to divine the secrets used to randomize the stack (i.e., some form of PRNG seed leak, like the remote clock), then it can brute force the offsets and propagate that way.

This can make some debugging and diagnostic tasks more difficult, so you have to make sure everything tolerates that. When you have many, many millions of lines of code and thousands of applications that people and businesses rely upon, you can't make that change lightly. OpenBSD made the changes in a controlled fashion, but they had to do it over the course of a year. It took a focused effort, and that's a much more tightly controlled system than something like Windows.

One of the neat things that was developed a few years ago at HP labs in the UK was the concept of a virus throttle, or a rate limiting enforcement for your network connections. While it broke nmap on Windows XP SP2, it prevents worms from scanning or even from trying targetted connections too aggressively. Worms can still propagate, but they do so much more slowly. Again, another security layer makes a big difference, but not it's not a total solution.

These mechanisms will stop a broad class of worms which use exploits to execute arbitrary code. But they won't stop things like password guessing attacks or trojan horses. Signed code execution helps there, but that relies on the person making the judgement to be aware of all of the ramifications of the choice, and to have the information presented accurately. This doesn't always happen, so with an avenue like this, attackers and worms will still have an avenue to propagate.

I'd like to see more of this wide-scale deployment of security protection mechanisms that have proven themselves in research and smaller scale installations like Linux or BSD. I'm happy to see Microsoft making these changes to Windows, and I expect them to continue. They've also done a decent first pass at enforcing some basic security mechanisms for end hosts with firewalls enabled initially (though the configuration isn't optimal), virus protection being demanded immediately and automatic Windows update enabled as well. These approaches combine to improve the security posture of the average end host, and I'd like to see all vendors take similar approaches.

How would you build a network infrastructure that can handle a worm infection without collapsing?

Any network which allows for a sufficient amount of specific control can work. This can include managed switches that allow for IP layer or VLAN level access control lists (ACLs), agents that disable the port on some malicious software event (like a buffer overflow being detected), or the like. You can even do this with sub-unit firewall appliances everywhere. The key is to have the same level of access control options that you have at the perimeter close to the end host. The key to remember is that the malicious events don't just come from the outside world; worms are one way that threat are brought inside.

It's a lot like a submarine and bulkheads. You have a hatch to the outside world that keeps water out, but you also have a series of doors inside that can contain any leaks to a single compartment. You want to contain the worm's traffic, and as specifically as possible, when a worm outbreak occurs.

Which type of disclosure should security researchers adopt to reduce the risk of spread of worms?

I don't know if full disclosure, responsible disclose, or irresponsible disclosure will make much of a difference. Interested attackers will always find a way, even if they have to use holes they find and exploit themselves. Since I began working for a commercial software vendor and I've been working with many commercial vendors, my views changed significantly and I was reminded what I was interested in in the first place: better software, and better security. If I don't get to disclose the vulnerability, I'm ok with that.

That being said, I don't advocate anything but working with the vendor if what you're interested in is more secure software. But I don't know if we stopped disclosing vulnerabilities that worms would go away. Full disclosure helps everyone in the end, I think, including the attackers, but it helps a whole host of people learn and fix bugs, even if they do so quietly. That's a benefit I can live with, and those are risks I accept.

One of the funny things we found in our [PDF] wormability research at Arbor Networks is that it appears that we have three different actors in the vulnerability to worm cycle in many cases: a vulnerability discoverer, an exploit author, and a worm author. The worm author uses the exploits written by others to propagate their worms. As long as they can do so, and as long as worm creators have the motivation to do so by spreading spamming wares, phishing site kits, spyware, and other forms of crimeware, this can keep being effective. But this doesn't mean I think we should shut down exploit authors, either, just because their product can be used for ill.

In the end I think that threats like worms will always be around, no matter what kind of knowledge we try and restrict. Instead, network operators, security admins and system admins will simply have to apply decent security practices, which have been shown time and time again to be effective at keeping malware out. Leading organizations do this quite well, so it's been shown that it can be done on a small scale and a large scale.

Do you think that the first reason for a worm to be spotted is the insane amount of traffic it generates?

In some cases yes, for instance SQL Slammer. In many cases, though, it's just the cascading changes and alerts. A growing number of hosts begin acting in the same way: scanning, attacking, and in general causing some havoc. Even if it's not very aggressive it's noticeable.

It's usually not the flood of traffic but the change in trends.

What algorithms do recent worms use to find targets?

Most worms rely on a handful of methods. Most network worms operate by generating a random IP address and scanning from that. Many recent worms use "island hopping" techniques where they generate addresses in the local (to the worm) /16, /8 and a random address to scan from. Some worms (like SQLslammer and Witty) just try random addresses.

By random I mean a psuedorandom number generator (PRNG), often seeded by the local system's uptime, process id, and some other stuff put through a poorly constructed random number generator. It's far from random (see [PDF] Toward Understanding Distributed Blackhole Placement), but is generally sufficient for most worm outbreaks.

Many application layer worms, like IM-based worms and mass mailers, generate their target list often by just going through your contact list. This helps them focus their efforts.

A few network worms have tried to use ARIN allocations to focus on populated networks, but many semi-random worms haven't had to do this and have spread successfully. There isn't a whole lot of creativity or rigor of thought when it comes to these algorithms, though they simply work.

Some propagation strategies like hitlists have been hypothesized and some researchers have claimed to have seen these in actions.

Is there any type of global coordination to avoid the same target, and in doing so ending up worms that are more stealthy?

In an effort to avoid hitting the same hosts or to focus on local networks and hosts, a few strategies have been proposed: using hitlists, hashing, and distributed hash tables, among others. These usually aren't needed, though, or even wanted. One of the success facets about network worms is that they keep trying the same hosts from various vantage points, which can let them slip past firewalls and other security mechanisms.

Stealth is very difficult to achieve with a worm, because of the way that it propagates and causes all of the victims to begin launching new attacks. However, a few simple things can assist the worm with its stealth. Worms like RBot and NeseBot got very far before they were analyzed due to their relatively slow speed (compared to worms like Nimda and Blaster), and the variety of attack mechanisms they use. It's hard to identify that a worm is at work when the cause isn't fully understood. By the time this family of worms was found it was installed on thousands of hosts.

Some worms are launched to create a network of zombies to be used as attack launchpads, or as spam senders. Since they often address only a specific IP range used by broadband users, they don't get press coverage and the average user doesn't run home to install patches. Then, a couple of times every year, we hear that a big worm hit the Internet, and everyone becomes a "security expert." Beyond what is covered by the big media, do you think that there is an underground life on the Internet made of stealthy worms that hit a small set of targets and... stay alive?

Indeed. We've seen a whole host of parties adopt the worm as a malware distribution platform, and this means that we have hundreds of authors competing with micro distributions of their worm. The populations can range from a few dozen machines to tens of thousands, most of which stay below the press' surface.


Privacy Statement
Copyright 2006, SecurityFocus