I'm just a random hacker from Germany, part of Phenoelit and interested in all kinds of protocols, devices, platforms and attacks. Among these happen to be a few Cisco products, which is probably why I write this.

Did you know Michael Lynn before the Cisco stuff became public?

Michael sent me the link to the BlackHat schedule where his talk was announced and asked me if I would be in Vegas (which I wasn't). That's all.

What is your opinion on Lynn's work with exploiting IOS? Is it something really new and hot?

I didn't see his talk personally (see above). For this particular bug, he apparently found an elegant way to not guess as many pointers as I did before when doing heap exploits on IOS. He also managed to get a VTY (non-Cisco people call it [a] shell), which is pretty cool. I would say it's an evolutionary step based on previous work, which it always is, even if people claim they invented something completely new. I have been waiting years for people to pick up the IOS hacking where others and myself left it, and I'm happy it finally happened.

When did you complete your first IOS exploit?

End of 2001.

Do you think other people were already exploiting IOS before you?

There have been non-code-exec bugs before, which is always a more reliable way of getting into a device, for example the HTTP exec level bug. But I'm sure that people executed code on IOS before my work. It would be arrogant to believe I was the first to do it.

Did you find any particular tip for exploit development in the uncensored version of Lynn's slides?

Yes and no. The addresses don't get you anything with over 10000 different IOS builds out there. The idea of finding the "I'm already crashing" flag is pretty nice.

So do you think that all the Cisco pressure on removing those assembly code snippets was unmotivated?

Yes, from a technical point of view. Which NDAs/Licenses/DMCAs were broken due to the code snippets on the slides is beyond my knowledge.

I think it's funny because you had a working exploit in 2001, and nearly 4 years later someone (Lynn) got something similar. But thanks to someone (Cisco) that chose to sue him, there was a big buzz, and all the people suddenly discovered that, "wow, IOS is exploitable, yes, you can get a shell there too". Now a lot of people want to be the first to reach the goal: make public some working shellcode.

Really? They should come out and talk to me. I'm not participating in this race, so if someone wants to discuss their shellcode techniques, drop me a mail. Unfortunately, I suspect it will take more time to solve the interesting problems with the code than the hype would fuel most people's motivation. What would be needed is an equivalent of the kernel32-base shell codes on Windows, which can certainly be done on IOS, but is probably harder to do. There is no prior VX research to build on. If there would be a good way to identify the image base in [the] shell code, you could parse the ELF headers from there and may find the functions you are looking for. But I guess this goes a bit far for an interview.

Do you think that we will start seeing more advisories from independent researchers addressing Cisco devices?

I certainly hope so. I also hope that they will at least try to talk to PSIRT [link] before releasing the information. There are some good people in PSIRT who need the support from the outside, due to little support from the inside.

Looking at your slides from a previous BlackHat/DEFCON, it's clear that you already exploited Cisco devices successfully. What was your method?

It's pretty much the same thing as Michael's. Exploitation of a vulnerability to execute arbitrary code always happens in three steps: triggering the vulnerability, getting code execution and writing code to do something. Triggering the vulnerability is obviously always different and depend[s] on the bug. The strategy for getting code execution with Cisco IOS heap overflows is the same as with many other heap overflows: overwrite the inline management information and have the victim memory management work with your data. This is what Michael Lynn and myself did. The difference is, that he overwrote less data and therefore got a better (more stable) code execution. Also, as I already mentioned, his third stage (the code) was different.

What were the differences between Lynn's and your approach?

[My approach] was three years earlier, much less stable and carried code that replaced the router's configuration or, in the later versions, patched the router's running image to get command line access.

Were you able to get a working shell (VTY) too?

The later shellcodes disabled the password checking, so any password would be valid, but [I] did not export a new VTY as Lynn apparently did.

Why didn't you try to get a VTY with your shellcode too? Did you think it was impossible because of the IOS design?

There were several aspects to it. One was, that I didn't know much about the IOS socket communication at the time I wrote the first exploit. The second was, that often port 23 is filtered by the device you are trying to take over, so you need to replace the configuration anyway. I guess I considered the config replacing code to be more reliable. Come to think of it, I also concentrated on the code execution a lot more than on the code.

Did your method leave any traces on the device?

A replaced configuration on the device could be considered a trace :)

It seems that some european teenagers were likely responsible for the IOS source code theft around May 2004. Until today we haven't heard of any new exploit to get a shell, or any rootkit used against IOS. Do you think it's because nobody used the source code to develop these, or because working with the source code you could become an invisible ghost on Cisco routers?

You don't need the source to become an invisible ghost on routers. I think the code is quite large and from what I have seen posted, it's not very nice code either. So I guess nobody actually sits down and reads the code. Even when you['ve] got the code, without at least one actual device, writing exploits and backdoors is hard if not impossible. Most people simply don't have a Cisco.

Doesn't IOS include any forensic or integrity tools?

There is nothing you would call [a] forensic or integrity tool. The images are protected by several interlocking checksums, but these were reverse engineered and cracked by a fellow hacker in a matter of hours. If someone took over your router, there is little you can do in terms of forensics. If the attacker changed the configuration, you may find it in the NVRAM. For everything else, the router must still be running. Then, you can see logged in users or go as far as inspecting the memory as hex dump. But I would say that for an average CCIE it's hard to spot an attacker on a router.

Is there any method that administrators should use to verify IOS image integrity and spot backdoors?

Have the image on known-to-be-good storage and compare the SHA-1 hashes with the image you are loading to the router via network.

On your slides for BlackHat Federal 2003, I read that someone told you that he was able to place a backdoor on IOS. Can you share any info about that?

It's actually pretty easy, as I have presented at Defcon 11 in the runtime image patching code. Of course you can do the same thing with an image, recalculate and replace the already mentioned checksums and give it to someone. That has been done in the past and since few people have valid CCO accounts to download their images directly from Cisco, it's easy to convince someone to install your backdoored image. Cisco should make all IOS images freely available. This would increase their hardware sales and improve security, since people would no longer need to actually pay for SSH support. Before the recent CCO compromise, everyone used someone else's account. But now, most people don't have an account to use anymore, including myself.

Does this big number of different IOS images work as a sort of protection against worms and automatic tools?

As much as still having a number of Windows 95 and 98 machines protects from the next Windows worm. No, the different CPU architectures used and the range of platforms is the biggest thing preventing automatic tools.

Lynn said that "Cisco plans to abstract the architecture of the router operating system in the future, which could have a side effect of making a single attack work against all routers. Rather then knowing the various memory addresses, or offsets, needed to compromise systems, a single offset could work". Is this a dangerous path?

Yes and no. It will simplify exploit development for sure, but it will also enable users to actually patch IOS and not replace entire images. It will also enable them to add security features to already rolled out routers. So it's a two-edged sword. Unless Cisco is going to load a JVM on their routers, the different CPU architectures are going to stay. But it will certainly open up a number of new vectors. Think loadable kernel module backdoors for IOS.

However there still is the big problem of patching. Lynn said "How do you ship the patch when the network won't be up so you can distribute it? Are you going to mail out a CD? But there's no CD drive." Any idea?

The issue is bigger than that. Cisco['s] IOS is not patched. You REPLACE the operating system with a new one. That's like flashing your PC BIOS. And more often than not, the configuration does not work anymore with the new version, or a routing protocol is broken or your special line card is not correctly supported. And remotely updating an IOS version is not a smart idea anyway.

I read that a lot of administrators don't install IOS upgrades as soon as they are released for these reasons. Do you think this behavior is equally diffused from small ISPs to big carriers?

Big carriers usually have their own IOS images, especially built for them. I don't know if they get updates earlier or later, but I suspect they get them earlier. I think the rule is that bigger carriers install updates only if they are required (they use the vulnerable service), smaller ISPs update later and the big enterprises never. I have seen enterprise routers with IOS version 9. The big carriers actually care about security but they don't tell everyone. The network and its availability is their business and money. Ask Raven Alder about this subject.

How should Cisco users protect a particular device such as a router, since it is directly connected to the internet and you cannot put a common firewall before it?

Common sense: pretty much nobody from the Internet has a good reason to talk to your router directly (as in the packet's destination IP address is the router's interface address). There might be exceptions to this rule, BGP for example, but these are just a few. A border router does not need to expose any service to the outside world, other than forwarding packets and sending a few ICMP messages IMHO. If your router does not parse attacker provided data, it cannot choke on it.

Are routers built by other companies more secure? Or maybe they are just a smaller market, and so a less interesting target?

Most of the smaller vendors actually produce much less secure devices. Especially the SOHO market is full of real bad routers. The other big vendors score differently. Overall, Cisco is the one that received [the] most attention (and even that is very little). People told you to use Firefox since IE is insecure. Since everyone uses Firefox, lots of bugs are found. It's the same with routers. If everyone starts to use Juniper, vulnerabilities will be found there.

Would it be possible to port an open-source OS such as Linux or BSD to these devices? Do you know any active project?

There have been ports of Linux to Cisco 2500/3000/4000 devices. But I don't think that's a scalable idea. Much of the power in Cisco routers comes from specialized hardware working hand in hand with IOS, for example when parts of the routing table are offloaded to special line cards. If you want to replace your Cisco IOS with open source software, I would recommend fast PC hardware and your preferred open source operating system. I hear good things about OpenBSD's BGP implementation.

Is there anything else you would like to add? I think you could have some interesting curiosities or experiences to share; maybe something about the rumored NSA backdoor, bundled features for eavesdropping, or industrial espionage (by the Chinese).

The bundled features for eavesdropping will be interesting, since you need a lot of code and functionality to make this work. Remember, the providers are not supposed to know if the police is listening in. So, what's sold to the general public as means to protect them from terrorists is probably going to be exactly the one feature with the most holes in it. Interesting, isn't it? And building this feature on top of SNMP probably wasn't the smartest idea.