PART 2: Thomas Ptacek and Nate Lawson.

Could you introduce yourself?

Thomas Ptacek: I'm a researcher at Matasano Security, a company I co-founded. I've been a published security researcher for over 10 years. My partners in this project are even more experienced: Nate Lawson found his first SunOS vulnerability in the early '90s, and designed the still-unbroken BD+ scheme that protects Blu-Ray disks. Peter Ferrie is a well known antivirus researcher, also with over 10 years of experience. It would be fair to say that Peter's work motivated us to start this project.

Nate Lawson: I am principal at Root Labs. I am currently assisting companies with design and evaluation of solutions involving embedded and kernel security and cryptography. A typical project might be to evaluate and improve a design for an authenticated boot ROM. I'm a FreeBSD committer in my spare time, maintaining the power management (i.e. ACPI) layer.

I'm most interested in designs that provide renewable security, as Tom mentions. The Blu-ray system I co-designed with others at Cryptography Research provides for disc-specific protection code that must be circumvented to descramble a movie.

Did you see the talk given by Joanna at Blackhat? Your reaction?

Thomas Ptacek: All three of us saw Joanna's talk, and I think I speak for the three of us when I say that she did a good job under tough circumstances. Joanna had just hours to respond to the points we raised in our presentation, and had clever answers to many of them. I'm hopeful that if you went to Black Hat and saw both talks, you watched an interesting debate and learned a lot about virtualization.

Researchers at Stanford, VMWare, Root Labs, Symantec, Matasano, and Joanna's own former employer COSEINC have all published methods to detect unexpected virtualization. Joanna has rebutted none of them. What she has done is point out the inevitable bugs that occur when code is ported from Intel to AMD chips. She has then fixed those bugs, and proved on stage that they do in fact work.

One thing the industry has learned from the drama at Black Hat is that it is undisputed that software can detect unexpected virtualization.

Regarding "sleep mode". There is no such thing as a "sleep mode" that makes code invisible to a running Microsoft Vista kernel. When Blue Pill "chickens out" and dives out of the hypervisor and back into the kernel, it must leave a backdoor to allow itself back in.

But that begs the question. We asked, "Joanna, if your strategy for dealing with virtualization detectors is to hide in the kernel, why bother virtualizing at all? Obviously you believe your code is more stealthy when it's "chickening out" in the kernel." Joanna responds, "Blue Pill has a really clever way of hiding in the kernel that is hard to detect". We agree. What we're asking is, why bother with virtualization at all?

Finally: the entire history of antivirus and malware detection is one of "tricky hacks". There is no rule book that says defenders must be clever, only that we must win. Sorry, Joanna: using "exotic timers" and "TLB sizing" to detect unexpected virtualization is not what we'd call a "pretty win". But we'll take that win anyways.

One of Joanna's comment was that you can just detect virtualization, not Blue Pill itself. What is your point of view?

Thomas Ptacek: Our research shows that if or when systems ever need to defend against malicious hypervisors, they can do so by searching out signs of unexpected virtualization and unauthorized hypervisors. Joanna is right to say that over the long term, we won't continue to be able to detect Blue Pill "by name", but wrong to suggest that we won't be able to spot unauthorized hypervisors.

Joanna's research over the last year, as well as our research over the last several months, and research from projects at Stanford and Google, all show that it is extraordinarily hard to hide a malicious hypervisor --- probably even harder than hiding conventional malware in the kernel. Joanna has improved Blue Pill in interesting ways, and I'm a fan of her work. But she has not shown that it is feasible to hide a hypervisor in a modern X86 system.

What makes you believe that we can't build a 100% invisible rootkit?

Thomas Ptacek: It's hard to build 100% undetectable rootkits because of a basic dilemma.

A rootkit has two goals: first, to serve the bidding of its master (usually by providing a backdoor entrance to a compromised computer), and second, to hide itself. But anywhere you try to a hide a backdoor will leave fingerprints. So rootkit authors and rootkit detectors have been in an arms race over the last ten years, each racing to get closer and closer to the OS kernel and the hardware so they can reprogram it to foil the opposing side.

The problem is, rootkits and detectors aren't on a level playing field. The OS and the hardware is on the detector's side. The closer a rootkit gets to the hardware (say, by virtualizing the whole machine to intercept detectors), the more work it has to do to keep the hardware from betraying it. We call this "cross-section", and if you're a rootkit author, you want to minimize it. Virtualized malware has a much bigger cross-section than conventional malware, because it gets between the running OS and the underlying hardware.

Eventually, the amount of work required to perfectly emulate an entire desktop computer, with all its under-documented exotic chipset features and hardware bugs, becomes intractable for rootkit authors: hard-to-detect rootkits get too expensive to write, and too slow to deploy stealthily.

Now, we're not saying we can easily detect all rootkits today. In the real world, rootkits are a huge pain for security teams to detect. We're just saying, we think we know how this game will end, and it isn't with virtualized rootkits foiling all possible detectors.

What detection techniques did you present at Blackhat?

Thomas Ptacek: Our team had been researching virtualized rootkit detection for over 9 months prior to Black Hat. In that time, working both together and independently, we conceived of several methods of detecting unauthorized hypervisors. We broke our methods down into three categories:

1. Side-channel attacks, an idea borrowed from cryptography, which detect the resources (time and memory) that a hypervisor steals from a virtual machine when it is forced to intercept operations the virtual machine requests from the underlying hardware.
2. Vantage point attacks, which detect important system functions (like exotic timers) that a hypervisor has failed to virtualize at all.
3. Hypervisor bugs, which detect coding mistakes made in a hypervisor.

Nate Lawson: Just to note, the actual implementation of one method from each of those 3 categories took about 2 weeks total of Tom and my spare time. Our key finding is that it will always be easier to detect a (hypervisor) rootkit than it is to write perfect cloaking code for one.

When you have a choice, it's always better to be on the side where software bugs benefit your goals. Our code is minimal and is less than 1000 LoC while New Blue Pill is about 7000 LoC. Adding support for hiding from our particular set of checks would increase the size of NBP even more. More code would mean more of a target for us to look for in the future. That's why we were confident in issuing the challenge.

Is there anything that hardware (not only cpus) manufacturers could do to improve security, or at least reduce the impact of virtualized malware?

Thomas Ptacek: I don't think virtualized malware is a hardware problem, any more than I think kernel malware is a hardware problem with protected memory. So far, the real-world impact of virtualized malware has been nil.

What about virtualization features included in modern cpus?

Thomas Ptacek: The reason so much attention is being paid to on-chip virtualization in X86 CPUs is that it's novel, and so it's fun to talk about. But the only reason it's a viable target for rootkit authors is that nobody is using it yet. Five years from now, everyone's desktop operating system will be virtualized by default; rootkits won't have any opportunity to load themselves into hypervisors directly, because there will already be a hypervisor present, and it won't want to share.

What's funny is, you could have made the same argument about protected memory, the chip feature that allows us to safely run a web server and a web browser on the same machine at the same time. If your operating system didn't configure and use the CPU's protected memory features, attackers would do it instead. But of course, no mainstream OS ignores protected memory, and soon no mainstream OS will ignore virtualization.

From a security standpoint, do you see any difference between AMD and Intel cpus virtualization features?

Thomas Ptacek: There are differences between AMD and Intel hardware, and those differences affect the way rootkit authors and rootkit detectors have to write their code. You can also make the argument that some versions of AMD's hardware are faster at virtualization today than other versions of Intel; tomorrow, the opposite will be true. But none of this matters to the debate; as Joanna demonstrated in her talk, with a little tweaking, the ideas our team came up with work just as well on AMD as on Intel.

Do you see a different scenario on other architectures that support virtualization?

Thomas Ptacek: Virtualization on exotic processors is much more closely tied to the hardware vendor's own software offerings (for instance, the PowerPC's virtualization was designed for the LPAR functionality in mainframe operating systems). People don't buy Sun SPARC hardware to run Linux on it; they buy it to run Solaris. These systems often already have hypervisor code running, and aren't good targets for virtualized malware.

What is your opinion on the security of software virtualization solutions (VMWare, Xen, VirtualPC, ...)?

Thomas Ptacek: I can't speak for the three of us on this: if you asked us individually I think you'd get three different answers. What I can say is that interesting research is being done into this; one of the best recent papers is from Tavis Ormandy at Google [PDF].

Tavis found a variety of ways that code running inside of a "guest" virtual machine could potentially disrupt or even hijack the host operating system.

Since ISPs are selling so-called VPS (virtual private servers) running on different types of virtualization software, I was wondering if you had any advice about choosing one.

Thomas Ptacek: While new vulnerabilities will always continue to be revealed in virtualization software, I'm confident and optimistic about virtualization. If you're in the market for a "virtual server", you are better off with a Xen or VMWare virtual machine than you are with a classical virtual host, where you share a single running operating system with many other customers.

I don't have any recommendation to make between VMWare and Xen; both are high-quality offerings. As both mature, they look more and more like each other, despite Xen's original distinction of using "paravirtualization".

Two things I would keep in mind if I was running sensitive applications on a virtual server:

1. Researchers will continue to find "guest hopping" vulnerabilities that allow code in one guest machine to hijack other guests, or the hypervisor itself. So far, these findings have been rare, but they do happen. If your hosting provider is vulnerable to one of these attacks, an attacker can buy a cheap account with them and hijack your server.
2. Many of the techniques we used to detect Blue Pill were inspired by techniques that cryptanalysts use to steal cryptographic secrets. On a virtual server, attackers can use these techniques not only to figure out what kind of hypervisor is running, but also potentially to steal cryptographic keys. Be cautious about deploying cryptographic software on virtual hosts.

I think readers might be interested in possible or future use of virtualization for defensive aims, such as using "blue pill" for invisible intrusion detection or antivirus processes.

Thomas Ptacek: Secure virtual machines are a key part of Intel's security strategy. Right now, security software has to trust the running operating system. If something goes wrong with the Windows kernel, security software isn't protected. In the future, as Intel sees it, security software will run above the operating system; even if you lose the kernel, your security code is still protected by a virtual machine boundary.

I think everyone would object to calling those "Blue Pills"; secure virtual machines predate Blue Pill, and are not themselves rootkits.