After more than eight years since its first release in Phrack magazine, Fyodor has announced Nmap 4.00. Curious as usual, Federico Biancuzzi interviewed Fyodor on behalf of SecurityFocus to discuss the new port scanning engine, version detection improvements, and the new stack fingerprinting algorithm under work by the community.
Could you introduce yourself?
Fyodor: I'm a long-time network security enthusiast with a particular interest in full disclosure and the offensive side of security. I have gained a lot from the security community over the years, and try to contribute back by releasing free tools such as my Nmap Security Scanner and publishing useful content on my websites, Insecure.Org and Seclists.Org. I am also an active member of the Honeynet Project. Writing has been a major recent focus of mine. Last year I co-authored a technical security novel named Stealing the Network: How to Own a Continent, and I'm almost finished with a network scanning book. This is all on top of my active and varied social life. OK, I'm just kidding about that last part. :)
You just released Nmap 4.00 after two years of work since 3.50. What are the most exciting changes?
Fyodor: Well, the Changelog shows more than 230 improvements since that release, so it is hard to choose just a few favorites. But some really do stand out. The port scanning engine has been rewritten to be much faster and (after the "diet Nmap" project) more memory efficient. The low-level packet sending subsystem has changed dramatically as well. Nmap can now send and route raw Ethernet frames rather than rely on the host's raw sockets implementation. This is critical for Windows, since Microsoft disabled raw sockets as of Windows XP SP2. And all platforms benefit from the new ARP scanning and MAC address spoofing functionality that this change allows.
Nmap 4.0 has new, better organized and more comprehensive documentation, including a rewritten man page available in seven languages. Huge improvements have also been made in version detection, which offers many new features and saw its signature database triple in size.
Many Nmap users pick runtime interaction as their favorite new feature. If you find yourself staring at the screen wondering when Nmap will finish, just press [enter] for an estimate. If you forgot to enable verbose mode, press 'v' to enable it. Or press 'V' to turn it off. Packet tracing and debugging can be enabled or disabled on a whim as well.
Rewriting the port scanning engine produced a much faster version, and thanks to the "diet Nmap" project it reduced memory consumption too. How did you modify the code to get these advantages? And what type of improvements can we expect?
Fyodor: There are dozens of new algorithms and techniques in the new engine. The most important is proably scanning multiple hosts at once (in addition to scanning multiple ports on each host concurrently). Other important performance changes include parallel DNS queries, "port scan pings" that allows Nmap to detect network reliability and performance even against heavily firewalled networks.
The best way to see the performance difference is to test Nmap 4.00 on your own network. All sorts of network characteristics play into this, so blanket generalizations like "it is twice as fast" don't hold water. And even external benchmarks may not be relevant to your organization. But I know people want to see numbers, so I just did a test scan of all 65K ports on www.insecure.org, seclists.org, and scanme.nmap.org. I added the -T4 option for aggressive timing. The tests were done from my home DSL line, which is limited to 128kbps upsream. The scanning machine is running 64-bit Linux (AMD Athlon64 processor), which uses significantly more RAM than normal 32-bit machines do. The memory consumption is based on the maximum 'VIRT' value displayed by the top command, which includes Nmap code, data, and shared library usage. I tested 3.50, 3.93 (which has the new engine but preceeds the "diet Nmap" efforts), and 4.00. Also note that these are heavily firewalled machines, which makes scanning them relatively slow. With all that out of the way, here are the results:
|Nmap Version||Execution Time||Maximum memory consumption|
As you can see, upgrading makes a huge difference. In some situations, the speed increase may be greater, in other cases worse. But Nmap 4.00 should never take longer than 3.50. If you find a case where it does, let me know and I'll try to fix it.
How does the new ARP scan work?
Fyodor: Well, Nmap has traditionally performed host discovery by spewing IP packets throughout the target networks and listening for replies. Obviously many firewalls block ICMP ping packets and other packet types. Nmap gets around this by sending a user-specified combination of IP packets in the hope that at least one will get through and elicit a response. Nmap can send TCP SYN or ACK packets to a list of ports, UDP packets to arbitrary ports, and ICMP netmask and timestamp request queries.
But on local Ethernet networks, which is one of the most common Nmap usage scenarios, there is a better solution. When Nmap tries to send a raw IP packet such as an ICMP echo request, the operating system must determine the destination hardware (MAC) address corresponding to the target IP so that it can properly address the Ethernet frame. This is often slow and problematic, since operating systems weren't written with the expectation that they would need to do millions of ARP requests against unavailable hosts in a short time period. Nmap now takes over this role. And when it receives an ARP response back, it knows the host is up so it doesn't even need to bother sending the IP packets. The net effect is that scanning your local network is now much faster and more accurate. Nmap automatically detects when conditions are proper for doing this, so you don't even need to remember an extra flag.
Can you tell us more about the version detection improvements?
Fyodor: I'd love to, as that is an area which has improved greatly. I also believe it is under-appreciated because many old time Nmap users still don't realize it exists. Version detection allows you to actually determine what service protocol (and often application name and version number) is running on an open port, rather than simply guessing based on the port number.
The biggest change is that the database has tripled in size to
3,153 signatures for 381 service protocols. We have most of the
common protocols (http, ftp, smtp, etc.) covered, as well as hundreds
of obscure ones from abc, acap, afp, and afs to zebedee, zebra, and
zenimaging. There is also a new
--version-intensity option which
lets you specify how hard Nmap tries to interrogate the port. A low
number leads to a faster scan, while higher numbers are a bit more
likely to identify the service.
Another new feature is the exclude directive. One problem people
occasionally reported with version detection is that it would make
their printers spew out page after page of Nmap service probes,
including binary DNS and X11 requests that came out as gibberish. It
turns out that many printers listen on port 9100 and simply print out
whatever garbage is sent there. This is a dumb (printer) feature, but
the new exclude directive works around this by making Nmap skip that
port (only for version detection purposes) by default. You can
override that with the new
Two more new features are the fallback directive and detailed softmatches. These make version detection faster and more accurate, without requires to even care about the implementation details. But readers who are curious anyway can learn more from our new version detection paper. Many of these version detection changes were done by Doug Hoyte, Dirk Mueller, Lionel Cons, Martin Macok, and Bo Jiang. I would also like to thank the literally thousands of users who submitted service fingerprints to the CGI that Nmap provides when it fails to identify a responsive service.