Story continued from Page 1
Has Windows support finally caught up in speed and reliability to the traditional UNIX version?
Fyodor: The reliability is similar, and I think the gap is narrowing in speed. When scanning across the Internet or heavily firewalled links, network latency, bandwidth, and reliability are often the gating factors, leaving you with the same performance on Windows as Linux. But for scans of fast local networks, UNIX systems have the edge. There are also several features such as IPv6 and SSL version detection that aren't yet supported under Windows. So if you have the choice, go with a Linux box or boot a LiveCD. But if all you have easily available is Windows, that should be fine. We treat bug reports in the Windows version just a seriously as any others, so please let us know if you encounter any problems.
What does the new
--badsum option do?
Fyodor: That is a clever option designed by long time Nmap user Ed3f. It tells Nmap to use an invalid TCP or UDP checksum for packets sent to target hosts. Since virtually all host IP stacks properly drop these packets, any responses received are likely coming from a firewall or IDS that didn't bother to verify the checksum. This helps you understand the target network firewall rules and policy, and saves you from spending a lot of time banging against hosts that don't actually exist. More details on this technique are available in Phrack 60.
How much did the community contribute to Nmap from its first public release to version 4.00?
Fyodor: Good question. We all know that Tenable decided to take the popular open source Nessus Security Scanner proprietary for version 3. In your recent Q&A with Ron Gula, he complained that "we had very little code submissions" and that their userbase is "mostly free users and not really code contributors." That concerned me, so I took a close look at community contributions to Nmap since 3.50 and presented the results at ShmooCon a couple weeks ago. I did find a serious problem with Nmap contributors: they are very hard to fit onto one slide. :) You can find a list of more than a hundred important contributors in the Nmap 4.0 Release Notes. These people have really made Nmap what it is today.
In fact, the passionate user community is the only reason Nmap has existed and been maintained for the last 8 years. My plan was to move on to other projects after Nmap was released in Phrack 51. But a groundswell of support and interest in the project motivated me to continue working. The earliest major contributor was Lamont Granquist, who ported the Linux-only initial release to *BSD and IRIX. Since then, hundreds of people have contributed code patches. Most are short (yet important) bug or portability fixes, but major contributions happen on occasion too. Examples are the initial Windows port (Ryan Permeh and Andy Lutomirski), the --exclude option (Mark-David McLaughlin and William McVey), and the nmap.xsl stylesheet for translating XML output to HTML (Benjamin Erb), and the IP protocol scan (Gerhard Rieger). Not all important contributors are programmers, either. Many thousands of people have helped in other ways, including bug reports, feature suggestions, document translation, assisting other users, and grass-roots Nmap evangelizing that is more effective than any marketing budget could buy.
While that help has been great, there is much more work to be done than there are resources available. More volunteers are always appreciated. Requests for help are posted to the (low volume) Nmap-hackers and (high volume) Nmap-dev mailing lists, so I encourage project supporters and users to join them. The Nmap-hackers list also includes important Nmap announcements and averages less than two messages per month.
On the Nmap-hackers list, you have proposed a new stack fingerprinting algorithm for remote OS detection. How does it work? Does it provide any new benefit?
Fyodor: New benefits? Nah, I just had nothing better to do than rewrite the whole system. :) But seriously, the second generation of OS detection will be a huge improvement. The technical details are here, but the main difference is that many more tests are included. These include congestion control (ECN bits), the selective ACK option, RST packet data, window scale values, and more. With the new system designed, the next step is code implementation. Former SoC student Zhao Lei is working on this now. Then comes the hardest task: recreating the fingerprint database. The old one has taken 7 years to develop and includes more than 1600 fingerprints. The reward for all of this work should be a more granular system that can distinguish between related systems that currently look the same to Nmap. For example, the current Nmap has a lot of trouble distinguishing between Windows 2000 and XP. The new system will also OS scan multiple hosts in parallel, while OS detection is currently handled one target at a time.
While Nmap has traditionally only done OS detection via this TCP/IP fingerprinting mechanism, version 4 can additionally guess the OS based on application heuristics. After all, many applications only run on one OS family and others happily report the platform they are running on. Version detection (-sV) now populates up to five fields when it determines their proper values: service name, vendor and application name, version number, device type, and hostname. There is also a miscellaneous field for KaZaA usernames, Apache module strings, etc.
Would you like to add some technical details?
Fyodor: Sure. Most of the tests utilize TCP/IP features that weren't invented or weren't commonly implemented when Nmap OS detection was released in 1998. There are too many to document in an interview, but here is a representative sampling:
- Congestion Control: In January 1999, RFC 2481 was released. This converts two formerly reserved bits in the TCP header into two new flags: ECN-Echo (ECE) and Congestion Window Reduced (CWR). This test sends a SYN packet to an open port on the target with both ECE and CWR flags set. It then records whether they are set in the SYN/ACK response. If the target system is ECN capable, it will respond with the ECE bit set and the CWR flag unset.
- Selective Acknowledgments: Selective acknowledgment (RFC 2018) improves TCP performance by allowing receivers to acknowledge all segments that have arrived successfully, even when there are gaps due to missing or late segments. Nmap's only concern is to determine whether this feature is supported or not. Nmap will send a SYN packet to the target with the new SACK-permitted option. If the response SYN/ACK includes the SACK option, Nmap knows the system supports this feature.
- RST packet data: One thing I have noticed by spending too much time in front of Tcpdump and Ethereal is that some systems send descriptive data content in their TCP RST packets, while most systems send just the required header. Examples are HP-UX 11 sending "RST tcp_close, during connect", Cisco sending "RST cisco". MacOS X has a habit of doing this too. It doesn't violate the protocol at all, but can make identification a cinch. The new system will include a 2-byte hash of any data received in RST packets.
- Window scale value: The one is an oversight in the original Nmap OS detection system. It notes when the window scaling TCP option is specified by the remote host, but does not examine the actual window scaling value. The new system rectifies this.