Digg this story   Add to del.icio.us  
Spam Wars Make Strange Bedfellows
Jon Lasser, 2003-03-05

The open-source community is closer than ever to curing the spam problem, but they'll have to hold their noses and help out Windows users to get there.

Ask two Internet users a question and you'll get three opinions, all of them fiery, heartfelt, and contradictory... unless the subject is unsolicited commercial e-mail. The only difference of opinion on spam will be who gets first crack at the cretins who send it out.

The situation is so bad that the engineers in the Internet Research Task Force have formed an Anti-Spam Research Group to explore solutions to the problem. When the heavy hitters of Internet architecture get together to solve a problem, you can be sure that it's serious.

Many statistics, and my own personal experience, indicate that spam is nearly half of all e-mail traffic today -- and that number is only going up. Robert Banz, technical architect at the University of Maryland, Baltimore County (UMBC), tells me that last week spam made up 43% of the university's 885,000 e-mail messages.

Finding and tagging or blocking spam wastes a lot of system administrator time: in addition to denying all SMTP access to particular well-established offenders, sysadmins must also build and test new versions of anti-spam software, and integrate these programs into existing e-mail architectures. Once you add in the time spent researching new anti-spam approaches and packages, writing documentation, and supporting users, spam can take a huge toll on a sysadmin's productivity.

Users' time, however, is more important. For most people, e-mail is a tool used to accomplish other tasks, rather than an end in itself. If spam makes e-mail too much of a hassle, they'll find other ways to accomplish their goals. If people don't use our systems, we're out of work.

The most promising new technique for sorting out spam is mathematical: the software takes note of words used in spam and non-spam e-mail, and uses these results to guess if a particular message is junk mail. If the system guesses wrong, the user corrects it, and results are factored in to future calculations.

Apple's OS X mail client and the forthcoming version 1.3 of Mozilla's mail client support statistical, or "Bayesian' filtering." Clients for statistical filters are also available for Linux or Unix workstations: ifile, bogofilter in particular.

But if your users read mail using a different client, like Microsoft Outlook, you won't find it easy to work with the statistical filters.

And that's a problem. While all of these statistical anti-spam tools are relatively easy to configure for site-wide operation, they need constant feedback in order to be even marginally effective. Every site can hack up some individual tools to provide feedback -- I've written several myself that allow feedback via special e-mail submission addresses or by saving mail to particular IMAP mailboxes. But tools like these tend to be site-specific, and are typically hacked together with Perl or shell scripts -- hardly suitable for a general release.

Missing Link
This scarcity of interoperable spam solutions is surprising and frustrating. Eric Raymond once claimed that the open-source community is good at scratching its itches -- that is, writing code to solve problems that directly affect us. This situation suggests that we might have a difficult time recognizing our itches.

When spam is half of our users' inboxes, that is our problem -- even if we, personally, have tools that let us manage it.

Moreover, Linux gained a toehold in corporate environments almost entirely on the strength of its interoperability, and that means solving the problems of clients running on other operating systems. Samba has done as much to encourage open source as Sendmail

How can we best solve the interoperability problem with regard to spam solutions? Web interfaces for Unix anti-spam tools are the first step. Many organizations using the heuristic-based SpamAssassin filtering program allow users to tweak their configurations via a Web interface.

But the real solution probably involves open-source plug-ins for commercial e-mail programs, most notably Outlook variants. Several companies have client-side ports of SpamAssassin, and several others even have ports of the technology to other server platforms. This suggests to me that the underlying technology is strong.

Why provide aid and comfort to users of proprietary, closed-source e-mail applications? For the same reason that we service these clients via SMTP, POP3, and HTTP: open source provides robust infrastructure via well-standardized protocols.

Hopefully the new IRTF Anti-Spam Working Group will look at standardizing an interface for anti-spam software. If open-source can't provide a simple, standard method for users to get rid of spam, end-users will find someone else to write software for their systems.

SecurityFocus columnist Jon Lasser is the author of Think Unix (2000, Que), an introduction to Linux and Unix for power users. Jon has been involved with Linux and Unix since 1993 and is project coordinator for Bastille Linux, a security hardening package for various Linux distributions. He is a computer security consultant in Baltimore, MD.
    Digg this story   Add to del.icio.us  
Comments Mode:
Spam Wars Make Strange Bedfellows 2003-03-06
Spam Wars Make Strange Bedfellows 2003-03-06
Anonymous (1 replies)
Spam Wars Make Strange Bedfellows 2003-03-08
Bayesian filtering for windows... 2003-03-06
Anonymous (2 replies)
Bayesian filtering for windows... 2003-03-08
Bayesian filtering for windows... 2003-03-10
Spam Wars Make Strange Bedfellows 2003-03-09
Spam Wars Make Strange Bedfellows 2003-03-09
Spammunition 2003-03-10
SpamAssassin for Outlook 2003-03-11
Jason Reusch
Spam Wars Make Strange Bedfellows 2003-03-13
Thomas Nilsen
Spam Wars Make Strange Bedfellows 2003-03-17


Privacy Statement
Copyright 2010, SecurityFocus