This is interesting work, but trying to fit a logistic curve to the data essentially assumes that the interval between finding each flaw is normally distributed. That seems reasonable enough for a 0-th order approximation but there are a host of reasons why it could be a long way out. This is easily seen when you look at the fitted graphs; Windows 9x plots fit surprisingly well, unitl recently when they seem to diverge a bit; while all the Linux fits are actually pretty lousy.

Thus it is hardly surprising that, despite the very generous margins of errors, some of the predictions have already been invalidated within a few months of publication. I wouldn't bet my house on any of the others standing.

I also ahave a few other minor criticisms. For example, the paper actually deals mainly with defect _density_, or defects per unit code size. Code size is measured by the "industry standard" unit of a thousand lines of code. This may indeed be a common practice but it is also a practice that is widely criticised because it is well known to be rather inaccurate for hand-coded apps and total nonsense for generated code.

They also allow for arbitrary corrections, such as allowing that Windows NT will have more known defects than Windows 95 because "as a server, it must have gone through more thorough testing". Erm -- we are only counting defects discovered _after_ release, so more thorough testing should have produced _fewer_!!

[ reply ]

Link to this comment: http://www.securityfocus.com/comments/articles/11399/33950#33950