Alex makes a good point in "MS Windows && Powerplants". A lot of people here are saying "they should have used Unix". While Unix would have been a much better choice, it is still marginal for mission critical tasks of this degree of sensitivity. Indeed, most if not all Unices come with a disclaimer that the software is not suitable for life threatening environments like life support machines, air traffic control, or - running nuke plants!! Microsoft is worse of course, but it's not the only other choice; there do exist genuine high availability OS'es which are designed specifically for the extremely high reliability required here, and QNX is one example. Others include AS/400 and S/390. Remember, even "five nines" availability means 5 minutes per year when it's broke, and for this kind of stuff, that's at best barely adequate. It also shouldn't be running on standard hardware, but super expensive high availability hardware, ECC memory, etc. And with super super expensive rad hardened chips - of course if everything works properly it will never be exposed to elevated radiation levels, but the worst time for your monitoring system to go down would be right in the middle of an emergency!

Other people are saying "why was this connected to the Internet?" Quite rightly, but I also ask why did these boxes even have ethernet cards at all? They would be getting their sensor data through special industrial protocols designed for high noise environments and communicating to the RAID 1 through fiber channel (which gives enough length to put primary and backup in different buildings, and is radiation tolerant), and I can't think of any other reason apart from laziness you would want any sort of network connection at all. Quite apart from the security issues involved, ethernet cards are one of the biggest sources of reliability problems, probably about fourth place behind power supply, disks and fans. The latter three are all essential so you make them at least dual redundant and hot pluggable. The ethernet card is not essential in this application so just drop it in the bin.

Don't tell me you need networking to get software patches; that's not how these sorts of systems work (or at least not how they're supposed to). Your patch goes on to the staging environment first, where it's tested to the n'th degree - i.e., thousands of test cases each iterated several million times, over the course of 6 months to a year or so. After everyone from the janitor through to the Chairman of the NRC is absolutely happy to sign their name to it, the patches are burned onto CD and you walk over to the backup production box and install them. When patched backup has been happy for a few days, you swap backup and live, and install there too.

Finally, I am highly doubtful about using any kind of SQL as the database engine, never mind MS SQL (although I'm less sure of this point than the previous two). Because we are dealing with modest amounts of data, and availability is far more critical than performance, I would seriously consider just using the (journalling) file system for data storage. Before you gasp in astonishment, the reason for that is as follows: availability is a function not only of MTTF (which everyone remembers) but also MTTR (which everyone forgets). There is just no way you can keep the MTTR of *any* database down to the few seconds required in this application, but with the filesystem, you can. If you absolutley must have an actual db, I understand DB2 is currently the high availability leader. MS SQL doesn't even rate serious consideration.

I get sick of people bashing the nuke industry, or exaggerating the (very real) problems of Microsoft. But this case is, in my opinion, an example of breathtaking negligence. I hope the NRC finds some way to send them to prison.

[ reply ]

Link to this comment: http://www.securityfocus.com/comments/articles/6767/21890#21890