Recently, a good friend of mine forwarded me an article from kerneltrap.org, which talked about a new heap implementation that's being introduced into an upcoming release of the OpenBSD operating system. This article was of specific interest to me, as I have been experimenting with the creation of a more secure heap implementation myself.
According to the article, this new heap implementation looks to be a candidate for release in OpenBSD 3.8, which will be their next official release. This is great news, and following along with other OpenBSD advances, promises to end up in widespread use due to the fact that (most of) it will be enabled by default.
More about heaps
For those of you who don't have a programming background, let me attempt a brief and non-technical explanation of what a heap is. Basically, it's just a management facility for most of the memory that applications use. When an application (such as Mozilla, your command shell, word processor, and so on) needs some memory, it will typically take this memory from the heap. The heap is responsible for handling these requests, which involve handing out chunks of memory when requested, and freeing the chunks when the application is finished with them.
For a more verbose (and perhaps less butchered from a technical perspective) description of a heap, check out this page on the wikipedia.
A problem can arise when programs don't make correct use of the memory that they receive from the heap - if they write before or past the chunks that they're given, they can corrupt information that the heap implementation uses to manage the chunks, and this can result in code execution exploits by fooling the heap into doing things by carefully crafting this management information. Secure heaps attempt to prevent this from happening, and catch this misuse of the heap before it turns into something more nasty.
As far as the current heap landscape goes, if you're running Linux, you're probably running "ptmalloc2," which is based off of "dlmalloc." On the other hand, if you're running one of the BSDs, you're probably running "phkmalloc," named after its author, Poul-Henning Kamp.
In practice, phkmalloc has been a little bit more difficult to exploit, though I don't think that this was so much an intentional feature as much as a side effect of the algorithm. It will suffice to say that the two algorithms are distinct and different.
Although phkmalloc has remained relatively unchanged over the years, the glibc implementation (ptmalloc2) has undergone several recent changes (a friend of mine confirms that version 2.3.4 contains numerous security enhancements), increasing the difficulty of exploiting heap corruption vulnerabilities. The glibc heap now contains safe unlinking, and a bunch of other features that raise the bar of exploitation significantly, however, these changes are still relatively minor - and are quite different from the fundamental changes that OpenBSD is making.
Fundamentally, they're both trying to solve the same problems, and both of them are quite effective - I certainly don't want to downplay the effects of the recent changes to ptmalloc2. Perhaps the only fundamental difference between the two, from the perspective of an end-user, is that the OpenBSD implementation will do a better job of preventing heap exploitation through heap data corruption (as opposed to heap management data corruption - sorry if the difference between the two isn't obvious), and the OpenBSD implementation will complain loudly (crash) almost every time there is a problem.