Story continued from Page 1
The most common obfuscation technique is self modifying code. This allows the binary executable stored on a hard drive or CD to be different than the executable image running in memory. As the code executes it modifies itself. This is designed to thwart the static analysis of files which is used by most AV products. If the bytes on disk dont look like bad bytes the software is deemed OK.
For each reverse engineering technique developed, a defensive anti-reverse engineering technique has been created in response. Most code analysts will attempt to use static analysis of the binary or code. The next step is to run the software and inspect it in memory. Run-time analysis with a debugger is often the reverse engineers most powerful tool.
Yet, anti-debugging techniques can be used to allow obfuscation to survive dynamic analysis. As the software runs, it tests for the presence of a debugger and changes its behavior or simply exits if a debugger is detected.
Time and future developments are on the side of the reverse engineer, because once a piece of software ships, it's obfuscation techniques are fixed. On the other hand, what the software engineer gets by using anti-reversing techniques is time, and sometimes only a small amount of time is enough.
The good news is — even if we cant determine if software is going to cause harm — it is relatively easy to detect if it is trying to hide its behavior. We can scan binary code statically and detect self modifying code, root kit behavior, and anti-debugging code. By looking for these indications we have a good idea whether or not the software developer is trying to hide something from us, the user.
In the future, we should expect software to clearly document its behavior and let us verify it before running the software. After all we dont use undocumented crypto algorithms, because we cant tell if they are weak or contain a backdoor. In the end, the simplest way to break away from the war of attrition is to keep code open and easy to reverse engineer. Only then can obfuscated code always be treated as bad code.