This paper classifies and presents several anti-debugging techniques used on Windows NT-based operating systems. Anti-debugging techniques are ways for a program to detect if it runs under control of a debugger. They are used by commercial executable protectors, packers and malicious software, to prevent or slow-down the process of reverse-engineering. We'll suppose the program is analyzed under a ring3 debugger, such as OllyDbg on Windows platforms. The paper is aimed towards reverse-engineers and malware analysts. Note that we will talk purely about generic anti-debugging and anti-tracing techniques. Specific debugger detection, such as window or processes enumeration, registry scanning, etc. will not be addressed here.

015C BA7201 MOV DX,0172
015F B80635 MOV AX,3506
0162 CD21 INT 21
0164 B425 MOV AH,25
0166 CD21 INT 21
0168 8DD3 LEA DX,BX
016A 06 PUSH ES
016B 1F POP DS
016C CD21 INT 21
The trick was that 8DD3 (LEA DX,BX) was an invalid opcode and caused the CPU to generate an INT 06. This same kind of thing can probably be used.
[ reply ]
Link to this comment: http://www.securityfocus.com/comments/infocus/1893/971#971