Using Memory Management to Detect and Extract Illegitimate Code for Malware Analysis

Exploits that successfully attack computers are mostly based on some form of shellcode, i.e., illegitimate code that is injected by the attacker to take control of the system. Detecting and gathering such code is the first step to its detailed analysis. The amount and sophistication of modern malware calls for automated mechanisms that perform such detection and extraction. In this paper we present a novel generic and fully automatic approach to detect the execution of illegitimate code and extract such code upon detection. The basic idea is to flag certain memory pages as non-executable and utilize a modified page fault handler to react on the attempt to execute code from them. Our modified page fault handler detects if legitimate code is about to be executed or if the code originates from an untrusted location. In such a case, the corresponding memory content is extracted and execution is continued to retrieve more illegitimate code.

We present an implementation of the approach for the Windows platform called XDetector, which involved reverse- engineering the proprietary memory management system of this operating system. Evaluation results using a large cor- pus of malicious PDF documents show that our system produces no false positives and has a very low false negative rate. To further demonstrate the universality of our approach, we also used it to detect shellcode execution in Flash Player, RealVNC client, and VideoLan Client.

Author(s):

Carsten Willems    
Ruhr University Bochum
Germany

Felix C. Freiling    
Friedrich-Alexander-Universität Erlangen-Nürnberg
Germany

Thorsten Holz    
Ruhr-University Bochum
Germany