Lorenzo Martignoni
Università degli Studi di Milano
Italy

Mihai Christodorescu
University of Wisconsin, Madison
USA

Somesh Jha
University of Wisconsin, Madison
USA

Malicious software (or malware) has become a growing threat as
malware writers have learned that signature-based detectors can be easily evaded by "packing" the malicious payload in layers of compression or encryption. State-of-the-art malware detectors have adopted both static and dynamic techinques to recover the payload of packed malware, but unfortunately such techniques are highly ineffective. In this paper we propose a new technique, called OmniUnpack, to monitor the execution of a program in real-time and to detect when the program has removed the various layers of packing. OmniUnpack aids malware detection by directly providing to the detector the unpacked malicious payload. Experimental results demonstrate the effectiveness of our approach. OmniUnpack is able to deal with both known and unknown packing algorithms and introduces a low overhead (at most 11% for packed benign programs).

Keywords: malware detection, malware unpacking, dynamic analysis, obfuscation, hardware protection

Read Paper (in PDF)