M3. Introduction to Reverse Engineering Malware

Reverse engineering involves deep analysis of the code, structure, and functionality of software using both static and dynamic methods. This tutorial will provide attendees with a basic foundation in reverse engineering malicious software and guidance for substantially increasing the depth of these skills in the future. Reverse engineering skills are crucial in understanding modern malicious software and this deep understanding, in turn, is necessary to evaluate the impact an attack has had on a system, to recover from the attack, and to craft solutions prevent future attacks. Reverse engineering is also useful for creating interoperable software, for verifying that software patches function as promised, and for the simple joy of understanding at a deep level how software works.

This tutorial provides attendees with knowledge (and some modest experience) in reverse engineering malware, covering a range of malware types, from "historical" (e.g., DOS boot sector viruses) through modern malware. The tutorial is modeled on the instructor's experiences in teaching full-semester, highly-immersive, reverse engineering courses to undergraduate and graduate students at the University of New Orleans. The tutorial is intended to appeal to the generally curious, to researchers for whom having malware analysis skills might be useful, and to academics considering introducing reverse engineering modules into their computer security curriculum. The training includes two instructor-assisted 'breakout' sessions in which teams of attendees statically analyze simple malware samples (on paper). Naturally, a one day session provides insufficient time for "mastering" even the basics of reverse engineering, but the tutorial provides a firm foundation on which to build additional skills for practice, research, and instruction. Static and dynamic analysis tools, including IDA Pro, and OllyDbg are demonstrated in the tutorial and detailed walkthroughs of malware source code reinforce the basic concepts that are introduced.

Prerequisites. Basic knowledge of assembler and systems concepts. Attendees should be either moderately comfortable with reading assembler or recall a time in which they were not completely uncomfortable doing so. The tutorial format will include time for attendees, in small groups, to tackle analysis of malware code samples (in hard copy) followed by a detailed walkthrough by the instructor. Any rust on preexisting assembler skills will be quickly sanded away. Attendees should also possess basic knowledge of systems, including compilation, linking, debuggers, concepts associated with executable file formats, etc. The course will only briefly touch upon legal issues associated with reverse engineering.

Outline:

1. Introduction (Brief, ~15 minutes)

   Course Overview. Instructor Background. Course Goals. Overview of Legal Issues and Disclaimer.

2. Reverse Engineering Background (1.5 hours)

   Why Learn / Teach Reverse Engineering? Overview of Historical and Current-generation Malware: Viruses, Worms, Trojans; Infection / Propagation strategies; Polymorphic / Metamorphic Malware. Tools for Static and Dynamic Analysis: Executable File Formats; Disassemblers; Debuggers; Tools for Live Analysis: Registry Monitoring, Filesystem Monitoring, System Call Tracing. Brief Refresher on Intel Assembler (w/ handouts / cheat sheets). PE Executable File Format Internals (w/ handouts).

3. First Immersion: Malware Sample # 1 (1.5 hours)

   Essential Background (w/ handouts / cheat sheets). IN TEAMS: Attendees Tackle Analysis of Malware Disassembly w/ Help of Instructor. Detailed Walkthrough by Instructor and Handout of Complete Solution for Further Study.

4. More Advanced Reverse Engineering: What You Need to Learn to Tackle Modern Malware (1 hour)

   Encrypted / Packed Executables. Anti-debugging / Anti-emulation / Anti-virtualization Techniques. Code obfuscation.

5. Second Immersion: Malware Sample # 2 (1.5 hours)

   Essential Background (w/ handouts / cheat sheets). IN TEAMS: Attendees Tackle Analysis of Malware Disassembly w/ Help of Instructor. Detailed Walkthrough by Instructor and Handout of Complete Solution for Further Study.

6. Summary / Wrap up / How to Develop Deeper Skills (Brief, ~15 minutes)

About the Instructor:

Dr. Golden G. Richard III is Professor of Computer Science, University Research Professor, and Director of the Greater New Orleans Center for Information Assurance (GNOCIA) at the University of New Orleans. At UNO, he teaches courses in digital forensics, reverse engineering, offensive computing, operating systems internals, and malware analysis, which are also his current areas of active research. Golden is a member of the United States Secret Service Electronic Crime Taskforce, holds a position on the Editorial Board of the Journal of Digital Investigation and the International Journal of Digital Crime and Forensics (IJDCF), is a member of the American Academy of Forensics Sciences (AAFS), a member of the ACM, IEEE, and USENIX, and serves as the academic liaison for USENIX to UNO. He is also a founding member and chairman of the non-profit that runs the Digital Forensics Research Workshop (DFRWS), the premiere venue for publishing digital forensics research. He earned a B.S. in Computer Science from the University of New Orleans and M.S. and Ph.D. degrees in Computer Science from The Ohio State University.

Dr. Richard has given tutorials at ACSAC, IPCCC, Mobicom, PDCS, USENIX ATC and USENIX Security on a variety of topics, including digital forensics, reverse engineering, and (in the more distant past) mobile computing concepts and service discovery protocols. He published a paper in CSET 2009 detailing his approach to teaching reverse engineering in academia, which, subject to depth and time limitations, underlies his approach in the proposed training.