M4. Introduction to Virtual Machine Introspection

By virtualizing hardware resources and allocating them based on need, virtualization has significantly increased the utilization of our computing capacities. It has pushed our modern computing paradigm from multi tasking computing to multi operating system computing. Located one layer below the operating system (OS), virtualization has become ubiquitous in the realm of enterprise computing today, underpinning cloud computing and data centers. It is expected to become ubiquitous on the desktop and mobile devices in the near future.

A compelling use case of virtualization in security is the virtual machine introspection (VMI), which pulls the guest OS state into the outside virtual machine monitor (VMM), or hypervisor and performs external monitoring of the runtime state of a guest OS. The introspection can be placed in a VMM, in another virtual machine (VM), or within any other part of the hypervisor, as long as it can inspect the runtime state of the guest OS—including CPU registers, memory, disk, and network. Because of such strong isolation, VMI has been widely adopted in many security applications such as intrusion detection, malware analysis, process monitoring, and memory forensics.

This introductory course aims to provide the attendees with the necessary knowledge of VMI. It starts from the basic concept, to the principles behind, and the enabled applications. In particular, based on years of experiences, the instructor will discuss how VMI works essentially, what the challenges are, and how to develop the practical VMI tools. Hands on experience with pre built VM with the corresponding toolsets will also be provided in this training.

Prerequisites:

• Basic knowledge of operating systems concepts, the security, and systems programming.

Outline:

1. Introduction (15 minutes)

   1. Course Overview

   2. Instructor Background

   3. Course Goals and Logistics

2. Basic Concepts (25 minutes)

   1. Isolation, mediation

   2. Passive, active monitoring

   3. In-VM vs. Out-of VM

3. Challenges in VMI (20 minutes)

   1. Trust

   2. The semantic gap

4. Techniques in bridging the semantic-gap (40 minutes)

   1. Manual approach

   2. Debugger-assisted

   3. Compiler-assisted

   4. Binary-code analysis assisted

   5. Guest-assisted

5. Deployment (20 minutes)

   1. Hardware based virtualization

   2. Hardware virtualization

   3. Software virtualization

   4. Emulation-based virtualization

   5. Additional hardware component

6. Applications (25 minutes)

   1. Kernel rootkit detection

   2. Malware analysis

   3. Memory forensics

7. Short Hands-on Project (30 minutes)

   1. Using kernel debugging tool to inspect kernel states

   2. Using volatility tool to perform memory introspection

8. Summary (5 minutes)

About the Instructor:

Dr. Zhiqiang Lin is an assistant professor of computer science, the director of systems and software security laboratory at The University of Texas at Dallas. He received his computer science PhD from Purdue University in 2011. His research interests lie in building new systems and automated techniques (using program analysis) to secure our computer systems including OS kernels and the running software. The particular interested security applications include the protection of hypervisor and operating system kernel, the inference of binary code for vulnerability discovery and malicious behavior analysis as well as the binary code rewriting and reuse, the investigation of the cyber-attacks such as intrusion detection and digital forensics, and the digital data recovery. More details of his research can be found at http://www.utdallas.edu/~zhiqiang.lin/s3.html.