Virtual Machine, Real Threats: Demystifying Virtualization Obfuscation for Resilient Software Security

Date of Award

Spring 2025

Project Type

Dissertation

Program or Major

Computer Science

Degree Name

Doctor of Philosophy

First Advisor

Dongpeng Xu

Second Advisor

Radim Bartos

Third Advisor

Elizabeth Varki

Abstract

Software obfuscation serves a dual purpose: It protects intellectual property and thwarts malware analysis, yet it can inadvertently enable advanced exploits and obstruct vulnerability discovery. This dissertation explores the multifaceted impact of obfuscation on software security, focusing primarily on virtualization-based obfuscators—widely recognized as among the most effective yet least understood forms of code obfuscation. This dissertation begins by demonstrating how conventional obfuscation can unintentionally facilitate sophisticated code-reuse attacks, enabling attackers to assemble more complex exploits. This dissertation then provides a systematic study of virtualization obfuscators, introducing a comprehensive taxonomy of VM diversification techniques, an automated tool to identify these techniques in real-world obfuscators, and an evaluation of how enhanced knowledge of VM internals can bolster deobfuscation. Finally, this dissertation investigates the challenge of vulnerability discovery in heavily virtualization obfuscated programs, proposing a hybrid fuzzing framework that combines runtime memory mutation and bottom-up fuzzing to effectively detect deep software flaws. By consolidating these efforts, this dissertation offers a roadmap for understanding, detecting, and mitigating modern obfuscation threats, ultimately empowering both the security research community and practitioners to build more resilient software.

This document is currently not available here.

Share

COinS