Unmasking Virtualized Malware: A New Approach to Binary Deobfuscation

by

Author: Denis Avetisyan

Researchers have developed a scalable technique to analyze and reverse virtualization-based obfuscation, a common malware defense evasion tactic.

Control flow graphs were successfully reconstructed from code obfuscated via virtual machines, utilizing both trace-based analysis and the Pushan deobfuscation technique, demonstrating the feasibility of recovering program structure despite intentional complexity.
Control flow graphs were successfully reconstructed from code obfuscated via virtual machines, utilizing both trace-based analysis and the Pushan deobfuscation technique, demonstrating the feasibility of recovering program structure despite intentional complexity.

Pushan recovers complete control flow graphs from obfuscated binaries through static analysis, avoiding the limitations of full-trace dynamic symbolic execution.

Despite advances in malware analysis, virtualization-based obfuscation remains a potent defense against both human analysts and automated systems due to its capacity to conceal program logic. This paper introduces ‘Pushan: Trace-Free Deobfuscation of Virtualization-Obfuscated Binaries’, a novel technique that overcomes limitations of existing approaches by recovering complete control flow graphs without reliance on execution traces or computationally expensive dynamic symbolic execution. Pushan achieves this through VPC-sensitive, constraint-free symbolic emulation, enabling the generation of high-quality C pseudocode suitable for effective program understanding. Can this approach unlock deeper insights into previously intractable malware samples and fundamentally reshape the landscape of binary analysis?

The Evolving Landscape of Code Obfuscation and Analytical Limitations

Contemporary software security increasingly employs virtualization-based obfuscation as a robust defense against reverse engineering efforts. This technique involves executing the original code within a custom-built virtual machine, shielding the underlying logic and data from direct inspection. Rather than simply scrambling code, virtualization alters the execution environment itself, introducing layers of indirection and complexity. The virtual machine interprets custom bytecode or instructions, effectively presenting a moving target to analysts. This approach makes it significantly more difficult for attackers to understand the software’s functionality, identify vulnerabilities, or extract sensitive information, as traditional debugging and disassembly tools are rendered largely ineffective against the virtualized code’s indirect representation.

The efficacy of established software analysis methods is increasingly challenged by the sophistication of modern code obfuscation techniques. Traditional static analysis, which examines code without execution, falters when confronted with self-modifying code – programs that alter their own instructions during runtime – as the code’s apparent structure diverges from its actual behavior. Similarly, dynamic analysis, relying on observing a program’s execution, struggles to trace consistent control flow through code that is actively reshaping itself, leading to incomplete or misleading insights. These techniques, once reliable for identifying vulnerabilities and understanding program logic, now encounter a moving target, requiring significant adaptations or entirely new approaches to effectively dissect and comprehend increasingly complex software systems. This limitation creates a critical impasse in fields like cybersecurity, where accurate analysis is paramount for threat detection and mitigation.

The increasing sophistication of software obfuscation techniques presents a substantial impediment to crucial security practices. Vulnerability research, once reliant on dissecting code to identify weaknesses, now encounters layers of deliberately misleading instructions and constantly shifting code structures, making static analysis increasingly ineffective. Similarly, malware analysts face protracted delays and heightened complexity when attempting to understand malicious software’s true functionality, as obfuscation obscures the underlying intent. Perhaps most critically, verifying software integrity-ensuring a program hasn’t been tampered with-becomes significantly more challenging when the code’s legitimate form is hidden behind layers of protective distortion, potentially leaving systems vulnerable to exploitation and compromising trust in digital applications.

Virtualization-based obfuscation protects code by translating it into bytecode and executing it within a fetch-decode-execute loop managed by a central dispatcher and individual virtual machine handlers.
Virtualization-based obfuscation protects code by translating it into bytecode and executing it within a fetch-decode-execute loop managed by a central dispatcher and individual virtual machine handlers.

Pushan: A Framework for Recovering Obfuscated Control Flow

Pushan is a deobfuscation framework engineered to address the challenges posed by virtualization-based obfuscation techniques. These techniques protect binaries by executing the original code within a virtual machine, effectively concealing the underlying control flow. Pushan aims to reconstruct this obscured control flow, enabling analysis of the original program logic. Unlike generic deobfuscation tools, Pushan is specifically designed to handle the complexities introduced by the virtual machine interpreter, providing a means to analyze binaries where traditional disassembly yields incomplete or misleading results. The framework’s functionality centers on recovering a complete and accurate representation of the program’s execution path despite the virtualization layer.

Pushan employs a ‘Flat Control Flow Graph’ (CFG) as its core deobfuscation technique. This CFG differs from traditional approaches by integrating the control flow of both the virtual machine interpreter and the protected guest code into a single, unified graph. By representing both execution contexts within a single structure, Pushan eliminates the need to separately analyze and correlate the behavior of the interpreter and the original program. This merged representation allows for direct tracing of execution across virtualization boundaries, enabling the reconstruction of the original program’s control flow despite the obfuscation layer. The flat CFG facilitates a holistic view of program execution, treating instructions from both the guest and host contexts as interconnected nodes within a unified control flow landscape.

Pushan’s core innovation lies in its ‘VPC Sensitivity’ which simultaneously monitors both the program counter (PC) of the analyzed binary and the virtual program counter (VPC) within the emulated environment. This dual-counter tracking is essential because VM-obfuscated code executes instructions within the virtual machine, altering the typical execution flow visible to static analysis. By correlating changes in the VPC with corresponding changes in the PC, Pushan establishes a precise mapping between the obfuscated execution path and the original program logic. This sensitivity allows Pushan to accurately trace control flow transfers that would otherwise be obscured by the virtualization layer, enabling the reconstruction of a complete and accurate control flow graph.

Evaluation of the Pushan framework demonstrates a high degree of accuracy in reconstructing control flow from virtualized binaries. Across a test suite of 1,028 VM-obfuscated targets, Pushan successfully reconstructed control flow graphs with a substantial degree of similarity to the original, un-obfuscated binaries in 999 instances. This represents a 97.17% success rate, indicating the framework’s efficacy in defeating virtualization-based obfuscation and recovering executable logic for analysis.

The push-and-deobfuscation workflow successfully recovers the original control flow graph (CFG) from a virtual machine-obfuscated program by iteratively refining a VPC-sensitive CFG through infeasibility elimination, path pruning, and symbolic execution to resolve loop exit conditions.
The push-and-deobfuscation workflow successfully recovers the original control flow graph (CFG) from a virtual machine-obfuscated program by iteratively refining a VPC-sensitive CFG through infeasibility elimination, path pruning, and symbolic execution to resolve loop exit conditions.

Enhancing Analytical Precision Through Symbolic Execution and Optimization

Pushan significantly broadens the applicability of symbolic execution techniques to obfuscated code that previously hindered analysis. Traditional symbolic execution often fails when confronted with the complexities introduced by obfuscation methods designed to impede reverse engineering. Pushan overcomes these limitations through a novel approach to control flow recovery and constraint handling, allowing for the successful symbolic execution of code exhibiting substantial obfuscation. This extended capability facilitates deeper analysis of malicious or proprietary software, enabling vulnerability discovery and behavioral understanding in scenarios where standard dynamic or static analysis methods prove insufficient.

Following control flow recovery, the framework utilizes standard compiler optimization techniques to simplify the recovered code. Specifically, Constant Propagation replaces variables with their known constant values, reducing computational complexity. Dead Assignment Elimination removes assignments to variables whose values are never subsequently used, further streamlining the code. The application of these optimizations post-recovery is crucial for improving the efficiency of subsequent analysis stages, such as symbolic execution, and for reducing the overall analysis time and resource consumption. These techniques are applied iteratively until no further simplification is possible.

The framework utilizes Constraint-Free Symbolic Emulation (CFSE) to enhance analysis scalability and efficiency by eliminating the need for traditional path constraint management. Conventional symbolic execution tracks constraints along each execution path to determine program behavior, a process that leads to exponential growth in the number of constraints and significantly impacts performance. CFSE avoids this by directly emulating program behavior without explicitly constructing and solving path constraints; instead, it relies on concrete value tracking and a specialized semantics to determine the possible outcomes of instructions. This approach reduces computational overhead and memory usage, enabling analysis of larger and more complex codebases, while still maintaining the precision necessary for identifying program properties.

MBA (Mixed Boolean and Arithmetic) expressions are frequently utilized within virtualization schemes to obscure control flow and data dependencies, presenting significant challenges for static analysis. These expressions combine Boolean logic with arithmetic operations on symbolic values, creating complex conditional statements and data transformations. The described framework’s ability to analyze such expressions stems from its constraint-free symbolic emulation, which avoids the state explosion typically associated with path constraint management in traditional symbolic execution. This allows for efficient traversal and evaluation of the mixed Boolean and arithmetic logic inherent in MBA expressions, enabling the recovery of control flow and data dependencies within virtualized code, and ultimately facilitating deobfuscation and security analysis.

Evaluation of the deobfuscation framework on the Tigress platform yielded a 95.6% success rate, measured by input/output testing. This performance was determined by processing 1000 obfuscated code samples and verifying successful deobfuscation based on matching expected outputs for given inputs, utilizing hash functions to confirm correct functionality after the deobfuscation process. The metric specifically assesses the framework’s ability to recover functional equivalence of the original code following obfuscation and subsequent analysis.

Pushan’s deobfuscation technique successfully reconstructed the complete function call graph for Netsky, surpassing the results of prior work as demonstrated by a comparison to both the original (middle) and previously deobfuscated (left) graphs.
Pushan’s deobfuscation technique successfully reconstructed the complete function call graph for Netsky, surpassing the results of prior work as demonstrated by a comparison to both the original (middle) and previously deobfuscated (left) graphs.

The Broad Implications for Malware Analysis and Software Integrity Assurance

Pushan dramatically simplifies the traditionally complex process of malware analysis, offering researchers a powerful tool to rapidly dissect and comprehend malicious code. Historically, reverse engineers faced substantial hurdles in understanding obfuscated or protected software, requiring significant time and expertise to reconstruct the underlying logic. This framework bypasses many of those obstacles by automating key aspects of the analysis, allowing for quicker identification of functionality and intent. By reducing the technical barrier to entry, Pushan empowers a broader range of security professionals and researchers to contribute to threat intelligence and accelerate the development of effective defenses against emerging malware. The speed and efficiency gained are particularly valuable when dealing with rapidly evolving threats and zero-day exploits, where timely analysis is critical.

Protected software, often secured with layers of obfuscation and virtualization, presents a significant challenge to verifying its intended function and identifying hidden weaknesses. Pushan addresses this by facilitating a more thorough analysis of these complex systems, effectively stripping away protective measures to reveal the underlying code and behavior. This capability is crucial for ensuring software integrity, as it allows researchers to confirm that the program operates as designed and hasn’t been tampered with. Furthermore, detailed analysis enabled by Pushan proactively uncovers potential vulnerabilities – flaws in the code that could be exploited by malicious actors – before they can be leveraged in real-world attacks, ultimately strengthening the overall security posture of the software and its users.

Pushan demonstrates considerable flexibility by successfully interfacing with prominent virtualization technologies, including VMProtect and Themida, which are frequently employed to obscure malicious code and protect software intellectual property. This adaptability isn’t limited to defensive technologies; the framework also integrates effectively with research platforms such as Tigress, facilitating dynamic analysis and deeper understanding of program behavior. By supporting these diverse environments, Pushan empowers analysts to dissect and interpret protected software regardless of the obfuscation techniques utilized, providing a unified approach to malware analysis and software integrity verification across a broad range of applications and security measures.

Deobfuscation, a core capability of the framework, extends beyond simply understanding malicious code; it plays a crucial role in protecting intellectual property and enforcing software licensing agreements. Many software developers employ obfuscation techniques to safeguard their code from reverse engineering and unauthorized use. However, this same obfuscation can hinder legitimate license verification processes and enable software piracy. By effectively reversing these techniques, the framework allows for accurate inspection of software internals, confirming whether a valid license is present and adhered to. This capability is especially valuable in scenarios involving subscription-based software or complex licensing schemes, where verifying compliance requires detailed code analysis and can proactively combat intellectual property theft by exposing unauthorized modifications or distributions.

Pushan’s capabilities were rigorously tested through participation in a Capture The Flag (CTF) challenge, demonstrating a high degree of accuracy in dynamic analysis. The framework successfully reproduced Application Programming Interface (API) traces – the detailed record of function calls a program makes – and precisely matched the descriptions provided in five independent write-ups detailing solutions to the challenge. This successful reproduction isn’t merely about mimicking results; it validates Pushan’s ability to accurately interpret the runtime behavior of complex software, even when deliberately obscured. The framework’s performance in this controlled environment suggests a robust capacity for reverse engineering and understanding obfuscated code, proving its potential beyond theoretical application and into practical security assessments.

Pushan employs a pipeline encompassing perception, planning, and control to successfully manipulate objects in cluttered environments.
Pushan employs a pipeline encompassing perception, planning, and control to successfully manipulate objects in cluttered environments.

The methodology presented within demonstrates a commitment to establishing provable correctness in program analysis, aligning with a fundamentally mathematical approach to software security. Pushan’s success in recovering complete control flow graphs without relying on exhaustive dynamic symbolic execution highlights the power of static analysis when grounded in rigorous formalisms. This resonates deeply with the assertion of John von Neumann: “The sciences do not try to explain why we exist, but how we exist.” Just as von Neumann sought to understand the ‘how’ of existence through mathematical frameworks, Pushan elucidates the ‘how’ of obfuscated code execution by precisely defining and recovering control flow – a core concept of the study – rather than relying on empirical observation alone. The pursuit isn’t merely to make obfuscation appear defeated, but to demonstrably prove its unraveling.

What Lies Ahead?

The presented work, while a demonstrable advance in the field of virtualization-obfuscated binary deobfuscation, merely addresses a symptom of a deeper malaise. The proliferation of obfuscation techniques isn’t driven by a desire for elegant code, but by a pragmatic need to obscure intent. Thus, future efforts must move beyond simply reversing obfuscation to developing methods that formally verify the absence of obfuscation-a provable guarantee of code clarity. To claim a solution ‘works’ based on empirical testing is, frankly, insufficient. A mathematically rigorous demonstration of correctness remains elusive.

Current approaches, even those leveraging symbolic execution, are fundamentally limited by the inherent complexity of modern binaries. Scaling these techniques demands a shift in perspective. Instead of attempting to analyze complete programs, focus should be directed towards developing modular verification systems – provably correct components that can be assembled to form larger, trustworthy systems. The notion of ‘VPC sensitivity’ is interesting, but ultimately a heuristic; a formal model of obfuscation’s impact on control flow is required, not simply its observation.

Ultimately, the true challenge isn’t deobfuscation, but the creation of compilers and development tools that prevent obfuscation in the first place. A system that prioritizes clarity and provability by design, rather than attempting to retrofit it onto deliberately obscured code, represents the only genuinely robust long-term solution. The pursuit of elegance, after all, is not merely aesthetic; it is a prerequisite for true security.

Original article: https://arxiv.org/pdf/2603.18355.pdf

Contact the author: https://www.linkedin.com/in/avetisyan/

See also:

2026-03-20 17:18