Tese submetida à "Graduate Faculty of George Mason University" como requisito parcial para a obtenção do grau de Doutor em "Computer Science", tendo como orientador Xinyuan Wang. Ficheiro de 4,26 MB em formato PDF (160 p.).

1. Introduction: Automated Forensic Analysis. 1.1 Automated Forensic Analysis Problem Model. 1.1.1 Attack Code. 1.1.2 Original Attack String. 1.1.3 Data Structure Exploited. 1.2 Contributions. 2. Related Work. 2.1 Offensive Techniques. 2.2 Defensive Techniques. 2.3 Open Problems. 3. Background: Malware Concepts and Considerations. 3.1 Exploitation. 3.1.1 Exploit Mechanics. 3.1.2 Organizing Exploits. 3.1.3 Example Malware Life Cycle: The Roving Bugnet. 3.2. Shellcode. 3.3 Obfuscation. 3.3.1 Selected Encoders. 3.3.2 Novel Incremental Encoder. 3.4 Analyzing Obfuscated Malware. 4. Automated Extraction of Obfuscated Code. 4.1 Overview. 4.1.1 Motivating Examples. 4.1.2 Overall CodeXt Architecture. 4.2 Design. 4.2.1 Necessary Conditions and Heuristics. 4.2.2 Locating Hidden Code. 4.2.3 Handling Self-modifying Code. 4.3 Methodology. 4.3.1 Online Specification Based Detection Component. 4.3.2 Dynamic Analysis Component. 4.4 Implementation. 4.4.1 Pre-Execution Processing. 4.4.2 Execution Processing. 4.4.3 Post-Execution Processing. 4.5 Empirical Evaluation. 4.5.1 Accuracy and Performance. 4.5.2 Locating the Hidden Code from Memory Dump. 4.5.3 Extracting Encoded Code. 4.5.4 Emulation Detection Evasion. 5. Automated Location of Attack String from Run-time Input. 5.1 Design. 5.1.1 Run-time Hidden Branch Coverage. 5.1.2 Taint Labels and Tracking. 5.1.3 Dataflow Validity Throughout Intermingled Code. 5.1.4 Monitoring Real-time Attacks. 5.2 Methodology and Implementation. 5.2.1 Symbolic Conditional Branch Exploration. 5.2.2 Labeling Taint Sources. 5.2.3 Symbolic Execution of Tainted Code. 5.2.4 Limiting Propagation. 5.2.5 Monitoring Real-time Attacks. 5.3 Empirical Evaluation. 5.3.1 Multiple Labels and Propagation. 5.3.2 Executing Symbolic Code and Tracking Decoding Keys. 5.3.3 Locating an Attack String During a Buffer Overflow. 5.3.4 Monitoring Executables under Attack. 5.3.5 Identifying an Attack String within Network Traffic. 5.3.6 Analytics Tool. 6. Conclusions and Future Work. A Disassembled Encoders and Execution Traces. A.1 ADMmutate Encoder Output. A.2 Clet Encoder Output. A.3 Alpha2 Encoder Output. B DASOSF Memory Dump. B.1 Dump in Human Readable Format. C CodeXt Code Tracing. C.1 Results of Searching for Start of Malicious Code. C.2 Handling Multiple Positives when Searching for Start of Malicious Code. C.3 Raw Data of Reasons for Negative Matches. D Attack String Location and Taint Tracking. D.1 Shikata-Ga-Nai Expression Simplification Example. D.2 Buffer Overflow Taint Tester. D.3 Vulnerable Server Source. References.