Binary and Malware Analysis

Course Objective

Gain insights in static and dynamic analysis, applied to binaries and malware, and tackle fun challenges along the way.

Upon completion of this course, students will:

- have knowledge of state-of-the-art techniques for static and dynamic binary analysis (K&U)

- have sufficient knowledge and experience to reverse engineer the behavior of real-world - benign or malicious program binaries (LS, AK&U)

- be able to recognize and overcome anti-analysis techniques in real binaries (K&U, AK&U, MJ)

- be able to recognize and overcome unpacking behavior in real binaries (K&U, AK&U, MJ)

- be able to perform static control-flow and data-flow analysis for real binaries (K&U, AK&U, MJ)

- be able to perform dynamic data-flow tracking for real binaries (K&U, AK&U, MJ)

- be able to perform binary lifting and symbolic analysis for real binaries (K&U, AK&U, MJ)

The acronyms above refer to the Dublin Descriptors covered by each learning objective, specifically: Knowledge and Understanding (K&U), Applied Knowledge and Understanding (AK&U), Making Judgments (MJ), Learning Skills (LS), and Communication Skills (CS).

Course Content

Binaries in general, and malware in particular, are very hard to analyze. Unlike with source code, you have no idea what the binary does, or even what the data structures look like - let alone what they mean! Security analysts, forensic experts, and reverse engineers often have to dig their way through such programs to figure out what the code is all about, and where the interesting pieces of information are.

How do they do this? What techniques and tools can they fall back on, and, conversely, what techniques do the malware authors use to prevent this?

This is a hands-on specialization course for motivated students, who will learn essential analysis techniques and methods in both static and dynamic analysis. Not only will they pick apart a malicious binary, they will also be working on a set of challenging assignments to find a secret buried deep inside a binary program.

During the course we will cover everything from static analysis (CFG analysis, reverse-engineering, deobfuscation, etc), dynamic analysis (e.g., taint analysis), symbolic execution, static and dynamic patching, and many other techniques for pulling apart a mysterious binary. In the assignments, students will be exposed to programs that actively fight static and dynamic analysis. To circumvent the binary's defenses, students will use binary patching. After studying the details of popular binary formats (ELF, PE, etc.), they will patch the binary -- while adhering to the format. In addition, they will work with all manner of state-of-art system tools to lift and analyze the binaries (such as Ghidra, Intel PIN, etc.).

Additional Information Teaching Methods

Lectures (with voluntary attendance), (mandatory) practical assignments, and (mandatory) assignment-specific checks, which may take the form of oral discussions or short written tests per assignment.

Method of Assessment

The final grade is determined by the quality and correctness of solutions to the individual assignments together with assignment-specific checks in which students demonstrate their understanding of the relevant course material and their submitted solutions. These checks may take the form of oral discussions or short written tests per challenge.. Each of the practical assignments must be passed with a grade >= 4 (minimal passing grade). The final grade is: (i) NS if no assignment has been submitted; (ii) NVD if at least one assignment has been submitted but the minimum grade requirements have not been satisfied; (iii) otherwise a weighted average of the assignments' grades. The course can be passed with a final grade of at least 5.5.

In principle, there is no resit opportunity for the practical assignments, nor for the corresponding assignment-specific checks. However, to provide a limited safety net, a student who receives a grade below the passing grade for one assignment may be offered one additional replacement assignment. This option can be used for at most one assignment during the course. If the replacement assignment is completed satisfactorily, the grade for the original assignment will be adjusted to the minimum passing grade.

Results for practical assignments and the corresponding assignment-specific checks may be transferred from previous years if their contents did not change significantly since the result was obtained (as determined by the examiner). If you submit an assignment, the new result replaces any transferred results.

We ensure that each student individually demonstrates their abilities through active plagiarism detection for the assignments, and through assignment-specific checks. These checks aim to verify that students understand the relevant course material, their submitted solution, and the choices they made while completing the assignment.

Literature

No set book. All material will be made available during the course.

Suggested supplementary reading material: Andriesse, D, “Practical Binary Analysis: Build Your Own Linux Tools for Binary Instrumentation, Analysis, and Disassembly”, No starch press, 2018.

Additional Information Target Audience

Master Computer Security (year 1)

Additional Information

Use of AI: Level 2a Use of generative AI is only permitted for orientation and brainstorming

In this course, generative AI (GenAI), such as ChatGPT or similar tools, may only be used for orientation and brainstorming, for example to explore ideas in a paper. GenAI may not be used to generate content that becomes part of the final submission (e.g., in a homework assignment). The final product must be based entirely on the student's own understanding and analysis. Incorrect use of GenAI is considered fraud. If established, the submitted work or assessment result will be declared invalid, and the Examination Board may impose further measures.

Recommended background knowledge