RE-011 Week 10: Anti-RE Tricks · RE-011 · Virtus Cyber Academy Classroom

Packing, control-flow obfuscation, anti-debug techniques, and how to recognize each one. Understanding what slows you down is the first step toward not being slowed down by it.

Reading (~45 min)

From Yurichev RE4B: read the "Obfuscation" and "Anti-debugging techniques" chapters. Yurichev covers these at a level appropriate for RE-011: conceptual understanding plus enough assembly detail to recognize them in practice.

Supplemental: read the Wikipedia article on UPX (the packer you will work with in this week's lab walk). UPX is the most common open-source packer; understanding how it works at a high level is sufficient for RE-011.

Lecture outline (~1.5 hr)

Part 1: Why anti-RE exists (10 min)

Software developers apply anti-RE techniques for several reasons:

Commercial software protection: prevent license bypass and cracking
Malware: avoid analysis by security researchers and AV engines
DRM: complicate reverse engineering of media protection
CTF design: add difficulty to challenges without adding algorithmic complexity

In RE-011, you encounter anti-RE in CrackMe challenges (where it is intentional and educational) and in the firmware analysis capstone (where it may appear in commercial embedded firmware). Understanding anti-RE is essential for recognizing when your analysis has hit an intentional obstacle versus a genuine complexity.

Part 2: Packing (25 min)

A packer compresses or encrypts the original binary and prepends a stub that decompresses/decrypts and runs the original at startup. The "packed" binary:

Contains no recognizable code in .text (the bytes are compressed/encrypted)
Has a small stub that is the only real code visible to static analysis
Decrypts the original code into memory at runtime, then jumps to the original entry point (OEP, Original Entry Point)

UPX is the most common open-source packer. A UPX-packed binary:

Has section names like UPX0, UPX1 instead of .text, .data
Has the string "UPX" in its binary content (by default)
Can be unpacked with upx -d binary (UPX self-identifies and includes an unpack routine)
The packed .text is empty or near-empty; UPX1 contains the compressed original

Detecting packing with static analysis:

strings output is very short (no meaningful strings -- they are compressed)
readelf -S shows unusual section names or sections with impossible characteristics (executable section with no data, or data section with execute permission)
The import table is minimal (only what the stub needs: LoadLibrary/GetProcAddress on Windows, or mmap/mprotect on Linux)
entropy analysis: compressed/encrypted data has high entropy (close to 8 bits/byte); normal code has lower entropy. Tools like binwalk -E or ent measure byte entropy.

RE-011 scope: RE-011 covers recognizing packing and using upx -d to unpack UPX-packed binaries. Full unpacking of custom packers (extracting the OEP, dumping from memory, fixing the import table) is RE-101 domain.

Part 3: Control-flow obfuscation (25 min)

Control-flow obfuscation techniques make disassembly harder to follow without changing the binary's observable behavior.

Control-flow flattening: All basic blocks are placed inside a large switch-based dispatcher. Instead of a function that calls A, then calls B, then calls C, you get a state machine where each block sets the "next state" variable and jumps back to the dispatcher. The disassembly looks like one enormous switch with no obvious call graph.

Recognizing it: a large function with a central switch, most blocks ending by assigning to the same local variable and jumping back to the same address. The decompiler shows it as a loop with a deeply nested switch.

Opaque predicates: A conditional jump whose condition always evaluates to the same value (always taken or never taken), but constructed so it is hard to simplify statically. Example: if (x * x >= 0) { ... } else { impossible_branch; } -- the else branch is dead code but clutters the disassembly. The opaque predicate adds a false branch that the analysis tool may follow.

Recognizing it: dead branches that have unreachable code. In the listing, branches that lead to int3 (breakpoint instruction), ud2 (undefined instruction), or immediately loop back to themselves are often opaque-predicate artifacts.

Instruction substitution: Replacing simple instructions with equivalent but more complex sequences. xor rax, rax (zero rax) might become push 0; pop rax. Simple operations replaced by longer, harder-to-pattern-match sequences.

Recognizing it: unusually verbose code for simple operations. Ghidra's decompiler simplifies many of these; the listing view shows them in full.

Jump trampolines / indirect calls: Instead of call function, the binary uses jmp [table_entry] or loads a function address from a computed location. This breaks static call graph analysis.

Recognizing it: call rax, call QWORD PTR [rbx+0x18], or similar computed calls. Ghidra marks these as CALL COMPUTED and cannot always determine the target statically.

Part 4: Anti-debug techniques (20 min)

Anti-debug techniques attempt to detect when the binary is being run under a debugger and alter behavior (crash, produce wrong output, exit) when detection fires.

ptrace self-attach (Linux): ptrace(PTRACE_TRACEME, 0, NULL, NULL) succeeds if the process has no debugger attached; it fails (returns -1) if the process is already being traced (because a debugger is already calling PTRACE_TRACEME). A binary that calls this and checks the return value can detect gdb.

Recognizing it: a ptrace call in the import table (visible in nm -D or readelf -s) or in an ltrace trace. In Ghidra, find the ptrace call site and look at what happens if it returns -1.

Timing checks: A debugger makes execution slow. A binary that measures elapsed time between two points and exits if the elapsed time is too large can detect single-stepping.

IsDebuggerPresent (Windows): A Windows API call that returns a bool indicating debugger attachment. Linux equivalent: reading /proc/self/status for the TracerPid field (nonzero if being traced).

Checksumming own code: A binary that hashes its own .text section at startup and exits if the hash does not match. Prevents binary patching (Week 11) from producing a working binary without also updating the checksum.

RE-011 scope: Recognizing these techniques and understanding conceptually how to bypass them. Active anti-debug bypass (patching ptrace calls, NOP-ing timing checks) is covered in Week 11 in the context of patching Lab 8.

Lab walk: Anti-RE analysis (~1 hr, ungraded)

Instructor-led: analyze a UPX-packed CrackMe. Steps:

file and strings on the packed binary -- observe the near-absence of strings.
binwalk -E or similar entropy visualization.
upx -d to unpack.
Re-analyze the unpacked binary with Ghidra and confirm it is now readable.

Also: examine a CrackMe with a ptrace anti-debug check. Find the ptrace call in Ghidra, understand the condition, and identify where you would patch to bypass it. (The patch itself is Lab 8.)

Independent practice (~3 hr)

Tool Journal: Document UPX: how to detect it, how to unpack it with upx -d, and what to expect afterward. Add a notes section on entropy analysis.
Anti-RE CrackMe: Find a CrackMe on crackmes.one rated 2-3 that uses anti-debug or packing (the CrackMe description usually says). Attempt static analysis. Document every anti-RE technique you find and how you recognized it.
CrackMe ladder: Continue the ladder. If you are stuck on a challenge because of anti-RE, document what you see and move to a challenge without it; the stuck challenges are good candidates for Lab 8.

Reflection prompts

UPX is the most common packer, and upx -d unpacks it in seconds. Why would a malware author use UPX? What does UPX protect against, and what does it fail to protect against?
Opaque predicates add dead code branches. A sufficiently advanced static analyser could evaluate constant expressions and remove them. Why don't all disassemblers do this by default? What risks would aggressive constant folding in a disassembler introduce?
An anti-debug technique based on timing assumes that debugging is slow. Modern hardware is fast, and a skilled debugger can step through code quickly. Under what conditions does a timing check fail as an anti-debug technique? Under what conditions does it succeed?

Week 10 of 14. Next: Binary patching -- objcopy, Ghidra's patch tool, and the smallest-possible-patch discipline.