Reverse Engineering Basics — Study Notes

Summary

Reverse Engineering (RE) is the process of analyzing a compiled program to understand its functionality without access to the original source code. It is widely used in cybersecurity for malware analysis, vulnerability research, and CTF challenges.

The goal is simple: take a binary → understand what it does → manipulate or exploit its logic.

Core Concepts

Binary vs Source Code

• Source Code → Human-readable (C, C++, Python)
• Binary → Machine code (compiled, not readable directly)

Example:

int a = 5 + 3;

Becomes assembly instructions like:

mov eax, 5
add eax, 3

Compilation Flow

[ Source Code ]
      ↓
[ Compiler (gcc, clang) ]
      ↓
[ Binary (ELF / PE) ]
      ↓
[ Execution by CPU ]

Binary Types

Linux (ELF)

• Format: ELF (Executable and Linkable Format)
• Used in Linux systems
• Common tools:
  • file
  • ldd
  • readelf

Windows (PE)

• Format: PE (Portable Executable)
• Used in Windows systems
• Tools:
  • PE-bear
  • x64dbg

Recon on Binary

Before reversing, always inspect the binary.

file binary

→ Identifies architecture (x86, x64)

checksec binary

→ Shows protections:

• NX
• PIE
• Canary
• RELRO

strings binary

→ Extract readable text (passwords, URLs, hints)

Static Analysis

Analyzing without running the binary.

Tools

• Ghidra
• IDA Pro
• Binary Ninja

What to Look For

• Functions
• Strings
• Conditions (if/else)
• Loops
• Hardcoded values

Example logic:

if (input == "admin123")
    access_granted();

Dynamic Analysis

Running the program and observing behavior.

Tools

• GDB
• x64dbg

Basic GDB Commands

gdb ./binary

break main
run
next
step
info registers
x/s

Assembly Basics

Registers (x86_64)

• rax → return value
• rbx, rcx, rdx → general purpose
• rsp → stack pointer
• rbp → base pointer

Common Instructions

mov rax, 5      ; assign
add rax, 3      ; addition
cmp rax, 8      ; compare
je label        ; jump if equal
jne label       ; jump if not equal
call func       ; function call
ret             ; return

Control Flow

Programs rely heavily on jumps.

cmp eax, 10
je success
jne fail

This translates to:

if (eax == 10)
    success();
else
    fail();

Strings & Input Validation

Most beginner RE challenges involve input checks.

cmp [input], 'A'
jne fail

Goal:

→ Find correct input that passes all checks

Common Techniques

1. String Hunting

strings binary | grep flag

2. Patching

Modify binary logic:

jne fail → je fail

3. Debugging

• Step through execution
• Modify registers
• Skip checks

4. Decompilation

Use tools to convert assembly → pseudo C

Typical CTF Flow

[ Get Binary ]
      ↓
[ file / checksec ]
      ↓
[ strings analysis ]
      ↓
[ Open in Ghidra ]
      ↓
[ Find main logic ]
      ↓
[ Identify condition ]
      ↓
[ Solve / patch ]

Key Takeaways

• Always start with basic recon (file, strings)
• Static analysis helps understand logic
• Dynamic analysis confirms behavior
• Most challenges rely on simple conditions
• Practice is everything in reverse engineering