Anti Reversing Techniques

Master binary protection techniques for authorized security analysis

✨ The solution you've been looking for

Verified
Tested and verified by our team
25450 Stars

Understand anti-reversing, obfuscation, and protection techniques encountered during software analysis. Use when analyzing protected binaries, bypassing anti-debugging for authorized analysis, or understanding software protection mechanisms.

reverse-engineering malware-analysis anti-debugging binary-analysis obfuscation unpacking virtualization security-research
Repository

See It In Action

Interactive preview & real-world examples

Live Demo
Skill Demo Animation

AI Conversation Simulator

See how users interact with this skill

User Prompt

I need to analyze this suspicious binary, but it's detecting my debugger with IsDebuggerPresent and PEB checks. How can I bypass these protections?

Skill Processing

Analyzing request...

Agent Response

Step-by-step guidance on identifying and bypassing specific anti-debug checks, including tool recommendations and manual patching techniques

Quick Start (3 Steps)

Get up and running in minutes

1

Install

claude-code skill install anti-reversing-techniques

claude-code skill install anti-reversing-techniques
2

Config

3

First Trigger

@anti-reversing-techniques help

Commands

CommandDescriptionRequired Args
@anti-reversing-techniques malware-analysis-anti-debug-bypassAnalyze a protected malware sample that employs anti-debugging techniquesNone
@anti-reversing-techniques packed-binary-analysisUnpack and analyze a commercially protected executableNone
@anti-reversing-techniques virtualized-code-analysisUnderstand and reverse virtualization-based protection mechanismsNone

Typical Use Cases

Malware Analysis Anti-Debug Bypass

Analyze a protected malware sample that employs anti-debugging techniques

Packed Binary Analysis

Unpack and analyze a commercially protected executable

Virtualized Code Analysis

Understand and reverse virtualization-based protection mechanisms

Overview

AUTHORIZED USE ONLY: This skill contains dual-use security techniques. Before proceeding with any bypass or analysis:

  1. Verify authorization: Confirm you have explicit written permission from the software owner, or are operating within a legitimate security context (CTF, authorized pentest, malware analysis, security research)
  2. Document scope: Ensure your activities fall within the defined scope of your authorization
  3. Legal compliance: Understand that unauthorized bypassing of software protection may violate laws (CFAA, DMCA anti-circumvention, etc.)

Legitimate use cases: Malware analysis, authorized penetration testing, CTF competitions, academic security research, analyzing software you own/have rights to

Anti-Reversing Techniques

Understanding protection mechanisms encountered during authorized software analysis, security research, and malware analysis. This knowledge helps analysts bypass protections to complete legitimate analysis tasks.

Anti-Debugging Techniques

Windows Anti-Debugging

API-Based Detection

 1// IsDebuggerPresent
 2if (IsDebuggerPresent()) {
 3    exit(1);
 4}
 5
 6// CheckRemoteDebuggerPresent
 7BOOL debugged = FALSE;
 8CheckRemoteDebuggerPresent(GetCurrentProcess(), &debugged);
 9if (debugged) exit(1);
10
11// NtQueryInformationProcess
12typedef NTSTATUS (NTAPI *pNtQueryInformationProcess)(
13    HANDLE, PROCESSINFOCLASS, PVOID, ULONG, PULONG);
14
15DWORD debugPort = 0;
16NtQueryInformationProcess(
17    GetCurrentProcess(),
18    ProcessDebugPort,        // 7
19    &debugPort,
20    sizeof(debugPort),
21    NULL
22);
23if (debugPort != 0) exit(1);
24
25// Debug flags
26DWORD debugFlags = 0;
27NtQueryInformationProcess(
28    GetCurrentProcess(),
29    ProcessDebugFlags,       // 0x1F
30    &debugFlags,
31    sizeof(debugFlags),
32    NULL
33);
34if (debugFlags == 0) exit(1);  // 0 means being debugged

Bypass Approaches:

 1# x64dbg: ScyllaHide plugin
 2# Patches common anti-debug checks
 3
 4# Manual patching in debugger:
 5# - Set IsDebuggerPresent return to 0
 6# - Patch PEB.BeingDebugged to 0
 7# - Hook NtQueryInformationProcess
 8
 9# IDAPython: Patch checks
10ida_bytes.patch_byte(check_addr, 0x90)  # NOP

PEB-Based Detection

 1// Direct PEB access
 2#ifdef _WIN64
 3    PPEB peb = (PPEB)__readgsqword(0x60);
 4#else
 5    PPEB peb = (PPEB)__readfsdword(0x30);
 6#endif
 7
 8// BeingDebugged flag
 9if (peb->BeingDebugged) exit(1);
10
11// NtGlobalFlag
12// Debugged: 0x70 (FLG_HEAP_ENABLE_TAIL_CHECK |
13//                 FLG_HEAP_ENABLE_FREE_CHECK |
14//                 FLG_HEAP_VALIDATE_PARAMETERS)
15if (peb->NtGlobalFlag & 0x70) exit(1);
16
17// Heap flags
18PDWORD heapFlags = (PDWORD)((PBYTE)peb->ProcessHeap + 0x70);
19if (*heapFlags & 0x50000062) exit(1);

Bypass Approaches:

; In debugger, modify PEB directly
; x64dbg: dump at gs:[60] (x64) or fs:[30] (x86)
; Set BeingDebugged (offset 2) to 0
; Clear NtGlobalFlag (offset 0xBC for x64)

Timing-Based Detection

 1// RDTSC timing
 2uint64_t start = __rdtsc();
 3// ... some code ...
 4uint64_t end = __rdtsc();
 5if ((end - start) > THRESHOLD) exit(1);
 6
 7// QueryPerformanceCounter
 8LARGE_INTEGER start, end, freq;
 9QueryPerformanceFrequency(&freq);
10QueryPerformanceCounter(&start);
11// ... code ...
12QueryPerformanceCounter(&end);
13double elapsed = (double)(end.QuadPart - start.QuadPart) / freq.QuadPart;
14if (elapsed > 0.1) exit(1);  // Too slow = debugger
15
16// GetTickCount
17DWORD start = GetTickCount();
18// ... code ...
19if (GetTickCount() - start > 1000) exit(1);

Bypass Approaches:

- Use hardware breakpoints instead of software
- Patch timing checks
- Use VM with controlled time
- Hook timing APIs to return consistent values

Exception-Based Detection

 1// SEH-based detection
 2__try {
 3    __asm { int 3 }  // Software breakpoint
 4}
 5__except(EXCEPTION_EXECUTE_HANDLER) {
 6    // Normal execution: exception caught
 7    return;
 8}
 9// Debugger ate the exception
10exit(1);
11
12// VEH-based detection
13LONG CALLBACK VectoredHandler(PEXCEPTION_POINTERS ep) {
14    if (ep->ExceptionRecord->ExceptionCode == EXCEPTION_BREAKPOINT) {
15        ep->ContextRecord->Rip++;  // Skip INT3
16        return EXCEPTION_CONTINUE_EXECUTION;
17    }
18    return EXCEPTION_CONTINUE_SEARCH;
19}

Linux Anti-Debugging

 1// ptrace self-trace
 2if (ptrace(PTRACE_TRACEME, 0, NULL, NULL) == -1) {
 3    // Already being traced
 4    exit(1);
 5}
 6
 7// /proc/self/status
 8FILE *f = fopen("/proc/self/status", "r");
 9char line[256];
10while (fgets(line, sizeof(line), f)) {
11    if (strncmp(line, "TracerPid:", 10) == 0) {
12        int tracer_pid = atoi(line + 10);
13        if (tracer_pid != 0) exit(1);
14    }
15}
16
17// Parent process check
18if (getppid() != 1 && strcmp(get_process_name(getppid()), "bash") != 0) {
19    // Unusual parent (might be debugger)
20}

Bypass Approaches:

1# LD_PRELOAD to hook ptrace
2# Compile: gcc -shared -fPIC -o hook.so hook.c
3long ptrace(int request, ...) {
4    return 0;  // Always succeed
5}
6
7# Usage
8LD_PRELOAD=./hook.so ./target

Anti-VM Detection

Hardware Fingerprinting

 1// CPUID-based detection
 2int cpuid_info[4];
 3__cpuid(cpuid_info, 1);
 4// Check hypervisor bit (bit 31 of ECX)
 5if (cpuid_info[2] & (1 << 31)) {
 6    // Running in hypervisor
 7}
 8
 9// CPUID brand string
10__cpuid(cpuid_info, 0x40000000);
11char vendor[13] = {0};
12memcpy(vendor, &cpuid_info[1], 12);
13// "VMwareVMware", "Microsoft Hv", "KVMKVMKVM", "VBoxVBoxVBox"
14
15// MAC address prefix
16// VMware: 00:0C:29, 00:50:56
17// VirtualBox: 08:00:27
18// Hyper-V: 00:15:5D

Registry/File Detection

 1// Windows registry keys
 2// HKLM\SOFTWARE\VMware, Inc.\VMware Tools
 3// HKLM\SOFTWARE\Oracle\VirtualBox Guest Additions
 4// HKLM\HARDWARE\ACPI\DSDT\VBOX__
 5
 6// Files
 7// C:\Windows\System32\drivers\vmmouse.sys
 8// C:\Windows\System32\drivers\vmhgfs.sys
 9// C:\Windows\System32\drivers\VBoxMouse.sys
10
11// Processes
12// vmtoolsd.exe, vmwaretray.exe
13// VBoxService.exe, VBoxTray.exe

Timing-Based VM Detection

1// VM exits cause timing anomalies
2uint64_t start = __rdtsc();
3__cpuid(cpuid_info, 0);  // Causes VM exit
4uint64_t end = __rdtsc();
5if ((end - start) > 500) {
6    // Likely in VM (CPUID takes longer)
7}

Bypass Approaches:

- Use bare-metal analysis environment
- Harden VM (remove guest tools, change MAC)
- Patch detection code
- Use specialized analysis VMs (FLARE-VM)

Code Obfuscation

Control Flow Obfuscation

Control Flow Flattening

 1// Original
 2if (cond) {
 3    func_a();
 4} else {
 5    func_b();
 6}
 7func_c();
 8
 9// Flattened
10int state = 0;
11while (1) {
12    switch (state) {
13        case 0:
14            state = cond ? 1 : 2;
15            break;
16        case 1:
17            func_a();
18            state = 3;
19            break;
20        case 2:
21            func_b();
22            state = 3;
23            break;
24        case 3:
25            func_c();
26            return;
27    }
28}

Analysis Approach:

  • Identify state variable
  • Map state transitions
  • Reconstruct original flow
  • Tools: D-810 (IDA), SATURN

Opaque Predicates

 1// Always true, but complex to analyze
 2int x = rand();
 3if ((x * x) >= 0) {  // Always true
 4    real_code();
 5} else {
 6    junk_code();  // Dead code
 7}
 8
 9// Always false
10if ((x * (x + 1)) % 2 == 1) {  // Product of consecutive = even
11    junk_code();
12}

Analysis Approach:

  • Identify constant expressions
  • Symbolic execution to prove predicates
  • Pattern matching for known opaque predicates

Data Obfuscation

String Encryption

 1// XOR encryption
 2char decrypt_string(char *enc, int len, char key) {
 3    char *dec = malloc(len + 1);
 4    for (int i = 0; i < len; i++) {
 5        dec[i] = enc[i] ^ key;
 6    }
 7    dec[len] = 0;
 8    return dec;
 9}
10
11// Stack strings
12char url[20];
13url[0] = 'h'; url[1] = 't'; url[2] = 't'; url[3] = 'p';
14url[4] = ':'; url[5] = '/'; url[6] = '/';
15// ...

Analysis Approach:

 1# FLOSS for automatic string deobfuscation
 2floss malware.exe
 3
 4# IDAPython string decryption
 5def decrypt_xor(ea, length, key):
 6    result = ""
 7    for i in range(length):
 8        byte = ida_bytes.get_byte(ea + i)
 9        result += chr(byte ^ key)
10    return result

API Obfuscation

 1// Dynamic API resolution
 2typedef HANDLE (WINAPI *pCreateFileW)(LPCWSTR, DWORD, DWORD,
 3    LPSECURITY_ATTRIBUTES, DWORD, DWORD, HANDLE);
 4
 5HMODULE kernel32 = LoadLibraryA("kernel32.dll");
 6pCreateFileW myCreateFile = (pCreateFileW)GetProcAddress(
 7    kernel32, "CreateFileW");
 8
 9// API hashing
10DWORD hash_api(char *name) {
11    DWORD hash = 0;
12    while (*name) {
13        hash = ((hash >> 13) | (hash << 19)) + *name++;
14    }
15    return hash;
16}
17// Resolve by hash comparison instead of string

Analysis Approach:

  • Identify hash algorithm
  • Build hash database of known APIs
  • Use HashDB plugin for IDA
  • Dynamic analysis to resolve at runtime

Instruction-Level Obfuscation

Dead Code Insertion

1; Original
2mov eax, 1
3
4; With dead code
5push ebx           ; Dead
6mov eax, 1
7pop ebx            ; Dead
8xor ecx, ecx       ; Dead
9add ecx, ecx       ; Dead

Instruction Substitution

 1; Original: xor eax, eax (set to 0)
 2; Substitutions:
 3sub eax, eax
 4mov eax, 0
 5and eax, 0
 6lea eax, [0]
 7
 8; Original: mov eax, 1
 9; Substitutions:
10xor eax, eax
11inc eax
12
13push 1
14pop eax

Packing and Encryption

Common Packers

UPX          - Open source, easy to unpack
Themida      - Commercial, VM-based protection
VMProtect    - Commercial, code virtualization
ASPack       - Compression packer
PECompact    - Compression packer
Enigma       - Commercial protector

Unpacking Methodology

1. Identify packer (DIE, Exeinfo PE, PEiD)

2. Static unpacking (if known packer):
   - UPX: upx -d packed.exe
   - Use existing unpackers

3. Dynamic unpacking:
   a. Find Original Entry Point (OEP)
   b. Set breakpoint on OEP
   c. Dump memory when OEP reached
   d. Fix import table (Scylla, ImpREC)

4. OEP finding techniques:
   - Hardware breakpoint on stack (ESP trick)
   - Break on common API calls (GetCommandLineA)
   - Trace and look for typical entry patterns

Manual Unpacking Example

1. Load packed binary in x64dbg
2. Note entry point (packer stub)
3. Use ESP trick:
   - Run to entry
   - Set hardware breakpoint on [ESP]
   - Run until breakpoint hits (after PUSHAD/POPAD)
4. Look for JMP to OEP
5. At OEP, use Scylla to:
   - Dump process
   - Find imports (IAT autosearch)
   - Fix dump

Virtualization-Based Protection

Code Virtualization

Original x86 code is converted to custom bytecode
interpreted by embedded VM at runtime.

Original:     VM Protected:
mov eax, 1    push vm_context
add eax, 2    call vm_entry
              ; VM interprets bytecode
              ; equivalent to original

Analysis Approaches

1. Identify VM components:
   - VM entry (dispatcher)
   - Handler table
   - Bytecode location
   - Virtual registers/stack

2. Trace execution:
   - Log handler calls
   - Map bytecode to operations
   - Understand instruction set

3. Lifting/devirtualization:
   - Map VM instructions back to native
   - Tools: VMAttack, SATURN, NoVmp

4. Symbolic execution:
   - Analyze VM semantically
   - angr, Triton

Bypass Strategies Summary

General Principles

  1. Understand the protection: Identify what technique is used
  2. Find the check: Locate protection code in binary
  3. Patch or hook: Modify check to always pass
  4. Use appropriate tools: ScyllaHide, x64dbg plugins
  5. Document findings: Keep notes on bypassed protections

Tool Recommendations

Anti-debug bypass:    ScyllaHide, TitanHide
Unpacking:           x64dbg + Scylla, OllyDumpEx
Deobfuscation:       D-810, SATURN, miasm
VM analysis:         VMAttack, NoVmp, manual tracing
String decryption:   FLOSS, custom scripts
Symbolic execution:  angr, Triton

Ethical Considerations

This knowledge should only be used for:

  • Authorized security research
  • Malware analysis (defensive)
  • CTF competitions
  • Understanding protections for legitimate purposes
  • Educational purposes

Never use to bypass protections for:

  • Software piracy
  • Unauthorized access
  • Malicious purposes

What Users Are Saying

Real feedback from the community

Environment Matrix

Dependencies

x64dbg debugger
IDA Pro or Ghidra
Python 3.6+ for scripting
Windows or Linux analysis environment

Framework Support

ScyllaHide ✓ (recommended) FLOSS for string analysis ✓ angr symbolic execution ✓ Triton DBA framework ✓ miasm reverse engineering ✓

Context Window

Token Usage ~5K-15K tokens for comprehensive protection analysis

Security & Privacy

Information

Author
wshobson
Updated
2026-01-30
Category
debugging