The Offensive Security Experienced Penetration Tester (OSEP) certification is a 48-hour proctored lab focused on evasion and advanced penetration testing. Beyond just creating payloads that bypass antivirus, the real challenge is building a payload pipeline that survives modern defense layers: behavioral analysis, ETW monitoring, AMSI hooking, sandbox detection, and analyst-driven incident response.
The Evasion Stack
Most security researchers stop at AV signature evasion. But in 2026, static signatures are only one layer. Modern detection happens at multiple levels:
- Behavioral Analysis: watching suspicious API calls, process hollowing, reflective DLL injection
- ETW (Event Tracing for Windows): kernel-level logging of process creation, image load, CreateRemoteThread
- AMSI (Antimalware Scan Interface): in-memory scanning of PowerShell, VBA, JavaScript, and .NET assemblies
- Sandbox Detection: detecting Hyper-V, VirtualBox, QEMU, and automated analysis environments
In OSEP, I learned that effective evasion requires a layered approach. You are not just hiding from one tool. You are building a path through the entire detection infrastructure.
Technique 1: ETW Patching Without DLL Injection
Patching EtwEventWrite at runtime without touching loaded DLLs is the key. The standard approach uses reflective DLL injection, but that itself is detectable. A better approach:
- Allocate RWX memory with VirtualAlloc
- Write a small stub that patches EtwEventWrite by overwriting its prologue with a RET instruction
- Use GetProcAddress to find EtwEventWrite in ntdll.dll
- Patch 1-2 bytes: replace the first instruction with a RET (0xC3)
This works because ETW calls are made at the beginning of many system functions. If EtwEventWrite returns immediately, no events are logged.
The tricky part: the patch must survive function prologue restoration and be applied early enough that your malicious activity is not logged before the patch takes effect.
Technique 2: AMSI Bypass with In-Memory Byte Replacement
AMSI.dll exports AmsiScanBuffer, which is called before PowerShell code is executed. Bypassing it without breaking PowerShell requires surgical precision.
The classic approach:
- Find AmsiScanBuffer in memory
- Overwrite the first 7 bytes with a stub that returns S_OK (0x80070000 in EAX)
- Use a series of NOP sleds to align the instruction pointer
What I learned: this works, but modern EDR solutions hook the patch point itself. A more reliable approach is to patch the scan verdict checking code inside amsi.dll itself, not just the export. This requires understanding the function's full control flow.
Technique 3: Living off the Land with Legitimate Tools
The most effective evasion is not creating a malicious binary at all. Windows ships with:
- certutil.exe: download files, encode/decode Base64
- bitsadmin.exe: background intelligent transfer (appears as network activity, not payload staging)
- mshta.exe: execute VBScript HTML applications (LOLBIN for script execution)
- rundll32.exe: execute code from DLL exports
- regsvcs.exe / regasm.exe: load .NET assemblies with arbitrary code
These tools are trusted, signed, and rarely blocked by application whitelisting. The detection gap is in chaining these tools: certutil to download, mshta to execute, rundll32 to load a DLL. Each link in the chain looks benign; the attack surface is in the orchestration.
Technique 4: Polyglot Payloads and File Format Confusion
A polyglot file is valid under multiple file format parsers. Example:
- A valid PE executable that is also valid JavaScript
- A VBA macro file that embeds a raw shellcode blob
The attack is execution. You send the file as a .pdf, but the victim's system executes it as .exe based on magic bytes or content type.
In OSEP, I built a polyglot that:
- Started with a valid JavaScript file
- Embedded raw shellcode after a comment boundary
- Crafted the file so it could be executed as both JS (via mshta.exe) and raw shellcode (via DLL injection)
This defeats content-based detection and file-type validation.
Technique 5: Sandbox Detection Without Artifact Scanning
Sandboxes leak timing artifacts. The classic check: is the current process running in a virtualized CPU with fewer cores than typical hardware?
More subtle: measure the time required to hash a large buffer. Virtual CPUs have measurable latency differences. Real hardware can hash 1GB in X milliseconds; sandbox VMs often have measurable overhead.
The flaw in this approach: it is probabilistic. A sandbox could have more resources, or the analyst could be running on fast hardware.
A better approach: check for specific kernel behaviors. In a Hyper-V guest, CPUID returns specific magic values. In QEMU, the ACPI table contains "BOCHS" as the OEM ID. These are not timing artifacts; they are static artifacts the sandbox cannot easily spoof without breaking its own functionality.
Technique 6: Timing-Based Payload Staging
Deliver stage 1 (the loader) immediately. Stage 2 (the real payload) is fetched after a delay-anywhere from 30 seconds to days.
The benefit: sandbox automation has a timeout. Default timeouts for automated analysis are 2-5 minutes. If your loader sleeps for 10 minutes before fetching stage 2, the sandbox analysis completes before the malicious behavior occurs.
The downside: obvious to a human analyst watching long-running processes.
In OSEP, the constraint is different: you are in an active engagement with an analyst watching. Timing delays are less useful because the analyst can wait. But in post-exploitation persistence, timing delays evade automated threat hunting.
Lesson Learned: Defense in Depth
No single evasion technique is foolproof. Modern defenses are layered, and each layer defeats different techniques.
The path to evasion is understanding the detection infrastructure:
- What logs does this tool create?
- What behaviors does it monitor?
- What artifacts does it leave?
Effective evasion addresses all three. In OSEP, the goal was not to create a tool so advanced it bypasses every possible defense. The goal was to understand the detection infrastructure well enough to route around it in a controlled lab environment.
In real engagements, evasion is only one vector. Social engineering, credential theft, supply chain compromise, and exploitation of legitimate tools are often more effective than shellcode encryption.
Resources
- MITRE ATT&CK Tactics: Defense Evasion
- Elastic Security research on ETW patching
- Adam Chester's work on AMSI bypass
- James Forshaw's sandbox escape research
The OSEP certification taught me that effective penetration testing is not about tools. It is about understanding systems deeply enough to find the gaps in their defenses.