Skip to content

AD2016: MarkImageAsNXCompatible

Summary

Property Value
ID AD2016
Name MarkImageAsNXCompatible
Category Security
Severity Error
Applies to PE (Windows)

Description

Binaries should be marked as NX compatible to help prevent execution of untrusted data as code. The NXCompat bit, also known as "Data Execution Prevention" (DEP) or "Execute Disable" (XD), triggers a processor security feature that allows a program to mark a piece of memory as non-executable.

This helps mitigate memory corruption vulnerabilities by preventing an attacker from supplying direct shellcode in their exploit.

How It Works

The rule checks for the IMAGE_DLLCHARACTERISTICS_NX_COMPAT flag (0x0100) in the PE optional header's DLL characteristics field.

When set, Windows enforces DEP for the process, preventing code execution from data pages (stack, heap, etc.).

Why This Matters

Data Execution Prevention (DEP) represents one of the most fundamental security improvements in computing history. It transforms exploitation from a straightforward exercise into a complex multi-step process.

The Pre-DEP Era

Before DEP (hardware NX bit), exploitation was straightforward:

1. Find buffer overflow
2. Overwrite return address with stack address
3. Place shellcode on stack
4. Control achieved

Success rate: ~100%
Skill required: Moderate

Shellcode could be placed anywhere in memory and executed directly.

How DEP Changed Exploitation

With DEP enforced, the stack, heap, and data sections are non-executable:

1. Find buffer overflow
2. Can't execute stack shellcode (DEP blocks it)
3. Must use Return-Oriented Programming (ROP)
4. Chain together existing code gadgets
5. Eventually call VirtualProtect or similar
6. Finally execute payload

Success rate: Lower, more fragile
Skill required: Significantly higher

Hardware vs Software DEP

Type Implementation Protection
Software DEP SEH validation only Limited
Hardware DEP (NX/XD) CPU-enforced Comprehensive

The NXCompat flag enables hardware DEP, which the CPU enforces on every memory access.

The NX Bit

Modern CPUs have a hardware "No eXecute" bit (called NX on AMD, XD on Intel):

  • Each memory page has an NX bit in its page table entry
  • CPU checks this bit on every instruction fetch
  • Attempting to execute from an NX page triggers an exception
  • Cannot be bypassed through software tricks

Exploit Technique Evolution

DEP forced exploit developers to invent new techniques:

Era Technique Complexity
Pre-DEP Direct shellcode Simple
Early DEP Return-to-libc Moderate
Modern ROP chains Complex
Current JIT-ROP, data-only Very complex

Why Some Binaries Lack NXCompat

Legitimate reasons are rare:

  1. JIT compilers: Must generate and execute code at runtime
  2. Self-modifying code: Some protection/obfuscation schemes
  3. Very old toolchains: Pre-Vista compilers didn't set the flag

None of these apply to typical applications.

Performance Considerations

DEP has zero performance overhead:

  • The NX bit is checked by hardware on every instruction fetch anyway
  • No additional instructions or checks needed
  • Memory usage is identical

Resolution

Enable DEP

Linker flag: 

link.exe /NXCOMPAT ...

This is enabled by default in modern Visual Studio.

Project Properties

  1. Open Project Properties
  2. Navigate to Linker → Advanced
  3. Set "Data Execution Prevention (DEP)" to "Yes (/NXCOMPAT)"

CMake

if(MSVC)
    add_link_options(/NXCOMPAT)
endif()

Remove /NXCOMPAT:NO

Search for any disabled DEP settings:

# Bad
link.exe /NXCOMPAT:NO ...

# Good
link.exe /NXCOMPAT ...

When to Suppress

This rule should rarely be suppressed. Exceptions might include:

  • JIT compilers: Must explicitly manage executable memory
  • Self-modifying code: Very specific legacy scenarios
  • Old driver models: Some kernel code requirements

Caveats

  • JIT compilers need to use VirtualAlloc with PAGE_EXECUTE_READ
  • Plugins or extensions must also be NX compatible
  • Mixing NX and non-NX modules can be problematic

Proper JIT Compilation

If you need to generate code at runtime:

// Allocate non-executable memory for writing
void* code = VirtualAlloc(NULL, size, MEM_COMMIT, PAGE_READWRITE);

// Write your code
memcpy(code, machineCode, size);

// Change to executable (non-writable)
DWORD oldProtect;
VirtualProtect(code, size, PAGE_EXECUTE_READ, &oldProtect);

// Now execute

DEP Process Policies

Policy Description
AlwaysOn DEP enabled, cannot be disabled
AlwaysOff DEP disabled (insecure)
OptIn DEP for system binaries only
OptOut DEP for all except opt-out

Modern Windows uses AlwaysOn for 64-bit processes.

References