NGINX can be hacked if you config it like this

Recently, security researchers had found a Zero Day in Nginx – one of the most popular tools that build up servers around the world. More importantly, this Zero Day existed since 2011. It is unclear that whether this security flaw is exploited in real world or not, but if your systems are using Nginx, it is recommended to upgrade Nginx to latest version!

Here, we dig a little deeper to understand the flaw, assuming that we already know what Nginx is, its role of Reverse Proxy, and how to configure it via file nginx.conf

1. How bad is it ?

This is a RCE flaw – Remote Code Execution – the most severe type of flaw that enable hackers to execute arbitrary commands on vulnerable servers. It means if hackers successfully exploited this flaw, they can steal data, install ransomware or shut servers down just by sending a malformed HTTP request.

2. How the hack happened ?

According to information on news, not every Nginx servers automatically have this vulnerability. This flaw exists only if system engineers configure the Nginx instance like this:

1. In a rewrite directive,
2. It uses an unnamed regex capture (e.g., using $1, $2 instead of named groups like (?<id>...)).
3. The replacement string contains a ? character (query part of an URL).
4. It is immediately followed by a set, if, or secondary rewrite directive.

For example, this part in file nginx.conf can open the flaw:

location / {
    # 1. Unnamed capture (.*) + 2. Question mark (?) in replacement
     rewrite ^/api/(.*)$ /internal?migrated=true;
    
    # 3. Chained 'set' directive referencing the unnamed capture
    set $original_endpoint $1;
}

2.1 Why does this specific configuration break Nginx?

Researchers found that Nginx’s script engine had a logic error which can be seen as a subtle bug. This bug is very hard to be observed, no matter how thoroughly testers test a server. This bug only happens under specific configurations like on above example – which I think it only can be seen by luck, or after a deep source code analysis.

Let’s look a little deep into the source code of version 1.31.2 – version that is vulnerable. This bug locates in this module: https://github.com/nginx/nginx/blob/release-1.31.2/src/http/ngx_http_script.c . Reading this file requires us to have serious experience on C programing language, Web development, Architecture of a Reverse Proxy and days to weeks to digest Nginx’s internal design. To save time, you can use Copilot there to read the code. I will skip the code explain part here and only summarize the bug.

Simply put, this bug happen because Nginx uses a two-pass process to handle directives rewrite:

1. First, it calculates the size of the memory buffer required to store urls
2. and then, it copies the text into allocated memory region.

As we might already know, many of the most well-known Remote Code Execution (RCE) vulnerabilities stem from Buffer Overflow and Heap Overflow errors. These vulnerabilities occur when a program copies data into a reserved memory region without properly checking data size. If the incoming data exceeds the capacity of the allocated memory, it overflows into adjacent memory, potentially make the program stop working, crash, or worst, allowing an attacker to execute arbitrary code.

This Nginx’s security flaw is a Heap Overflow error that is triggered by a mismatch in memory size calculation for storing urls. Notice that it is not because developers forgot to check the memory size, but miscalculate the size due to a complex logic. This Heap Overflow happens like so:

2.2 How does Heap Overflow happen in Nginx?

Since this is a bug, so here are some contextual information:

• is_args: this variable is a 1-bit flag in the script engine that marks whether we’re currently processing URL query string arguments (the part after character ?). When is_args=1, it signals that the string containing special characters (/, ?, &, etc.) and must be URI-escaped (e.g., character “/" becomes string “%2F").
• escape function: converts a string containing special characters into percent-encoded format. This a common practice when handling special characters in an URL. The escaped string usually has the length larger than the original string.

And the memory size miscalculation happens like so:

1. Request arrives with malformed URL containing query string (?) and regex capture variables (like $1).
2. Regex matching happens and captures special characters (e.g., /, ?, &) into buffer.
3. Length calculation phase starts without is_args flag set (line 1164-1169).
4. Capture length is calculated assuming no escaping because is_args=0 (line 1363-1373).
5. Buffer is allocated only for the unescaped size (line 1185).
6. Copy/execution phase begins with is_args=1 is set from parent context (line 1404-1412).
7. Escape function runs on captures because is_args=1, resulting new string that is longer than the original string if it contains special characters.
8. Escaped string is copied to allocated buffer causing heap overflow because it is longer than the buffer.
9. Adjacent heap metadata gets overwritten enabling arbitrary code execution via heap exploitation.
10. RCE then can happen if hacker is skillful enough

There is a proof of concept here: https://github.com/depthfirstdisclosures/nginx-rift that can demonstrate RCE on a vulnerable Nginx settings. We will learn about RCE via Heap Overflow in next posts.

2.2.2 How does the Heap look like ?

For a simplified view, when a program starts, the operating system (such as Windows) allocates a region of memory for it. This memory contains separate areas for the program’s instructions (code), global and static variables, local variables, and a region used to store objects created dynamically at runtime, known as the heap. In RAM, this memory layout can be illustrated as follows:

Higher Memory Addresses
+----------------------+
|      Stack           |  ← Local variables, function calls
|         ↓            |
+----------------------+
|                      |
|    Free Memory       |
|                      |
+----------------------+
|         ↑            |
|       Heap           |  ← Dynamically allocated memory
|  [Object A]          |
|  [Object B]          |
|  [Free Block]        |
|  [Object C]          |
+----------------------+
|   Global / Static    |
+----------------------+
|      Program Code    |
+----------------------+
Lower Memory Addresses

Unlike the stack, the heap is not neatly organized. After many allocations and deallocations, it may look like:

Heap
+-----------+
| Object A  |
+-----------+
|   Free    |
+-----------+
| Object B  |
+-----------+
| Object C  |
+-----------+
|   Free    |
+-----------+
| Object D  |
+-----------+

A heap overflow occurs when a program writes beyond the boundary of one allocated object. For example, after allocating 64 bytes for Object B, it writes 100 bytes to region of Object B, causing overwriting 36 bytes to the region of Object C. This can corrupt Object C, potentially causes app crashes or even arbitrary code execution if it is exploited accurately.

2.2.3 How does the malformed URL look like ?

The malformed URL is actually not syntactically invalid. It is a perfectly valid HTTP request, but it contains many special characters that require URI escaping and eventually, can trigger the Heap Overflow bug. For example with above sample vulnerable Nginx configuration, the request can look like

GET /api/++++++++++++++++++++++++++++++++++++++++++++++++++++ HTTP/1.1

Notice that there are a lot of special characters “+” there. Let say there is N bytes of that bunch of character “+” , after it is escaped, the actual size will increase to 3N bytes, since the escaped “+” is “%2B”.

3. How to defend ?

3.1 ASLR

A good news is the RCE won’t happen easily on today computer thanks to ASLR mechanism – Address Space Layout Randomization.

Address Space Layout Randomization (ASLR) is a computer security technique that defends against memory corruption exploits, such as buffer overflows & heap overflows. It works by randomly arranging the memory locations every time a program runs, making it extremely difficult for attackers to locate and inject code. ASLR is enabled by default on every computers. This fact removes the fear of RCE and downgrade the risk to DoS level (Denial of Service) where attackers, at most, cause the server crashed and your system can be down for a while.

The proof of concept above works because it intentionally disable ASLR on the server, so that researchers can calculate memory sizes and memory addresses accurately. In real world, this will not work because the memory addresses will be different on each server, so attackers only can rely on luck or they must possess another level of hacking – which seems unreal for now.

3.2 Upgrade Nginx

To eliminate the risk of DoS, upgrade Nginx to latest versions is recommended. Although this flaw exists only under specific Nginx configuration, there is a chance that similar bugs can exists beyond our awareness.

3.3 Watch out the Crashes

Since RCE will not happen easily, vulnerable servers still crash and can cause disrupts to the business operations. In practice, system engineers usually setup system to automatically restart the Nginx if it is down, but restarting is not enough since it can be crashed again by the same attack. So, watch out crash events, via system logs or any monitor tools, and beware when it crashes because app crashes are often symptoms of a Zero Day security flaw. Developers hate app crashes, but hackers love it because if an app can crash, it means it can be hacked.