NGINX can be hacked if you configure it like this

Recently, security researchers found a zero-day vulnerability in Nginx, one of the most popular tools that powers servers around the world. More importantly, this zero-day has existed since 2011. It is unclear whether this security flaw has been exploited in the real world, but if your systems are using Nginx, it is recommended to upgrade to the latest version!

Here, we dig a little deeper to understand the flaw, assuming that we already know what Nginx is, its role as a Reverse Proxy, and how to configure it via the file nginx.conf.

1. How bad is it ?

This is an RCE flaw – Remote Code Execution – the most severe type of flaw that enables hackers to execute arbitrary commands on vulnerable servers. This means that if hackers successfully exploit this flaw, they can steal data, install ransomware, or shut servers down just by sending a malformed HTTP request.

2. How did the hack happen ?

According to news reports, not every Nginx server automatically has this vulnerability. This flaw exists only if system engineers configure an Nginx instance like this:

1. In a rewrite directive,
2. It uses an unnamed regex capture (e.g., using $1, $2 instead of named groups like (?<id>...)).
3. The replacement string contains a ? character (query part of a URL).
4. It is immediately followed by a set, if, or secondary rewrite directive.

For example, this part in file nginx.conf can open the flaw:

location / {
    # 1. Unnamed capture (.*) + 2. Question mark (?) in replacement
     rewrite ^/api/(.*)$ /internal?migrated=true;
    
    # 3. Chained 'set' directive referencing the unnamed capture
    set $original_endpoint $1;
}

2.1 Why does this specific configuration break Nginx?

Researchers found that Nginx’s script engine had a logic error, which can be seen as a subtle bug. This bug is very hard to observe, no matter how thoroughly testers test a server. It only occurs under specific configurations, as in above example, and I believe can only be discovered by luck or after a deep source code analysis.

Let’s look a little deeper into the source code of version 1.31.2—the version that is vulnerable. This bug is located in this module: https://github.com/nginx/nginx/blob/release-1.31.2/src/http/ngx_http_script.c. Reading this file requires serious experience in the C programming language, web development, and the architecture of a reverse proxy, along with days to weeks to digest Nginx’s internal design. To save time, you can use Copilot to read the code. I will skip the code explanation part here and only summarize the bug.

Simply put, this bug happened because Nginx uses a two-pass process to handle directives rewrite:

1. First, it calculates the size of the memory buffer required to store urls
2. and then, it copies the text into allocated memory region.

As we might already know, many of the most well-known Remote Code Execution (RCE) vulnerabilities stem from Buffer Overflow and Heap Overflow errors. These vulnerabilities occur when a program copies data into a reserved memory region without properly checking the size of the data. If the incoming data exceeds the capacity of the allocated memory, it overflows into adjacent memory, potentially causing the program to stop working, crash, or worse, allowing an attacker to execute arbitrary code.

This Nginx security flaw is a Heap Overflow error that is triggered by a mismatch in memory size calculation for storing URLs. It is important to note that this issue does not arise because developers forgot to check the memory size, but rather due to a miscalculation stemming from complex logic. This Heap Overflow occurs as follows:

2.2 How did Heap Overflow happen in Nginx?

Since this was a bug, so here are some contextual information:

• is_args: this variable is a 1-bit flag in the script engine that marks whether we’re currently processing URL query string arguments (the part after character ?). When is_args=1, it signals that the string containing special characters (/, ?, &, etc.) and must be URI-escaped (e.g., character “/" becomes string “%2F").
• escape function: converts a string containing special characters into percent-encoded format. This is a common practice when handling special characters in an URL. The escaped string usually has the length larger than the original string.

And the memory size miscalculation occurred as follows:

1. Request arrives with malformed URL containing query string (?) and regex capture variables (like $1).
2. Regex matching happens and captures special characters (e.g., /, ?, &) into buffer.
3. Length calculation phase starts without is_args flag set (line 1164-1169).
4. Capture length is calculated assuming no escaping because is_args=0 (line 1363-1373).
5. Buffer is allocated only for the unescaped size (line 1185).
6. Copy/execution phase begins with is_args=1 is set from parent context (line 1404-1412).
7. Escape function runs on captures because is_args=1, resulting new string that is longer than the original string if it contains special characters.
8. Escaped string is copied to allocated buffer causing heap overflow because it is longer than the buffer.
9. Adjacent heap metadata gets overwritten enabling arbitrary code execution via heap exploitation.
10. RCE then can happen if hacker is skillful enough

Here is a proof of concept: https://github.com/depthfirstdisclosures/nginx-rift that demonstrates RCE on vulnerable Nginx settings. We will learn about RCE via heap overflow in the next posts.

2.2.2 How does the Heap look like ?

For a simplified view, when a program starts, the operating system (such as Windows) allocates a region of memory for it. This memory contains separate areas for the program’s instructions (code), global and static variables, local variables, and a region used to store objects created dynamically at runtime, known as the heap. In RAM, this memory layout can be illustrated as follows:

Higher Memory Addresses
+----------------------+
|      Stack           |  ← Local variables, function calls
|         ↓            |
+----------------------+
|                      |
|    Free Memory       |
|                      |
+----------------------+
|         ↑            |
|       Heap           |  ← Dynamically allocated memory
|  [Object A]          |
|  [Object B]          |
|  [Free Block]        |
|  [Object C]          |
+----------------------+
|   Global / Static    |
+----------------------+
|      Program Code    |
+----------------------+
Lower Memory Addresses

Unlike the stack, the heap is not neatly organized. After many allocations and deallocations, it may look like:

Heap
+-----------+
| Object A  |
+-----------+
|   Free    |
+-----------+
| Object B  |
+-----------+
| Object C  |
+-----------+
|   Free    |
+-----------+
| Object D  |
+-----------+

A heap overflow occurs when a program writes beyond the boundary of one allocated object. For example, after allocating 64 bytes for Object B, it writes 100 bytes to region of Object B, causing overwriting 36 bytes to the region of Object C. This can corrupt Object C, potentially causes app crashes or even arbitrary code execution if it is exploited accurately.

2.2.3 How does the malformed URL look like ?

The malformed URL is actually not syntactically invalid. It is a perfectly valid HTTP request, but it contains many special characters that require URI escaping and eventually, can trigger the Heap Overflow bug. For example with above sample vulnerable Nginx configuration, the request can look like

GET /api/++++++++++++++++++++++++++++++++++++++++++++++++++++ HTTP/1.1

Notice that there are a lot of special characters “+” there. Let say there is N bytes of that bunch of character “+” , after it is escaped, the actual size will increase to 3N bytes, since the escaped “+” is “%2B”.

3. How to defend ?

3.1 ASLR

A good news is the RCE won’t happen easily on today computer thanks to ASLR mechanism – Address Space Layout Randomization.

Address Space Layout Randomization (ASLR) is a computer security technique that defends against memory corruption exploits, such as buffer overflows & heap overflows. It works by randomly arranging the memory locations every time a program runs, making it extremely difficult for attackers to locate and inject code. ASLR is enabled by default on every computers. This fact removes the fear of RCE and downgrade the risk to DoS level (Denial of Service) where attackers, at most, cause the server crashed and your system can be down for a while.

The proof of concept above works because it intentionally disable ASLR on the server, so that researchers can calculate memory sizes and memory addresses accurately. In real world, this will not work because the memory addresses will be different on each server, so attackers only can rely on luck or they must possess another level of hacking – which seems unreal for now.

3.2 Upgrade Nginx

To eliminate the risk of DoS, upgrade Nginx to latest versions is recommended. Although this flaw exists only under specific Nginx configuration, there is a chance that similar bugs can exists beyond our awareness.

3.3 Watch out the Crashes

Since RCE will not happen easily, vulnerable servers can still crash and disrupt business operations. In practice, system engineers usually set up systems to automatically restart Nginx if it goes down. However, restarting is not enough, as it can crash again from the same attack. Therefore, monitor crash events through system logs or any monitoring tools, and be cautious when it crashes, as app crashes are often symptoms of a zero-day security flaw. Developers dislike app crashes, but hackers thrive on them because if an app can crash, it means it can be hacked.