The Beauty Of Film Grain: Must You Lose It In The Encode? Visionular’s encoding technology serves the unique characteristics of film grain.
Nathan Veer
Director of Product Marketing
“There’s a magic to the grain and the color quality that you get with film.” Judd Apatow
When it comes to film grain, most of us fall into one camp or the other. Either it’s a problem that needs to be removed because it makes for complex encodes and difficult digital delivery, or it is THE ingredient that makes your footage look cinematic and, well, not digital.
Blockbuster movies such as Creed, the latest Star Wars movies, Inception, and the latest James Bond, “No Time To Die” were all shot on film by legends including Tarantino, Nolan, Spielberg, and Fukunaga.
Love or hate it, film grain is what gives so many films a specific look or feel. In fact, some noise can improve the perceived picture quality due to a phenomenon called stochastic resonance. Certain noise (such as Film Grain) will resonate with the original weak signal resulting in better picture quality. But that’s another topic.
When it comes to encoding such footage for digital delivery, attempts are often made to remove that grain (aka noise) to reduce digital complexity and potential pixelation. However, it’s a new day in encoding. Not only is it possible, but it is critical to maintain film grain throughout the encoding process, especially of premium content.
The biggest challenge with film grain is that it is a kind of noise, which is irregular and random in distribution and size. This violates the basic assumption of many video encoding algorithms. Video encoders typically prefer predictable and consistent elements in the footage.
Visionular has applied the film grain encoding technology that the AV1 standard supports with their Aurora1 AV1 Encoder in a unique way. Traditional encoding techniques can generate uneven grain distribution or even just poor retention of the grain itself. The Aurora1 encoder takes full advantage of the film grain processing capabilities of the AV1 standard resulting in faster encoding times and smaller file sizes without quality loss.
To address different types of grains, Visionular’s film grain specific technology uses three different approaches.
-
Grain Synthesis
-
Grain Preservation
-
Synthesis and Preservation
Grain Synthesis
This process removes the grain in the video through a denoising process and estimates the grain parameters. The encoder retains these parameters and then sends them to the decoder along with the compressed video stream. At this point, the grain is re-generated using the estimated parameter and then added back to the reconstructed video.
This method allows one to preserve the grain while still achieving a lower bit rate and higher quality than directly encoding the grain.
When re-applying the noise, the decoding process uses a 32×32 block which is randomly selected from a 64×64 template. These 32×32 blocks are re-assembled in the same orientation next to each other and can often produce patterns and noticeable artifacts.
Visionular took a creative approach to address this challenge by selecting grain seeds and only using seeds with less obvious patterns for synthesis. The results are a much more natural reconstructed film grain.
Grain Preservation
Sometimes it is necessary to preserve the original grain directly in the encoding process, for example, the input video is already compressed. In this case, it’s difficult to estimate the original
Film grain parameters because it looks a lot like noise which many encoders are expressly designed to eliminate!
A typical video encoder does not handle film grain well because grains are small making them easy to be discarded during quantization. Once the grain disappears in the reference frame, it’s hard to bring them back. Actually, it’s impossible. And what looks like noise to an algorithm is an artful component to the artist (producer, director of photography). To solve this issue, we re-evaluate and adjust key algorithms in the video encoder, including Motion Estimation, Intra Prediction, Mode decision, RDO, and post processes.
Via these efforts, Aurora1’s grain preservation capabilities can make the grains distributed uniformly in one frame and across different frames but with the same original pattern. This is a unique capability and not found in other AV1 encoders. In fact, other codec standards do not support the capability natively. Making AV1 especially suitable for premium video content-encoding.
Preservation+Synthesis
The first two methods have complementary strengths. Grain synthesis is good at synthesizing high-frequency Gaussian noise but struggles with complex or irregular grain. Grain preservation can preserve complex or irregular difficult to preserve fine grain noise (but it will cost a lot of bits). For complex situations like this, Visionular combines the methods which look something like this:
-
- Decompose the grain.
- Retain and encode the grain that is difficult to generate with the Grain Preservation.
- Use the Grain Synthesis method for the more fine-grain.
In the example below, the grain synthesis method can retain the high-frequency gaussian noise. In contrast, the grain preservation method can retain complex/irregular grains. You end up getting the best of both worlds!
Left: Retains the grain in the encode. Right: Combines both methods of retention and synthesis.
As you can see, combining both methods to address complex grain patterns results in a natural-looking grain that is evenly distributed across the entire frame and fits the original artistic intent of the producer.
If you’d like to learn more about Visinular’s breakthrough encoding technology and see how it works firsthand, reach out to us, and we’ll help you take it for a test drive.
Continue reading...
At Foothill Ventures, we believe in startup companies that ride the transformative power of major technology shifts such as deep learning in computer vision. Visionular’s founders are world-class technologists in their field of video codec and AI-driven optimization. We feel privileged to support their adventure with our resources and experience.
I invested in Visionular because the team is at the forefront of innovations in video encoding and image processing for real-time low latency video communications and premium video streaming applications.
