2021 MSU Codec Study Subjective Results

2021 MSU Subjective Codec Eval Results 1fps
AV1 codec implementations compared.
  • The Visionular Aurora1 AV1 encoder beat all AV1 implementations for the third year in a row using both subjective and objective quality metrics.
  • Aurora1 beat all encoder implementations for the quality and efficiency use-case based on subjective viewing evaluation.
  • Aurora1 enables services to save more than 60% compared to x265 reference encoder while maintaining perceptually identical visual quality.

challenge.

The annual Moscow State University Codec Study completed by the Graphics and Media Lab Video Group represents the best round up of video codec implementations in the video streaming and delivery industry. Video encoding engineers worldwide rely on this comprehensive and objective study to guide their codec selection decisions and the best implementation for their use case. 

Visionular first participated in the 2019 study where our Aurora1 AV1 encoder came out on top for all objective metrics, a remarkable achievement, given the implementation was new. In 2020, we also ranked at the top as the best performing encoder.

For this year’s study, 2021, we again participated. The competition was steep, and yet, Aurora1 once again beat all competing AV1 encoder implementations and ranked a close second only to the S266 encoder using subjective viewing. We also joined the winning group for the Best Speed-Quality tradeoff. 

method.

This immense effort to evaluate 16 video encoders covering H.264 AVC, H.265 HEVC, AV1, VP9, and VVC utilized 15 video sequences at Full HD resolution. The process of video sequence selection involved voting among the participants, organizers, and an independent expert. The MSU team analyzed more than 1,500,000 video sequences and selected representative examples to comprise the test set.

Video encoder selection and codec standards are dependent on the use case. What works well for VOD may not be appropriate for a live or real-time deployment. Thus, the MSU comparison consists of two parts, corresponding to fast encoding and offline encoding (Fast and Slow).

For each use case, codec developers could provide encoding parameters. If a developer declined to provide codec parameters, evaluators either used the same parameters from our prior study or, if none were available, they did their best to choose the optimum parameters for the test. Each encoder tested had to satisfy the minimum speed requirement for their respective use case as follows while encoding at 1080p resolution.

  • Online – Fast (30 fps)
  • Offline – Slow (1 fps)

All comparisons used a machine based on an Intel Core i7-8700K (Coffee Lake)processor @ 3.7GHz with 32 GB of RAM running Windows 10 or Ubuntu. The YUV-SSIM metric was used as the primary objective quality indicator, though PSNR and VMAF were also measured.

Researchers used the Subjectify.us crowdsourcing platform and involved more than 10,800 participants where they received 529,171 pairwise answers. The Bradley-Terry model was used to compute global rank.

results.

Ranked subjectively, Aurora1 beat rav1e and QAV1 including VP9 and popular HEVC implementations.

msu 2021 bitrate quality subjective results

To understand how the results were tabulated, you can view a free abridged version of the report by clicking here.

The subjective scores were generated using Subjectify.us a platform that enabled the researchers to conduct subjective comparisons of video-processing methods to determine the best performing encoders based on the operational criteria. Watch the Subjectify video here.

Rate-Distortion Curves

Relative-bitrate shows the average bitrate dependence on relative encoding time for a fixed quality output. The y-axis shows the ratio of a codec’s bitrate under test to the reference codec’s bitrate for a fixed quality.

A lower value indicates a better-performing codec. For example, a value of 70% means the codec can encode the sequence using 30% fewer bits than the reference codec (x265 was the reference used for this study).

The x-axis shows the relative encoding time. Larger values indicate a slower codec. A value of 2.5 means the codec works 2.5 times slower, on average than the reference codec (x265).

Judging from the mean quality scores, first place based on quality went to S266, second place to Aurora AV1, and third place to QAVS3. Notably, VVC and AVS3 are considered niche codecs because of their limited hardware decoder support, while AV1 device penetration is growing rapidly.

rd curve subjective eval full hd 2021 msu study

Aurora1 AV1 encoder achieved:

  • 60% reduced bitrate at the same quality over the Reference x265 using the Subjective metric.
  • 54% reduced bitrate at the same quality over Reference x265 using Y-VMAF.
  • 139% advantage over SVT-VP9 using the Subjective metric.
  • Achieved a 15% bitrate savings over QAV1, and a massive 53% savings over rav1e.
  • Aurora1 was the best performing codec for quality/speed (Y-VMAF) while ranking second overall using the Subjective metric.

The following graph compares coding efficiency of the Aurora1 AV1 encoder with the other AV1 implementations in the study, as well as x265, and VP9 using the VMAF quality metric.

VMAF MSU Codec Study 2021 Subjective Results

The following graph compares all encoders tested across all content for the 1fps use-case using the subjective, Y-VMAF, and YUV-SSIM comparison metrics.

msu 2021 subjective results all encoders all quality metrics

Get more info on AV1.

How We Help Hudl “Up” Their Video Quality Game

How We Help Hudl "Up" Their Video Quality Game Is your streaming business burdened by escalating storage and CDN delivery...
Read More

Optimizing User Generated Content for the World

Optimizing User Generated Content for the World YouTube has become the go-to platform for user-generated content (UGC), with over 2...
Read More

Using AI & ML to Enhance the Visual Appeal of Video

Using AI and Machine Learning, our technology can enhance and improve the perceptual quality of any video.
Read More

Putting the Aurora5 HEVC Encoder to the Test

The Aurora5 HEVC encoder was tested by a major video platform against x264 and x265 and the results were blistering.
Read More

Making Sense of VMAF, SSIM, and PSNR

This article provides the video engineer with real-world and technical insights for how to correctly select a quality metric.
Read More

Aurora1 AV1 vs. x264 Real-Time Performance

Aurora1 AV1 was compared to x264 at 3 different speed settings. Aurora1 showed a 40% bitrate savings while being 25-95%...
Read More

AI+Codec Unlocks Superior Visual Quality

Advanced AI tools combined with core codec optimization are a silver bullet to deliver superior visual quality across all video...
Read More

Screen Content Coding With AV1

AV1 Screen Content Coding Tools can enable a 50% bitrate reduction over OPEN H264 while providing a demonstrably sharper image.
Read More

Fast AV1 Encoding Is Finally Here

For years “codec complexity” ergo “high computational cost” has hung over the AV1 codec, but with Aurora1 this is no...
Read More

The Beauty Of Film Grain: Must You Lose It In The Encode?

You no longer have to choose between a clean encode and beautiful film grain. You can have both!
Read More

How To Deliver HDR-Like Video In An SDR Package

Now you can deliver HDR-like quality video while keeping it in an SDR format reaching a much larger audience and...
Read More

HUYA Delivers Improved Viewing Experience With Visionular

HUYA delivers a world class live streaming video experience for their users enabled by Visionular technology and solutions.
Read More

What’s More Important, QOE or ROI?

ShareChat, is one of India’s largest short-form video platforms, together with Visionular, they were able to boost platform efficiency, decrease...
Read More

Zoe Liu LVS Interview Oct 2021

Zoe Liu talks with LiveVideoStack about the personal triumphs and key learnings as a first time entrepreneur and the co-founder...
Read More

AV1 For WebRTC – New Benchmarks

Read how AV1's screen content coding tools and exceptional compression efficiency make it perfectly suited for RTC applications.
Read More

Aurora1 AV1 vs. NVIDIA NVENC HEVC

See our Aurora1 is able to cut the bitrate needed for 1080p60 cloud gaming content in half while preserving quality...
Read More

Exploring the AV1 Image Format AVIF

Read how AV1's screen content coding tools and exceptional compression efficiency make it perfectly suited for RTC applications.
Read More

AV1 for WebRTC – x264 vs. Aurora1

Read how AV1's screen content coding tools and exceptional compression efficiency make it perfectly suited for RTC applications.
Read More

Dan Rayburn Innovation Blog Post

It's time to modernize from H.264 - 870 million users in Asia receive AV1 and HEVC encoded content every month.
Read More

Big Q4 for AV1

In Q4, Visionular took part in many activities that served to grow the ecosystem and provide momentum for the standard.
Read More
{"dots":"false","arrows":"true","autoplay":"true","autoplay_interval":3000,"speed":600,"loop":"true","design":"design-1"}

At Foothill Ventures, we believe in startup companies that ride the transformative power of major technology shifts such as deep learning in computer vision. Visionular’s founders are world-class technologists in their field of video codec and AI-driven optimization. We feel privileged to support their adventure with our resources and experience.

Dr. Xuhui Shao
Managing Partner, Foothill Ventures

We'd love to tell you more about our AV1 encoder. Hit the button so we can chat.