2021 MSU Codec Study Subjective Results
AV1 codec implementations compared.
- The Visionular Aurora1 AV1 encoder beat all AV1 implementations for the third year in a row using both subjective and objective quality metrics.
- Aurora1 beat all encoder implementations for the quality and efficiency use-case based on subjective viewing evaluation.
- Aurora1 enables services to save more than 60% compared to x265 reference encoder while maintaining perceptually identical visual quality.
challenge.
The annual Moscow State University Codec Study completed by the Graphics and Media Lab Video Group represents the best round up of video codec implementations in the video streaming and delivery industry. Video encoding engineers worldwide rely on this comprehensive and objective study to guide their codec selection decisions and the best implementation for their use case.
Visionular first participated in the 2019 study where our Aurora1 AV1 encoder came out on top for all objective metrics, a remarkable achievement, given the implementation was new. In 2020, we also ranked at the top as the best performing encoder.
For this year’s study, 2021, we again participated. The competition was steep, and yet, Aurora1 once again beat all competing AV1 encoder implementations and ranked a close second only to the S266 encoder using subjective viewing. We also joined the winning group for the Best Speed-Quality tradeoff.
method.
This immense effort to evaluate 16 video encoders covering H.264 AVC, H.265 HEVC, AV1, VP9, and VVC utilized 15 video sequences at Full HD resolution. The process of video sequence selection involved voting among the participants, organizers, and an independent expert. The MSU team analyzed more than 1,500,000 video sequences and selected representative examples to comprise the test set.
Video encoder selection and codec standards are dependent on the use case. What works well for VOD may not be appropriate for a live or real-time deployment. Thus, the MSU comparison consists of two parts, corresponding to fast encoding and offline encoding (Fast and Slow).
For each use case, codec developers could provide encoding parameters. If a developer declined to provide codec parameters, evaluators either used the same parameters from our prior study or, if none were available, they did their best to choose the optimum parameters for the test. Each encoder tested had to satisfy the minimum speed requirement for their respective use case as follows while encoding at 1080p resolution.
- Online – Fast (30 fps)
- Offline – Slow (1 fps)
All comparisons used a machine based on an Intel Core i7-8700K (Coffee Lake)processor @ 3.7GHz with 32 GB of RAM running Windows 10 or Ubuntu. The YUV-SSIM metric was used as the primary objective quality indicator, though PSNR and VMAF were also measured.
Researchers used the Subjectify.us crowdsourcing platform and involved more than 10,800 participants where they received 529,171 pairwise answers. The Bradley-Terry model was used to compute global rank.
results.
Ranked subjectively, Aurora1 beat rav1e and QAV1 including VP9 and popular HEVC implementations.
To understand how the results were tabulated, you can view a free abridged version of the report by clicking here.
The subjective scores were generated using Subjectify.us a platform that enabled the researchers to conduct subjective comparisons of video-processing methods to determine the best performing encoders based on the operational criteria. Watch the Subjectify video here.
Rate-Distortion Curves
Relative-bitrate shows the average bitrate dependence on relative encoding time for a fixed quality output. The y-axis shows the ratio of a codec’s bitrate under test to the reference codec’s bitrate for a fixed quality.
A lower value indicates a better-performing codec. For example, a value of 70% means the codec can encode the sequence using 30% fewer bits than the reference codec (x265 was the reference used for this study).
The x-axis shows the relative encoding time. Larger values indicate a slower codec. A value of 2.5 means the codec works 2.5 times slower, on average than the reference codec (x265).
Judging from the mean quality scores, first place based on quality went to S266, second place to Aurora AV1, and third place to QAVS3. Notably, VVC and AVS3 are considered niche codecs because of their limited hardware decoder support, while AV1 device penetration is growing rapidly.
Aurora1 AV1 encoder achieved:
- 60% reduced bitrate at the same quality over the Reference x265 using the Subjective metric.
- 54% reduced bitrate at the same quality over Reference x265 using Y-VMAF.
- 139% advantage over SVT-VP9 using the Subjective metric.
- Achieved a 15% bitrate savings over QAV1, and a massive 53% savings over rav1e.
- Aurora1 was the best performing codec for quality/speed (Y-VMAF) while ranking second overall using the Subjective metric.
The following graph compares coding efficiency of the Aurora1 AV1 encoder with the other AV1 implementations in the study, as well as x265, and VP9 using the VMAF quality metric.
The following graph compares all encoders tested across all content for the 1fps use-case using the subjective, Y-VMAF, and YUV-SSIM comparison metrics.
