A quick evaluation of AWS AV1
On March 20, AWS announced its video transcoding support for AV1, the royalty free high compression video codec, as part of Elemental MediaConvert service (source).
As of May 21, 2020, MediaConvert AV1 supports resolutions up to HD (1080p) and is priced at about 20X of H.264 (2-pass), and 2.5X of HEVC (2-pass quality optimized):
The price is probably proportional to the computation resources consumed for each codec, plus a first-to-market premium for AV1.
Naturally we are curious about it’s performance in terms of compression and speed, and so set off to do a quick experiment.
Test Configuration
I picked a 1080p test video that is particularly hard to compress and full of visual sensitive elements: Crowd Run (source). (I favored this video even more after an ex-colleague revealed his presence as one of the runners!)
I ran MediaConvert AV1 jobs to transcode the test video at four target bitrates: 2Mbps, 4Mbps, 6Mbps, and 8Mbps. QVBR was used for rate control and all the other options are left unchanged.
Transcoding Speed
Most video services care about how fast transcoding can be done. For user generated content platforms, video uploaders expect to see their videos published instantly and transcoding delay becomes an important part of the user experience. For premium content or professionally generated videos, transcoding speed is also important since those videos are typically long and encoded at slower speed setting.
Encoding speed is also one of the essential dimensions for encoder evaluation (besides bitrate and quality). Most encoders provide multiple speed modes where slower modes run more complex algorithms to squeeze more compression at the cost of higher computation cost.
The transcoding times for my AWS AV1 jobs are shown below:
The source video is 6 seconds long with 150 frames (25 fps). Thus the average transcoding speed is about 2.4 fps, or ~1/10 of “real time”.
Compression Performance
It’s hard to say anything about bitrate or quality without comparing them to a baseline. So I also encoded the same video with x264 using the following options:
-preset veryslow -g 60 -crf 23 -maxrate 2000k -bufsize 4000k
-preset veryslow -g 60 -crf 23 -maxrate 4000k -bufsize 8000k
-preset veryslow -g 60 -crf 23 -maxrate 6000k -bufsize 12000k
-preset veryslow -g 60 -crf 23 -maxrate 8000k -bufsize 16000k“-g 60” is to make keyframe placement consistent with AWS AV1.
The combination of “-crf”, “-maxrate”, and “-bufsize” enables constrained VBR rate control to match AWS AV1's QVBR, which is basically constant quality with maximum bitrate constrained.
“-preset veryslow” is practically the slowest preset x264 has (ignoring “placebo” which is never used in practice). It is still faster than AV1: on my 8 core 3.6GHz Intel i7–7820 CPU, x264 encodes 9–11 frames per second (4–5 times faster than AWS AV1).
To compare compression efficiency, I plotted the quality vs. bitrate curves for PSNR, SSIM, and VMAF as below:
For all the three metrics, AWS AV1 consistently delivers higher quality at the same bitrate. The margin is more prominent for PSNR than SSIM or VMAF: approximately 15% to 35% bitrate saving at the same PSNR, or 0.6~1 point PSNR gain at the same bitrate.
How about the subjective quality? Taking the 6Mbps output videos for x264 and AWS AV1 as an example, since their actual bitrates are the closest to each other.
The x264 output definitely has more visible compression artifacts, while the AV1 output is smoother and looks more natural overall.
This is not surprising. AV1 is expected to have better quality than x264 at same bitrate. Now let’s put x264 8Mbps next to AV1 6Mbps:
Although x264 still has many compression artifacts standing out, the sharper details of x264 is also noticeable compared to the blurriness and deformation of AV1. I’d say it’s a much closer call now.
Here you have my anecdotal evaluation of AWS MediaConvert AV1. It’s only one video being tested but intentionally chosen to give the encoders a real challenge. Adoption of new codecs is mostly driven by hard problems that cannot be solved by existing codecs after all. So I believe the evaluation is not thorough but meaningful nevertheless.
Putting everything together: modestly higher compression efficiency, longer transcoding time, and high transcoding $cost, I feel AWS AV1 is probably not a compelling option to upgrade from h.264 or HEVC yet.
But the AV1 community is growing strong and moving fast. Actually, libaom 2.0 was just released this week, running tens to hundreds times faster than libaom 1.0! The landscape may be very different if we look again after 6 months.
Exciting time.
