Which Media Formats To Use
Audio
Use MP3: it’s supported by all browsers: Chrome, Edge and Safari on desktop, and Chrome and Safari on mobile. In native apps on all OSs — Windows, macOS, Android and iOS. And in hardware like car stereos, speakers with USB ports, home stereos, portable music players, and a lot of long-tail devices. If it plays music, it plays MP3.
MP3 has already surpassed the abilities of the human ear in frequency, dynamic range and other ways, so using a newer format doesn’t let you encode music in a way that sounds better than MP3 can achieve.
MP3 is also patent-free.
While MP3 is the universal codec, the best codec is Opus — it produces files half the size of mp3 for the same quality. It also has better quality than other codecs like AAC and Vorbis, which are themselves better than MP3.
Opus has low latency, so it works well for live calls, where you don’t want the other side to hear you a second after you’ve spoken. WhatsApp uses Opus to have an acceptable quality even on very low bandwidth cellular connections. As does Youtube:
Opus is supported in Chrome, Edge and Chrome for Android. It’s supported natively in Windows 10 (from May 2019) and Android (from Sep 2019). And in players like VLC and in media editing frameworks like FFmpeg. And in some transcoding services like Bitmovin.
The fly in the ointment is that Opus isn’t supported by Apple devices, whether Safari, iOS Safari, iOS native apps or Mac native apps. So, if you use Opus, you have to fall back to some other format, as Youtube does when playing in Safari. Or bundle your own copy of the Opus codec, which makes sense for (say) the Spotify mobile app. At scale, these options are worth it, but for most people and companies, MP3 is good enough, and audio files small enough, that the complexity of trying to optimise it further isn’t justified.
Opus files can have different extensions like .opus, .mka, .webm or .m4a. This is because Opus is a codec, not a container format. Opus needs to be stored in a container format like Maktroska (with a .opus or .mka extension), WebM (with a .webm extension) or MP4 (with a .m4a extension).
Don’t re-encode your existing mp3s into Opus, since applying lossy compression to an already lossily compressed file will degrade it.
If you want a lossless format, use FLAC. It won’t sound any better, but avoids generation loss when saving the result of editing for further editing. Other than such narrow use cases, FLAC isn’t the right format.
Video
Use H.264. It’s supported in all browsers — Chrome, Edge and Safari on desktop, and Chrome and Safari on iOS. It’s also supported in native apps on all OSs — Windows, macOS, Android and iOS. All cameras and video editors support it. And all kinds of long-tail use cases. If it plays video, it plays H.264.
Unfortunately, H.264 is patent-encumbered.
If you want better compression than H.264, and are willing to sacrifice universal compatibility, instead settling for compatibility with the latest versions of browsers (Chrome, Edge, Safari) and OSs (Windows, macOS, Android, iOS), use VP9. Players like VLC support it, as do cloud encoding services like Amazon and Bitmovin. VP9 has been designed to be decoded efficiently without hardware support, and has let Youtube support UltraHD since 2015. On modern hardware, VP9 performance is excellent: my 2020 iPad Pro can play back VP9 at UltraHD with HDR at 60 FPS. Even my five-year old 27-inch iMac can play it at Ultra HD with HDR, albeit at 24 FPS.
If you want better compression than VP9, and are willing to sacrifice some more compatibility, use HEVC. HEVC is supported in Safari, iOS Safari, macOS and iOS. On VLC and FFmpeg. And in cameras like the iPhone, some Android phones, and GoPros. HEVC is not supported in Chrome, Edge or Chrome on Android. Or natively on Windows 10 or Android.
If you want the best compression, with even less compatibility, use AV1 [1]. AV1 is supported on Chrome and Edge, Android, playback software like VLC, media editing frameworks like FFmpeg, and media transcoding services like Bitmovin. But not Safari, Safari on iOS or Chrome on Android. Or on Windows, macOS or iOS. So you wouldn’t be wrong if you considered AV1 a format for the future, not for today.
Images
For images, use PNG — it’s lossless, and supports transparency, 16 bits per channel color depth and animations.
For photos, you can use JPEG to achieve smaller files, using lossy compression, which looks equally good to the eye. JPEG doesn’t support transparency or animation. It’s limited to 8 bits per channel, which prevents HDR.
If you want a more modern format, use HEIC, which can encode a burst of photos captured using your camera, or a cinemagraph like
… in one file. Not only that, they compress better than individual files. You can also use this for exposure or focus stacking, and for 3D photos, storing both the left and right photos in one file. Or to store the depth for each pixel. A HEIC file can have an instruction to tell the viewer to display these individual frames one after another, for animation. HEIC can store the same image at different resolutions. It can also split an image into tiles, which allows you have a 100-megapixel panorama that’s incrementally decoded as you pan around. HEIC supports auxiliary data like depth maps and even audio. And thumbnails. And transparency. And high dynamic range.
HEIC can be hardware-accelerated. On Apple hardware, it uses the HEVC hardware encoder and decoder, because HEIC is HEVC. It uses the same codec but for a still image.
Leaving all these features aside, HEIC also compresses to half the size as JPEG. A 12 megapixel HEIC photo on the iPhone takes only around 1 MB. HEIC also supports lossless compression, which is again smaller than PNG. So, whether you want lossy or lossless compression, you’ll save space by using HEIC over JPEG or PNG.
Unlike legacy image formats like JPEG and PNG, modern ones like WebP, HEIC and AVIF are derived from video codecs (VP9, HEVC and AV1, respectively).
[1] Not be confused with the legacy AVI format. The new one is AV ONE. The naming is stupid, since it causes needless confusion.