Video playback
Welcome to our Video playback series! This will be a series of multiple posts, where we will add more content over time.
Ahhh yes… the wonders of video playback. Who doesn’t like videos? And who doesn’t like applications with videos? Ever since the uprising of the video playback apps, we have a whole generation who associate “watching TV” with actually “streaming video inside an App” and no longer watching broadcast channels. Back in my day we hopped across broadcast frequencies over a whopping 25 channels until we found something interesting! Channel up was our app! Young punks.
Old people puns a side, if you’re building an application there is a high chance you’re thinking of video playback. Unless you’re building a music app, though the same would still apply. If you’re building a no video, no audio application like a video game or textual info app, more power to you. Have a great time.
How is this related to Lightning?
It technically isn’t directly related. Meaning the Lightning project does not provide a complex video player solution. This is for a number of reasons, mostly because Video Playback solutions are very complicated and we can only focus on so many things at once. To do this right you need a team dedicated to video playback alone. Besides if you look at the competition; React or Vue do not provide video playback solutions either. Likely for the same reasons. We’re drawing a line between “Application Dev Framework” and video playback.
So why create this? Well… it’s a very common use case. Meaning we see a lot of apps being built in Lightning that do something with video playback. This post is created to help guide and provide additional information regarding video playback on a typical device we target with the Lightning project. As there are a bunch of gotcha’s and generic advice that we hope will make you more successful with video playback in your Lightning project.
Let’s start with the most obvious about video playback:
- Video playback on an embedded device is much harder then you think it is
- Video playback takes expertise and/or investment to do it right
- The complexer your video needs, the more expensive/time consuming the investment will be
Before you dive into this I’d like you to answer few questions:
- What type of video playback do you require? Codecs and containers
- What type of video streaming will we be doing? Progressive or adaptive?
- Does it have DRM or is it in the clear?
- Will there be advertisements?
- What kind of metrics will we be collecting about your video playback?
- What kind of device and device capabilities does your target device have?
Consider this your homework. If you don’t have answers to many of the above questions, you might run into trouble down the road. It’s important to understand what kind of video we’re dealing with and what your target device can handle. Without having a full picture of the entire scope it’s going to be a very bumpy road.
Foreword
Okay before we get into the details lets start with:
Video playback on an embedded device is one the most difficult tasks of them all
Video playback on an embedded device can get really complex. Why? Because embedded devices are very bespoke, very different each time around. Surely the browser tries to unify a lot of the playback interfaces but ultimately the capabilities of the device are determined by the encoders/decoders available on the actual hardware. Why is that? Because embedded devices tend to be low powered, low resources. Because of their lack of CPU power they can not decode audio/video in software like your PC does (basically decoding video on the CPU). To compensate that these devices have dedicated hardware video decoding capabilities, specialized chips that are really good at playing video and tend to look really good on big screens (hardware scalers ftw!). The downside? they all have bespoke drivers/kernel modules to make that happen.
And each device has different ones, with different hardware capabilities. Add in a large set of different options to chose from when streaming your content with different containers, codecs and streaming protocols it quickly turns into a bit of a complexer thing.
Having one of the two (or both) options below helps tremendously:
- A skilled engineer that has a lot of experience in playback on an embedded device
- A paid video playback solution
Having #1, an experienced video engineer who’s fluent in STBs or embedded devices, is invaluable to your project. Video playback is highly specific and having someone who understands what happens beneath the browser will set you up for success. They’re out there, they’re called (video) integration or embedded engineers, they’re not cheap but will pay themselves back in full in a very short time for your project. Experience is key in making your app playback video successfully and within a short timeline.
That brings us to #2, a paid video playback solution. “But I like open source”… 9 out of 10 times this probably means something else. You like free, but “free” isn’t always really for free. Sure an open source video playback solution doesn’t require a license and thus is “free”. But that does mean you need to configure/fine tune and tweak the solution to your exact needs. And that is where #1 comes in, if you want to go open source and have option #1 you will be fine. If you don’t, you should highly consider getting a paid video playback solution because they will take care of #1 for you and you can negotiate support for your video playback needs as part of the commercial engagement. Making it “someone else’s” problem is great if you do not have an experience or engineering to do it yourself.
Trying “wing it” with an open source video player without experience or know how of what you’re doing for video playback on embedded systems is not something we would encourage. Unless your deadlines don’t mean a thing and you have all the time in the world to figure this out (and inherently build that experience as you go). You will run into issues and without having the expertise on how to deal with that it will be a bumpy road full of meaningful learnings that will take up a lot of your time.
Lastly… do you own the video source? The distribution pipeline or do you consume the content from another party? If you own the pipeline you can typically have more freedom by adjusting encoding profiles and such to meet the needs of the device. This is less of a problem with newer devices as they tend to support a wider variety of audio/video profiles. But may help if you get stuck, sometimes it is cheaper to adjust the encoder/packager slightly versus trying to spend months to make it work with the software you have or fiddle with the drivers on the device itself. If you don’t own the video distribution channel no worries, it’s a last resort and you might be fine dependent on how finicky the device is that you are dealing with is.
Chapters
- The basics of video playback
- Video playback on an embedded device
- The Video element
- Media Source extensions
- DRM and Encrypted Media extensions
- Native players
- Conclusions