TV broadcast streams on Firefox OS products

This article discusses the mechanisms we have made available on Firefox OS for processing and displaying TV broadcast streams.

TV is an emerging platform in the web arena — it has only recently started leveraging web technologies. We currently treat the Web on TV as an exciting challenge and expect Smart TVs to feature more interesting interactive web app-based experiences as time goes on. Firefox OS gives platform providers a new opportunity to embrace open web technologies and run existing web apps across more platforms.

Currently however, the most common usage of Smart TVs is still TV broadcasting, which requires us to implement platform-specific features for Firefox OS support.

TV broadcasting streams

TV broadcasting stream technologies have already been thoroughly developed the past and have become a competitive point among TV manufacturers. Those technologies involve engineering effective AV sync of media streams on TVs, which means in order to provide a smooth viewing experience with an optimal frame rate, manufacturers have to tune the performance of hardware decoding and rendering.

Bringing TV broadcasting streams to Firefox OS

TV broadcasting streams are not natively supported on Firefox OS, which has its own rendering pipeline originally designed for Web browsers. Because of this, Firefox OS currently can’t support a hardware composer, required for TV manufacturers to utilise hardware acceleration when decoding and rendering TV broadcasting streams. In addition, because such existing technologies have been very reliable, there is currently little incentive for them to abandon existing technologies.

In Firefox, Gecko is responsible for handling both the rendering pipeline, and the process of rendering the current frame result into the framebuffer. However, TV manufacturers’ hardware renders have their own framebuffer as well. These two framebuffers are independent and could block each other. To solve the issue, we decided to implement DomHwMediaStream, a subset of DomMediaStream.

As the above diagram shows, the API is designed to create an overlay image and add it into a MediaStream’s primary video track. Its size can be adjusted to meet TV programs’ specifications. In addition, because DomMediaStream is the DOM wrapper for MediaStreams, it already has some basic functionality for handing TV broadcasting streams built in. Developers can therefore leverage these existing APIs to access relative information.

Let's look at a code example:

// Retrieve all the available TV tuners
navigator.tv.getTuners().then(
    function(aTuners) {
      for ( var i = 0; i < aTuners.length; ++i ) {
          var tuner = aTuners[i];
            // While the source of tuner changed, get the newest info of this mediaStream
          tuner.oncurrentsourcechanged = function(aEvent) {
            getTracksInfo(tuner.stream);
        };
      }    
    },
    function(aError) {
      console.error( "Fail to get tuners: " + aError );
    }
);
// Retrieve all the tracks information.
function getTracksInfo(aStream) {
    var tracks = aStream.getTracks();
    for ( var i = 0; i < tracks.length; ++i ) {
        var track = tracks[i];
        console.log( "track info id: " + track.id );
        console.log( "track info kind: " + track.kind );
        console.log( "track info label: " + track.label );
    }
}

In DomHwMediaStream, OverlayImage is handled directly by the GFX compositor. Because OverlayImage — essentially a transparent layer — will be rendered to the framebuffer, users can see TVs framebuffers through Gecko’s framebuffer.

Advantages and Disadvantages

This approach has both advantages and disadvantages. Let's look at the advantages first.

• Since we do not handle AV sync in Gecko, TV manufacturers are able to continue using their AV sync module and hardware render alongside it, as before.
• We can invite more content providers to leverage Web APIs, allowing web developers to use familiar web methods to create TV-related apps. For example, the MediaStream API provides a MediaStreamTrack interface for accessing VideoStreamTrack and AudioStreamTrack. In addition, the MediaStream API defines other interfaces suitable for manipulating broadcasting streams. If there is any special requirement requiring a modified API, manufacturers can inherit the DOMMediaStream object and add/override.
• When using our own API, we don't need to copy pixel data from the hardware renderer’s framebuffer to Gecko’s framebuffer.

Now let's talk about the disadvantages:

• The most obvious one is that we cannot apply CSS effects on TV broadcasting streams. As mentioned above, because the Gecko and TV hardware framebuffers are independent, we are not able to support CSS effects for TV broadcasting streams unless we copy the pixel data of the TV’s framebuffer to Gecko’s layer, which will cause a performance hit.
• The other one is Web Audio. Currently, we don’t support the Web Audio API for TV broadcasting streams because TV audio tracks are processed (decoded, A-V synced) in the TV hardware, not in Gecko (we are looking to solve this in the future.) The Web Audio API could allow developers to provide users with simple but powerful controls, for example volume and switching audio channel layouts, up to advanced functions such as mixing audio channels or cross fading between TV channels.

Conclusion

The Web's strength is as a ubiquitous delivery platform, but we should remember that some platforms still require unique functions. For TVs, an effective broadcasting stream is vital, therefore we have designed a rendering path that allows both web and TV content to thrive in the same environment, attracting more developers to join this ecosystem. Regarding CSS effect support in the future, we are aiming to implement a hardware composer on which TV manufacturers can directly render streaming images. This way, the hardware composer would make it possible for manufacturers to customize their screen effects.

Although TV is a relatively fresh device for web developers, we envisage that the capability of cross-platform web technologies will bring more opportunities for innovation.