Support PNG subtitles in SMPTE-TT

[fguinet]

Currently the ExoPlayer code assumes subtitles/text-tracks are always receive in text format (TTML).
The metadata stream of the transport stream may contain SMPTE-TT data, an extension of TTML that extends TTML with binary blobs, encoded in base64.
The base64 data are part of the TTML xml structure and may for example contain bitmap subtitles or be used to tunnel other formats.
Is it plan to implements SMPTE-TT + PNG support including the display of the base64-decoded bitmap subtitles in the player ?

[rvergara1]

Same issue #179

[AquilesCanta]

As stated above, this is a duplicate of #179. Thanks, @rvergara1.

[ojw28]

Re-opening specifically to track support for bitmaps inside of SMPTE-TT. Marking as an enhancement. Realistically, I don't think we'll be looking at this any time soon, so if someone wants to work on a pull request please feel free. It's probably not too hard to add support now that plumbing of bitmaps through Cue and SubtitlePainter is in place.

[ghost]

Hi @ojw28

Just to weigh in on the use-cases here.

Many of the source channels for live TV ONLY deliver subtitles in image based DVB-SUB format.

Since we don't have the luxury of being able to deliver a text based format on many channels and it cannot be be guaranteed that there will be capability in the head end to provide OCR based conversion from image rendering to text - thus SMPTE-TT is considered a 'must have' for OTT/MPEG-DASH delivery of live TV in Scandinavia and Eastern Europe.

In saying that, If we were to have a go at implementing this, I guess we would need to add extraction /decoding of TTML metadata to TtmlDecoder? Any other gotchas in there?

[ojw28]

Yes, I think it's just a case of adding support in TtmlDecoder (and related classes in the text/ttml package). I don't think there are any gotchas, and you shouldn't need to change anything outside of that package.

[nigelmegitt]

SMPTE-TT is considered a 'must have' for OTT/MPEG-DASH delivery of live TV in Scandinavia and Eastern Europe

A valid and possibly more current alternative would be IMSC's Image Profile, which is similarly a variant of TTML and includes some SMPTE-TT concepts. It is also under consideration for CMAF.

[JerCodeWhisperer]

Hi, we are looking to use this feature. At the present time is it being worked on or is it planned?

[szaboa]

@ojw28 @AquilesCanta

I intend to work on this, is it all right for you?

What I see after quickly checking the code that the image attribute from the header needs to be parsed, then I need to create a bitmap from the base64 string and just pass forward to the cue.

Am I missing something perhaps?

[ojw28]

@szaboa - Feel free to give this a go!

Re your question, I'm not sure as we haven't had time to properly investigate, but what you describe sounds plausible! The main thing I'm not sure on is how the implementation will fit into the existing TTML code, which is relatively complex (e.g. TtmlNode). Figuring that out is probably the main challenge.

[szaboa]

I think I am on the right track, the sub is already displayed. But need to research a bit the TTML specs, because I am not sure if image based subs can have regions, and also I would like to extend the TTML decoder unit test cases.

[nicusorflorin]

Will this allow IMSC subs to be rendered as well?

[ojw28]

@szaboa has implemented support for this (thanks!), which is now available in the dev-v2 branch. @szaboa - Can this issue be closed, or is there more work still to be done?

[szaboa]

@ojw28
Np, looking forward to contribute more.

From my side this issue can be closed. If there is a demand for IMSC image subs then I can do it under another issue in the future.