[HLS][EIA608] Add possibility to detect is closed captions presented in stream

[zayass]

We use ExoPlayer to support EIA608 in HLS. And cannot find any way to detect is cloased captions realy presented in specified stream.

Eia608TrackRenderer switch to PREPARED state when detect TrackInfo with mime type MimeTypes.APPLICATION_EIA608 but this TrackInfo appears in HlsSampleSource even if EIA608 cloased captions not presented in stream. But when stream type detected as h264 https://github.com/google/ExoPlayer/blob/master/library/src/main/java/com/google/android/exoplayer/hls/parser/TsExtractor.java#L337.

I think correct way to detect is eia608 track presented is observe to first SEI package with EIA608 payload so maybe here https://github.com/google/ExoPlayer/blob/master/library/src/main/java/com/google/android/exoplayer/hls/parser/SeiReader.java#L60

Do I understand correctly that is the track state(enabled, disabled, unavailable) should be determinated on the state of corresponding track renderer?

[ojw28]

Hmm. If the first caption is several minutes into the content, are there any guarantees that there will be a (presumably empty) EIA608 payload in the stream before that point (ideally the guarantee would be that it would be present in the first frame)?

[ojw28]

It looks from the HLS spec like the proper way to indicate that the stream has EIA608 captions is in the manifest, with an EXT-X-MEDIA tag of type CLOSED-CAPTIONS (not SUBTITLES, which is for distinct WebVTT subtitling), which includes an INSTREAM-ID attribute. See here.

We don't support this right now, but it looks like a better way of doing things to me.

[daynesh]

What version of the HLS spec is the ExoPlayer currently implemented against?

[AquilesCanta]

We added support for #EXT-X-MEDIA:TYPE="CLOSED-CAPTIONS" in 8965508.

[goffioul]

The solution above is HLS-only. What about other formats, like plain MPEG-TS? Is it worth opening an issue for this?

As far as I understand, the app must know in advance whether the stream will contain closed-captions. If you use default behavior, you'll end up with a closed-captions track exposed, even if the stream does not have any closed-captions. Say you don't set FLAG_OVERRIDE_CAPTION_DESCRIPTORS, DefaultTsPayloadReaderFactory will automatically inject a CEA608 track. It can be overridden by ATSC-specific descriptors, but this is not gonna happen in European TV broadcasts.

Is it possible to delay the creation of the CEA608 track until the first few frames are read? I understand the concern about "what if captions data are not present in the first frame". But from my experience, I have yet to see a US TV broadcast that didn't contain CEA608 data on every single frame (when there's no data, either cc_count is 0 or the data is filled with "fa 00 00", but the data is always present).

The only other option from the app point of view is to set FLAG_OVERRIDE_CAPTION_DESCRIPTORS and predefine the caption formats manually. But then it makes it hard to build an app that can target both US and non-US markets.

[ojw28]

Is the same also true of European TV broadcasts (i.e. that there's CEA608 data in every frame)?

[nigelmegitt]

@ojw28 I'm not aware of any European TV broadcasts using CEA608 at all - the old analogue equivalent in use in Europe is Teletext, which is still widely used, for the time being anyway...

[ojw28]

Got it, thanks. It doesn't seem unreasonable to use FLAG_OVERRIDE_CAPTION_DESCRIPTORS for the time being. It feels like targeting both markets should be as simple as an if statement. It only becomes difficult if your app is simultaneously targeting both markets and does not know what market the content being played came from, which seems somewhat unlikely in a legitimate use cases (if you believe differently, a concrete example where this would happen would be helpful).

[goffioul]

@ojw28 In Europe, you wouldn't have any CEA608 data in the stream, that means no caption SEI message, no caption user data. Those would only be present in US streams. And if there's no caption data, the SEI message / user data would still be present, but with empty data.

The situation that you describes a not a ligitimate use case is exactly mine. Detecting what's in the stream to be played should be the responsibility of the media framework. The app shouldn't have to know in advance the content of the streams.

Anyway, I've added CEA608 detection to my ExoPlayer version and now caption tracks are added dynamically, based on the actual presence of caption data.

[ojw28]

By the same argument, telling the media player what tracks are present in the stream in a sensible way should be the responsibility of the stream/container ;). I'm aware this doesn't help; just pointing it out!

Can you explain what your use case is in a concrete way? Where is the media being played coming from such that you don't know its source? It is all side-loaded?

[goffioul]

I understand your point of view, as a media framework developer, that some containers or streaming system are not straightforward to handle, because they do not provide the information or the headers that others do. As such, you may consider that because of better alternatives you don't want to support them. But my reality, as an app developer, is a bit different, I have to deal with the streaming systems that are present at my customers site, coming out of quite expensive hardware they don't want or plan to change.

We develop an app that can be deployed to play live TV broadcasts on multiple market (Australia, Asia, Europe, North America). The streams can come from different hardware from Exterity, DVEO, UCrypt... they can be raw streams, transcoded streams (in which case, the transcoding may strip CEA608 data), the hardware can be 1 year old or 10 years old, the streams can come from cable TV providers, satellite providers, digital terrestrial providers... In the end, we don't want to make the app assume what's gonna be in the stream, there are too many combinations.

[ojw28]

I think I'd argue it's working as intended that it costs you more time and effort to support a fragmented ecosystem that requires playback of legacy streaming formats. It acts as a natural forcing function that encourages upgrading to newer tech ;). Broadcasters we've worked with have typically been able to provide well constructed HLS and/or DASH streams, so it's not like doing so is out of reach.

It also makes sense for us to allocate our resources according to actual ExoPlayer usage, as well as projected future usage. The reality is that although we see quite a few issues reported about this type of use case, the actual proportion of usage these cases account for is a small and decreasing minority. Hence there's only so much time we can reasonably devote to them. We will of course continue to do our best within this time.

Sounds like you have a solution for your use case anyway, which is good news :).

[goffioul]

Sounds like you have a solution for your use case anyway, which is good news :).

Yes, I do have now proper closed-captions detection.