Support additional passthrough audio formats

[ebr11]

I'm having trouble getting Exo (2.0.4) to play mkv files with AC-3 or DTS (DCA) audio. This is on a device that should support it (Nvidia Shield hooked to an AVR with support for those and more).

All my searches have come up with the basic answer that this error indicates an unsupported audio track but, given the above hardware, I don't understand how that could be. Is there some sort of additional configuration necessary to make pass thru of these tracks to the AVR possilble? I've just used the default everything from the example/demo app.

Thanks for any direction...

12-02 16:44:33.601 E/AudioTrack: AudioFlinger could not create track, status: -22
12-02 16:44:33.605  E/AudioTrack-JNI: Error -22 initializing AudioTrack
12-02 16:44:33.605  E/android.media.AudioTrack: Error code -20 when initializing AudioTrack.
12-02 16:44:33.620  E/ExoPlayerImplInternal: Renderer error.
                                                                      com.google.android.exoplayer2.ExoPlaybackException
                                                                          at com.google.android.exoplayer2.audio.MediaCodecAudioRenderer.processOutputBuffer(MediaCodecAudioRenderer.java:345)
                                                                          at com.google.android.exoplayer2.mediacodec.MediaCodecRenderer.drainOutputBuffer(MediaCodecRenderer.java:890)
                                                                          at com.google.android.exoplayer2.mediacodec.MediaCodecRenderer.render(MediaCodecRenderer.java:477)
                                                                          at com.google.android.exoplayer2.ExoPlayerImplInternal.doSomeWork(ExoPlayerImplInternal.java:431)
                                                                          at com.google.android.exoplayer2.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:265)
                                                                          at android.os.Handler.dispatchMessage(Handler.java:100)
                                                                          at android.os.Looper.loop(Looper.java:148)
                                                                          at android.os.HandlerThread.run(HandlerThread.java:61)
                                                                          at com.google.android.exoplayer2.util.PriorityHandlerThread.run(PriorityHandlerThread.java:40)
                                                                       Caused by: com.google.android.exoplayer2.audio.AudioTrack$InitializationException: AudioTrack init failed: 0, Config(48000, 252, 49152)
                                                                          at com.google.android.exoplayer2.audio.AudioTrack.checkAudioTrackInitialized(AudioTrack.java:909)
                                                                          at com.google.android.exoplayer2.audio.AudioTrack.initialize(AudioTrack.java:461)
                                                                          at com.google.android.exoplayer2.audio.MediaCodecAudioRenderer.processOutputBuffer(MediaCodecAudioRenderer.java:337)
                                                                          at com.google.android.exoplayer2.mediacodec.MediaCodecRenderer.drainOutputBuffer(MediaCodecRenderer.java:890) 
                                                                          at com.google.android.exoplayer2.mediacodec.MediaCodecRenderer.render(MediaCodecRenderer.java:477) 
                                                                          at com.google.android.exoplayer2.ExoPlayerImplInternal.doSomeWork(ExoPlayerImplInternal.java:431) 
                                                                          at com.google.android.exoplayer2.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:265) 
                                                                          at android.os.Handler.dispatchMessage(Handler.java:100) 
                                                                          at android.os.Looper.loop(Looper.java:148) 
                                                                          at android.os.HandlerThread.run(HandlerThread.java:61) 
                                                                          at com.google.android.exoplayer2.util.PriorityHandlerThread.run(PriorityHandlerThread.java:40)

[ebr11]

Okay, I may have answered my own question - if you guys can just confirm if this is correct and we can close this...

I was using the example in the docs to create my media source and I noticed it wasn't passing in a handler or event logger but the demo app was. I added these to my constructor call and now DD and DTS are passing through as they should.

I suspect the Handler was what was really needed. Is that correct?

[andrewlewis]

One possible explanation for passthrough not working is not passing an AudioCapabilities instance to the renderer, but that seems to be different from the change you're describing (and SimpleExoPlayer does this already, in case you're using that).

Could you paste the exact diff and we can take a look? Thanks.

[ebr11]

Yes, I am using the SimpleExoPlayer and I just changed the parameters back to null that I changed before and the audio is not throwing an error anymore so I do not have an explanation at this point.

I think Exo must have determined that the audio should work or it wouldn't even have tried to send it, correct? So I don't know why it failed with the above errors and I also don't know why it doesn't now - unless it was just some sort of HDMI handshake issue or something.

As a related question, should TrueHD and DTS-HDMA work this way as well? The AVR supports them but Exo is passing through just core DD and DTS instead of the lossless versions.

[andrewlewis]

Yes, we check the audio renderer capabilities before trying to use it, so the error is unexpected. Is it possible that some override was in place that caused ExoPlayer to try to use passthrough even if the HDMI connection didn't advertise it (e.g. an Android TV platform setting to force passthrough)? If you can find some way to reproduce the original error, please provide some instructions so we can investigate further.

We don't yet support TrueHD though we might add support soon as AudioFormat.ENCODING_DOLBY_TRUEHD was added in API 24. I'm not sure about DTS-HDMA. My understanding is that ENCODING_IEC61937 can be used to support arbitrary passthrough formats. If the original issue does not need further investigation I will mark this as an enhancement to add support for those encodings.

[ebr11]

Yes, support for passing through those HD formats is my primary goal and I cannot seem to reproduce the above error.

Given a DTS-HD-MA or other bitstream format track, how would I tell Exo to treat it as IEC61937?

Also, would the ffmpeg extension get me pass-thru on these formats or would they be software decoded by ffmpeg?

Thanks very much for your help.

[andrewlewis]

MediaCodecAudioRenderer is used for passthrough. FfmpegAudioRenderer should always decode to PCM.

Support for passthrough of other formats using IEC 61937 is an enhancement. I will use this issue to track that. Thanks!

[ebr11]

On the DTS-HD MA issue, it is partially supported in that audio plays. However, it is only sending the Core DTS stream, not the lossless MA one even though the device does report that it supports DTS_HD.

Not sure if it is relevant but I can see in the track map that there is no difference to Exo (at least reported here) between a DTS HD stream and a straight DTS one. The below map contains two audio streams. The first one is DTS HD MA and the second one is straight DTS but, in the map, they are identical and, when played, the HD track comes through as straight DTS.

Hopefully, this is helpful. Thanks.

Tracks [ Renderer:0 [ Group:0, adaptive_supported=N/A [ [X] Track:0, id=1, mimeType=video/avc, res=1920x1080, supported=YES ] ] Renderer:1 [ Group:0, adaptive_supported=N/A [ [X] Track:0, id=2, mimeType=audio/vnd.dts, channels=6, sample_rate=48000, language=eng, supported=YES ] Group:1, adaptive_supported=N/A [ [ ] Track:0, id=3, mimeType=audio/vnd.dts, channels=6, sample_rate=48000, language=eng, supported=YES ] ]

[andrewlewis]

Was the stream extracted from a Matroska file? That container uses the same codec identifier for DTS and DTS-HD. Currently we just assume DTS for Matroska tracks. To make this work properly we'd need to inspect the samples to distinguish these two formats, which is inconvenient in the current model.

[ebr11]

Yes it is in an MKV. If I use MKV ToolNix I can see the difference (that's how I confirmed I wasn't just feeding it bad data).

In our processes that inspect these tracks, for DTS it is almost always "DCA" and then you have to look at the profile value to see that it is DTS or DTS-HD.

Is there anything I can do to make it pass the actual HD track?

[andrewlewis]

Does it work if you just specify MimeTypes.AUDIO_DTS_HD in the MIME type in MatroskaExtractor.java?

I'm afraid we don't have a sensible way to do this other than forking the Extractor at the moment. You can pass your own ExtractorsFactory to ExtractorMediaSource.

[ebr11]

YES!

Okay, so now I just need to know what properties may exist to allow me to determine it is actually an HD track. Of course, if I use my modified extractor for a non HD track it fails.

EDIT: Since I have this information in my app, I just am handling it from there. Thanks for the tips.

[ebr11]

Hi. Is there any ETA on when we might get native support for passing through these advanced audio codecs? Thanks.

[andrewlewis]

Support for ENCODING_IEC61937 is not being prioritized at the moment as we haven't seen a lot of demand for it so far. If you're asking about something different, please clarify.

[cncb-gh]

You can pass your own ExtractorsFactory to ExtractorMediaSource.

Sorry if this is the wrong place to ask this, but could you please explain how to do this in more detail to have the DTS-HD track play instead of just DTS? Thanks.

[andrewlewis]

@cncb-gh Copy MatroskaExtractor and add in your logic for determining whether the content is DTS or DTS-HD. (Note: setting DTS-HD all the time will likely cause DTS streams to fail to play, so you'd need some way to distinguish between the formats if you want to support both.) Then, when building an ExtractorMediaSource pass in your own ExtractorsFactory that returns a new instance of your forked extractor. Here is the required setter.

[cncb-gh]

@andrewlewis Thanks very much for your help. Would it be feasible for there to be a "flag" for the MatroskaExtractor (set in DefaultExtractorsFactory.setMatroskaExtractorFlags()) to indicate that it is a DTS-HD track to avoid having to fork the MatroskaExtractor?

Also, is there a place to get general help with ExoPlayer? I have not been able to locate a discussion board or something similar.

[ojw28]

A flag sounds pretty hacky. We should just improve the extractor so that it's able to determine the mime type correctly (without any external input). Unless that's completely infeasible for some reason.

[ebr11]

That would be awesome because then we could remove our hack that supports this :).

[cncb-gh]

Hacky, yes, but maybe less hacky than everyone copying/forking the whole class just to change one thing? And since this is low priority maybe a flag would be reasonable in the meantime until the extractor could be improved? Thanks.

[andrewlewis]

@ebr11, @cncb-gh, @IanDBird, @drhill Any ideas if there's signaling in Matroska containers that distinguishes DTS and DTS-HD? The codec ID always seems to be A_DTS. The only way I'm aware of to address this properly is to start reading an audio sample and parse it to check which type of stream is in use, which is likely to be quite complicated (not sure if it could be classed as infeasible though!).

[ebr11]

Hi. The codec is always the same but the codec_profile (as read by ffprobe or similar) will be different for the HD version(s).

[cncb-gh]

Sorry, I can't help other than to say that MediaInfo is able to identify the "profile" correctly so this might be some code to see how to do it: https://github.com/MediaArea/MediaInfo

[IanDBird]

My understanding is that the container will not specify the profile used, and only the codec. Therefore the only reliable way to detect any DTS extension is to detect it from an audio sample.

[cncb-gh]

Any progress on this? I am not developing in Java, so a simple copy of MatroskaExtractor is not feasible for me. Adding some kind of flag in DefaultExtractorsFactory.setMatroskaExtractorFlags() would still be helpful for me.

[andrewlewis]

@cncb-gh No progress I'm afraid. The option of adding a flag was considered above but we decided it was too hacky, and that instead we should identify the media.

Looking at the code again, I think one option that would work and not be too obtrusive would be to output a new format for the DTS-HD track once we've started reading the first sample and have identified that it's indeed DTS-HD instead of DTS. I tried this out by hard-coding a format change on reading the first audio sample from DTS to DTS-HD and it seemed to work (the A/V receiver correctly showed the new format). The remaining piece to submit this workaround would be having a way to identify that the track is DTS-HD by looking at the start of the first sample. If someone on this thread knows what's needed to distinguish the two formats based on one access unit, please leave a comment and we can get this fixed. Thanks.

[cncb-gh]

I'm sorry that I don't have the technical knowledge to help with this, but were you able to work this out? Thanks.

[andrewlewis]

I'm afraid not. We still need info on how to identify DTS vs DTS HD based on the start of the first sample (if this is indeed possible).

[ebr11]

What about the possibility of just raw passthru using IEC 61937? Any more information on how that may be done?

[andrewlewis]

I'm afraid I don't have any information about how to do that. Perhaps the AOSP sources under audio_utils/spdif can help but I haven't looked at them in detail.

One concern I have with the IEC 61937 approach is future-proofing for new audio formats: there doesn't seem to be an API to get access to all possible encodings supported by the receiver including those that are not yet defined in the current Android build's AudioFormat encoding constants. So, as an example, although we can in theory do passthrough for arbitrary formats on an API 24 build I'm not sure we can detect that TrueHD passthrough is possible as the encoding constant for that was added in API 25. One way to fix that would be to expose the raw HDMI EDID information to apps. If anyone has ideas about how to make this work please reply here. (Asking the audio framework team about this is tracked by [internal: b/131736540] and I will post a follow-up here if I get any more information.)

I suggest we continue to use this issue to track using IEC 61937. I've filed #6225 to keep the DTS-HD discussion separate as it's mostly independent of this.

[ebr11]

Sounds good. Thanks very much.

[andrewlewis]

Apparently there is a limitation in the IEC 61937 implementation that high bitrate formats are not supported [internal: b/128866205]. So to address this fully we'd need that to be fixed in addition to allowing apps to query HDMI EDID audio descriptors.