Multiple audio streams in audioTracks property

[deadbeef84]

Is your feature request related to a problem? Please describe.

I'm using HLS.js together with broadcast media, and typically our media has 1 video stream and 8 mono audio streams. When playing the media, I need to be able to play all streams simultaneously. Unfortunately I haven't been able to make hls.js play all streams if I separate them into 8 streams, so as a workaround I've merged them into a single 8 channel stream. To do this I have to set a 7.1 channel layout, which creates several other issues (inconsistent channel order, compatibilty issues with other players/NLEs).

Describe the solution you'd like

I'm not exactly sure what the HLS spec allows, but what I'm looking for is to have 8 mono audio tracks that I can access using the video elements audioTracks property.

Additional context

I'm using the .ts file for other purposes than hls, which means it has to be a single file containing all streams.

[robwalch]

Hi @deadbeef84,

When playing the media, I need to be able to play all streams simultaneously.

To clarify, you want HLS.js to adopt audioTracks, not only to switch between an active track like with Safari HLS playback but also to enable and mix multiple audio tracks? I'm not sure that is possible. With Safari HLS playback, enabling one audioTrack disables the others.

I'm not exactly sure what the HLS spec allows, but what I'm looking for is to have 8 mono audio tracks that I can access using the video elements audioTracks property.

In the HLS spec EXT-X-MEDIA audio are alternatives of the same content. They are not tracks to be mixed.

https://datatracker.ietf.org/doc/html/draft-pantos-hls-rfc8216bis#section-4.4.6.2.1

"The EXT-X-MEDIA tag is used to relate Media Playlists that contain alternative Renditions of the same content."

While HLS.js does not create audioTracks for each audio option available in an HLS asset, it does allow switching of the active track by setting hls.audioTrack to an index of hls.audioTracks. In v1.5, you can configure audioPreference and change the active track using hls.setAudioOption(selectionPrefs_or_hlsTrackObject) (#5532).

[robwalch]

I'm using the .ts file for other purposes than hls, which means it has to be a single file containing all streams.

Ah so you are asking for"Support for Multiple Audio Streams in single segment MPEG-2 TS" (#3931), along with mixing of streams via audioTracks. I misunderstood your comment (#3931 (comment)) as a request for HTMLMediaElement audioTrack switching (not mixing as I don't think UAs offer that as a feature of the AudioTrackList API).

[robwalch]

I think you're in the realm of custom streaming app. You could fork parts of HLS.js to do the TS demuxing, but you'd probably want to use the Web Audio API https://www.w3.org/TR/webaudio/ to mix. This is outside the scope of HLS.js.

[deadbeef84]

I believe the behavior I'm describing is outside of the HLS spec? Anyway, the browser should be able to handle multiple audio streams when loading media directly using <video> or <audio>. Unfortunately hls.js will only use the first audio stream.

The spec says:

If multiple audio tracks are enabled simultaneously, they are mixed.

I can then use AudioContext.createMediaElementSource() to get the mixed audio, or possibly createMediaStreamTrackSource (not very supported, and unsure if it applies to all media elements or only streams) to get the audio of each individual track.