Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

API enhancements for audio (and subtitle) track selection #5532

Closed
robwalch opened this issue Jun 1, 2023 · 0 comments · Fixed by #5978
Closed

API enhancements for audio (and subtitle) track selection #5532

robwalch opened this issue Jun 1, 2023 · 0 comments · Fixed by #5978

Comments

@robwalch
Copy link
Collaborator

robwalch commented Jun 1, 2023

Is your feature request related to a problem? Please describe.

Select audio track by language and/or features such as number of channels or description

Describe the solution you'd like

  • Provide fields such as language, channels, and characteristics
  • Provide audio and subtitle rendition groups which can each contain a set of tracks, groups and corresponding levels
  • Limit level selection based on active renditions
  • Provide rendition selection API that can handle complex level and track selection HLS Manifest with multi audio is not supported by player #5302

Additional context

#5704 Added hls.allAudioTracks and hls.allSubtitleTracks getters. These are not directly selectable, but list all MVP media options. The rendition groups will be a list of options that share the same attributes with the exception of those described in HLS Spec section 4.4.6.1.1 Rendition Groups.

Proposed solution

Two new new config options and methods will allow (pre) selection of audio and subtitles, implicitly by providing one ore more attributes for which a match must be found, or explicitly, using the methods, by passing in a MediaPlaylist instance (one of hls.allAudioTracks or hls.allSubtitleTracks).

export type AudioSelectionOption = {
  lang?: string;
  characteristics?: string;
  channels?: string;
  name?: string;
  audioCodec?: string;
  groupId?: string;
};

export type SubtitleSelectionOption = {
  lang?: string;
  characteristics?: string;
  name?: string;
  groupId?: string;
};

/**
 * Find and select the best matching audio track, making a level switch when a Group change is necessary.
 * Returns the selected track, or null when no matching track could be found.
 */
setAudioOption(audioOption: MediaPlaylist | AudioSelectionOption): MediaPlaylist | null {}
/**
 * Find and select the best matching subtitle track, making a level switch when a Group change is necessary.
 * Returns the selected track, or null when no matching track could be found.
 */
setSubtitleOption(audioOption: MediaPlaylist | SubtitleSelectionOption): MediaPlaylist | null {}

// New config options
export type SelectionPreferences = {
  audioPreference: AudioSelectionOption,
  subtitlePreference: SubtitleSelectionOption
};

Already added in dev (v1.5.0):

  /**
   * Get the complete list of audio tracks across all media groups
   */
  get allAudioTracks(): Array<MediaPlaylist> {}

  /**
   * get the complete list of subtitle tracks across all media groups
   */
  get allSubtitleTracks(): Array<MediaPlaylist> {}

Matching options:

First look for a matching option in the active group. If one is found, select the track and return it.

Otherwise, look for a matching option across all options and groups. As a level change is required, select the match and level that best matches the current codec set, bitrate, and channel count for audio.

If no match is found, log a warning including the provided input, and return null.

For the config options, matching will be performed on start, overriding the default AUTOSELECT/FORCED behavior, and potentially overriding the initial selection criterial for audioCodec and channels in abr-controller.

TBD: calling the setter methods successfully may update corresponding config option values. Either this, or another player instance specific value may be needed to coordinate changes on level switch.

Switching:
Generally, tracks a flushed on manual selection. Taking special care to smooth switch on changes of the same language and characteristics is a point for further enhancement.

@robwalch robwalch added Feature proposal Needs Triage If there is a suspected stream issue, apply this label to triage if it is something we should fix. labels Jun 1, 2023
@robwalch robwalch added this to the 1.5.0 milestone Jun 1, 2023
@robwalch robwalch removed the Needs Triage If there is a suspected stream issue, apply this label to triage if it is something we should fix. label Sep 28, 2023
robwalch added a commit that referenced this issue Nov 3, 2023
robwalch added a commit that referenced this issue Nov 4, 2023
robwalch added a commit that referenced this issue Nov 5, 2023
robwalch added a commit that referenced this issue Nov 10, 2023
robwalch added a commit that referenced this issue Nov 13, 2023
robwalch added a commit that referenced this issue Nov 16, 2023
robwalch added a commit that referenced this issue Nov 26, 2023
robwalch added a commit that referenced this issue Nov 26, 2023
eowino added a commit to DiceTechnology/hls.js that referenced this issue Jan 15, 2024
* chore(deps): update dependency chromedriver to v118 (video-dev#5919)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

* chore(deps): update dependency lint-staged to v15 (video-dev#5920)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

* chore(deps): update tjenkinson/gh-action-auto-merge-dependency-updates action to v1.3.5 (video-dev#5922)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

* chore(deps): update dependency lint-staged to v15.0.1

* chore(deps): update dependency lint-staged to v15.0.2

* chore(deps): update dependency chromedriver to v118.0.1

* chore(deps): update dependency @rollup/plugin-replace to v5.0.4

* chore(deps): update dependency wrangler to v3.13.2

* chore(deps): update dependency wrangler to v3.14.0

* chore(deps): update dependency @types/chai to v4.3.9

* chore(deps): update dependency @rollup/plugin-commonjs to v25.0.7

* chore(deps): update typescript-eslint monorepo to v6.8.0

* chore(deps): update typescript-eslint monorepo to v6.9.0

* chore(deps): update dependency @types/mocha to v10.0.3

* chore(deps): update dependency @types/chart.js to v2.9.39

* chore(deps): update dependency @types/sinon-chai to v3.2.11

* chore(deps): update dependency sinon to v16.1.1

* chore(deps): update dependency eslint-plugin-import to v2.29.0

* chore(deps): update dependency sinon to v16.1.3

* chore(deps): update dependency eslint to v8.52.0

* chore(deps): update dependency wrangler to v3.15.0

* chore(deps): update dependency @rollup/plugin-replace to v5.0.5

* Add named exports for classes and enums to ESM output
Resolves video-dev#5630

* chore(deps): update dependency @microsoft/api-documenter to v7.23.10

* chore(deps): update dependency @microsoft/api-extractor to v7.38.1

* chore(deps): update dependency @microsoft/api-documenter to v7.23.11

* chore(deps): update dependency @microsoft/api-extractor to v7.38.2

* chore(deps): update typescript-eslint monorepo to v6.9.1

* chore(deps): update typescript-eslint monorepo to v6.10.0

* chore(deps): update dependency eslint to v8.53.0

* Update README.md

* Update README.md

* Update README.md

* chore(deps): update dependency @types/mocha to v10.0.4

* chore(deps): update dependency selenium-webdriver to v4.15.0

* chore(deps): update dependency @types/chart.js to v2.9.40

* chore(deps): update dependency @types/chai to v4.3.10

* chore(deps): update dependency @types/sinon-chai to v3.2.12

* Fix detach attach behavior dropping one of two SourceBuffers

* Use Content Steering Pathways to manage Redundant Streams (video-dev#5970)

* Use Content Steering Pathways to manage Redundant Streams and resolve their errors
* Ensure correct Pathway penalization on Playlist loading errors
* Do not reload Content Steering manifest while media is ended or detached

* chore(deps): update babel monorepo to v7.23.3

* Remove use of `self` from `enableLogger` (video-dev#5936)

Fixes video-dev#5905

* Refactor CMCD controller and tests to use the common media library utilities (video-dev#5903)

* refactor CMCD controller and test to use common media library
* update build script to transpile the @svta package
* add ability to specify cmcd keys

* chore(deps): update dependency lint-staged to v15.1.0

* chore(deps): update dependency @microsoft/api-extractor to v7.38.3

* chore(deps): update typescript-eslint monorepo to v6.11.0

* chore(deps): update typescript-eslint monorepo to v6.12.0

* chore(deps): update dependency @microsoft/api-documenter to v7.23.12

* Fix regression introduced with video-dev#5689 Lazy init CEA608 parsers found in video-dev#5953 (video-dev#5986)

* Fix issues with long cea608 captions. video-dev#5952

In mp4-tools.ts

* Fixed parsing for sei_message, by always consuming the entire message,
  before parsing the message according to payload_type

* Fixed payloadType / payloadSize parsing to ensure they never exceed
  255, as the field is restricted to 8 bytes.

* chore(deps): update dependency node to v20 (video-dev#5928)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

* chore(deps): update dependency sinon to v17 (video-dev#5944)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

* chore(deps): update actions/setup-node action to v4 (video-dev#5948)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

* chore(deps): update dependency chromedriver to v119 [security] (video-dev#5965)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

* chore(deps): update dependency prettier to v3.1.0 (video-dev#5983)

* chore(deps): update dependency prettier to v3.1.0

* Run prettier

---------

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Tom Jenkinson <tom@tjenkinson.me>

* Configure typescript, eslint and prettier caches (video-dev#5990)

* Rollup 4 (video-dev#5886)

* Update karma-rollup-preprocessor to version that works in watch mode (video-dev#5991)

* chore(deps): update dependency @svta/common-media-library to v0.5.1

* chore(deps): update dependency wrangler to v3.16.0

* chore(deps): update dependency wrangler to v3.17.1

* chore(deps): update dependency rollup to v4.5.2

* chore(deps): update dependency eslint to v8.54.0

* API enhancements for audio and subtitle selection
Resolves video-dev#5532

* Remove note about "canplay" that references code removed from the example (related to autoplay policy)
Closes video-dev#3153

* Fix issues parsing sei_messages

In mp4-tools.ts

* Fixed parsing for sei_message to remove the incorrect masking to 8
  bits for the sei message size.

* Remove SEI payload type masking

* chore(deps): update dependency @types/chart.js to v2.9.41

* chore(deps): update dependency @types/chai to v4.3.11

* chore(deps): update dependency @types/mocha to v10.0.5

* chore(deps): update dependency @types/mocha to v10.0.6

* Expand isSupported check to test other codecs
Resolves video-dev#6004

* Add `videoPreference` config option for HDR/SDR VIDEO-RANGE selection and priority
Resolves video-dev#2489

* Recover from media error after MediaSource ended following SourceBuffer update error event

* Fix exception on 2019 Tizen where MediaCapabilities is undefined

* Add `isMSESupported` check
Add named exports for and expose statically: `isSupported`, `isMSESupported`, and `getMediaSource`

* chore(deps): update dependency @rollup/plugin-alias to v5.1.0

* chore(deps): update dependency rollup to v4.6.0

* chore(deps): update dependency rollup to v4.6.1

* chore(deps): update typescript-eslint monorepo to v6.13.0

* chore(deps): update typescript-eslint monorepo to v6.13.2

* chore(deps): update babel monorepo to v7.23.5

* Remove use of deprecated WebKitDataCue and hand Cue instantiation and custom property setting errors
Fixes video-dev#6020

* Add polyfill for isSafeInteger

* Fix esds box parsing for for usac audio

* Update README Compatibility section

* chore(deps): update dependency @svta/common-media-library to v0.6.0

* chore(deps): update dependency wrangler to v3.18.0

* chore(deps): update dependency wrangler to v3.19.0

* chore(deps): update dependency eslint to v8.55.0

* chore(deps): update dependency eslint-config-prettier to v9.1.0

* chore(deps): update dependency lint-staged to v15.2.0

* chore(deps): update dependency @microsoft/api-documenter to v7.23.13

* chore(deps): update dependency @microsoft/api-extractor to v7.38.4

* chore(deps): update dependency @microsoft/api-extractor to v7.38.5

* chore(deps): update dependency @microsoft/api-documenter to v7.23.14

* Use addEventListener for MediaKeySession events
video-dev#6034

* chore(deps): update dependency selenium-webdriver to v4.16.0

* fix(latency-controller): only sync live stream

* chore(deps): update actions/github-script action to v7 (video-dev#5996)

* Ignore #EXT-X-INDEPENDENT-SEGMENTS so that it is not added to Fragment tagList

* Fix handling of the DATERANGE END-ON-NEXT attribute

* chore(deps): update dependency rollup to v4.7.0

* chore(deps): update dependency rollup to v4.9.0

* Store deployments in json file, and generate md and txt file from that (video-dev#6044)

* Fix deployment branch update commit messages

Just noticed this has been broken for a while

* Fix path to deployment readme script

* Fix path to script again

* Add the final `/` at the end of deployment url

* Remove tab at end of deployments readme

Which is causing the list to be really spaced out for some reason

* chore(deps): update dependency typescript to v5.3.3 (video-dev#5999)

* chore(deps): update dependency typescript to v5.3.3

* Include api extractor changes

---------

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Tom Jenkinson <tom@tjenkinson.me>

* Exclude PS4 from TextDecoder use

On PS4 TextDecoder is defined but it is partially implemented and does not function properly. This will force manual decoding approach for PS4 platform.

* Ignore #EXT-X-INDEPENDENT-SEGMENTS (video-dev#6047)

Fixes video-dev#6039

* chore(deps): update babel monorepo to v7.23.6

* chore(deps): update dependency prettier to v3.1.1

* chore(deps): update typescript-eslint monorepo to v6.14.0

* chore(deps): update typescript-eslint monorepo to v6.15.0

* chore(deps): update dependency wrangler to v3.20.0

* chore(deps): update dependency wrangler to v3.22.0

* Fix base-stream-controller onHandlerDestroying callback evocation
Remove circular references left after destroying player

* chore(deps): update dependency eslint-plugin-import to v2.29.1

* chore(deps): update dependency eslint to v8.56.0

* chore(deps): update dependency @svta/common-media-library to v0.6.1

* chore(deps): update dependency rollup to v4.9.1

* chore(deps): update dependency @microsoft/api-documenter to v7.23.15

* chore(deps): update dependency @microsoft/api-extractor to v7.39.0

* chore(deps): update dependency chromedriver to v120 (video-dev#6052)

* chore(deps): update github/codeql-action action to v3 (video-dev#6058)

* chore(deps): update dependency wrangler to v3.22.1

* Abort fetch loader as long as loading has not ended
Fixes video-dev#6054

* chore(deps): update dependency chromedriver to v120.0.1

* chore(deps): update typescript-eslint monorepo to v6.16.0

* chore(deps): update typescript-eslint monorepo to v6.17.0

* chore(deps): update babel monorepo to v7.23.7

* chore(deps): update dependency rollup to v4.9.2

* chore(deps): update dependency rollup to v4.9.4

* Fix codec parsing for AVC streams (video-dev#6077)

* Force auto level on emergency switch down (video-dev#6082)

Update estimates on frag load timeout
Do not abort request in _abandonRulesCheck
Remove two segment forward buffer length limit in _abandonRulesCheck
Reset estimate when candidate bitrate is lower than adjusted estimate
Resolves video-dev#6079

* chore(deps): update dependency wrangler to v3.22.2

* chore(deps): update dependency wrangler to v3.22.4

* chore(deps): update dependency @microsoft/api-documenter to v7.23.16

* chore(deps): update dependency @microsoft/api-extractor to v7.39.1

* Null CMCD callbacks on destroy (video-dev#6098)

* Fix regression where subtitle options with AUTOSELECT and FORCED are enabled at start (video-dev#6094)

* Do not enable subtitle options with AUTOSELECT=YES attribute
* Update and add initial selection tests for subtitle-controller
* Only pick forced subtitle option if it is the only one
Add default field to audio and subtitle selection options and forced field to subtitle selection option
* Address TextTrack change event overriding subtitle preference
Fix _TRACKS_UPDATED and _TRACK_SWITCH event order when preference is selected
* Do not auto select subtitle options with FORCED=YES attribute

* Update artifact actions (video-dev#6099)

* Update functional tests to run on Safari using MacOS 13 (video-dev#6101)

* Update functional tests to run on Safari using MacOS 13

* Skip smooth switch test in Safari on streams with overlapping appends

* Omit VOD "ended" event tests with overlapping appends from Safari

* chore(deps): update dependency chai to v4.4.0

* chore(deps): update dependency chai to v4.4.1

* chore(deps): update typescript-eslint monorepo to v6.18.0

* chore(deps): update typescript-eslint monorepo to v6.18.1

* Use AAC SBR (HE-AAC) workaround on Pale Moon (video-dev#6111)

---------

Co-authored-by: hlsjs-ci <40664919+hlsjs-ci@users.noreply.github.com>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Rob Walch <rwalch@apple.com>
Co-authored-by: Rob Walch <robwalch@users.noreply.github.com>
Co-authored-by: Casey Occhialini <1508707+littlespex@users.noreply.github.com>
Co-authored-by: Joey Ekstrom <jekstrom@xumo.com>
Co-authored-by: Tom Jenkinson <tom@tjenkinson.me>
Co-authored-by: Tom Jenkinson <tjenkinson@users.noreply.github.com>
Co-authored-by: Evan Burton <eburton2@apple.com>
Co-authored-by: 曾智锋 <zengzhifeng@cvte.com>
Co-authored-by: Agajan Jumakuliyev <agajan.tm@gmail.com>
Co-authored-by: Jakub Perżyło <qizot1405@gmail.com>
Co-authored-by: Pat Nafarrete <pnaf@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant