HLS Interstitials Support

[robwalch]

This PR will...

Add support for HLS Interstitials parsing, playback, and events.

Design Details

The InterstitialsController handles HLS and media input and events for the primary HLS asset. It passes date range data to the scheduler to produce an array of Interstitial events and an array of event and primary items called the schedule. InterstitialsController has:

• an InterstitialsSchedule which produces Interstitial events and a schedule of items on level update. The InterstitialsSchedule has
  • an array of InterstitialEvent objects. Each event contains properties and getters for finding its place on the playback timeline as well as an array of interstitial assets
• an array of HLSAssetPlayer instances call the player queue. Each HLSAssetPlayer wraps a child instance of Hls used to preload and stream Interstitial assets.
• the responsibility of loading asset lists. The InterstitialsController adds a loader to the corresponding InterstitialEvent while loading. The loader is removed and replaced by the asset list response once the request is complete.
• the responsibility of scheduling Interstitial playback and maintaining the current playing and buffering schedule items and assets. The schedule advances according to primary currentTime changes on timeupdate and seeking events, advancement of the combined buffer on media append, and asset playback and buffer advancement and completion.

API Enhancements

Configuration options

• Interstitials parsing and playback are enabled via config option interstitialsController. Setting this to null turns off Interstitial parsing and playback.
• To turn off Interstitial playback without removing parsing or schedule updates and buffered-to events, set config option enableInterstitialPlayback to false (allowing for custom playout and ad managers)
• Config option interstitialAssetListLoadPolicy defines the loading policy of X-ASSET-LIST JSON
• Several config options have been added specifically for Interstitial asset player instances:
  • primarySessionId identifies the parent player session that spawned the asset player (read from hls.sessionId)
  • assetPlayerId is used to identify logs from asset players
  • timelineOffset is used to offset MSE appends of Interstitial content (not all Interstitial assets are appended inline at offsets; most require a MediaSource reset)

New top-level API on Hls instances

• bufferedToEnd getter returns a boolean indicating if EOS has been appended (media is buffered from currentTime to end of stream)
• bufferingEnabled getter returns a boolean indicating whether fragment loading has been toggled with pauseBuffering() and resumeBuffering()
• interstitialsManager getter returns an InterstitialsManager (or null). The InterstitialsManager is an interface that provides access to Interstitial program and timeline data as well as methods for seeking across items and skipping Interstitial events.
• latestLevelDetails getter returns the LevelDetails of the most up-to-date HLS variant Playlist data
• sessionId getter returns the session UUID assigned to the Hls instance
  • The sessionId value is used when assigning a _HLS_primary_id query parameter to interstitial requests
• startLoad() now includes a second optional argument to skip seeking on start (otherwise, HLS.js seeks following to the first optional startPosition argument on append)
• hasEnoughToStart getter returns whether enough is buffered to seek to start position (fix: media buffer is not empty but video element buffer is empty #6571)
• startPosition getter returns the resolved startPosition that playback will begin at once media is appended
• transferMedia() method detaches and returns MediaSource and SourceBuffers non-destructively
• url getter returns the URL resolved from loadSource(url)

Updated top-level API on Hls instances

• attachMedia() enhancements for transferring MediaSource and SourceBuffers between Hls instances
• recoverMediaError() now seeks to the value of currentTime before the source reset is performed (BaseStreamController always honor config.startPosition when attaching(recovering) #6297)

New Events for Interstitials

• ASSET_LIST_LOADING when a request is made for an X-ASSET-LIST JSON object
• ASSET_LIST_LOADED when a response is received for an X-ASSET-LIST JSON object
• INTERSTITIALS_UPDATED when Interstitials are added, removed, or the schedule is updated following a variant playlist update or updated asset durations from X-ASSET-LIST JSON or asset playlist and media parsing
• INTERSTITIALS_BUFFERED_TO_BOUNDARY when the forward buffer reaches the boundary of the following schedule item (Interstitial event or primary segment)
• INTERSTITIAL_ASSET_PLAYER_CREATED when an asset player instance is created to stream an Interstitial asset (will always be before attaching media to the asset player)
• INTERSTITIAL_STARTED when streaming of an Interstitial event containing one or more assets has begun (may occur before X-ASSET-LIST JSON is loaded or playback has started)
• INTERSTITIAL_ENDED when streaming of an Interstitial event containing one or more assets has ended - before resuming primary or starting the next event
• INTERSTITIAL_ASSET_STARTED when streaming of an Interstitial asset has begun (following the beginning of the event or the end of the last asset)
• INTERSTITIAL_ASSET_ENDED when streaming of an Interstitial asset has ended (before the next asset or the event ending)
• INTERSTITIAL_ASSET_ERROR when an error occurs starting or streaming an Interstitial asset (this can include non-fatal errors such as stalling and errors that will end streaming of the asset, resulting in the schedule advancing to the next asset or fallback to primary)
• INTERSTITIALS_PRIMARY_RESUMED when playback of primary content has begun or resumed from an Interstitial event
• BUFFERED_TO_END when the last audio and video segments in the playlist have been appended (EOS signaled on all SourceBuffers)
• AUDIO_TRACK_UPDATED similar to LEVEL_UPDATED fired for any update to audio group playlists
• SUBTITLE_TRACK_UPDATED similar to LEVEL_UPDATED fired for any update to subtitle group playlists

Updated Events

• MEDIA_ATTACHING, MEDIA_ATTACHED, MEDIA_DETACHING, and MEDIA_DETACHED include additional information (depending on whether media is being transferred)

New Errors for Interstitials

• Type: NETWORK_ERROR
  • details: ASSET_LIST_LOAD_ERROR network error loading asset list
  • details: ASSET_LIST_LOAD_TIMEOUT network timeout error loading asset list
  • details: ASSET_LIST_PARSING_ERROR asset list was not valid JSON or missing required data
• type: OTHER_ERROR details: INTERSTITIAL_ASSET_ITEM_ERROR an issue interrupted or prevented asset playback. This will result in skipping the remainder of the asset or falling back to primary content. The event error will contain more details. This type of error differs from the INTERSTITIAL_ASSET_ERROR events forwarded from asset player errors.

Build Constants

• Interstitials is removed from the hls.light.js build by the __USE_INTERSTITALS__ build const/directive.

Known Issues

Should address before release (in order of priority):

• Asset lists are not being loaded when enableInterstitialPlayback is set to false. They should be loaded. The schedule needs to reflect the path taken when primary playback is chosen over interstitial playback.
• video element requires the autoplay or play() to be called on canplaythough after MediaSource reset from detach/attach for most schedule transitions (when appendInPlace is false). resolved with e1e515a (thanks to @matvp91 for calling this out)
• InterstitialsManager primary, playout and integrated durations are non-finite for Live streams. These should be finite for easier mapping of controls. resolved with 5842cc2
• hls.interstitialsManager.skip() does not skip Interstitial events that do not reset MediaSource (where InterstitialEvent appendInPlace is true) resolved with 0021cfd e84bc30 2c3c372
• In some cases the MediaSource is reset at the end of a post roll or playback ends at the end of the post roll, but attempting to play from that state does not restart the program.

No plan to address before release:

• There is no API provided for defining client-side Interstitials (file an issue stating your interest and use case).
• Primary subtitles TextTrack cues are not edited to avoid overlapping with Interstitial events scheduled without MediaSource reset (where InterstitialEvent appendInPlace is true). Note that this only applies to Interstitials that overlap and replace primary segments completely which is not the case for most Interstitials. If you have content where this is a concern, please file an issue.
• Option to enforce X-RESTRICT=JUMP by applying seekable range. Current enforcement is implemented on "seeking" and calls to seekTo that restrict schedule advancement.
• Present abutting interstitials and their assets in a gapless mode. This will reduce the number of MediaSource resets when not in appendInPlace mode and allow for single element and timeline playback of breaks independent of the primary player,

Resolves issues:

• Resolves HLS Interstitials Support #5730
• Resolves BaseStreamController always honor config.startPosition when attaching(recovering) #6297
• Resolves Seeking breaks stream with DISCONTINUITY #4905

Checklist

• changes have been done against master branch, and PR does not conflict
• new unit / functional tests have been added (whenever applicable)
• API or design changes are documented in API.md

[matvp91]

@robwalch, as discussed earlier, this is the case where only the first fragment is fetched / buffered when an interstitial is scheduled beyond the main timeline's duration: https://feature-interstitials.hls-js-4zn.pages.dev/demo/?src=https%3A%2F%2Fstitcher.mixwave.stream%2Fdirect%2F2eb5dfd5-7256-49f5-95c5-e86e1661e733%2Fmaster.m3u8%3Fparams%3DeyJpbnRlcnN0aXRpYWxzIjpbeyJ0aW1lT2Zmc2V0IjowLCJhc3NldElkIjoiNDZiYjRmZWYtNDQwYS01NTNhLWI1MTktMDg5ZWZmNDMwNmY2IiwidHlwZSI6ImFkIn0seyJ0aW1lT2Zmc2V0Ijo2MCwiYXNzZXRJZCI6IjQ2YmI0ZmVmLTQ0MGEtNTUzYS1iNTE5LTA4OWVmZjQzMDZmNiIsInR5cGUiOiJhZCJ9XX0%3D&demoConfig=eyJlbmFibGVTdHJlYW1pbmciOnRydWUsImF1dG9SZWNvdmVyRXJyb3IiOnRydWUsInN0b3BPblN0YWxsIjpmYWxzZSwiZHVtcGZNUDQiOmZhbHNlLCJsZXZlbENhcHBpbmciOi0xLCJsaW1pdE1ldHJpY3MiOi0xfQ==

The params payload is a base64 encoded string with the following info:

{
  "interstitials": [{
    "timeOffset": 0,
    "assetId": "46bb4fef-440a-553a-b519-089eff4306f6",
    "type": "ad"
  }, {
    "timeOffset": 60,
    "assetId": "46bb4fef-440a-553a-b519-089eff4306f6",
    "type": "ad"
  }]
}

When you exclude the timeOffset=60 (as it's the offset exceeding the main timeline duration), it'll continue buffering fragments normally.

[robwalch]

this is the case where only the first fragment is fetched / buffered when an interstitial is scheduled beyond the main timeline's duration

Thanks for the repro. I've pushed a fix to the scheduler that corrects an assumption that the previous event before the last primary segment must always be the last interstitial event. ef72aa0

        // last primary segment
        const timelineStart = primaryPosition;
        const integratedStart = integratedTime;
        const segmentDuration = primaryDuration - primaryPosition;
        integratedTime += segmentDuration;
        const playoutStart = playoutDuration;
        playoutDuration += segmentDuration;
        schedule.push({
-          previousEvent: interstitialEvents[interstitialEvents.length - 1],
+          previousEvent: schedule[schedule.length - 1].event || null,

That caused a lookup here (

hls.js/src/controller/interstitials-controller.ts

Line 1530 in 8490088

const bufferingIndex = this.findItemIndex(bufferingItem);

) to return an invalid index and setting the incorrect buffering item and preventing primary segments from loading.

[matvp91]

Checked it out and your change also fixed a bug where subtitle selection would only fetch the text playlist but not the fragments. See (

hls.js/src/controller/base-stream-controller.ts

Line 443 in e1e515a

const bufferingItem = interstitials?.bufferingItem;

), it bailed out early, _loadFragForPlayback was never called. Although I was running the same stream with the interstitial offset exceeding the main timeline duration so I'm not sure how representable the bug was in general. Nonetheless, it's good to know.

Minor: hls.interstitialsManager.integrated.seekTo($0.duration) is ignored. From a UI perspective, a user could seek to the end of an asset.

Thought 1: let's say there's an interstitial at 10, hls.interstitialsManager.integrated.seekTo(15) seeks over it but the timeline is internally adjusted to trigger the interstitial. Would that imply manually adjusting interstitial.resumeOffset (X-RESUME-OFFSET)? As far as I know, there's no spec definition to indicate a dynamic resume offset (to match the last requested main timeline time).

Thought 2: interstitial.hasPlayed indicates that an interstitial has been consumed before, would it make sense to have an event to influence whether an interstitial can be ignored? I think this goes hand in hand with a client API to change interstitials and throw them out of the schedule once they've finished. Wdyt?