Buffering fails when switch codecs #5369

PavelFomin90 · 2023-04-03T15:48:34Z

What version of Hls.js are you using?

1.2.0

What browser (including version) are you using?

Version 111.0.5563.146 (Official Build) (x86_64)

What OS (including version) are you using?

MacOs BigSur 11.6

Test stream

No response

Configuration

{
  enableWorker: true,
  lowLatencyMode: true,
  backBufferLength: 60,
  startLevel: 3,
  liveSyncDurationCount: 2,
}

Additional player setup steps

No response

Checklist

The issue observed is not already reported by searching on Github under https://github.com/video-dev/hls.js/issues
The issue occurs in the stable client (latest release) on https://hlsjs.video-dev.org/demo and not just on my page
The issue occurs in the latest client (main branch) on https://hlsjs-dev.video-dev.org/demo and not just on my page
The stream has correct Access-Control-Allow-Origin headers (CORS)
There are no network errors such as 404s in the browser console when trying to play the stream

Steps to reproduce

Start with level with one codec eg. CODECS="avc1.640029, mp4a.40.2"
Switch to another level with other codec eg. CODECS="av01.0.00M.10, mp4a.40.2"
Buffering fails

Expected behaviour

buffering not fails when switch levels with different codecs

What actually happened?

When switching levels with different codecs buffer-controller uses different remuxers and in the case of CODECS="avc1.640029, mp4a.40.2" it return 2 tracks (audio and video separately)
in the case of CODECS="av01.0.00M.10, mp4a.40.2" it return 1 audiovideo track
and onBufferCodecs in buffer-controllers couldn't switch tracks buffers correctly in this case

Also it's not metter using setting enableWorker true or false

Console output

Using Hls.js config: {debug: true, enableWorker: false, lowLatencyMode: true, backBufferLength: 60, startLevel: 3, …}
logger.ts:74 [log] > Debug logs enabled for "Hls instance"
hls.ts:371 [log] > stopLoad
hls.ts:339 [log] > loadSource:https://pavelfomin.ru/rutube/a.m3u8
stream-controller.ts:550 [log] > [stream-controller]: Trigger BUFFER_RESET
hls.ts:310 [log] > attachMedia
level-controller.ts:191 [log] > [level-controller]: manifest loaded, 5 level(s) found, first bitrate: 2531000
buffer-controller.ts:142 [log] > 1 bufferCodec event(s) expected
hls.ts:361 [log] > startLoad(-1)
level-controller.ts:272 [log] > [level-controller]: switching to level 3 from -1
level-controller.ts:584 [log] > [level-controller]: Attempt loading level index 3 with URL-id 0 https://river-7-701.rutube.ru/hls-vod/4RSeR_bhfd9uRuX_Oo1pYQ/1680543184/1712/0x5000c500db69685f/9ab239e7f68547f0be8b48b0c5529212.mp4.m3u8?i=3840x2160_5139
base-stream-controller.ts:1505 [log] > [stream-controller]: STOPPED->IDLE
base-stream-controller.ts:1505 [log] > [subtitle-stream-controller]: STOPPED->IDLE
buffer-controller.ts:791 [log] > [buffer-controller]: Media source opened
stream-controller.ts:615 [log] > [stream-controller]: Level 3 loaded [1,56], cc [0, 0] duration:295.71799999999996
buffer-controller.ts:683 [log] > [buffer-controller]: Updating Media Source duration to 295.718
base-stream-controller.ts:660 [log] > [stream-controller]: Loading fragment initSegment cc: 0 of [1-56] level: 3, target: 0
base-stream-controller.ts:1505 [log] > [stream-controller]: IDLE->FRAG_LOADING
base-stream-controller.ts:1505 [log] > [stream-controller]: FRAG_LOADING->IDLE
base-stream-controller.ts:504 [log] > [stream-controller]: Buffered main sn: initSegment of level 3 (frag:[NaN-NaN] > buffer:)
base-stream-controller.ts:660 [log] > [stream-controller]: Loading fragment 1 cc: 0 of [1-56] level: 3, target: 0
base-stream-controller.ts:1505 [log] > [stream-controller]: IDLE->FRAG_LOADING
transmuxer-interface.ts:201 [log] > [transmuxer-interface, main]: Starting new transmux session for sn: 1 p: -1 level: 3 id: 1
        discontinuity: true
        trackSwitch: true
        contiguous: false
        accurateTimeOffset: true
        timeOffset: 0
        initSegmentChange: true
base-stream-controller.ts:1505 [log] > [stream-controller]: FRAG_LOADING->PARSING
stream-controller.ts:1263 [log] > [stream-controller]: Init audiovideo buffer, container:video/mp4, codecs[level/parsed]=[av01.0.00M.10, mp4a.40.2/mp4a.40.2,av01.0.00M.10]
buffer-controller.ts:752 [log] > [buffer-controller]: creating sourceBuffer(video/mp4;codecs=mp4a.40.2,av01.0.00M.10)
audio-stream-controller.ts:118 [log] > [audio-stream-controller]: InitPTS for cc: 0 found from main: 0
base-stream-controller.ts:371 [log] > [stream-controller]: Loaded fragment 1 of level 3
transmuxer.ts:253 [log] > [transmuxer.ts]: Flushed fragment 1 of level 3
base-stream-controller.ts:1505 [log] > [stream-controller]: PARSING->PARSED
base-stream-controller.ts:504 [log] > [stream-controller]: Buffered main sn: 1 of level 3 (frag:[0.000-5.367] > buffer:[0.000-5.367])
base-stream-controller.ts:1505 [log] > [stream-controller]: PARSED->IDLE
base-stream-controller.ts:660 [log] > [stream-controller]: Loading fragment 2 cc: 0 of [1-56] level: 3, target: 5.367
base-stream-controller.ts:1505 [log] > [stream-controller]: IDLE->FRAG_LOADING
base-stream-controller.ts:1505 [log] > [stream-controller]: FRAG_LOADING->PARSING
base-stream-controller.ts:371 [log] > [stream-controller]: Loaded fragment 2 of level 3
transmuxer.ts:253 [log] > [transmuxer.ts]: Flushed fragment 2 of level 3
base-stream-controller.ts:1505 [log] > [stream-controller]: PARSING->PARSED
base-stream-controller.ts:504 [log] > [stream-controller]: Buffered main sn: 2 of level 3 (frag:[5.367-10.733] > buffer:[0.000-10.733])
base-stream-controller.ts:1505 [log] > [stream-controller]: PARSED->IDLE
base-stream-controller.ts:660 [log] > [stream-controller]: Loading fragment 3 cc: 0 of [1-56] level: 3, target: 10.733
base-stream-controller.ts:1505 [log] > [stream-controller]: IDLE->FRAG_LOADING
base-stream-controller.ts:1505 [log] > [stream-controller]: FRAG_LOADING->PARSING
base-stream-controller.ts:371 [log] > [stream-controller]: Loaded fragment 3 of level 3
transmuxer.ts:253 [log] > [transmuxer.ts]: Flushed fragment 3 of level 3
base-stream-controller.ts:1505 [log] > [stream-controller]: PARSING->PARSED
base-stream-controller.ts:504 [log] > [stream-controller]: Buffered main sn: 3 of level 3 (frag:[10.733-16.100] > buffer:[0.000-16.100])
base-stream-controller.ts:1505 [log] > [stream-controller]: PARSED->IDLE
base-stream-controller.ts:660 [log] > [stream-controller]: Loading fragment 4 cc: 0 of [1-56] level: 3, target: 16.1
base-stream-controller.ts:1505 [log] > [stream-controller]: IDLE->FRAG_LOADING
base-stream-controller.ts:1505 [log] > [stream-controller]: FRAG_LOADING->PARSING
base-stream-controller.ts:371 [log] > [stream-controller]: Loaded fragment 4 of level 3
transmuxer.ts:253 [log] > [transmuxer.ts]: Flushed fragment 4 of level 3
base-stream-controller.ts:1505 [log] > [stream-controller]: PARSING->PARSED
base-stream-controller.ts:504 [log] > [stream-controller]: Buffered main sn: 4 of level 3 (frag:[16.100-21.467] > buffer:[0.000-21.467])
base-stream-controller.ts:1505 [log] > [stream-controller]: PARSED->IDLE
base-stream-controller.ts:660 [log] > [stream-controller]: Loading fragment 5 cc: 0 of [1-56] level: 3, target: 21.467
base-stream-controller.ts:1505 [log] > [stream-controller]: IDLE->FRAG_LOADING
base-stream-controller.ts:1505 [log] > [stream-controller]: FRAG_LOADING->PARSING
base-stream-controller.ts:371 [log] > [stream-controller]: Loaded fragment 5 of level 3
transmuxer.ts:253 [log] > [transmuxer.ts]: Flushed fragment 5 of level 3
base-stream-controller.ts:1505 [log] > [stream-controller]: PARSING->PARSED
base-stream-controller.ts:504 [log] > [stream-controller]: Buffered main sn: 5 of level 3 (frag:[21.467-26.833] > buffer:[0.000-26.833])
base-stream-controller.ts:1505 [log] > [stream-controller]: PARSED->IDLE
base-stream-controller.ts:660 [log] > [stream-controller]: Loading fragment 6 cc: 0 of [1-56] level: 3, target: 26.833
base-stream-controller.ts:1505 [log] > [stream-controller]: IDLE->FRAG_LOADING
base-stream-controller.ts:1505 [log] > [stream-controller]: FRAG_LOADING->PARSING
base-stream-controller.ts:371 [log] > [stream-controller]: Loaded fragment 6 of level 3
transmuxer.ts:253 [log] > [transmuxer.ts]: Flushed fragment 6 of level 3
base-stream-controller.ts:1505 [log] > [stream-controller]: PARSING->PARSED
base-stream-controller.ts:504 [log] > [stream-controller]: Buffered main sn: 6 of level 3 (frag:[26.833-32.200] > buffer:[0.000-32.200])
base-stream-controller.ts:1505 [log] > [stream-controller]: PARSED->IDLE
base-stream-controller.ts:660 [log] > [stream-controller]: Loading fragment 7 cc: 0 of [1-56] level: 3, target: 32.2
base-stream-controller.ts:1505 [log] > [stream-controller]: IDLE->FRAG_LOADING
base-stream-controller.ts:1505 [log] > [stream-controller]: FRAG_LOADING->PARSING
base-stream-controller.ts:371 [log] > [stream-controller]: Loaded fragment 7 of level 3
transmuxer.ts:253 [log] > [transmuxer.ts]: Flushed fragment 7 of level 3
base-stream-controller.ts:1505 [log] > [stream-controller]: PARSING->PARSED
base-stream-controller.ts:504 [log] > [stream-controller]: Buffered main sn: 7 of level 3 (frag:[32.200-37.567] > buffer:[0.000-37.567])
base-stream-controller.ts:1505 [log] > [stream-controller]: PARSED->IDLE
base-stream-controller.ts:660 [log] > [stream-controller]: Loading fragment 8 cc: 0 of [1-56] level: 3, target: 37.567
base-stream-controller.ts:1505 [log] > [stream-controller]: IDLE->FRAG_LOADING
base-stream-controller.ts:1505 [log] > [stream-controller]: FRAG_LOADING->PARSING
base-stream-controller.ts:371 [log] > [stream-controller]: Loaded fragment 8 of level 3
transmuxer.ts:253 [log] > [transmuxer.ts]: Flushed fragment 8 of level 3
base-stream-controller.ts:1505 [log] > [stream-controller]: PARSING->PARSED
base-stream-controller.ts:504 [log] > [stream-controller]: Buffered main sn: 8 of level 3 (frag:[37.567-42.933] > buffer:[0.000-42.933])
base-stream-controller.ts:1505 [log] > [stream-controller]: PARSED->IDLE
base-stream-controller.ts:660 [log] > [stream-controller]: Loading fragment 9 cc: 0 of [1-56] level: 3, target: 42.933
base-stream-controller.ts:1505 [log] > [stream-controller]: IDLE->FRAG_LOADING
base-stream-controller.ts:1505 [log] > [stream-controller]: FRAG_LOADING->PARSING
base-stream-controller.ts:371 [log] > [stream-controller]: Loaded fragment 9 of level 3
transmuxer.ts:253 [log] > [transmuxer.ts]: Flushed fragment 9 of level 3
base-stream-controller.ts:1505 [log] > [stream-controller]: PARSING->PARSED
base-stream-controller.ts:504 [log] > [stream-controller]: Buffered main sn: 9 of level 3 (frag:[42.933-48.300] > buffer:[0.000-48.300])
base-stream-controller.ts:1505 [log] > [stream-controller]: PARSED->IDLE
base-stream-controller.ts:660 [log] > [stream-controller]: Loading fragment 10 cc: 0 of [1-56] level: 3, target: 48.3
base-stream-controller.ts:1505 [log] > [stream-controller]: IDLE->FRAG_LOADING
hls.ts:427 [log] > set currentLevel:2
hls.ts:468 [log] > set loadLevel:2
level-controller.ts:272 [log] > [level-controller]: switching to level 2 from 3
level-controller.ts:584 [log] > [level-controller]: Attempt loading level index 2 with URL-id 0 https://river-7-701.rutube.ru/hls-vod/gfEQNngs0BJeEDr5YOnhbQ/1680543184/1764/0x5000c500e34c79ab/6298ef8f27504cea8e6ae00676194805.mp4.m3u8?i=1280x720_4961
base-stream-controller.ts:1505 [log] > [stream-controller]: FRAG_LOADING->IDLE
buffer-controller.ts:876 [log] > [buffer-controller]: Removing [0,295.71799999999996] from the audiovideo SourceBuffer
stream-controller.ts:615 [log] > [stream-controller]: Level 2 loaded [1,75], cc [0, 0] duration:295.7
base-stream-controller.ts:660 [log] > [stream-controller]: Loading fragment 3 cc: 0 of [1-75] level: 2, target: 5.43
base-stream-controller.ts:1505 [log] > [stream-controller]: IDLE->FRAG_LOADING
transmuxer-interface.ts:201 [log] > [transmuxer-interface, main]: Starting new transmux session for sn: 3 p: -1 level: 2 id: 1
        discontinuity: false
        trackSwitch: true
        contiguous: false
        accurateTimeOffset: true
        timeOffset: 4
        initSegmentChange: true
mp4-remuxer.ts:86 [log] > [mp4-remuxer]: ISGenerated flag reset
mp4-remuxer.ts:75 [log] > [mp4-remuxer]: initPTS & initDTS reset
mp4-remuxer.ts:80 [log] > [mp4-remuxer]: reset next timestamp
adts.ts:59 [log] > manifest codec:mp4a.40.2, ADTS type:2, samplingIndex:4
adts.ts:245 [log] > parsed codec:mp4a.40.5, rate:44100, channels:2
base-stream-controller.ts:1505 [log] > [stream-controller]: FRAG_LOADING->PARSING
stream-controller.ts:1243 [log] > [stream-controller]: Init audio buffer, container:audio/mp4, codecs[selected/level/parsed]=[mp4a.40.2/mp4a.40.2/mp4a.40.5]
stream-controller.ts:1254 [log] > [stream-controller]: Init video buffer, container:video/mp4, codecs[level/parsed]=[avc1.640029/avc1.640029]
base-stream-controller.ts:371 [log] > [stream-controller]: Loaded fragment 3 of level 2
transmuxer.ts:253 [log] > [transmuxer.ts]: Flushed fragment 3 of level 2
base-stream-controller.ts:1505 [log] > [stream-controller]: PARSING->PARSED

Chrome media internals output

No response

The text was updated successfully, but these errors were encountered:

robwalch · 2023-04-04T01:16:13Z

When switching levels with different codecs buffer-controller uses different remuxers and in the case of CODECS="avc1.640029, mp4a.40.2" it return 2 tracks (audio and video separately)
in the case of CODECS="av01.0.00M.10, mp4a.40.2" it return 1 audiovideo track
and onBufferCodecs in buffer-controllers couldn't switch tracks buffers correctly in this case

This is not an issue with the codecs, but rather with the segment type and track configuration.

You can try using hls.recoverMediaError() to reset source buffers to match the selected container, but otherwise HSL.js (and MSE) does not support seamless switching between unmuxed "audio" and "video" tracks to muxed "audiovideo" in fmp4.

fmp4 is always pass-through so if you provide muxed "audiovideo" mp4, you need to do so in all variants.

PavelFomin90 · 2023-04-04T08:55:18Z

Thank you for answer.
We'll try to fix streams

PavelFomin90 added Bug Needs Triage If there is a suspected stream issue, apply this label to triage if it is something we should fix. labels Apr 3, 2023

robwalch added Stream Issue Not Supported and removed Needs Triage If there is a suspected stream issue, apply this label to triage if it is something we should fix. Bug labels Apr 4, 2023

PavelFomin90 closed this as completed Apr 4, 2023

robwalch mentioned this issue Jul 3, 2023

Handle mixed sourcebuffer errors #5623

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Buffering fails when switch codecs #5369

Buffering fails when switch codecs #5369

PavelFomin90 commented Apr 3, 2023

robwalch commented Apr 4, 2023 •

edited

Loading

PavelFomin90 commented Apr 4, 2023

Buffering fails when switch codecs #5369

Buffering fails when switch codecs #5369

Comments

PavelFomin90 commented Apr 3, 2023

What version of Hls.js are you using?

What browser (including version) are you using?

What OS (including version) are you using?

Test stream

Configuration

Additional player setup steps

Checklist

Steps to reproduce

Expected behaviour

What actually happened?

Console output

Chrome media internals output

robwalch commented Apr 4, 2023 • edited Loading

PavelFomin90 commented Apr 4, 2023

robwalch commented Apr 4, 2023 •

edited

Loading