Backward-seeking frame in live / non live hls video get stuck #4956
Comments
salesh
added
Needs Triage
|
Hi @salesh, By default, HLS.js seeks forward to the start of the combined buffer (audio and video) when seeking into an area where only one (audio or video) is present, or the start of the combined buffer is less than I only see this as an issue if you are expecting HLS.js to provide frame accurate seeks when your content has jagged audio and video track start and end times. You can still play over these boundaries or keep seeking until crossing the next segment boundary for the next segment to load. The config options above will prevent the forward seek. If frame accurate seeking is important you may also want to consider segmenting using unmuxed fmp4. With video only playlists, video should always load when missing. |
robwalch
added
answered
and removed
Needs Triage
|
Hi @robwalch Thanks so much for responding and helping! |
|
np @salesh! Feel free to file an enhancement if you think this behavior should change in specific playback modes (like seeking while paused, or with low/no buffer hole tolerance).
We have steps to repro here when including the |
|
Hey @robwalch First of all, thank you for your responses and eagerness to help. We have Webrtc/HLS combination - we use Webrtc to provide our customers with a live sports game (that's why audio for video is important for us) and sometimes they want to go back and review some situation - that's why frame seeking is important for us. [1] ❓ Can you maybe explain to me why there are holes in the first place?
If I understood it good this happens when
and
[2] ❓ Can we do something from the side of configuration to improve What we tried: 1) ❌
(still audio is important for us)
It seems like the video buffer gets data but other buffers are stuck and this still prevents us from seeking 2) ❌
It does load segments when seeking backward, but there are gaps in the buffer which cause stalling - I guess 3) ❌ We tried here to get the previous one and connect buffer segments but it's failing. 4) ✅
At the moment this is the only thing that is working for us (we don't hit the wall I guess it recovers). [3] ❓I am not sure if 4) is correct from the bigger picture. What do you think about it?
since we couldn't make it with 2) and 3) any advice on where should I check in the code and where should this happen? |
|
just a small reminder ping if you find some time to comment about this @robwalch |
|
Hi @salesh,
Which holes exactly? Please provide specific examples. MSE is fickle. Appending media in advancing order works well with with well segmented media. That is not always the case when appending in reverse order.
There are three settings that play a roll in muxing transport stream and audio files (they have no impact on fmp4): I would not recommend using them as track and segment timing is a product of your encoding and packaging. Player configuration does not impact how media is appended.
Good. This should be much easier to fix. Where you able to determine why audio segments did not load? Isn't all the information here (buffer hole, frag look up, playlist time not matching media time)? Could you share the stream and steps to reproduce, so that we can look at fixing the audio segment loading in that use-case?
You'll need to elaborate on what these code samples are. I can't tell if this is code you modified or not. Use ````diff` as the format to make the change clearer, and state what you changed and why before the code block: Track stalls while paused and seeking, so that gap-controller will seek over small gaps in this playback state: // The playhead is moving, no-op
- if (currentTime !== lastCurrentTime) {
+ if (currentTime !== lastCurrentTime || (media?.paused && seeking)) {
this.moved = true;
|
|
Hey @robwalch Thank you for your response.
[1] Go to https://hls-js.netlify.app/demo const seekFrame = () => {
I guess this is the answer to my [1] question.
Thank you for sharing this knowledge with me
I didn't want to bother you since our producer (AntMedia) produces jagged audio - as long this is not fixed it would be a mistake to point the finger at
Sorry for this one. In 0.14 -> it will recover with some loss frames [1] The first thing we tried is changing the protected onMediaSeeking() {
const { config, fragCurrent, media, mediaBuffer, state } = this;
const currentTime: number = media ? media.currentTime : 0;
const bufferInfo = BufferHelper.bufferInfo(
mediaBuffer ? mediaBuffer : media,
currentTime,
config.maxBufferHole
);
this.log(
`media seeking to ${
Number.isFinite(currentTime) ? currentTime.toFixed(3) : currentTime
}, state: ${state}`
);
+ // check if we are soon in a unbuffered area when paused and seeking backwards
+ const backwardSearch = media?.seeking && this.lastCurrentTime && this.lastCurrentTime > media?.currentTime
+ if (media?.paused && backwardSearch) {
+ if (this.levels && this.levelLastLoaded !== null && this.levels[this.levelLastLoaded]) {
+ const level = this.levels[this.levelLastLoaded];
+ const levelDetails = level?.details;
+ const duration = level?.details?.targetduration;
+ if (levelDetails && duration) {
+ const targetBufferTime = media?.currentTime - duration + 2 * config.maxBufferHole;
+ const backBufferInfo = BufferHelper.bufferInfo(
+ this.mediaBuffer ? this.mediaBuffer : media,
+ targetBufferTime,
+ config.maxBufferHole
+ );
+ if (!backBufferInfo.len) {
+ const frag = this.getNextFragment(targetBufferTime, levelDetails);
+ if (frag && !frag.loader) {
+ this.log("Load next segement for backward seeking");
+ this.loadFragment(frag, levelDetails, targetBufferTime);
+ }
+ }
+ }
+ }
+ }
+
if (state === State.ENDED) {
this.resetLoadingState();
} else if (fragCurrent && !bufferInfo.len) {
// check if we are seeking to a unbuffered area AND if frag loading is in progress
const tolerance = config.maxFragLookUpTolerance;
const fragStartOffset = fragCurrent.start - tolerance;
const fragEndOffset =
fragCurrent.start + fragCurrent.duration + tolerance;
const pastFragment = currentTime > fragEndOffset;
// check if the seek position is past current fragment, and if so abort loading
if (currentTime < fragStartOffset || pastFragment) {
if (pastFragment && fragCurrent.loader) {
this.log(
'seeking outside of buffer while fragment load in progress, cancel fragment load'
);
fragCurrent.abortRequests();
}
this.resetLoadingState();
}
}
if (media) {
this.lastCurrentTime = currentTime;
}
// in case seeking occurs although no media buffered, adjust startPosition and nextLoadPosition to seek target
if (!this.loadedmetadata && !bufferInfo.len) {
this.nextLoadPosition = this.startPosition = currentTime;
}
// Async tick to speed up processing
this.tickImmediate();
}This one failed - it does load segments when seeking backward, but there are gaps in the buffer which cause stalling - our guess [2] the second approach was changing private _handleTransmuxComplete(transmuxResult: TransmuxerResult) {
const id = 'main';
const { hls } = this;
const { remuxResult, chunkMeta } = transmuxResult;
const context = this.getCurrentContext(chunkMeta);
if (!context) {
this.warn(
`The loading context changed while buffering fragment ${chunkMeta.sn} of level ${chunkMeta.level}. This chunk will not be buffered.`
);
this.resetStartWhenNotLoaded(chunkMeta.level);
return;
}
const { frag, part, level } = context;
const { video, text, id3, initSegment } = remuxResult;
const { details } = level;
// The audio-stream-controller handles audio buffering if Hls.js is playing an alternate audio track
const audio = this.altAudio ? undefined : remuxResult.audio;
// Check if the current fragment has been aborted. We check this by first seeing if we're still playing the current level.
// If we are, subsequently check if the currently loading fragment (fragCurrent) has changed.
if (this.fragContextChanged(frag)) {
return;
}
this.state = State.PARSING;
if (initSegment) {
if (initSegment.tracks) {
this._bufferInitSegment(level, initSegment.tracks, frag, chunkMeta);
hls.trigger(Events.FRAG_PARSING_INIT_SEGMENT, {
frag,
id,
tracks: initSegment.tracks,
});
}
// This would be nice if Number.isFinite acted as a typeguard, but it doesn't. See: https://github.com/Microsoft/TypeScript/issues/10038
const initPTS = initSegment.initPTS as number;
const timescale = initSegment.timescale as number;
if (Number.isFinite(initPTS)) {
this.initPTS[frag.cc] = initPTS;
hls.trigger(Events.INIT_PTS_FOUND, { frag, id, initPTS, timescale });
}
}
// Avoid buffering if backtracking this fragment
if (video && remuxResult.independent !== false) {
if (details) {
const { startPTS, endPTS, startDTS, endDTS } = video;
if (part) {
part.elementaryStreams[video.type] = {
startPTS,
endPTS,
startDTS,
endDTS,
};
} else {
if (video.firstKeyFrame && video.independent && chunkMeta.id === 1) {
this.couldBacktrack = true;
}
+ const bufferInfo = this.getMainFwdBufferInfo();
+ if (video.independent && bufferInfo?.nextStart &&
+ endDTS < bufferInfo.nextStart && endDTS + this.config.maxBufferHole >= bufferInfo.nextStart) {
+ // connect buffer segments
+ if (this.levels) {
+ const levelDetails = this.levels[this.currentLevel]?.details;
+ if (levelDetails) {
+ const frag = this.getNextFragment(video.startPTS, levelDetails);
+ if (frag) {
+ this.resetTransmuxer();
+ this.flushBufferGap(frag)
+ this.fragmentTracker.removeFragment(frag);
+ this.nextLoadPosition = frag.start;
+ // this.backtrack(frag);
+ return;
+ }
+ }
+ }
+ }
if (video.dropped && video.independent) {
// Backtrack if dropped frames create a gap after currentTime
const bufferInfo = this.getMainFwdBufferInfo();
const targetBufferTime =
(bufferInfo ? bufferInfo.end : this.getLoadPosition()) +
this.config.maxBufferHole;
const startTime = video.firstKeyFramePTS
? video.firstKeyFramePTS
: startPTS;
if (targetBufferTime < startTime - this.config.maxBufferHole) {
this.backtrack(frag);
return;
}
// Set video stream start to fragment start so that truncated samples do not distort the timeline, and mark it partial
frag.setElementaryStreamInfo(
video.type as ElementaryStreamTypes,
frag.start,
endPTS,
frag.start,
endDTS,
true
);
}
}
frag.setElementaryStreamInfo(
video.type as ElementaryStreamTypes,
startPTS,
endPTS,
startDTS,
endDTS
);
if (this.backtrackFragment) {
this.backtrackFragment = frag;
}
this.bufferFragmentData(video, frag, part, chunkMeta);
}
} else if (remuxResult.independent === false) {
this.backtrack(frag);
return;
}
if (audio) {
const { startPTS, endPTS, startDTS, endDTS } = audio;
if (part) {
part.elementaryStreams[ElementaryStreamTypes.AUDIO] = {
startPTS,
endPTS,
startDTS,
endDTS,
};
}
frag.setElementaryStreamInfo(
ElementaryStreamTypes.AUDIO,
startPTS,
endPTS,
startDTS,
endDTS
);
this.bufferFragmentData(audio, frag, part, chunkMeta);
}
if (details && id3?.samples?.length) {
const emittedID3: FragParsingMetadataData = {
id,
frag,
details,
samples: id3.samples,
};
hls.trigger(Events.FRAG_PARSING_METADATA, emittedID3);
}
if (details && text) {
const emittedText: FragParsingUserdataData = {
id,
frag,
details,
samples: text.samples,
};
hls.trigger(Events.FRAG_PARSING_USERDATA, emittedText);
}
}We tried here to get the previous one and connect buffer segments but it's failing. ❌ [3] Here comes a solution that worked for us (we forked hls.js since it was important for us to get out of those stuck positions) public poll(lastCurrentTime: number, activeFrag: Fragment | null) {
const { config, media, stalled } = this;
if (media === null) {
return;
}
const { currentTime, seeking } = media;
const seeked = this.seeking && !seeking;
const beginSeek = !this.seeking && seeking;
this.seeking = seeking;
// The playhead is moving, no-op
- if (currentTime !== lastCurrentTime) {
+ if (currentTime !== lastCurrentTime || (media?.paused && seeking)) {
this.moved = true;
if (stalled !== null) {
// The playhead is now moving, but was previously stalled
if (this.stallReported) {
const stalledDuration = self.performance.now() - stalled;
logger.warn(
`playback not stuck anymore @${currentTime}, after ${Math.round(
stalledDuration
)}ms`
);
this.stallReported = false;
}
this.stalled = null;
this.nudgeRetry = 0;
}
return;
}
// Clear stalled state when beginning or finishing seeking so that we don't report stalls coming out of a seek
if (beginSeek || seeked) {
this.stalled = null;
}
// The playhead should not be moving
if (
(media.paused && !seeking) ||
media.ended ||
media.playbackRate === 0 ||
!BufferHelper.getBuffered(media).length
) {
return;
}
const bufferInfo = BufferHelper.bufferInfo(media, currentTime, 0);
const isBuffered = bufferInfo.len > 0;
const nextStart = bufferInfo.nextStart || 0;
// There is no playable buffer (seeked, waiting for buffer)
if (!isBuffered && !nextStart) {
return;
}
if (seeking) {
// Waiting for seeking in a buffered range to complete
const hasEnoughBuffer = bufferInfo.len > MAX_START_GAP_JUMP;
// Next buffered range is too far ahead to jump to while still seeking
const noBufferGap =
!nextStart ||
(activeFrag && activeFrag.start <= currentTime) ||
(nextStart - currentTime > MAX_START_GAP_JUMP &&
!this.fragmentTracker.getPartialFragment(currentTime));
if (hasEnoughBuffer || noBufferGap) {
return;
}
// Reset moved state when seeking to a point in or before a gap
this.moved = false;
}
// Skip start gaps if we haven't played, but the last poll detected the start of a stall
// The addition poll gives the browser a chance to jump the gap for us
if (!this.moved && this.stalled !== null) {
// Jump start gaps within jump threshold
const startJump =
Math.max(nextStart, bufferInfo.start || 0) - currentTime;
// When joining a live stream with audio tracks, account for live playlist window sliding by allowing
// a larger jump over start gaps caused by the audio-stream-controller buffering a start fragment
// that begins over 1 target duration after the video start position.
const level = this.hls.levels
? this.hls.levels[this.hls.currentLevel]
: null;
const isLive = level?.details?.live;
const maxStartGapJump = isLive
? level!.details!.targetduration * 2
: MAX_START_GAP_JUMP;
if (startJump > 0 && startJump <= maxStartGapJump) {
this._trySkipBufferHole(null);
return;
}
}
// Start tracking stall time
const tnow = self.performance.now();
if (stalled === null) {
this.stalled = tnow;
return;
}
const stalledDuration = tnow - stalled;
if (!seeking && stalledDuration >= STALL_MINIMUM_DURATION_MS) {
// Report stalling after trying to fix
this._reportStall(bufferInfo);
if (!this.media) {
return;
}
}
const bufferedWithHoles = BufferHelper.bufferInfo(
media,
currentTime,
config.maxBufferHole
);
this._tryFixBufferStall(bufferedWithHoles, stalledDuration);
}With this fix, we are able to avoid stuck position and this skipping hole ✔️
Correct I created #5093 if you see this as an acceptable solution |
…n jagged start of loaded segment #4956
|
Closed with #5257 |
|
Hey @robwalch regarding #5093 (comment) As I mentioned in #5093 (comment) I didn't have any kind of issue with But today I did tests with 1.4.0 -> download zip -> add
Did something change in the release? |
Maybe, but it looks like the issue is now only reproducible with content that has alt-audio tracks, and not with muxed content (Big Buck Bunny test stream). I also can't repro in either case while paused in Chrome. Please close and file a new issue with steps to reproduce including a reference to a test stream that the issue can be reproduced with, and include which platforms you are experiencing the issue in. |
What do you want to do with Hls.js?
We are using
hls.jsin combination withngx-videogularfor live-streaming and playing videos.(We are using Simple Live/Ant Media for producing streams)
The issue is happening in both live and non-live HLS videos and it's reproducible on the demo site.
How reproduce?
[1] Go to https://hls-js.netlify.app/demo
[2] seek somewhere in middle and pause the video
[3] add function in the console
[4] call
seekFrame()from console[5] keep calling till you reach to
[warn] > skipping hole, adjusting currentTime from 459.988593 to 460.088593Currently, we are on the old version
"hls.js": "^0.14.17"In
0.14-> it will recover with some loss framesIn `latest -> it will not recover and we are stuck there.
Not sure how much configuration of our video is important since this is happening same for demo videos
Please let me know if I can provide you with some more information...Any kind of suggestion would be really appreciated
What have you tried so far?
Tried configuration from #2327
❌
The text was updated successfully, but these errors were encountered: