currentTime and video playback time are inconsistent

[CertainPassenger]

Description

Hello everybody,

first of all, big thank you for adding Video support to Skia!

I've noticed a small discrepancy in the currenTime reported by useVideo and the actual playback time of the video. There appears to be a small offset of 2 - 3 frames or in my case when testing with a 30 fps video of 66 milliseconds (see Pciture 1 below). Interestingly this offset changes between subsequent loops of the video (see Picture 3 below). It also appears that the last frames are not displayed, since the currentTime is larger than the video time (10 seconds vs. 9.93 seconds, see Picture 2 below). Maybe this could be related to #2493? Additionally, I've noticed that the discrepancy is quite a bit larger when loading the video from an url vs loading it from the assets (66 milliseconds when loading from assets, 150 milliseconds when loading from url). This offset appears to be consistent throughout the video playback.

I want to use this approach to draw a dynamic overlay over the video, highlighting objects in the video. However, since the currentTime and video time are slightly out of sync, as a result, my highlighting moves slightly ahead of the video.

Thank you very much for your help!

P.s.: It's probably impossible to also retrieve the exact frame via currentFrame in addition to the currentTime, but might be a nice feature :)

Version

1.3.8

Steps to reproduce

1. Create new blank expo project with typescript support (yarn create expo-app --template)

2. Install necessaries dependencies (expo-build-properties to set minSdkVersion to 26, expo-asset to load video from filesystem). Here are my exact dependencies.

  "dependencies": {
    "@shopify/react-native-skia": "^1.3.8",
    "expo": "~51.0.20",
    "expo-asset": "~10.0.10",
    "expo-build-properties": "~0.12.3",
    "expo-dev-client": "~4.0.20",
    "expo-status-bar": "~1.12.1",
    "expo-system-ui": "~3.0.7",
    "react": "18.2.0",
    "react-native": "0.74.3",
    "react-native-reanimated": "^3.14.0"
  }

3. Run minimum example

import {
  Canvas,
  ColorMatrix,
  Fill,
  ImageShader,
  Picture,
  Skia,
  createPicture,
  matchFont,
  useVideo,
} from "@shopify/react-native-skia";
import { Pressable, useWindowDimensions, Platform } from "react-native";
import { useDerivedValue, useSharedValue } from "react-native-reanimated";
import { useAssets } from "expo-asset";

// create a font to draw text overlay
const fontFamily = Platform.select({ ios: "Helvetica", default: "serif" });
const fontStyle = {
  fontFamily,
  fontSize: 22,
  fontStyle: "italic",
  fontWeight: "bold",
};
const font = matchFont(fontStyle as any);

export default function App()  {

  // initialize video
  const paused = useSharedValue(false);
  const { width, height } = useWindowDimensions();
  const [assets] = useAssets([require("./assets/test_video.mp4")]);

  const { currentFrame, currentTime } = useVideo(
    assets ? assets[0].localUri : "",
    {
      paused,
    }
  );

  // useDerivedValue to draw currentTime
  const currentPicture = useDerivedValue(() => {
    return createPicture((canvas) => {
      const paint = Skia.Paint();
      paint.setColor(Skia.Color("cyan"));
      paint.setStrokeWidth(3);
      canvas.drawText(`Video Timestamp: ${(currentTime.value / 1000.0).toFixed(2)}`, width / 2 - 135, height / 2 - 30, paint, font);
    });
  });

  return (
    <Pressable
      style={{ flex: 1 }}
      onPress={() => (paused.value = !paused.value)}
    >
      {currentFrame && currentPicture && (
        <Canvas style={{ flex: 1 }}>
          <Fill>
            <ImageShader
              image={currentFrame}
              x={0}
              y={0}
              width={width}
              height={height}
              fit="contain"
            />
            <ColorMatrix
              matrix={[
                0.95, 0, 0, 0, 0.05, 0.65, 0, 0, 0, 0.15, 0.15, 0, 0, 0, 0.5, 0,
                0, 0, 1, 0,
              ]}
            />
          </Fill>
          <Picture picture={currentPicture}></Picture>
        </Canvas>
      )}
    </Pressable>
  );
};

4. Pause video by clicking on it. Expected: currentTime == videoTime. Actual: currentTime > videoTime

Snack, code example, screenshot, or link to a repository

Example Video used in these tests: [Example Video](https://cloud.uni-konstanz.de/index.php/s/d4M2imiSbWiWHkY/download/test_video.mp4

1. When pausing the playback of the video, the currentTime (which is drawn as a text overlay) is 0.83 seconds, the timestamp in the video is 0.77 seconds)
   

2. It appears the last two frames are not being played, since the currentTime reached the limit of 10 seconds, but the video is only at 298 frames and 9.93 seconds.
   

3. After the video looped once, the currentTime and videoTime seem to be in sync, however sometimes the discrepancy was sometimes also 3 milliseconds.
   

[wcandillon]

is this issue Android only?

[CertainPassenger]

Unfortunately, I can only provide information for Android, since I don't have an iPhone to test this. Sorry!

[CertainPassenger]

I was able to get my hands on a MacBook to check if this issue also occurs on iPhones (emulator) and was able to recreate this issue.

Below I have some screenshots showing the offset at the beginning and at the end of the video. Similar to Android I have quite large of an offset at the beginning with around 200 milliseconds when loading the video directly via the url (Screenshot 1). The offset is still there when loading from assets, but similar to Android the offset is smaller. However, unlike Android, the offset is not consistent throughout the video playback and gets smaller or vanishes at the end of the video (Screenshot 2) or sometimes the video time even overtakes the video time (Screenshot 3). An additional issue I encountered is, that sometimes the currentTime loops, but not the video (see Screenshot 4).

Screenshot 1: Large offset at beginning of video

Screenshot 2: No offset at end of video

Screenshot 3: Negative offset at end of video

Screenshot 4: currentTime loops, but not the video

[dev-env571]

We have roughly a similar use case and tested it on a physical Apple device (Ipad as well as IPhone). On these devices, the problem also occurs.

Is there any new information/progress on this issue?