How do I use segmentation videos to segment object in video?

[gajraj-gan]

Hello there,
I am making a remotion video where I want to display a video specifically a object in that video, I have a segmentation video consisting of 0s and 1s which only highlights that particular object, how do I render a video in which only that video is visible using remotion??
I tried mounting two Video elements in my own custom component, but it does not seem to work.

import React, { useEffect, useRef, useState } from 'react';
import {
  AbsoluteFill,
  continueRender,
  delayRender,
  useCurrentFrame,
  useVideoConfig,
  Video
} from 'remotion';

type CustomOffthreadVideoProps = {
  inputVideoSrc: string;
  segmentationVideoSrc: string;
};

export const CustomOffthreadVideo: React.FC<CustomOffthreadVideoProps> = ({
  inputVideoSrc,
  segmentationVideoSrc
}) => {
  const inputVideoRef = useRef<HTMLVideoElement>(null);
  const segmentationVideoRef = useRef<HTMLVideoElement>(null);
  const canvasRef = useRef<HTMLCanvasElement>(null);
  const frame = useCurrentFrame();
  const { width, height } = useVideoConfig();
  const [handle] = useState(() => delayRender('Fetching dimensions...'));
  const [video1Loaded, setVideo1Loaded] = useState(false);
  const [video2Loaded, setVideo2Loaded] = useState(false);

  useEffect(() => {
    if (video1Loaded && video2Loaded) {
      continueRender(handle);
    }
  }, [video1Loaded, video2Loaded]);

  useEffect(() => {
    const inputVideo = inputVideoRef.current;
    const segmentationVideo = segmentationVideoRef.current;
    const canvas = canvasRef.current;

    if (!video1Loaded || !video2Loaded || !inputVideo || !segmentationVideo || !canvas) {
      return;
    }

    const context = canvas.getContext('2d');

    if (!context) {
      return;
    }

    const renderFrame = () => {
      if (inputVideo.readyState >= 2 && segmentationVideo.readyState >= 2) {
        context.clearRect(0, 0, width, height);

        context.drawImage(inputVideo, 0, 0, width, height);
        const imageData = context.getImageData(0, 0, width, height);

        context.drawImage(segmentationVideo, 0, 0, width, height);
        const segmentationData = context.getImageData(0, 0, width, height);

        for (let i = 0; i < imageData.data.length; i += 4) {
          const alpha = segmentationData.data[i] / 255;
          imageData.data[i + 3] = alpha * 255;
        }

        context.putImageData(imageData, 0, 0);
      }
    };

    renderFrame();
  }, [frame, width, height, video1Loaded, video2Loaded]);

  return (
    <AbsoluteFill>
      <AbsoluteFill>
        <Video
          onLoad={() => setVideo1Loaded(true)}
          ref={inputVideoRef}
          src={inputVideoSrc}
          style={{ opacity: 0 }}
        />
        <Video
          onLoad={() => setVideo2Loaded(true)}
          ref={segmentationVideoRef}
          src={segmentationVideoSrc}
          style={{ opacity: 0 }}
        />
      </AbsoluteFill>
      <AbsoluteFill>
        <canvas ref={canvasRef} width={width} height={height} />
      </AbsoluteFill>
    </AbsoluteFill>
  );
};

[JonnyBurger]

Good question, probably you want to base it off the pixel manipulation example: https://www.remotion.dev/docs/video-manipulation

Where exactly does it stop working?

[gajraj-gan]

It is working, I remove the states associate with onLoad!

[gajraj-gan]

Thanks for your help!

[gajraj-gan]

Altho I wonder if the video may go out of sync anytime? Since they are playing together? Can that ever happen? @JonnyBurger

[JonnyBurger]

During rendering, it should not because the frame extractor of OffthreadVideo is deterministic and waits for each frame.

During preview, it is a bit more complicated. I think it is not guaranteed that they are in sync. @happylinks has sponsored Remotion for us to look into this problem with WebCodecs. I'm working on a solution for the future, but it's not there yet.

[gajraj-gan]

I think there is a bug, sometimes onVideoFrame does not get called. Then it just gets stuck.