Broken (repeating, hallucination) subtitles

[cupiditys]

On each video I upload, the subtitles repeat for about 4 minutes before coming back. I tried with LargeV2 and V3 models.

[umlx5h]

This is a problem I have been aware of for some time.
What runtime are you using, CPU, Vulkan or CUDA?

Re-enabling ASR will recreate the subtitles from the playback position, so that may help.
(Subtitles after the start position will be deleted, but before will remain.)

I reproduced it even when I fed the WAV files directly to whisper.net, so the cause may be on the library side (whisper.net or whisper.cpp) .

debug code: output wav

LLPlayer/FlyleafLib/MediaPlayer/SubtitlesASR.cs

Lines 211 to 216 in b82d422

	// Output wav file for debugging
	//using (FileStream fs = new($"subtitlewhisper-{chunkCnt}.wav", FileMode.Create, FileAccess.Write))
	//{
	// waveStream.WriteTo(fs);
	// waveStream.Position = 0;
	//}

I will do a detailed investigation later.

[cupiditys]

I was using CPU in the gif, tried using cuda and it has the same issue (seems worse really, but unsure).

re-enabling does work but just a bit tedious since you have to do it multiple times

[umlx5h]

Thanks, I donn't have a nvidia gpu so that helps.
I have confirmed it reproduces by Vulkan using Radeon as well, I will investigate it later.

[umlx5h]

I have confirmed that when I feed a specific WAV file directly to whisper.net, it does not reproduce on the CPU, but definitely on Vulkan.

Apparently there are two types of repeating problems: the one that occurs on the CPU must be a problem on the whisper.cpp side, which is the problem of the previous subtitle repeating when silent. which should be resolved when the next audio comes in.

The problem that occurs with vulkan and cuda is that the previous subtitle repeats with audio present, which should be fixed.

I will check whether whisper.net or whisper.cpp has the vulkan/cuda problem.

---

I bought an nvidia GPU and have confirmed that I can reproduce this with a combination of CPU/CUDA and LargeV3.

I need to set up an environment where I can run whisper.cpp directly in a Windows environment.

[umlx5h]

It seems to be a problem on the whisper.cpp or original whisper models, so we will have to wait patiently for it to be fixed in an upgraded version.

sandrohanea/whisper.net#337
ggml-org/whisper.cpp#2191

There are several ways to deal with this.

• Change to a different model.
  • More likely to occur with LargeV3 and LargeV3 Turbo; less likely with LargeV2
• Change runtime options
  • Change to Cuda or Vulkan or CPU
• Change whisper parameters
  • Set NoContext to True
• Re-run ASR from the repeat point.
  • If ASR is enabled and you select it again, it will be re-generated from the playback point.
• Set ASR Chunk Size to a smaller size in the settings.
  • There is a possibility to eliminate repeats because the whisper is run again at short intervals.

[carbonatedturtle]

It seems to be a problem on the whisper.cpp or original whisper models, so we will have to wait patiently for it to be fixed in an upgraded version.

sandrohanea/whisper.net#337 ggerganov/whisper.cpp#2191

There are several ways to deal with this.

* Change to a different model.
  
  * More likely to occur with `LargeV3` and `LargeV3 Turbo`; less likely with `LargeV2`

* Change runtime options
  
  * Change to Cuda or Vulkan or CPU

* Re-run ASR from the repeat point.
  
  * If ASR is enabled and you select it again, it will be re-generated from the playback point.

* Set `ASR Chunk Size` to a smaller size in the settings.
  
  * There is a possibility to eliminate repeats because the whisper is run again at short intervals.

I've tried everything here and it just keeps repeating the same 3 or 4 lines over and over, while the video keeps playing. I am trying to translate from a stream of live Japanese TV, so maybe this player just isn't able to keep up with translations of live videos yet, but it definitely seems like a bug when I can see the translated subs populating in the sidebar while it cycles through the same few lines over and over.

[umlx5h]

@carbonatedturtle

I have a few questions.

1. can you provide the URL of the reproduced video?
2. what runtime options and whisper model you are using
3. does the repeat occur in the silent audio part? or during speech?

I can output a WAV file here and feed it directly to whisper to see if there is a problem on the library side.

I can see the translated subs populating

This refers to the translation of whisper? If you are using the LLPlayer translation, the translation process is completely irrelevant.

[carbonatedturtle]

@carbonatedturtle

I have a few questions.

1. can you provide the URL of the reproduced video?

2. what runtime options and whisper model you are using

3. does the repeat occur in the silent audio part? or during speech?

I can output a WAV file here and feed it directly to whisper to see if there is a problem on the library side.

I can see the translated subs populating

This refers to the translation of whisper? If you are using the LLPlayer translation, the translation process is completely irrelevant.

Thanks for the reply.

1. https://raw.githubusercontent.com/luongz/iptv-jp/refs/heads/main/jp.m3u (The default channel has no video stream so you'll need to select another from the playlist.)

2. I've tried all 5 runtime options and Tiny, LargeV2, LargeV3, LargeV3Turbo.

3. The subtitles cycle through the same few lines regardless of what audio is playing while they continue to populate in the sidebar.

This refers to the translation of whisper? If you are using the LLPlayer translation, the translation process is completely irrelevant.

I'm unsure about this. I was just referring to the subtitles that populate in the sidebar as the stream continues to play.

Edit: Here's a video showing what's happening.

[umlx5h]

This is unrelated to this issue and appears to be an issue with ASR not being available on the live stream.
The problem with the title is a different issue, with the same subtitles being generated over and over again.

If it is not possible to seek throughout the video, ASR may not be available.
I will do some research, but it seems difficult to use ASR if it is not possible to seek throughout the video.
(I might add a validation to make it unavailable).

I've created an another issue.
#48

[umlx5h]

@carbonatedturtle
For the time being, ASR cannot be performed in the live stream.
Unfortunately, it seems to be difficult to support ASR of live streams with the current implementation.

#48 (comment)
a170a58