Limitations

This page is the honest list. Everything here is a real constraint, not a setting that has been overlooked. Where a workaround exists, it is given.

Speaker separation

WhisperSubs cannot label who is speaking, and cannot split a cue by narrator. There is no toggle for this because there is no data to expose.

Whisper is a speech-to-text model. It predicts text tokens from audio. Speaker identity never appears in its training targets or in its output vocabulary, so nothing anywhere inside it computes "who said this".

The two whisper.cpp flags that look like they solve it

-di / --diarize is documented as stereo audio diarization. The implementation is a loudness comparison between the left and right channels: if the left is more than 1.1x louder it prints speaker 0, if the right is louder it prints speaker 1, otherwise it prints ?. It is gated on the input having exactly two channels. It was built for recordings where each speaker is hard-panned to their own channel, such as a two-mic interview or a phone call. Film and TV dialogue is mixed to the centre and arrives at near-identical energy in both channels, so almost every line falls into the ? branch. It does not recognise voices at all.

-tdrz / --tinydiarize is genuine speaker-turn detection, with severe constraints. It requires the specific ggml-small.en-tdrz.bin checkpoint, which is English-only (finetuned on the AMI meeting corpus) and a small model. WhisperSubs downloads the large-v3 family because accuracy matters on film dialogue. Switching to tinydiarize would remove transcription entirely for every non-English user and make English output materially worse. It still would not answer the question: it emits a bare [SPEAKER_TURN] marker meaning "somebody else started talking" — not who, and not how many people. It is explicitly experimental, at roughly 71% turn recall on its own benchmark.

What real diarization would take

Proper diarization (WhisperX, whisper-diarization, NVIDIA NeMo) is a second machine-learning pipeline running alongside Whisper: voice-activity detection, then neural speaker embeddings, then clustering, then aligning that speaker timeline onto Whisper's word timestamps.

In practice that means a Python and PyTorch runtime sitting next to this plugin's self-contained native binary, plus pyannote's gated models, which require a Hugging Face account and a manually accepted licence before they can be downloaded at all. It is typically the slowest and most memory-hungry stage of the pipelines that use it, often comparable to the transcription itself.

The accuracy is the harder problem. The pyannote speaker-diarization-3.1 model card publishes that pipeline's own error rates, measured the strict way: no forgiveness collar, overlapped speech included, no manual voice activity detection and no manual speaker count. On that setup it reports 18.8% DER on AMI (headset mix, only_words) and 21.7% on DIHARD 3 (full). Neither set is film audio. The closest one in the same table is AVA-AVD, 351 five-minute clips cut from 117 movies, and there the pipeline reports 50.0%.

DER counts missed speech, false alarms and speaker confusion against total speech time. At 50% on movie clips there is no version of this that ships as correct. The reasons are what film audio is made of: overlapping dialogue, background score and effects masking voice differences, and short speaker turns.

Even at its best, it produces anonymous cluster labels — SPEAKER_00, SPEAKER_01 — never character or actor names. Getting from those to "Ellie" and "Joel" is a further unsolved step that needs per-show voice enrollment.

.srt also has no speaker field. Even perfect diarization has to be rendered by convention: Netflix's style guide uses a leading hyphen per speaker, one speaker per line, with a bracketed name when the speaker is not visible on screen.

The one path that exists elsewhere

OpenAI offers a gpt-4o-transcribe-diarize model that does diarization server-side and returns speaker-labelled output through a diarized_json response format (labels come back as "A", "B", and so on, unless you supply reference samples).

WhisperSubs negotiates srt and verbose_json only. It neither requests nor parses diarized_json, so pointing a remote worker at that model today does not work. It is also a paid cloud service that uploads your audio, which is the opposite of why most people run this plugin.

Workaround

Run diarization outside the plugin, on the finished subtitle or the media file: WhisperX or whisper-diarization. WhisperSubs produces the transcript; those tools add the speaker layer.

Run-on lines with no punctuation

whisper.cpp applies no cue-length cap of its own — --max-len defaults to 0, meaning unlimited. When the model drops into its no-punctuation mode, a whole utterance lands in one enormous cue.

The fix is a setting. In Dashboard → Plugins → WhisperSubs, set Maximum subtitle line length to 47. That passes --max-len 47 together with --split-on-word, so a capped line breaks on a word boundary instead of mid-word. Broadcast subtitling caps a line near 42 characters.

The default is 0, so upgrading changes nothing for Whisper and existing subtitles are untouched. Regenerate an item to see the difference.

Two scope notes:

• The setting reaches the Jellyfin host's own whisper-cli, and its own crispasr for Canary targets. Canary is capped even at 0: it uses 42, or 84 for Bulgarian, Greek, Russian and Ukrainian, because crispasr counts bytes and those letters take two each. A remote worker, a CrispASR server included, segments its own output, so set WHISPER_MAX_LEN=47 on a whisper worker as well. See Remote workers and hosted providers.
• A --max-len placed in Custom Whisper Arguments wins, because custom arguments are appended last and whisper-cli takes the last value.

This is a guard against the worst case, not a cure for missing punctuation. Whisper normally punctuates. When it produces none at all, the cause is usually a small model (use Large V3 Turbo (Q5), the 574 MB recommended default) or a wrong language guess, which applies one language's punctuation conventions to another language's words. If you know the audio language, set it explicitly instead of leaving it on auto.

Translation

Two separate limits apply, and the second one is easy to hit by accident.

English is the only target for non-English audio

Whisper's --translate only produces English. It was trained for X to English and nothing else, so no other target can be reached from inside whisper.cpp. The upstream requests are whisper.cpp#1219 and whisper.cpp#1956 (a forced-decoding workaround, unreliable), and an OpenAI maintainer confirmed the limit in openai/whisper#1167.

English audio is the exception. More target languages translates it into 24 European languages with NVIDIA Canary. That includes a film in another language that also carries an English-tagged audio track, such as a dub: Canary translates from that track. Audio in any other language still translates to English only. Canary has no direct path between two non-English languages, and when we tried French to Spanish and Spanish to French or German anyway, it returned mixed-language cues, dropped cues or an empty file, with exit code 0 and no warning. The plugin therefore never asks it for such a pair.

Canary output is experimental:

• In testing, one speech segment in eight came out untranslated: English text inside a Spanish file, the same segment on every run.
• Its translation into English was weaker than Whisper's, which is why the English subtitle always comes from Whisper.
• Its quality on film audio with music, effects and overlapping speech has only been checked on short clips.

WhisperSubs will not add a text-to-text translation step. Translating an existing .srt is a different pipeline stage with a different API contract, different failure modes and a separate dependency. Coupling subtitle generation to a translation service's availability doubles the config surface for something orthogonal to speech-to-text.

Workaround

For a non-English title, generate the English subtitle here, then translate the file with a dedicated tool. Lingarr mounts your media directories and translates subtitles in place, with LibreTranslate, DeepL, Google, OpenAI, Anthropic, Gemini, DeepSeek, Bing, Yandex, Azure and local Ollama backends. Sublarr is another option with Sonarr/Radarr webhook support.

The recommended model cannot translate

Warning

The recommended model, Large V3 Turbo (Q5) (ggml-large-v3-turbo-q5_0.bin), cannot translate. Neither can Large V3 Turbo (F16). Both are flagged canTranslate: false in the model catalog.

whisper.cpp's distilled turbo models were fine-tuned without the translate task, so --translate emits the source language instead of English.

WhisperSubs does not block this. It logs a warning and runs the job anyway. The result is source-language text written into an English-named file: with the default filename template {name}.{lang}.{label}{.type}, that is Movie.en.WhisperSubs.translated.srt. Nothing in the file name, and nothing in Jellyfin's subtitle picker, says the text is not English. The only signal is the warning line in jellyfin.log.

The same applies to Forced Only and Full + Forced in an English-primary library, where foreign-language inserts are translated to English: a turbo model emits the source language, and again only a log warning is raised.

To translate reliably, open the plugin setup page and download and activate one of these instead:

Model	Size	Note
Large V3 (Q5)	1081 MB	Best choice for non-English audio and translation
Large V3 (F16)	3095 MB	Maximum accuracy, 3-4x slower than turbo
Medium (Q5)	539 MB	Similar size to turbo-q5, slower and less accurate

Then regenerate the affected items.

Forced subtitles in a non-English library stay in the source language

Forced subtitles are translated to English only when the title's primary language is English. When the primary language is something else, Whisper has no path to that target, so the foreign-language inserts are kept as transcribed rather than written as mislabelled English into a .<lang>.forced file. A French insert in a Spanish film ends up as French text, not Spanish.

Forced subtitle generation is slow

Forced mode runs a multi-step pipeline: audio extraction, VAD-based speech segmentation, then per-chunk language detection on every ~30-second segment, before any transcription starts. For a 2-hour film that is roughly 240 individual detection calls up front. On CPU that phase alone can take 10-20+ minutes per title. GPU acceleration helps significantly.

A dedicated ~148 MB base model is downloaded for the detection pass specifically to keep it cheap, but the call count is inherent to the approach.

If you do not need foreign-dialogue-only tracks, use Full mode.

Language detection falls back to the CPU on a GPU that gets it wrong

Language detection runs on the GPU. Before the first detection with a given whisper-cli and model, the plugin runs a short English clip through it, and the answer has to be English with real confidence. whisper.cpp v1.8.4 on Vulkan with an Intel UHD 770 failed that: it named every stretch Dutch at p = 0.010. When the check fails, the Jellyfin log says so once at Warning, and detection runs with --no-gpu until Jellyfin restarts or the whisper-cli binary changes, including an engine swapped behind a wrapper script. The bundled builds use whisper.cpp v1.9.5, which passes on that GPU. A binary you supply yourself goes through the same check.

One worker still means one item at a time

Transcription concurrency is capped by the pool's total concurrency, which is the sum of every worker's max concurrency. A default install has one local worker at max concurrency 1, so everything runs one item at a time: the scheduled task's queued requests, its library sweep, and manual Generate and Generate All alike.

You can raise a worker's own max concurrency, but whisper.cpp runs one transcription per GPU context, so a single-GPU worker should stay at 1. Raising the ceiling means adding workers. See Remote workers and hosted providers.

Qwen3-ASR cues are timed by VAD unless the aligner is installed

With the Qwen3-ASR engine the speech model returns text per speech segment and no timestamps of its own. The Silero VAD model cuts the audio into segments, each segment becomes one cue, and Maximum subtitle line length splits a long one into pieces timed in proportion to their length. Where music runs under dialogue, VAD can keep one segment open for twenty seconds or more, and that cue then shows early and stays late. Whisper times cues from its own decoder and does not have this failure.

The fix is the Word timing download in the same panel: the Canary CTC aligner places every word, and in testing the cue that started 22 s early with VAD timing started on the word with the aligner. It costs about twice the run time and knows only English and the 24 Canary languages; a title in any other language keeps VAD timing. Even with it, a cue can still span a pause when the text split falls there, because cues are cut by length and punctuation, not by silence. Align subtitles to speech snaps cue starts to speech afterwards in both cases.

Qwen3-ASR does not translate, so Translation to English keeps using the Whisper model even with Qwen3-ASR selected, and a title that needs English still needs a Whisper model that can translate.

In the forced-subtitle pass of a Qwen3-ASR title, a foreign line in a language outside Qwen3-ASR's 30 (Catalan, Galician, Basque, Norwegian and others) is transcribed by the Whisper model instead, so it holds a full Whisper model in memory while it runs. Detecting languages with the small detection model often names Spanish as Galician or Catalan, and each such line then costs a Whisper run and comes out wrong; Detect languages with the transcription model makes those detections rare.

Pause during playback suspends on Linux only

Pause generation during playback stops the engine processes (whisper-cli, crispasr, FFmpeg extraction, vocal separation) with SIGSTOP and continues them with SIGCONT, on Linux. Nothing is lost in any pass, including the forced-subtitle pass, which cannot resume from partial output. A job that has been held for four hours continues anyway and is not stopped again.

On macOS and Windows the old behaviour stays: the job is cancelled and retried, which is cheap for a full subtitle (Whisper writes as it goes, Qwen3-ASR saves ten-minute windows) and expensive for a forced one. macOS is excluded on purpose: a stopped child process sends .NET's own child-process bookkeeping into a busy loop there, and every later wait on a process hangs.

Two things to know on Linux. A suspended engine keeps its memory, GPU memory included, until it continues. And if Jellyfin is killed outright while an engine is suspended outside a container, that process stays stopped until you end it yourself; in a container it goes away with the container.

Engine auto-download is Linux only

The setup page downloads prebuilt whisper-cli binaries for two platforms:

Platform	Variants offered
linux-x64	CPU, CPU compatibility (no AVX), CUDA 12, CUDA 12 compatibility, Vulkan, Vulkan compatibility, AMD ROCm
linux-arm64	CPU, CPU compatibility (no AVX)

Every other platform gets no variants at all. On Windows or macOS, build or install whisper.cpp yourself and point Whisper Binary Path at the binary. Model downloads are unaffected and still work everywhere.

Lyrics generation is experimental

.lrc lyrics generation for music libraries is off by default. Whisper is trained on speech, not singing, so accuracy on music is well below what the same model achieves on dialogue.

Image-based subtitles are never converted

PGS and VOBSUB tracks are images. WhisperSubs does not OCR them — it transcribes the audio instead. By default an image track does not count as "already subtitled", so a text subtitle is still generated alongside it. Turning on Count image-based subtitles as present suppresses that, at the cost of leaving the title without a searchable text track.

Hosted models that return text without timestamps

A subtitle needs timings. response_format=json returns the transcript text and no timestamps at all, so it cannot be synchronized, and WhisperSubs rejects such a response rather than fabricating timings.

On OpenAI, timestamps are available on whisper-1 only. gpt-4o-transcribe and gpt-4o-mini-transcribe accept json and text only, so they return no timings and cannot be used for subtitles.

The .srt WhisperSubs writes is not always the provider's own srt response. When an endpoint refuses srt, the plugin converts the segments of a verbose_json response into SRT itself. Both paths still need the provider to return timestamps. On OpenRouter, timestamps exist only on its OpenAI-compatible models.

Details, and the matching upload-size limits, are in Remote workers and hosted providers.