Whisper realtime translates non-English speech despite 99-language claim

[oscarbrey]

Impact

This is currently blocking the multilingual launch of our production macOS dictation application. The realtime openai/whisper-large-v3 endpoint translates non-English speech into English instead of transcribing it in the original language.

Primary reproduction: raw WebSocket protocol

Using Together's documented realtime WebSocket protocol:

1. Connect to:
   wss://api.together.ai/v1/realtime?intent=transcription&model=openai/whisper-large-v3&input_audio_format=pcm_s16le_16000&turn_detection=none
2. After session.created, send:

   {"type":"transcription_session.updated","session":{"language":"auto"}}

3. Append 16 kHz mono PCM16 audio and commit the buffer.
4. Repeat with language: "es".

For audio saying:

Hola, buenos días.

Both Auto and explicit Spanish return:

Hello, good morning.

Independent confirmation

I reproduced the same behavior with Together's official Python SDK v2.28.0 using client.beta.realtime.transcription(...), with both language="auto" and language="es". A second Spanish recording returned an empty result when Spanish was explicitly selected.

Controls

The same recordings are transcribed correctly by:

• Together Parakeet realtime streaming
• Together Whisper REST with explicit language="es"

This isolates the problem to the Together Whisper realtime path rather than the audio or client encoding.

Expected behavior

Realtime Whisper should preserve the source language, consistent with the model page's claim that streaming Whisper supports 99 languages:

https://www.together.ai/models/whisper-large-v3-streaming

Requested action

Please confirm whether realtime Whisper is currently English-only, fix the endpoint or correct the documentation, and provide an ETA or production-safe workaround. I can provide the original recordings and complete WebSocket event logs immediately.

@yadavsahil197 @sbeurnier @zainhas @blainekasten — tagging the public author and reviewers of the realtime SDK implementation because this is critical for our launch.

[yadavsahil197]

@oscarbrey thanks for flagging the issue.
The team is looking into the automatic language detection for Whisper and we are also fixing the issue where session update with language was not being respected.
Meanwhile, to unblock you, I suggest you can pass the language as a query param in the websocket url. As per your example, you can update the ws url to the following.

wss://api.together.ai/v1/realtime?intent=transcription&model=openai/whisper-large-v3&input_audio_format=pcm_s16le_16000&turn_detection=none&language=es

This will set the language correctly for the session.