Alibaba Coin Team Introduces Qwen3.5-LiveTranslate-Flash: Real-time Multimedia Translation Across 60 Languages ​​with 2.8 Second Response Time

Interpretation is one of the most difficult problems in applied AI. You ask a model to translate speech before the speaker finishes the sentence. Every extra second of delay shatters the illusion of real-time communication. Alibaba’s Qwen team has been eliminating this with every release. Their latest models, Qwen3.5-LiveTranslate-Flashbrings it Latency is up to 2.8 seconds It expands input language coverage to 60 languages.

A meaningful leap from the previous version

Qwen3-LiveTranslate-Flash handles 18 input languages ​​with a response time of about three seconds. Qwen3.5-LiveTranslate-Flash This leads to 2.8 secondsexpands input coverage to 60 languages, and adds speech output in 29 languages. This is more than a 3x expansion in language coverage on the input side. For developers building multilingual products, this reduces the need to switch models for each language in most global enterprise scenarios.

The latency improvement comes from the technology for processing what the team calls “read units.” Instead of waiting for a complete sentence to arrive before producing output, the model decides when enough meaning has accumulated in a passage to commit to translation. It streams the output continuously while the speaker is still talking. This is the same basic logic as semantic unit prediction but with a tighter implementation that shave off an additional 200 ms.

The vision is now a first-class entry

Most translation systems treat audio as the only input signal. This works well in clean studio conditions. It crashes in a crowded conference room, a noisy business hall, or anywhere with overlapping sounds and poor acoustics.

Qwen3.5-LiveTranslate-Flash takes a different approach. It analyzes visual information in parallel with on-screen audio text, physically displayed objects, lip movements, and gestures. When a word is phonetically ambiguous or the audio flow deteriorates, visual context fills the gap and increases the accuracy of the translation decision. This is not a secondary feature. In real-world publishing, audio quality is rarely guaranteed. Having a vision channel means that the model handles the messy reality of live interpretation more gracefully than audio-only systems.

Audio reproduction occurs in real time

This is the part that stands out the most in the Qwen3.5 release. Standard translation systems replace the speaker’s voice with a generic synthetic voice. Qwen3.5-LiveTranslate-Flash instead reproduces the distinctive voice features of a native speaker during the translation itself. One spoken sentence is enough for the model to make this phonetic adaptation.

To listeners on the receiving end, the translated output sounds like the same person speaking the target language rather than a robotic substitute. This is important in simultaneous interpretation of live conferences, multilingual live broadcasts or international client calls. The experience feels significantly more humane than what current systems offer.

Configure domain-specific keywords

One consistent failure mode of translation models in professional settings is proper nouns and specialized vocabulary. A model that translates a medical abstract may consistently mistranslate the name of the medication. The legal interpretation session is divided into a technical legal period.

Qwen3.5-LiveTranslate-Flash addresses this problem by configuring dynamic keywords at runtime. Developers can enter a glossary of brand names, medical terms, legal terms, or technical vocabulary, and the form handles these terms more reliably. This is not available in most general-purpose translation APIs, and fills a real gap for domain-specific enterprise deployments.

Standard performance

In FLEURS and CoVoST2 – two major standards for multilingual speech translation – Qwen3.5-LiveTranslate-Flash outperforms major commercial alternatives. FLEURS tests translation quality across a wide range of language pairs under real audio conditions. CoVoST2 covers 21 translation directions of speech, making it a practical proxy for the performance of multilingual pipelines.

Visual explanation of Marktechpost

 Install dependencies
pip install websocket-client pyaudio

 Set your API key as an environment variable
export DASHSCOPE_API_KEY="your_key_here"

import json, websocket, os

API_KEY = os.getenv("DASHSCOPE_API_KEY")
API_URL = (
    "wss://dashscope-intl.aliyuncs.com"
    "/api-ws/v1/realtime"
    "?model=qwen3-livetranslate-flash-realtime"
)

def on_open(ws):
    print("Connected to Qwen3.5-LiveTranslate-Flash")

def on_message(ws, message):
    data = json.loads(message)
    print("Translation event:", data)

def on_error(ws, error):
    print("Error:", error)

ws = websocket.WebSocketApp(
    API_URL,
    header=["Authorization: Bearer " + API_KEY],
    on_open=on_open,
    on_message=on_message,
    on_error=on_error
)
ws.run_forever()

import base64, pyaudio

 Audio input config: 16kHz, 16-bit PCM mono
INPUT_RATE    = 16000
INPUT_CHUNK   = 1600   100ms per chunk
INPUT_FORMAT  = pyaudio.paInt16
INPUT_CHANNELS = 1

def on_open(ws):
     1. Send session config first
    session_cfg = {
        "type": "session.update",
        "session": {
            "input_audio_transcription": {
                "language": "zh"   source: Chinese
            },
            "translation": {
                "target_language": "en"   target: English
            }
        }
    }
    ws.send(json.dumps(session_cfg))

     2. Stream microphone audio
    pa = pyaudio.PyAudio()
    stream = pa.open(
        rate=INPUT_RATE, channels=INPUT_CHANNELS,
        format=INPUT_FORMAT, input=True,
        frames_per_buffer=INPUT_CHUNK
    )
    while True:
        chunk = stream.read(INPUT_CHUNK)
        audio_b64 = base64.b64encode(chunk).decode()
        ws.send(json.dumps({
            "type": "input_audio_buffer.append",
            "audio": audio_b64
        }))

import cv2, base64, threading, time

def stream_video_frames(ws):
    cap = cv2.VideoCapture(0)   0 = default camera
    while True:
        ret, frame = cap.read()
        if not ret:
            break
         Encode frame as JPEG → base64
        _, buf = cv2.imencode(".jpg", frame)
        img_b64 = base64.b64encode(buf).decode()
        ws.send(json.dumps({
            "type": "input_image_buffer.append",
            "image": img_b64
        }))
        time.sleep(0.5)   ~2fps is sufficient

 Run video streaming in a separate thread
threading.Thread(
    target=stream_video_frames,
    args=(ws,), daemon=True
).start()

 Add to your session.update payload
session_cfg = {
    "type": "session.update",
    "session": {
        "input_audio_transcription": {
            "language": "zh"
        },
        "translation": {
            "target_language": "en"
        },
         Inject domain keywords here
        "keywords": [
            {"source": "达芬奇机器人",  "target": "da Vinci Surgical System"},
            {"source": "腹腔镜",      "target": "laparoscope"},
            {"source": "实体瘤",      "target": "solid tumor"}
        ]
    }
}
ws.send(json.dumps(session_cfg))

Key takeaways

• Qwen3.5-LiveTranslate-Flash provides instant multimedia translation across 60 input languages ​​and 29 speech output languages ​​with a response time of 2.8 seconds.
• The model uses enhanced visual understanding – reading lip movements, gestures and on-screen text – to maintain accuracy in noisy or degraded audio environments.
• Real-time audio reproduction replicates the original speaker’s audio file in the translated output using just one spoken sentence.
• Semantic unit prediction via Reading Unit processing enables continuous stream output without waiting for complete sentences, reducing latency to 2.8 seconds.
• Dynamic keyword configuration allows developers to introduce domain-specific glossaries at runtime, improving translation reliability for technical, medical, and legal terms.

verify Technical details. Also, feel free to follow us on twitter Don’t forget to join us 150k+ mil SubReddit And subscribe to Our newsletter. I am waiting! Are you on telegram? Now you can join us on Telegram too.

Do you need to partner with us to promote your GitHub Repo page, face hug page, product release, webinar, etc.? Contact us