Tencent WeMM-Video-2B open-source launch: What can the 2B audio and video understanding model do?
HyphenTech · Field notes

Tencent WeMM-Video-2B open-source launch: What can the 2B audio and video understanding model do?

4 min read
Categories:
Tags:
Summary

Tencent WeMM-Video-2B open-source launch: What can the 2B audio and video understanding model do?

Info: Machine translation This post was machine-translated from the Chinese original. Wording may be rough in places — the Chinese version is authoritative.

Tencent Hunyuan has released a new model repository on Hugging Face: tencent/WeMM-Video-2B. According to the official model page, it is a dense 2B checkpoint in the WeMM-Video series, belonging to a bilingual model family focused on joint audio-video understanding and reasoning; The weight is BF16 Safetensors, officially marked as a 6.12 GiB weight file. The repository page also lists four quantization variants: FP8 Block128, AWQ W4A16, GGUF Q4_K_M, and NVFP4 W4A16. The license section lists Apache License 2.0.

Official Model Page

The problem this model solves is very specific: given a video, can it simultaneously hear what is said and what happens, and combine both to answer the question? The official website provides the answer yes—it processes video frames and the original audio track together to describe events, transcribe speech, and reason about what is seen and heard; It can also ask questions only for silent video, standalone audio, images, or plain text, with answers in Chinese or English.

How exactly does it handle video and sound?

According to the architecture on the official model page, the visual encoder and audio encoder turn the media into tokens, and the language model processes these tokens together with the text; Video frames come with timestamps. This explains a practical difference: only viewing the image versus simultaneously viewing the image plus sound are two different input forms, not two different ways of saying the same thing. In the Python call example, these two methods are separate—video only handles the frame, video_audio carries the original audio track, and both pass the file path in the video field.

AI Mechanism Illustration

The two official examples clearly illustrate its positioning: one is “a 20-minute meeting starts at 09:45, when does it end?” and the other is “what do the planets in this chart look like?” The former relies on timestamps and time arithmetic, while the latter is a single-image understanding. In other words, the focus is not on generating videos, but on reading the video and then writing the answers into text.

The official training scale is also stated: this family first pre-trained on about 1.2 trillion modal tokens, then used multi-teacher OPD, anti-duplication MrDPO, and specialized reinforcement learning for post-training. This is the official training quote and does not represent any third-party independent evaluation.

How to run after downloading

The shortest official path is to run a sample script from the root directory of the WeMM-Video repository to load an MP4 with the audio track:

python examples/transformers_video.py --model tencent/WeMM-Video-2B \
  --video ./clip.mp4 --prompt "Describe what happens and summarize what you hear."

The official team specifically mentions that GPU VRAM must be reserved for full weight files, media input, and generation cache. This determines the answer to “whether it can run locally”—whether it can run depends on whether your VRAM can simultaneously load the model, assets, and decoding cache, not just the model itself. The official also reminds you to use --revision to fix the model and processor versions to a specific Hub submission to avoid pulling inconsistent versions.

Official deployment requirements

Deployment isn’t just about installing a package

Warning To make it a continuously invokable service, the official team requires installing the corresponding WeMM runtime patch first. The two service solutions provided on the page specify the patched version: vLLM 0.18.0 (with another patched vLLM 0.31.0 for BF16) and SGLang 0.5.16 / 0.5.19. In other words, starting directly with a community-ready vLLM or SGLang version is likely not the officially verified path.

MODEL=tencent/WeMM-Video-2B
vllm serve "$MODEL" --served-model-name "$MODEL" --trust-remote-code --dtype bfloat16 \
  --host 127.0.0.1 --port 8000 --max-model-len 8192 \
  --no-enable-prefix-caching --mamba-ssm-cache-dtype float32 \
  --limit-mm-per-prompt '{"image":4,"video":2,"audio":2}' \
  --media-io-kwargs '{"video":{"fps":2,"min_frames":4,"max_frames":16,"total_pixels":262144}}'

After launching the service, the request goes through an OpenAI-compatible chat interface, with media directly inlined into the request: text is placed in messages[].content, images use image_url, audio use audio_url, video use video_url, and use_audio distinguish whether to send frames only or audio tracks together. Sampling frequency and maximum frame count are controlled in the video_url; for example, in the official example, fps is 2 and max_frames is 16.

Managed Status

Official Verification Record

The model has two dialogue modes: Instruct is the default mode. When set to apply_chat_template, passing enable_thinking=False gives a direct answer; passing enable_thinking=True triggers a reasoning response. When using video reasoning mode for tasks that require time arithmetic or cross-segment checking, this switch is worth trying first.

How far has the official verification been reached, and which ones have yet to be verified?

The verification section on the official model page is very restrained: the patched vLLM loaded this checkpoint on L20 and H20, and completed generation checks on text, images, audio, and video with audio tracks; The local checkpoint also passed direct loading with Transformers and a single CPU text generation check. This is the scope of the official report verification, which does not include consumer graphics cards nor third-party independent reviews.

It should also be noted that there is a practical limitation in the release status: the Inference Providers section in the repository shows that no inference providers have deployed this model. The page directly states “not deployed by any Inference Provider” and provides an entry point for platform requests. In other words, there is no ready-made managed API to call directly; the only option is to deploy it yourself.

How to judge if it’s worth building on your own

Putting these conditions together, a practical judgment is: this is a 2B audio and video understanding model with open weights, lenient licensing (Apache 2.0), and clear task positioning. The best use case is to build your own service, perform video summarization, voice transcription, and mixed text-and-text inference locally or on your own machine; The unsuitable use case is to treat it as an “online interface you can call today.” The threshold for the former lies in memory budget, version alignment of official patches, and experience deploying SGLang/vLLM; The latter doesn’t hold up not because the model is weak, but because the hosting provider hasn’t gone live yet.

Tip If you want to get started, follow the official order for acceptance: first, run an MP4 with audio tracks in the repository root directory using the Transformers example to confirm you can use both frame and sound simultaneously; Then install the corresponding runtime patches according to the serving guide, and start the service using the official vLLM or SGLang commands; Finally, use the project’s built-in chat.py client to test text, image, audio, video, and video-audio requests, confirming all five can return and resume your own service. The L20 and H20 mentioned in the verification section are just hardware it has verified; when applied to other cards, you still need to measure the VRAM and speed yourself.

The next verifiable questions are two things: how much VRAM and time the quantized variants can save on regular graphics cards, and the stability of the five input modes on real business materials. Neither official material has given a conclusion on these points, so you’ll need to test it yourself after deployment; The warehouse page shows that this series includes 21 entries and is still being updated, indicating that this audio-video line is still ongoing.

Official information

Self-made software

Versions, changes and supported installers all come from the same public release records.

Share:

Comments

Loading comments…