ByteDance Unveils Lance 3B: A Unified Model for Vision and Language Understanding and Generation

ByteDance Research has officially open-sourced Lance, its native unified multimodal large model.

While the AI industry typically relies on models with hundreds of billions or trillions of parameters, or those assembled by combining separate components, Lance represents a major breakthrough. It delivers full functionality with an exceptionally lightweight activation parameter count of 3B (3 billion), effectively bridging the technical divide between "understanding models (VLM)" and "generative models (DiT/Diffusion)".

Key Highlights:

Natively Unified: Built from scratch rather than stitched together, it integrates image and video understanding, generation, and cross-modal editing into a single system.

Comprehensive Performance: A single model seamlessly handles $X rightarrow T$ (text/video understanding), $X rightarrow I$ (image generation/editing), and $X rightarrow V$ (video generation/editing) — three core output tasks.

Open Source & Free: Released under the highly permissive Apache 2.0 license. Weights are fully available on Hugging Face, and the entire process can be run on a modest budget of 128 A100 GPUs.

Technical Breakdown: How Does It Achieve "Synchronization" Between Contrasting Demands?

In traditional AI architectures, "understanding" and "generation" capabilities have often been at odds: understanding tasks require filtering noise to extract high-level semantic features, whereas generation tasks focus on low-level textures, geometric structures, and temporal dynamics.

To address this industry-wide challenge, Lance employs a clever "shared context + parallel capability decoupling" design:

1. Unified Interleaved Sequence and Dual-Stream Expert Architecture

All text, image, and video inputs are first tokenized and converted into a unified "interleaved sequence." This sequence is then processed by a Dual-Stream MoE (Multi-Expert) architecture, allowing experts dedicated to "understanding" and "generation" to operate independently, effectively resolving capability conflicts.

Understanding Side: Text tokens and visual inputs utilize the embedding layer of Qwen2.5-VL and the ViT encoder to accurately extract high-level semantic visual tokens.

Generation Side: Visual inputs are compressed by Wan2.2's powerful 3D causal VAE, achieving $16times$ spatial down-sampling and $4times$ temporal down-sampling, thereby preserving detailed dynamic continuous representations.

2. MaPE (Modal-Aware Rotational Position Encoding)

Mixed visual tokens (images, text, and videos) in long sequences can lead to "boundary confusion" hallucinations. Lance introduces the MaPE mechanism, which adds fixed time offsets to different modal groups. This elegant design enables strong spatial and temporal boundary identification without disrupting the internal structure and temporal order of images and videos.

[Unified Interleaved Sequence] ──► [MaPE Modal Boundary Isolation] ──► [Dual-Stream Expert Architecture (MoE)]

Four-Phase Extreme Training: A "Lean Battle" with 128 GPUs

Contrasting with the "brute-force aesthetics" of major companies using thousands of GPUs, Lance's training process demonstrates high financial efficiency. The entire lifecycle is strictly capped at a maximum budget of 128 GPUs, progressing through four tightly integrated phases:

Phase 1: Pre-training (1.5T Tokens) —— Processes 1B image-text pairs and 140M video-text pairs to establish a robust foundation for multimodal understanding.

Phase 2: Continuous Training (300B Tokens) —— Incorporates editing, subject-driven generation, and multi-modal understanding data to activate multi-task synergy.

Phase 3: Supervised Fine-tuning (72B Tokens) —— Extensively injects human instructions, focusing on instruction following and visual identity consistency.

Phase 4: Reinforcement Learning (GRPO Algorithm) —— Utilizes group-based relative policy optimization and, notably, employs PaddleOCR as a reward model to specifically address AI issues such as inaccurate text rendering and misalignment between text and images.

Outstanding Results: 3B Model Defeats 7B Giants Across Multiple Domains

Thanks to cross-task data collaboration (where the model deepens understanding while learning to generate, and enhances generation while learning to understand), the 3B-sized Lance achieved remarkable performance across various benchmarks:

Video Generation (VBench): Scored 85.11 points! It surpassed similar all-in-one models like TUNA (84.06) and outperformed specialized video generation models such as HunyuanVideo (83.33) and Wan2.1-T2V (83.69).

Image Generation (GenEval): Scored 0.90, firmly securing a top position among global open-source models.

Video Understanding (MVBench): Scored 62.0 points, significantly exceeding the dedicated understanding model Show-o2 (7B, 55.7 points), which is twice the size of Lance.

Industry Shock: Deployment Costs for Multimodal Applications Will Plunge Dramatically

Lance's open source represents a significant industry disruption, particularly for booming fields such as AI short films, intelligent agents (Agent) collaboration, and interactive media.

Previously, developing an AI tool capable of understanding scripts, generating storyboards, and modifying visuals in real-time while maintaining character consistency required deploying, scheduling, and stitching together multiple large models (one for VLM semantics, one for Diffusion image generation, and one for temporal video). This not only caused system latency but also made pipeline alignment a nightmare for developers.

Now, Lance3B achieves "left eye sees, right eye edits, both hands create" with a single model