Unifying Visual Understanding and Generation via Text-Aligned Representations

This paper presents a multimodal framework that attempts to unify visual understanding and generation within a shared discrete semantic representation. At its core is the Text-Aligned Tokenizer (TA-Tok), which converts images into discrete tokens using a text-aligned codebook projected from a large language model's (LLM) vocabulary. By integrating vision and text into a unified space with an expanded vocabulary, our multimodal LLM, Tar, enables cross-modal input and output through a shared interface, without the need for modality-specific designs. Additionally, we propose scale-adaptive encoding and decoding to balance efficiency and visual detail, along with a generative de-tokenizer to produce high-fidelity visual outputs. To address diverse decoding needs, we utilize two complementary de-tokenizers: a fast autoregressive model and a diffusion-based model. To enhance modality fusion, we investigate advanced pre-training tasks, demonstrating improvements in both visual understanding and generation. Experiments across benchmarks show that Tar matches or surpasses existing multimodal LLM methods, achieving faster convergence and greater training efficiency.

Framework. Tar is a unified multimodal LLM for both visual understanding and generation, which consists of an autoregressive LLM, a visual tokenizer TA-Tok and a visual de-tokenizer. Different from previous works, Tar leverages fully discrete, text-aligned visual tokens, eliminating the need of modality-specific designs like visual projectors. We can train Tar using the standard next-token prediction objective.

Visual Tokenizer. The key design of Tar is a text-aligned visual tokenizer, TA-Tok. It adds a vector quantization module to pretrained SigLIP2, thus converting input images into semantic, discrete tokens. Unlike other discrete tokenizers (e.g., VQVAE), TA-Tok directly leverages LLM's token embeddings as its codebook. The visual token can be represented by a transformed LLM token. Therefore, training unified MLLM with TA-Tok is similar to adding foreign languages to the LLM.

De-Tokenizer. Since the visual tokenizer TA-Tok is fully text-aligned, it cannot decode images directly like VQVAE. Instead, we propose visual de-tokenizers to decode visual tokens back to images. Here are two variants: an autoregressive model and a diffusion-based model. The AR de-tokenizer works well with discrete visual tokens from TA-Tok, while for the diffusion-based de-tokenier, we can leverage pretrained models for fast adaptation.