New Model Shows You Don't Need Massive Language Training to Create Images from Text

Rethinking how image generators learn

Most popular text-to-image systems today, like Stable Diffusion, rely on huge amounts of prior language training. They read billions of internet captions first to learn what words mean, then combine that knowledge with image data to generate pictures.

A new paper from Stanford University asks a simple question: do you actually need all that text-only pretraining? The answer, according to the researchers, is no.

They created a family of models called Halcyon, trained entirely from scratch on 100 million image-text pairs from a dataset called GPIC. Instead of teaching the model language separately before showing it images, Halcyon learns both at the same time. This approach is called early fusion.

Faster and cheaper performance

The results surprised many in the field. A smaller version of Halcyon, with 520 million parameters, outperformed LlamaGen-XL — a much larger late-fusion model that learns text and images separately — while using just 18.2 times less pretraining compute.

An even larger Halcyon model, with 1.6 billion parameters, beat another strong late-fusion baseline called JiT-T2I-XXL while using 275 times less pretraining compute. The authors measured performance using gFD, a standard benchmark score for image generation quality, where lower is better.

The controlled experiments also showed that adding text-only pretraining did not improve the final text-to-image results.

What this means for the future

For years, the default assumption in the AI community has been that strong text-to-image generation requires massive language pretraining first. This research challenges that idea by showing that multimodal autoregressive models can learn both text and vision together from the start and still produce high-quality images.

Autoregressive models generate content one piece at a time — for images, they predict pixels or image tokens step by step — rather than denoising an entire image all at once like diffusion models do.

The team also showed that Halcyon can be adapted to caption images by replacing a technique called caption dropout with a different training method called I2T supervision. This meant the model could learn to describe images in text without sacrificing its ability to generate images from text prompts, all within the same pretraining budget.

Code and models for Halcyon are publicly available online.

This story reports on a trend for information only. It is not advice or encouragement to do any of it.

Key facts

• Halcyon was trained from scratch on 100M image-text pairs without prior language pretraining
• Halcyon-0.52B outperformed LlamaGen-XL with 18.2x less pretraining compute
• Halcyon-1.6B outperformed JiT-T2I-XXL with 275x less pretraining compute
• Text-only pretraining was shown to not improve text-to-image generation
• Halcyon can also learn image captioning within the same training budget

Sources

1. Paper page - Text-to-Image Generative Modeling Without Language Pretraining
2. Latent diffusion model — en.wikipedia.org
3. Large language model — en.wikipedia.org
4. Stable Diffusion — en.wikipedia.org
5. LTX (world model) — en.wikipedia.org