An arena-style contest pits the world’s best text-to-image generators against each other.
What’s new: Artificial Analysis, a testing service for AI models, introduced the Text to Image Arena leaderboard, which ranks text-to-image models based on head-to-head matchups that are judged by the general public. At the time of this writing, Midjourney v6 beats more than a dozen other models models in its ability to generate images that reflect input prompts, though it lags behind competitors in speed.
How it works: Artificial Analysis selects two models at random and feeds them a unique prompt. Then it presents the prompt and resulting images. Users can choose which model better reflects the prompt. The leaderboard ranks the models based on Elo ratings, which scores competitors relative to one another.
- Artificial Analysis selects models to test according to “industry significance” and unspecified performance tests. The goal is to identify and compare the most popular, high-performing models, especially those that are available via APIs. (Midjourney, which has no API, is an exception.) Only 14 models meet this threshold, but Artificial Analysis says it is refining its criteria and may include more models in the future.
- Users who have voted at least 30 times can see a personalized leaderboard based on their own voting histories.
- Separate from the Text to Image Arena, Artificial Analysis compares each model’s average time to generate and download an image, calculated by prompting each model four times a day and averaging the time to output over 14 days. It also tracks the price to generate 1,000 images.
Who’s ahead?: As of this writing, Midjourney v6 (Elo rating 1,176), which won 71 percent of its matches, holds a slim lead over Stable Diffusion 3 (Elo rating 1,156), which won 67 percent. DALL·E 3 HD holds a distant third place, barely ahead of the open-source Playground v2.5. But there are tradeoffs: Midjourney v6 takes 85.3 seconds on average to generate an image, more than four times longer than DALL·E 3 HD and more than 13 times longer than Stable Diffusion 3. Midjourney v6 costs $66 per 1,000 images (an estimate by Artificial Analysis based on Midjourney’s policies, since the model doesn’t offer per-image pricing), nearly equal to Stable Diffusion 3 ($65), less than DALL·E 3 HD ($80), and significantly more than Playground v2.5 ($5.13 per 1,000 images via the Replicate API).
Behind the news: The Text to Image Arena is a text-to-image counterpart of the LMSys Chatbot Arena, which lets users write a prompt, feed it to two large language models, and pick the winner. imgsys and Gen-AI Arena similarly let users choose between images generated by different models from the same prompt (Gen-AI Arena lets users write their own). However, these venues are limited to open models, which excludes the popular Midjourney and DALL·E.
Why it matters: An image generator’s ability to respond appropriately to prompts is a subjective quality. Aggregating user preferences is a sensible way to measure it. However, individual tastes and applications differ, which makes personalized leaderboards useful as well.
We’re thinking: The user interface for some image generators implicitly asks users to judge images. For example, Midjourney defaults to generating four images and asks users which they want to render at higher resolution. This can give the image generator valuable feedback about which image users like. Perhaps data gathered by an arena could feed an algorithm like reinforcement learning from human feedback to help generators learn to produce output that people prefer.