Best DiffusionBench: On Holistic Evaluation of Diffusion Transformers alternatives.
Live source-backed alternatives to DiffusionBench: On Holistic Evaluation of Diffusion Transformers for Image generation. Alternatives are selected from the same task category and update whenever the best-of index rebuilds.
DiffusionBench: On Holistic Evaluation of Diffusion Transformers
Diffusion transformer (DiT) research on image generation has converged to a single evaluation setup: class-conditional generation on ImageNet. While methods improve the FID and related metrics, it is increasingly unclear whether they reflect real progress in generative modeling. The natural alternative, i.e., text-to-image (T2I) generation, is perceived as too costly or inconvenient to train and evaluate and is often skipped. We argue that this perception no longer holds. We introduce NanoGen, a unified DiT training and evaluation framework. NanoGen matches state-of-the-art DiT baselines on ImageNet and, with 12 lines of configuration change, also trains competitive text-to-image models. It currently supports RAE, VAE, pixel-space, and MeanFlow diffusion methods under both ImageNet and T2I setups. Under NanoGen, training T2I requires comparable compute to ImageNet. After training 21 latent diffusion models with NanoGen, we observe that method ranking shows no strong correlation between ImageNet and T2I generation: Pearson correlation is between -0.377 and -0.580 across three metrics. This suggests that a method which improves class-conditional ImageNet FID may show no corresponding improvement on T2I, clearly indicating the necessity of evaluating DiTs on both tasks. To this end, we summarize ImageNet and text-to-image results, which yields DiffusionBench, a holistic benchmark for DiT research. We recommend reporting DiffusionBench in place of ImageNet alone: methods that improve DiffusionBench are more likely to reflect broader progress. cs.CV Diffusion transformer (DiT) research on image generation has converged to a single evaluation setup: class-conditional generation on ImageNet. While methods improve the FID and related metrics, it is increasingly unclear whether they reflect real progress in generative modeling. The natural alternative, i.e., text-to-image (T2I) generation, is perceived as too costly or inconvenient to train and evaluate and is often skipped. We argue that this perception no longer holds. We introduce NanoGen, a unified DiT training and evaluation framework. NanoGen matches state-of-the-art DiT baselines on ImageNet and, with 12 lines of configuration change, also trains competitive text-to-image models. It currently supports RAE, VAE, pixel-space, and MeanFlow diffusion methods under both ImageNet and T2I setups. Under NanoGen, training T2I requires comparable compute to ImageNet. After training 21 latent diffusion models with NanoGen, we observe that method ranking shows no strong correlation between ImageNet and T2I generation: Pearson correlation is between -0.377 and -0.580 across three metrics. This suggests that a method which improves class-conditional ImageNet FID may show no corresponding improvement on T2I, clearly indicating the necessity of evaluating DiTs on both tasks. To this end, we summarize ImageNet and text-to-image results, which yields DiffusionBench, a holistic benchmark for DiT research. We recommend reporting DiffusionBench in place of ImageNet alone: methods that improve DiffusionBench are more likely to reflect broader progress. Research signal collected from arXiv metadata; Gemini enrichment can add a clearer summary. cs.CV benchmark eval evaluation
Replicate Official Models
Matched image generation, text-to-image, text to image; 3 source links; official model catalog signal; access model: Paid API
fal Model APIs
Matched image generation, text-to-image, text to image; 3 source links; official model catalog signal; access model: Paid API
| # | Alternative | Kind | Access | Fit | Why it appears | Source |
|---|---|---|---|---|---|---|
| 01 | Replicate Official Models | service | Paid API | RDR89 | Matched image generation, text-to-image, text to image; 3 source links; official model catalog signal; access model: Paid API | replicate.com |
| 02 | fal Model APIs | service | Paid API | RDR88 | Matched image generation, text-to-image, text to image; 3 source links; official model catalog signal; access model: Paid API | fal.ai |
| 03 | Runware Model API | service | Paid API | RDR87 | Matched image generation, text-to-image, text to image; 3 source links; official model catalog signal; access model: Paid API | runware.ai |
| 04 | artokun/comfyui-mcp | repo | Open source | RDR78 | Matched image generation, text-to-image, text to image; 1 source link; access model: Open source; freshly updated | github.com |
| 05 | Intermediate Text Representation Guided Text-to-Image Generation for Enhancing One-and-Only Alignment | paper | Research-only | RDR77 | Matched image generation, text-to-image, text to image; 1 source link; access model: Research-only; freshly updated | arxiv.org |
| 06 | aqm857886159/Nomi | repo | Open source | RDR77 | Matched image generation, text-to-image, text to image; 1 source link; access model: Open source; freshly updated | github.com |
| 07 | SeFi-Image: A Text-to-Image Foundation Model with Semantic-First Diffusion | paper | Research-only | RDR75 | Matched image generation, text-to-image, text to image; 1 source link; access model: Research-only | arxiv.org |