$\textbf{Lumina-T2X}$ : Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformers
- [2024-08-06] 🎉🎉🎉 We have released Lumina-mGPT, the next-generation of generative models in our Lumina family! Lumina-mGPT is an autoregressive transformer capable of photorealistic image generation and other vision-language tasks, e.g., controllable generation, multi-turn dialog, depth/normal/segmentation map estimation.
- [2024-07-08] 🎉🎉🎉 Lumina-Next is now supported in the diffusers! Thanks to @yiyixuxu and @sayakpaul! HF Model Repo.
- [2024-06-26] We have released the inference code for img2img translation using
Lumina-Next-T2I
. CODE ComfyUI - [2024-06-21] 🥰🥰🥰 Lumina-Next has a jupyter nootbook for inference, thanks to canenduru! LINK
- [2024-06-21] We have uploaded the
Lumina-Next-SFT
andLumina-Next-T2I
to wisemodel.cn. wisemodel repo - [2024-06-19] We have released the
Lumina-T2Audio
(Text-to-Audio) code and model for music generation. MODEL - [2024-06-17] 🚀🚀🚀 We have support both inference and training (including Dreambooth) of SD3, implemented in our Lumina framework! CODE
- [2024-06-17] 🥰🥰🥰 Lumina-Next supports ComfyUI now, thanks to Kijai! LINK
- [2024-06-08] 🚀🚀🚀 We have released the
Lumina-Next-SFT
model, demonstrating better visual quality! MODEL - [2024-06-07] We have released the
Lumina-T2Music
(Text-to-Music) code and model for music generation. MODEL DEMO - [2024-06-03] We have released the
Compositional Generation
version ofLumina-Next-T2I
, which enables compositional generation with multiple captions for different regions. model. DEMO - [2024-05-29] We updated the new
Lumina-Next-T2I
Code and HF Model. Supporting 2K Resolution image generation and Time-aware Scaled RoPE. - [2024-05-25] We released training scripts for Flag-DiT and Next-DiT, and we have reported the comparison results between Next-DiT and Flag-DiT. Comparsion Results
- [2024-05-21] Lumina-Next-T2I supports a higher-order solver. It can generate images in just 10 steps without any distillation. Try our demos DEMO.
- [2024-05-18] We released training scripts for Lumina-T2I 5B. README
- [2024-05-16] ❗❗❗ We have converted the
.pth
weights to.safetensors
weights. Please pull the latest code and usedemo.py
for inference. - [2024-05-14] Lumina-Next now supports simple text-to-music generation (examples), high-resolution (1024*4096) Panorama generation conditioned on text (examples), and 3D point cloud generation conditioned on labels (examples).
- [2024-05-13] We give examples demonstrating Lumina-T2X's capability to support multilingual prompts, and even support prompts containing emojis.
- [2024-05-12] We excitedly released our
Lumina-Next-T2I
model (checkpoint) which uses a 2B Next-DiT model as the backbone and Gemma-2B as the text encoder. Try it out at demo1 & demo2 & demo3. Please refer to the paper Lumina-Next for more details. - [2024-05-10] We released the technical report on arXiv.
- [2024-05-09] We released
Lumina-T2A
(Text-to-Audio) Demos. Examples - [2024-04-29] We released the 5B model checkpoint and demo built upon it for text-to-image generation.
- [2024-04-25] Support 720P video generation with arbitrary aspect ratio. Examples
- [2024-04-19] Demo examples released.
- [2024-04-05] Code released for
Lumina-T2I
. - [2024-04-01] We release the initial version of
Lumina-T2I
for text-to-image generation.
Warning
Since we are updating the code frequently, please pull the latest code:
git pull origin main
We have supported Lumina-Next in the diffusers.
Note
You should install the development version of diffusers (main
branch) before diffusers releasing the new version.
pip install git+https://github.com/huggingface/diffusers
and you can try the code below:
from diffusers import LuminaText2ImgPipeline
import torch
pipeline = LuminaText2ImgPipeline.from_pretrained(
"/mnt/hdd1/xiejunlin/checkpoints/Lumina-Next-SFT-diffusers", torch_dtype=torch.bfloat16
).to("cuda")
image = pipeline(prompt="Upper body of a young woman in a Victorian-era outfit with brass goggles and leather straps. Background shows an industrial revolution ciyscape with smoky skies and tall, metal structures", height=1024, width=768).images[0]
For more details about training and inference of Lumina framework, please refer to Lumina-T2I, Lumina-Next-T2I, and Lumina-Next-T2I-Mini. We highly recommend you to use the Lumina-Next-T2I-Mini for training and inference, which is an extremely simplified version of Lumina-Next-T2I with full functionalities.
In order to quickly get you guys using our model, we built different versions of the GUI demo site.
Image Generation: [node1] [node2] [node3]
Image Compositional Generation: [node1]
Music Generation: [node1]
Using Lumina-T2X
as a library, using installation command on your environment:
pip install git+https://github.com/Alpha-VLLM/Lumina-T2X
If you want to contribute to the code, you should run command below to install pre-commit
library:
git clone https://github.com/Alpha-VLLM/Lumina-T2X
cd Lumina-T2X
pip install -e ".[dev]"
pre-commit install
pre-commit
- Lumina-Text2Image (Demos✅, Training✅, Inference✅, Checkpoints✅, Diffusers✅)
- Lumina-Text2Video (Demos✅)
- Lumina-Text2Music (Demos✅, Inference✅, Checkpoints✅)
- Lumina-Text2Audio (Demos✅, Inference✅, Checkpoints✅)
- $\textbf{Lumina-T2X}$: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformers
We introduce the
🌟 Features:
- Flow-based Large Diffusion Transformer (Flag-DiT): Lumina-T2X adopts the flow matching formulation and is equipped with many advanced techniques, such as RoPE, RMSNorm, and KQ-norm, demonstrating faster training convergence, stable training dynamics, and a simplified pipeline.
-
Any Modalities, Resolution, and Duration within One Framework:
-
$\textbf{Lumina-T2X}$ can encode any modality, including mages, videos, multi-views of 3D objects, and spectrograms into a unified 1-D token sequence at any resolution, aspect ratio, and temporal duration. - By introducing the
[nextline]
and[nextframe]
tokens, our model can support resolution extrapolation, i.e., generating images/videos with out-of-domain resolutions not encountered during training, such as images from 768x768 to 1792x1792 pixels.
-
-
Low Training Resources: Our empirical observations indicate that employing larger models,
high-resolution images, and longer-duration video clips can significantly accelerate the convergence
speed of diffusion transformers. Moreover, by employing meticulously curated text-image and text-video pairs featuring high aesthetic quality frames and detailed captions, our
$\textbf{Lumina-T2X}$ model is learned to generate high-resolution images and coherent videos with minimal computational demands. Remarkably, the default Lumina-T2I configuration, equipped with a 5B Flag-DiT and a 7B LLaMA as the text encoder, requires only 35% of the computational resources compared to Pixelart-$\alpha$ .
720P Videos:
Prompt: The majestic beauty of a waterfall cascading down a cliff into a serene lake.
video_720p_1.mp4
video_720p_2.mp4
Prompt: A stylish woman walks down a Tokyo street filled with warm glowing neon and animated city signage. She wears a black leather jacket, a long red dress, and black boots, and carries a black purse. She wears sunglasses and red lipstick. She walks confidently and casually. The street is damp and reflective, creating a mirror effect of the colorful lights. Many pedestrians walk about.
video_tokyo_woman.mp4
360P Videos:
video_360p.mp4
multi_view.mp4
Note
Attention: Mouse over the playbar and click the audio button on the playbar to unmute it.
Prompt: Semiautomatic gunfire occurs with slight echo
Generated Audio:
semiautomatic_gunfire_occurs_with_slight_echo.mp4
Groundtruth:
semiautomatic_gunfire_occurs_with_slight_echo_gt.mp4
Prompt: A telephone bell rings
Generated Audio:
a_telephone_bell_rings.mp4
Groundtruth:
a_telephone_bell_rings_gt.mp4
Prompt: An engine running followed by the engine revving and tires screeching
Generated Audio:
an_engine_running_followed_by_the_engine_revving_and_tires_screeching.mp4
Groundtruth:
an_engine_running_followed_by_the_engine_revving_and_tires_screeching_gt.mp4
Prompt: Birds chirping with insects buzzing and outdoor ambiance
Generated Audio:
birds_chirping_repeatedly.mp4
Groundtruth:
birds_chirping_repeatedly_gt.mp4
Note
Attention: Mouse over the playbar and click the audio button on the playbar to unmute it. For more details check out this
Prompt: An electrifying ska tune with prominent saxophone riffs, energetic e-guitar and acoustic drums, lively percussion, soulful keys, groovy e-bass, and a fast tempo that exudes uplifting energy.
Generated Music:
electrifying.ska.mp4
Prompt: A high-energy synth rock/pop song with fast-paced acoustic drums, a triumphant brass/string section, and a thrilling synth lead sound that creates an adventurous atmosphere.
Generated Music:
high_energy.song.mp4
Prompt: An uptempo electronic pop song that incorporates digital drums, digital bass and synthpad sounds.
Generated Music:
uptempo-electronic.mp4
Prompt: A medium-tempo digital keyboard song with a jazzy backing track featuring digital drums, piano, e-bass, trumpet, and acoustic guitar.
Generated Music:
medium-tempo.mp4
Prompt: This low-quality folk song features groovy wooden percussion, bass, piano, and flute melodies, as well as sustained strings and shimmering shakers that create a passionate, happy, and joyful atmosphere.
Generated Music:
low-quality-folk.mp4
We present three multilingual capabilities of Lumina-Next-2B.
Generating Images conditioned on Chinese poems:
Generating Images with multilingual prompts:
Generating Images with emojis:
We support diverse configurations, including text encoders, DiTs of different parameter sizes, inference methods, and VAE encoders.AAdditionally, we offer features such as 1D-RoPE, image enhancement, and more.
Core member for code developlement and maintence:
Dongyang Liu, Le Zhuo, Junlin Xie, Ruoyi Du, Peng Gao
@article{gao2024lumina-next,
title={Lumina-Next: Making Lumina-T2X Stronger and Faster with Next-DiT},
author={Zhuo, Le and Du, Ruoyi and Han, Xiao and Li, Yangguang and Liu, Dongyang and Huang, Rongjie and Liu, Wenze and others},
journal={arXiv preprint arXiv:2406.18583},
year={2024}
}
@article{gao2024lumin-t2x,
title={Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformers},
author={Gao, Peng and Zhuo, Le and Liu, Chris and and Du, Ruoyi and Luo, Xu and Qiu, Longtian and Zhang, Yuhang and others},
journal={arXiv preprint arXiv:2405.05945},
year={2024}
}