使用自定义数据集微调 Stable Diffusion 3.5 LoRA 文生图模型
环境要求
Package Version ------------------------ ------------
absl-py 2.3.1 accelerate 1.11.0 aiohappyeyeballs 2.6.1 aiohttp 3.13.1 aiosignal 1.4.0 annotated-types 0.7.0 anyio 4.11.0 attrs 25.4.0 bitsandbytes 0.48.1 certifi 2025.10.5 charset-normalizer 3.4.4 click 8.3.0 datasets 4.2.0 diffusers 0.35.2 dill 0.4.0 filelock 3.20.0 frozenlist 1.8.0 fsspec 2025.9.0 gitdb 4.0.12 GitPython 3.1.45 grpcio 1.75.1 h11 0.16.0 hf-xet 1.1.10 httpcore 1.0.9 httpx 0.28.1 huggingface-hub 0.35.3 idna 3.11 importlib_metadata 8.7.0 Jinja2 3.1.6 Markdown 3.9 MarkupSafe 3.0.3 modelscope 1.31.0 mpmath 1.3.0 multidict 6.7.0 multiprocess 0.70.16 networkx 3.5 numpy 2.3.4 nvidia-cublas-cu11 11.11.3.6 nvidia-cuda-cupti-cu11 11.8.87 nvidia-cuda-nvrtc-cu11 11.8.89 nvidia-cuda-runtime-cu11 11.8.89 nvidia-cudnn-cu11 9.1.0.70 nvidia-cufft-cu11 10.9.0.58 nvidia-curand-cu11 10.3.0.86 nvidia-cusolver-cu11 11.4.1.48 nvidia-cusparse-cu11 11.7.5.86 nvidia-nccl-cu11 2.20.5 nvidia-nvtx-cu11 11.8.86 packaging 25.0 pandas 2.3.3 peft 0.17.1 Pillow 9.5.0 pip 24.0 platformdirs 4.5.0 propcache 0.4.1 protobuf 6.33.0 psutil 7.1.1 pyarrow 21.0.0 pydantic 2.12.3 pydantic_core 2.41.4 python-dateutil 2.9.0.post0 pytz 2025.2 PyYAML 6.0.3 regex 2025.10.22 requests 2.32.5 safetensors 0.6.2 sentencepiece 0.2.1 sentry-sdk 2.42.1 setuptools 65.5.0 six 1.17.0 smmap 5.0.2 sniffio 1.3.1 sympy 1.14.0 tensorboard 2.20.0 tensorboard-data-server 0.7.2 tokenizers 0.22.1 torch 2.4.1+cu118 torchaudio 2.4.1+cu118 torchvision 0.19.1+cu118 tqdm 4.67.1 transformers 4.57.1 triton 3.0.0 typing_extensions 4.15.0 typing-inspection 0.4.2 tzdata 2025.2 urllib3 2.5.0 wandb 0.22.2 Werkzeug 3.1.3 xxhash 3.6.0 yarl 1.22.0 zipp 3.23.0
相关介绍
Stable Diffusion 3.5 是 Stability AI 于 2024 年 10 月 22 日正式发布的最新图像生成模型，是 Stable Diffusion 3.0 系列的升级版本。主要版本包括 Stable Diffusion 3.5 Large（80 亿参数）、Stable Diffusion 3.5 Large Turbo（速度提升）以及 Stable Diffusion 3.5 Medium（适合消费级硬件）。技术核心采用 MM-DiT（Modified Multimodal Diffusion Transformer），集成 Query-Key Normalization，简化微调过程。
优势包括图像质量提升、提示词遵循能力领先、多样化输出及商业友好（年收入不超过 100 万美元可免费商用）。局限性在于结构稳定性与 Flux 模型相比略逊一筹，且对硬件有一定需求（Large 版本需 10G 以上显卡）。
微调 Stable Diffusion 3.5 LoRA 文生图模型
下载项目
官方源代码：https://github.com/seochan99/stable-diffusion-3.5-text2image-lora.git
Linux 环境下克隆仓库：
git clone https://github.com/seochan99/stable-diffusion-3.5-text2image-lora.git
准备数据集
准备一个用于模型微调的数据集，目录结构如下：
dataset/
├── images/
│ ├── landscape_001.jpg
│ ├── portrait_001.jpg
│ └── ...
└── metadata.jsonl
metadata.jsonl 内容示例：
{"image":"images/landscape_001.jpg","caption":"a breathtaking mountain landscape with snow-capped peaks, a crystal clear lake reflecting the sky, and pine trees in the foreground"}
{"image":"images/portrait_001.jpg","caption":"a professional portrait of a young woman with curly brown hair, wearing a blue denim jacket, smiling softly, natural lighting"}
下载预训练模型
Hugging Face
import os
os.environ['HF_ENDPOINT']='https://hf-mirror.com'
from huggingface_hub import snapshot_download
snapshot_download(repo_id="stabilityai/stable-diffusion-3.5-large", local_dir="stabilityai/stable-diffusion-3.5-large")
Model Scope
from modelscope import snapshot_download
model_dir = snapshot_download('AI-ModelScope/stable-diffusion-3.5-medium', local_dir="stabilityai/stable-diffusion-3.5-medium")
进行训练
python stable-diffusion-3.5-text2image-lora-main/train_text_to_image_lora_sd35.py \
--pretrained_model_name_or_path stabilityai/stable-diffusion-3.5-medium \
--train_data_dir sd3_test/stable-diffusion-3.5-text2image-lora-main/examples/dataset/ \
--output_dir sd3_test/stable-diffusion-3.5-text2image-lora-main/outputs/sd35-lora \
--resolution 1024 --train_batch_size 1 --gradient_accumulation_steps 4 \
--num_train_epochs 50 --rank 64 --learning_rate 0.0001 \
--text_encoder_lr 1e-05 --lr_scheduler cosine --lr_warmup_steps 500 \
--lr_num_cycles 1 --mixed_precision fp16 --validation_epochs 2 \
--num_validation_images 2 --checkpointing_steps 500 --report_to tensorboard \
--dataloader_num_workers 2 --seed 42 --precondition_outputs 1 \
--random_flip --gradient_checkpointing --train_text_encoder
进行预测
python stable-diffusion-3.5-text2image-lora-main/inference.py \
--model_path stabilityai/stable-diffusion-3.5-medium \
--lora_path stable-diffusion-3.5-text2image-lora-main/outputs/sd35-lora/pytorch_lora_weights.safetensors \
--prompt "a professional portrait of a young woman with curly brown hair, wearing a blue denim jacket, smiling softly, natural lighting" \
--num_images 4 --seed 42
参考
[1] Patrick Esser et al., Scaling Rectified Flow Transformers for High-Resolution Image Synthesis. 2024. [2] https://huggingface.co/stabilityai/stable-diffusion-3.5-large [3] https://huggingface.co/stabilityai/stable-diffusion-3.5-medium