模型简介
Qwen3-VL 是由阿里云 Qwen 团队开发的多模态大语言模型系列。其空间感知能力大幅提升，2D grounding 从绝对坐标变为相对坐标，支持判断物体方位、视角变化、遮挡关系，能实现 3D grounding。OCR 支持更多语言及复杂场景，覆盖范围扩展至 32 种语言。
主要技术改进包括：
- 采用 MRoPE-Interleave，原始 MRoPE 将特征维度按照时间（t）、高度（h）和宽度（w）的顺序分块划分，Qwen3-VL 采取 t,h,w 交错分布的形式，实现对时间、高度和宽度的全频率覆盖，提升对长视频的理解能力。
- 引入 DeepStack 技术，融合 ViT 多层次特征，提升视觉细节捕捉能力和图文对齐精度。将以往多模态大模型单层输入视觉 tokens 的范式，改为在大型语言模型 (LLM) 的多层中进行注入。
- 将原有的视频时序建模机制 T-RoPE 升级为文本时间戳对齐机制，实现帧级别的时间信息与视觉内容的细粒度对齐。
环境配置
conda create -n Qwen3-vl python=3.10
conda activate Qwen3-vl
pip install accelerate
pip install qwen-vl-utils==0.0.14
uv pip install -U vllm>=0.11.0
下载代码
git clone https://github.com/QwenLM/Qwen3-VL
下载权重文件
pip install modelscope
modelscope download --model Qwen/Qwen3-VL-2B-Instruct
推理代码
修改模型路径和图片地址后运行以下脚本：
from transformers import Qwen3VLForConditionalGeneration, AutoProcessor
import torch
from PIL import Image
def load_qwen3_vl_4b_model():
model = Qwen3VLForConditionalGeneration.from_pretrained(
"Qwen/Qwen3-VL-4B-Instruct",
torch_dtype=torch.bfloat16,
device_map="auto",
attn_implementation="flash_attention_2"
)
processor = AutoProcessor.from_pretrained("Qwen/Qwen3-VL-4B-Instruct")
return model, processor
def process_multimodal_query(model, processor, image_path, text_query):
image = Image.open(image_path).convert('RGB')
messages = [
{
"role": "user",
"content": [
{"type": "image", "image": image},
{"type": "text", "text": text_query}
]
}
]
inputs = processor.apply_chat_template(
messages, tokenize=True, add_generation_prompt=True, return_dict=True, return_tensors="pt"
)
generated_ids = model.generate(**inputs, max_new_tokens=128, do_sample=True, temperature=0.7, top_p=0.8)
generated_ids_trimmed = [out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)]
output_text = processor.batch_decode(generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False)
return output_text[0] if output_text else ""
if __name__ == "__main__":
model, processor = load_qwen3_vl_4b_model()
image_path = "example.jpg"
query = "描述这张图片中的场景和主要对象"
result = process_multimodal_query(model, processor, image_path, query)
print("模型回复：", result)
微调部分
5.1 使用 LLaMA-Factory 项目进行微调
5.1.1 下载项目
git clone https://github.com/hiyouga/LLaMA-Factory
5.1.2 创建虚拟环境
conda create -n llama-factory python=3.12
conda activate llama-factory
pip install -e ".[torch,metrics]" --no-build-isolation
pip uninstall torch torchvision
pip install torch==2.8.0 torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
5.2 准备微调数据集
5.2.1 所需数据集格式
LLaMA-Factory 微调需要特定的数据格式。注意 Qwen3-VL 使用 0-1000 的相对坐标，需进行归一化。准备好数据集后，需修改 LLaMA-Factory/data/dataset_info.json 文件增加所需文件夹配置。
5.2.2 YOLO 格式转换为 qwen3-vl-grounding 格式
转换代码如下：
import os
import json
from tqdm import tqdm
IMAGE_DIR = "images"
LABEL_DIR = "labels"
OUTPUT_JSON = "qwen3_vl_grounding_mllm.json"
CLASS_ID2NAME = {0: "house"}
USER_PROMPT = (
"<image>\n"
"Locate all objects in this image and output the bbox coordinates "
"in JSON format using relative coordinates in the range [0, 1000]."
)
def yolo_to_xyxy_relative(xc, yc, w, h):
x_min = xc - w / 2
y_min = yc - h / 2
x_max = xc + w / 2
y_max = yc + h / 2
x_min = max(0.0, min(1.0, x_min))
y_min = max(0.0, min(1.0, y_min))
x_max = max(0.0, min(1.0, x_max))
y_max = max(0.0, min(1.0, y_max))
return [x_min, y_min, x_max, y_max]
def scale_to_qwen_coords(xyxy_rel, scale=1000):
x_min, y_min, x_max, y_max = xyxy_rel
return [
int(round(x_min * scale)),
int(round(y_min * scale)),
int(round(x_max * scale)),
int(round(y_max * scale)),
]
def collect_image_files(image_dir):
exts = {".jpg", ".jpeg", ".png", ".bmp", ".webp"}
files = []
for fname in os.listdir(image_dir):
if os.path.splitext(fname)[1].lower() in exts:
files.append(fname)
return sorted(files)
def main():
image_files = collect_image_files(IMAGE_DIR)
if not image_files:
print(f"No images found in {IMAGE_DIR}")
return
dataset = []
for img_name in tqdm(image_files, desc="Converting"):
img_path = os.path.join(IMAGE_DIR, img_name)
img_rel_or_abs = os.path.abspath(img_path)
base, _ = os.path.splitext(img_name)
label_path = os.path.join(LABEL_DIR, base + ".txt")
if not os.path.exists(label_path):
continue
bboxes_qwen = []
cls_ids = []
with open(label_path, "r", encoding="utf-8") as f:
for line in f:
line = line.strip()
if not line:
continue
parts = line.split()
if len(parts) < 5:
continue
cls_id = int(parts[0])
xc = float(parts[1])
yc = float(parts[2])
w = float(parts[3])
h = float(parts[4])
xyxy_rel = yolo_to_xyxy_relative(xc, yc, w, h)
xyxy_qwen = scale_to_qwen_coords(xyxy_rel, scale=1000)
bboxes_qwen.append(xyxy_qwen)
cls_ids.append(cls_id)
if not bboxes_qwen:
continue
objects = []
for cid, box in zip(cls_ids, bboxes_qwen):
obj = {"cls_id": cid, "bbox_2d": box}
if cid in CLASS_ID2NAME:
obj["cls_name"] = CLASS_ID2NAME[cid]
objects.append(obj)
answer_obj = {"objects": objects}
answer_str = json.dumps(answer_obj, ensure_ascii=False)
sample = {
"conversations": [
{"from": "human", "value": USER_PROMPT},
{"from": "gpt", "value": answer_str}
],
"images": [img_rel_or_abs]
}
dataset.append(sample)
with open(OUTPUT_JSON, "w", encoding="utf-8") as f:
json.dump(dataset, f, ensure_ascii=False, indent=2)
print(f"Done. Wrote {len(dataset)} samples to {OUTPUT_JSON}")
if __name__ == "__main__":
main()
将生成的 json 文件放在 LLaMA-Factory/data 路径下面。
5.3 使用 LLaMA-Factory 可视化界面进行微调
cd LLaMA-Factory
5.3.1 启动可视化界面
llamafactory-cli webui
5.3.2 修改训练参数
关键参数包括语言、模型、模型路径、模型下载源、计算类型等。其中计算类型为 Pure_bf16 更省显存。
修改 dataset_info.json，在文件最前面添加如下配置：
"qwen3_vl_grounding_mllm": {
"file_name": "qwen3_vl_grounding_mllm.json",
"formatting": "sharegpt",
"columns": {
"messages": "conversations",
"images": "images"
}
}
保存训练参数、载入训练参数、开始训练。
5.3.3 对话测试模型
点击 chat、选择训练好模型路径、点击加载模型、进行问答。输入图片和提示词进行测试。
