Qwen3.5 小尺寸模型开源：昇腾适配完成及部署指南

方案二：源码构建 如果您希望更灵活地控制环境，也可通过源码完整构建。前提是保证环境成功安装了 CANN 8.5.0。

从源码安装 vllm-ascend 后，需要将 vllm、vllm-ascend、transformers 升级至主分支特定版本以确保兼容性：

# 升级 vllm
git clone https://github.com/vllm-project/vllm.git
cd vllm
git checkout a75a5b54c7f76bc2e15d3025d6 git fetch origin pull/34521/head:pr-34521
git merge pr-34521
VLLM_TARGET_DEVICE=empty pip install -v .

# 升级 vllm-ascend
pip uninstall vllm-ascend -y
git clone https://github.com/vllm-project/vllm-ascend.git
cd vllm-ascend
git checkout c63b7a11888e9e1caeeff8
git fetch origin pull/6742/head:pr-6742
git merge pr-6742
pip install -v .

# 重新安装 transformers
git clone https://github.com/huggingface/transformers.git
cd transformers
git reset --hard fc9137225880a9d03f130634c20f9dbe36a7b8bf
pip install .

如需部署多节点环境，您需要在每个节点上分别完成环境配置。

服务启动与测试

单节点部署（A3 系列）

执行以下脚本进行在线推理。这里设置了一些关键的环境变量来优化 NPU 性能，比如 expandable_segments 和 HCCL_OP_EXPANSION_MODE。

export PYTORCH_NPU_ALLOC_CONF="expandable_segments:True"
export HCCL_OP_EXPANSION_MODE="AIV"
export HCCL_BUFFSIZE=1024
export OMP_NUM_THREADS=1
export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libjemalloc.so.2:$LD_PRELOAD
export TASK_QUEUE_ENABLE=1

vllm serve /root/.cache/modelscope/hub/models/vllm-ascend/Qwen3.5-2B/ \
    --served-model-name "qwen3.5" \
    --host 0.0.0.0 \
    --port 8010 \
    --data-parallel-size 1 \
    --tensor-parallel-size 4 \
    --max-model-len 5000 \
    --max-num-batched-tokens 16384 \
    --max-num-seqs 128 \
    --gpu-memory-utilization 0.94 \
    --trust-remote-code \
    --async-scheduling \
    --allowed-local-media-path / \
    --mm-processor-cache-gb 0 \
    --enforce-eager \
    --additional-config '{"enable_cpu_binding":true, "multistream_overlap_shared_expert": true}'

发送请求测试

服务启动后，可以使用 curl 向模型发送文本请求：

curl http://localhost:8000/v1/completions \
-H "Content-Type: application/json" \
-d '{
    "prompt": "The future of AI is",
    "path": "/path/to/model/Qwen3.5-2B/",
    "max_tokens": 100,
    "temperature": 0
}'

执行结束后，您可以看到模型生成了连贯的文本。此外，该模型也支持多模态输入，可以发送包含图片的请求：

curl http://localhost:8000/v1/completions \
-H "Content-Type: application/json" \
-d '{
    "model": "qwen3.5",
    "messages": [
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": [
            {"type": "image_url", "image_url": {"url": "https://modelscope.oss-cn-beijing.aliyuncs.com/resource/qwen.png"}},
            {"type": "text", "text": "What is the text in the illustrate?"}
        ]}
    ]
}'

基于 SGLang 的部署流程

环境准备

NPU 运行时环境所需的依赖已集成到 Docker 镜像中，用户可直接拉取该镜像进行部署。

# Atlas 800 A3
swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:main-cann8.5.0-a3
# Atlas 800 A2
swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:main-cann8.5.0-910b

# start container
docker run -itd --shm-size=16g --privileged=true --name ${NAME} \
--privileged=true --net=host \
-v /var/queue_schedule:/var/queue_schedule \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /usr/local/sbin:/usr/local/sbin \
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
-v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
--device=/dev/davinci0:/dev/davinci0 \
--device=/dev/davinci1:/dev/davinci1 \
--device=/dev/davinci2:/dev/davinci2 \
--device=/dev/davinci3:/dev/davinci3 \
--device=/dev/davinci4:/dev/davinci4 \
--device=/dev/davinci5:/dev/davinci5 \
--device=/dev/davinci6:/dev/davinci6 \
--device=/dev/davinci7:/dev/davinci7 \
--device=/dev/davinci8:/dev/davinci8 \
--device=/dev/davinci9:/dev/davinci9 \
--device=/dev/davinci10:/dev/davinci10 \
--device=/dev/davinci11:/dev/davinci11 \
--device=/dev/davinci12:/dev/davinci12 \
--device=/dev/davinci13:/dev/davinci13 \
--device=/dev/davinci14:/dev/davinci14 \
--device=/dev/davinci15:/dev/davinci15 \
--device=/dev/davinci_manager:/dev/davinci_manager \
--device=/dev/hisi_hdc:/dev/hisi_hdc \
--entrypoint=bash \
quay.io/ascend/sglang:${tag}

权重下载

请确保 Qwen3.5-2B 的模型权重已正确放置于指定路径。

服务启动与测试

单节点部署

执行以下脚本进行在线推理。这里调整了 CPU 亲和性和内存交换策略以提升性能。

# high performance cpu
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

# bind cpu
export SGLANG_SET_CPU_AFFINITY=1
unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING
source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export STREAMS_PER_DEVICE=32
export HCCL_BUFFSIZE=1000
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=lo
export GLOO_SOCKET_IFNAME=lo

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --attention-backend ascend \
    --device npu \
    --tp-size 1 \
    --chunked-prefill-size -1 --max-prefill-tokens 120000 \
    --disable-radix-cache \
    --trust-remote-code \
    --host 127.0.0.1 \
    --mem-fraction-static 0.8 \
    --port 8000 \
    --cuda-graph-bs 16 \
    --enable-multimodal \
    --mm-attention-backend ascend_attn

发送请求测试

curl --location http://127.0.0.1:8000/v1/chat/completions \
--header 'Content-Type: application/json' \
--data '{
    "model": "qwen3.5",
    "messages": [
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {"url": "/image_path/qwen.png"}
                },
                {"type": "text", "text": "What is the text in the illustrate?"}
            ]
        }
    ]
}'

结果返回中包含模型对图片内容的分析。当前版本仍在持续优化性能，后续会带来更好的体验。

注意：以上内容及代码仓中提到的数据集和模型仅作示例使用，仅供非商业用途学习与参考。如果您基于示例使用这些数据集和模型，请注意遵守对应的开源协议（License），避免产生相关纠纷。