NPU训练最佳实践 swift 华为昇腾npu

Ne0inhk

25 Dec 2024 — 7 min read

NPU训练最佳实践

原文：

作者: ,

环境准备

实验环境：8 * 昇腾910B3 64G (设备由提供, 感谢对modelscope和swift的支持～) # 创建新的conda虚拟环境(可选) conda create -n swift-npu python=3.10 -y conda activate swift-npu # 设置pip全局镜像 (可选,加速下载) pip config set global.index-url https://mirrors.aliyun.com/pypi/simple/ # 安装ms-swift(当前推荐从源码安装, 待发版后可直接pip安装) git clone https://github.com/modelscope/swift.git cd swift pip install -e '.[llm]'

Collecting oss2 (from ms-swift==2.3.0.dev0)
  Downloading https://mirrors.aliyun.com/pypi/packages/bf/af/b2d6dee157e56cfd4489173d3a2aebdf57fc70a8e9901edc140e516065ac/oss2-2.18.6.tar.gz (283 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 283.8/283.8 kB 269.4 kB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Collecting pandas (from ms-swift==2.3.0.dev0)
  Downloading https://mirrors.aliyun.com/pypi/packages/01/c6/d3d2612aea9b9f28e79a30b864835dad8f542dcf474eee09afeee5d15d75/pandas-2.2.2-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (15.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 15.6/15.6 MB 265.3 kB/s eta 0:00:00
Collecting peft<0.12.0,>=0.11.0 (from ms-swift==2.3.0.dev0)
  Downloading https://mirrors.aliyun.com/pypi/packages/19/99/c5e0292a6d2a62e95c3dfe674ce9e8f8a9fe5d4835d3c9bb9b3e016f02ae/peft-0.11.1-py3-none-any.whl (251 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 251.6/251.6 kB 279.8 kB/s eta 0:00:00
Collecting requests (from ms-swift==2.3.0.dev0)
  Downloading https://mirrors.aliyun.com/pypi/packages/f9/9b/335f9764261e915ed497fcdeb11df5dfd6f7bf257d4a6a2a686d80da4d54/requests-2.32.3-py3-none-any.whl (64 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 64.9/64.9 kB 318.9 kB/s eta 0:00:00
Collecting rouge (from ms-swift==2.3.0.dev0)
  Downloading https://mirrors.aliyun.com/pypi/packages/32/7c/650ae86f92460e9e8ef969cc5008b24798dcf56a9a8947d04c78f550b3f5/rouge-1.0.1-py3-none-any.whl (13 kB)
Collecting safetensors (from ms-swift==2.3.0.dev0)
  Downloading https://mirrors.aliyun.com/pypi/packages/e9/35/f73397a7745f954ef44d6b2416676c743a3d53447336b0da1d2b19c688ca/safetensors-0.4.3-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 278.3 kB/s eta 0:00:00
Collecting tensorboard (from ms-swift==2.3.0.dev0)
  Downloading https://mirrors.aliyun.com/pypi/packages/0a/32/2e8545fb0592f33e3aca5951e8b01008b76d61b440658cbdc37b4eaebf0b/tensorboard-2.17.0-py3-none-any.whl (5.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.5/5.5 MB 241.8 kB/s eta 0:00:00
Collecting tqdm (from ms-swift==2.3.0.dev0)
  Downloading https://mirrors.aliyun.com/pypi/packages/18/eb/fdb7eb9e48b7b02554e1664afd3bd3f117f6b6d6c5881438a0b055554f9b/tqdm-4.66.4-py3-none-any.whl (78 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 78.3/78.3 kB 264.9 kB/s eta 0:00:00
Collecting transformers<4.44,>=4.33 (from ms-swift==2.3.0.dev0)
  Downloading https://mirrors.aliyun.com/pypi/packages/ad/ff/b3e311e58b9c90b149fb957953b228287d7c9fe78df9a3a72e8715c5fc56/transformers-4.43.3-py3-none-any.whl (9.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.4/9.4 MB 268.5 kB/s eta 0:00:00
Collecting transformers_stream_generator (from ms-swift==2.3.0.dev0)
  Downloading https://mirrors.aliyun.com/pypi/packages/42/c2/65f13aec253100e1916e9bd7965fe17bde796ebabeb1265f45191ab4ddc0/transformers-stream-generator-0.0.5.tar.gz (13 kB)
  Preparing metadata (setup.py) ... done
Collecting trl>=0.9.6 (from ms-swift==2.3.0.dev0)
  Downloading https://mirrors.aliyun.com/pypi/packages/a5/c3/6565c2c376a829f99da20d39c2912405195ec1fa6aae068dc45c46793e72/trl-0.9.6-py3-none-any.whl (245 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 245.8/245.8 kB 248.2 kB/s eta 0:00:00
Collecting charset_normalizer (from ms-swift==2.3.0.dev0)
  Downloading https://mirrors.aliyun.com/pypi/packages/b8/60/e2f67915a51be59d4539ed189eb0a2b0d292bf79270410746becb32bc2c3/charset_normalizer-3.3.2-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (138 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 138.2/138.2 kB 285.4 kB/s eta 0:00:00
Collecting cpm_kernels (from ms-swift==2.3.0.dev0)
  Downloading https://mirrors.aliyun.com/pypi/packages/af/84/1831ce6ffa87b8fd4d9673c3595d0fc4e6631c0691eb43f406d3bf89b951/cpm_kernels-1.0.11-py3-none-any.whl (416 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 416.6/416.6 kB 238.0 kB/s eta 0:00:00
Collecting fastapi (from ms-swift==2.3.0.dev0)
  Downloading https://mirrors.aliyun.com/pypi/packages/a4/d4/eb78f7c2648a3585095623f207d7e4b85a1be30347e01e0fdcd1d7d167a9/fastapi-0.111.1-py3-none-any.whl (92 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 92.2/92.2 kB 211.5 kB/s eta 0:00:00
Collecting gradio>=3.40.0 (from ms-swift==2.3.0.dev0)
  Downloading https://mirrors.aliyun.com/pypi/packages/c5/49/ea7450fb03b926110181b758ff30a8426a6aaa60ae99f149ec3e7bacf04e/gradio-4.40.0-py3-none-any.whl (12.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.5/12.5 MB 241.4 kB/s eta 0:00:00
Collecting openai (from ms-swift==2.3.0.dev0)
  Downloading https://mirrors.aliyun.com/pypi/packages/43/18/8425b246cefad230b2574018bba0c039d70750b0a270eba1c9576a1fcab2/openai-1.37.1-py3-none-any.whl (337 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 337.0/337.0 kB 287.1 kB/s eta 0:00:00
Collecting sentencepiece (from ms-swift==2.3.0.dev0)
  Downloading https://mirrors.aliyun.com/pypi/packages/fd/46/316c1ba6c52b97de76aff7b9da678f7afbb52136afb2987c474d95630e65/sentencepiece-0.2.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 230.6 kB/s eta 0:00:00
Collecting tiktoken (from ms-swift==2.3.0.dev0)
  Downloading https://mirrors.aliyun.com/pypi/packages/b9/ab/f9c7675747f259d133d66065106cf732a7c2bef6043062fbca8e011f7f4d/tiktoken-0.7.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.0/1.0 MB 195.4 kB/s eta 0:00:00
Collecting uvicorn (from ms-swift==2.3.0.dev0)
  Downloading https://mirrors.aliyun.com/pypi/packages/3a/2f/97a928fb0e8cea9ddf5d5cb77e978f470a612d879e94aa90ea8fa6db8ef0/uvicorn-0.30.4-py3-none-any.whl (62 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.8/62.8 kB 276.0 kB/s eta 0:00:00
Collecting filelock (from datasets<2.19->ms-swift==2.3.0.dev0)
  Downloading https://mirrors.aliyun.com/pypi/packages/ae/f0/48285f0262fe47103a4a45972ed2f9b93e4c80b8fd609fa98da78b2a5706/filelock-3.15.4-py3-none-any.whl (16 kB)
Collecting pyarrow>=12.0.0 (from datasets<2.19->ms-swift==2.3.0.dev0)
  Downloading https://mirrors.aliyun.com/pypi/packages/81/3c/0580626896c842614a523e66b351181ed5bb14e5dfc263cd68cea2c46d90/pyarrow-17.0.0-cp310-cp310-manylinux_2_28_aarch64.whl (38.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 38.7/38.7 MB 256.4 kB/s eta 0:00:01

# 安装torch-npu pip install torch-npu decorator # 如果你想要使用deepspeed (控制显存占用,训练速度会有一定下降) pip install deepspeed # 环境对齐 (通常不需要运行. 如果你运行错误, 可以跑下面的代码, 仓库使用最新环境测试) pip install -r requirements/framework.txt -U pip install -r requirements/llm.txt -U

测试环境是否安装正确，NPU能否被正常加载： from transformers.utils import is_torch_npu_available import torch print(is_torch_npu_available()) # True print(torch.npu.device_count()) # 8 print(torch.randn(10, device='npu:0'))

查看NPU的P2P连接，这里看到每个NPU都通过7条HCCS与其他NPU互联 (valle) root@valle:~/src# npu-smi info -t topo NPU0 NPU1 NPU2 NPU3 NPU4 NPU5 NPU6 NPU7 CPU Affinity NPU0 X HCCS HCCS HCCS HCCS HCCS HCCS HCCS 144-167 NPU1 HCCS X HCCS HCCS HCCS HCCS HCCS HCCS 144-167 NPU2 HCCS HCCS X HCCS HCCS HCCS HCCS HCCS 96-119 NPU3 HCCS HCCS HCCS X HCCS HCCS HCCS HCCS 96-119 NPU4 HCCS HCCS HCCS HCCS X HCCS HCCS HCCS 0-23 NPU5 HCCS HCCS HCCS HCCS HCCS X HCCS HCCS 0-23 NPU6 HCCS HCCS HCCS HCCS HCCS HCCS X HCCS 48-71 NPU7 HCCS HCCS HCCS HCCS HCCS HCCS HCCS X 48-71 Legend: X = Self SYS = Path traversing PCIe and NUMA nodes. Nodes are connected through SMP, such as QPI, UPI. PHB = Path traversing PCIe and the PCIe host bridge of a CPU. PIX = Path traversing a single PCIe switch PXB = Path traversing multipul PCIe switches HCCS = Connection traversing HCCS. NA = Unknown relationship.

查看NPU状态, npu-smi命令详解可以查看 (valle) root@valle:~/src# npu-smi info +------------------------------------------------------------------------------------------------+ | npu-smi 24.1.rc1.b030 Version: 24.1.rc1.b030 | +---------------------------+---------------+----------------------------------------------------+ | NPU Name | Health | Power(W) Temp(C) Hugepages-Usage(page)| | Chip | Bus-Id | AICore(%) Memory-Usage(MB) HBM-Usage(MB) | +===========================+===============+====================================================+ | 0 910B3 | OK | 101.8 43 0 / 0 | | 0 | 0000:C1:00.0 | 0 0 / 0 3318 / 65536 | +===========================+===============+====================================================+ | 1 910B3 | OK | 92.0 39 0 / 0 | | 0 | 0000:C2:00.0 | 0 0 / 0 3314 / 65536 | +===========================+===============+====================================================+ | 2 910B3 | OK | 102.0 40 0 / 0 | | 0 | 0000:81:00.0 | 0 0 / 0 3314 / 65536 | +===========================+===============+====================================================+ | 3 910B3 | OK | 99.8 40 0 / 0 | | 0 | 0000:82:00.0 | 0 0 / 0 3314 / 65536 | +===========================+===============+====================================================+ | 4 910B3 | OK | 98.6 45 0 / 0 | | 0 | 0000:01:00.0 | 0 0 / 0 3314 / 65536 | +===========================+===============+====================================================+ | 5 910B3 | OK | 99.7 44 0 / 0 | | 0 | 0000:02:00.0 | 0 0 / 0 3314 / 65536 | +===========================+===============+====================================================+ | 6 910B3 | OK | 103.8 45 0 / 0 | | 0 | 0000:41:00.0 | 0 0 / 0 3314 / 65536 | +===========================+===============+====================================================+ | 7 910B3 | OK | 98.2 44 0 / 0 | | 0 | 0000:42:00.0 | 0 0 / 0 3315 / 65536 | +===========================+===============+====================================================+

微调

以下介绍LoRA的微调, 全参数微调设置参数--sft_type full即可.

模型大小	NPU数量	deepspeed类型	最大显存占用量
7B	1	None	1 * 28 GB
7B	4	None	4 * 22 GB
7B	4	zero2	4 * 28 GB
7B	4	zero3	4 * 22 GB
7B	8	None	8 * 22 GB
14B	1	None	1 * 45 GB
14B	8	None	8 * 51 GB
14B	8	zero2	8 * 49 GB
14B	8	zero3	8 * 31 GB

单卡训练

通过如下命令启动单卡微调: （注意: 如果微调期间出现nan的情况, 请设置--dtype fp32.） # 实验环境: 昇腾910B3 # 显存需求: 28 GB # 运行时长: 8小时 ASCEND_RT_VISIBLE_DEVICES=0 \ swift sft \ --model_type qwen1half-7b-chat \ --dataset blossom-math-zh \ --num_train_epochs 5 \ --sft_type lora \ --output_dir output \

数据并行训练

我们使用其中的4卡进行ddp训练 # 实验环境: 4 * 昇腾910B3 # 显存需求: 4 * 22 GB # 运行时长: 2小时 NPROC_PER_NODE=4 \ ASCEND_RT_VISIBLE_DEVICES=0,1,2,3 \ swift sft \ --model_type qwen1half-7b-chat \ --dataset blossom-math-zh \ --num_train_epochs 5 \ --sft_type lora \ --output_dir output \

Deepspeed训练

ZeRO2: # 实验环境: 4 * 昇腾910B3 # 显存需求: 4 * 28GB # 运行时长: 3.5小时 NPROC_PER_NODE=4 \ ASCEND_RT_VISIBLE_DEVICES=0,1,2,3 \ swift sft \ --model_type qwen1half-7b-chat \ --dataset blossom-math-zh \ --num_train_epochs 5 \ --sft_type lora \ --output_dir output \ --deepspeed default-zero2 \

ZeRO3: # 实验环境: 4 * 昇腾910B3 # 显存需求: 4 * 22 GB # 运行时长: 8.5小时 NPROC_PER_NODE=4 \ ASCEND_RT_VISIBLE_DEVICES=0,1,2,3 \ swift sft \ --model_type qwen1half-7b-chat \ --dataset blossom-math-zh \ --num_train_epochs 5 \ --sft_type lora \ --output_dir output \ --deepspeed default-zero3 \

推理

原始模型: ASCEND_RT_VISIBLE_DEVICES=0 swift infer --model_type qwen1half-7b-chat

LoRA微调后: ASCEND_RT_VISIBLE_DEVICES=0 swift infer --ckpt_dir xxx/checkpoint-xxx --load_dataset_config true # merge-lora并推理 ASCEND_RT_VISIBLE_DEVICES=0 swift export --ckpt_dir xx/checkpoint-xxx --merge_lora true ASCEND_RT_VISIBLE_DEVICES=0 swift infer --ckpt_dir xxx/checkpoint-xxx-merged --load_dataset_config true

部署

NPU不支持使用vllm进行推理/部署加速, 但是可以使用原生pytorch进行部署.

原始模型: ASCEND_RT_VISIBLE_DEVICES=0 swift deploy --model_type qwen1half-7b-chat

LoRA微调后: ASCEND_RT_VISIBLE_DEVICES=0 swift deploy --ckpt_dir xxx/checkpoint-xxx --load_dataset_config true # merge-lora并推理 ASCEND_RT_VISIBLE_DEVICES=0 swift export --ckpt_dir xx/checkpoint-xxx --merge_lora true ASCEND_RT_VISIBLE_DEVICES=0 swift deploy --ckpt_dir xxx/checkpoint-xxx-merged --load_dataset_config true