DCU BW1000 使用 llama.cpp 推理 Qwen3-Coder-30B 模型失败记录

安装 llama.cpp

下载 llama.cpp 源代码

git clone https://gitcode.com/GitHub_Trending/ll/llama.cpp

编译 llama.cpp

cd llama.cpp cmake -B build cmake --build build --config Release

加入路径

export PATH=/root/llama.cpp/build/bin:$PATH

或者也可以直接使用 make install

cd build make install 

但是安装好后报错

root@crdnotebook-2027598444851879937-denglf-12859:~/llama.cpp/build# llama-cli
llama-cli: error while loading shared libraries: libmtmd.so.0: cannot open shared object file: No such file or directory
root@crdnotebook-2027598444851879937-denglf-12859:~/llama.cpp/build# llama-gguf
llama-gguf: error while loading shared libraries: libggml-base.so.0: cannot open shared object file: No such file or directory

原来是没有把路径加入的缘故，加入路径后问题解决：

export PATH=/root/llama.cpp/build/bin:$PATH

模型下载

安装 modelscope

pip install modelscope

下载

from modelscope import snapshot_download
snapshot_download('tclf90/Qwen3-Coder-30B-A3B-Instruct-AWQ', cache_dir="models")

推理

用 llama-cli 推理

llama-cli -m models/tclf90/Qwen3-Coder-30B-A3B-Instruct-AWQ

报错：

root@crdnotebook-2027598444851879937-denglf-12859:~# llama-cli -m models/tclf90/Qwen3-Coder-30B-A3B-Instruct-AWQ
Loading model... |gguf_init_from_file_impl: failed to read magic
llama_model_load: error loading model: llama_model_loader: failed to load model from models/tclf90/Qwen3-Coder-30B-A3B-Instruct-AWQ
llama_model_load_from_file_impl: failed to load model
llama_params_fit: encountered an error while trying to fit params to free device memory: failed to load model
gguf_init_from_file_impl: failed to read magic
llama_model_load: error loading model: llama_model_loader: failed to load model from models/tclf90/Qwen3-Coder-30B-A3B-Instruct-AWQ
llama_model_load_from_file_impl: failed to load model
common_init_from_params: failed to load model 'models/tclf90/Qwen3-Coder-30B-A3B-Instruct-AWQ'
srv    load_model: failed to load model, 'models/tclf90/Qwen3-Coder-30B-A3B-Instruct-AWQ'

Failed to load the model

检查发现模型路径应为：stelterlab/Qwen3-Coder-30B-A3B-Instruct-AWQ。问题是该模型在指定仓库中不存在。

尝试用 transformers 推理

from transformers import AutoModelForCausalLM, AutoTokenizer
model_name = "/root/models/tclf90/Qwen3-Coder-30B-A3B-Instruct-AWQ"
# load the tokenizer and the model
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name, torch_dtype="auto", device_map="auto"
)
# prepare the model input
prompt = "Write a quick sort algorithm."
messages = [
    {"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True,
)
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)
# conduct text completion
generated_ids = model.generate(
    **model_inputs, max_new_tokens=65536
)
output_ids = generated_ids[0][len(model_inputs.input_ids[0]):].tolist()
content = tokenizer.decode(output_ids, skip_special_tokens=True)
print("content:", content)

也是失败

File /opt/conda/lib/python3.10/site-packages/transformers/quantizers/quantizer_awq.py:48, in AwqQuantizer.validate_environment(self, **kwargs)
46 def validate_environment(self, **kwargs):
47 if not is_gptqmodel_available():
---> 48 raise ImportError(
49     "Loading an AWQ quantized model requires gptqmodel. Please install it with `pip install gptqmodel`"
50 )
52 if not is_accelerate_available():
53 raise ImportError("Loading an AWQ quantized model requires accelerate (`pip install accelerate`)")
ImportError: Loading an AWQ quantized model requires gptqmodel. Please install it with `pip install gptqmodel`

总结

未调通，暂时搁置。

llama.cpp 是因为指定仓库没有那个模型，导致模型不匹配。 transformers 是因为库的问题，需要重新安装 torch 等库，导致需要的库无法安装上，推理失败。

调试

报错 ImportError: Loading an AWQ quantized model requires gptqmodel.

File /opt/conda/lib/python3.10/site-packages/transformers/quantizers/quantizer_awq.py:48, in AwqQuantizer.validate_environment(self, **kwargs) 46 def validate_environment(self, **kwargs): 47 if not is_gptqmodel_available(): ---> 48 raise ImportError( 49 "Loading an AWQ quantized model requires gptqmodel. Please install it with pip install gptqmodel" 50 ) 52 if not is_accelerate_available(): 53 raise ImportError("Loading an AWQ quantized model requires accelerate (pip install accelerate)") ImportError: Loading an AWQ quantized model requires gptqmodel. Please install it with pip install gptqmodel

安装提示执行

pip install gptqmodel

安装失败，

Exception: Unable to detect torch version via uv/pip/conda/importlib. Please install torch >= 2.7.1 [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed to build 'gptqmodel' when getting requirements to build wheel

用 conda 试试

conda install gptqmodel

也失败了。

PackagesNotFoundError: The following packages are not available from current channels:

• gptqmodel