Retinaface+CurricularFace 人脸识别服务 Kubernetes StatefulSet 部署实战

该镜像包含预训练模型、优化推理脚本及完整 Python 环境，开箱即用。

本地测试验证

建议在集群部署前先在本地验证镜像功能，提前发现环境问题。

# 1. 拉取镜像
docker pull your-registry/retinaface-curricularface:latest

# 2. 运行容器并进入
docker run -it --gpus all your-registry/retinaface-curricularface:latest bash

# 3. 激活环境并测试
cd /root/Retinaface_CurricularFace
conda activate torch25
python inference_face.py

正常输出应显示人脸特征提取结果，例如：

检测到人脸，开始特征提取...
图片 1 特征维度：512
图片 2 特征维度：512
相似度得分：0.856
判定结果：同一人

此步骤可验证 GPU 驱动、CUDA 环境、模型完整性及推理脚本运行情况。

Kubernetes StatefulSet 部署配置

创建命名空间和存储

首先创建独立命名空间：

# 1-namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: face-recognition
  labels:
    name: face-recognition

接着配置存储类与 PVC，用于保存模型文件、日志及临时文件：

# 2-storageclass.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: kubernetes.io/gce-pd
parameters:
  type: pd-ssd
  replication-type: none
---
# 3-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: face-model-pvc
  namespace: face-recognition
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: fast-ssd
  resources:
    requests:
      storage: 10Gi

StatefulSet 核心配置

这是最关键的部分，重点在于 serviceName、GPU 资源申请及探针配置：

# 4-statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: face-recognition
  namespace: face-recognition
  labels:
    app: face-recognition
spec:
  serviceName: "face-recognition-service"
  replicas: 3
  selector:
    matchLabels:
      app: face-recognition
  template:
    metadata:
      labels:
        app: face-recognition
    spec:
      nodeSelector:
        accelerator: nvidia-tesla-t4
      containers:
        - name: face-recognition-container
          image: your-registry/retinaface-curricularface:latest
          imagePullPolicy: Always
          resources:
            limits:
              nvidia.com/gpu: 1
              memory: "8Gi"
              cpu: "4"
            requests:
              nvidia.com/gpu: 1
              memory: "4Gi"
              cpu: "2"
          env:
            - name: MODEL_DIR
              value: "/models"
            - name: LOG_LEVEL
              value: "INFO"
          ports:
            - containerPort: 8000
              name: http
              protocol: TCP
          command: ["/bin/bash", "-c"]
          args:
            - |
              source /root/miniconda3/bin/activate torch25
              if [ ! -f /data/models/retinaface.pth ]; then
                echo "复制模型文件到持久化存储..."
                cp -r /root/Retinaface_CurricularFace/models/* /data/models/
              fi
              cd /root/Retinaface_CurricularFace
              python http_server.py --port 8000 --model-dir /data/models
          livenessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 60
            periodSeconds: 30
            timeoutSeconds: 10
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: /ready
              port: 8000
            initialDelaySeconds: 90
            periodSeconds: 20
            timeoutSeconds: 5
          volumeMounts:
            - name: model-storage
              mountPath: /data/models
              readOnly: false
            - name: log-storage
              mountPath: /var/log/face-recognition
              readOnly: false
      volumeClaimTemplates:
        - metadata:
            name: model-storage
          spec:
            accessModes: ["ReadWriteOnce"]
            storageClassName: "fast-ssd"
            resources:
              requests:
                storage: 10Gi
        - metadata:
            name: log-storage
          spec:
            accessModes: ["ReadWriteOnce"]
            storageClassName: "fast-ssd"
            resources:
              requests:
                storage: 5Gi

注意 serviceName 是 StatefulSet 的硬性要求，用于生成稳定 DNS；volumeClaimTemplates 确保每个 Pod 拥有独立存储。

服务暴露配置

使用 Headless Service 配合 Ingress 暴露服务：

# 5-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: face-recognition-service
  namespace: face-recognition
spec:
  clusterIP: None
  selector:
    app: face-recognition
  ports:
    - port: 8000
      targetPort: 8000
      name: http
---
# 6-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: face-recognition-ingress
  namespace: face-recognition
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/proxy-body-size: "10m"
spec:
  ingressClassName: nginx
  rules:
    - host: face-recognition.your-domain.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: face-recognition-service
                port:
                  number: 8000

部署与验证

执行部署

按顺序应用配置文件：

kubectl apply -f 1-namespace.yaml
kubectl apply -f 2-storageclass.yaml
kubectl apply -f 3-pvc.yaml
kubectl apply -f 4-statefulset.yaml
kubectl apply -f 5-service.yaml
kubectl apply -f 6-ingress.yaml

验证部署状态

检查各资源状态是否正常：

kubectl get statefulset -n face-recognition
kubectl get pods -n face-recognition -l app=face-recognition
kubectl get pvc -n face-recognition
kubectl get svc -n face-recognition
kubectl logs -f face-recognition-0 -n face-recognition

预期 Pod 状态为 Running，且 READY 列显示 1/1。

功能测试

通过端口转发或进入 Pod 内部测试服务：

# 端口转发测试
kubectl port-forward face-recognition-0 8000:8000 -n face-recognition
curl http://localhost:8000/health

# Pod 内部测试
kubectl exec -it face-recognition-0 -n face-recognition -- bash
cd /root/Retinaface_CurricularFace
conda activate torch25
python inference_face.py

生产环境优化建议

性能优化配置

添加预热钩子与安全上下文以提升性能和安全性：

containers:
  - name: face-recognition-container
    lifecycle:
      postStart:
        exec:
          command: ["/bin/bash", "-c", "python /root/Retinaface_CurricularFace/warmup.py"]
    securityContext:
      capabilities:
        add: ["IPC_LOCK"]
      privileged: false
      readOnlyRootFilesystem: true
    resources:
      limits:
        nvidia.com/gpu: 1
        memory: "12Gi"
        cpu: "6"
        ephemeral-storage: "20Gi"
      requests:
        nvidia.com/gpu: 1
        memory: "8Gi"
        cpu: "4"
        ephemeral-storage: "10Gi"

监控与日志收集

集成 Prometheus 指标收集：

# 7-monitoring.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: face-recognition-metrics
  namespace: face-recognition
data:
  prometheus-scrape.yaml: |
    - job_name: 'face-recognition'
      scrape_interval: 30s
      static_configs:
        - targets:
            - 'face-recognition-0.face-recognition-service.face-recognition.svc.cluster.local:8000'
            - 'face-recognition-1.face-recognition-service.face-recognition.svc.cluster.local:8000'
            - 'face-recognition-2.face-recognition-service.face-recognition.svc.cluster.local:8000'
      metrics_path: '/metrics'

在推理脚本中增加指标记录逻辑，包括请求计数、延迟及相似度分布。

自动扩缩容配置

根据负载自动调整副本数：

# 8-hpa.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: face-recognition-hpa
  namespace: face-recognition
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: StatefulSet
    name: face-recognition
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 80
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Percent
          value: 10
          periodSeconds: 60

常见问题与解决方案

GPU 相关问题

Pod 无法调度，提示 No GPU available

1. 检查节点标签：kubectl get nodes --show-labels | grep gpu
2. 给节点打标签：kubectl label nodes <node-name> accelerator=nvidia-tesla-t4
3. 安装 NVIDIA 设备插件

CUDA 版本不匹配

1. 检查节点 CUDA 版本：nvidia-smi
2. 确保镜像 CUDA 版本与节点一致

存储相关问题

PVC 一直处于 Pending 状态

1. 检查 StorageClass 配置
2. 查看 PVC 事件：kubectl describe pvc <pvc-name>
3. 检查存储配额

服务发现问题

Pod 之间无法通过 DNS 名称访问

StatefulSet 的 Pod 拥有固定 DNS 名称格式：<pod-name>.<service-name>.<namespace>.svc.cluster.local。可通过 nslookup 验证解析情况。

性能优化问题

第一次推理延迟很高

原因通常是模型加载耗时。可通过 postStart 钩子实现模型预热，或调整就绪探针初始延迟，确保模型完全加载后再接收流量。

总结与最佳实践

本方案完成了 Retinaface+CurricularFace 人脸识别模型在 Kubernetes 上的 StatefulSet 部署。核心优势在于利用 StatefulSet 提供稳定的网络标识和持久化存储，结合 HPA 实现弹性伸缩，并通过完善的监控保障高可用性。

对于小规模部署，可从 2 个副本起步，关注核心指标；中大规模部署建议引入节点亲和性及多级缓存；超大规模场景可考虑模型服务网格及跨区域容灾。技术部署需随业务增长持续优化，本方案可作为构建稳定人脸识别服务的坚实基础。