Kubernetes 与 AI 集成：部署与管理实战指南

2. 训练工作负载部署

对于一次性训练任务，使用 Batch Job 比 Deployment 更合适，它能在任务完成后自动清理 Pod。注意设置 restartPolicy: Never 以避免不必要的重试，同时明确 GPU 配额。

apiVersion: batch/v1
kind: Job
metadata:
  name: ai-training-job
  namespace: default
spec:
  completions: 1
  parallelism: 1
  template:
    metadata:
      labels:
        app: ai-training
    spec:
      restartPolicy: Never
      containers:
        - name: training
          image: pytorch/pytorch:latest
          command: ["python", "/app/train.py"]
          resources:
            requests:
              cpu: "4"
              memory: "16Gi"
              nvidia.com/gpu: "1"
            limits:
              cpu: "8"
              memory: "32Gi"
              nvidia.com/gpu: "1"
          volumeMounts:
            - name: data
              mountPath: /data
            - name: code
              mountPath: /app
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: ai-data-pvc
        - name: code
          configMap:
            name: training-code

3. 推理服务部署

推理服务通常采用 Deployment 配合 Service 暴露端口。这里设置了副本数为 3 以应对流量波动，并通过 ClusterIP 实现内部通信。资源限制需根据实际 QPS 压测结果调整。

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ai-inference
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
      app: ai-inference
  template:
    metadata:
      labels:
        app: ai-inference
    spec:
      containers:
        - name: inference
          image: tensorflow/serving:latest
          ports:
            - containerPort: 8501
          resources:
            requests:
              cpu: "2"
              memory: "8Gi"
              nvidia.com/gpu: "1"
            limits:
              cpu: "4"
              memory: "16Gi"
              nvidia.com/gpu: "1"
          volumeMounts:
            - name: model
              mountPath: /models
      volumes:
        - name: model
          persistentVolumeClaim:
            claimName: model-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: ai-inference-service
  namespace: default
spec:
  selector:
    app: ai-inference
  ports:
    - port: 8501
      targetPort: 8501
  type: ClusterIP

4. 自动扩缩容配置

HPA 可以根据 CPU 和内存指标自动调整副本数。对于 AI 推理场景，建议结合自定义指标（如请求队列长度）进行更精准的扩缩容，此处展示基于资源利用率的基础配置。

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: ai-inference-hpa
  namespace: default
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: ai-inference
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 70

四、AI 工作负载优化

1. 数据处理优化

数据预处理往往涉及大量 I/O，StatefulSet 能保证网络标识的稳定性，适合分布式 Spark 任务。注意挂载持久化卷以加速数据读取。

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: data-processor
  namespace: default
spec:
  serviceName: data-processor
  replicas: 3
  selector:
    matchLabels:
      app: data-processor
  template:
    metadata:
      labels:
        app: data-processor
    spec:
      containers:
        - name: processor
          image: apache/spark:latest
          command: ["spark-submit", "--master", "k8s://https://kubernetes.default.svc:443", "--deploy-mode", "cluster", "/app/process.py"]
          resources:
            requests:
              cpu: "4"
              memory: "16Gi"
            limits:
              cpu: "8"
              memory: "32Gi"
          volumeMounts:
            - name: data
              mountPath: /data
            - name: code
              mountPath: /app
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: data-pvc
        - name: code
          configMap:
            name: processing-code

2. 模型管理

引入 ArgoCD 实现 GitOps 流程，将模型版本控制纳入 CI/CD 体系。自动化同步与自愈功能可减少人工干预风险。

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: model-management
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/susu/model-repo.git
    targetRevision: HEAD
    path: models
  destination:
    server: https://kubernetes.default.svc
    namespace: default
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

3. 监控与告警

Prometheus Operator 提供了标准化的监控方案。通过 ServiceMonitor 采集指标，并定义 PrometheusRule 触发 GPU 利用率或推理延迟告警。

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: ai-workload-metrics
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: ai-inference
  endpoints:
    - port: metrics
      interval: 15s
---
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: ai-workload-alerts
  namespace: monitoring
spec:
  groups:
    - name: ai-workload
      rules:
        - alert: GPUUtilizationHigh
          expr: nvidia_gpu_utilization > 80
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: GPU utilization high
            description: GPU utilization is above 80%
        - alert: ModelInferenceLatencyHigh
          expr: model_inference_latency_seconds > 0.5
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: Model inference latency high
            description: Model inference latency is above 500ms

五、常见问题

1. GPU 资源不足

若遇到显存瓶颈，可尝试配置 GPU 资源配额限制，利用时间分片技术共享 GPU，或在业务允许范围内开启自动扩缩容。

2. 数据处理瓶颈

分布式处理框架能缓解单机压力，同时优化存储层访问路径，必要时引入内存缓存机制提升吞吐。

3. 模型部署延迟

优化模型加载逻辑，预加载常用模型至本地缓存，或采用多模型服务架构减少切换开销。

六、最佳实践总结

1. 资源管理：精细化配置 GPU 和 CPU 配额，避免资源争抢。
2. 工作负载调度：依据训练、推理或数据处理特性选择 Job、Deployment 或 StatefulSet。
3. 数据管理：确保存储持久化且 I/O 性能满足需求。
4. 自动扩缩容：基于实时负载动态调整资源规模。
5. 监控告警：覆盖 GPU 状态与模型性能指标。
6. 模型管理：利用 GitOps 工具统一管控模型版本。

七、总结

Kubernetes 与 AI 集成是现代云原生架构的重要趋势。遵循上述最佳实践，能够构建出高效、可靠且易于维护的 AI 工作负载管理系统，为业务创新提供坚实底座。