Kubernetes 与 AI 集成最佳实践

2. 训练工作负载部署

apiVersion: batch/v1
kind: Job
metadata:
  name: ai-training-job
  namespace: default
spec:
  completions: 1
  parallelism: 1
  template:
    metadata:
      labels:
        app: ai-training
    spec:
      restartPolicy: Never
      containers:
        - name: training
          image: pytorch/pytorch:latest
          command: ["python", "/app/train.py"]
          resources:
            requests:
              cpu: "4"
              memory: "16Gi"
              nvidia.com/gpu: "1"
            limits:
              cpu: "8"
              memory: "32Gi"
              nvidia.com/gpu: "1"
          volumeMounts:
            - name: data
              mountPath: /data
            - name: code
              mountPath: /app
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: ai-data-pvc
        - name: code
          configMap:
            name: training-code

3. 推理服务部署

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ai-inference
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
      app: ai-inference
  template:
    metadata:
      labels:
        app: ai-inference
    spec:
      containers:
        - name: inference
          image: tensorflow/serving:latest
          ports:
            - containerPort: 8501
          resources:
            requests:
              cpu: "2"
              memory: "8Gi"
              nvidia.com/gpu: "1"
            limits:
              cpu: "4"
              memory: "16Gi"
              nvidia.com/gpu: "1"
          volumeMounts:
            - name: model
              mountPath: /models
      volumes:
        - name: model
          persistentVolumeClaim:
            claimName: model-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: ai-inference-service
  namespace: default
spec:
  selector:
    app: ai-inference
  ports:
    - port: 8501
      targetPort: 8501
  type: ClusterIP

4. 自动扩缩容配置

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: ai-inference-hpa
  namespace: default
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: ai-inference
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 70

四、AI 工作负载优化

1. 数据处理优化

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: data-processor
  namespace: default
spec:
  serviceName: data-processor
  replicas: 3
  selector:
    matchLabels:
      app: data-processor
  template:
    metadata:
      labels:
        app: data-processor
    spec:
      containers:
        - name: processor
          image: apache/spark:latest
          command: ["spark-submit", "--master", "k8s://https://kubernetes.default.svc:443", "--deploy-mode", "cluster", "/app/process.py"]
          resources:
            requests:
              cpu: "4"
              memory: "16Gi"
            limits:
              cpu: "8"
              memory: "32Gi"
          volumeMounts:
            - name: data
              mountPath: /data
            - name: code
              mountPath: /app
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: data-pvc
        - name: code
          configMap:
            name: processing-code

2. 模型管理

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: model-management
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/susu/model-repo.git
    targetRevision: HEAD
    path: models
  destination:
    server: https://kubernetes.default.svc
    namespace: default
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

3. 监控与告警

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: ai-workload-metrics
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: ai-inference
  endpoints:
    - port: metrics
      interval: 15s
---
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: ai-workload-alerts
  namespace: monitoring
spec:
  groups:
    - name: ai-workload
      rules:
        - alert: GPUUtilizationHigh
          expr: nvidia_gpu_utilization > 80
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: GPU utilization high
            description: GPU utilization is above 80%
        - alert: ModelInferenceLatencyHigh
          expr: model_inference_latency_seconds > 0.5
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: Model inference latency high
            description: Model inference latency is above 500ms

五、常见问题

1. GPU 资源不足

解决方案：

1. 配置 GPU 资源配额
2. 使用时间分片共享 GPU
3. 考虑使用自动扩缩容

2. 数据处理瓶颈

解决方案：

1. 使用分布式数据处理
2. 优化数据存储和访问
3. 考虑使用内存缓存

3. 模型部署延迟

解决方案：

1. 优化模型加载时间
2. 使用模型缓存
3. 考虑使用多模型服务

六、最佳实践总结

1. 资源管理：合理配置 GPU 和 CPU 资源
2. 工作负载调度：根据工作负载类型选择合适的调度策略
3. 数据管理：优化数据存储和访问
4. 自动扩缩容：根据负载自动调整资源
5. 监控告警：配置 GPU 和模型性能监控
6. 模型管理：使用 GitOps 管理模型版本

七、总结

Kubernetes 与 AI 集成是现代云原生架构的重要趋势。按照本文的最佳实践，你可以构建一个高效、可靠的 AI 工作负载管理系统。