Prometheus Pushgateway 安装配置与临时任务监控实践

4. 验证服务：

ps aux | grep pushgateway
curl http://localhost:9091/metrics
journalctl -u pushgateway -f

5. （可选）配置防火墙放行 9091 端口：

sudo firewall-cmd --permanent --add-port=9091/tcp
sudo firewall-cmd --reload

此时访问 http://<ip>:9091 即可看到 Pushgateway Web UI。访问 http://<ip>:9091/metrics 可查看其自身暴露的 Go 运行时及进程指标。

2.2 Docker 安装

使用官方镜像快速部署：

docker pull prom/pushgateway
docker run -d --name=pg -p 9091:9091 prom/pushgateway

启动后访问 http://<ip>:9091 即可。

3. Prometheus 配置 Pushgateway

编辑 Prometheus 配置文件 prometheus.yml，添加对应的 scrape job：

scrape_configs:
  - job_name: 'pushgateway'
    honor_labels: true
    static_configs:
      - targets: ['localhost:9091']
        labels:
          app: "pushgateway"

注意：必须配置 honor_labels: true，否则 Pushgateway 推送的 job 和 instance 标签会被 Prometheus 覆盖。

重启 Prometheus 服务：

systemctl restart prometheus
systemctl status prometheus

访问 Prometheus UI (http://<ip>:9090) 的 Targets 页面，确认 Pushgateway 状态为 UP。

4. 推送数据到 Pushgateway

Pushgateway 提供标准 HTTP API 用于接收指标。默认推送 URL 格式为： http://<pushgateway-addr>:9091/metrics/job/{JOB_NAME}{/LABEL_NAME/LABEL_VALUE...} 其中 job 为必填项，后续可追加任意标签对（如 instance/<INSTANCE_NAME>）以区分数据来源。

4.1 推送简单指标

echo "test_metric 123456" | curl --data-binary @- http://192.168.42.140:9091/metrics/job/test_job

推送成功后，Pushgateway 会自动生成 push_time_seconds 和 push_failure_time_seconds 指标。在 Prometheus UI 中即可查询到 test_metric。

4.2 推送复杂指标（带类型与帮助信息）

cat <<EOF | curl --data-binary @- http://192.168.42.140:9091/metrics/job/some_job/instance/some_instance
# TYPE some_metric counter
some_metric{label="val1"} 42
# TYPE another_metric gauge
# HELP another_metric Just an example.
another_metric 2398.283
EOF

4.3 删除指标

• 删除指定 job 和 instance 的数据：

  curl -X DELETE http://192.168.42.140:9091/metrics/job/some_job/instance/some_instance
  

• 删除指定 job 下的所有数据：

  curl -X DELETE http://192.168.42.140:9091/metrics/job/some_job
  

5. 真实场景应用示例

5.1 Shell 脚本监控备份任务

模拟备份任务并推送耗时与状态指标：

#!/bin/bash
JOB_NAME="daily_backup"
INSTANCE="server01"
PUSHGATEWAY_URL="http://localhost:9091"

start_time=$(date +%s)
echo "Starting backup..."
sleep 3
backup_success=1 # 1 表示成功，0 表示失败
end_time=$(date +%s)
duration=$((end_time - start_time))

cat <<EOF | curl --data-binary @- ${PUSHGATEWAY_URL}/metrics/job/${JOB_NAME}/instance/${INSTANCE}
# HELP backup_duration_seconds Duration of the backup job in seconds
# TYPE backup_duration_seconds gauge
backup_duration_seconds ${duration}
# HELP backup_success Whether the backup succeeded (1) or failed (0)
# TYPE backup_success gauge
backup_success ${backup_success}
EOF
echo "Metrics pushed to Pushgateway."

执行后，Prometheus 中可查询到： backup_duration_seconds{job="daily_backup", instance="server01"} 3 backup_success{job="daily_backup", instance="server01"} 1

5.2 Python 脚本监控数据处理任务

import requests
import time

def push_metrics(job, instance, records_processed, success):
    metrics = f"""# HELP data_records_processed Number of records processed
# TYPE data_records_processed gauge
data_records_processed {records_processed}
# HELP data_job_success Job success status (1 = success, 0 = failure)
# TYPE data_job_success gauge
data_job_success {int(success)}"""

    url = f"http://localhost:9091/metrics/job/{job}/instance/{instance}"
    response = requests.post(url, data=metrics.encode('utf-8'))
    if response.status_code == 202:
        print("Metrics pushed successfully.")
    else:
        print(f"Failed to push metrics: {response.status_code}")

# 模拟任务执行
start = time.time()
try:
    records = 1500
    time.sleep(2)
    success = True
except Exception as e:
    records = 0
    success = False

push_metrics(
    job="data_pipeline",
    instance="worker-node-01",
    records_processed=records,
    success=success
)

执行后，Prometheus 中将生成对应的 data_records_processed 与 data_job_success 指标。

总结

Pushgateway 作为 Prometheus 生态的补充组件，通过'推送 + 中转'机制有效解决了短生命周期任务无法被拉取监控的痛点。在实际应用中，只需合理配置 job 与 instance 标签，即可将 Shell 脚本、Python 批处理等临时作业的执行状态、耗时及业务指标无缝接入 Prometheus 监控体系。配合规范的指标清理策略，可确保监控数据的准确性与时效性。