SeaweedFS 分布式文件系统集群部署指南

3.2 关键配置解析

Master 节点配置

seaweedmaster1:
  image: chrislusf/seaweedfs:4.02
  ports:
    - "9333:9333" # Master HTTP API 端口
    - "19333:19333" # Master gRPC 端口
    - "9323:9323" # 监控指标端口
  command: 'master -port=9333 -ip=seaweedmaster1 -ip.bind=0.0.0.0 -metricsPort=9323 -peers=seaweedmaster1:9333,seaweedmaster2:9334,seaweedmaster3:9335'

关键参数说明：

• -peers 参数指定了所有 Master 节点，这是实现 Master 高可用的关键
• 每个 Master 节点使用不同的端口（9333、9334、9335）避免冲突
• -metricsPort 启用监控指标收集，便于 Prometheus 抓取数据

Volume 节点配置

seaweedvolume1:
  image: chrislusf/seaweedfs:4.02
  command: 'volume -max=30 -port=8080 -ip.bind=0.0.0.0 -mserver=seaweedmaster1:9333,seaweedmaster2:9334,seaweedmaster3:9335 -publicUrl=172.16.20.19:18080'

关键参数说明：

• -max=30 限制每个 Volume 节点最多创建 30 个卷（约 900GB 存储）
• -mserver 指定所有 Master 节点，确保 Volume 服务能正确注册
• -publicUrl 设置外部访问地址，确保文件 URL 可正确访问

Filer 和 S3 网关配置

Filer 服务提供了高级文件系统功能，如目录结构、权限管理等。S3 网关则使 SeaweedFS 可以兼容 Amazon S3 生态系统，方便现有应用迁移。

3.3 部署步骤

1. 准备环境：
   • 确保所有节点已安装 Docker 和 Docker Compose
   • 创建必要的目录结构并设置适当的权限
   • 确保网络配置正确，各容器间可相互通信
2. 验证服务：
   • 访问 Master 状态页面：http://主机 IP:9333/cluster/status?pretty=y
   • 检查 Volume 节点状态：http://主机 IP:8080/status?pretty=y
   • 验证 Filer 服务：http://主机 IP:8888/
3. 配置监控：
   • 访问 Grafana：http://主机 IP:3000（默认用户名/密码：admin/admin）
   • 配置 Prometheus 数据源，导入 SeaweedFS 监控看板

启动集群：

# 在包含 docker-compose.yml 的目录中执行
docker-compose up -d

3.4 数据持久化与备份策略

在您提供的配置中，数据持久化是通过 Docker 卷映射实现的：

volumes:
  - /user_path:/data

数据保护建议：

1. 定期备份 Master 元数据：虽然 Master 节点无状态，但元数据备份很重要
2. 设置合理的复制策略：通过 -defaultReplication 参数控制数据复制份数
3. 监控存储空间使用：设置适当的卷大小和数量限制，避免存储耗尽

系统优化与故障处理

4.1 性能优化建议

1. 小文件存储优化：SeaweedFS 内部会自动将小文件合并存储，这大大提高了海量小文件的存储效率。对于极度的小文件密集场景，可以考虑调整 Volume 大小。
2. 网络优化：确保存储节点间网络带宽充足，延迟低。使用专用网络进行节点间数据传输。
3. 内存配置：适当调整 JVM 参数（如果使用 Java 客户端），确保有足够内存处理文件操作。

4.2 常见问题及解决方案

1. Volume 节点无法注册：
   • 检查 Master 节点地址是否正确可达
   • 验证网络配置和防火墙规则
   • 查看 Volume 节点日志排除错误
2. 文件上传失败：
   • 检查 Volume 节点存储空间是否充足
   • 验证文件大小是否超过单个 Volume 容量限制
   • 检查复制策略设置是否与集群规模匹配
3. 性能下降：
   • 监控系统资源使用情况（CPU、内存、磁盘 IO）
   • 检查网络带宽和延迟
   • 考虑增加 Volume 节点分散负载

总结

SeaweedFS 作为一个现代化分布式文件系统，通过其独特的架构设计解决了海量小文件存储的挑战。与 MinIO 相比，SeaweedFS 在小文件处理性能和许可证友好度方面具有明显优势。其简单的架构使得运维更加直观，而完整的 S3 兼容性又确保了与现有工具的集成能力。

Docker Compose 完整配置

services:
  seaweedmaster1:
    image: chrislusf/seaweedfs:4.02
    ports:
      - "9333:9333"
      - "19333:19333"
      - "9323:9323"
    volumes:
      - /user_data_path:/data
    command: 'master -port=9333 -ip=seaweedmaster1 -ip.bind=0.0.0.0 -metricsPort=9323 -peers=seaweedmaster1:9333,seaweedmaster2:9334,seaweedmaster3:9335'
    networks:
      seaweednet:
        aliases:
          - seaweedmaster1

  seaweedmaster2:
    image: chrislusf/seaweedfs:4.02
    ports:
      - "9334:9334"
      - "19334:19334"
      - "9324:9324"
    volumes:
      - /user_data_path:/data
    command: 'master -port=9334 -ip=seaweedmaster2 -ip.bind=0.0.0.0 -metricsPort=9324 -peers=seaweedmaster1:9333,seaweedmaster2:9334,seaweedmaster3:9335'
    networks:
      seaweednet:
        aliases:
          - seaweedmaster2

  seaweedmaster3:
    image: chrislusf/seaweedfs:4.02
    ports:
      - "9335:9335"
      - "19335:19335"
      - "9325:9325"
    volumes:
      - /mnt/ztqbzx/seaweedfs/master3:/data
    command: 'master -port=9335 -ip=seaweedmaster3 -ip.bind=0.0.0.0 -metricsPort=9325 -peers=seaweedmaster1:9333,seaweedmaster2:9334,seaweedmaster3:9335'
    networks:
      seaweednet:
        aliases:
          - seaweedmaster3

  seaweedvolume1:
    image: chrislusf/seaweedfs:4.02
    ports:
      - "18080:8080"
    volumes:
      - /user_data_path:/data
    command: 'volume -max=30 -port=8080 -ip.bind=0.0.0.0 -mserver=seaweedmaster1:9333,seaweedmaster2:9334,seaweedmaster3:9335 -publicUrl=172.16.20.19:18080'
    depends_on:
      - seaweedmaster1
      - seaweedmaster2
      - seaweedmaster3
    networks:
      seaweednet:
        aliases:
          - seaweedvolume1

  seaweedvolume2:
    image: chrislusf/seaweedfs:4.02
    ports:
      - "18081:8080"
    volumes:
      - /user_data_path:/data
    command: 'volume -max=30 -port=8080 -ip.bind=0.0.0.0 -mserver=seaweedmaster1:9333,seaweedmaster2:9334,seaweedmaster3:9335 -publicUrl=172.16.20.19:18081'
    depends_on:
      - seaweedmaster1
      - seaweedmaster2
      - seaweedmaster3
    networks:
      seaweednet:
        aliases:
          - seaweedvolume2

  seaweedvolume3:
    image: chrislusf/seaweedfs:4.02
    ports:
      - "18082:8080"
    volumes:
      - /user_data_path:/data
    command: 'volume -max=30 -port=8080 -ip.bind=0.0.0.0 -mserver=seaweedmaster1:9333,seaweedmaster2:9334,seaweedmaster3:9335 -publicUrl=172.16.20.19:18082'
    depends_on:
      - seaweedmaster1
      - seaweedmaster2
      - seaweedmaster3
    networks:
      seaweednet:
        aliases:
          - seaweedvolume3

  filer:
    image: chrislusf/seaweedfs:4.02
    ports:
      - "8888:8888"
      - "18888:18888"
    command: 'filer -ip=filer -ip.bind=0.0.0.0 -master=seaweedmaster1:9333,seaweedmaster2:9334,seaweedmaster3:9335 -port=8888'
    depends_on:
      - seaweedmaster1
      - seaweedmaster2
      - seaweedmaster3
      - seaweedvolume1
      - seaweedvolume2
      - seaweedvolume3
    networks:
      seaweednet:
        aliases:
          - filer

  seaweeds3_1:
    image: chrislusf/seaweedfs:4.02
    ports:
      - "8333:8333"
    volumes:
      - /user_path/conf:/etc/seaweedfs
    command: 's3 -filer=filer:8888 -port=8333 -ip.bind=0.0.0.0 -config=/etc/seaweedfs/s3.json'
    depends_on:
      - filer
    networks:
      seaweednet:
        aliases:
          - seaweeds3-1

  seaweeds3_2:
    image: chrislusf/seaweedfs:4.02
    ports:
      - "8334:8333"
    volumes:
      - /user_path/conf:/etc/seaweedfs
    command: 's3 -filer=filer:8888 -port=8333 -ip.bind=0.0.0.0 -config=/etc/seaweedfs/s3.json'
    depends_on:
      - filer
    networks:
      seaweednet:
        aliases:
          - seaweeds3-2

  cronjob:
    image: chrislusf/seaweedfs:4.02
    command: 'cronjob -master=seaweedmaster1:9333,seaweedmaster2:9334,seaweedmaster3:9335'
    depends_on:
      - seaweedmaster1
      - seaweedmaster2
      - seaweedmaster3
      - seaweedvolume1
      - seaweedvolume2
      - seaweedvolume3
    networks:
      seaweednet:

  seaweedfs-haproxy:
    image: haproxy:latest
    container_name: seaweedfs_haproxy
    ports:
      - "9332:9332"
      - "8335:8335"
      - "8404:8404"
    volumes:
      - /user_path/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro
    networks:
      seaweednet:
        aliases:
          - swfhaproxy
    restart: unless-stopped

  prometheus:
    image: prom/prometheus:latest
    container_name: seaweedfs_prometheus
    user: "root"
    ports:
      - "9090:9090"
    volumes:
      - /user_path/prometheus.yml:/etc/prometheus/prometheus.yml
      - /user_path/prom_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.console.libraries=/etc/prometheus/console_libraries'
      - '--web.console.templates=/etc/prometheus/consoles'
      - '--storage.tsdb.retention.time=200h'
      - '--web.enable-lifecycle'
    networks:
      - "seaweednet"
    restart: unless-stopped

  grafana:
    image: grafana/grafana:latest
    container_name: seaweedfs_grafana
    user: "472:472"
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin!@#grafana
    networks:
      - "seaweednet"
    restart: unless-stopped

networks:
  seaweednet:
    name: seaweedfs-cluster
    driver: bridge

HAProxy 负载均衡配置

# 全局配置
global
  daemon
  maxconn 4096
  log 127.0.0.1 local0 notice
  user haproxy
  group haproxy

# 默认配置
defaults
  mode http
  log global
  option dontlognull
  timeout connect 5000ms
  timeout client 50000ms
  timeout server 50000ms
  retries 3

# 前端配置 - SeaweedFS Master 集群状态页和 API 入口
frontend seaweedfs_master_front
  bind *:9332
  mode http
  option httplog
  default_backend seaweedfs_master_back

# 后端配置 - SeaweedFS Master 服务器
backend seaweedfs_master_back
  mode http
  balance roundrobin
  option httpchk GET /cluster/status
  server master1 seaweedmaster1:9333 check inter 5s fall 3 rise 2
  server master2 seaweedmaster2:9334 check inter 5s fall 3 rise 2
  server master3 seaweedmaster3:9335 check inter 5s fall 3 rise 2

# 前端配置 - SeaweedFS S3 Gateway 入口
frontend seaweedfs_s3_front
  bind *:8335
  mode http
  option httplog
  default_backend seaweedfs_s3_back

# 后端配置 - SeaweedFS S3 Gateway 服务器
backend seaweedfs_s3_back
  mode http
  balance roundrobin
  option tcp-check
  default-server inter 5s fall 3 rise 2
  server s3_gateway1 seaweeds3-1:8333 check
  server s3_gateway2 seaweeds3-2:8333 check

# HAProxy 统计监控界面
listen stats
  bind *:8404
  mode http
  stats enable
  stats uri /haproxy?stats
  stats refresh 30s
  stats auth admin:cGYA7r7cwTRZPStVMS7m

S3 网关配置文件

{
  "identities": [
    {
      "name": "health_check",
      "actions": ["Read:/health"]
    },
    {
      "name": "admin",
      "credentials": [
        {
          "accessKey": "admin",
          "secretKey": "admin"
        }
      ],
      "actions": ["Admin", "Read", "Write", "List", "Tagging"]
    }
  ]
}