Java Web 大文件分块上传与断点续传实现方案

大文件传输系统技术方案

需求分析与技术挑战

主要面临以下技术挑战：

1. 超大文件处理：单文件 100G 的高效传输
2. 文件夹结构保留：完整保留层级结构，非打包传输
3. 高稳定性断点续传：浏览器刷新/关闭不丢失进度
4. 多环境兼容性：跨平台 (Windows/macOS/Linux)、跨浏览器
5. 安全要求：支持 SM4/AES 加密，传输与存储安全
6. 高并发下载：避免服务器资源耗尽
7. 现有系统集成：兼容 JSP/Spring Boot/Vue/React 等技术栈

技术架构设计

整体架构

[客户端] ---HTTPS(加密)---> [Nginx 负载均衡] ---> [应用服务集群]
                                      |
                                      v
                              [阿里云 OSS] <---加密存储---> [文件处理服务]
                                                              <---> [MySQL 集群]

核心技术选型

1. 文件分片：采用动态分片策略 (默认 10MB/片，大文件自动调整)
2. 断点续传：基于 Redis+MySQL 的双重进度保存机制
3. 文件夹处理：虚拟文件系统 (VFS) 维护目录结构
4. 加密模块：国密 SM4 与 AES 可插拔式加密组件
5. 传输优化：TCP BBR 拥塞控制算法 + 多线程传输

核心功能代码示例

前端上传组件 (Vue2 示例)

export default {
  data() {
    return {
      files: [],
      progress: 0,
      uploadId: '',
      chunkSize: 10 * 1024 * 1024, // 10MB
      concurrentLimit: 3
    };
  },
  methods: {
    handleFileChange(e) {
      this.files = Array.from(e.target.files);
      this.prescanFiles(this.files);
    },
    async prescanFiles(files) {
      const { data } = await this.$http.post('/api/upload/prepare', {
        files: files.map(f => ({
          name: f.name,
          size: f.size,
          relativePath: f.webkitRelativePath || ''
        }))
      });
      this.uploadId = data.uploadId;
    },
    async startUpload() {
      for (const file of this.files) {
        await this.uploadFile(file);
      }
    },
    async uploadFile(file) {
      const totalChunks = Math.ceil(file.size / this.chunkSize);
      const chunks = Array(totalChunks).fill().map((_, i) => ({
        index: i,
        start: i * this.chunkSize,
        end: Math.min(file.size, (i + 1) * this.chunkSize)
      }));
      // 并发控制上传
      const queue = [];
      for (let i = 0; i < chunks.length; i++) {
        const chunk = chunks[i];
        const blob = file.slice(chunk.start, chunk.end);
        queue.push(this.uploadChunk(file, blob, chunk));
        if (queue.length >= this.concurrentLimit) {
          await Promise.all(queue);
          queue.length = 0;
        }
      }
      await Promise.all(queue);
      // 完成上传
      await this.$http.post('/api/upload/complete', {
        uploadId: this.uploadId,
        fileName: file.name,
        fileSize: file.size,
        totalChunks,
        filePath: file.webkitRelativePath || ''
      });
    },
    async uploadChunk(file, blob, chunk) {
      const formData = new FormData();
      formData.append('file', blob);
      formData.append('uploadId', this.uploadId);
      formData.append('chunkIndex', chunk.index);
      formData.append('fileName', file.name);
      formData.append('filePath', file.webkitRelativePath || '');
      try {
        await this.$http.post('/api/upload/chunk', formData, {
          onUploadProgress: progressEvent => {
            const percent = Math.round((progressEvent.loaded / progressEvent.total) * 100);
            this.updateProgress(file.name, chunk.index, percent);
          }
        });
      } catch (error) {
        console.error('上传分片失败:', error);
        await this.uploadChunk(file, blob, chunk); // 自动重试
      }
    },
    updateProgress(fileName, chunkIndex, percent) {
      // 更新进度逻辑
    }
  }
};

后端核心处理 (Spring Boot 示例)

@RestController
@RequestMapping("/api/upload")
public class FileUploadController {
    @Autowired
    private FileStorageService storageService;
    @Autowired
    private RedisTemplate redisTemplate;

    @PostMapping("/prepare")
    public ResponseEntity prepareUpload(@RequestBody UploadPrepareRequest request) {
        String uploadId = UUID.randomUUID().toString();
        // 初始化上传任务
        UploadSession session = new UploadSession();
        session.setUploadId(uploadId);
        session.setFiles(request.getFiles());
        session.setStatus(UploadStatus.INITIALIZED);
        session.setCreatedAt(new Date());
        // 存储到 Redis 和数据库
        redisTemplate.opsForValue().set("upload:" + uploadId, session);
        storageService.saveUploadSession(session);
        return ResponseEntity.ok(new UploadPrepareResponse(uploadId));
    }

    @PostMapping("/chunk")
    public ResponseEntity uploadChunk(
            @RequestParam("file") MultipartFile file,
            @RequestParam("uploadId") String uploadId,
            @RequestParam("chunkIndex") int chunkIndex,
            @RequestParam("fileName") String fileName,
            @RequestParam(value = "filePath", required = false) String filePath) {
        // 验证上传会话
        UploadSession session = (UploadSession) redisTemplate.opsForValue().get("upload:" + uploadId);
        if (session == null || session.getStatus() != UploadStatus.INITIALIZED) {
            return ResponseEntity.status(HttpStatus.BAD_REQUEST).body("无效的上传会话");
        }
        try {
            // 加密存储分片
            byte[] encryptedChunk = CryptoUtils.encrypt(file.getBytes(), storageService.getEncryptionAlgorithm(), storageService.getEncryptionKey());
            String chunkKey = String.format("%s/%s/%d", uploadId, fileName, chunkIndex);
            storageService.storeChunk(chunkKey, encryptedChunk, filePath);
            // 更新进度
            session.getChunkStatuses().put(chunkKey, ChunkStatus.COMPLETED);
            redisTemplate.opsForValue().set("upload:" + uploadId, session);
            return ResponseEntity.ok().build();
        } catch (Exception e) {
            return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR).body("分片上传失败：" + e.getMessage());
        }
    }

    @PostMapping("/complete")
    public ResponseEntity completeUpload(@RequestBody UploadCompleteRequest request) {
        // 验证所有分片已上传
        // 合并分片
        // 生成最终文件元数据
        // 清理临时数据
        return ResponseEntity.ok(new UploadCompleteResponse(request.getFileName(), finalPath));
    }
}

文件加密模块 (Java 实现)

public class CryptoUtils {
    private static final String AES_ALGORITHM = "AES/CBC/PKCS5Padding";
    private static final String SM4_ALGORITHM = "SM4/CBC/PKCS5Padding";

    public static byte[] encrypt(byte[] data, String algorithm, String key)
            throws NoSuchAlgorithmException, NoSuchPaddingException, InvalidKeyException, IllegalBlockSizeException, BadPaddingException, InvalidAlgorithmParameterException {
        Cipher cipher = Cipher.getInstance(getCipherAlgorithm(algorithm));
        SecretKeySpec keySpec = new SecretKeySpec(generateKey(algorithm, key), algorithm);
        IvParameterSpec iv = generateIv(algorithm);
        cipher.init(Cipher.ENCRYPT_MODE, keySpec, iv);
        return cipher.doFinal(data);
    }

    public static byte[] decrypt(byte[] encryptedData, String algorithm, String key)
            throws NoSuchAlgorithmException, NoSuchPaddingException, InvalidKeyException, IllegalBlockSizeException, BadPaddingException, InvalidAlgorithmParameterException {
        Cipher cipher = Cipher.getInstance(getCipherAlgorithm(algorithm));
        SecretKeySpec keySpec = new SecretKeySpec(generateKey(algorithm, key), algorithm);
        IvParameterSpec iv = generateIv(algorithm);
        cipher.init(Cipher.DECRYPT_MODE, keySpec, iv);
        return cipher.doFinal(encryptedData);
    }

    private static String getCipherAlgorithm(String algorithm) {
        return algorithm.equalsIgnoreCase("SM4") ? SM4_ALGORITHM : AES_ALGORITHM;
    }

    private static byte[] generateKey(String algorithm, String key) {
        // 密钥生成逻辑
    }

    private static IvParameterSpec generateIv(String algorithm) {
        // IV 生成逻辑
    }
}

系统特性与优势

1. 高可靠断点续传机制

设计了三级恢复机制确保断点续传的可靠性：

• 客户端缓存：LocalStorage 保存基本进度信息
• 服务端 Redis 缓存：实时更新上传状态
• 数据库持久化：定时同步确保数据不丢失

2. 文件夹结构处理方案

采用虚拟文件系统 (VFS) 技术维护原始目录结构。 数据结构示例：

{
  "uploadId": "xyz123",
  "root": {
    "name": "project",
    "type": "directory",
    "children": [
      {
        "name": "src",
        "type": "directory",
        "children": [
          {
            "name": "main.js",
            "type": "file",
            "size": 1024,
            "chunks": [{"id": 1, "status": "completed"}]
          }
        ]
      }
    ]
  }
}

3. 性能优化措施

• 动态分片策略：根据网络状况自动调整分片大小 (1MB-20MB)
• 智能重试机制：指数退避算法处理失败分片
• 内存优化：采用零拷贝技术处理大文件，避免内存溢出