基于 Milvus 与混合检索的云厂商文档智能问答系统：Java SpringBoot 实践

文档类型识别与分块策略路由

不同类型的云文档需要不同的分块策略，我们设计了基于内容分析的智能路由：

@Component
public class SmartChunkingRouter {
    public List<DocumentChunk> chunkByContentAnalysis(ParsedDocument doc) {
        DocumentType docType = analyzeDocumentType(doc);
        switch (docType) {
            case SPECIFICATION:
                // 规格文档：检测参数表，保持表格完整性
                return chunkSpecificationDocument(doc);
            case PRICING:
                // 价格文档：检测价格矩阵，按服务项分块
                return chunkPricingDocument(doc);
            case TUTORIAL:
                // 教程文档：按操作步骤和示例分块
                return chunkTutorialDocument(doc);
            case API_REFERENCE:
                // API 文档：按端点和参数说明分块
                return chunkApiDocument(doc);
            default:
                // 默认策略：递归字符分块
                return recursiveTextSplit(doc, 800, 120);
        }
    }

    private DocumentType analyzeDocumentType(ParsedDocument doc) {
        // 基于关键词、结构特征和元数据识别文档类型
        String content = doc.getContent();
        if (containsPricingTable(content)) {
            return DocumentType.PRICING;
        } else if (containsApiEndpoints(content)) {
            return DocumentType.API_REFERENCE;
        } else if (containsSpecParameters(content)) {
            return DocumentType.SPECIFICATION;
        } else if (containsTutorialMarkers(content)) {
            return DocumentType.TUTORIAL;
        }
        return DocumentType.GENERAL;
    }
}

结构化元数据提取

为每个文档块提取丰富的元数据，为后续检索过滤奠定基础：

public class DocumentChunk {
    private String id;
    private String content;
    private Map<String, String> metadata;

    // 核心元数据字段
    private String documentSource;      // 文档来源：aliyun/tencent/huawei
    private String productCategory;     // 产品类别：compute/storage/network
    private String chunkType;           // 块类型：concept/parameter/price/example
    private String sectionTitle;        // 章节标题
    private String productName;         // 产品名称：ECS/RDS/VPC
    private String documentVersion;     // 文档版本
    private Date updateTime;            // 更新时间
}

Milvus 向量存储与混合检索实现

Milvus 2.3+ 版本原生支持混合检索（Hybrid Search），为我们提供了完美的技术基础。

集合 Schema 设计与优化

针对云文档的特点，我们设计了专门的集合结构：

@Data
@MilvusEntity(collectionName = "cloud_docs_chunks")
public class DocumentChunkEntity {
    // 主键字段
    @MilvusField(name = "chunk_id", isPrimaryKey = true)
    private String chunkId;

    // 内容字段（用于稀疏检索）
    @MilvusField(name = "content", dataType = DataType.VarChar, maxLength = 65535)
    private String content;

    // 稠密向量字段（768 维 BGE-M3 向量）
    @MilvusField(name = "dense_vector", dataType = DataType.FloatVector, dim = 768)
    private List<Float> denseVector;

    // 稀疏向量字段（BM25 权重表示）
    @MilvusField(name = "sparse_vector", dataType = DataType.SparseFloatVector)
    private Map<Long, Float> sparseVector;

    // 元数据字段（用于过滤和增强检索）
    @MilvusField(name = "doc_source", dataType = DataType.VarChar, maxLength = 50)
    private String docSource;

    @MilvusField(name = "product_name", dataType = DataType.VarChar, maxLength = 100)
    private String productName;

    @MilvusField(name = "chunk_type", dataType = DataType.VarChar, maxLength = 50)
    private String chunkType;

    @MilvusField(name = "tags", dataType = DataType.Array, elementType = DataType.VarChar)
    private List<String> tags;
}

混合检索的核心实现

混合检索的关键在于同时执行向量相似度搜索和关键词权重搜索，并将结果智能融合：

@Service
public class HybridSearchEngine {
    @Autowired
    private MilvusServiceClient milvusClient;

    public SearchResults hybridSearch(SearchRequest request) {
        // 1. 查询分析与路由
        QueryAnalysisResult analysis = analyzeQuery(request.getQuery());

        // 2. 并行执行两种检索
        CompletableFuture<List<SearchResult>> denseFuture = executeDenseVectorSearch(request, analysis);
        CompletableFuture<List<SearchResult>> sparseFuture = executeSparseVectorSearch(request, analysis);

        // 3. 结果融合与重排
        return CompletableFuture.allOf(denseFuture, sparseFuture).thenApply(v -> {
            List<SearchResult> denseResults = denseFuture.join();
            List<SearchResult> sparseResults = sparseFuture.join();

            // 基于查询类型动态调整权重
            float denseWeight = analysis.isSemanticQuery() ? 0.7f : 0.3f;
            float sparseWeight = 1.0f - denseWeight;

            // 加权分数融合
            List<SearchResult> fusedResults = fuseResults(denseResults, sparseResults, denseWeight, sparseWeight);

            // 重排提升精度
            return rerankResults(request.getQuery(), fusedResults);
        }).join();
    }

    private QueryAnalysisResult analyzeQuery(String query) {
        // 分析查询类型：概念性查询 vs 精确查询
        QueryAnalysisResult result = new QueryAnalysisResult();

        // 检测精确查询模式（产品型号、规格代码、价格查询）
        Pattern specPattern = Pattern.compile("[A-Z]{2,}\\.[a-z0-9]+\\.[a-z0-9]+");
        Pattern pricePattern = Pattern.compile("价格 | 费用 | 计费 | 成本");
        boolean isExactQuery = specPattern.matcher(query).find() || pricePattern.matcher(query).find() || containsExactProductCodes(query);
        
        result.setSemanticQuery(!isExactQuery);
        result.setExactQuery(isExactQuery);

        // 提取查询中的产品名称和关键词
        result.setProductNames(extractProductNames(query));
        result.setKeywords(extractKeywords(query));
        return result;
    }
}

查询权重动态调整算法

根据查询类型的分析结果，动态调整混合检索的权重分配：

public class WeightAdjustmentStrategy {
    public static SearchWeights calculateWeights(QueryAnalysisResult analysis) {
        SearchWeights weights = new SearchWeights();
        
        if (analysis.isExactQuery()) {
            // 精确查询：偏向关键词匹配
            weights.setDenseWeight(0.2f); // 语义权重 20%
            weights.setSparseWeight(0.8f); // 关键词权重 80%
            weights.setMetadataBoost(1.5f); // 元数据匹配增强
        } else if (analysis.isSemanticQuery()) {
            // 语义查询：偏向向量匹配
            weights.setDenseWeight(0.7f); // 语义权重 70%
            weights.setSparseWeight(0.3f); // 关键词权重 30%
            weights.setMetadataBoost(1.1f); // 元数据匹配轻微增强
        } else {
            // 混合查询：平衡权重
            weights.setDenseWeight(0.5f);
            weights.setSparseWeight(0.5f);
            weights.setMetadataBoost(1.3f);
        }

        // 根据查询长度微调
        int queryLength = analysis.getQueryLength();
        if (queryLength < 10) {
            // 短查询更依赖关键词
            weights.setSparseWeight(weights.getSparseWeight() + 0.1f);
            weights.setDenseWeight(weights.getDenseWeight() - 0.1f);
        }
        return weights;
    }
}

SpringBoot 微服务集成

异步文档处理管道

文档处理是计算密集型任务，我们采用全异步管道设计：

@Service
@Slf4j
public class AsyncDocumentPipeline {
    @Autowired
    private ThreadPoolTaskExecutor documentProcessor;

    @Async("documentProcessor")
    public CompletableFuture<ProcessResult> processDocumentAsync(MultipartFile file) {
        return CompletableFuture.supplyAsync(() -> parseDocument(file), documentProcessor)
            .thenApplyAsync(this::analyzeDocumentType, documentProcessor)
            .thenApplyAsync(this::chunkDocument, documentProcessor)
            .thenApplyAsync(chunks -> generateEmbeddings(chunks), documentProcessor)
            .thenApplyAsync(chunks -> generateSparseVectors(chunks), documentProcessor)
            .thenApplyAsync(chunks -> storeInMilvus(chunks), documentProcessor)
            .exceptionally(ex -> {
                log.error("文档处理失败：{}", ex.getMessage());
                return ProcessResult.failure(ex.getMessage());
            });
    }

    // 批量处理优化
    public CompletableFuture<List<ProcessResult>> batchProcess(List<MultipartFile> files) {
        List<CompletableFuture<ProcessResult>> futures = files.stream()
            .map(this::processDocumentAsync)
            .collect(Collectors.toList());
        return CompletableFuture.allOf(futures.toArray(new CompletableFuture[0]))
            .thenApply(v -> futures.stream().map(CompletableFuture::join).collect(Collectors.toList()));
    }
}

REST API 设计

提供简洁清晰的 API 接口：

@RestController
@RequestMapping("/api/v1/rag")
@Tag(name = "智能文档问答", description = "基于 RAG 的云文档智能问答接口")
public class RagController {

    @PostMapping("/documents")
    @Operation(summary = "上传文档到知识库")
    public ResponseEntity<UploadResponse> uploadDocument(
            @RequestParam("file") MultipartFile file,
            @RequestParam(value = "docSource", required = false) String docSource) {
        CompletableFuture<ProcessResult> future = documentPipeline.processDocumentAsync(file, docSource);
        return ResponseEntity.accepted().body(UploadResponse.accepted(future));
    }

    @PostMapping("/query")
    @Operation(summary = "查询知识库")
    public Flux<String> queryKnowledgeBase(@RequestBody QueryRequest request) {
        // 流式返回结果
        return Flux.create(sink -> {
            try {
                // 1. 检索相关文档块
                SearchResults results = searchEngine.hybridSearch(request);
                // 2. 构建 LLM 上下文
                String context = buildContext(results);
                // 3. 流式调用大模型
                streamLlmResponse(request.getQuestion(), context, sink);
            } catch (Exception e) {
                sink.error(e);
            }
        });
    }

    @GetMapping("/search/similar")
    @Operation(summary = "语义相似搜索")
    public ResponseEntity<List<SearchResult>> semanticSearch(
            @RequestParam String query,
            @RequestParam(defaultValue = "10") int topK) {
        List<SearchResult> results = searchEngine.semanticSearch(query, topK);
        return ResponseEntity.ok(results);
    }
}

性能优化与生产部署

向量检索性能调优

# application.yml Milvus 配置部分
milvus:
  host: ${MILVUS_HOST:localhost}
  port: 19530
  # 连接池配置
  connection-pool:
    max-size: 20
    min-size: 5
    connect-timeout-ms: 5000
    keep-alive-timeout-ms: 180000
  # 索引优化配置
  index:
    dense-vector:
      type: HNSW
      params:
        M: 16
        efConstruction: 200
    sparse-vector:
      type: SPARSE_INVERTED_INDEX
      params:
        drop_ratio_build: 0.2
  # 查询参数优化
  search:
    anns-field: dense_vector
    metric-type: IP
    params:
      nprobe: 16
    top-k: 50
    offset: 0

缓存策略设计

针对云文档查询特点，设计多层缓存策略：

@Component
@Slf4j
public class QueryCacheManager {
    @Autowired
    private RedisTemplate<String, Object> redisTemplate;

    // 本地缓存（Caffeine）用于热点查询
    private Cache<String, CacheEntry> localCache = Caffeine.newBuilder()
        .maximumSize(1000)
        .expireAfterWrite(5, TimeUnit.MINUTES)
        .build();

    public SearchResults getCachedResults(String queryHash, String filtersHash) {
        String cacheKey = buildCacheKey(queryHash, filtersHash);

        // 1. 检查本地缓存
        CacheEntry entry = localCache.getIfPresent(cacheKey);
        if (entry != null && !entry.isExpired()) {
            log.debug("本地缓存命中：{}", cacheKey);
            return entry.getResults();
        }

        // 2. 检查 Redis 分布式缓存
        SearchResults redisResults = (SearchResults) redisTemplate.opsForValue().get(cacheKey);
        if (redisResults != null) {
            log.debug("Redis 缓存命中：{}", cacheKey);
            // 回填本地缓存
            localCache.put(cacheKey, new CacheEntry(redisResults));
            return redisResults;
        }
        return null;
    }

    public void cacheResults(String queryHash, String filtersHash, SearchResults results, Duration ttl) {
        String cacheKey = buildCacheKey(queryHash, filtersHash);
        // 1. 存入本地缓存
        localCache.put(cacheKey, new CacheEntry(results));
        // 2. 存入 Redis，设置 TTL
        redisTemplate.opsForValue().set(cacheKey, results, ttl);
        log.debug("缓存已更新：{}, TTL: {}秒", cacheKey, ttl.getSeconds());
    }

    // 缓存键生成策略：结合查询语义和过滤条件
    private String buildCacheKey(String queryHash, String filtersHash) {
        return String.format("rag:search:%s:%s", queryHash, filtersHash);
    }
}

系统监控与评估

关键监控指标

构建完整的监控体系，跟踪系统健康状态：

@Component
@Slf4j
public class SystemMetricsCollector {
    // 检索质量指标
    private AtomicLong totalQueries = new AtomicLong(0);
    private AtomicLong semanticQueries = new AtomicLong(0);
    private AtomicLong exactQueries = new AtomicLong(0);
    private AtomicLong hybridQueries = new AtomicLong(0);

    // 性能指标
    private AtomicLong averageRetrievalTime = new AtomicLong(0);
    private AtomicLong averageRerankTime = new AtomicLong(0);
    private AtomicLong averageLlMTime = new AtomicLong(0);

    // 准确率指标
    private Map<String, AtomicLong> chunkTypeHits = new ConcurrentHashMap<>();
    private Map<String, AtomicLong> productHits = new ConcurrentHashMap<>();

    public void recordQuery(QueryAnalysisResult analysis, long retrievalTime, List<SearchResult> results) {
        totalQueries.incrementAndGet();
        if (analysis.isSemanticQuery()) {
            semanticQueries.incrementAndGet();
        } else if (analysis.isExactQuery()) {
            exactQueries.incrementAndGet();
        } else {
            hybridQueries.incrementAndGet();
        }

        // 更新平均检索时间（滑动平均）
        long currentAvg = averageRetrievalTime.get();
        long newAvg = (currentAvg * 99 + retrievalTime) / 100;
        averageRetrievalTime.set(newAvg);

        // 记录命中类型分布
        if (!results.isEmpty()) {
            for (SearchResult result : results) {
                String chunkType = result.getChunkType();
                chunkTypeHits.computeIfAbsent(chunkType, k -> new AtomicLong(0)).incrementAndGet();
                String productName = result.getProductName();
                if (productName != null) {
                    productHits.computeIfAbsent(productName, k -> new AtomicLong(0)).incrementAndGet();
                }
            }
        }
    }

    public MetricsReport generateReport() {
        MetricsReport report = new MetricsReport();
        report.setTimestamp(Instant.now());
        report.setTotalQueries(totalQueries.get());
        report.setSemanticQueryRatio(totalQueries.get() > 0 ? (double) semanticQueries.get() / totalQueries.get() : 0);
        report.setAverageRetrievalTimeMs(averageRetrievalTime.get());
        report.setChunkTypeDistribution(new HashMap<>(chunkTypeHits));
        report.setProductDistribution(new HashMap<>(productHits));
        return report;
    }
}

检索质量评估

设计自动化评估流程，持续优化系统：

@Service
public class RetrievalEvaluator {
    // 评估检索系统在不同类型查询上的表现
    public EvaluationResult evaluateOnTestSet(TestDataset testSet) {
        EvaluationResult result = new EvaluationResult();
        for (TestCase testCase : testSet.getTestCases()) {
            // 执行检索
            SearchResults searchResults = searchEngine.hybridSearch(SearchRequest.fromTestCase(testCase));

            // 计算精度指标
            double precision = calculatePrecision(testCase.getRelevantIds(), searchResults.getResultIds());
            double recall = calculateRecall(testCase.getRelevantIds(), searchResults.getResultIds());
            double ndcg = calculateNDCG(testCase.getRelevantIds(), searchResults.getScoredResults());

            // 按查询类型聚合统计
            String queryType = classifyQueryType(testCase.getQuery());
            result.addMetric(queryType, "precision", precision);
            result.addMetric(queryType, "recall", recall);
            result.addMetric(queryType, "ndcg", ndcg);

            // 记录失败案例用于分析
            if (precision < 0.5) {
                result.addFailureCase(testCase, searchResults);
            }
        }
        return result;
    }

    // A/B 测试不同检索策略
    public ABTestResult compareStrategies(SearchStrategy strategyA, SearchStrategy strategyB, TestDataset testSet) {
        ABTestResult result = new ABTestResult();
        for (TestCase testCase : testSet.getTestCases()) {
            SearchResults resultsA = executeSearch(strategyA, testCase);
            SearchResults resultsB = executeSearch(strategyB, testCase);

            // 人工评估或自动评估
            double scoreA = evaluateResults(resultsA, testCase);
            double scoreB = evaluateResults(resultsB, testCase);
            result.recordComparison(testCase, scoreA, scoreB);
        }
        return result.calculateWinner();
    }
}

总结与展望

本文详细介绍了基于 Milvus 和 Java SpringBoot 构建云厂商文档智能问答系统的完整方案。通过混合检索架构，系统能够同时处理语义查询和精确查询；通过结构化分块策略，保持了云文档的技术细节完整性；通过动态权重调整，优化了不同类型查询的检索效果。

未来，我们计划在以下方向进一步优化：

1. 多模态检索扩展：支持云架构图、流程图等图像内容的检索
2. 个性化推荐：基于用户角色和历史查询，提供个性化文档推荐
3. 实时知识更新：建立文档变更监控，自动同步最新内容到知识库
4. 跨厂商统一检索：构建统一的查询接口，跨云厂商比较产品特性

云文档智能问答系统的建设是一个持续迭代的过程，随着大模型技术和向量数据库技术的快速发展，我们相信这类系统将变得更加智能、高效，成为云原生时代不可或缺的基础设施。