LLM RAG 检索增强生成原理与应用详解

LLM RAG 检索增强生成原理与应用详解 | 极客日志

from langchain_community.document_loaders import UnstructuredHTMLLoader
loader = UnstructuredHTMLLoader("example_data/fake-content.html")
data = loader.load()

from langchain.text_splitter import MarkdownHeaderTextSplitter

markdown_document = "# Foo\n\n ## Bar\n\nHi this is Jim\n\nHi this is Joe\n\n ### Boo \n\n Hi this is Lance \n\n ## Baz\n\n Hi this is Molly"
  
headers_to_split_on = [  
("#", "Header 1"),  
("##", "Header 2"),  
("###", "Header 3"),  
]  

markdown_splitter = MarkdownHeaderTextSplitter(headers_to_split_on=headers_to_split_on)  
md_header_splits = markdown_splitter.split_text(markdown_document)  
md_header_splits

from langchain_experimental.text_splitter import SemanticChunker
from langchain_openai.embeddings import OpenAIEmbeddings

# This is a long document we can split up.
with open("../../state_of_the_union.txt") as f:
    state_of_the_union = f.read()
    
text_splitter = SemanticChunker(OpenAIEmbeddings())

docs = text_splitter.create_documents([state_of_the_union])  
print(docs[0].page_content)

metadatas = [{"document": 1}, {"document": 2}]  
documents = text_splitter.create_documents(  
[state_of_the_union, state_of_the_union], metadatas=metadatas  
)  
print(documents[0])

模型	中文支持
M3E	是
text2vec	是
OpenAIEmbeddings	是

from langchain_openai import OpenAIEmbeddings  

embeddings_model = OpenAIEmbeddings(openai_api_key="...")

embeddings = embeddings_model.embed_documents(
    [
        "Hi there!",
        "Oh, hello!",
        "What's your name?",
        "My friends call me World",
        "Hello World!"
    ]
)
len(embeddings), len(embeddings[0])

from langchain_community.document_loaders import TextLoader
from langchain_openai import OpenAIEmbeddings
from langchain.text_splitter import CharacterTextSplitter
from langchain_community.vectorstores import LanceDB

import lancedb

db = lancedb.connect("/tmp/lancedb")
table = db.create_table(
    "my_table",
    data=[
        {
            "vector": embeddings.embed_query("Hello World"),
            "text": "Hello World",
            "id": "1",
        }
    ],
    mode="overwrite",
)

# Load the document, split it into chunks, embed each chunk and load it into the vector store.
raw_documents = TextLoader('../../../state_of_the_union.txt').load()
text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)
documents = text_splitter.split_documents(raw_documents)
db = LanceDB.from_documents(documents, OpenAIEmbeddings(), connection=table)

@abstractmethod
def add_texts(
    self,
    texts: Iterable[str],
    metadatas: Optional[List[dict]] = None,
    **kwargs: Any,
) -> List[str]:
    """Run more texts through the embeddings and add to the vectorstore."""
    
@abstractmethod
def similarity_search(
    self, query: str, k: int = 4, **kwargs: Any
) -> List[Document]:
    """Return docs most similar to query."""
    
@classmethod
@abstractmethod
def from_texts(
    cls: Type[VST],
    texts: List[str],
    embedding: Embeddings,
    metadatas: Optional[List[dict]] = None,
    **kwargs: Any,
) -> VST:
    """Return VectorStore initialized from texts and texts."""

template = """Answer the question based only on the following context:  

{context}  

Question: {question}  
"""

Name	Index Type	Uses an LLM	When to Use	Description
Vectorstore	Vectorstore	No	If you are just getting started and looking for something quick and easy.	This is the simplest method and the one that is easiest to get started with. It involves creating embeddings for each piece of text.
Vectorstore + Document Store	Vectorstore + Document Store	No	If your pages have lots of smaller pieces of distinct information that are best indexed by themselves, but best retrieved all together.	This involves indexing multiple chunks for each document. Then you find the chunks that are most similar in embedding space, but you retrieve the whole parent document and return that (rather than individual chunks).
Vectorstore + Document Store	Vectorstore + Document Store	Sometimes during indexing	If you are able to extract information from documents that you think is more relevant to index than the text itself.	This involves creating multiple vectors for each document. Each vector could be created in a myriad of ways - examples include summaries of the text and hypothetical questions.
Vectorstore	Vectorstore	Yes	If users are asking questions that are better answered by fetching documents based on metadata rather than similarity with the text.	This uses an LLM to transform user input into two things: (1) a string to look up semantically, (2) a metadata filer to go along with it. This is useful because oftentimes questions are about the METADATA of documents (not the content itself).
Any	Any	Sometimes	If you are finding that your retrieved documents contain too much irrelevant information and are distracting the LLM.	This puts a post-processing step on top of another retriever and extracts only the most relevant information from retrieved documents. This can be done with embeddings or an LLM.
Vectorstore	Vectorstore	No	If you have timestamps associated with your documents, and you want to retrieve the most recent ones	This fetches documents based on a combination of semantic similarity (as in normal vector retrieval) and recency (looking at timestamps of indexed documents)
Any	Any	Yes	If users are asking questions that are complex and require multiple pieces of distinct information to respond	This uses an LLM to generate multiple queries from the original one. This is useful when the original query needs pieces of information about multiple topics to be properly answered. By generating multiple queries, we can then fetch documents for each of them.
Any	Any	No	If you have multiple retrieval methods and want to try combining them.	This fetches documents from multiple retrievers and then combines them.

COLLECTION_NAME = "state_of_the_union_test"  

db = PGVector.from_documents(  
embedding=embeddings,  
documents=docs,  
collection_name=COLLECTION_NAME,  
connection_string=CONNECTION_STRING,  
)
query = "What did the president say about Ketanji Brown Jackson"  
docs_with_score = db.similarity_search_with_score(query)

for doc, score in docs_with_score:  
print("-" * 80)  
print("Score: ", score)  
print(doc.page_content)  
print("-" * 80)

docs_with_score = db.max_marginal_relevance_search_with_score(query)

for doc, score in docs_with_score:  
print("-" * 80)  
print("Score: ", score)  
print(doc.page_content)  
print("-" * 80)

def maximal_marginal_relevance(
    query_embedding: np.ndarray,
    embedding_list: list,
    lambda_mult: float = 0.5,
    k: int = 4,
) -> List[int]:
    """Calculate maximal marginal relevance."""
    if min(k, len(embedding_list)) <= 0:
        return []
    if query_embedding.ndim == 1:
        query_embedding = np.expand_dims(query_embedding, axis=0)
    similarity_to_query = cosine_similarity(query_embedding, embedding_list)[0]
    most_similar = int(np.argmax(similarity_to_query))
    idxs = [most_similar]
    selected = np.array([embedding_list[most_similar]])
    while len(idxs) < min(k, len(embedding_list)):
        best_score = -np.inf
        idx_to_add = -1
        similarity_to_selected = cosine_similarity(embedding_list, selected)
        for i, query_score in enumerate(similarity_to_query):
            if i in idxs:
                continue
            redundant_score = max(similarity_to_selected[i])
            equation_score = (
                lambda_mult * query_score - (1 - lambda_mult) * redundant_score
            )
            if equation_score > best_score:
                best_score = equation_score
                idx_to_add = i
        idxs.append(idx_to_add)
        selected = np.append(selected, [embedding_list[idx_to_add]], axis=0)
    return idxs

LLM RAG 检索增强生成原理与应用详解

前言

一、RAG 基本介绍

二、RAG 技术实现

2.1 数据加载（Document Loaders）

2.2 数据处理（Text Splitters）

2.2.1 数据分割

2.2.2 数据信息（Metadata）

2.2.3 分割参数

2.3 数据向量化（Text embedding models）

2.4 向量数据库（Vector stores）

2.5 数据召回（Retrievers）

2.5.1 数据召回算法

三、RAG 实践挑战与优化

3.1 检索精度优化

3.2 评估体系构建

3.3 延迟与成本考量

四、总结

更多推荐文章

相关免费在线工具

LLM RAG 检索增强生成原理与应用详解

前言

一、RAG 基本介绍

二、RAG 技术实现

2.1 数据加载（Document Loaders）

2.2 数据处理（Text Splitters）

2.2.1 数据分割

2.2.2 数据信息（Metadata）

2.2.3 分割参数

2.3 数据向量化（Text embedding models）

2.4 向量数据库（Vector stores）

2.5 数据召回（Retrievers）

2.5.1 数据召回算法

三、RAG 实践挑战与优化

3.1 检索精度优化

3.2 评估体系构建

3.3 延迟与成本考量

四、总结

微信扫一扫，关注极客日志

更多推荐文章

相关免费在线工具