The fundamental Retrieval-Augmented Technology (RAG) pipeline makes use of an encoder mannequin to search out comparable paperwork given a question.
That is additionally Semantic Search It’s because the encoder converts the textual content right into a high-dimensional vector illustration (referred to as an embedding) the place semantically comparable textual content is positioned shut to one another.
Earlier than large-scale language fashions (LLMs) created these vector embeddings, the BM25 algorithm was a highly regarded search algorithm. BM25 focuses on necessary key phrases and appears for his or her actual matches inside the accessible paperwork. This method Key phrase Search.
If you wish to take your RAG pipeline to the subsequent stage, strive the next: Hybrid SearchHybrid search combines some great benefits of key phrase and semantic search to enhance search high quality.
On this article, we clarify the idea and implement all three search approaches in Python.
desk of contents
· RAG Acquisition
∘ Keyword search on BM25
∘ Semantic Search with Dense Embeddings
∘ Semantic or Hybrid Search?
∘ Hybrid Search
∘ Putting it all together
·…
