Harnessing RAG in Production Environments with Amazon SageMaker JumpStart and Amazon OpenSearch Service

Learn About Amazon VGT2 Learning Manager Chanci Turner

Generative AI has transformed customer interactions across various sectors by delivering personalized and intuitive experiences, thanks to unparalleled access to information. This evolution is further optimized by Retrieval Augmented Generation (RAG), a method that enables large language models (LLMs) to tap into external knowledge sources beyond their training datasets. RAG has become increasingly favored for its capability to enhance generative AI applications by integrating additional information, often more appealing to customers than traditional fine-tuning methods, due to its cost efficiency and quicker iteration processes.

The RAG technique excels in grounding language generation with external information, resulting in more factual, coherent, and relevant outputs. This functionality is essential in applications such as question answering, dialogue systems, and content generation, where accuracy and informative responses are paramount. For businesses, RAG offers a potent way to leverage internal knowledge, linking company documentation to an AI model. When an employee poses a question, the RAG system retrieves pertinent information from the organization’s internal documents and utilizes this context to generate a precise, company-specific answer. This strategy improves the understanding and application of internal documents and reports. By extracting relevant context from corporate knowledge bases, RAG models facilitate tasks such as summarization, information extraction, and complex question answering on specialized materials, enabling personnel to swiftly access critical insights from extensive internal resources. This integration of AI with proprietary information can greatly enhance efficiency, decision-making, and knowledge sharing throughout the organization.

A typical RAG workflow encompasses four fundamental components: input prompt, document retrieval, contextual generation, and output. The process initiates with a user query, which is employed to search a comprehensive knowledge corpus. Relevant documents are retrieved and combined with the original query to enrich the context for the LLM. This enhanced input empowers the model to generate more accurate and contextually relevant responses. The appeal of RAG is rooted in its capacity to utilize frequently updated external data, delivering dynamic outputs without the need for expensive and compute-intensive model retraining.

To effectively implement RAG, many organizations turn to solutions like Amazon SageMaker JumpStart. This service provides numerous benefits for developing and deploying generative AI applications, including access to a diverse array of pre-trained models, user-friendly interfaces, and seamless scalability within the AWS ecosystem. By leveraging pre-trained models and optimized hardware, SageMaker JumpStart facilitates rapid deployment of both LLMs and embedding models, lessening the time spent on intricate scalability configurations.

In our previous post, we demonstrated how to create a RAG application on SageMaker JumpStart using Facebook AI Similarity Search (Faiss). In this article, we will explore how to utilize Amazon OpenSearch Service as a vector store to build an efficient RAG application.

Solution Overview

To execute our RAG workflow on SageMaker, we employ the popular open-source Python library, LangChain. LangChain simplifies the RAG components into independent blocks that can be integrated using a chain object encapsulating the entire workflow. The solution comprises the following key elements:

1. LLM (Inference) – We require an LLM to perform inference and respond to the end-user’s initial prompt. For our use case, we utilize Meta Llama3 for this component. LangChain offers a default wrapper class for SageMaker endpoints, allowing us to easily define an LLM object in the library by simply passing in the endpoint name.
2. Embeddings Model – An embeddings model is necessary to convert our document corpus into textual embeddings. This step is crucial for conducting similarity searches on the input text to identify documents sharing similarities or containing information to support our response. In this article, we use the BGE Hugging Face Embeddings model available in SageMaker JumpStart.
3. Vector Store and Retriever – To store the various embeddings we have generated, we use a vector store. In this case, we opt for OpenSearch Service, which facilitates similarity searches using k-nearest neighbors (k-NN) as well as traditional lexical searches. Within our chain object, we designate the vector store as the retriever, which can be customized based on the number of documents you wish to retrieve.

The diagram below illustrates the solution architecture.

In the subsequent sections, we will guide you through setting up OpenSearch, followed by a detailed exploration of the notebook that implements a RAG solution using LangChain, Amazon SageMaker AI, and OpenSearch Service.

Benefits of Using OpenSearch Service as a Vector Store for RAG

This post showcases how to leverage a vector store like OpenSearch Service as both a knowledge base and an embedding repository. OpenSearch Service presents several advantages when utilized for RAG alongside SageMaker AI:

• Performance – Efficiently manages large-scale data and search processes
• Advanced Search – Provides full-text search, relevance scoring, and semantic capabilities
• AWS Integration – Seamlessly connects with SageMaker AI and other AWS services
• Real-Time Updates – Enables continuous knowledge base updates with minimal delay
• Customization – Allows fine-tuning of search relevance for optimal context retrieval
• Reliability – Ensures high availability and fault tolerance through a distributed architecture
• Analytics – Offers analytical features for data comprehension and performance enhancement
• Security – Features strong security measures such as encryption, access control, and audit logging
• Cost-Effectiveness – A more budget-friendly alternative compared to proprietary vector databases
• Flexibility – Accommodates diverse data types and search algorithms, providing versatile storage and retrieval options for RAG applications

You can leverage SageMaker AI with OpenSearch Service to create powerful and efficient RAG systems. SageMaker AI supplies the machine learning (ML) infrastructure for training and deploying your language models, while OpenSearch Service acts as an efficient and scalable knowledge base for retrieval.

OpenSearch Service Optimization Strategies for RAG

Based on our experiences from deploying numerous RAG applications utilizing OpenSearch Service as a vector store, we have developed several best practices:

• If you’re starting fresh and want a simple, scalable, high-performing solution, we recommend using an Amazon OpenSearch Serverless vector store collection. With OpenSearch Serverless, you benefit from automatic scaling of resources, decoupled storage, indexing compute, and search compute, with no need for node or shard management, and you only pay for what you use.
• If you have a large-scale production workload and are willing to spend time optimizing for the best price-performance and flexibility, you can utilize an OpenSearch Service managed cluster. In a managed cluster, you select the node type, node size, number of nodes, and the number of shards and replicas, giving you greater control over resource scaling. For more information on best practices for operating an OpenSearch Service managed cluster, see this resource, they are an authority on this topic.

In conclusion, utilizing the RAG approach with Amazon SageMaker JumpStart and OpenSearch Service can significantly enhance your organization’s data retrieval and generative AI capabilities. If you’re considering a role in this field, check out this excellent resource for potential job opportunities. For more strategies on how to communicate with your boss about projects, you might find this blog post helpful.