RAG Architecture: A beginner’s guide

Why Every Chatbot and Its Grandma Is Getting Smarter: Everything about a “RAG”

Introduction

You’ve probably noticed it, chatbots are suddenly everywhere, and they’re not just answering “What time do you close?” anymore. They’re writing emails, planning your vacations, and in cases like Apple Intelligence, probably keeping your life more organised than you do. We’re not talking about general chatbots like ChatGPT or Gemini anymore, but highly specific chatbots, specific to a particular application, say a ChatGPT-like assistant, for your United Airlines app. But what’s behind this leap from “meh” to “mind-blowing”? Spoiler alert: It’s not just those big, fancy Large Language Models (LLMs) you keep hearing about.

The real MVP? Retrieval-Augmented Generation (RAG). It’s the quiet hero making sure your chatbot doesn’t hallucinate that your next meeting is with Elon Musk. Retrieval-Augmented Generation is already powering practical solutions across diverse industries, enhancing user experiences and operational efficiency. Apple Intelligence uses RAG-like systems to personalize interactions on our iPhones, combining user data with live retrievals for tasks like setting reminders or answering device-settings specific queries. The chatbot that you see on Amazon, “Rufus” is a RAG implementation. This article talks about Rufus being a RAG implementation while this is an informative blog on the implementation of Rufus. Now that we know the cool kids are using it, let’s peel back the curtain and see how RAG is helping AI chatbots go from basic to brilliant.

If you’re too busy to read the entire article, the following image does a pretty good job, an overview of the entire RAG process, with the solid black line marking what users see and the coloured lines representing all that goes on under the hood. Think of it as the TL;DR of RAG visualized!

RAG 101: What It Is and Where It All Began

At its core, RAG is a clever marriage of two powerful capabilities: the vast linguistic prowess of Large Language Models (LLMs) and the precision of targeted information retrieval. The idea is simple but transformative take a user’s question, enrich it with relevant, up-to-date context from external sources like databases or proprietary documents like PDFs and feed it into an LLM to produce highly accurate and context-aware responses.

LLMs

At this point, you might ask, Hey! I’ve been using ChatGPT, and I’ve heard about LLMs, how is RAG different? While LLMs like GPT-4 and Llama 3 are undeniably impressive, they have limitations. These models are trained on large datasets up to a specific point in time, meaning they lack knowledge of recent events or proprietary information. They’re also known to “hallucinate,” confidently generating plausible but incorrect answers when they don’t know something. Picture this: you ask an LLM, one that isn’t regularly updated, about the latest version of an iPhone. A hallucinating LLM chatbot, like the early versions of ChatGPT, might concoct product specifications and details that seem realistic but have no basis in fact. RAG addresses these gaps by integrating reliable, task-specific data into the process, enhancing both accuracy and relevance.

RAGs bring detail, context and specificity. Think of companies that aim to tailor a Large Language Model (LLM) or chatbot to their specific applications or platforms. A prime example of this is Apple Intelligence, which utilizes OpenAI’s LLMs but adapts them to suit its unique ecosystem. This personalization allows users to perform tasks that are specific to devices like the iPhone, such as setting reminders, thereby enhancing the user experience with technology that feels distinctly Apple.

I can think of one particular analogy, well-suited to this scenario. In preparing this article, I’ve acted as a RAG system. Behind the scenes, I’ve sifted through numerous relevant documents, absorbing everything there is to know about RAG. This isn’t just passive reading; it’s active learning and synthesising, reading articles specific to LLMs and RAG, akin to how a RAG system would embed and retrieve relevant vectors from vast databases.

In this article…

Now, armed with this comprehensive understanding, I use my (somewhat limited) writing skills not just to regurgitate facts but to present them in a manner that’s concise, detailed, and accessible, suitable for everyone from beginners to experts. This process of distilling complex information into an engaging and informative narrative is reflective of how RAG leverages its architectural components to enhance the utility and accuracy of LLM outputs.

Thus, by reading this article, you’re not just reading about RAG; you’re experiencing an illustration of how RAG fundamentally operates: taking a broad query (“What is RAG?”), conducting a deep and thorough retrieval of information, and generating a response that is both informative and easy to digest, much like the best RAG systems aspire to do.

Origins of RAG

Retrieval Augmented Generation (RAG) has its roots in the quest to enhance the factual accuracy and relevance of AI-generated content, particularly in knowledge-intensive tasks. The concept was first introduced in the 2020 paper “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks” by Patrick Lewis and colleagues. Initially, RAG was conceived as a method to improve the performance of language models on tasks that required access to external knowledge sources. Traditional language models, while adept at generating coherent text, often struggle with providing accurate information, especially when dealing with specialized or up-to-date knowledge. RAG addressed this limitation by integrating a retrieval mechanism that could access relevant documents during the generation process, effectively combining the strengths of retrieval-based and generative models.

Over time, the application of RAG has evolved beyond its original scope. Today, it is widely used to enhance large language models by providing them with access to external databases, proprietary documents, and real-time information. Moreover, with AI agents, we’re not just stopping at creating answers. These are smart systems powered by LLMs that can do more than just respond, they can act on their own. For example, imagine asking your AI assistant to plan a vacation. Instead of just giving you a list of destinations, an AI agent could search for flights, book a hotel, schedule activities, and even send you reminders about your trip. It’s like having a digital assistant that doesn’t just give advice but actually gets things done for you.

Why should you care?

Simply because RAG helps address some of the drawbacks of an LLM-based application

Large Language Models (LLMs) are undoubtedly impressive, but they’re far from perfect. One of their most glaring issues is their tendency to hallucinate, confidently spouting information that sounds plausible but has no factual basis. This is a famous paper about hallucinating LLMs. For example, an LLM might claim that a historical figure invented something they didn’t or mix up timelines. While this can be amusing in casual settings, it’s catastrophic in high-stakes applications like medical diagnosis, financial analysis or law as seen here.

Conformity

Another major drawback is conformity. LLMs often produce outputs that align too closely with common patterns in their training data, even when those patterns don’t fit the query’s context. This results in generic, predictable, and sometimes outright irrelevant responses. Imagine asking an LLM about a niche scientific concept, only to receive an overly simplified or misleading explanation rooted in unrelated general knowledge.

Asch Conformity Test

I ran a popular social experiment, called the Asch Conformity Test (enter Asch test reference here) using LLMs as test subjects, to test whether LLMs succumb to pressure and change their answer to please their environment. The LLM was told it was in a group with other subjects, all of whom gave wrong answers to any question asked, to see if the LLM would also succumb to the social pressure. Although high-risk fields did have comprehensive safeguards against conformity, some questions in domains like math and general knowledge were prone to conformity. Here are a couple of hand-picked examples of conformity.

Information

Finally, LLMs struggle with stale information. These models are trained on snapshots of data and lack the ability to update themselves dynamically. If you ask about an event or product released after the training cutoff, the LLM won’t have a clue or may fabricate an answer, further exacerbating the hallucination problem. This limitation is particularly problematic in domains like technology or news, where up-to-date information is critical. Over time, they may also experience model drift, a gradual decline in accuracy and relevance as the world evolves and the static training data remains unchanged. What’s trendy today might be tomorrow’s cringe, so your model needs to be updated constantly to look cool. Regular updates and mechanisms like retrieval-augmented generation (RAG) are vital for combating model drift and keeping the system aligned with current knowledge. 

Enter RAG

This is where Retrieval-Augmented Generation (RAG) comes to the rescue. Unlike LLMs that rely solely on static training data, RAG supplements the model’s responses with live retrieval mechanisms. By pulling relevant and updated chunks of information from a vector database or external knowledge source, RAG ensures the model is grounded in facts, reducing the chances of hallucinations.

Additionally, RAG improves the specificity and relevance of responses. By directly incorporating task- or domain-specific data into the prompt, RAG can counter the tendency of LLMs to conform to generic patterns. Need a detailed explanation of a new scientific paper or the latest company policies? RAG allows the system to retrieve and integrate those precise details, creating outputs that feel both personalized and contextually accurate. A good RAG pipeline will include clear instructions to only use relevant information that is retrieved and not its own information to prevent it from hallucinating or conforming.

RAG doesn’t eliminate all issues, it’s not a silver bullet, but it addresses critical weaknesses in LLMs, making it an essential tool for applications that demand accuracy, relevance, and recency.

The Secret Sauce: Unpacking the Ingredients of a RAG Pipeline

Step 1: Sourcing the Ingredients: From PDFs to audio

The first step in a RAG pipeline involves gathering and preparing your data sources. This could be anything from company documents, databases, or external datasets that are relevant to your domain and users’ possible queries. From the user’s side, at the start of the RAG process, prompts are crucial. They are the initial queries or statements that users input into the system, which direct the LLM on what information is being sought. However, prompts are more than just questions; they set the stage for the type of information that needs to be retrieved and how the LLM should formulate its response. This stage is akin to entering search terms into Google, but here, the LLM needs to understand not just what you’re asking but how to approach the answer creatively and contextually.

Step 2: Prepping the Ingredients – Parsing, Chunking, and Embedding

Once you’ve gathered your raw data, it’s time to transform it into a format that your RAG pipeline can use effectively. Think of this as cleaning and preparing your ingredients before cooking. Raw data often comes in diverse forms, documents, images, tables, or even PDFs, and not all of it is ready for LLM consumption. This is where data preprocessing begins.

Parsing

Parsing is the first step in preprocessing. Raw data often contains unnecessary elements like page numbers, headers, or irrelevant metadata. These must be cleaned or removed to make the data usable. In some cases, parsing involves converting data into plain text. For instance, an image or scanned PDF might require Optical Character Recognition (OCR) to extract readable text. Tables or structured data might need careful extraction to maintain the relationships between rows and columns. The goal here is to get everything into a neat, consistent format, usually text, that can flow smoothly into the next steps.

Chunking

After parsing, you’ll typically have large documents or datasets. Here’s where chunking comes in, the process of splitting this data into smaller, more digestible pieces. Each chunk is easier to process, retrieve, and use effectively. Chunking isn’t arbitrary. If the chunks are too large, they risk overloading the model, causing it to lose focus on specific details. On the other hand, chunks that are too small may miss out on critical context. Sometimes chunks don’t have to be set in stone, they could be varying in length. Every page or every post could be one chunk.

Finding the right chunk size is like deciding whether to dice vegetables finely or leave them in larger pieces, it depends on the recipe and the tools you’re using. Splitting a document into chunks of 200-300 words often provides a good balance between context and specificity. These smaller chunks allow the LLM to pull out precise and relevant details later during retrieval.

Embeddings

Once the data is neatly chunked, it’s time to embed these chunks into a mathematical format the LLM can work with. This is where embeddings come in, a cornerstone of RAG pipelines. Embeddings are high-dimensional numerical vectors that represent the meaning and context of the text.

Vectors

But what exactly is a vector? A vector is a mathematical object, a list of numbers, that represents data in a multi-dimensional space. In the context of RAG, vectors encode the semantic meaning of words, phrases, or chunks of text. Basically, vectors are a numeric representation of words, with each word or part of the word associated with a vector. For instance, the word “king” might be represented as a vector like [0.1, 0.7, 0.3, …], and “queen” might have a similar vector [0.1, 0.6, 0.4, …]. Their closeness in this multi-dimensional space reflects their semantic similarity. This explanation can give rise to a question, how is an embedding different from a vector? Simply put, an embedding is a specialized type of vector that represents the meaning, context, or features of data (like words, sentences, or images) in a numerical format.

Embedding models, such as BERT, generate these vectors to capture not only the meaning of individual words but also their contextual relationships within a sentence or chunk of text. For instance, an embedding model understands that the phrase “raining cats and dogs” refers to heavy rain rather than literal animals. These embeddings are crucial because they allow us to mathematically compare chunks of text with user queries. This similarity scoring enables a RAG system to identify and retrieve the most relevant pieces of information in response to a user’s prompt.

Example

This image illustrates this concept by visualizing the embeddings of four words in a 2D space. Notice how the model seemingly encodes the concept of gender through the blue arrows. The vectors connecting “king” to “queen” and “husband” to “wife” suggest that the model captures gender relationships as a transformable feature in the vector space, highlighting its ability to represent semantic nuances.

Once the embeddings are generated, they need to be stored somewhere accessible. This is where vector databases come into play. A vector database is a specialized storage system optimized for handling high-dimensional vector data. Think of it as a meticulously organized spice rack, where each jar (embedding) is labelled and stored for quick access. When a query comes in, the database efficiently searches for the closest semantic matches based on numerical representations. The magic of vector databases lies in their ability to pull out the most meaningful data with lightning speed, even from millions of embeddings.

Analysis of Example

This approach marks a leap beyond traditional lexical search, which relies on matching exact keywords in the query to those in documents. While effective for simple tasks, lexical methods often fail to grasp the underlying meaning of the text, like mistaking “car maintenance tips” as unrelated to “vehicle repair advice.” Tools like Apache Solr, a popular implementation of lexical search, excel in speed but struggle with such semantic nuances. In contrast, embeddings capture the context and relationships within data, enabling similarity metrics like cosine similarity to identify relevant chunks even when wording varies. In reality, a combination of both approaches is used depending on the scenario.

At this point, we’ve transformed raw, unstructured data into neatly chunked, semantically meaningful embeddings stored in an optimized database. This step is critical because it enables the RAG pipeline to efficiently retrieve and process the right information later. Without embeddings, the system would struggle to understand semantic relationships between chunks of text, and without vector databases, searching vast datasets would become slow and unwieldy. By parsing, chunking, and embedding the data, we’ve laid the foundation for what comes next: retrieving the most relevant information to augment the LLM’s response.

Retrieval: Finding the Right Ingredients

Imagine you’re cooking a gourmet dish. Retrieval in a RAG pipeline is like searching through your pantry for the freshest ingredients. Sure, you might have shelves full of supplies, but finding the perfect ones, say, ripe tomatoes for a marinara sauce instead of a squishy, sad-looking one, is essential for a standout meal. Similarly, retrieval ensures that the LLM has the best possible data chunks to work with, setting the stage for a great response. A retriever generates relevant chunks for a given prompt or query. 

Once the input documents have been split into smaller chunks and embedded into vector representations, the retrieval process begins. Here’s how it unfolds:

Converting query to vector

When a user submits a query, the system converts it into a vector using the same embedding model applied to the data chunks. This ensures that the query can be compared to the stored chunks in terms of semantic meaning rather than just matching keywords. The query vector is compared with the chunk vectors in the vector database using similarity metrics, such as cosine similarity. This allows the system to identify the chunks most relevant to the query. The system dives into the vector database, scanning for the most relevant pieces of information. To enhance the process, retrieval sometimes combines different methods, like pairing vector search with keyword-based techniques.

Retrieval isn’t just for plain text. It can handle structured data from databases, traverse relationships in graph databases, or even tap into live APIs for the latest information. The versatility of retrieval makes it a core component of the RAG pipeline, capable of adapting to whatever task is at hand. The goal of this process is to provide the system with just the right amount of context to work with. Too little, and the LLM might miss key details; too much, and it risks being overwhelmed by noise. This careful balance ensures the model has everything it needs to craft an informed, accurate response.

Augmentation: Setting the Stage for the Final Dish

Once the retrieval process delivers the finest ingredients (relevant data chunks), augmentation is like mixing those ingredients into a cohesive recipe. This step ensures the retrieved chunks are combined with the user’s original query to provide the LLM with the perfect context, documents and direction to craft its response. Think of it as preparing a mise en place for a chef, laying out all the chopped vegetables, measured spices, and sauces before the cooking begins. This could be something akin to setting a scene. Here’s a relatable manually augmented example prompt, “You are a Teaching assistant, grade all these papers, here are documents that contain the rubrics, here is the question paper, and so on. So you are supplying context and the relevant documents.

Context Window

The challenge here is the context window. Much like a plate has limited space for a meal, the LLM has a fixed amount of information it can handle at once. This means you can’t toss in every single piece of retrieved information. Instead, you need to carefully select and arrange the most relevant parts. The goal is to give the LLM everything it needs without overwhelming it or wasting valuable context space.

Augmentation

Typically, augmentation involves pairing the retrieved chunks with clear instructions, such as: “Answer the question using the provided context only. If the context is unavailable, use your general knowledge. If you don’t know, say so.” This step ensures the AI knows exactly how to use the information it’s given, kind of like providing a recipe card to ensure the chef doesn’t go rogue and add cinnamon to a lasagna.

Interestingly, there are two ways to handle this step. In one variant, the corresponding textual content of the retrieved chunks is provided to the LLM, along with the user query. In another approach, raw embeddings of the retrieved chunks are directly fed into the LLM, along with the embedding of the query itself. While this hasn’t been highlighted in the flowchart, this embedding-based method can be an alternative for augmenting prompts, especially when working with models that can process embeddings directly. It allows the system to bypass the need for textual context and focus purely on the numerical representation of meaning.

Why can’t I just stop at Retrieval?

“Hey Nirmal, so it looks like I have found the relevant documents after the retrieval stage, so why can’t I just do retrieval? Why do I need an LLM ?” Great question! Think of retrieval as Google. You have all the files (or recipes, in our analogy), but it doesn’t necessarily summarize them, weed out irrelevant details, or tailor the answer to your exact needs. An LLM acts as the personal chef who takes all the retrieved ingredients (data) and crafts a dish (response) that’s concise, relevant, and perfectly tailored.

The Magic of LLMs: The Head Chef at Work

Once the context is perfectly prepared, it’s handed off to the Large Language Model (LLM), the master chef of this process. This is where the magic happens. This is also the part most people are now familiar with. Armed with the transformer architecture and the attention mechanism, the LLM identifies the most important parts of the context and combines them with its own knowledge to create a tailored, detailed response. At this stage, the LLM acts like a seasoned chef, weaving together the retrieved context and the user’s original prompt to generate a coherent, insightful, and tailored response. This process is commonly referred to as the “generation” phase, the pièce de résistance of the RAG pipeline.

Transformer Architecture and Attention

The key to an LLM’s ability to weave together sentences and create great text responses exactly how the user would like comes down to two major reasons, the transformer architecture and attention. Transformers are the cornerstone of modern deep learning models and have revolutionised how machines understand and process language. This groundbreaking paper introduced the Transformer architecture, which forms the backbone of modern LLMs like GPT and BERT. At their core, transformers process text as sequences and excel in capturing the relationships between words across these sequences.

Attention

Unlike older methods that read word-by-word or look at limited windows of context, transformers look at all the words in a sentence, or even a document, simultaneously. They focus on how each word relates to others, creating a nuanced understanding of the text. This is achieved through something called attention, a mechanism that helps the model determine which words are important in understanding a given sentence. For instance, a sentence like “my cat sat on the bed, * a bunch of sentences in between * and my dog barked at it. Here, “it” could refer to any of the objects.

A transformer uses attention to weigh the importance of these relationships, assigning higher importance to the more relevant word, like “cat,” ensuring it correctly interprets the sentence. Under the hood, the attention mechanism changes vector values for each of these words, depending on the other words in the document. To put it simply, a transformer is an architecture, a blueprint for how a model processes input data.

The attention mechanism is a key innovation within the transformer architecture. It’s the specific process that enables the model to decide which parts of the input are most relevant to a task, like understanding the relationship between words in a sentence. In our use case, transformers use the attention mechanism to determine which parts of the context are most important for the task at hand. Think of it as the model’s way of asking, “Out of all this information, what should I focus on to craft the best response?” By analysing both the retrieved text and the user’s query, the LLM combines this external context with its pre-trained knowledge, producing a response that’s not just relevant but often feels intuitively accurate.

Adaptability

The beauty of this approach lies in its adaptability. Without requiring extensive fine-tuning, the LLM can dynamically adjust its output based on the information retrieved. This means it can handle queries about niche topics or proprietary data (a major use case for RAG implementations), something it would otherwise struggle with if limited to its training data alone. It’s not just answering; it’s contextualizing, all thanks to the harmony between retrieval, augmenting and generation. This synergy ensures the final output is not just a reflection of the retrieved data but a well-rounded answer crafted with the user’s intent at the forefront.

Although a lot of the above information is a combination of reading and watching videos from multiple sources, 3Blue1Brown’s video on LLMs and Transformers is an excellent visual explanation of the core concept behind an LLM like ChatGPT and a major point of reference for some of the information on this article.

How YOU can cook up a RAG pipeline

As explained earlier, the first step is data preparation and chunking. Gather relevant documents, whether they’re product manuals, research papers, or customer support logs. Split these documents into manageable chunks, short enough for efficient retrieval yet meaningful in context. Tools like LangChain’s TextSplitters make this process straightforward by splitting texts based on tokens or logical boundaries (e.g., paragraphs or sentences). Ensure your chunk size balances granularity and context, as overly small chunks might lose meaning, while large ones may exceed the LLM’s context window. The following image is from LangChain documentation on text splitters.

Embedding Model

Next, feed these chunks into an embedding model to convert them into dense numerical representations. These embeddings capture the meaning and relationships of the text. Popular pre-trained models like OpenAI’s Ada or BERT embeddings can be used here. The resulting vectors are stored in a vector database, such as Pinecone, Weaviate, or FAISS, designed for fast and scalable similarity searches. The following image is from LangChain documentation on embedding models.

When the system receives a query, it embeds the query text into a vector using the same embedding model. This query vector is then compared against stored embeddings using a similarity metric like cosine similarity. The top relevant chunks are retrieved and passed to the augmentation phase. The following image is from LangChain documentation on vector stores.

Next, the retrieved chunks are integrated with the user’s prompt to create an enriched input for the LLM. The following image is from a LangChain tutorial on building RAGs

Tools like LangChain simplify this step, letting you format the augmented prompt with clear instructions on how the retrieved context should be used. The LLM (e.g., GPT-4, Llama 2) takes this augmented prompt and generates a detailed, context-aware response. OpenAI’s GPT4 API is generally easy to use and works well with LangChain. LangChain provides pretty good documentation and tutorials for everything related to RAG.

Training a RAG System: Beyond Implementation

While implementing a RAG pipeline involves setting up retrieval, augmentation, and generation components, training it is a different ballgame. Training focuses on fine-tuning specific components to ensure the system delivers high-quality, contextually accurate responses over time. Here’s where the distinction lies: implementation gets the RAG pipeline running; training optimizes it for your specific use case.

In a RAG pipeline, not all parts need frequent training. For instance, the embedding model used for generating vectors may require periodic fine-tuning to better represent the nuances of your domain especially if it is constantly changing. Similarly, ranking models or cross-encoders used in retrieval (e.g., ColBERT) might benefit from updates based on user feedback or new data. Fine-tuning these components ensures the system remains relevant as your dataset or requirements evolve.

While RAG systems excel without constant retraining of the LLM, there are cases where fine-tuning the model is beneficial, like handling specialised domains (e.g., legal or medical) or tailoring responses to a specific tone. However, fine-tuning an LLM is resource-intensive, so most implementations focus on improving retrieval and augmentation instead. That said, for niche use cases, a hybrid approach with a fine-tuned LLM can elevate performance significantly.

Taste Testing your RAG recipe: Metrics

Like any good dish, your RAG pipeline needs a taste test to ensure it’s delivering the right flavour. Start with retrieval precision, which checks if the system grabs the most relevant chunks. Then, evaluate the response quality with metrics like BLEU or ROUGE, ensuring the final dish (response) is coherent, flavorful, and satisfies the user’s appetite for information. You can read more about Natural Language evaluation metrics here and RAG metrics here, but a balanced combination of automated tests and real-world taste testers (users) ensures your RAG pipeline serves up Michelin-star answers.

Room for improvement: Making RAGs even better

This article does not go deep into the implementation of RAGs. If you’ve been motivated to engineer a RAG pipeline, here is how you can improve it.

Hybrid Retrieval Methods

• Hybrid retrieval methods, like combining vector search with ML methods, or lexical search can enhance retrieval.  

Dynamic Chunk Sizes

• Similarly, dynamic chunk sizes, or deciding the right chunk size can also be a good improvement strategy. 

Graphs

• Exploring graph databases and knowledge graphs can open up new possibilities for retrieval. LangGraph looks like a good tool in RAG pipelines allowing models to reason over relationships and dependencies within data, providing deeper insights. 

Cross-Encoders and Ranking Models

• Incorporating cross-encoders or ranking models like ColBERT into the retrieval pipeline can enhance the relevance of retrieved chunks. Fine-grained ranking ensures that only the most pertinent data is passed to the LLM.

LLM as a Judge

• Leveraging an LLM as a judge to evaluate intermediate and final outputs. For example, during augmentation, an LLM can assess the relevance of retrieved chunks to ensure they provide the best context for the query. Similarly, in the generation phase, an LLM can score responses on their relevance, completeness, understandability and ability to generate answers limited to information being retrieved, enabling continuous refinement of the pipeline.

Combining RAG

• While RAG excels without modifying the base LLM, combining RAG with task-specific fine-tuning can optimize performance. For instance, fine-tuning the embedding model used for retrieval can improve the relevance of vector representations.

Future of RAG

• Looking ahead, RAG systems can evolve into more advanced AI agents capable of agentic workflows. These agents go beyond answering questions, they can take proactive actions, make decisions, and complete tasks autonomously.

Cameron Wolfe’s A Practitioners Guide to Retrieval Augmented Generation (RAG) was the inspiration for the improvements section and is a great place to learn about practical considerations and improvements to your RAG implementations.

RAG vs. Other Methods: Finding the Right Fit

Retrieval-Augmented Generation (RAG) is a practical solution for enhancing Large Language Models (LLMs), but it’s not the only contender in the field. Let’s compare RAG with methods like fine-tuning, training from scratch, and prompt engineering, as each approach comes with its unique trade-offs between cost, complexity, and effectiveness.

Prompt Engineering

Prompt engineering, for instance, is straightforward and cost-effective. It’s about crafting clever instructions that guide the LLM toward the desired output. While this method doesn’t require additional infrastructure or data systems, its potential is constrained by the base model’s capabilities. If your needs go beyond general knowledge or require regularly updated information, prompt engineering may fall short.

Fine-tuning

Fine-tuning takes customization further by adjusting the LLM’s weights to adapt it to specific tasks or domains. This approach enhances expressiveness, allowing models to better handle niche jargon or tasks. However, fine-tuning can be resource-intensive and lacks the real-time adaptability of RAG, especially when new information is needed.

Training from scratch provides unparalleled control but comes at a steep cost in terms of both finances and technical effort. It’s ideal for highly specialized domains where you need complete transparency and control over the data used. However, for most use cases, this level of investment isn’t necessary.

RAG

RAG strikes a middle ground by enabling LLMs to dynamically integrate external data during inference. It avoids the high costs of retraining or fine-tuning while offering the flexibility to work with up-to-date, domain-specific data. Unlike fine-tuning, RAG doesn’t alter the model’s behaviour or biases, its strength lies in its ability to inject context on the fly. However, it does come with the complexity of setting up and maintaining a retrieval pipeline. The right approach is always specific to the problem at hand and the resources available. Sometimes a combination of approaches, like a RAG pipeline, which uses a fine-tuned mode, l could prove to be ideal. This is why experimentation and metrics are extremely important.

Video Explanation

• Learn how to implement RAG (Retrieval Augmented Generation) from scratch, straight from a LangChain software engineer. This Python course teaches you how to use RAG to combine your own custom data with the power of Large Language Models. (Runtime: 2 hrs 33 mins)

References

• “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks” by Patrick Lewis et al. (2020)
• “Attention is All You Need” by Vaswani et al. (2017)
• “Language Models are Few-Shot Learners” by Brown et al. (2020)
• “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” by Devlin et al. (2018)
• “Factual Probing of Knowledge Graphs with Language Models” by Petroni et al. (2019)
• “Beyond Accuracy: Behavioral Testing of NLP Models with CheckList” by Ribeiro et al. (2020)
• LangChain Docs
• LangGraph Docs
• ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• https://youtu.be/wjZofJX0v4M?si=KRh2OlJKkRckXSX8
• https://python.langchain.com/docs/concepts/retrieval/
• https://cameronrwolfe.substack.com/p/a-practitioners-guide-to-retrieval?utm_source=profile&utm_medium=reader2
• https://www.nvidia.com/en-us/glossary/retrieval-augmented-generation/
• https://www.databricks.com/resources/ebook/train-llms-your-data?itm_data=glossary-rag-ad2-jul24
• https://www.databricks.com/blog/LLM-auto-eval-best-practices-RAG

Related Articles

1. End-to-End Implementation of a RAG Model for a Specific Task
2. What is a Vector Database and How is it used for RAG?

Author: Nirmal Prabhu, MS in Computer Science, John Hopkins University