What building a RAG system taught me about shipping AI that actually works

Over the last couple of weeks I built a RAG system from scratch. The result is NutriClaim {Lab}, a health and nutrition claim verification tool that validates claims against peer-reviewed research from PubMed. It’s in beta, functional, and almost production-ready. Building it taught me things that no tutorial bothered to mention.

In this post I want to share the lessons that matter most, especially if you’re a founder or a senior leader evaluating whether RAG is the right bet for your business.

Before diving into the learnings, I’ll briefly cover when RAG makes sense and where it applies. Beyond health and nutrition, into internal knowledge bases, customer service, analytics, and more. If you already know what RAG is and just want the hard-earned lessons, feel free to jump straight to that section.

When to use a RAG system

Let’s start with the basics. A RAG system Retrieves relevant information, uses it to Augment the original prompt, and Generates a more accurate, grounded response. In plain English: instead of relying solely on what an LLM already knows, you plug it into a data source (a vector database, a document store, a knowledge base) and let it pull in the right context at query time.

What does that actually get you?

Access to private or proprietary knowledge. The LLM combines its reasoning with your data, data it would otherwise never see.
Fewer hallucinations. Give the model specific, relevant information and it’s far less likely to make things up.
Fresh information. You can feed it live or recently updated data without retraining anything.
An updatable knowledge base. With fine-tuning, new information means retraining the model. With RAG, you just update the data source.
Controlled tone, format, and references. Because you’re modifying the prompt, you can instruct the model to cite sources, adopt your brand voice, or respond in a specific format. This is one of the most underrated capabilities of the whole approach.

Where does it make the most sense?

One of my favorite applications, and the one I’ll be focusing on heavily over the coming months, is Analytics Copilots. Commercial LLMs don’t have access to your business data, nor should they. With a RAG layer, you can build a system that queries your actual numbers, asks follow-up questions, and surfaces insights grounded in reality. I’ve seen this working in tools like Dot, a Slack-based agent that answers questions, performs analyses, and generates reports automatically.

For eCommerce, product discovery systems are a compelling use case. With access to a retailer’s product database, enriched with customer service history, internal knowledge, and reviews, an LLM can tailor recommendations to individual customers. Amazon’s Rufus is probably the most visible example.

Customer support automation is now table stakes for many businesses. With the right FAQs, past tickets, communication guidelines, and a solid knowledge base, a well-built RAG can save thousands of hours and significant money, without the unpredictability of a generic LLM running loose.

According to a McKinsey report on knowledge-worker productivity, employees spend an average of 1.8 hours a day searching for information. That’s over 22% of their time. A RAG-powered internal knowledge assistant can reduce that substantially, at the cost of a few thousand dollars a year. Moody’s did exactly this, connecting its proprietary financial research and ratings data to build a copilot for financial analysts that generates up-to-date analyses grounded in its own research body.

RAG also shines for document intelligence: making sense of large volumes of unstructured documents like PDFs, reports, contracts, manuals, and emails. Legal research, scientific research, compliance. Anything where the answer lives somewhere inside a mountain of text.

A simple filter

If you’re evaluating RAG for your business, ask yourself whether one or more of these apply:

The information is private or proprietary.
Accuracy is important.
The knowledge changes frequently.
The knowledge base is wide, varied, and/or complex.

“Do you ship to Alaska?” Accurate, proprietary, no one else has it. “What campaigns are driving the most revenue this month?” Grounded in real data that changes daily. In NutriClaim {Lab}, the data spans over four thousand documents with varied formats and content: complex, frequently updated, accuracy-critical. You get the idea.

Things I learned

Now to the meat of the article. Here are the lessons that actually moved the needle.

Chunking is the secret sauce

After finishing the first version of the RAG pipeline, I ran to try the tool. The results were terrible. Takeaways made no sense, linked documents were few and usually unrelated, and most of the time no documents survived the reranking step at all, even for topics where there clearly should have been.

I set a team of AI debugging agents to work, and they identified that the chunking strategy was broken. I had chosen a semantic similarity approach that groups sentences together for as long as they’re semantically close, splitting into a new chunk when the distance exceeds a threshold. I chose it because I wanted each chunk to be topically coherent and self-contained. Naively, I assumed it would produce logical chunks. It didn’t.

On closer examination, chunks were everywhere. Reference sections at the end of papers, containing nothing but citations, accounted for around 15% of all chunks. There were also floating connecting sentences with no standalone meaning, tables split into disconnected parts (title, body, and caption as separate chunks), and orphaned paragraphs that had been cut off due to text length limits.

The fix: exclude the noise, group chunks logically, and add overlap splits for text that was too long. That single change moved the system from 20% ready to 80% ready.

The lesson is simple, and it’s one I’ll repeat for every RAG project I work on: no matter how sophisticated the LLM, if the data going in is poor, the answers coming out will be worse. RAG’s core job is to supply relevant, accurate context. If it fails at that, nothing else matters.

If a vendor pitches you a RAG solution without a rigorous data preparation and chunking strategy, that’s a red flag.

You don’t always need the most advanced model

NutriClaim {Lab}‘s pipeline uses several LLMs at different steps: one to categorize papers by type (RCT, Meta Analysis, Systematic Review, and so on), one to expand queries to match the corpus, one to extract and summarize relevant evidence, and a final one to synthesize the answer.

Initially I used Claude Sonnet for everything. Response times were slow, and I was burning through API credits fast.

So I started treating each step as its own optimization problem. What does this step actually require? Speed? Reliability? Quality? And what’s it worth paying for?

My first move was switching to Gemma 3 for all steps except synthesis. Cheaper, and built into Cloudflare Workers AI. One fewer call to Anthropic’s API. The result: papers were being classified as “Other” across the board, and a single query was taking over 30 seconds to return.

Back to the drawing board. After benchmarking Qwen3, Llama, Gemma 3, and DeepSeek, Llama came out on top for classification. Not the fastest, but way cheaper than any Anthropic model and 100% accurate on paper classification. Since classification only runs on ingestion and can be parallelized, it was the right call.

For evidence extraction, summarizing the relevant bits from the top 12 papers before feeding them to the synthesizer, I needed something fast, reliable, and JSON-consistent. I ran 11 test queries through each candidate model and used DeepSeek to evaluate the outputs. Qwen3 was the winner: second fastest overall, 99% cheaper than Sonnet, highly reliable JSON output, and native to the Workers AI ecosystem. Running the 12 extractions in parallel brought that step down to around three seconds.

For the final synthesis step, I started with Sonnet. Excellent quality, but three times more expensive than the next option and up to 80 times more than the cheapest. After another round of comparisons (ruling out DeepSeek after it clocked in at 18 seconds per query, Mistral and GPT-OSS for JSON unreliability, and Llama 70B Fast for being anything but), I was left with Qwen3, Haiku 4.5, and Sonnet 4.6. Qwen3’s quality wasn’t quite there. I ended up trading 0.18 points in quality for a 60% reduction in response time and a 70% reduction in cost. Haiku 4.5 for the win.

The broader point: choosing the right model for each step is as much an art as it is a science. It depends on accuracy requirements, latency tolerance, cost constraints, and the format of the output. Getting it right requires testing, not assumption.

The deceivingly long distance between working and finished

With all the AI development tooling available today, you can have a working prototype in an hour or two. Maybe less. And it will look impressive. The search bar works, the page looks good, there are animations.

But that’s not a product. That’s a demo.

AI has made the gap between idea and prototype almost disappear. It’s also made iteration much faster: “add an animation to the pop-up”, “collapse that menu into a hamburger on mobile.” So you can move from working prototype to refined working prototype quickly. This creates a dangerous illusion that you’re nearly done.

You’re not.

NutriClaim {Lab} is a good example. After two hours, I could show it to friends and family and they were genuinely impressed. But if any of them had actually tried to validate a claim, they would have found it was unusable. All claims returned “Not enough evidence”, it responded to questions completely outside its domain, and the referenced evidence was meaningless.

Most of my time went into making the chunking reliable, getting classification right, making the pipeline efficient, bringing response time down to something acceptable, choosing the right model for each step, and making sure the system wasn’t saying anything that would embarrass me or misrepresent the science. That is a lot of work, and none of it is visible from the outside.

NutriClaim {Lab} is in beta, and I’m deliberate about that label. It covers 39 topics across 12 categories, solid coverage for a system built on a curated set of peer-reviewed papers, but still a slice of the full nutrition and health landscape. Expanding the knowledge base is an ongoing process, not a gap in the system. Does it work? Yes. Is it reliable within its scope? Yes. Would I expand it to cover every health claim imaginable without more data? No, and that’s the point.

The same principle applies to any RAG deployment: it has to be reviewed, validated, and stress-tested before it goes anywhere near real users. The last thing you want is a RAG-powered customer service bot running wild on your site without being confident it’s actually helping. If you’re not sure it will improve the experience, you’re better off sticking with humans for now.

Beware of the costs

I was initially using Sonnet 4.6 for everything. A dollar covered testing, ingestion, and several rounds of trying the app in development. Now imagine releasing that to a site with thousands of visitors a day.

Each query in a production RAG system can cost anywhere between $0.0003 and $0.05, depending on the models involved. That range matters enormously at scale. Take an eCommerce site with 100,000 daily visits, 3% of users engaging with a chat, and three questions each. That’s 9,000 requests a day, anywhere from $3 to $450, daily.

If your average revenue per visit is $3, your margin is 10%, and you chose the premium option without thinking about it, that RAG eats 50% of your profit. Unless it moves the conversion needle significantly, you’ve built something that costs more than it returns.

This has to be planned, modeled, and tested before you go live. Choosing the right architecture and models isn’t just a technical decision. It’s a commercial one.

Execution time adds up

RAG adds steps to every request, and every step takes time.

Here’s where the time goes in NutriClaim {Lab}:

Step	Average time (ms)	Part of RAG
Topic Classification	1,544	Yes
Query Expansion	121	Yes
Evidence Extraction	6,506	Yes
Reranking	828	Yes
Vector Retrieval	1,172	Yes
Final Synthesis	5,764	No

Of the 15,935ms total, 10,171ms (nearly two thirds) comes from the RAG pipeline itself. And that’s after several rounds of optimization: faster models, parallelization, trimming steps that weren’t earning their time.

For a claim verification tool, 16 seconds is on the edge of acceptable. For a customer support chat, it would be disqualifying.

When you’re designing a RAG system, response time isn’t an afterthought. It shapes what the system can and can’t be used for. Most RAG components involve trade-offs between speed, cost, and quality, and there’s no universal formula. You need to understand those trade-offs, run the numbers, and make deliberate choices based on what your specific use case actually demands.

Is RAG the right move for your business?

Building NutriClaim {Lab} was an exercise in humility as much as it was a technical one. The technology works, and when it’s done right, it can be genuinely transformative. But “done right” takes more than spinning up a vector database and pointing an LLM at it. It takes careful data preparation, deliberate model selection, cost modeling, and enough patience to stress-test it before anyone else sees it.

If you’re a founder or a senior leader thinking seriously about where RAG fits in your business, I’d love to have that conversation. Not a sales pitch. Just an honest 30 minutes to talk through your data, your use case, and whether this is the right investment right now. Book a free call and let’s figure it out together.