I've lost count of how many "emergency consulting" calls start the same way: "We hooked up a RAG pipeline to our data, it worked great in testing, and now it's confidently making things up in front of customers." Sound familiar?
The problem isn't the technology—it's that we've collectively underestimated what it takes to go from a cool demo to a reliable production system. Let me walk you through the landmines I see teams step on most often.
RAG Is Not a Magic Wand
Retrieval-Augmented Generation has become the default answer to "how do we make an LLM work with our data?" And honestly, the concept is sound—retrieve relevant context, feed it to the model, get grounded responses. Simple, right?
Not even close.
Last quarter, I worked with a legal tech company that had built what looked like a solid RAG system over their contract database. In demos, it nailed every question. In production, it started mixing clauses from different contracts into single responses—because their chunking strategy was splitting documents at arbitrary character limits instead of semantic boundaries. Their users were getting legally incorrect answers that sounded perfectly authoritative.
The RAG Pitfalls Nobody Warns You About
- Garbage in, garbage out (but fancier): If your data is messy, RAG doesn't clean it up—it just retrieves messy context and the LLM polishes it into convincing-sounding nonsense
- Chunking matters more than you think: Bad chunking strategy = retrieved context that's missing critical information or blending unrelated content
- Retrieval relevance ≠ answer quality: Your vector search might return the "right" documents but the wrong passages within them
- Stale data is dangerous data: If your source data updates frequently, your embeddings need to keep up—and most teams don't build that pipeline
The fix isn't to abandon RAG. It's to treat it as one component in a larger system that includes data curation, intelligent chunking, re-ranking, and—critically—output validation. Building a RAG pipeline takes a week. Building one that works reliably in production takes considerably longer, and it demands serious attention to your data quality, your retrieval strategy, and your vector database architecture.
Your AI Guidelines Gap Is Bigger Than You Think
Here's a conversation I had last month with a VP of Engineering: "We've got AI guidelines." Great, let me see them. "Well, they're mostly around not using ChatGPT for code reviews." That's... not what I meant.
When you're deploying LLMs as part of your product or internal operations, you need governance that actually covers the hard questions. And the hardest one I see teams dodge is intellectual property.
Think about what happens when you send proprietary data to a third-party API for inference. Even if the provider promises they don't train on your data (and most reputable ones don't), your data still travels over a network, sits in memory on someone else's infrastructure, and gets logged somewhere. For a lot of industries, that's already a compliance problem.
What Real AI Governance Looks Like
- Data flow mapping: Know exactly what data touches your LLM, where it goes, and who can access it at every step
- IP classification: Before anything goes near a model, classify it—public, internal, confidential, restricted. Then enforce those boundaries
- Vendor contracts with teeth: Generic ToS isn't enough. Get specific about data retention, training exclusions, and audit rights
- Bias and fairness audits: Especially if your LLM touches customer-facing decisions—lending, hiring, support prioritization
This isn't about creating a 200-page policy document that nobody reads. It's about having clear, enforceable rules that your team actually follows. The companies that get this right tend to start small—pick the three riskiest things and nail those first.
Model Updates Will Break Your Production System
This one burns people badly, and it happens more than you'd expect.
A client of mine had a customer support chatbot running on GPT-4. It was performing beautifully—high satisfaction scores, low escalation rates. Then OpenAI pushed a model update. Nothing dramatic, just a routine version bump. Within 48 hours, their escalation rate doubled. The model had subtly changed how it handled ambiguous requests, and their carefully tuned prompts were now producing wishy-washy non-answers instead of confident resolutions.
This is the dirty secret of building on top of third-party models: you don't control the foundation. A model provider optimizing for one set of benchmarks can quietly degrade performance on your specific use case.
The Model Volatility Problem
- Response patterns can shift between model versions, breaking downstream logic
- New safety filters might block legitimate use cases you depend on
- Performance improvements on general benchmarks don't guarantee improvements on your domain
- Pricing and rate limits can change, blowing up your unit economics overnight
The teams that survive this have one thing in common: they treat model changes like they treat code deployments. Which brings me to the most important thing most teams aren't doing.
You Need an EVAL Framework (Yesterday)
If you're deploying LLMs without a structured evaluation framework, you're flying blind. Full stop. And no, "we test it manually before shipping" doesn't count.
A proper EVAL framework is what lets you confidently answer the question: "If we switch from Model A to Model B, will our users notice?" Without it, every model change is a gamble.
The Four Pillars of LLM Evaluation
- Benchmarking: Compare models on your actual use cases, not generic leaderboards. Build domain-specific test sets that reflect the real queries your system handles—including the weird edge cases
- Red teaming: Actively try to break your system. Prompt injection, adversarial inputs, edge cases that exploit model weaknesses. If you're not doing this, someone else will
- Human-in-the-loop review: Automated metrics catch the obvious problems. Subject matter experts catch the subtle ones—the responses that are technically correct but practically useless
- Automated CI/CD for models: Treat model swaps like code deployments. Run your eval suite automatically, gate releases on quality thresholds, and have a rollback plan
Accuracy
Does the model get the right answer on your domain-specific test cases?
Consistency
Does it give stable answers to semantically equivalent questions?
Latency
Does it meet your response time requirements under real load?
Cost
What's the per-query cost, and does your unit economics still work?
Building this takes effort upfront, but it pays for itself the first time a model update would have silently degraded your product. Instead of discovering problems through angry customer tickets, you catch them in your pipeline.
The Long Game: Private LLMs
I'm going to say something that might be controversial: if your business depends heavily on LLMs, you should be building toward running your own models. Not tomorrow—but it should be on your roadmap.
Here's why. Every time I see a company fully dependent on a single provider's API, I see a business with a single point of failure they don't control. Pricing changes, rate limits, model deprecations, policy updates—any of these can disrupt your operations, and you'll find out about them in a blog post, not a planning meeting.
The Hybrid Approach That Actually Works
The smartest teams I work with aren't going all-in on either side. They're running a hybrid strategy:
A Practical Path to Private LLMs
- Start with APIs for prototyping. Use commercial models to validate your use case fast and cheap. Don't over-invest in infrastructure before you've proven the value
- Build your eval framework early. This is what makes model-switching possible later. It's much harder to retrofit than to build from the start
- Identify your high-value, high-sensitivity workloads. These are the candidates for private deployment first—where the IP risk or cost justifies the infrastructure investment
- Start with capable open-source foundations. Models like Llama, Mistral, or DeepSeek give you a strong starting point. Fine-tune on your domain data to close the gap with commercial models
- Invest in inference infrastructure gradually. You don't need a GPU cluster on day one. Start with cloud GPU instances, then move to dedicated hardware as your workload justifies it
The goal isn't to replace commercial APIs entirely. It's to have options. When your critical workloads run on models you control, you sleep better at night—and you negotiate better rates with API providers when they know you can walk away.
The Bottom Line
Deploying LLMs in production is genuinely hard, and that's okay. The companies that succeed aren't the ones with the fanciest models—they're the ones that treat deployment as an engineering discipline, not a science experiment.
Get your data pipeline right before you worry about which model to use. Build governance that your team will actually follow. Invest in evaluation infrastructure that catches problems before your users do. And start building toward a future where you're not entirely dependent on someone else's API.
None of this is glamorous. But the gap between a cool demo and a reliable production system? That's where the real value is created.