RAG in production: what the CTO office should own vs delegate

RAG is indexing, security, freshness, evaluation, and cost with a language model attached. CX owns corpus meaning and escalation; the CTO office owns how data enters the index and who can query what. When those two ownerships blur, every wrong answer becomes a meeting instead of a ticket.

Why the demo is not the program

A demo is a single happy path over a frozen corpus, asked by the person who built it. Production is an open query distribution over a corpus that changes without telling you, asked by people who do not know what the system cannot answer. Those are different engineering problems, and the second one is mostly not about the model.

This is why naming the failing stage matters more than upgrading the model. If tables in your source PDFs are flattened into unstructured strings at ingestion, no model recovers the column relationships later. If your retriever returns semantically adjacent documents but misses the one page with the exact term, the model will write a fluent answer from the wrong evidence. Each of these lives at a different stage, has a different owner, and needs a different fix.

What the CTO office keeps

The split is not political, it follows from who can be paged. The CTO office owns everything with a runbook and an alert. CX owns everything that requires a judgement about what a good answer means for a customer. Evaluation is the one row that splits: engineering owns the harness that runs it, the business owns the ground truth that defines a correct answer. Hand the ground truth to engineering and you get a system that scores well on questions nobody asks.

Access is the boundary you cannot delegate

The OWASP Top 10 for LLM Applications 2026 lists vector and embedding weaknesses as LLM09, promoted from LLM08 in the 2025 edition with a widened scope. Read the entry as an operating warning rather than a security checklist: the retrieval layer is part of your trust boundary, and similarity search decides what the model is permitted to know.

The practical consequence for a CTO office is that authorization has to be enforced before retrieval, on the server, using the caller’s identity, and never as a filter the client can influence. Tenants and cohorts get isolated indexes rather than shared indexes with a metadata predicate. Raw similarity scores do not get returned to callers, because a score is a membership oracle. These are architecture decisions, not configuration, which is why they cannot travel to a vendor in a statement of work.

Freshness, deletion, and silent degradation

Stale indexes are the most common production complaint and the least dramatic failure. Nothing breaks. Latency stays inside the SLA, error rates stay at zero, and the system keeps answering questions using a policy that was superseded two quarters ago. Deletion is the sharper version of the same problem: a document removed at source is not removed from your index unless something makes that happen, so your retriever can serve content the business believes it has retracted.

Freshness therefore needs the same treatment as any other production dependency, which means named alerts rather than good intentions.

• Index age per corpus, with a threshold the business agreed to and an owner who is paged when it is crossed.
• Reindex job success and duration, because a job that silently stops succeeding looks identical to a fresh index.
• Deletion propagation lag from source system to index, measured, not assumed.
• Retrieval quality on a fixed evaluation set, run on a schedule, so degradation is visible before users report it.

Evaluation is a standing asset, not a launch gate

Most teams build an evaluation set once, use it to justify the launch decision, and never run it again. That inverts its value. A fixed set of questions with known correct sources is the only instrument that detects quality decay caused by a growing corpus, a shifting query vocabulary, or a dependency upgrade that quietly changed embedding behaviour. Without it, your first signal is a customer complaint, and by then you are debugging backwards through a system with no baseline.

Cost governance belongs to the same owner as the index

RAG cost is not the token bill, it is the product of corpus size, refresh frequency, retrieval depth, and reranking. Each of those is a decision somebody made, usually without seeing the others. Reranking the top twenty candidates instead of the top three improves answers and multiplies cost per query. Connecting a fifth source triples embedding spend and rarely triples value. Put those levers in one place with the person who owns the index, and cost becomes a design conversation instead of a quarterly surprise.

Where a vendor boundary is legitimate

Managed retrieval infrastructure is a reasonable purchase. Managed ownership is not. The test is whether your team can run an incident on that layer at 2am without the vendor, and whether access boundaries, observability, and data export are contractual rather than promised on a call. NIST AI 600-1, the generative AI profile of the AI Risk Management Framework, is useful here precisely because it organizes obligations around govern, map, measure, and manage rather than around features. Read your vendor boundary against those four verbs and the gaps become obvious.

Inherit VoC governance tiers rather than inventing a parallel scheme for retrieval. Pilot workflow on one corpus using the same 90-day discipline as any CX-led program, and keep the build versus buy review honest about which layer you intend to operate.