What production RAG actually cost us

The first question about generative AI is often which model to use. A more useful question is where the money and engineering effort go once the system reaches production. At Chantelle, we ran retrieval-augmented generation first on a hosted API and then on a model we fine-tuned and hosted ourselves. In both cases, the model was the smallest part of the overall cost.

Here is the short answer, because it is the whole point: the dominant costs of production RAG are ingestion, data quality, evaluation, and integration. The model, whether a metered API call or a self-hosted fine-tune, is the smallest recurring line. The model was the cheapest line right up until we chose to own it, and even then the real cost was never the model.

Where does the money actually go in production RAG?

Into getting documents in, keeping them clean, evaluating the answers, and wiring the thing into the systems people already use. Raw model inference is usually the smallest recurring cost, not the largest.

A demo makes the model look like the product. In production, it is one component in a larger system of ingestion, evaluation and integrations. Getting documents in was the real project: source formats are inconsistent, content is duplicated, stale or contradictory, and engineers spend most of their time deciding what to ingest, how to split it and how to keep it current. The integrations were the second major cost. A retrieval system that cannot reach the tools people use is still an experiment, so orchestrating flows in n8n, connecting Magento and handling failures and retries accounted for much of the work.

We started on a hosted API, and the model was the cheapest line

In 2025 the stack was deliberately boring: PostgreSQL with pgvector for retrieval, Gemini for generation, n8n to orchestrate ingestion and workflows, Valkey for caching. Inference was a metered API call. You can forecast it, cap it, and cache around it, and we put Valkey in front of the workload so we were not paying to re-answer the same questions. It flattened the bill and sped up responses at the same time.

Set that against ingestion and data quality, which are human costs, recurring, and much harder to cap. An engineer maintaining an ingestion pipeline does not get cheaper the way an API call does. If you are budgeting an AI project and the model line is the big one, you have either not shipped yet or you are measuring the wrong thing.

So why build and host our own model?

Not because the API was expensive. It was cheap. We moved to our own fine-tuned Gemma for three reasons that the API could not give us. Quality: a model fine-tuned on our own ideal task examples beat the generic endpoint on our actual questions. Control: we owned the latency, the versioning, and we were not exposed to someone else’s model changing under us. And, honestly, to prove what was possible in-house, which matters when you are trying to move an organisation.

Picking the model was its own small exercise in not chasing size. I benchmarked Gemma against Qwen and against larger Gemmas on the job we actually had, navigating multi-language e-commerce sites, and the 12B won: stronger multilingual behaviour than Qwen, where the bigger models were not worth their price for our workload. Size is the easiest thing to over-buy in this field.

Owning a model is not free. You trade a metered API call for a GPU bill and the work of serving it, in our case fine-tuning with QLoRA and serving with vLLM on a spot-instance fleet. On its own, for one assistant, that trade is hard to justify. What justified it was that we stopped treating the model as one app’s dependency and started treating it as a platform.

Owning the model turned it into a platform

Once the fine-tuned Gemma was ours, the assistant was only the first thing we ran on it. We pointed Playwright agents at it to walk the critical user-journey paths across all our production sites, the way a customer would, and tell us when something broke. And when a ticket moved into QA in ClickUp, an n8n workflow had the model draft the test case and Playwright run it against staging, then posted the results back onto the ticket as a comment, what it did and what it found, before a human tester ever picked it up. One model we owned, several workloads, all sharing the same GPU fleet we had already paid for.

Sharing one model across several workloads changes the cost comparison. A metered API remains priced per request, while a self-hosted model can use the same infrastructure for additional workloads. For one chatbot, self-hosting would probably not have been worthwhile. For a shared platform used by several teams, the economics were more favourable.

What actually made it work?

Retrieval quality and honest evaluation, not prompt cleverness. This did not change when the model did.

A mediocre prompt over excellent retrieval beats a brilliant prompt over mediocre retrieval, every time. If the system pulls the right passages, the model does its job; if it pulls the wrong ones, no amount of prompt engineering saves you, and you have built a confident machine for being wrong. Evaluation was the other half. We measured retrieval and answer quality against a test set instead of trusting the vibe of a good demo, and we ran that evaluation in CI so a change that made answers worse failed the build. Vibes are how AI projects get greenlit and how they quietly fail six months later.

So what did it really cost?

Even self-hosted, the model was not the expensive part. We ran the fine-tuned Gemma 12B around the clock for about €150 a month: spot instances only, in an AWS region with deep G4, G5 and G6 capacity, with a Packer image built to run on any of those GPU generations so we almost always caught cheap spot capacity, and multi-token prediction to speed up generation. €150 a month, for a model that served the assistant and the test agents alike. The expensive part was still ingestion, data quality, evaluation, and integration, plus the engineering around them. Owning the model added a small, predictable GPU bill and removed the per-call cost, and it paid off because we ran more than one thing on it. The line everyone obsesses over stayed the small one in both regimes.

If you are a founder or an engineering leader being asked to “do something with AI,” budget your effort and your money where they actually go. Assume the model is the cheapest part. Put your people on ingestion, data quality, evaluation, and integration, because that is what decides whether you ship something dependable or something that only shines in a demo. And if you do decide to own a model, do it because you have more than one use for it, because a model you own is a platform and a model you rent is a feature.

The practical lesson is to budget for the whole system, not just inference.