What It Costs to Build an Enterprise AI Assistant

The honest answer is “it depends”, but that is not useful on its own, so this article makes it precise. You can reason about the cost of an enterprise RAG assistant by splitting it into two buckets: one-time build costs and ongoing run costs.

One finding tends to surprise teams. The language model API bill, the number everyone fixates on, is usually not the largest cost. The engineering to build the system properly, and the effort to maintain it, typically dominate. This guide walks through both, with representative figures and a worked example. For the architecture these costs attach to, see the pillar guide, Enterprise RAG: Building AI Knowledge Assistants.

A note before the numbers. Model pricing has fallen steadily and changes often. Treat every figure here as a snapshot taken in September 2026, and verify current provider pricing before you model spend.

Two kinds of cost

Build costs are one-time and mostly labor. Run costs are ongoing and scale with usage. Conflating them is how budgets go wrong. A system cheap to build can be expensive to run at scale, and the reverse is also true.

Build costs: mostly engineering

The one-time cost of a RAG assistant is dominated by skilled engineering time, not infrastructure. The major line items:

• Discovery and scoping. Defining the use case, the corpus, the users, and the reliability bar.
• Data ingestion and preparation. Frequently the largest build effort. Enterprise documents are messy, live across systems, and need cleaning, parsing, and structuring before indexing. Underestimating this is the most common planning error.
• Chunking and initial indexing. Splitting the corpus and generating the initial embeddings.
• The retrieval and generation application. Hybrid search, reranking, grounding, and citation handling.
• An evaluation harness. Not optional for production, and covered in reducing hallucinations in enterprise RAG systems.
• Security and access-control integration. Permission-aware retrieval and the controls in enterprise RAG security and data governance are build-time work, not an afterthought.
• Interface and integration into the tools users already work in.

Because this bucket is labor, its cost varies enormously with team, scope, and geography, so a single dollar figure would mislead. The useful takeaway is structural. Data preparation, retrieval quality, and security are where the build hours go, and they are what separate a durable system from a demo.

Run costs: the per-query economics

Ongoing cost is driven mostly by one thing, the language model calls, plus several smaller components.

Language model generation (the dominant variable)

Every answer costs input tokens plus output tokens. The input is your system prompt, the retrieved context, and the user’s question. The output is the generated answer. Providers bill per million tokens, separately for input and output.

As of September 2026, representative Anthropic rates run from $1 input and $5 output per million tokens for a small fast model, Claude Haiku 4.5. A mid-tier model, Claude Sonnet, is $3 input and $15 output. A frontier model, Claude Opus, is $5 input and $25 output [1]. Other major providers offer comparable tiers. Prompt caching cuts the cost of cached input by around 90 percent, and batch processing halves token costs for non-interactive work [1][2].

A worked example makes the scale concrete. Suppose a context-heavy query sends about 4,000 input tokens: the system prompt, five retrieved chunks, and the question. It generates 500 output tokens.

• On a mid-tier model at $3 and $15, that is roughly $0.012 in input and $0.0075 in output, about 2 cents per query.
• On a small model at $1 and $5, it is under 1 cent per query.
• At 100,000 queries per month, the mid-tier case is on the order of $2,000 per month, and the small-model case closer to $650, before caching. With prompt caching applied to the stable parts of the prompt, the input portion drops sharply [1].

The point is not the exact figures, which will change. It is the structure. Cost is query volume times tokens per query times model rate, and every one of those three is a lever.

Cost per answer, by model tier

• Small (Haiku 4.5) 0.65¢
• Mid-tier (Sonnet) 1.95¢
• Frontier (Opus) 3.25¢

Same query on every tier: 4,000 input tokens and 500 output tokens, before caching.

The smaller components

• Embeddings. Cheap per token, incurred once for the initial corpus, then for updates and for embedding each query. Small relative to generation for most workloads.
• Vector database. A hosting cost that scales with corpus size and query volume. Managed services price by storage and usage.
• Reranking. A per-query cost that improves quality for a modest amount, discussed as a lever below.
• Infrastructure, monitoring, and logging. Ongoing operational overhead.
• Maintenance. Re-indexing as content changes, keeping the evaluation set current, adapting to model updates, and human oversight. This is recurring engineering, and it is easy to forget in a first budget.

Check current provider pricing for each of these components, as rates vary by vendor and change frequently.

The levers that control spend

Ongoing cost is highly controllable. In rough order of impact:

Right-size the model. Model choice is the single biggest lever on the bill. Route simple, high-volume queries to a small model, and reserve a frontier model for the genuinely hard ones. The price gap between tiers is several-fold, so matching model to task difficulty cuts spend dramatically [1].

Use prompt caching. The system prompt and any reused context are the same on every call. Caching them means paying full price once, then a fraction on subsequent reads. That is around a 90 percent reduction on the cached input [1][2]. For an assistant with a consistent system prompt and repeated context, this is one of the highest-return optimizations available.

Keep the context tight. Retrieving 20 chunks when 5 would do inflates input tokens on every single query. It also hurts accuracy, since models use information less reliably when it is buried in a long context [5]. Reranking to pass a small, high-quality set of chunks lowers cost and improves answers at the same time. Cost control and reliability point the same way here.

Batch non-interactive work. For anything that does not need a real-time response, batch processing halves token cost [1].

Skip retrieval for small corpora. If the knowledge base fits in the context window, on the order of a few hundred pages, place it directly in the prompt and rely on caching. That avoids the cost and complexity of a retrieval pipeline entirely [2]. RAG earns its infrastructure when the corpus is too large to fit, or changes constantly.

The RAG-versus-fine-tuning cost question

Teams sometimes assume fine-tuning a model on their data will be cheaper than running retrieval. For a body of knowledge that changes, it usually is not. Fine-tuning adds the cost of generating training data, the compute to train, and, critically, the cost of retraining every time the underlying knowledge changes [3]. The research also finds that models absorb new facts poorly through standard fine-tuning, so the spend does not even buy reliable recall [4]. RAG shifts cost into retrieval infrastructure and per-query context tokens, but it updates by changing the index rather than retraining. For evolving enterprise knowledge, retrieval typically has the lower total cost of ownership, and the fuller comparison is in RAG vs fine-tuning for enterprise knowledge.

The bottom line

For most enterprise assistants, the language model bill is manageable and, with the levers above, highly controllable. The real cost centers are elsewhere. There is the one-time engineering to build the system properly, especially data preparation, retrieval quality, and security. Then there is the ongoing maintenance and oversight to keep it accurate and safe. Budget for those, not just for tokens.

And keep the stakes in proportion. The cost of a hallucinated answer or a data leak reaching a customer dwarfs any savings from shaving the token bill. Spend on getting reliability and security right, and optimize the API cost with routing, caching, and tight retrieval. That ordering is what keeps an enterprise AI assistant both affordable and worth trusting.

---

Trying to budget an AI knowledge assistant for your organization? The build cost depends on your corpus and your reliability and security requirements far more than on token rates.

---

Sources

1. Anthropic. Pricing (accessed September 2026; figures are point-in-time and subject to change). https://www.anthropic.com/pricing
2. Anthropic (2024). Introducing Contextual Retrieval (prompt caching cost and latency, and the long-context alternative to RAG). https://www.anthropic.com/engineering/contextual-retrieval
3. Balaguer, A., Benara, V., de Freitas Cunha, R., et al. (2024). RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculture. arXiv:2401.08406. https://arxiv.org/abs/2401.08406
4. Ovadia, O., Brief, M., Mishaeli, M., & Elisha, O. (2024). Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs. EMNLP 2024. https://arxiv.org/abs/2312.05934
5. Liu, N. F., Lin, K., Hewitt, J., et al. (2024). Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics, 12:157-173. https://arxiv.org/abs/2307.03172