Beyond the chatbot: a multi-tenant RAG assistant where the guarantees live in the architecture

Software architecture · AWS · Django · FastAPI · LangGraph · Celery · pgvector

Building a chatbot that answers from a handful of PDFs takes an afternoon. Building a platform on which many organizations upload large volumes of documents and query them from the web app, from a widget on their own website, from a messaging app and from an API is a different job altogether. There, the hard questions are not about the language model: what prevents one customer’s documents from ending up in the answer given to another, what prevents paying twice for the same model call, what happens to a long indexing job when a new version is released.

For a technology services company we designed and built the architecture of a platform of this kind, offered to its customers as a service. Each organization works in one or more knowledge spaces, each with its own documents, permissions, and model and retrieval configuration. We followed a single principle: in a multi-tenant AI product, isolation, cost and service continuity must be properties of the architecture, not precautions entrusted to each individual feature. A guarantee that depends on someone remembering a filter will, sooner or later, fail.

Two services, but a single owner for identity and schema

In the early versions everything lived in a single Django backend, including the conversational pipeline and indexing. It is the right choice at the start, and Django remains, for us, the best-suited tool for a domain rich in models and permissions. As the product grew, however, the two halves started pulling in opposite directions: platform governance consists of short, transactional requests, while the AI side keeps requests open for the whole duration of the stream, spends its time in calls to model providers and changes dependencies at a much faster pace. We split them into two services.

Splitting into two services is the easy part. The real risk is splitting the authority too: two services that each decide who the user is, or that each modify the schema, will sooner or later diverge, and the divergence is discovered in production. That is why the two services share the same PostgreSQL, but authority stays with the Control Plane.

The schema belongs to the Django migrations: the AI Runtime accesses the database through asynchronous SQLAlchemy models regenerated from the schema after every migration, not written by hand, and shared enumerations live in a package imported by both. Identity, too, has a single owner. When a request reaches the Runtime, from any channel, the Runtime signs a very short-lived service token, with an asymmetric key, containing the user’s original credential, and asks the Control Plane to resolve it into user, space and effective permissions; on its side, it only adds defense-in-depth checks.

The pipeline makes these rules verifiable. The Runtime tests that use the database run on the schema produced by the Control Plane migrations, and a check prevents modifying, renaming or deleting a migration that has already been merged, except for approved exceptions: a mismatch surfaces on the pull request, not in production. In a release, migrations come before the services, and the Runtime starts only once the schema is up to date. Adding a channel, then, means adding a type of request to resolve, not duplicating the access rules.

Isolation between customers is a property of every query

In a RAG system, a customer’s data also lives in the document chunks and their vectors. The most common solution is to place a dedicated vector database alongside the application database, with a per-customer filter on every search. It works, but it moves isolation into a second system, with transactions and guarantees different from those of the rest of the data, and turns every deletion into an operation to be coordinated across two stores.

We chose to keep vectors, chunks and application data in the same Aurora PostgreSQL, with pgvector. Isolation is expressed as a join on the documents table, in the same transaction and with the same guarantees as the rest of the data. Deleting a document is an operation on a single database. Full-text search, which works alongside semantic search, uses the same engine. On top of isolation, a permission system based on access groups determines which applications, tools and agents each organization and each user can use.

Each chunk also records the embedding model that produced its vector. It is a small detail with an important consequence: vectors from different models coexist, and changing the model on one space does not invalidate the others.

In practice, a new customer is an organization and a few spaces, not a new deployment, and there is no second store to keep in sync.

The quality of an answer is decided at two moments

A RAG system answers well only if it finds the right passages, and the right passages are prepared when the document comes in, not only when the question arrives.

On the way in: context and metadata, paying for models only where needed

The file reaches S3 directly from the browser, through a signed URL, and from there processing is entirely asynchronous. Where the feature is enabled for the organization, only PDFs containing relevant images go through a vision model capable of reading tables, diagrams and scans; the others are extracted as text page by page, and non-PDF formats become Markdown with MarkItDown. The cost of the vision model is paid only where it is needed.

The next problem is less visible: a chunk extracted from a long document often does not stand on its own. “The term is thirty days” is useless if you do not know which contract and which clause it refers to. Before the vectors are computed, each chunk is therefore enriched with information that places it within the document, following the contextual retrieval technique, with parameters configurable per space: an archive of contracts and a collection of product sheets are not indexed the same way.

In parallel, a second task extracts the document’s metadata with a LangGraph graph and normalizes the essential fields against the values already present, so that the same customer or the same category does not appear spelled three different ways. The two tasks are independent and the document status is derived from both: the user can always see which part of the processing has finished.

At question time: two searches and a single judge

Semantic search alone has a well-known weakness: it finds passages that say the same thing in other words, but tends to dilute codes, acronyms and identifiers. That is why the conversation is rewritten twice: a sentence for the vector search, a few keywords for PostgreSQL full-text search. The two searches run in parallel, and the semantic one returns passages of text large enough to contain a whole line of reasoning.

The most important choice is where to rank. Ranking each search on its own produces two rankings that nobody compares. We merge and deduplicate the candidates, and have a reranking model rank them only once: that is where it is decided which passages actually reach the model.

Metadata enters retrieval as a hard filter, when the question explicitly concerns a category of documents, or as boosting, which moves relevant results up without excluding the others. A conversation can be restricted to one or more document groups, and if the selected group is empty the search returns nothing instead of returning everything: it is the kind of detail that, when wrong, shows a user documents they had excluded.

Every stage is logged: generated queries, the candidates from each search, deduplication, final ranking and scores. When an answer is unconvincing, it is possible to reconstruct exactly what the model saw and why. Answers cite their sources by title and pages.

External dependencies live behind a contract of our own

The core of semantic indexing relies on an open-source RAG library with valuable retrieval algorithms, but designed for a different context: it handled its own persistence, talked to models through an interface different from that of the chosen providers, and was not designed for many customers on a single database. Faced with a dependency like this, there are two obvious paths, both expensive: rewrite it, or keep adapting to it.

Before choosing, we broke it down module by module, estimating the cost of rewriting each one. The retrieval logic was the most valuable part and the most expensive to reimplement; many other modules were infrastructure we wanted to replace anyway. So we decided to keep the algorithms and replace everything that connects them to the outside world: store and vector index reimplemented on the platform’s tables, with per-space isolation in every query; the library’s own persistence disabled; adapters for the models of all providers, with token counting; reranking moved after the merge with full-text search.

The agents do not see the library: they see a LangChain retriever like any other. It is the same reason we use LangChain and LangGraph: they give a common shape to components whose interface would otherwise change with every provider. Every model is a chat model, every agent is a graph with explicit nodes that can be read, traced and tested one by one, and capabilities such as history, memory, tracing and evaluation are inherited by declaring them. Prompts and models are configuration that each space can override: a new model from an already integrated provider is a record, not a code change.

Finally, token consumption is recorded per agent step, model, conversation and space, and is the basis for cost analysis: what a space costs can be read from the data, without reconstructing it after the fact.

The knowledge graph: a bridge, not an owner of the data

Similarity search answers local questions well, but struggles when the answer requires connecting entities scattered across different documents: which components appear in several projects, which procedures recur. For these questions we integrated a lexical graph of topics, statements, facts and entities, linked to the chunks they come from. It is built with AWS’s open-source GraphRAG toolkit and queried in OpenCypher, compatible with both Neo4j and Amazon Neptune.

The risk, with a toolkit of this kind, is letting it do everything. It is the fastest way to experiment, but then the toolkit computes the embeddings itself, queries the graph and generates an answer that the application then has to generate again, with duplicate paid calls. That is why the toolkit is a bridge for building and traversing the graph, not the owner of the data or of authorization: persistence, model calls and access controls remain with the platform.

The expensive part is paid only once

Graph ingestion is split into two phases with very different costs. In extraction, a language model analyzes each chunk; the result is validated against the expected schema and saved in PostgreSQL with its provenance: prompt fingerprint, model, parser and toolkit versions. A chunk already extracted with the same provenance is not extracted again. In construction, the graph is rebuilt entirely from the saved data, without model calls and with stable identifiers, so rebuilding the same document twice produces the same graph. Construction starts only when metadata and extraction are complete, and is claimed with a row lock so that it is queued only once.

On the vector side the toolkit has no store of its own: it reads the same chunk table as the rest of the platform, through a read-only index that rejects any write, and existing spaces required no data migrations. The graph can therefore be rebuilt at any time without paying for extraction again and without duplicating the vectors.

Isolation that does not trust a single check

On the graph, each knowledge space is a tenant. It is the point where an isolation error would cost the most, and there we chose to check twice. The tenant identifier is always derived from the already authorized space, never from the request. The toolkit’s multi-tenant layer is wrapped by an adapter of our own that rewrites every query and refuses to serve a space other than the one it was created for.

After the traversal, every returned identifier must belong to the expected tenant, otherwise the request fails with an isolation error. The surviving evidence is then re-authorized in SQL: an active document, from the right space, with all phases completed, consistent with the conversation’s filters and document groups, and indexed with the same embedding model. Finally, the graph’s tracing spans never record document text.

The graph agent is a LangGraph graph like the others, with the same inherited capabilities and without the toolkit’s answer generation and reranking, which are the platform’s job. If something goes wrong, the error is explicit: no silent fallback to classic search that would mask the problem. The graph is enabled space by space.

Workers: each work rhythm has its own lane

Background jobs are not all alike. A message from a conversational channel must be answered within a few seconds; indexing a document or extracting the graph can take minutes. In a single queue, a bulk upload delays the answer to a user who is waiting. So we split the Celery tasks into three lanes, each with its own SQS queue and its own infrastructure.

In the first two lanes each worker runs one task at a time and reserves only one message. By default a Celery process reserves four: convenient for short tasks, but with jobs that take minutes a message sits stuck behind an indexing job while another worker is idle. We give up a little density in exchange for predictable behavior: capacity is measured in running tasks and is increased by adding ECS tasks.

As a consequence, autoscaling follows the queue, not the CPU. The metric is the number of waiting messages per active worker, with a different target for each of the two lanes: when the queue grows, ECS adds workers quickly; when it drains, it removes them cautiously. A spike in uploads becomes more workers, and a period of inactivity does not keep idle workers running.

No interrupted work, not even during a release

Autoscaling and releases share a blind spot: to reduce capacity or replace a version, ECS sends SIGTERM to the containers and, once the stopTimeout expires, kills them with SIGKILL. It does not know whether a worker is halfway through an indexing job, and Celery’s warm shutdown waits for the task to finish only as long as ECS allows it.

The most common solution is to raise the stopTimeout of every container until it covers the longest task. On Fargate, however, the limit is two minutes, while an indexing job or a graph extraction can take longer. And even where it would be enough, a stop time tuned to the worst case slows down every release and every scale-in, even when the workers are idle. The alternative, accepting the interruption and requeuing, means redoing work that includes model calls already paid for.

We chose to reverse the responsibility: it is the worker that tells ECS when it cannot be stopped. A supervisor wraps the Celery process and receives, through a local Unix socket, the lifecycle signals of every task: start, completion, failure, revocation. It keeps a registry of active tasks with their maximum deadline, derived from each one’s time limit, and enables ECS task scale-in protection for exactly that long. While the worker is working, the autoscaler cannot pick it; as soon as it is free again, the protection drops.

The same supervisor handles releases. At regular intervals it compares its own version with that of the deployment the service considers primary; when it finds that it has been superseded, it asks Celery for a warm shutdown: it finishes the task in progress, takes no new ones and exits. A repeated stop signal escalates to a progressively more forceful shutdown.

Two details make the mechanism reliable, not merely functional. Calls to the ECS API are rate-limited and are made only when the deadline changes significantly, so a worker running many short tasks does not saturate the API. And the registry discards on its own the tasks that are past their deadline: if an event is lost, the worker does not stay protected forever, blocking scale-in.

The result is that we release and scale in at any time without interrupting an indexing job in progress, with normal stop times for all the other containers. A release does not require a window in which nobody is uploading documents.

Short tasks by rule, not by chance

Protection works better the shorter the tasks are, and the project made this a rule: a task that keeps a worker busy for too long is a design flaw, not a configuration problem. Multi-step processing is broken down into short, idempotent steps that save their progress in the database and enqueue the next step. A restart costs one step, not the whole job, and a worker’s protection never lasts longer than necessary.

For the rare jobs that cannot be split there is the third lane, which does not use a worker: the queue is connected through EventBridge Pipes to ECS, which starts a dedicated task for each message. The container decodes the Celery envelope, runs the task and exits. The task code does not change; only the process running it changes, and that process lives exactly as long as the job.

What the product gains

The client offers the platform as a service, and growing it does not require rethinking it. A new customer is an organization and a few spaces; a new channel is a type of request to resolve in the Control Plane; a new model from an already integrated provider is configuration; a new agent is a graph that inherits the existing capabilities, and a new retrieval technique, such as the knowledge graph, is enabled space by space. Isolation and releases do not have to be rethought at every step, because they do not depend on the feature being added.

This is how we approach AI products: the model is the component that changes most often, and precisely for that reason it cannot be where the guarantees live. The guarantees live in the architecture around it: a single owner for identity and schema, isolation inside every query, expensive work paid for once, workers that know when they cannot be stopped. Each of these choices costs more upfront than a shortcut; together, they allow the product to grow in customers, documents and features without going back to the foundations.