Orchestrating a RAG System: Static Pipelines vs Agentic Control

How do you orchestrate a RAG system?

Orchestration is the code that decides which step runs next: rewriting the question, retrieving passages, reranking them, calling the model, checking the answer and returning the sources. The model writes the text, but the orchestration layer sets the order and the conditions.

Anthropic separates two kinds of systems. Workflows are systems where LLMs and tools are orchestrated through predefined code paths ^[1]. Agents, on the other hand, are systems where LLMs dynamically direct their own processes and tool usage ^[1].

A RAG system can use either. The choice decides how predictable the system is, what it costs per question and how hard it is to test.

When is a static pipeline enough?

A static pipeline runs the same steps in the same order for every question. The arXiv review of RAG architectures names the same split as orchestration layers, static pipelines versus agentic controllers, and describes static pipelines as systems that adhere to predetermined, rule-based operations ^[2].

For a search tool over a fixed set of product manuals, that fixed path is often the right first build. A typical pipeline looks like this:

• Retrieve the top passages for the question, with metadata filters such as product version.
• Rerank the passages and drop the ones below a score threshold.
• Generate an answer from the remaining passages only, with a link to each one.
• Return a "no answer found" response when nothing passes the threshold.

Anthropic gives the same starting advice: it recommends finding the simplest solution possible, and only increasing complexity when needed ^[1]. That may mean not building agentic systems at all ^[1].

When does an agentic controller earn its cost?

An agentic controller lets the model choose the next step. That helps when the first search shows what to search for next, such as a question that needs facts from two product releases. The cost is real. Agentic systems often trade latency and cost for better task performance ^[1].

One pattern in Anthropic's guide fits RAG well. In the orchestrator-workers pattern, a central LLM dynamically breaks down tasks, delegates them to worker LLMs, and synthesizes their results ^[1]. The subtasks are determined by the orchestrator based on the specific input ^[1], not written in advance.

For RAG, the workers can be retrievers over different indexes, such as product docs, support tickets and release notes. The orchestrator decides which indexes to query and merges what comes back. Each extra decision adds a model call, latency and another place for failure, so keep the tool list short, cap the number of steps, and log every choice the controller makes.

Take a question such as "Did the connector change between the two releases, and what do the migration notes say?" A static pipeline retrieves once, so it may miss the migration notes for the second release. A controller can search the release index first, read the change it finds, and then query the migration notes for that specific change. The extra round trip is the price of that second search, and it is worth paying only for questions that need it.

How do you keep a multi-step run on track?

A long run needs state that lives outside the model. Anthropic's post on long-running agents says the core challenge is that they must work in discrete sessions ^[3]. Its fix uses an initializer that sets up the environment, and a coding agent that makes incremental progress in every session, while leaving clear artifacts for the next session ^[3].

The same post describes a failure that a RAG controller can repeat. A later agent instance would look around, see that progress had been made, and declare the job done ^[3]. A controller that has found a few passages can stop early in the same way.

The fix is a log the controller writes after each step. The log holds the queries already run, the sources already read and the sub-questions still open. The controller finishes only when every sub-question is answered or marked as not supported by the sources. Keep the log in a store the controller reads on each step, not only in the prompt.

How do you test the orchestration layer?

Test the steps and the whole run separately. A single end-to-end score hides where a failure starts. The arXiv review reports that standardized benchmarks are underutilized, while custom evaluation criteria are predominant ^[2], so a team usually has to write its own test set.

Check four things:

• Retrieval: does the expected source appear in the top results for each test question?
• Routing: does the controller choose the right index or tool?
• Grounding: does each claim in the answer trace back to a retrieved passage?
• End to end: does a basic run still pass after each change?

The test set needs more than easy lookups. Include questions with a single source, questions that need two sources, questions the corpus cannot answer, and questions where two sources disagree. The last two matter most for grounding, because a system that always produces an answer fails them without any visible error.

Anthropic's long-running agents post asks the agent to run through a basic end-to-end test before implementing a new feature ^[3]. Apply the same rule to a RAG pipeline: run the end-to-end check before each new step is added, not only before a release.

Which pattern should you build first?

Build the static pipeline first and measure it on your test set. Move a question type to an agentic controller only when the pipeline fails on it, and only at the step that needs a decision. Put each step behind a function with a fixed input and output, so the controller can replace one step without a rewrite of retrieval.

Record the version of each step in the logs as well. When answer quality changes after a release, the version record shows whether retrieval, routing or generation changed, so the fix can target one step.

The next concrete step is to collect real user questions from your logs, mark the source that answers each one, and run the pipeline against that set before you change anything.