RAG decision card: which retrieval method to pick for your agent

Pull this card out when you're designing an agent that answers from your own data, or when an existing agent keeps getting answers wrong. It doesn't explain theory. It walks you through the decision: questions first, then matching the method, then what the agent has to be given, and finally the signs you chose badly. The four methods on the card are database filters, SQL queries, full context and chunk-based vector search.

1. Before you pick a method

Write the answers down; don't keep them in your head. Each one narrows the choice.

• List ten real questions the agent will get. Real means the ones people actually ask, not the ones that are easy to test.
• For each question, note what a person would have to look at to answer correctly: one row, a few rows, the whole table, a whole document, one paragraph out of many documents.
• Name the type of data: a table with rows and columns, or running text (procedures, notes, policies, conversation records).
• Check whether the question needs calculation: a total, an average, a ranking, a comparison between periods.
• Check whether order matters: a timeline, process steps, how a meeting unfolded.
• Estimate the size: a few documents of a few pages, or hundreds and thousands.
• Note how often the data and its structure change: new products, new columns, new document versions.

If the questions on your list point to different answers, that's normal. One agent can need two methods at once.

2. Matrix: question type → method

Two rules settle the borderline cases:

• A question about the whole (table or document) never goes to chunk search. The agent will see one piece and answer as if it saw everything.
• A question about numbers in a table never goes to the model to eyeball. The database does the math; the model only writes the query and reads the result.

3. Checklist for each method

Filters

• The agent's instructions contain the full list of valid values for every field it filters on, spelled exactly as the database spells them (capitalization included).
• The instructions state the date format the database uses, with an example.
• You know who updates the instructions when a new product or category appears or the date format changes.
• The agent has a calculator or another way to add up results if the question needs a sum.
• You've checked one question by hand: the same filter in a spreadsheet gives the same result as the agent.

SQL

• The agent knows the table name, the column names, what they mean and some example values.
• If the database structure changes often, the agent has a tool to check the current schema before it writes a query.
• The agent has read-only access. A query written by a model shouldn't be able to change or delete anything.
• The calculations (sums, grouping, sorting) happen inside the query, not in the model after it pulls the rows.
• You've compared one agent answer against a pivot table built by hand on the same data.

Full context

• You've chosen how to supply it: a tool the agent calls itself, text pasted into the system prompt, or documents inserted dynamically on every question.
• If there are several documents and questions usually concern one, use the tool. The agent then reads only what it needs.
• If the documents change, insert them dynamically so that a change in the source doesn't require editing the agent.
• You've checked how much text one question processes and whether the cost is acceptable at the number of questions you expect.
• You know at what size the collection will stop fitting in the model's context window, and what you'll do then.

Vector search

• The collection is large enough that reading all of it on every question makes no sense.
• Answers sit in single spots in the text, not in a document as a whole.
• You've set how many chunks the agent retrieves and tested a few values. More chunks give a fuller picture and cost more.
• If the agent needs to know which document a chunk came from, the chunks carry metadata: title, date, position in the document.
• Tables don't go into the vector database as text. They stay in the database and get handled by filters or SQL.

4. Signs you picked wrong

• The agent confidently names a highest or lowest value that doesn't match the table. It's calculating on a chunk. Switch to SQL.
• The agent's average or total differs from the spreadsheet's. Same problem. The database should do the math.
• A filter returns zero results even though the data exists. The agent typed a value with different spelling or a date in another format. Add the list of valid values and the date format to the instructions.
• A summary skips parts you know were in the document. The agent is summarizing what it found, not the whole. Switch to full context.
• The timeline comes out in the wrong order. Chunks don't carry their position in the document. Use full context, or metadata with the position.
• The cost of a single answer climbs with every new document. Full context no longer fits the size of the collection. Consider vector search for single-fact questions.
• The agent reads every document even when the question concerns one. The context is pasted into the system prompt. Turn it into a tool the agent calls selectively.
• After you add a new product, the agent stops finding it. The instructions still hold the old list of values. That's a job for whoever owns the agent, not for the model.

5. One line to finish

The question picks the method, not the trend. Start with a list of real questions, note what a person would look at for each one, and give the agent exactly that, no less and no more.