Making firm knowledge accessible to AI: RAG is not the only answer
RAG splits documents into chunks and pulls out the most similar ones. In file work, the most similar passage is rarely the right one.
Anyone looking to make their own firm’s knowledge available to a language model gets the same answer almost every time: RAG. For a large share of cases it is the right one. It is just neither the only answer nor, for typical file work, the most obvious.
What RAG actually does
RAG stands for retrieval augmented generation, a model that is handed relevant passages before it answers. Technically it works like this: the documents are split into chunks, each chunk is translated into a string of numbers that represents its meaning (a vector), and those vectors sit in a database of their own. When a question comes in, it too is turned into a vector, and the system pulls out the chunks whose vectors are closest to the question.
The last step is the one that matters. The model does not see the file. It sees a handful of excerpts that a search procedure picked out beforehand. Whatever that procedure fails to find does not exist as far as the answer is concerned.
Where the approach works well
On large, uniform collections where the answer really does sit in one chunk. Standard clauses, contract provisions, information sheets, internal handbooks: if you want to know how the firm words a limitation of liability, you need exactly one passage, and similarity search finds it reliably. The bigger the collection, the clearer the advantage, because the search barely slows down while a human takes ever longer to leaf through.
Where it breaks in file work
The weakness sits in the assumption that the most similar passage is the right one. That holds as long as a question has one source. It stops holding as soon as the answer is spread across several documents, as soon as chronology matters, or as soon as the decisive passage does not contain the search term at all.
The running of a deadline is the clearest example. It does not follow from the chunk that looks most like the word “deadline” but from a service date on the acknowledgment of receipt, from a provision that appears in no file, and from a public holiday nobody wrote down. A similarity search will reliably deliver something here that looks like an answer.
In file work this case is not the exception but the rule. Files are ordered chronologically, not thematically. The facts emerge from the sequence of letters, and a sequence cannot be cut into excerpts without losing precisely what matters.
Agentic search: the model works through the filing itself
There is a second route that does without a vector database. The model gets tools instead of excerpts: list a directory, read a file, search the full text for a term. With those it works through the file the way a person opens a binder. Structure first, then the document that looks relevant given that structure, then the next one. This design is called agentic search, because the model picks the search steps itself instead of having them handed to it.
Several things get simpler. There is no vector database to run and keep current. There is no reindexing when a document changes, because the next query reads the current version anyway. Access is traceable, because the log records which file was read, not merely that some chunk with high similarity was selected. And the file system’s permissions still apply: anyone who may not open a directory gets no content out of it either. In a firm with confidentiality walls that is no side point.
The price is model calls. Where RAG searches once and answers once, agentic search reads several files in sequence and calls the model each time. That takes longer and costs more, and how much more depends on how orderly the filing is. What operating decisions like this mean financially is covered in the post on what AI costs a law firm.
The prior question is the filing, not the procedure
That shifts the question. It is not “RAG or agentic search” but whether the filing is structured well enough for any procedure to find something in it. A collection with directories called “Miscellaneous”, files called “Scan_final_v3.pdf” and several versions of the same brief with no recognizably valid one makes both routes bad. RAG indexes the disorder along with everything else; agentic search gets stuck in it.
That is the uncomfortable answer, because it sounds like work that shows nothing for itself: fix the structure, standardise the naming, remove duplicates. This work is coming anyway. Do it before choosing the tool and you end up with a choice between two procedures rather than between two bad results.
Without text recognition, no procedure finds anything
With scanned documents there is one more step before all this. A PDF out of the scanner is an image. To any search procedure such a file is empty, however well it otherwise works. So before any thought about search comes text recognition across the collection, and with older files a spot check on whether the recognition holds up.
In the end the choice between the two is an architecture decision, and the same rule applies here as elsewhere: the simplest design that meets the requirement. For a large clause library that is RAG. For a file where the chronology carries the answer, it is orderly filing with read access. If you need both, build both, one after the other and not as one procedure for everything.
The content of this blog was created partly with the support of AI.