My RAG found 6 of 13 answers. Four fixes got it to 12.

I built a chatbot that answers questions from a business's own documents. For 7 of 13 test questions, it never even found the paragraph with the answer.

This kind of chatbot is called RAG, short for retrieval-augmented generation. It works in two steps. First it searches your documents for the passages most related to the question (the retrieval). Then it hands those passages to a language model, like the one behind ChatGPT, which writes the answer (the generation). The model only sees what the search gives it. If the search picks the wrong passages, the model writes a confident answer from the wrong material.

So before changing anything else, I tested the search on its own. I wrote 13 questions about one long PDF, and for each one I listed words that a correct passage must contain. Then I checked one thing: did the search find a passage with the answer?

How my search works

My search looks for passages in two ways at the same time, then combines the results.

The first way searches by meaning. It turns the question into an embedding, finds the chunks whose embeddings are closest, and keeps only the ones with a similarity score of 0.4 or more. The second way searches by words. It finds chunks that contain the words in the question, like the search box on a website, and has no cutoff.

To combine the two lists, each chunk gets points for its position in each list: more for first place, fewer for tenth. A chunk near the top of both lists ends up first overall. This method is called reciprocal rank fusion. It uses positions instead of scores because the two searches score on different scales. A meaning score of 0.6 and a word score of 0.08 can't be compared, but first place is first place in both.

Keep the two halves in mind. The combined result hid a problem in one of them for most of the night.

WITH by_meaning AS (
  SELECT id, ROW_NUMBER() OVER (ORDER BY embedding <=> $question) AS rnk
  FROM chunks WHERE 1 - (embedding <=> $question) >= 0.4
), by_words AS (
  SELECT id, ROW_NUMBER() OVER (ORDER BY ts_rank_cd(tsv, q) DESC) AS rnk
  FROM chunks, websearch_to_tsquery('english', $words) q WHERE tsv @@ q
)
SELECT id, SUM(1.0 / (60 + rnk)) AS score
FROM (SELECT * FROM by_meaning UNION ALL SELECT * FROM by_words) both
GROUP BY id ORDER BY score DESC LIMIT 10;

The night in one chart

I made four fixes, one at a time, and ran the same 13 questions after each. The blue line is the number that mattered: how often the combined search found the answer. The amber line counts the questions where meaning search had any match at all, a chunk scoring 0.4 or more. It went down twice before it went up.

The combined number looked fine because word search was doing the work. Meaning search was contributing less and less underneath.

Fix 1: the PDF reader was eating spaces

Before a PDF can be searched, a library pulls the text out of it. I printed what the search was returning and found this:

The library I used, PyPDF, dropped the spaces between words in some PDFs. To word search, "Theproblemstatementtheagentreceives" is one long word that nobody will ever type. To the embedding model, it's close to gibberish. I switched to a different library, PyPDFium2, which keeps the spaces.

Fix 2: the chunks were too small

My chunks were 500 characters long, about a short paragraph, with 50 characters shared between neighbouring chunks. That was too short. Most answers were split across two chunks, and neither chunk held the whole answer. I made them 1,200 characters with 200 shared, so a typical answer fits in one chunk.

My guess is that a bigger chunk covers more topics, so its embedding is a blurrier summary of what it's about. Word search didn't care, and the combined result hid the drop.

Fix 3: an instruction where it didn't belong

Some embedding models are trained to see a short instruction before each question, such as "Represent this sentence for searching relevant passages:". I was adding that instruction to every question. The model I was using, all-MiniLM-L6-v2, was never trained with it, so the extra words made every question's embedding less accurate. I changed the code to add the instruction only for models that expect it.

Fix 4: a model made for this job

all-MiniLM-L6-v2 is a general model, built to compare any two sentences. bge-small-en-v1.5 is trained to match a short question with the longer passage that answers it. Both turn text into a list of 384 numbers, so I could switch without changing the database. I made new embeddings for every chunk and ran the questions again.

The average similarity score of the best match went from 0.39 to 0.65. With the old model, that average was below the 0.4 cutoff, so meaning search threw away many good chunks. At the start, two questions were answered only thanks to word search: the best meaning scores for them were 0.15 and 0.18.

Where these numbers come from

The test is one script and a file of 13 questions. Each question lists the words a correct passage must contain, so the script can check the search results without asking a language model, and a run takes seconds. I ran it before and after every change and compared the results. Here's the first run against the last:

$ python eval/run_eval.py --compare baseline.json fix4_bge.json

  retrieval_recall                 46.2%      92.3%   UP +46.1%
  vector_hit                       61.5%     100.0%   UP +38.5%
  mean_top_similarity            0.3852    0.6485   UP +0.2633

In that output, retrieval_recall is the blue line and vector_hit is the amber one.

Later I added a mode that also asks the language model to write each answer and checks it. That caught a different problem: 11 of 13 answers included tags like "[Document 4]", which mean nothing to a customer. After I changed how passages are labelled for the model, that dropped to 0, and a rerun still got 13 of 13 right. That's the next post.