Foundry Local RAG

Problem

Whenever I wanted to ask an AI a question about a document and trust the answer, two things got in the way. The first was privacy: sending an internal report to a cloud model is not acceptable in most real settings. The second was hallucination, where the model invents an answer it does not actually have.

I built this assistant to address both by running it entirely offline, with no internet connection. For the test domain I picked a report on cold chains and refrigeration. A technical, narrow text makes it clear when the model drifts outside what it actually knows.

Approach

I split the system into two stages. In the indexing stage the PDF is converted to text and cut into chunks of about 200 words, with a 30-word overlap between them so sentences are not sliced in half. Each chunk is embedded with qwen3-embedding-0.6b and stored in SQLite as a float32 blob, which removes the need for a separate vector database.

In the query stage I embed the user's question with the same model and compare it against the stored vectors using cosine similarity. The key decision sits here: if the best match scores below 0.35, the system refuses to answer and never sends the question to the model. If the score is high enough, the retrieved chunks go to phi-3.5-mini and the answer is grounded in them. I ran the models on Microsoft Foundry Local, exposed both a Streamlit interface and a command line, and handled PDF extraction with pdfplumber.

Outcome

I tested the system against a set of ten questions and it behaved as expected on all ten. It answered the six in-scope questions, with similarity scores between 0.58 and 0.73, and rejected the four out-of-scope ones, scoring between 0.17 and 0.26.

The most useful finding was about the threshold. Running the similarity check before the model, rather than after, stopped hallucinations that a prompt instruction like "say you don't know" could not prevent on its own. Putting the guardrail in the retrieval layer worked better than putting it in the instructions.

The limits are clear too. On CPU each answer takes around 40 seconds. Two-column or boxed PDF sections sometimes produce garbled text fragments. The smaller embedding model occasionally under-answers on specific figures. The system is tuned for English, and cross-lingual performance drops noticeably.

What I learned

Even with small, local models, a careful retrieval design is enough to build a question-answering system you can rely on. The most solid way to prevent hallucination turned out to be a threshold rule on the retrieval side, not better wording in the prompt. My next step, improving PDF extraction and embedding quality, should reduce the missing answers on numeric questions.