Local RAG · Portfolio Project

Teaching a Laptop-Scale AI to Read My Technical Library

I own 148 technical books from years of Humble Bundles — Python, data science, machine learning, AI. This project turns those ~48,800 pages of PDFs into a chatbot that answers questions from my actual books, with citations, running entirely on my own laptop — no cloud, no API bills, no data leaving the machine.

148 books · 48.8k pages 96,546 searchable chunks 100% local & private $0 running cost 3B parameter model

Skills & Technologies Demonstrated

Python RAG Architecture LangGraph Vector Databases Hybrid Search (BM25 + Embeddings) Prompt Engineering Local LLMs (Ollama) ChromaDB Observability & Debugging Gradio

The big picture

The whole system, end to end — from a folder of PDFs to a chatbot answering questions in the browser.

Figure 1 — Data flows left to right: 148 PDFs are ingested once into two search indexes (96,546 chunks), then every question runs through the LangGraph pipeline and back to the user.

🧭 In plain English: Think of it like a research assistant who has read all my books. When I ask a question, it first finds the relevant pages, then writes an answer using only those pages — and tells me which book each fact came from, so it can't just make things up.

What RAG is, and why it beats a bigger model here

Retrieval-Augmented Generation: instead of trusting a small model's memory, we look up the exact pages from my own books, put them in front of the model, and let it answer only from that supplied text — with citations.

Problem

A small local model doesn't "know" much and invents confident, wrong answers.

Approach

Split the job: a retriever finds facts, the model only rewords them fluently.

Result

A 3B model punches far above its weight, and every claim is traceable to a page.

The core idea is a separation of concerns. The retriever owns facts; the LLM owns fluency. Because the model never has to recall anything — it just paraphrases text placed directly in front of it — a tiny model becomes genuinely useful.

Why not fine-tune, or run a 70B model? Fine-tuning bakes knowledge into weights (expensive, static, painful to update when I add books). A 70B model won't fit comfortably on a laptop. RAG keeps knowledge as swappable data on disk, runs on hardware I own, updates instantly on re-ingest, and gives a provenance trail for every answer.

The stack — and the reasoning behind each choice

Everything open-source, everything local through Ollama.

qwen2.5:3b

Chat / reasoning LLM · ~1.9 GB

Why: the sweet spot of the Qwen2.5 family for a laptop — strong instruction-following at a size that stays resident in memory and answers in seconds.

nomic-embed-text

Embedding model

Why: an open, locally-runnable embedder with a long context window, well-suited to dense technical prose and code.

ChromaDB

Persistent vector store

Why: embedded, file-backed, no server to run. Embed once, query forever. Metadata travels with every vector.

rank_bm25

Lexical (keyword) search

Why: pure-Python, in-memory BM25. Vectors miss literal tokens (sklearn, argmax); BM25 scores them exactly.

LangGraph

Pipeline orchestration

Why: models the flow as an explicit graph over a typed state. Trivial to insert a new step and attach a persistent checkpointer.

PyMuPDF

PDF text extraction

Why: fast, accurate, page-level extraction with page numbers preserved — the metadata that later powers citations.

Gradio

Browser chat UI

Why: a production-feeling web chat in ~90 lines, session state and streaming baked in. No frontend build step.

SqliteSaver + logs

Memory & observability

Why: conversation survives restarts, and every turn is logged as a full trace so answer quality is debuggable from evidence.

How one question flows through the system

Figure 2 — Each stage neutralises a specific failure mode of a small local model.

# the whole graph, wired in a line and compiled with persistent memory
workflow.add_edge(START, "rewrite_query")
workflow.add_edge("rewrite_query", "retrieve")
workflow.add_edge("retrieve", "generate")
workflow.add_edge("generate", "verify_citations")
app = workflow.compile(checkpointer=SqliteSaver.from_conn_string(DB).__enter__())

1 Ingestion internals

ingest_books.py — run once, and again whenever books are added. On my library: 148 PDFs → 48,779 pages → 96,546 chunks.

Figure 3 — Pages are split into overlapping chunks; every one of the 96,546 chunks carries the metadata that powers citations like "practicaldeeplearning (p. 163)".

🧭 In plain English: Books are too big to hand to the AI whole, so they're sliced into page-sized snippets. Each snippet keeps a little label saying which book and page it came from — that's what lets every answer cite its source.

Chunking — the part that decides quality

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1200, chunk_overlap=300,
    separators=[
        "\n```", "```\n",   # prefer breaking AT code-fence edges
        "\n\n\n", "\n\n",   # then paragraph boundaries
        "\n", ". ", " ", "",  # then lines, sentences, words
    ],
)

Why the code-fence separators? The splitter tries separators in order and only drops to a lower one when a chunk is still too big. Putting ``` boundaries first means it prefers to cut at the edge of a code block rather than slice a Python example in half — critical when the whole library is programming books.

2 Hybrid retrieval & Reciprocal Rank Fusion

rag_chatbot.py — the heart of retrieval quality.

Figure 4 — Two searches run in parallel over the 96,546-chunk corpus. "doc B" ranks highly in both, so fusion floats it to the top.

🧭 In plain English: One search understands meaning ("plotting a distribution" ≈ "drawing a histogram"). The other matches exact words (a library name like sklearn). Running both and combining them catches results either one alone would miss.

The RRF formula

def reciprocal_rank_fusion(list1, list2, k=60):
    scores, doc_map = {}, {}
    for ranked in (list1, list2):
        for rank, doc in enumerate(ranked):
            key = doc.page_content[:120]          # dedup key: content prefix
            scores[key] = scores.get(key, 0) + 1 / (k + rank + 1)
            doc_map[key] = doc
    return [doc_map[k] for k in sorted(scores, key=lambda x: scores[x], reverse=True)]

Why RRF instead of blending raw scores? Cosine similarity and BM25 scores live on totally different scales — you can't just add them. RRF discards the magnitudes and keeps only the ranks, so no normalisation or hand-tuned weight is needed. k=60 (from the original RRF paper) softens the gap between ranks so one index can't wholly dominate.

3 Grounding & guardrails

Problem

Small models invent book titles and answer from training data, not my library.

Approach

An explicit source allow-list in the prompt, plus a deterministic post-check.

Result

Answers cite only books actually retrieved; anything else is redacted in code.

The anti-hallucination context header

Retrieved chunks are formatted with an explicit list of titles the model may cite — the single source of truth for what "exists":

sources_header = (
    "Available Sources (ONLY cite titles from this exact list):\n"
    + "\n".join(f"- {t}" for t in sorted(book_titles))
)

The deterministic citation verifier

After generation, a non-LLM node checks the answer. A quoted string only counts as a book citation when it appears in citation context (after words like book, titled, guide) — and when one fails the check, only the title is redacted; the answer survives:

# Ignore code first — string literals in code are never citations
prose_only = re.sub(r'```.*?```', ' ', answer, flags=re.DOTALL)

# A quote is a citation ONLY in citation context ('the book "X"', 'titled "Y"')
cite_pattern = r'(?:book|titled|guide|primer|reference)\s+["“]([^"”]{10,100})["”]'
candidates = re.findall(cite_pattern, prose_only, flags=re.IGNORECASE)

# Redact unverified titles — never nuke the whole answer
for bad in hallucinated:
    answer = answer.replace(f'"{bad}"', "[title removed — not in retrieved sources]")

Why so surgical? This design was earned the hard way. Earlier versions flagged any quoted string and replaced the whole answer with a refusal — and the trace logs proved every single flag was a false positive (see the debugging journal below). Prompts nudge; code enforces; and the enforcement itself has to be tested against real transcripts.

4 The product — what the user sees

rag_ui.py wraps the whole pipeline in a Gradio browser chat.

Figure 5 — The Gradio UI: a grounded answer with a working code snippet, and every book that was consulted listed underneath for full transparency.

Clean browser chat at localhost:7860 — no terminal needed.
Each browser session gets its own memory thread (a UUID), so conversations stay isolated.
The books used to answer are shown under every reply.
Every exchange is written to a per-session trace log.

Real session logs

Every turn writes a full trace: the rewritten query, every retrieved chunk with book/page/position, the raw answer, and the verifier's decision. These are unedited excerpts from my own sessions.

A healthy turn, end to end

Note the rewriter expanding "EDA" and the retrieval spanning four different books:

chat_logs/·····.log

[2026-07-08 11:54:38] ==== Turn ====
Question: Can you teach me how to do python with pandas for EDA
Rewritten query: How to use Python Pandas for Exploratory Data Analysis (EDA)
Retrieved (10 chunks):
  - pythonforexcelusers (p. 36) [2026 - Python The Good Stuff by No Starch] chunk 112/818
  - practicalstatisticsfordatascientists (p. 35) [2019 - Data Analysis & ML by O'Reilly] chunk 92/776
  - machinelearningpocketreference (p. 26) [2021 - Pocket Reference Guides by O'Reilly] chunk 38/388
  - python3formachinelearning (p. 160) [2021 - Data Science & Analytics by Mercury] chunk 265/665
  ... (6 more)
Citation check: no titles redacted
Final answer: Certainly! Here's a step-by-step guide on how to use Python with
Pandas for Exploratory Data Analysis (EDA)... [full 8-step tutorial with code]

The trace that caught a real bug

This excerpt is why observability matters. The model gave a good, grounded answer about Claude — quoting the phrase "You're absolutely right" directly from one of my Claude Code books. The old verifier mistook that quoted phrase for a fabricated book title and destroyed the answer:

chat_logs/·····.log · before the fix

[2026-07-08 11:53:22] ==== Turn ====
Question: can you give me tips on using Claude
Retrieved (10 chunks):
  - 101claudecodetips abattle testedfieldguideforagenticcoding (p. 81) chunk 158/166
  - claudecodeautomation mcpskillsandproductionagenticworkflows (p. 90) chunk 133/152
  ... (8 more)
Raw answer: Certainly! Here are some tips for using Claude based on the provided
excerpts: 1. **Be precise**: Claude loves clear questions. ...
4. **Check the diff when you see "You're absolutely right"**: Claude might be
matching the conversational shape of agreement rather than agreeing...
Citation check: stripped hallucinated title(s): ['You're absolutely right']
Final answer: I don't have information about that in the books I have access to.

A good answer, wiped by an over-eager guardrail. The trace made the root cause obvious in seconds: the verifier treated every quoted string as a book title. After the fix (citation-context detection + redaction), the same question:

same question · after the fix

Question: can you give me tips on using Claude
Citation check: no titles redacted
Final answer: Certainly! Here are some tips for using Claude based on the
provided excerpts:
1. **Be precise**: Claude performs better when it receives clear and specific questions.
2. **Talk to it like a friend**: Claude likes context, so use a conversational tone.
3. **Recognize patterns of failure**: The book mentions five common pitfalls that
   teams often encounter while using Claude Code...

🧭 In plain English: Because the system keeps a detailed diary of every step it takes, I could see exactly where a good answer was being destroyed — and prove the fix worked by replaying the same question.

Debugging journal — the bugs that shaped the design

Most of the real engineering wasn't features — it was containing a small model's failure modes. Every fix below was found by reading the trace logs, not by guessing.

① The verifier kept eating valid answers

Before

The verifier treated ANY quoted string as a potential book title and replaced the whole answer with a refusal. First it flagged code strings like print("Predicted labels:", preds); after excluding code, it still flagged quoted phrases — including a tip quoted from my own book ("You're absolutely right"). Audit result: every flag it ever raised was a false positive.

After

Two-part redesign: a quote only counts as a citation in citation context (after book / titled / guide…), and a failed check redacts just the title — [title removed — not in retrieved sources] — instead of destroying the answer.

② The rewriter sabotaged retrieval

Before

Two failure modes from the logs: hyphen-glued garbage (how-to-use-claude-and-seem-like-a-hacker) that neither index could match, and topic injection — "tips on using Claude" became "…for Python programming", pulling grep and PyTorch chunks instead of my five Claude books.

After

The prompt forbids adding topics the user didn't mention and forbids hyphen-joining; a code guard rejects any rewrite with >3 hyphens or >200 chars and falls back to the user's original wording.

③ Over-refusal vs. hallucination — the prompt see-saw

Before

A refusal-first prompt made the model bail on answerable questions. Loosening it brought hallucinations back — and sometimes the model glued the refusal sentence onto the END of a complete, correct tutorial.

After

The prompt asserts the context is relevant, tells the model to ignore irrelevant excerpts rather than force them in, and keeps one tight, exact-wording refusal. A deterministic post-step strips any refusal sentence glued onto a substantive answer.

Meta-lesson: with a small model, the prompt, the retrieval, and the deterministic post-check are three dials that must be tuned together — tightening anti-hallucination loosened over-refusal, and vice versa. The trace log was what made this tractable: you cannot tune what you cannot see.

Running it yourself

The repo ships only code — you bring your own PDFs. My library, vectorstore, and chat logs never leave my machine.

# 0. Prerequisites
ollama pull qwen2.5:3b
ollama pull nomic-embed-text
pip install -r requirements.txt

# 1. Add your PDFs to ./books (sub-folders become categories), then ingest
python ingest_books.py --clear

# 2a. Chat in the terminal
python rag_chatbot.py

# 2b. …or launch the web UI at http://localhost:7860
python rag_ui.py

Key learnings

Retrieval > model size. A well-fed 3B model beats a bigger, ungrounded one for library Q&A. Effort on chunking and hybrid search paid off more than any model swap would.
Hybrid search is not optional for technical content — vectors and BM25 fail on opposite query types, and RRF gets both for free without score normalisation.
Guardrails need guardrails. My anti-hallucination check destroyed more answers than hallucination ever did, until real transcripts proved it and drove a surgical redesign.
Chunk boundaries decide answer quality — code-fence-aware splitting was the difference between whole and mangled code examples.
Observability is a feature. Full per-turn trace logs turned "the bot feels worse today" into specific, fixable root causes — every bug in the journal above was found in the logs.
Local is genuinely viable — 148 books, 96k chunks, ingestion, embeddings, LLM, memory, and UI all run on a laptop, offline, for free, with data never leaving disk.

About this project

A personal project exploring how far a fully-local RAG system can go on consumer hardware — combining retrieval engineering, prompt design, and evidence-driven debugging to make a small model reliable. Built end-to-end in Python: ingestion, hybrid retrieval, the LangGraph pipeline, and the web UI. All source code is in this repo.

👤 Aaron Long GitHub → Along777 ✉ longaa1897@gmail.com