paper-rag

ingest writes chunks and citation metadata in; ask reads chunks back through hybrid retrieval; cite/mine read the metadata directly, skipping the vector store entirely.

ingest → chroma + bm25

doc_key (PMID or sha256) is written on every chroma entry and reused for delete-then-reinsert dedup. Every ingest that touches chroma also fully rebuilds the BM25 index — since BM25's IDF weights need the whole corpus, not just the new chunks. metadata.json (dashed path) is written straight from the citation fetch — it never enters either index, which is what lets cite.py/mine.py answer without touching them (next figure).

ask → hybrid retrieval

Both legs query the same collection; --dense-only / --lexical-only bypass fusion for debugging. --where filters (pmid, type) apply natively to the dense leg, post-hoc to the lexical leg — BM25 has no metadata filter of its own. Result JSON keeps each hit's dense rank and lexical rank alongside the fused score; rrf_k defaults to 60.

Falls back to dense-only — with a warning — if a collection exists but its BM25 pickle doesn't (e.g. papers ingested before hybrid search shipped).

cite / mine → metadata.json, no index

"First author", "which journal", "which year" are facts already sitting in metadata.json — asking the vector store to relevance-rank for them would be answering a lookup with a guess. glob + filter is plain Python; neither chroma nor bm25 is ever touched. /paper-rag:mine defaults --author from config.json's author_name, set once via /paper-rag:whoami.

RAG home — file & folder layout

<home>/                              # resolve_home(): --home > $PAPER_RAG_HOME > .paper-rag-home > homes.json "current" > ~/.local/share/paper-rag
├── config.json                       # {"embedding_model": "...", "author_name": "..."}
├── chroma/                            # chromadb.PersistentClient(path=chroma_dir(home))
│   └── ... one collection per embedding model: papers__{model_slug}
├── bm25/
│   └── {model_slug}.pkl                # pickled BM25 index, rebuilt whole on every ingest touching that model's collection
└── papers/
    └── <doc_key>/                    # doc_key = resolved PMID, else sha256 of the local file
        ├── metadata.json              # authors, year, journal, doi, doc_key, has_fulltext
        ├── fulltext.md                # docling-converted markdown — only written when has_fulltext
        └── figures/
            └── fig_0.png, fig_1.png, ...  # PictureItem crops, PDF sources only
~/.config/paper-rag/
└── homes.json                        # {"homes": {name: path}, "current": name} — multiple libraries, switched by name
<project>/.paper-rag-home                # optional per-project override: a raw path, or "name:<name>" into homes.json

Every paper gets its own doc_key directory under papers/ — the same key that tags each chroma chunk and bm25 entry, so --force re-ingest deletes-then-reinserts by directory, not by search. chroma/ and bm25/ are keyed by embedding_model, not doc_key — switching models gets you a fresh collection and index, not a mixed one.