The data model

Library layout

/ref:init <path> runs init_repo.py, which creates these directories and an empty log.md, then records the root in ~/.config/ref-manager/config.json. This is its actual output on an empty scratch library:

$ python3 skills/ref-manager/scripts/init_repo.py lib
library initialized at …/scratchpad/concepts-api/lib
config written to …/scratchpad/concepts-api/.config/ref-manager/config.json

Everything below the top level is created by later commands, on first write. The tree marks created by init, user-owned content, and comments.

<LIBRARY>/
  log.md                          # "# ref-manager log"
  papers/
    <pmid>/
      meta.json                   # bibliographic authority, citekey, extraction_tier (add.py)
      authorship.json             # ordered author list, raw_order_preserved (add.py)
      funding.json                # funding observations + state (add.py, fetch.py)
      raw/<sha256>/               # immutable, content-addressed
        response.json             #   metadata envelope as resolved (add.py)
        source.jats | source.html | source.plain_text   # (fetch.py)
        source.pdf, attachment.json                     # (attach.py)
      current.json                # {"version": "v-…"} — atomic pointer
      versions/<v-id>/
        manifest.json             # source/converter/hash, or extractor/model/prompt versions
        source.md                 # converted full text
        figures/  figures.json    # figure assets, captions, cached vision entries
        claims.json               # claims emitted by this extraction run
      claim_registry.json         # cross-version claim history, supersession links
      notes.md                    # /ref:note — never touched by extraction
      annotations.json            # /ref:pull-annotations — upsert, tombstones on upstream delete
      corrections.json            # /ref:verify review overlays
      citations.json              # append-only cited-by observations (D25)
      related.json                # /ref:related snowball candidates
  projects/
    <slug>/
      project.yaml                # slug, scope, questions[] (JSON content, .yaml name)
      papers.yaml                 # membership: relevance, priority, reading_status, screening
      screening.jsonl             # append-only decisions
      tables/<batch>/             # /ref:compare: table.json, manifest.json, edits.json
      summaries/<batch>/          # /ref:summarize: summary.md, evidence.json, manifest.json
      briefs/<key>/               # /ref:brief: <snapshot_id>/ (answer.md, evidence.json, manifest.json), latest.json, edits.json
      reviews/<batch>/            # /ref:review: appraisals/<pmid>.json, grade.json, manifest.json
      checks/<id>/                # /ref:check-citations: input.md, findings.json, manifest.json (+ references.bib/.csl.json)
      prisma/<snapshot>/          # /ref:review --prisma: flow.md, flow.csv, manifest.json; latest.json
  people/<slug>.json              # name variants, ORCID, confirmed/candidate/rejected matches
  labs/                           # created, no command writes lab records yet
  grants/<slug>.json              # funder, award_number, approved_aliases
  reports/<label>/                # /ref:report: manifest.json, publications.csv, report.md
  studies/
    studies.jsonl  datasets.jsonl  methods.jsonl
  graph/
    concepts.jsonl  relations.jsonl                     # authoritative graph records
    people_relations.jsonl                              # regenerated by graph_people.py
  okf/                            # generated: concepts/ papers/ people/ grants/ index.md log.md
  index/
    catalog.sqlite                # rebuildable FTS5 projection
    citekeys.json                 # allocated citekeys
    .library.lock                 # library-wide flock
    .locks/<pmid>.lock            # per-paper flock
    aliases/<kind>.json            # slug registry: live slugs + rename aliases
  queries/<slug>.yaml              # saved PubMed query + immutable runs[]
  exports/
    <batch>/                      # /ref:export: references.bib, references.csl.json, manifest.json
    papers/<batch>/                # /ref:export-papers: references.bib, PDFs, manifest.json
    papers/note_pushes.json       # per-paper pushed-note hashes (D28)

Set-valued artifacts written without --project land at the library root instead (tables/, summaries/, briefs/, reviews/, checks/); init does not pre-create those. For the per-paper directory drawn as a figure, see Paper directory anatomy.

Ownership & recovery

Files are the authority. SQLite and okf/ are projections you can delete and rebuild. Content you write sits in files that regeneration never opens for writing.

Atomic writes

• JSON and text. atomic_write_json writes a temp file next to the target (.<name>.*.tmp), fsyncs it, then os.replaces it into place. JSON is written with sorted keys and two-space indent.
• Versions. commit_version(paper_dir, version_id, write_fn) creates versions/.staging-<id>/, lets the caller fill it, renames it to versions/<id>/, and only then rewrites current.json. If write_fn raises, the staging directory is removed and current.json is left alone.
• Readers ignore staging. catalog.py rebuild follows current.json only, so a crash mid-commit leaves nothing half-indexed.

Locks

Both locks are fcntl.flock exclusive locks. pmid_lock serializes one paper's writes without blocking other papers, which is why batch commands report a result per PMID. library_lock guards citekey allocation, slug registries and graph commits.

Recovery and stage state

The catalog is dropped and recreated on every rebuild, with no in-place migration. relation.py refresh recomputes stale on edges. PLAN.md also describes a per-paper state.json with stage checkpoints, but no script writes one. Recovery comes from staging plus atomic pointer swaps, not from a checkpoint file.

Extraction tiers

meta.json carries extraction_tier. lib_schema.validate_meta accepts exactly three values:

• Promotion only goes up. extract.py ranks unavailable < abstract < full and never lowers the tier when a later run is thinner.
• Having full text is a separate fact. fetch.py and attach.py set meta.full_text and commit a source.md version, but they do not promote the tier. Only a committed extraction does.
• If no evidence tier is given, extract.py uses "full" when meta.full_text is true and "abstract" otherwise.
• Selectors filter on it with --tier abstract|full|any, and each resolution reports counts per tier (below).

The claim contract

A claim is one reported finding. The ref-extractor subagent proposes claims as JSON (contract). extract.py fills in the defaults, validates each claim with validate_claim, assigns IDs, and commits.

Real claim · versions/v-ff60b1f75514/claims.json (scratch library, placeholder PMID)

[
  {
    "adjustment_context": "unknown",
    "claim_id": "c-2a41dbb8e59f",
    "cohort_identity": "unknown",
    "comparator": "non-autistic controls",
    "content_hash": "23d6faf2ca79f3fa25c048bf9021d18d11f0901a1b6517da5cbaaf9fe48fa9de",
    "direction": "decrease",
    "effect_measure": "unknown",
    "effect_value": "unknown",
    "evidence_span": "Autistic adults showed reduced cortical thickness compared with controls.",
    "evidence_tier": "abstract",
    "excluded_from_synthesis": false,
    "intervention": "autism diagnosis",
    "locator": "abstract",
    "outcome": "cortical thickness",
    "pmid": "90000001",
    "population": "autistic adults",
    "source_hash": "",
    "status": "active",
    "study_design": "cohort",
    "study_type": "cohort",
    "timepoint": "unknown",
    "uncertainty_interval": "unknown",
    "version_id": "v-ff60b1f75514"
  }
]

Stable claim IDs across reruns

claim_registry.json stores every claim ever committed for the paper. On a rerun, assign_claim_ids compares each incoming claim with the claims that were active at the same locator before the run started:

• Same content hash: the claim keeps its claim_id.
• No hash match, and exactly one unconsumed prior claim at that locator: the new claim gets a new ID with supersedes; the old one becomes status: superseded with superseded_by, and stays in the registry.
• No hash match, and zero or several prior claims: the claim gets a fresh ID and no supersession is asserted.

The version manifest records schema_version (phase4-v1), extractor, model, prompt_version, study_type_confidence and claim_count.

Relations & edges

graph/relations.jsonl holds typed edges between concepts. Each edge names the claims that support it. validate_relation accepts five types:

Required edge fields: relation_id, type, subject_concept_id, object_concept_id, supporting_claims[] (at least one {pmid, claim_id}), source_version_ids[], review_state (unreviewed|reviewed), rationale (str or null), stale (bool), created_at, updated_at. relation.py refresh, which also runs under /ref:index, sets stale: true when a supporting claim is no longer active. The review decision and rationale are kept.

Researcher and grant edges

graph_people.py regenerates graph/people_relations.jsonl as a separate store:

Corrections overlay lifecycle

/ref:verify appends entries to papers/<pmid>/corrections.json. Entries are never edited in place, except that their status flips. Shape (validate_correction): correction_id (cor-…), target_type, target_id, decision (accept|edit|reject), reviewer, timestamp, evidence_locator, status (active|pending_review). A claim review also stores original_value, replacement_value and rationale.

1. Recorded with status: active. A claim review fails if the claim_id is not in the registry. Reviewing a claim that has already been superseded is allowed.
2. Reject also sets excluded_from_synthesis: true on the registry entry. The claim is kept for audit, and retrieval skips it because it queries status='active' AND excluded_from_synthesis=0.
3. Evidence changes. After every extraction commit, and on verify.py show, revalidate_corrections flips any claim correction whose target now has superseded_by to pending_review. Corrections on unchanged claims are left untouched.
4. Re-review. A new decision is appended against the current claim. Earlier entries stay in the file.

Projects, reading states, screening

A paper exists once in papers/ but can be a member of many projects. Each membership has its own state.

// projects/thesis-ch3/project.yaml  (JSON content)
{"slug": "thesis-ch3", "scope": "…", "questions": [{"id": "q1", "text": "…"}]}

// projects/thesis-ch3/papers.yaml — one entry per member
{"papers": [{"pmid": "…", "relevance": null, "priority": null, "reading_status": null,
             "why_saved": null, "screening": null, "added_at": "…"}]}

• Reading states (project.READING_STATES): to_screen, to_read, reading, read. Only /ref:project add-paper --reading-status and /ref:queue set them. A new membership starts at null. Acquisition and extraction never change them.
• Question IDs are slugs that must be unique within their project. The same ID can appear in another project.
• Screening (screen.py decide) appends {pmid, decision, reason, timestamp, search_run?} to screening.jsonl, where decision is included|excluded|pending and reason is required. It also copies the latest decision onto membership.screening. Screening a non-member adds it as a member. history reads the full log.
• Excluding a paper in one project has no effect on its membership in any other project.

Studies, datasets, methods

These are three separate JSONL registries in studies/. Their IDs are library-global slugs. A shared dataset does not make two papers the same study, and only create-study groups papers.

compare.py groups table rows by recorded study, prisma.py reports studies separately from reports, and --study <id> selects every PMID in a study.

Researchers, authorship, funding, citations

Researcher identity

people/<slug>.json: slug, name_variants[], orcid (optional), affiliations[], confirmed_publications[] ({pmid, author_index}), candidate_publications[] (from /ref:discover), rejected_publications[] (kept so the same candidate is not suggested again). Identity is a confirmed link to one specific author position. Name similarity alone never creates one.

Authorship roles

authorship.json keeps the author list in the order PubMed returned it (last, first, raw, optional affiliation/is_group, plus complete, default true). publications.classify_role assigns exactly one role:

The independent flags shared_first, shared_senior and corresponding come only from verify.py review-author-contribution, which refuses to run without --evidence-statement.

Funding observation kinds

funding.json also has an overall state, which starts at not_checked when there are no GrantList entries. fetch.py only raises it to a stronger kind. PLAN.md §3c also lists user_assigned_output; no script writes that kind.

Citation observations (D25)

papers/<pmid>/citations.json is a list that audit.py citations only appends to. validate_citation_observation accepts three entry kinds:

// a real observation
{"source": "pmc_elink", "query": "<exact query>", "count": 12,
 "coverage": "citing articles indexed in PMC; not a total citation count", "retrieved_at": "…"}
// a failed lookup — never a zero
{"status": "check_failed", "reason": "…", "retrieved_at": "…"}
// no PMCID, so no PMC cited-by is possible
{"status": "no_pmcid", "reason": "paper has no PMCID", "retrieved_at": "…"}

Markers must not carry a count. latest_real_observation skips markers, so a failed check leaves the previous dated count visible. show-citations prints null when there is no real observation, which means unknown, not zero. The staleness threshold defaults to 180 days (--stale-days). The count in the example above is illustrative. As commands/ref-audit.md states, the PubMed MCP server in this environment has no cited-by count source, so --citations has nothing real to record until one is connected.

Identifier contract

There are three kinds of identifier, defined in lib_ids.py (PLAN §3d, D27).

1 · Slugs: user-minted, human-facing

SLUG_RE = re.compile(r"^[a-z0-9]+(-[a-z0-9]+)*$")   # and 1 ≤ len ≤ 64

Only lowercase letters, digits and single inner hyphens are allowed: no leading, trailing or doubled hyphen. A collision raises SlugError naming the existing record, and no suffix is added. rename_slug renames under the library lock and records the old slug as an alias in index/aliases/<kind>.json, so resolve_slug still follows it.

2 · Citekeys: the one exception that gets a suffix

make_citekey_base(last, year, word) returns clean(last) + year + clean(word), where clean lowercases and drops everything outside [a-z0-9]. add.py takes the first author's last (else the first token of raw, else anon), year (else n.d.), and the first [A-Za-z0-9]+ run of the title (else untitled). allocate_citekey appends -2, -3, … under the library lock and records the result in index/citekeys.json. A citekey is never regenerated. Real output from two papers with the same author, year and first title word:

90000001: added (citekey=rivera2024cortical)
90000002: added (citekey=rivera2024cortical-2) — no abstract available — metadata-only record, not fabricated (§3a)

3 · Opaque IDs: machine-facing

gen_opaque_id(prefix) returns the prefix plus secrets.token_hex(6), i.e. 12 hex characters, generated at commit time. Prefixes in use include v- (versions), c- (claims) and cor- (corrections). Nothing outside the library should parse or construct one. Content hashes (raw/<sha256>/, content_hash) belong to this class too.

External codes

PMID is the identity key for papers (D11). add.py deduplicates only by PMID, and a DOI shared with another PMID produces an inconsistency: warning without merging. ORCID is an optional attribute of a person. Award numbers live in grants/<slug>.json as award_number and approved_aliases. MeSH terms and other codes resolve to a concept's slug through its aliases.

Selector grammar

Every set-valued script wires the same arguments through lib_selector.add_selector_args and resolves them with resolve() before doing any work.

[pmids ...] [--project P] [--question Q] [--screened {included,excluded,pending}]
[--read] [--queue STATE] [--query SLUG] [--run RUN_ID] [--search EXPR]
[--from-file PATH] [--study ID] [--concept ID] [--tier {abstract,full,any}] [--exclude [PMID ...]]

resolve() returns the list together with the expression and counts. Callers copy selector_expression and pmids into their artifact manifests: compare, summarize, appraise, brief, check_citations, export and export_papers all do.

{
  "pmids": ["…"],
  "selector_expression": "--project thesis-ch3 --screened included --tier full",
  "report": {
    "count": 12,
    "by_extraction_tier": {"abstract": 0, "full": 12, "unavailable": 0, "missing_record": 0},
    "by_human_verification_state": {"not_yet_tracked": 12},
    "by_retraction_errata_status": {"none": 11, "unknown": 1}
  }
}

Key design decisions

Decisions from PLAN.md §0 that the implemented phases put into practice.