Appraisal is not a quality score. It is a judgement about whether a specific result can answer a specific question.

The Order, Every Time

1. Identify the result being appraised — population, intervention/exposure, comparator, outcome, effect measure, follow-up window, and whether the question is about assignment to an intervention or adherence to it.
2. Pick the instrument from the design, not from the paper's self-description. A paper can call itself "prospective" or "real-world" without changing the tool it needs.
3. Appraise the specific result, not the article's general polish. Risk can differ across outcomes inside one paper.
4. Anchor every non-unclear judgement to a span. A domain rationale without source offsets is reasoning without auditability — see Evidence kernel.
5. Convert domains to the overall judgement using the tool's algorithm. Do not average, vote, or compensate a serious domain with several reassuring ones.
6. Carry the appraisal into synthesis as a constraint on claims. A high-risk or abstract-only study can still be described; it should not support strong effect language.

Which Tool, And Why

Routing is driven by the extracted design plus the corpus record's article_types. Exactly one tool per record.

Strategy by question type

RoB 2 — Randomized Trials

Five domains, stored in canonical order:

Overall: low requires low risk in all five; some_concerns means concerns in at least one with high risk in none; high means high risk in at least one, or concerns in several domains in a way that substantially lowers confidence.

Effect of interest, and per-outcome appraisal

RoB 2 is assessed per result, for a stated effect of interest: the effect of assignment (intention-to-treat) or of adhering (per-protocol). Domain 2 branches on that choice, so name it in the domain-2 rationale. Default to assignment unless the extraction shows the question is about adherence.

It is also assessed per outcome. When a trial contributes a blinded lab measure and an unblinded self-report, appraise the one carrying the review's primary question and say which. Do not average across outcomes.

Signalling questions, and the rules that must not be short-circuited

Y  = yes      PY = probably yes      PN = probably no      N = no      NI = no information

1. Answer the signalling questions first, from the text, then derive the domain judgement. Never pick a judgement and reverse-engineer answers to it.
2. PY and PN carry the same algorithmic weight as Y and N — they encode inference from partial reporting, and the rationale must say what the inference rested on.
3. NI is not a synonym for PN. NI means the paper is silent. A domain resting on NI for a bias-critical question is at minimum some_concerns; it is never low.
4. Cluster trials add a recruitment-bias domain; crossover trials add period and carryover content. Use the variant's domain names verbatim, or name the domain descriptively and say in the rationale that the variant was applied.

ROBINS-I — Non-Randomized Intervention Studies

Overall = the worst domain judgement. No averaging, no compensation: one critical domain makes the study critical regardless of the other six.

Target-trial framing is mandatory before domain 1

ROBINS-I judges the study against the hypothetical randomized trial it is trying to emulate. Write that trial down first:

Target trial
  Eligibility : who would have been randomized
  Treatment   : the intervention strategies being compared, with start time
  Assignment  : randomized at time zero, no blinding assumed
  Outcome     : the outcome and its measurement
  Follow-up   : starts at assignment; contrasts of interest (ITT-analogue)

Two corollaries follow:

• The confounding domain needs a pre-specified confounder list. Decide what the target trial would have balanced, then ask whether the study measured and controlled it. "Adjusted for age and sex" is serious when the field's known confounders are baseline severity and socioeconomic status. State the list.
• Time-zero alignment must be checked, not assumed. Misalignment of eligibility, assignment, and start of follow-up — immortal time bias, prevalent-user bias — belongs to domains 1 and 2.

Newcastle-Ottawa — Cohort And Case-Control

Three groups, one star per item except Comparability (two), maximum 9. overall_judgement is a star-count string such as "7/9".

1. Record each item as its own domain entry — yes (star awarded) or no — with a rationale saying what earned or lost it.
2. Declare the Comparability confounders in the protocol or in the rationale. The instrument leaves "principal confounder" to the reviewer; leaving it undeclared makes the score unreproducible.
3. The NOS defines no good/fair/poor thresholds. Any cutoff is the reviewer's and must be declared before it is used. Never present a threshold as if it shipped with the instrument.
4. A star count is not a risk-of-bias judgement. Do not turn "7/9" into "low risk of bias" in prose. Report the count, the group breakdown, and the specific losses.

AMSTAR-2 — Systematic Reviews

Sixteen items, seven of them critical. The rating is driven by flaws in the critical set, weighted against non-critical weaknesses.

QUADAS-2 — Diagnostic Accuracy

Four domains rated for risk of bias; the first three additionally rated for applicability — whether the study's patients, index test, and reference standard match this review question.

Seven domain entries are stored: four risk-of-bias, then three applicability, each tagged with domain_group so the report layer groups them rather than printing seven undifferentiated rows.

The appraisal target is mandatory

QUADAS-2 appraises a specific index test against a specific reference standard, not the paper as a whole:

{
  "target_type": "index_test",
  "target_id": "troponin-I-99th-percentile",
  "population": "Adults presenting to ED with suspected ACS",
  "index_test": "High-sensitivity troponin I, single draw at presentation",
  "reference_standard": "Adjudicated final diagnosis of MI per Fourth Universal Definition"
}

A paper reporting two assay thresholds, or troponin vs. CK-MB, needs one record per index test. Do not average across them. The appraiser pulls index_test, reference_standard, threshold, verification, and interval_index_reference from the extraction's diagnostic_accuracy[] block rather than re-reading the paper — judging bias from data the extractor already anchored to spans.

Traps

• Case-control enrichment (diseased cases vs. healthy controls instead of a consecutive clinically-relevant sample) is high risk on patient selection even if everything else is well reported — enrichment inflates accuracy.
• An index test interpreted with knowledge of the reference standard (or vice versa) is high. "Blinded" claimed without saying who was blinded to what is unclear.
• An imperfect reference standard is an applicability concern on domain 3, not a risk-of-bias fix. Do not silently substitute a better one in the rationale.
• An interval long enough for the condition to resolve, progress, or be treated is high on flow and timing. State the interval when it is reported.
• Excluding indeterminate results from the analysis without putting them in the 2×2 is high on flow and timing.

There is no single QUADAS-2 overall score by design — the instrument deliberately keeps the two axes apart so a reader is not handed a false single number. This pipeline still requires an overall_judgement, so it is populated as the worst risk-of-bias domain (low < unclear < high), with the applicability picture stated separately. A low overall must never be read as low applicability concern.

PROBAST — Prediction Models

The same seven-entry shape as QUADAS-2: four risk-of-bias domains, three applicability.

A validation study of someone else's model is appraised as its own PROBAST record — never folded into the development paper's.

Checklist Tools — CASP And JBI

These three report a count, with the lost items named. They are not risk-of-bias instruments and their scores must never be relabelled as one.

The none Case

tool: "none" is a legitimate, common outcome — not a failure. The record carries domains: [] and an overall_judgement of "unclear", the reason goes into the appraiser receipt, and it surfaces in the report's risk-of-bias section as a pipeline limitation. Never invent a pseudo-domain just to hold an explanation.

{
  "schema_version": 1,
  "pmid": "12345678",
  "evidence_id": "pmid:12345678",
  "tool": "none",
  "domains": [],
  "overall_judgement": "unclear",
  "grade": null,
  "evidence_basis": "abstract_only"
}

What not to do: reach for RoB 2 because an abstract says "randomized" while no methods text exists; appraise a narrative review with AMSTAR-2; or appraise a guideline with any of these tools. If guideline quality matters to the question, say so as a gap — AGREE II is the recognised instrument and is out of scope here.

Saying "Unclear" Honestly

The single largest failure mode of automated appraisal is converting silence into reassurance.

BAD   "Allocation concealment was likely adequate given the journal's standards."
GOOD  "Allocation concealment not described (Methods, 'Randomisation' paragraph); no
       information on who held the sequence. Some concerns."

BAD   "No information on blinding, so low risk assumed."
GOOD  "Outcome assessors' awareness of assignment not reported; the outcome is a
       clinician-rated scale, which is susceptible to assessor knowledge. Some concerns."

BAD   "Confounding adequately handled."
GOOD  "Adjusted for age and sex only; baseline symptom severity and prior treatment —
       pre-specified as the principal confounders — were neither measured nor adjusted.
       Serious."

BAD   "Missing data unclear."                  (unclear with no reason: schema violation)
GOOD  "Attrition stated as 18% overall but not by arm, and no analysis of whether
       missingness relates to outcome. Some concerns."

BAD   "Study appears well conducted overall."  (global impression, no domain anchor)
GOOD  Per-domain judgements only; the overall follows the tool's algorithm.

Abstract-Only Evidence

evidence_basis is carried on both the extraction and the appraisal record, and must agree with the corpus record's fulltext.status.

For abstract-only evidence the tool is "none", domains are [], and the overall is "unclear" — unconditionally, even when the abstract states a method detail. A record with five unclear domains and one weakly-supported judgement communicates less than an honest "not appraised: abstract only". GRADE may still be applied at the body-of-evidence level, but the abstract-only studies must show up there as risk-of-bias or indirectness limitations rather than being ignored.

GRADE — Certainty Per Outcome

GRADE rates certainty per outcome, across the body of studies contributing to it — not per study. An individual appraisal record's grade object captures that study's contribution; the outcome-level rating is produced at synthesis.

When NRSI are appraised with ROBINS-I against a target trial, GRADE permits starting at High and downgrading from the ROBINS-I judgement. That choice must be declared in the protocol and stated in the report — it is not the default here.

Five downgrade domains

Three upgrade domains

Applicable only to bodies not already downgraded for risk of bias — in practice, observational evidence. Never upgrade randomized evidence that already starts High.

start (design)
  - risk of bias        (0 / -1 / -2)
  - inconsistency       (0 / -1 / -2)
  - indirectness        (0 / -1 / -2)
  - imprecision         (0 / -1 / -2)
  - publication bias    (0 / -1 / -2)
  + upgrades            (observational, unbiased bodies only)
= certainty : high | moderate | low | very_low        (floor at very_low)

• One outcome, one rating. Rate the outcomes that matter to the question, including harms.
• Do not double-count: a problem downgraded under risk of bias is not also downgraded under imprecision for the same reason.
• Record the reason for every non-zero domain. A certainty rating with no stated downgrade reasons is not a GRADE rating.
• Certainty is about the estimate, not the importance. "Low certainty" never means "no effect".

Certainty wording — use these shapes verbatim

Downgrading Claim Strength

Appraisal changes the verbs the report is allowed to use. This is the link between Stage 6 and Stage 7:

Mixed-Design Bodies

• Randomized and non-randomized intervention evidence answer different versions of the same causal question unless the non-randomized studies explicitly emulate a target trial. Keep them stratified.
• Primary studies and systematic reviews are not independent evidence for the same effect when the review includes those same studies. Use reviews for context, search leads, and consistency checks, then privilege the freshest primary extraction.
• One paper can contribute appraised evidence for one outcome and only descriptive context for another. Name the outcome appraised.
• When every usable record is abstract-only or tool: "none", the synthesis must say the body was not critically appraised by an in-scope instrument — never write certainty language implying a completed risk-of-bias assessment.

Where Appraisals Live

In standalone repo mode, appraisals are project-scoped: data/papers/appraisals/<project>/<paper>--<target>.json, one file per paper-and-target within a project, indexed from the registry's appraisals map. This is deliberate — risk of bias and applicability depend on the review question, outcome, population, index test, comparator, or model target. The same paper can be appraised for one manuscript and not_appraised for another at the same time. A paper's extraction stays in the shared pool; its appraisal does not. The wiki-backed pipeline instead writes one run-scoped record per paper under workspace/appraisals/, which has no project dimension.

Framework Citations