AfaraBook a call
← All articles

November 17, 2026

Medical Imaging AI: Why Scan-Level Annotation Quality Decides Model Performance

Medical Imaging AI: Why Scan-Level Annotation Quality Decides Model Performance

Abstract scan grids and annotation boxes representing careful review of medical imaging data

Meta description: Medical imaging AI depends on reliable clinical annotations and independent evaluation. Learn how to build, measure and validate imaging datasets.

Medical imaging AI works when its labels reflect the clinical question and its performance is tested on genuinely independent patients. Better architecture cannot compensate for inconsistent contours, missing negative cases, or a test set that shares patients with training data. Annotation and evaluation quality shape what a model learns, and whether its reported results are meaningful.

Where imaging models actually break

Many problems start before model training. A few common ones can make a model appear stronger, or weaker, than it is.

  • Label noise: Readers draw inconsistent boxes or use labels differently. The model is then trained on conflicting examples.
  • Slice-versus-volume mismatch: A label on one CT or MRI slice may not answer a question about the whole scan. A lesion may appear across multiple slices, and its context can matter.
  • Missing negatives: A dataset with few normal scans or lookalike findings does not show how the model behaves when the target condition is absent.
  • Reader disagreement: Some findings are genuinely ambiguous. Treating one reader’s opinion as unquestioned truth hides that uncertainty.
  • Dataset leakage: If images from one patient appear in training and testing, the model may recognise patient-specific patterns instead of learning to generalise.

These issues affect radiology AI, but the same logic applies to pathology slides, dermatology images and ophthalmology scans. The label must match the prediction task, and evaluation must reflect how the system will be used.

Why the reader has to understand the clinical question

A bounding box from someone who has never reported a scan is a guess if they do not understand what the box is meant to represent. Is it the visible boundary of a finding, the region that supports a diagnosis, or an area that needs follow-up? These are different annotation tasks.

Start with the use case. For example, “find a pulmonary nodule on a chest CT” needs clear rules for what counts as a nodule, how to handle uncertain findings and whether the label covers each slice or the full 3D lesion. Without those decisions, two careful annotators may produce different labels for the same scan.

Clinical expertise does not remove all disagreement. It helps make the task and its limits explicit. Use credentialed clinicians whose scope fits the work (such as radiographers, radiologists, physicians, pathologists or ophthalmologists) and provide the full imaging context they need to apply the protocol.

What a defensible annotation protocol looks like

A useful protocol is a working guide for consistent decisions, not simply a list of label names. Before annotation begins, write down:

  1. A label ontology: The classes, definitions and relationships between labels. Distinguish, for example, “not present,” “not assessable” and “uncertain.”
  2. The clinical task: What the model will predict, for whom, and at what level, image, series, study or patient.
  3. Annotation rules: How to draw a boundary, handle multiple findings, record uncertainty and mark technically inadequate images.
  4. A pilot batch: Have readers label a small, varied sample independently. Review disagreements, revise ambiguous instructions and repeat as needed.
  5. A multi-reader plan: Define which cases receive independent review, how disagreements are adjudicated and who has final decision authority.
  6. Versioned records: Keep the protocol version, annotator role, review history and changes linked to each annotation.

Do not assume a consensus label is automatically correct. An adjudicator should have the expertise to resolve the specific question, and the record should preserve the original reads as well as the final outcome. For some tasks, retaining uncertainty or multiple expert labels is more informative than forcing a single answer.

Measuring agreement properly

Agreement metrics should fit the annotation task. They describe how closely readers match; they do not, by themselves, prove that the label is clinically valid.

For segmentation, the Dice coefficient measures overlap between two masks: 1 means perfect overlap and 0 means none. Dice is useful for comparing contours, but a single average can hide poor performance on small lesions. Consider reporting results by lesion size and adding a boundary measure when the distance between contours matters.

For categorical classification by two readers, Cohen’s kappa measures agreement beyond what would be expected by chance, based on each reader’s class frequencies. For more than two readers, a multi-rater measure such as Fleiss’ kappa may be appropriate. If categories are ordered, such as severity grades, weighted kappa can give partial credit to near-matches.

Percent agreement is easy to explain but can mislead when classes are imbalanced. If most scans are negative, readers can agree on many negatives while disagreeing on the rarer positive cases that matter most. Report agreement alongside class counts and class-specific results. There is no universal kappa or Dice cutoff that makes every imaging dataset acceptable; set thresholds for the task and clinical risks before the full annotation run.

Abstract comparison of two scan contours with measurement guides for evaluating annotation agreement

Evaluation that survives regulatory review

A good evaluation asks whether the model works on patients it has not seen, not whether it can reproduce its training examples. Split at the patient level: all images, series and studies from one patient must stay in one partition. The RSNA guide to cross-validation for medical imaging explains why patient-level separation matters when patients contribute multiple examinations.

Where the data allow, hold out sites as well as patients to check whether performance depends on a particular scanner, protocol or institution. Keep training, validation and test sets distinct. Do not repeatedly use the final test set to choose models or thresholds; doing so turns it into part of development.

Build a test set around the intended use. Include relevant positive and negative cases, clinically important subgroups and realistic acquisition variation. Then perform structured error analysis: review false negatives, false positives, borderline findings and failures by site, modality and relevant patient group. For segmentation, inspect where contours diverge and whether the differences change the clinical interpretation.

For a regulatory submission, documentation should make the evidence traceable: how the data were collected and selected, how the reference standard was established, who annotated, what training and quality checks they received, and how disagreements were handled. The FDA’s AI-enabled device software guidance discusses these kinds of lifecycle and submission considerations. It is draft guidance, not a guarantee of approval. Requirements depend on the device and submission.

If data include protected or personal information, plan access, de-identification and governance from the start. HIPAA and GDPR may apply depending on the data and organisations involved; an IRB or ethics review may also be relevant to the research setting. DICOM metadata can help link images to studies and patients, but identifiers need careful handling. See the DICOM standard for information about the format.

A short checklist before you buy or build an imaging dataset

Before development begins, check that:

  • Labels match a defined clinical question and prediction level.
  • The ontology distinguishes negative, uncertain and unassessable cases.
  • Annotation instructions are written, versioned and tested in a pilot.
  • Annotators have relevant clinical credentials and task-specific guidance.
  • Independent reads, adjudication and quality review are documented.
  • Agreement metrics fit the task and are reported with class counts.
  • All images from one patient stay in a single data split.
  • Test data represent the intended population and are kept out of model selection.
  • Error analysis and annotation provenance are available for review.

FAQ

How many annotated images do you need to train a medical imaging model?

There is no single number that works for every task. The amount depends on the condition’s prevalence, image variation, label complexity and model approach. Start with a pilot that tests annotation consistency and data coverage. Then use learning curves and performance estimates on patient-separated data to see whether more examples improve results.

What is inter-reader agreement in radiology AI?

It is the degree to which different qualified readers give the same labels or contours to the same cases. It helps reveal ambiguous instructions, difficult findings and uncertainty in the reference labels. The right metric depends on whether readers are classifying, grading or segmenting.

Can AI annotate medical images without human radiologists?

AI can assist with pre-labelling, prioritisation or quality checks. For clinical-grade datasets, human review remains important, especially for uncertain cases, rare findings and reference labels used to assess a model. The appropriate level of review depends on the task and intended use.

What is patient-level splitting in medical AI?

It means assigning every image and examination from one patient to only one dataset partition: training, validation or test. This prevents related scans from appearing on both sides of an evaluation and giving an overly optimistic estimate of performance.

A practical next step

If you are assessing a dataset or planning an annotation pipeline, start by checking whether the label definitions and patient-level splits match the intended clinical use. Afara is a credentialed healthcare professional network, including physicians, nurses, pharmacists and radiographers from the global south, supporting clinical-grade annotation and evaluation.