You asked whether there is a human in the loop, and the vendor said yes. These are the questions to ask next, what a good answer sounds like, and the red flags worth listening for.
When you evaluate a vendor that reads, extracts or transforms health data with artificial intelligence (AI), one question now comes up in almost every meeting: is there a human in the loop? The answer is nearly always yes, which is why the question on its own rarely tells you much.
What matters is how that human is involved. Review that happens after data has reached your systems is very different from review that holds a record back until a clinically trained person has checked it. A reviewer whose corrections fix one record is very different from one whose corrections improve every record after it. The ten questions below help you find out which kind of loop a vendor has built, and they apply whether you are a digital health startup, a laboratory, a hospital or a pharma or insurer programme.
The questions are grouped into four lenses: where and when review happens, who does it, what happens next, and what proof the vendor can show.
If you have not yet seen it, our guide to the 5 levels of human-in-the-loop sets out the ladder these questions map onto, from no review at all to a closed feedback loop.
Review after data has been used measures errors rather than preventing them. By the time a spot check finds a misread result, it may already sit in a patient record, a claim or an analysis.
A good answer sounds like: "Any record that is flagged is held back until a clinically trained reviewer has checked it. Nothing flagged is released unchecked."
Red flag: "We audit a sample of outputs every month."
Many systems send a record for review only when the model is unsure. The most dangerous errors, though, are the ones a model is confident about, such as a glucose result of 5.5 read as milligrams per decilitre instead of millimoles per litre. Clinical checks catch what confidence scores miss.
A good answer sounds like: "Flags come from model confidence and from clinical checks, such as values outside a plausible range, units that do not fit the test, or test names we have not seen before."
Red flag: "Anything below our confidence threshold goes to review."
A review process that works in a pilot can struggle at scale. If the queue grows faster than the team, reviewers either fall behind or start approving items quickly, and quality quietly slips.
A good answer sounds like: "Flag rates fall as the system learns your sources, and we can show you review turnaround and flag rates over time for clients like you."
Red flag: "We will just add more reviewers," with no data on flag rates or turnaround.
A reviewer without clinical training can approve a value that looks tidy but makes no clinical sense. How reviewers are engaged also matters: people employed by the vendor are directly accountable and bound by its confidentiality and security obligations, while a loosely managed external pool is harder to hold to the same standard.
A good answer sounds like: "Our reviewers are clinically trained, employed by us, bound by confidentiality, and trained on the specific data types you send."
Red flag: "Our annotation partner handles that," with no detail on who the partner's people are.
Human review means people outside your organisation may see health information. You need to know which systems and countries the data passes through, who has access and whether that fits your obligations under local privacy law and any in-country processing requirements.
A good answer sounds like: "Here is the data flow. Reviewers see only what they need, data is pseudonymised where possible, every access is logged, and processing can stay in your country where required."
Red flag: The vendor cannot say which countries data passes through, or review is not covered in its sub-processor list or data processing agreement.
People who review AI output tend to accept what the system suggests, especially under time pressure. The EU AI Act names this automation bias explicitly in its human oversight requirements.1 A good vendor designs its review process around it rather than assuming experience is enough.
A good answer sounds like: "Reviewers work from written guidance, a share of their decisions is second-checked, and we monitor override rates and workload."
Red flag: "Our reviewers are experienced, so they just know," or an approval rate close to 100% presented as a sign of quality.
This question separates a review team from a review system. If a correction fixes only the record in front of the reviewer, the same local test name or report layout will be flagged again tomorrow, and review effort will grow with every new source.
A good answer sounds like: "Each correction is captured and, after clinical sign-off, updates the mappings for your sources, so the same issue is not flagged twice."
Red flag: "The reviewer fixes the record," and nothing else changes.
Most organisations have their own codes, test names and reporting conventions, often built up over years. A vendor that forces you onto its schema can break the continuity of patient histories and push the mapping work back onto your team.
A good answer sounds like: "We map to your ontology once at the start, then results are mapped to it automatically. If you do not have one yet, we can help you set one up for what you are trying to do."
Red flag: "You will need to adopt our standard schema," or mapping that has to be redone by hand for each new source.
Headline accuracy figures are usually measured on the vendor's own test set. What matters is performance on your documents, in your languages, formats and units, measured at the level of individual fields rather than whole documents.
A good answer sounds like: "We measure field-level accuracy after review, and we are happy to run a sample of your own documents so you can check the results yourselves."
Red flag: A single accuracy number with no explanation of how it was measured, or testing in English only.
When a clinician, auditor or regulator asks where a value came from, you need to be able to answer. Singapore's AI in Healthcare Guidelines, for example, expect healthcare organisations to put governance in place for the AI they deploy, proportionate to the potential for patient harm.2 A full trail for each record is part of being able to show that.
A good answer sounds like: "For any record we can show the source document, the AI output, what the reviewer changed and when, and the final record."
Red flag: "We keep the final values," with no record of what the AI produced or what a reviewer changed.
Human review is the part of a health data pipeline that is easiest to promise and hardest to run well. Building and keeping a clinically trained review team, with guidance, quality checks and a feedback loop, takes real time and effort, and when it is missing the work falls back on your own clinicians and staff. There are two ways we can support you.
