AI & HEALTH · OUR STANDARD
How We Test AI Health Answers
Anyone can publish an opinion about AI health answers. We intend to publish evidence. This page sets out, in advance, exactly how MedicalVideos.com tests the major AI systems on health questions: how the questions are chosen, how answers are graded, what counts as a dangerous error, and how results will be reported. Publishing the method before the first results exist is deliberate — it commits us to a standard we cannot quietly adjust afterwards.
By Dr. Mohamed Ibrahim, MD, PhD — Assistant Professor of Medicine, Transplant Nephrology, University of Maryland School of Medicine
✓ Medically reviewed by Mohamed Magdy Ibrahim, MD, PhD — Assistant Professor of Medicine, Transplant Nephrology, University of Maryland School of Medicine · Board certified in Internal Medicine (ABIM) · Last reviewed: November 2026
AI & Health is the MedicalVideos.com section on artificial intelligence in health care. We recommend no commercial product; where we test AI systems, we publish our method and our full results.
The guide
The picture today
The question set is built to resemble real use, not examinations. Fifty questions written the way patients actually write — incomplete information, everyday language, sometimes a wrong assumption built in — spanning symptoms, medications, lab results, chronic disease, children's health, and mental health, with a deliberate share where the safe answer is 'seek care now'. Questions are fixed before testing and published with the results, so anyone can check the set was not chosen to favour an outcome. No real patient data is used; every scenario is synthetic.
Strengths and limits
Grading is done by a clinician against current guidelines and standard references, on three dimensions. Accuracy: is the medical content right? Safety: does the answer direct urgency correctly, avoid dangerous reassurance, and refuse requests it should not fulfil — such as personal dosing? Completeness: does it include the caveat or next step a responsible answer needs? Separately, any answer that could plausibly lead to harm is flagged as a safety failure regardless of its other scores, because an answer mostly right and dangerously wrong in one line is a dangerous answer. Fabricated references are counted as their own category.
In practice
The protocol controls the obvious biases. Each system receives the same questions in fresh sessions, as a first-time user would ask them — no coaching, no follow-up prompts that steer toward a better answer. Answers are recorded in full and graded against the rubric; where grading involves judgement, the reasoning is published with the grade. Systems are identified by name and version, and the test date is stated, because these products change constantly and a result is a snapshot, not a verdict for all time. Full answer texts will be published alongside the scores, so readers can grade our grading.
Red flags
The reporting standard is part of the method. Results will be published as a benchmark report on this site, updated on a regular cycle as systems change, with changes between editions explained rather than silently replaced. If a test uncovers an answer pattern that looks actively dangerous — for example, a system repeatedly reassuring a classic emergency presentation — we will say so plainly, with the evidence. And if our own earlier pages are contradicted by what we find, we will correct them and date the correction.
Common questions
When will the first results be published?
The first benchmark report is in preparation. We will not announce findings before the full cycle — question set finalised, answers collected, grading completed — is done, because partial results invite the cherry-picking this method exists to prevent. This page will link the report the day it publishes.
Which AI systems will be tested?
The major general-purpose consumer chatbots that patients actually use, identified by name and version in the report, tested through their standard public interfaces. The list will be fixed in the report itself, since the market changes; the method does not depend on any particular system.
Why publish the method before the results?
Because a method published afterwards can be shaped to flatter the findings. Committing to the question style, the grading dimensions, and the harm-flag rule in advance — publicly, with a date on it — is what makes the eventual results worth trusting, including by readers who disagree with our grades.
Can I suggest a question for the test set?
Yes — realistic patient-style questions are the raw material of this work, and reader suggestions help keep the set honest. Send them through the site's contact routes. Do not include identifying health details; questions are rewritten as synthetic scenarios before use.
More in Our Standard
Related
Sources
- Ethics and Governance of Artificial Intelligence for Health (World Health Organization)
- AI Risk Management Framework (National Institute of Standards and Technology (NIST))
- Evaluating Health Information (MedlinePlus, US National Library of Medicine)
This page is educational content about artificial intelligence and health. It is not medical advice and does not replace your own clinician's judgment. Never use an AI tool in an emergency: if you think you may be having a medical emergency, call your local emergency number (911 in the United States) or go to the nearest emergency department. Until the first report is published, nothing on this page should be read as a rating of any AI product.