Hospital BenchDossier01
2023 paper, arXiv v2
EHRSHOT ↗
The denominator is a prediction time, not just a patient.
UnitA labeled prediction event; patients can contribute multiple labels.MeasureAUROC and AUPRC
Hospital prediction and decision benchmark analysis
Two views of hospital evidence.
Independent EHRSHOT and MIMIC-CDM analysis: longitudinal prediction, few-shot denominators, diagnostic simulation and the limits of hospital benchmark transfer.
Prediction and diagnosis follow different paths
- 01Fix patient partitions
- 02Define prediction time
- 03Sample training and validation labels
- 04Evaluate held-out labels
AUROC and AUPRC [1]
- 01Select the eligible cohort
- 02Reveal information
- 03Collect the conclusion
- 04Score by pathology
Per-class diagnostic accuracy [4]
2,295 train; 2,232 validation; 2,212 test. EHRSHOT partitions refer to patients. MIMIC-CDM covers four selected abdominal pathologies.
01 /
Inspect the task
Trace the actual input, output and evaluation setting for each named benchmark.
02 /
Read the evidence
Explore cited cohort counts and selected historical results with their measurement boundaries.
03 /
Make the inference explicit
Separate the authors’ observations from our analytical interpretation and proposed evaluation questions.
Our analytical question
EHRSHOT tests prediction from longitudinal coded records. MIMIC-CDM tests diagnostic decisions from selected retrospective cases. This independent analysis puts their inputs, denominators and information settings side by side. Explore what a prediction label represents, why a selected disease cohort cannot establish general emergency-care performance, and where published evidence stops. We report aggregate metadata and historical study results, with no patient material, new model runs or implied affiliation with the benchmark authors.
Read the original evidence closely
The benchmark, unpacked.
Dossier02
2024 paper; dataset metadata v1.1
MIMIC-IV-Ext Clinical Decision Making ↗
Requesting evidence is harder than receiving the finished case.
UnitOne patient case within a selected abdominal pathology.MeasurePer-class diagnostic accuracy
An original analytical tool
Prediction or diagnostic action?
Compare EHRSHOT’s prediction targets with MIMIC-CDM’s information settings. Each row states the actual unit being scored.
8 of 8 evidence entries shown
- Input
- Coded record up to prediction time
- Output
- Binary outcome
- Measure
- AUROC / AUPRC
- Input
- Coded history before discharge
- Output
- Binary outcome
- Measure
- AUROC / AUPRC
- Input
- Coded record during admission
- Output
- Binary outcome
- Measure
- AUROC / AUPRC
- Input
- Prior structured record
- Output
- Four-way class
- Measure
- AUROC / AUPRC aggregation
- Input
- Prior coded timeline
- Output
- Future diagnosis label
- Measure
- AUROC / AUPRC
- Input
- Prior coded record
- Output
- Fourteen-label finding prediction
- Measure
- AUROC / AUPRC
- Input
- History, then requested findings
- Output
- Diagnosis and treatment proposal
- Measure
- Per-class diagnostic accuracy
- Input
- Prepared case evidence up front
- Output
- Diagnosis
- Measure
- Per-class diagnostic accuracy
Different cohorts, outputs and metrics make a combined hospital-accuracy number inappropriate. [1][4]
Original analysis / methods and interpretation
What the score leaves unsaid.
Benchmark analysis
How many examples does an EHRSHOT result really use?
Separate EHRSHOT’s patients, prediction labels, pretraining population and positive/negative few-shot sampling.
Benchmark analysis
What does MIMIC-CDM reveal about gathering diagnostic evidence?
Interpret interactive and full-information diagnostic results in the restricted four-condition MIMIC-CDM cohort.
Benchmark analysis
When should a hospital use EHRSHOT or MIMIC-CDM?
Choose between structured-record prediction and diagnostic simulation by matching the benchmark’s inputs, targets and evidence limits.
Questions, answered
Read the result in context.
Specific tasks. Stated conditions.
Inspect every source.
Does EHRSHOT evaluate clinical note understanding?
The released benchmark uses structured coded records and excludes free-text notes. Its results do not directly establish note reading or summarization performance.
Is an EHRSHOT label the same as a patient?
No. One patient can contribute multiple labeled prediction events. The paper reports patient and label counts separately.
What is MIMIC-CDM-FI?
It is the full-information comparison setting, in which a prepared case is supplied up front. The interactive setting requires the model to request findings.
Does MIMIC-CDM represent every abdominal-pain presentation?
No. It selects four target pathologies and requires specified modalities. Its reported accuracy cannot establish general emergency-care specificity or positive predictive value.
Working tool / saved on this device
Prepare a reproducible benchmark reading
Use this secondary worksheet to record the version, task and evidence boundary of the benchmark you are considering. Checked items indicate documented decisions, not measured performance.
Evidence you can inspect. Benchmark dossiers distinguish published facts from our interpretation, with source versions and access notes attached.
Download the evidence ↗