📌 Highlights
EndoSCOP-V reframes endoscopy multimodal evaluation as a structured, multi-turn video reasoning problem. A case is not reduced to whether a disease appears in one frame; the benchmark asks models to progress through disease identification, anatomical localization, morphology, measurement, grading, complications, clinical reasoning, and action recognition. Its 1,307 videos include 22% multi-disease procedures, with as many as six diseases in one case, and its questions are expanded from clinician-filled EHR reports.
The strongest open-weight result is Qwen3.6-27B at 59.8% strict overall accuracy and 53.4% strict NDR accuracy, while the clinician panel reaches 77.9% on both measures. The gap is not uniform: action recognition is comparatively strong, but anatomical localization, grading, and clinical reasoning remain weak. The frame study further shows that video contributes directly to biopsy and visualization recognition, whereas bleeding and stricture recognition are best supported by the oracle keyframe.
The paper's strongest methodological feature is its separation of per-turn reasoning from dialogue-error propagation. By inserting gold parent answers before dependent questions, the benchmark measures what a model can answer about a known finding. That makes the results useful for capability diagnosis, but it also means the reported scores should not be interpreted as end-to-end autonomous reporting performance.
- Core claim: long-horizon video and structured clinical dependencies expose weaknesses hidden by image-level benchmarks.
- Strongest open-weight result: Qwen3.6-27B, at 59.8% strict overall and 53.4% strict NDR.
- Reference point: the clinician panel scores 77.9% strict overall and NDR.
- Most important limitation: single-institution data with predominantly single-rater case labels.
🎯 Introduction
Gastrointestinal endoscopy is a temporal procedure in which an 8–15 minute examination produces several thousand high-resolution frames. The endoscopist must localize lesions, characterize morphology, assess severity, decide whether to biopsy or intervene, and compose a structured report that becomes part of the patient record. These decisions are conditional: once a finding is identified, follow-up questions about organ, sub-location, wall position, grade, measurement, and action become meaningful. Figure 1 captures this chain from disease identification through increasingly specific report fields.
Prior endoscopy benchmarks largely test image-level or single-turn behavior. Even where disease coverage is broad, each condition is often examined shallowly through one recognition decision, rather than through the operator-level sequence required by a clinical episode. Static protocols also miss events that unfold over time, while labels generated from captions or post hoc language-model transformations can weaken the connection between the answer and the original clinical record. EndoSCOP-V addresses these limitations by grounding questions in structured EHR fields, using native procedural video, and evaluating multi-turn reporting across 54 WEO MST 3.0 diseases plus four non-disease findings.
The paper's objective is therefore broader than building another disease classifier. It aims to measure whether multimodal models can support the reporting workflow at the level of individual clinical fields and dependent reasoning steps. The benchmark separates disease recognition from downstream reasoning, supplies a clinician reference, and reports both aggregate performance and task-specific weaknesses.
🔬 Methodology
The dataset contains 1,307 real-world procedural clips: 1,004 EGD cases and 303 colonoscopy cases. The clips total 63.2 hours and average approximately 174 seconds; 97% were recorded at 1080p. The corpus was collected under institutional ethics approval, recorded on Olympus endoscopes, and released after PHI overlays were cropped. The benchmark includes 54 diseases drawn from WEO Minimal Standard Terminology 3.0 and four non-disease findings. Multi-disease cases are retained rather than filtered out: 291 of 1,307 cases contain two or more strict diseases, with individual procedures containing up to six.
The question construction process is deliberately constrained. An LLM-assisted converter drafts candidate questions from EHR field labels, but board-certified endoscopists review every retained stem, remove questions that require unavailable evidence such as tactile feedback or histopathology, rewrite language, and author distractors. The final 1,033 templates cover nine task groups and 30 sub-tasks. Per-case questions are produced deterministically only when the relevant report fields are populated, and conditional children are emitted only when their parent answer triggers them. This produces 20,274 turns, averaging 15.5 turns per case.
Evaluation proceeds through disease identification, anatomical context, and per-disease detail phases. Models receive uniformly sampled frames and a fixed gastroenterologist system prompt; the clinician-marked keyframe is used for dataset construction and ablation but is not supplied in the main evaluation. Large-context models receive 100 frames, while Hulu-Med and ColonR1 receive 16, making frame budget a confound in cross-tier comparisons. Responses use binary, single-choice, or multi-select formats. Exact match is the primary metric, while multi-select answers also use the partial-credit formula below. Markovian context pruning and oracle forcing reveal correct parent answers before dependent turns, isolating the tested reasoning step from previous model mistakes.
The protocol is appropriate for capability diagnosis because it answers a focused question: given the relevant preceding clinical fact, can the model recover the next report field from the video? Its trade-off is equally important: the scores do not represent a natural dialogue in which the model's own earlier errors accumulate. The frame ablation separately tests 2,256 questions across four video-exclusive sub-tasks on 1,081 cases, comparing an oracle keyframe, 25 frames, 50 frames, and full video across five Qwen-family models.
Partial multi-select score
Meaning: This controls partial credit for multi-select answers: correct selections contribute recall, spurious selections are penalized at half rate, and the score is floored at zero.
📊 Experiments
The main benchmark is EndoSCOP-V, evaluated as Set A across 20,274 questions for the open-weight models, with ColonR1 scored on its eligibility-filtered subset of 9,897 questions across 668 cases and 28 disease classes. Set B is the clinician baseline: 15 endoscopists answered seven randomly drawn cases each, approximately 1,500 questions from 105 cases, and achieved 77.9% strict accuracy against EHR ground truth. Set C is a proprietary-model spot-check in which Gemini 3-flash, ChatGPT 5.4-mini, and Claude Sonnet 4.6 each answered a different random draw of approximately 100 questions. The comparison with clinicians is therefore between-subjects and directional.
Strict exact-match accuracy is the primary metric. The paper reports strict overall accuracy, strict NDR accuracy with Disease-Rejection items excluded, partial overall accuracy, partial NDR accuracy, and separate EGD and colonoscopy results. Qwen3.6-27B is the strongest open-weight model at 59.8% strict overall and 53.4% strict NDR, while the clinician panel reaches 77.9% on both. Gemini 3-flash is highest in the proprietary spot-check at 61.4% strict overall and 55.1% strict NDR, but the approximately 100-question sample has confidence intervals of approximately ±10 percentage points. The best proprietary result is only 1.6 percentage points above the best open-weight overall result.
Task-level results show a recognize-but-fail-to-characterize pattern. Disease identification is stronger than localization, morphology, grading, or clinical reasoning. The best open-weight scores are 76.5% for DI with Qwen3.5-27B, 47.2% for AL with Qwen3.6-27B, 48.3% for MA with Qwen3.6-27B, 46.5% for GS with Qwen3-VL-32B, and 46.5% for CR with Hulu-Med-7B. Action recognition is highest overall, reaching 80.7% for Qwen3.5-27B. The paper cautions that DI-N, QN-S, and AR-V each have fewer than 50 questions, so those point estimates should not support fine-grained model ranking.
The frame ablation provides the clearest evidence that the benchmark contains genuinely temporal tasks. Averaged over five models, full video improves AR-V visualization by 13.8 percentage points over the oracle keyframe and AR-B biopsy recognition by 6.5 points. AR-B peaks at 80.2% with 50 frames, while AR-V reaches 54.3% with full video. Complication recognition behaves differently: bleeding is 82.0% from the keyframe versus 76.2% from full video, and stricture and obstruction is 50.6% versus 48.5%. All four video-versus-keyframe paired tests report p < 0.001, but the direction of the effect depends on the sub-task.
The consensus check supports the broad ranking while exposing label uncertainty. Five raters re-annotated 100 cases, labeling 2,044 of 2,148 questions after excluding ties. Krippendorff's alpha is 0.77 and Fleiss' Kappa is 0.76; the recorded EHR label matches consensus on 90.7% of labeled questions. Re-scoring against consensus changes each model by at most 1.5 percentage points, with mean absolute change of 0.8 points, and the ranking remains stable. This reduces concern that the headline model gap is solely caused by single-rater noise, while leaving subjective grading and measurement as meaningful sources of uncertainty.
Table 3
Caption: Accuracy (%). Best per column in bold.
| Rank | Model | Strict Overall | Strict NDR | Partial Overall | Partial NDR | EGD | Colonoscopy |
|---|---|---|---|---|---|---|---|
| 1 | Qwen3.6-27B | 59.8 | 53.4 | 62.6 | 57.0 | 60.5 | 57.2 |
| 2 | Qwen3.5-9B | 58.4 | 51.6 | 62.0 | 56.5 | 60.3 | 52.8 |
| 3 | Qwen3.5-27B | 58.2 | 52.2 | 61.2 | 56.0 | 58.3 | 56.2 |
| 4 | Qwen3-VL-32B | 56.4 | 48.8 | 60.3 | 53.3 | 56.5 | 53.9 |
| 5 | Hulu-Med-32B | 55.0 | 46.9 | 58.8 | 51.2 | 55.9 | 52.5 |
| 6 | Hulu-Med-14B | 54.6 | 47.0 | 56.9 | 49.2 | 55.5 | 51.4 |
| 7 | Qwen3-VL-8B | 54.2 | 46.0 | 58.4 | 51.1 | 54.9 | 51.5 |
| 8 | Hulu-Med-7B | 54.2 | 45.8 | 57.1 | 49.0 | 55.0 | 50.9 |
| 9 | Hulu-Med-4B | 51.7 | 43.2 | 56.0 | 48.3 | 52.5 | 49.0 |
| 10 | MedGemma-27B | 51.6 | 43.0 | 54.4 | 47.2 | 52.3 | 47.6 |
| 11 | Qwen3.5-4B | 51.5 | 46.9 | 55.2 | 50.9 | 53.8 | 45.1 |
| 12 | MedGemma-4B | 46.7 | 37.9 | 51.3 | 43.3 | 48.0 | 43.0 |
| 13 | MedGemma-1.5-4B | 44.5 | 38.2 | 48.0 | 42.2 | 43.9 | 45.1 |
| 14 | ColonR1† | 43.4 | 35.6 | 46.5 | 39.2 | 46.9 | 40.1 |
| Gemini 3-flash | 61.4 | 55.1 | 63.3 | 57.4 | 62.3 | 59.9 | |
| ChatGPT 5.4-mini | 59.4 | 53.2 | 62.7 | 56.6 | 60.4 | 57.4 | |
| Claude Sonnet 4.6 | 58.3 | 52.4 | 61.2 | 56.1 | 59.1 | 56.5 | |
| Clinician panel | 77.9 | 77.9 | 79.1 | 79.2 | 78.7 | 77.1 |
Why It Matters: The table shows the central performance gap, the small separation between the best open-weight and proprietary spot-check results, and the consistent EGD-versus-colonoscopy split.
Table 4
Caption: Strict accuracy (%) by clinical task across the 14 open-weight models, where n is the total number of questions per task across the 1,307 cases.
| Model | DI | AL | MA | PF | QN | GS | CF | CR | AR | n |
|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3.6-27B | 76.0 | 47.2 | 48.3 | 51.6 | 64.7 | 40.3 | 61.3 | 40.7 | 79.5 | |
| Qwen3.5-9B | 75.4 | 40.1 | 45.1 | 52.6 | 52.7 | 40.9 | 59.0 | 43.4 | 77.1 | |
| Qwen3.5-27B | 76.5 | 46.6 | 43.8 | 46.9 | 58.6 | 46.0 | 59.1 | 37.5 | 80.7 | |
| Qwen3-VL-32B | 74.9 | 41.8 | 42.5 | 44.2 | 58.0 | 46.5 | 57.4 | 35.1 | 75.5 | |
| Hulu-Med-32B | 66.8 | 39.6 | 45.0 | 52.6 | 54.4 | 42.9 | 60.5 | 38.7 | 72.8 | |
| Hulu-Med-14B | 70.4 | 29.5 | 47.7 | 49.0 | 52.5 | 38.5 | 58.9 | 35.1 | 70.2 | |
| Qwen3-VL-8B | 70.7 | 38.2 | 43.9 | 46.2 | 56.7 | 45.3 | 50.5 | 35.3 | 74.1 | |
| Hulu-Med-7B | 67.2 | 36.7 | 44.5 | 46.2 | 50.7 | 42.2 | 61.5 | 46.5 | 65.7 | |
| Hulu-Med-4B | 65.2 | 34.2 | 40.2 | 44.7 | 57.0 | 40.6 | 53.6 | 42.1 | 68.6 | |
| MedGemma-27B | 58.9 | 42.2 | 41.8 | 49.8 | 38.6 | 43.7 | 59.1 | 40.9 | 72.7 | |
| Qwen3.5-4B | 61.4 | 38.8 | 42.3 | 47.5 | 56.3 | 37.1 | 52.6 | 41.1 | 76.9 | |
| MedGemma-4B | 63.5 | 35.6 | 34.3 | 41.6 | 19.4 | 38.6 | 45.8 | 29.8 | 65.2 | |
| MedGemma-1.5-4B | 60.0 | 37.6 | 32.9 | 38.3 | 26.3 | 28.8 | 45.3 | 26.8 | 64.4 | |
| ColonR1 | 61.7 | 27.1 | 36.6 | 35.7 | 29.4 | 23.6 | 44.6 | 23.7 | 64.6 |
Why It Matters: The task breakdown reveals that recognition and action-event categories are substantially stronger than anatomical localization, grading, and clinical reasoning, which aggregate accuracy alone would conceal.
Table 5
Caption: Strict accuracy (%) on the four video-exclusive sub-tasks, mean over five models, equal weight per model. Paired McNemar tests (video vs. keyframe, pooled over model × question outcomes) give p < 0.001 in all four sub-tasks.
| Video-exclusive sub-task | n | Keyframe (oracle) | 25 frames | 50 frames | Full video | Video −keyframe |
|---|---|---|---|---|---|---|
| AR-V | 41 | 40.5 | 50.7 | 51.2 | 54.3 | +13.8 |
| AR-B | 856 | 72.5 | 78.6 | 80.2 | 79.1 | +6.5 |
| CF-B | 718 | 82.0 | 77.9 | 77.2 | 76.2 | −5.8 |
| CF-S | 641 | 50.6 | 49.6 | 49.0 | 48.5 | −2.1 |
Why It Matters: The ablation distinguishes tasks that genuinely require temporal evidence from tasks for which a localized frame is sufficient, and shows that most frame-count gains arrive before 50 frames.
Figure 1
Caption: Figure 1: Endoscopy reporting is inherently multi-turn. A single finding cascades through a chain of follow-up questions (for example, Disease? →Ulcer, Organ? →Duodenum, Sub-location? → Duodenal Bulb, Wall? →Posterior, Grade? →IIB) that mirrors how an endoscopist composes a structured EHR report. EndoSCOP-V evaluates models on this multi-turn dialogue rather than on isolated single-shot recognition. Illustration generated with Gemini 3.1 Pro.
Why It Matters: The figure makes the benchmark's central design choice concrete: one observed finding produces dependent questions whose answers progressively fill a structured clinical report.
🔮 Conclusion
EndoSCOP-V shows that current multimodal models remain substantially below clinicians on structured endoscopy reporting, especially when format-trivial disease-rejection questions are removed. The strongest open-weight model reaches 53.4% strict NDR accuracy against the clinician panel's 77.9%. The deficit is concentrated in composing multiple visual cues into clinically meaningful fields: localization, multi-variable grading, measurement, and clinical reasoning.
The benchmark also establishes that video is not uniformly beneficial in the same way for every task. Temporal evidence improves biopsy and visualization recognition, while a localized keyframe is stronger for bleeding and stricture recognition. The results support an assistive reporting role in which models can draft or prioritize fields, but clinicians must verify the resulting report before clinical use.
🛠️ Future Research Improvements
The most important next step is external validation. A multi-institutional corpus and an independent clinician panel would test whether the reported difficulty gradient survives changes in equipment, patient population, reporting style, and institutional practice. The current consensus study is useful but remains within the same endoscopist panel, and most cases still have single-rater labels.
Evaluation should also move from isolated structured fields toward realistic assistance workflows. Paired comparisons on identical cases would enable stronger statistical conclusions for proprietary and open models. Future studies could measure report-editing time, clinically important omissions, calibration, uncertainty, ranked differentials, and the effect of retrieval or timestamp-aware frame selection. Because the benchmark's uniform sampling intentionally includes pre-lavage and quick-movement segments, experiments should separately report performance under slower, lavaged, well-lit viewing conditions.
Methodologically, adaptive temporal processing is a promising direction. The ablation suggests that 25–50 frames capture much of the action-recognition benefit, but full video is not always best for complications. A system that first localizes candidate events and then allocates frames to the relevant task could reduce context use without assuming that every report field needs the same temporal evidence.
🏭 Potential Industry Use Scenarios
The evidence supports assistive rather than autonomous deployment. A reporting workstation could use a model to propose structured fields, surface likely anatomical locations, flag possible biopsy or visualization events, and organize a draft report for clinician review. The benchmark's deterministic field structure is particularly compatible with systems that populate discrete EHR entries instead of generating unrestricted prose.
Video-aware quality assurance is another plausible use. Temporal models could identify whether a biopsy or visualization event occurred, while localized-frame models could support targeted review of suspected bleeding or obstruction. The frame ablation suggests that a product need not always process every frame at maximum context length; task-specific sampling may provide useful efficiency, although the paper does not measure latency, compute cost, or clinical workflow benefit.
Any deployment should preserve human verification, expose the evidence frames behind each proposed field, and distinguish high-confidence recognition from uncertain grading or etiological inference. The current gap to clinicians, the single-institution data, and the absence of free-form uncertainty evaluation make independent validation and prospective safety assessment necessary before clinical decision support.
💬 Critical Analysis
The paper is strongest where its evaluation design matches the clinical dependency structure of the task. Deterministic expansion from clinician-filled reports gives every question a traceable source, while endoscopist review of stems and distractors limits the risk that a language model defines the target. Retaining multi-disease cases is also important: real procedures can contain several pathologies, and case-level Disease-Rejection distractors prevent a finding from being incorrectly treated as absent merely because another disease co-occurs. The 100-case consensus check further shows that model rankings are not driven only by the recorded label, since re-scoring changes accuracy by at most 1.5 percentage points and preserves the ranking.
The main interpretive caution is oracle forcing. It is a principled way to isolate each reasoning step, but it removes the operational cost of an earlier recognition error. A model may score well on a dependent grading question after receiving the correct disease and still fail to produce a usable report when its own disease prediction is wrong. Conversely, the reported results are valuable precisely because they identify downstream knowledge gaps without conflating them with dialogue memory. Builders should therefore treat the benchmark as a diagnostic decomposition, not as a complete end-to-end safety test.
The comparisons also require care. Cross-tier results confound model capability with frame count because some models receive 100 frames and others 16. The proprietary comparison uses different cases from the clinician baseline and has approximately ±10 percentage-point confidence intervals. The clinician baseline is based on the same panel involved in annotation, and consensus raters come from that panel as well. Finally, some sub-task cells are small, including AR-V with n = 41, so apparent model differences at that granularity should not be overinterpreted. Despite these caveats, the consistent gap across aggregate metrics, task categories, and qualitative capability patterns supports the paper's central conclusion: current multimodal systems are not yet reliable substitutes for expert endoscopy reporting.