📌 Highlights
The paper's central result is that conversational interaction materially improves the perceived usability and practical acceptability of clinical text-to-SQL for healthcare professionals. The authors report a mean SUS increase from 68.2 for the single-turn interface to 85.9 for the conversational interface, with significant improvements across positive acceptance constructs and strong forced-choice preference for the conversational system.
The dialogue analysis adds an important builder-facing finding: realistic clinician-authored questions are much more likely to require clarification than predefined benchmark questions. Most clarification triggers involved clinical or EHR-domain decisions rather than surface wording alone.
- 14 physicians completed the main within-subject comparison.
- 93% reported greater overall satisfaction with the conversational interface.
- 86% preferred it for real-world adoption and 71% trusted its results more.
- Clarification was triggered by 24.8% of free questions versus 1.8% of fixed questions.
- The main caveat is limited external validity from the small cohort and MIMIC-IV Demo setting.
🎯 Introduction
The task is natural-language access to relational EHR data. Clinical information is distributed across interconnected tables and heterogeneous terminology, so extracting a cohort can require both SQL proficiency and detailed knowledge of the database schema. LLM-based text-to-SQL reduces the need to write SQL directly, but prior medical evaluations have emphasized SQL generation and execution accuracy on fully specified benchmark questions.
The paper argues that this evaluation setting omits a central property of clinical use: healthcare professionals may not know or state every cohort definition, measurement operationalization, aggregation choice, or data-provenance requirement in their initial question. The study therefore asks whether conversational interaction helps real users express intent, resolve misunderstandings, trust returned answers, and consider adoption, while also measuring how often clarification is needed in naturally formulated queries.
🔬 Methodology
The main study used a counterbalanced within-subjects design in which 14 physicians used both interfaces. Phase 1 used three participant-selected questions from a predefined pool of 45 MIMIC-IV Demo queries: 15 requiring one table, 15 requiring two tables, and 15 requiring three tables. Phase 2 allowed unrestricted clinical or research queries. The conversational interface could request clarification before SQL generation and suggest follow-up tasks; the single-turn interface inferred intent and returned an answer with an explanation in one interaction.
The system queried the MIMIC-IV Clinical Database Demo, a publicly available de-identified subset containing structured data for 100 ICU patients, including admissions, ICU stays, and procedures, but no free-text clinical notes. GPT-5.4 was selected for the main study based on response latency and stable tool-calling performance. Usability was measured with SUS and UTAUT constructs, while dialogue logs from the main study and two pilot studies were annotated into Q, FX-Q, SF-Q, CL-Q, ANS, S-ANS, and C-ANS turn types. The extracted SQL snippets show the kind of filtering constraints involved in the task.
SQL predicate (page 4)
Meaning: This predicate filters the clinical records to rows whose diagnosis matches the specified condition.
SQL predicate continuation (page 4)
Meaning: This condition further restricts the query to records with sequence number 1, illustrating how clinical cohort or event-selection criteria are encoded in SQL.
📊 Experiments
The evaluation used the MIMIC-IV Clinical Database Demo rather than a conventional broad text-to-SQL benchmark. The controlled phase used 45 predefined clinical queries spanning one-, two-, and three-table tasks; the open-ended phase captured naturally formulated questions. SUS scores range from 0 to 100, with higher values indicating greater usability. UTAUT measures covered Performance Expectancy, Effort Expectancy, Self-Efficacy, Anxiety, Behavioral Intention, and the average of the four positively oriented constructs. Paired t-tests or Wilcoxon signed-rank tests were selected after testing normality of paired differences.
The conversational interface improved mean SUS from 68.2 to 85.9. SUS, PE, EE, SE, BI, and Avg. UTAUT (P) were significantly higher for the conversational interface at p<0.05, while ANX did not differ significantly. Reported effect sizes were large for SUS, PE, BI, and Avg. UTAUT (P), with |dz|>1.3, and medium-to-large for EE and SE, with |dz|=0.75 and |dz|=0.72. In forced-choice comparisons, 13 of 14 participants reported greater satisfaction with the conversational interface, 12 preferred it for real-world adoption while two would use both depending on the task, and 10 reported greater trust while four reported comparable trust.
The dialogue analysis covered 183 sessions from 21 healthcare professionals across the main and pilot studies. Forty-two clarification exchanges were observed; 40 were successfully resolved, and 36 of those were resolved in a single round. Cohort or population definition was the most frequent trigger at 35.7%, followed by operational definition of a measure at 23.8% and aggregation unit or grouping criterion at 19.0%. Together, cohort definition, measure operationalization, and measurement source or provenance accounted for 64.3% of clarification exchanges. No conventional model-accuracy ablation is reported in the main-body evidence.
tab:clarification_occurrence
Caption: Clarification occurrence and resolution in the conversational multi-turn interface, and user acceptance of agent-proposed follow-up suggestions.
| Preceding Turn Type | Turn Sequence | C-ANS / Total | Rate |
|---|---|---|---|
| Question (Q) | Q → C-ANS | 40 / 161 | 24.8% |
| Fixed Question (FX-Q) | FX-Q → C-ANS | 1 / 56 | 1.8% |
| Suggested Follow-up Question (SF-Q) | SF-Q → C-ANS | 1 / 60 | 1.7% |
| Clarification Response (CL-Q) | CL-Q → C-ANS | 6 / 46 | 13.0% |
| Total | 48 / 323 | 14.9% |
Why It Matters: This table directly quantifies how much more clarification is required for freely formulated questions than for fully specified questions.
tab:clarification_resolution
Caption: Clarification occurrence and resolution in the conversational multi-turn interface, and user acceptance of agent-proposed follow-up suggestions.
| Required Rounds | Turn Sequence | Exchanges (Turns) | Proportion |
|---|---|---|---|
| 1 round | C-ANS → CL-Q | 36 (36) | 85.7% |
| 2 rounds | C-ANS → CL-Q → C-ANS → CL-Q | 2 (4) | 4.8% |
| 3 rounds | C-ANS → CL-Q → C-ANS → CL-Q → C-ANS → CL-Q | 2 (6) | 4.8% |
| Unresolved | C-ANS → (no further CL-Q recorded) | 2 (2) | 4.8% |
| Total | 42 (48) | 100% |
Why It Matters: Most clarification exchanges were resolved in one round, but the table also exposes multi-round and unresolved cases that matter for interaction design.
tab:followup_acceptance
Caption: Clarification occurrence and resolution in the conversational multi-turn interface, and user acceptance of agent-proposed follow-up suggestions.
| Category | Turns |
|---|---|
| All S-ANS turns | 249 |
| Excluded: Session-final S-ANS turns (unobservable) | 66 |
| Eligible S-ANS turns | 183 (= 249 - 66) |
| Accepted (followed by an SF-Q) | 60 |
| Acceptance rate | 32.8% (60/183) |
Why It Matters: The acceptance analysis distinguishes the availability of follow-up suggestions from their immediate usefulness to users.
fig:design
Caption: Overview of the user-study design. Participants completed a within-subject comparison of single-turn and conversational multi-turn text-to-SQL interfaces, consisting of a controlled comparison on pre-defined questions (Phase 1) followed by open-ended querying (Phase 2).
Why It Matters: This figure clarifies how the study separates controlled interface comparison from more ecologically realistic open-ended querying.
fig:participant_background
Caption: Distribution of participant background characteristics ($n=14$), including EHR (or EMR)/CDW usage frequency, SQL proficiency, statistical software proficiency, and conversational AI usage frequency.
Why It Matters: The participant profile establishes that the main cohort had substantial EHR experience but generally limited direct database-querying expertise.
fig:usability
Caption: System usability, technology acceptance, and interface preference for the single-turn versus conversational multi-turn interfaces ($n=14$). (a) SUS scores (0--100); the dashed line indicates the industry-average score of approximately 68. (b) UTAUT construct scores (1--5): Performance Expectancy (PE), Effort Expectancy (EE), Self-Efficacy (SE), Anxiety (ANX), Behavioral Intention (BI), and the average of the four positively oriented constructs, Avg.\ UTAUT (P). Bars represent the mean and error bars indicate the standard deviation. Statistical significance is based on the primary paired test for each metric (Section~sec:method:usability_analysis:statistical_comparison_procedure): $^*p<0.05$...
Why It Matters: This is the central outcome figure: it combines usability, technology acceptance, anxiety, and forced-choice preference evidence for the two interfaces.
fig:clarification
Caption: Distribution of the 42 clarification exchanges by dominant trigger category. Categories shaded in blue indicate triggers requiring EHR- or clinical-domain knowledge.
Why It Matters: The figure identifies which kinds of ambiguity a clinical text-to-SQL system must resolve conversationally.
🔮 Conclusion
The authors conclude that conversational multi-turn interaction is an important component of practical natural-language access to EHR data. Across usability, technology acceptance, satisfaction, adoption preference, and perceived trust, the conversational interface consistently outperformed or matched the single-turn interface in this cohort.
The dialogue evidence explains why: naturally formulated clinical questions frequently contain unresolved domain decisions that cannot be reliably inferred from surface text alone. At the same time, the findings do not establish universal superiority for every task; two participants indicated that they would prefer the single-turn interface for simple, well-defined questions requiring quick answers.
🛠️ Future Research Improvements
The clearest next step is to validate the interaction findings with larger cohorts, more professional roles, and full-scale clinical databases. The current evidence comes from 14 physicians in the main comparison and a MIMIC-IV Demo dataset containing structured data for 100 ICU patients, so performance and interaction needs may differ in larger, noisier, or more longitudinal environments.
Future systems should improve context-aware follow-up suggestion generation, since only 32.8% of eligible suggestions were followed by an accepted suggested question. They should also evaluate clarification policies against SQL correctness, execution reliability, clinical completeness, and user verification behavior, not only perceived usability.
The reported concerns about context preservation, excessive clarification, incomplete clinical-variable coverage, and response reliability suggest testing explicit ambiguity taxonomies, structured clarification controls, provenance displays, and recovery behavior when the database cannot support the requested analysis.
🏭 Potential Industry Use Scenarios
The evidence supports cautious use of conversational clinical text-to-SQL for exploratory cohort definition, clinical research data discovery, quality-improvement analysis, and other structured EHR queries where users need to refine an information request over several turns. These scenarios align with the paper's motivation that EHR data can support research, quality improvement, and decision support but remain difficult to access directly.
A practical deployment should position the system as an interactive query assistant rather than an autonomous clinical decision-maker. Users should be able to inspect the interpreted cohort, clarify operational definitions, review data provenance, and verify returned results before using them in research or operational workflows.
A hybrid product could expose direct single-turn querying for simple, well-defined requests and conversational clarification for ambiguous or multi-step tasks, reflecting the participants' view that the two modes may be complementary.
💬 Critical Analysis
The study is strongest where it connects controlled usability measurement to naturally occurring dialogue behavior. The within-subject design controls for differences in SQL proficiency, EHR experience, and query formulation style, while the open-ended phase reveals ambiguity that fixed-question benchmarks remove. The convergence of SUS, UTAUT, forced-choice, qualitative, and turn-level evidence makes the case for conversational support more persuasive than a single satisfaction measure would.
The main inferential limitation is scope. A 14-physician study over a small structured ICU dataset can demonstrate a meaningful interaction pattern without establishing generality across hospitals, specialties, user roles, database schemas, or clinical workflows. The use of GPT-5.4 in the main study also means that the measured experience reflects one model and one interface implementation; the paper does not isolate which gains come from dialogue policy, model capability, result presentation, or other system factors.
The clarification findings are actionable but should not be interpreted as a complete taxonomy of clinical ambiguity. The authors report that cohort definition, measure operationalization, and provenance dominate the observed exchanges, yet the sample and task setting may shape that distribution. Finally, follow-up suggestions appear useful as an affordance but only moderately aligned with immediate user needs, so deployment should measure whether dialogue improves factual correctness and downstream decisions rather than relying on preference or trust alone.