Can LLM Agents Be Forward-Deployed Engineers? Requirement Discovery and Delivery for Enterprise Text-to-SQL

📌 Highlights

The paper studies an end-to-end FDE task: discover customer requirements from dialogue, then deliver a post-trained and served text-to-SQL system.

Across 15 discovery models, Claude Opus 5.5 has the highest reported mean requirement recovery at 81.7%; the open-weight models in the table range from 41.8% to 49.4%.

Only two of eight delivery runs meet all three acceptance targets. The human FDE's shipped system reaches 79.90% Spider-dev execution accuracy, compared with 76.11% for the best passing agent run.

• Discovery is measured over 10 independent runs per model and ambiguity level.
• Delivery targets are Spider-dev execution accuracy ≥60%, production SQL compilation failures ≤10%, and CPU p90 latency <15 seconds.
• The authors identify incomplete discovery, unreliable execution, budget-management problems, and benchmark-to-production gaps as bottlenecks.

🎯 Introduction

The paper frames forward-deployed engineering as work that connects an incomplete business request to a deployed system through requirement discovery, technical design, implementation, and delivery. It asks whether LLM agents can discover customer requirements and deliver a system that satisfies them, where the workflow bottlenecks occur, and which practices improve reliability.

The authors argue that existing agent evaluations commonly begin with a complete task specification. Their study begins earlier, with an ambiguous customer request, and grounds the evaluation in a completed enterprise Oracle-dialect text-to-SQL engagement, including the human FDE's requirements and shipped system.

The objective is to evaluate both phases of this work: eliciting acceptance criteria and operational constraints through dialogue, then planning and executing a post-training recipe against those criteria.

🔬 Methodology

The reference engagement supplies annotated customer knowledge and deployment constraints. The discovery specification contains 17 core business, data, and evaluation requirements plus six decisions the customer had left open. A simulated customer, powered by Gemini 2.5 Pro, answers agent questions from a fact sheet; agents interact through ask, request-an-artifact, inspect, and submit actions under three ambiguity levels. Gemini 2.5 Flash judges whether withheld facts were disclosed and correctly included in the submitted requirements.

For delivery, the agent receives discovered requirements and data. In the decoupled workflow, a planner produces a structured plan covering the base model, data processing, post-training, evaluation, and profiling; the plan is validated and frozen before an execution agent runs it. In the autonomous workflow, one coding agent plans and executes in one session. The evaluation includes four backbones in both workflows on OpenCode and one Codex CLI run for a harness comparison.

The data includes Oracle-dialect variants of Spider, BIRD, and SParC, with SParC flattened to question–SQL pairs, and 100 production questions. Spider-dev and BIRD-dev are used for evaluation and selection. Delivery reports execution accuracy on Spider-dev and BIRD-dev, production SQL compilation failures, and p90 CPU inference latency; only Spider accuracy, production failures, and latency define the acceptance targets.

📊 Experiments

For discovery, the authors evaluate 15 models across three ambiguity levels, with 10 independent runs per model and ambiguity level. Recovery rate is the fraction of withheld facts correctly submitted; elicitation count measures withheld facts both disclosed in dialogue and correctly submitted; ask turns count clarification actions, which may contain multiple questions. Claude Opus 5.5 reports the highest mean recovery at 81.7% ± 14.0, while Gemma 4 31B leads the open-weight group at 49.4% ± 17.2. A human audit of three models agrees with the LLM judge on 70.6% of recovery labels and 82.4% of disclosure labels.

For delivery, training data includes Oracle-dialect Spider, BIRD, and SParC variants plus 100 production questions. Spider-dev and BIRD-dev support evaluation and selection; production failures measure SQL validity, and p90 CPU latency is profiled over five independent runs. The thresholds are Spider-dev execution accuracy ≥60%, production SQL compilation failures ≤10%, and p90 latency <15 seconds; BIRD-dev is reported for reference. Two of eight OpenCode runs pass all targets: decoupled GPT-5.6 Sol and autonomous GPT-6.1 Sol. The human FDE row has 79.90 Spider EX, 25.30 BIRD EX, six production failures, and 9.26 seconds p90.

The paper reports that benchmark accuracy alone does not ensure production validity: autonomous Grok 4.7 has 75.82 Spider EX and 44.29 BIRD EX but 13 production failures, while autonomous GLM-5.3-Flash has 67.50 Spider EX and 23 production failures. In a harness ablation with GPT-5.6 Sol, autonomous workflow, Llama 3.1 8B, and the same budget and evaluator, Codex passes at 10.8 seconds p90 while OpenCode misses at 16.2 seconds.

fig:overview

Caption: A forward deployment agent: discovery, then delivery. In discovery, a discovery agent turns the ambiguous request of a customer simulated from a real engagement into identified requirements (1)--(4), establishes by experiment what the customer cannot answer (the SQL dialect), and gets sign-off. In delivery, a planner agent turns the identified requirements into an experiment plan and an execution agent runs it, profiling on the target hardware and evaluating on (1)--(4), returning model artifacts and an accuracy, profiling and cost report.

Why It Matters: The figure summarizes the two-stage workflow and shows how customer dialogue, experimental resolution of unknowns, planning, execution, and deployment checks connect.

🔮 Conclusion

The authors conclude that agents can recover many, but not all, withheld requirements and that delivery remains unreliable in this case: only 25% of delivery runs satisfy all three acceptance criteria. They identify incomplete discovery, autonomous execution failures, poorly coordinated compute budgets, insufficient comparative validation, and benchmark-to-production differences as the principal bottlenecks.

Their proposed mitigations are human review of discovered requirements and the execution plan, plus explicit experiment-environment specifications that cover hardware, budget scope, and production checks.

🛠️ Future Research Improvements

The paper's bounded single-engagement setting motivates testing requirement discovery and delivery across more customers, SQL dialects, and organizational contexts; the authors explicitly caution that customer needs and communication styles may differ.

Further evaluation could measure whether human review of requirements and plans improves pass rates, and whether systematic comparisons of base models and post-training methods use available compute more effectively. The paper identifies these as gaps but does not report such follow-up experiments.

🏭 Potential Industry Use Scenarios

The evaluated workflow is relevant to enterprise teams customizing text-to-SQL systems: an agent can help elicit acceptance criteria, identify unresolved questions, plan post-training, and profile a candidate model on target CPU hardware. The evidence supports treating these agents as supervised engineering aids in this setting, because discovery can omit or misrepresent requirements and most delivery runs did not pass all targets.

A practical deployment process could pair agent-generated requirements and plans with human approval, then gate release on production SQL validity and latency checks in addition to development benchmark accuracy. The study presents this as a mitigation direction rather than evidence of broad production readiness.

💬 Critical Analysis

A strength of the study is that it evaluates requirement discovery before a task specification is complete and compares delivery against a human FDE's shipped system. Its combined metrics also expose tradeoffs hidden by benchmark accuracy alone: the highest Spider and BIRD scores in the delivery table do not necessarily satisfy production SQL validity.

The evidence is narrower than the framing of general FDE capability: it comes from one engagement, four delivery backbones, and a small number of delivery runs. The discovery judge's imperfect agreement with a human audit adds uncertainty to the measured recovery rates.

The paper reports that all delivery runs use LoRA SFT and that base models and post-training methods were not systematically compared. Consequently, the reported pass rate characterizes these tested configurations and execution conditions, not the ceiling for agent-led delivery. The reported harness ablation also comprises a single comparison, so it is evidence of a possible harness effect in this setup rather than a general causal result.