DistDebug-Bench: Can LLM Agents Diagnose Bugs in Distributed Systems?

📌 Highlights

DistDebug-Bench asks a concrete question: can an agent identify the pre-fix source-code blocks that explain a real distributed-system failure, given an incident report, supporting artifacts, and repository access? This focuses evaluation on diagnosis before repair, a step that patch-generation benchmarks can obscure.

The dataset contains 120 incidents from 11 distributed systems, five programming languages, and 10 failure patterns, with 218 gold causal blocks. Its evidence is often indirect: 75 incidents lack a usable project-internal stack trace, and 83 reports do not name the gold method in their title or description.

In the reported sweep, GPT-6 Astra with Mini-SWE-Agent achieves 67.7% block coverage and 83.9% file coverage. Yet repeated runs are not fully stable: 15 of 120 bugs are flaky for this configuration. The paper's central takeaway is that finding a relevant component is easier than precisely explaining the causal code.

• Diagnosis is scored against human-reconciled pre-fix causal spans.
• Weak evidence and cross-component causes expose major gaps.
• Coverage can improve while precision falls, so both matter.

🎯 Introduction

The paper positions distributed-system debugging beyond the scope of many established software benchmarks. Code-generation suites test short programs, repository-context benchmarks measure cross-file use, and issue-resolution benchmarks often judge whether an agent produces a patch that passes tests. Those settings are useful, but a passing patch does not directly show that the agent identified the root cause; a symptom can be suppressed without resolving the underlying failure.

Distributed systems raise a distinct diagnosis challenge. They span many subsystems and layered abstractions, and failures can involve interactions across modules, network calls, and service tiers. Reports may contain only partial evidence such as logs or stack traces, while the observed symptom may be downstream from the originating fault. The agent must form and test explanations, trace causal relationships, and revise an initial search when the evidence does not support it.

The paper's objective is to evaluate this code-level localization step directly. Given an incident report, its evidence bundle, and the pre-fix repository snapshot, an agent must identify the source blocks that best explain the observed failure. Human-annotated causal regions provide the scoring target, allowing the authors to study whether agents can navigate from symptoms to root causes rather than only generate repairs.

🔬 Methodology

Benchmark instances are built from public issue reports and fixing changes in production-scale distributed systems. The authors retain closed issues with an associated fix and a concrete failure description. They favor realistic evidence, including logs, stack traces, reproduction steps, linked issues, or symptom descriptions, while removing or ignoring later thread comments that disclose the cause. Fixes must support a precise label; large refactorings and fixes with more than 100 changed source lines are generally excluded unless the causal span remains separable.

Annotators derive a minimal pre-patch causal span from the fix and surrounding code. Each label records a file, function or method, and line range. Tests, helper refactors, defensive hardening, and cleanup are excluded unless needed to explain the original failure. Three annotators label each issue independently and reconcile a final set; LLM output is not used to create the released gold labels.

At evaluation time, the agent sees the incident report, issue-side artifacts, pre-fix repository, and structured output requirements. It does not see the fix or gold labels. The evaluator validates output, canonicalizes symbols, and scores predicted blocks against the annotated regions. Metrics include coverage and precision at file, function, line, and block levels, plus mention coverage from the natural-language explanation. Block coverage is primary because it is less sensitive to exact line-boundary choices; line coverage provides a stricter overlap check. Repeated attempts support measurement of within-bug variation and flaky diagnoses.

📊 Experiments

The central benchmark comprises 120 common incidents across 11 distributed systems, five programming languages, and 10 failure patterns. The paper characterizes the evidence structure as well as aggregate performance: 96 reports contain code identifiers, 28 contain stack traces, and 20 contain logs. In 75 cases no usable project-internal stack trace is present. Of the 120 reports, 83 do not mention the exact gold method in the title or description. The labels contain 218 blocks; 45% of incidents need multiple blocks and nearly a quarter span multiple files.

The main model sweep runs 13 configurations on Mini-SWE-Agent. The authors also compare GPT-5.2-Codex and o4-mini across Mini-SWE-Agent, Codex CLI v0.128.0, and OpenCode v1.4.6. Each bug/configuration is attempted three times; results average attempts within bugs and then weight the 120 bugs equally. The authors report coverage and precision at several code granularities, use block coverage as the primary metric, and provide bug-bootstrap confidence intervals. These are author-reported experiments, not demonstrations of a deployed debugging product.

GPT-6 Astra with Mini-SWE-Agent reaches 67.7% block coverage, 83.9% file coverage, and 53.9% line coverage. GPT-5.5 reaches 65.1% block coverage, compared with 56.4% for GPT-5.2-Codex, but its block precision is 34.7% versus 43.1%, and it predicts 3.6 rather than 2.3 blocks per diagnosis. Evidence breakdowns show the gap: GPT-6 Astra reaches 94.8% block coverage on no-stack incidents whose report names the method, versus 65.0% on no-stack opaque incidents and 40.5% on cross-component opaque incidents. GPT-5.2-Codex similarly falls from 82.8% on no-stack method-named cases to 44.7% on no-stack opaque cases.

Run-to-run instability is also material. GPT-6 Astra has 15 flaky bugs out of 120 under the paper's definition. For GPT-5.5, 11 of 42 no-stack opaque bugs are flaky, compared with 2 of 16 no-stack method-named bugs. Harness comparisons for GPT-5.2-Codex show a coverage-precision tradeoff: Mini-SWE-Agent has 56.4% block coverage and 43.1% precision, Codex CLI 46.5% and 49.3%, and OpenCode 50.0% and 47.7%. The source extraction did not provide a clean machine-readable full table, so these prose-verified comparisons are reported directly rather than reconstructed as a numeric table.

fig:teaser

Caption: task flow.

Why It Matters: Shows the diagnosis workflow: agents receive an incident and pre-fix repository context, then identify the source blocks that explain the failure.

fig:bench-characterization

Caption: workload characterization.

Why It Matters: Summarizes the heterogeneous evidence, distance from symptoms to causes, and non-local structure that make the benchmark challenging.

fig:eval-evidence-block-coverage

Caption: Block coverage by incident evidence. Columns group the issue text's strongest code anchor; rows group the first crash stack frame. Coverage drops most when reports lack a precise anchor and the crash site is absent or cross-component. Panel counts give bugs per cell; whiskers show 95\% bug-bootstrap CIs (none for the single-bug cell).

Why It Matters: Breaks down block coverage by report anchors and crash location, making the weak-evidence performance gap visible.

Figure ure

Caption: Figure 2 summarizes the workload beyond headline dataset size. The characterization highlights three properties that make diagnosis difficult and later enable fine-grained evaluation breakdowns: the issue-side evidence is heterogeneous, the symptom is often far from the causal code, and many gold labels require non-local or multi-block explanations.

Why It Matters: Provides a caption-aware crop describing benchmark workload properties; its caption overlaps with the characterization figure, so treat it as supplementary visual context.

🔮 Conclusion

The paper concludes that current agents can often navigate to relevant repository regions but still struggle to identify precise causal locations, especially when incident reports lack direct code references or usable stack traces. Its trace analysis points to inconsistent tool use and limited revision of early hypotheses, with repeated attempts sometimes producing different diagnoses.

The results motivate agent designs that make exploration, hypothesis revision, and validation explicit. Better search breadth alone or stronger models do not remove the weak-evidence problem in the reported evaluation. DistDebug-Bench provides a way to measure whether such changes improve causal localization while retaining precision.

🛠️ Future Research Improvements

The authors propose adaptive context construction: when an agent searches for a term, tooling could surface related class definitions, symbol uses, call paths, and neighboring context. This could reduce sensitivity to small differences in shell commands that send otherwise similar attempts into different parts of a repository.

They also recommend hypothesis-aware context management. Separate contexts for competing explanations, plus active pruning of hypotheses contradicted by evidence, may help prevent an agent from deepening an early but incorrect local theory.

A third direction is to make correctness signals easier to use. Harnesses could bring exceptions, crash sites, logs, and documentation into a consistent evidence-checking process, or use an evaluator agent to assess whether a proposed diagnosis explains the full behavior. These are proposed design directions; the supplied main-body evidence does not establish that they improve benchmark scores.

🏭 Potential Industry Use Scenarios

The paper motivates use in production incident response, where engineers must connect symptoms in reports, logs, and stack traces to source-level causes across components. A diagnosis agent built around this task could help an engineer prioritize files or code regions to inspect before deciding on a repair. The benchmark supports evaluating that localization capability separately from patch generation.

A practical workflow could use structured causal spans as review suggestions, attach the supporting incident evidence, and ask an engineer to confirm whether the proposed region explains the observed failure. In distributed storage, coordination, or stream-processing systems, this could help organize investigation across layered code and services. These are plausible applications inferred from the paper's problem setting; the paper does not report a production deployment, operational time savings, or an industry field trial.

For teams adapting the benchmark, evaluation should preserve both coverage and precision and include repeated runs. A tool that points to more code may help broaden an investigation, but the reported GPT-5.5 comparison shows that higher block coverage can accompany lower precision and more predicted blocks. Deployment decisions therefore need human review and evidence traceability, especially for opaque reports where performance remains weaker.

💬 Critical Analysis

A strength of DistDebug-Bench is that it isolates a practical step that patch-based evaluation can miss: identifying the code that explains an incident before proposing a fix. Its labels are human-curated and reconciled, and its characterization captures multi-block and cross-file causes as well as heterogeneous report evidence. The authors also examine repeated attempts and tool trajectories, which expose failure patterns that a single aggregate score could conceal.

The results require careful interpretation. Block coverage is a proxy based on the authors' annotated causal regions; it does not cover every plausible intervention point. The paper's own audit found a small number of incompletely covered attempts that independent reviewers considered correctly localized, so a zero or partial score is not always proof of bad reasoning. Conversely, high file coverage should not be read as precise diagnosis: block coverage and precision are lower, and some configurations return broader sets of blocks.

Generalization is also bounded by the collection process. Incidents are selected for diagnosability from issue-side evidence and for fixes that allow precise labeling, which is necessary for a usable benchmark but may omit ambiguous cases engineers encounter in production. The authors add leakage controls and closed-book probes, yet acknowledge that pretraining exposure cannot be ruled out. Finally, the proposed adaptive retrieval, hypothesis management, and correctness-checking ideas remain research directions in the supplied evidence; their effectiveness needs direct controlled evaluation.