Well-Specified and Well-Tested? Assessing SWE-bench-Style Datasets with a Self-Evolving Agent

📌 Highlights

EXLABEL treats SWE dataset annotation as an iterative learning process rather than a fixed rubric lookup. It accumulates transferable judgment patterns from historical cases, retrieves relevant experience for new cases, and revises its knowledge after comparing predictions with golden annotations.

The paper's strongest evidence is consistent cross-backbone performance: EXLABEL achieved the best F1 in all 12 combinations of task, subset, and backbone model. Its average-F1 improvement over the best baseline was 44.2%/49.9% for task clarity and 24.0%/18.4% for test coverage on the Python/Multilingual subsets.

The main caveat is filtering risk. EXLABEL's FPR is nonzero and reaches 30.1% in the reported multilingual test-coverage setting, so downstream usefulness depends on whether improved retained-data quality offsets the loss of valid instances.

• Core task: detect underspecified or inadequately tested SWE instances.
• Core mechanism: structured summaries, hybrid retrieval, reranking, reflection, and experience-bank updates.
• Downstream result: EXLABEL-filtered training data produced 66.2% resolution on SWE-bench Verified.
• Main verification question: how well do results transfer beyond the cleaned, agreement-only SWE-TQ benchmark?

🎯 Introduction

SWE-bench-style instances combine a natural-language task description, a golden patch, test suites, and an executable environment. The paper focuses on two quality dimensions: task clarity, which asks whether the description is sufficient and unambiguous for solving the issue, and test coverage, which asks whether the tests adequately and precisely verify correctness.

The prior limitation is twofold. Manual annotation by professional engineers is expensive and difficult to scale, while existing automated methods rely on fixed, general rubrics that struggle with nuanced cases and can produce high false-positive rates. The paper's objective is to emulate the way human annotators accumulate experience by building an adaptive, reusable judgment bank.

The motivating Django example illustrates the distinction: a test that checks an exact deprecation-warning message may reject valid implementations even though the task only requires deprecation behavior. EXLABEL uses this type of cross-instance principle to improve test-coverage judgments.

🔬 Methodology

EXLABEL has an experience accumulation phase and an experience reuse phase. Accumulation starts with an empty bank and processes labeled instances through rollout, reflection, and experience update. Rollout summarizes the task, golden patch, and test patch along task type, task explanation, and test explanation; retrieves candidate experience using BM25 and dense embeddings; reranks candidates; and predicts a label and explanatory notes.

Reflection compares the prediction with the golden annotation and generates a structured experience item. Understanding captures an abstract problem pattern, Successes records transferable strategies, and Mistakes records recurring reasoning failures. The update stage applies insertion, modification, or deletion operations to maintain a compact, adaptive bank. For new unlabeled instances, reuse repeats summarization, retrieval, reranking, and prediction with the accumulated experience as context.

The benchmark construction is part of the methodology. SWE-TQ-PYTHON derives from SWE-bench human annotations, while SWE-TQ-MULTILINGUAL derives from Multi-SWE-bench and covers C, C++, Go, Java, JavaScript, Rust, and TypeScript in addition to Python. The authors retain unanimous binary labels, split historical and test data, and manually remove annotation patterns inconsistent with the stated criteria.

• Retrieval representation: task type, task explanation, and test explanation.
• Retrieval configuration: Top-10 candidates followed by reranking to Top-3.
• Reflection fields: Understanding, Successes, and Mistakes.
• Update operations: Insert, Modify, and Delete.

📊 Experiments

The main evaluation uses SWE-TQ-PYTHON and SWE-TQ-MULTILINGUAL with Qwen3.8-27B, DeepSeek-V4-Flash, and GPT-5.6 Luna. Baselines are Vanilla, SimpleRAG, SPICE, Codex, and Claude Code. F1 is the primary metric because it balances precision and recall for detecting unsolvable or unverifiable instances; FPR measures the proportion of valid instances mistakenly filtered. Token usage and monetary cost are also reported.

EXLABEL achieved the best F1 in all 12 task/subset/backbone combinations. Its average relative improvement over the best baseline was 44.2% and 49.9% for task clarity on Python and Multilingual, and 24.0% and 18.4% for test coverage. The paper reports average FPRs of 3.3% for task clarity and 15.1% for test coverage. EXLABEL used 6.0k–21.0k tokens per instance, compared with 25.6k–265.5k for Codex and Claude Code in the reported settings.

The ablations show that the experience bank is the most important component: removing it reduced task-clarity F1 by 25.5%–35.7% and test-coverage F1 by 49.3%–84.0% in the reported Python and Multilingual settings. Removing reranking or either reflection field also reduced F1, with the combined removal of Successes and Mistakes producing larger degradation than removing either field alone.

For downstream effectiveness, the authors sampled 5k SWE-rebench instances, generated trajectories with DeepSeek-V4-Flash, filtered instances with GPT-5.6 Luna-backed assessors, fine-tuned Qwen3.5-9B-Base on retained resolved trajectories, and evaluated on SWE-bench Verified. EXLABEL retained 1,087 trajectories and achieved 66.2% resolution, compared with 54.4% for Full Data and 57.8% for the best baseline shown in the table, at a reported cost of $0.46 per retained trajectory.

• Datasets and benchmarks: SWE-TQ-PYTHON, SWE-TQ-MULTILINGUAL, SWE-rebench, and SWE-bench Verified.
• Metrics: F1↑, FPR↓, token usage, monetary cost, and downstream Resolved%↑.
• Best reported downstream result: 66.2% resolution on SWE-bench Verified.
• Ablation interpretation: retrieval quality and structured positive/error experience are complementary.

fig: motivation

Caption: A motivating example of django-15869 in SWE-bench.

Why It Matters: Shows how a test can over-specify an exact warning message that the task description does not require, motivating experience-guided test-coverage assessment.

fig: overview

Caption: Overview of .

Why It Matters: Summarizes the accumulation and reuse phases, including rollout, reflection, retrieval, prediction, and experience-bank updates.

fig: task_features

Caption: An example of the summarized descriptions along three dimensions for django-14453 in SWE-bench.

Why It Matters: Illustrates the structured representation used to bridge code patches and natural-language retrieval.

fig: experience

Caption: An example of generated experience about xarray-5362 in SWE-bench for test coverage assessment.

Why It Matters: Demonstrates how reflection converts one annotation outcome into reusable Understanding, Successes, and Mistakes.

fig: effective_case

Caption: A test coverage assessment case of django-15869 to reveal the reason for 's better performance.

Why It Matters: Provides a concrete case where retrieved experience helps identify an over-specified test that baseline systems accept.

fig: inconsistent_cases

Caption: Examples of typical annotation error patterns in task clarity (left) and test coverage assessments (right).

Why It Matters: Connects recurring annotation failures to the need for accumulated, reusable error patterns.

🔮 Conclusion

The paper argues that SWE dataset quality assessment benefits from experience that evolves across instances. EXLABEL combines structured instance understanding, hybrid retrieval, reranking, self-reflection, and explicit experience-bank maintenance to identify task descriptions and test suites that are likely to undermine solvability or verifiability.

Across SWE-TQ-PYTHON and SWE-TQ-MULTILINGUAL, the method consistently outperformed the evaluated baselines in F1. The downstream experiment further indicates that filtering can improve coding-agent training: the EXLABEL-filtered model achieved 66.2% resolution on SWE-bench Verified using 1,087 retained trajectories.

The evidence also establishes a practical trade-off. Filtering removes some valid instances, but the downstream result suggests that, under the reported setup, the retained trajectories were more useful for training than the larger unfiltered collection.

🛠️ Future Research Improvements

A natural next step is to evaluate robustness under less curated label conditions. SWE-TQ uses unanimous annotator agreement and removes cases that two authors judge inconsistent with the annotation criteria; testing disagreement-aware learning or soft labels could reveal whether EXLABEL remains effective on the ambiguity that production dataset pipelines actually encounter.

The reported multilingual results aggregate seven programming languages into one subset. Per-language evaluation, language-specific experience banks, and cross-language transfer experiments would clarify whether the experience representation captures general software-engineering principles or mainly benefits from distributional similarity.

Further work should measure threshold calibration and the quality-versus-retention trade-off explicitly. The reported FPR range, especially for test coverage, indicates that deployment may require operating points tailored to whether the priority is preserving valid data or aggressively removing unreliable instances.

The downstream study should be extended across more training models, trajectory generators, issue sources, and held-out benchmarks to determine whether the 66.2% resolution result reflects a broadly reusable filtering effect.

🏭 Potential Industry Use Scenarios

A dataset-maintenance team could run EXLABEL before releasing or refreshing coding-agent training corpora, flagging issue descriptions that lack sufficient information and tests that overfit implementation details. The output notes could support targeted human review instead of requiring every instance to receive the same level of manual inspection.

Benchmark providers could use separate task-clarity and test-coverage assessments to annotate evaluation suites, report quality distributions, and remove or quarantine instances likely to produce misleading model comparisons.

Organizations collecting synthetic or automatically mined software issues could apply the method as a quality gate before expensive trajectory generation. The reported downstream setup specifically supports filtering before sampling, because it reduces sampling expenditure on instances later rejected by quality assessment.

Human-in-the-loop review systems could use the experience bank as an auditable memory of recurring success patterns and mistakes, while routing high-uncertainty or high-impact cases to engineers.

💬 Critical Analysis

The strongest aspect of the paper is mechanism-level evidence. The experience bank is not only proposed conceptually; removing it causes large F1 drops, and removing reranking or reflection fields also degrades performance. The case study provides a plausible explanation for the gains: accumulated guidance helps the model recognize when a test checks details that the task does not require.

The benchmark design is thoughtful in separating temporal test coverage evaluation from random task-clarity evaluation and in cleaning recurring annotation inconsistencies. However, these choices also narrow the evaluation distribution. Complete-agreement filtering removes disagreement cases, and the severe class imbalance makes F1 and FPR behavior dependent on the selected test composition.

The results do not show that repository exploration is unnecessary in every setting; they show that the evaluated general-purpose coding agents did not match EXLABEL on these quality-assessment tasks. The method also depends on LLM calls for summarization, prediction, reflection, reranking, and bank maintenance, so latency, provider changes, prompt sensitivity, and experience-bank drift deserve separate operational study.

The downstream result is compelling but bounded. EXLABEL retains 1,087 resolved trajectories versus 2,309 for Full Data and achieves higher SWE-bench Verified resolution, yet the experiment does not establish how much of the gain comes from filtering criteria, trajectory selection, model-specific compatibility, or the particular sampled SWE-rebench distribution.

• Strength: strong ablation support for the proposed experience-guided mechanism.
• Strength: evaluates both annotation accuracy and downstream training utility.
• Caveat: benchmark labels are filtered for agreement and manually cleaned.
• Caveat: test-coverage false-positive rates remain substantial in some settings.
• Open question: whether the experience bank remains stable under continual data and repository evolution.