📌 Highlights
IdeaTaste's central finding is that LLM agents can approximate the literature and broad direction of human research more readily than the concrete approaches and research plans that human communities actually publish. Across six models and four areas, citation matching reaches about 26% of the human level and idea matching about 18%, with no population reaching half of the human level on either metric.
- Agents are closer to human communities in what they read than in what they propose.
- Richer field context and larger populations improve coverage or spread more than human-aligned idea matching.
- Blinded reviewers find the largest AI-versus-human gap in expected success rather than presentation.
- Unmatched proposals are generally judged weaker, not simply novel alternatives absent from the human record.
🎯 Introduction
AI scientists increasingly perform literature grounding, ideation, experimentation, and paper writing, but successful execution depends on choosing worthwhile problems and methods. The paper calls this judgment research taste and notes that existing evaluations mostly rate one idea at a time, use fixed targets, or rely on model judges, while population studies often measure internal similarity rather than agreement with human research choices.
IdeaTaste addresses this gap by treating a community's subsequent publications as an external reference. It freezes an area's literature through 2025, generates proposals from that snapshot, and compares them with papers published by the same community in 2026. The benchmark therefore evaluates both what agents read and what they formulate, while using diversity to distinguish broad population coverage from repeated proposals.
🔬 Methodology
For each of four areas—machine learning systems, code generation, world models, and automated research—the corpus U contains papers published through 2025 and the reference H contains screened 2026 papers. Both are built around 20 to 22 area landmarks: the corpus follows landmark citations and adds papers cited by at least two citing papers, while the reference combines landmark-citing papers with topic-search results and removes surveys. The resulting corpus sizes range from 30,264 papers in automated research to 94,674 in code generation, while the reference sizes range from 1,136 to 5,188 papers.
Agents write one proposal within a fixed 20-step budget and can plan, search U, pin retrieved papers, revise, compress history into notes, or submit. The four context conditions are free, topic, landscape, and frontier: they progressively expose an area topic, task/method tag structure, or peer proposals and traces from earlier rounds. Each population contains 300 proposals, written in two rounds of 150; peer records remain separate from the citable literature.
At the citation level, documents are compared through nearest-neighbor citation overlap and directional matching. At the idea level, a language-model judge extracts five slots—contribution type, problem, motivation, proposed approach, and research plan—and evaluates slot agreement among retrieved nearest documents, with rarer slots receiving greater weight. The benchmark applies the same metrics to human papers against one another to establish the human level.
tab:corpus
Caption: Data statistics. Left: sizes of the corpus $U$, published through 2025, and the human reference $H$, published in 2026. Right: papers in the human reference by publication month. Collection ends in August 2026, so the last month is incomplete.
| Area | Corpus U | Reference H |
|---|---|---|
| Machine learning systems | 88,184 | 5,188 |
| Code generation | 94,674 | 3,375 |
| World models | 50,271 | 1,867 |
| Automated research | 30,264 | 1,136 |
Why It Matters: The corpus and reference sizes show that the benchmark spans both large established literatures and smaller emerging areas.
equation 1
Meaning: These directional averages measure how closely each AI proposal matches its nearest human paper and how well the AI population covers the human papers. Their asymmetry distinguishes narrow overlap from broad coverage.
equation 2
Meaning: Q measures whether proposals cite enough papers relative to the human mean citation-set size \tau, while S measures how broadly the population spreads citations across the corpus. Citation diversity is their product.
equation 3
Meaning: M_idea averages weighted slot coverage of human papers by AI proposals, while D_idea subtracts peer overlap from one so that higher values represent more varied AI ideas.
📊 Experiments
The evaluation covers machine learning systems, code generation, world models, and automated research, using six models: Llama 4 Scout, Claude Haiku 4.5, Gemini 3.5 Flash-Lite, DeepSeek-V4-Flash, Qwen3.8-27B, and K2 Horizon 375B A23B. Four models were released in 2026 and may overlap the reference period; Llama 4 Scout and DeepSeek-V4-Flash were trained before 2026 and cannot have seen 2026 papers. Populations are scored on citation matching, citation diversity, idea matching, and idea diversity, where higher values mean closer agreement or greater variety.
Across more than 90,000 proposals, the reported average is about 26% of the human level for citation matching and 18% for idea matching, and no population reaches half of the human level on either metric. The gap is narrowest in automated research and widest in machine learning systems and code generation. For five of six models, idea matching falls further behind the human level than citation matching; once a direction is supplied, some populations cite up to three times as diversely as the community, yet no population reaches the human level of idea diversity.
The context and scale studies do not close the gap. Models trained before 2026 match more human ideas—about 26% of the human level versus 14% for models that may have seen the reference—while richer context changes citation diversity by about threefold but leaves matching within a narrow band. The fullest context ranks last on idea matching in 19 of 24 settings. When Gemini 3.5 Flash-Lite and DeepSeek-V4-Flash populations grow from 300 proposals to the size of the human reference, matching rises gently and levels off below the human level.
The blinded human evaluation uses 11 experienced reviewers and 192 documents, split evenly between proposals and 2026 papers, balanced across areas and models, with two reviews per document. Reviewers score quality, feasibility, expected success, overall rating, confidence, and whether a document appears AI-generated. LLM proposals average 50.39 on expected success versus 70.70 for human papers, while presentation is 58.16 versus 62.50. The per-slot results show that problems and motivations match human papers much more often than proposed approaches and research plans.
tab:human
Caption: Human evaluation of idea quality. Reviewers score 2026 papers and LLM agent proposals without knowing their source, and scores are mapped onto $[0,100]$. ``Judged AI" is the share of reviews that label a document as AI-generated. Match is decided by slot-level judge .
| Population | Soundness | Presentation | Contribution | Feasibility | Success | Rating | Confidence | Judged AI |
|---|---|---|---|---|---|---|---|---|
| Human | 49.83 | 62.50 | 53.99 | 75.39 | 70.70 | 49.48 | 56.51 | 39.06 |
| LLM agent (mean) | 38.54 | 58.16 | 45.31 | 63.80 | 50.39 | 38.54 | 59.90 | 86.46 |
| Llama 4 Scout | 13.54 | 30.21 | 17.71 | 55.47 | 35.94 | 22.50 | 60.94 | 81.25 |
| Claude Haiku 4.5 | 38.54 | 57.29 | 41.67 | 67.97 | 50.78 | 39.38 | 55.47 | 84.38 |
| Gemini 3.5 Flash-Lite | 39.58 | 56.25 | 45.83 | 64.06 | 58.59 | 35.62 | 62.50 | 90.62 |
| DeepSeek-V4-Flash | 51.04 | 75.00 | 63.54 | 67.19 | 61.72 | 46.88 | 60.16 | 87.50 |
| Qwen3.8-27B | 44.79 | 70.83 | 52.08 | 60.94 | 46.88 | 44.38 | 59.38 | 87.50 |
| K2 Horizon 375B A23B | 43.75 | 59.38 | 51.04 | 67.19 | 48.44 | 42.50 | 60.94 | 87.50 |
Why It Matters: The blinded evaluation shows that proposals trail human papers most strongly on expected success and are frequently identified as AI-generated.
tab:slots
Caption: Per-slot over the slots of the idea summary, namely problem, motivation, proposed approach, and research plan. Each entry is the share of human papers a population matches on that slot.
| Population | ML systems Problem | ML systems Motivation | ML systems Approach | ML systems Plan | Code generation Problem | Code generation Motivation | Code generation Approach | Code generation Plan | World models Problem | World models Motivation | World models Approach | World models Plan | Automated research Problem | Automated research Motivation | Automated research Approach | Automated research Plan |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Human | 28.32 | 42.58 | 3.99 | 7.21 | 24.74 | 33.19 | 4.21 | 3.64 | 25.12 | 29.41 | 7.12 | 6.16 | 19.10 | 38.29 | 4.49 | 2.99 |
| Llama 4 Scout free | 7.09 | 17.62 | 0.81 | 1.45 | 6.73 | 6.99 | 1.87 | 1.07 | 4.02 | 5.25 | 1.18 | 0.91 | 6.95 | 14.96 | 3.35 | 0.44 |
| Llama 4 Scout topic | 10.00 | 20.34 | 0.87 | 0.94 | 6.13 | 10.10 | 1.63 | 2.61 | 7.02 | 10.28 | 2.09 | 1.23 | 5.28 | 14.08 | 1.58 | 0.53 |
| Llama 4 Scout landscape | 8.54 | 18.04 | 0.87 | 2.35 | 2.93 | 4.30 | 1.13 | 1.01 | 1.71 | 2.25 | 0.37 | 1.45 | 5.99 | 15.32 | 2.38 | 0.62 |
| Llama 4 Scout frontier | 12.10 | 23.46 | 1.29 | 1.52 | 7.50 | 10.16 | 1.63 | 0.77 | 5.52 | 8.03 | 2.25 | 1.39 | 4.23 | 13.12 | 1.41 | 0.09 |
| DeepSeek-V4-Flash free | 1.04 | 1.12 | 0.23 | 0.08 | 0.89 | 1.21 | 0.00 | 0.00 | 2.95 | 1.98 | 0.27 | 0.00 | 2.11 | 3.43 | 0.26 | 0.00 |
| DeepSeek-V4-Flash topic | 1.77 | 3.01 | 0.10 | 0.04 | 1.63 | 1.69 | 0.12 | 0.00 | 2.68 | 3.00 | 0.64 | 0.21 | 3.35 | 4.75 | 0.44 | 0.09 |
| DeepSeek-V4-Flash landscape | 1.31 | 2.18 | 0.12 | 0.12 | 1.19 | 1.60 | 0.18 | 0.00 | 3.48 | 4.18 | 0.16 | 0.05 | 2.38 | 3.43 | 0.62 | 0.09 |
| DeepSeek-V4-Flash frontier | 1.04 | 2.04 | 0.06 | 0.02 | 1.42 | 2.04 | 0.21 | 0.00 | 2.36 | 2.73 | 0.11 | 0.00 | 3.08 | 4.23 | 0.26 | 0.00 |
Why It Matters: The slot decomposition shows that both models match human problems and motivations more often than concrete approaches and plans, locating the main gap inside idea formulation.
fig:setup
Caption: Overview of IdeaTaste. The literature corpus is frozen through 2025. In the upper branch, the human research community goes on to publish 2026 papers that cite work in the corpus. In the lower branch, a population of LLM agents retrieves from the corpus and writes research proposals. Matching compares the two branches, and diversity compares proposals with one another.
Why It Matters: This figure defines the benchmark's temporal split and makes clear that AI proposals and human papers are compared from corresponding pre- and post-cutoff conditions.
fig:maincase
Caption: Cases from the blind review, with the scores and the source guess of one reviewer under each document. Case 1 pairs a proposal with the 2026 paper whose idea the judge marks as overlap, and Case 2 is three documents with no counterpart in the human record.
Why It Matters: The cases show how matched and unmatched documents can vary in quality and how reviewers perceive their source.
fig:tagcoverage
Caption: Tag use in automated research: the task a document works on, the method it uses, and the task--method pair. Left: how many distinct tags of each kind each population uses. Right: the cumulative share of human tag mass each population reaches, with human tags ordered from the most to the least used and the human curve as the ceiling.
Why It Matters: The direction-coverage analysis separates knowing which tasks and methods matter from combining them into the specific directions pursued by humans.
fig:case
Caption: Two human papers, each paired with two AI proposals from a single language model. Under the stored automatic judgments, all four proposals match their human paper on problem and motivation, and only proposal B also matches on proposed approach and research plan.
Why It Matters: The paired examples make the slot-level failure concrete: agents can share a problem and motivation while proposing a different mechanism and research plan.
fig:scale
Caption: Matching against population size. Rows are Gemini 3.5 Flash-Lite and DeepSeek-V4-Flash from top to bottom, and columns are the four areas. Color marks the method, topic or free, and marker shape the level, circles for and squares for .
Why It Matters: The scaling curves test whether simply sampling more proposals closes the gap and show that matching rises slowly and levels off below the human level.
🔮 Conclusion
IdeaTaste finds a consistent separation between literature selection and idea formulation. LLM populations can cover many of the tasks and methods a community works on, but they seldom produce the concrete approaches and research plans later represented in human papers.
The paper concludes that newer models, richer context, and larger populations do not remove this limitation. The strongest practical implication is that improving AI research taste requires better translation from a recognized problem into a mechanism likely to work, not merely broader retrieval or more proposal samples.
🛠️ Future Research Improvements
The most direct next step is to improve the problem-to-approach transition. The experiments suggest that agents already receive enough topical and methodological signal to identify broad directions, so future systems should explicitly generate, critique, and test competing mechanisms and execution plans before submission.
The benchmark should also be extended across later publication cycles and additional research areas, with careful controls for training-data overlap and stronger human validation of slot-level judgments. Scaling experiments should continue to distinguish population coverage from genuine improvement in concrete proposal quality.
- Add mechanism-level critiques and feasibility checks before proposal submission.
- Evaluate proposals against multiple future publication windows rather than one partial 2026 reference period.
- Audit language-model judging and abstract-slot compression against domain-expert annotations.
- Measure whether execution or pilot experiments improve expected-success scores for proposals that initially miss the human record.
🏭 Potential Industry Use Scenarios
A research-planning copilot could use the benchmark structure to separate literature coverage from proposal quality: first verify that an agent retrieves the relevant task and method space, then evaluate whether its proposed mechanism and research plan are concrete and feasible. This would be useful for internal R&D ideation, where polished but low-probability proposals can consume substantial experimentation time.
Organizations could also use future-publication evaluation as an offline test for research agents before deployment, comparing proposal populations with later work in their target domain. The paper does not establish production benefits, so these scenarios should be treated as evaluation and triage applications rather than validated commercial outcomes.
- R&D portfolio triage based on problem, approach, feasibility, and expected success.
- Benchmarking literature-grounded research copilots before granting them experiment budgets.
- Generating diverse candidate directions while filtering proposals with weak execution plans.
- Monitoring whether an internal research agent improves mechanism-level alignment over successive model or prompt revisions.
💬 Critical Analysis
The benchmark's strongest design choice is its population-level, post-cutoff reference: it evaluates what a community actually pursued rather than asking a judge to predict value from a single proposal. The separation of citation and idea metrics, together with direction coverage and population scaling, supports the paper's central interpretation that the gap lies inside a direction rather than only in selecting the direction.
The evidence is not a complete measure of scientific value. Future publication is an observable record of community choice, not a proof that every published approach is objectively superior, and the 2026 reference ends in August with an incomplete final month. Corpus construction, area definitions, abstract compression, nearest-neighbor retrieval, and language-model judging each introduce additional measurement choices.
The blinded review strengthens the interpretation because unmatched proposals are rated lower in feasibility and expected success, but it remains a sample of 192 documents reviewed twice by 11 reviewers. The model-release comparison also cannot fully isolate knowledge exposure from model capability or prompting differences. Builders should therefore use IdeaTaste as a complementary benchmark for research taste, alongside direct expert review and execution-based validation.
- Strength: compares AI proposal populations with subsequent human research under an explicit temporal split.
- Strength: decomposes failure into citation selection, direction coverage, idea slots, diversity, and human-rated quality.
- Caveat: publication overlap is an imperfect proxy for scientific merit and may favor community conventions.
- Caveat: the benchmark's conclusions depend on the selected areas, landmarks, cutoff window, slot schema, retrieval configuration, and judge.