When, Where, How: Active In-Context Learning for Tabular Foundation Models

📌 Highlights

The paper’s central result is that active in-context learning for frozen tabular foundation models is budget-dependent. Hybrid diversity acquisition leads during the cold start, TANN overtakes it at about 50 labels, and EPIC overtakes it at about 60 labels in the 100-label learning-curve setting.

EPIC achieves a final mean Cohen’s κ of 0.578 in the cross-model baseline summary and improves on Hybrid on 15 of 20 datasets with a mean final difference of +0.030 and p=0.011 before Holm correction. TANN is statistically close to EPIC while requiring substantially less computation.

On the eight large-pool benchmarks, EPIC retains 80.3% and 87.0% of full-context Cohen’s κ with 5% and 10% of the pool labeled, compared with 72.5% and 78.9% for Random.

The main caveats are the transductive requirement for target inputs, classification-only EPIC, greedy batch selection, and the absence of an evaluated adaptive schedule that combines early diversity with later target-aware acquisition.

• When: diversity is strongest during the first few dozen labels.
• Where: labels near uncertain target inputs are especially valuable.
• How: in-context hypothetical-label evaluation makes predictive information gain practical for frozen models.
• Engineering trade-off: TANN captures much of EPIC’s benefit at roughly one-eighth of EPIC’s large-dataset runtime.

🎯 Introduction

Tabular foundation models such as TabPFN and TabICL adapt to a new supervised task through labeled context rows rather than parameter updates. Because labels may require expert annotation, measurement, or time, the selection of rows to label is a direct control over how close the frozen model comes to its full-data performance.

Prior active in-context learning rules transfer margin, core-set, hybrid, and proxy-filtered acquisition strategies to frozen models, but they apply a fixed criterion throughout acquisition, do not use the inputs on which the model will be deployed, and score candidates by proxies such as candidate uncertainty or distance to the labeled set. The paper therefore asks how diversity, uncertainty, and target-aware information gain change as the context grows.

The paper’s objective is twofold: introduce target-aware acquisition rules for target inputs that are already visible, and determine whether expected predictive information gain can be computed directly through a frozen model’s in-context conditioning rather than parameter posteriors or repeated retraining.

🔬 Methodology

The setting is transductive active in-context learning. An unlabeled pool supplies candidates, a target set represents the inputs whose predictions matter, and an oracle reveals queried labels at unit cost. The context starts with one uniformly drawn row per class and grows in batches of 10 until 100 labels are acquired or half of a small pool is labeled. The main comparison uses TabICL v2 with eight ensemble members and default inference settings.

TANN ranks target inputs by margin uncertainty and assigns each of the most uncertain targets its nearest still-unassigned pool row. Distance can be measured in TabICL’s context-dependent row representation or in standardized feature space. This makes TANN target-aware without explicitly modeling how a hypothetical label changes the model’s target predictions.

EPIC instead evaluates the expected information carried by a candidate label about target predictions. For each candidate and possible class label, the frozen model receives an augmented context and produces target predictions; the expected entropy reduction is then averaged over sampled target inputs. The implementation uses a target sample of size 256, a shortlist of at most 64 candidates, eight ensemble members, and greedy top-10 batch selection.

The workflow is summarized in the source figure, while the two equations below capture the core nearest-neighbor and predictive-information mechanisms.

📊 Experiments

The main benchmark is the twenty OpenML classification datasets used by Tab-AICL. They range from 145 to 581,012 rows, 4 to 94 features, and 2 to 7 classes; datasets above 10,000 rows are uniformly subsampled to 10,000. The eight large-pool datasets have pools of 1,358 to 7,000 rows, while the remaining twelve have pools of 101 to 700 rows. The paper also reports one indicative out-of-time weather regression task from TabReD, but EPIC itself is evaluated as a classification method.

Each seed creates a stratified 70/30 pool/test split, initial context, and acquisition randomness. Rules are evaluated over ten seeds using normalized AULC of test Cohen’s κ and final Cohen’s κ at the labeling budget; higher values are better. Competitors include Random, Margin, Coreset, Hybrid, Proxy-Hybrid, Target-Random, Target-Coverage, TANN-feat, TANN, and EPIC. Additional comparisons use frozen TabPFN v3, XGBoost-Margin, CatBoost-Margin, and LabelSpreading-Random.

The learning curves show the main temporal pattern: Hybrid leads during the first 40 labels by up to 0.05 κ over Random, TANN overtakes Hybrid at about 50 labels, and EPIC overtakes it at about 60 labels. At the final budget, EPIC reaches 0.578 mean κ in the cross-model baseline summary, while Hybrid reaches 0.549 and Random 0.536. EPIC improves on Hybrid on 15 of 20 datasets with a +0.030 final difference and p=0.011 before Holm correction.

Label efficiency and transfer support the same conclusion. At 5% and 10% of large pools, EPIC retains 80.3% and 87.0% of full-context κ, versus 72.5% and 78.9% for Random. When the acquired labels are reused by other models, EPIC-selected labels produce final κ values of 0.543 for CatBoost and 0.530 for XGBoost, compared with 0.525 and 0.498 under their own margin acquisition.

fig:workflow

Caption: Workflow for the Expected Predictive Information Gain in context for tabular foundation models.

Why It Matters: This figure clarifies the end-to-end EPIC workflow: maintain a labeled context, evaluate hypothetical candidate labels, measure their effect on target predictions, and acquire the highest-scoring rows.

fig:ranks_final

Caption: Mean rank of every acquisition rule over the twenty datasets (1 is best), for Cohen's $ $ at the maximal label budget, with TabICL as the backbone.

Why It Matters: It summarizes the final-budget comparison and supports the claim that target-aware rules, especially EPIC, become strongest at the labeling budget.

fig:learning-curves

Caption: Learning curves on the fifteen datasets with a budget of 100 labels. (a) Cohen's $ $ after each acquisition round, averaged over datasets (each averaged over seeds). (b) Difference to Random; bands show $ 1$ standard error over datasets. Hybrid leads in the cold start; Target-NN and IC-EPIG overtake it after about 50 and 60 labels, respectively.

Why It Matters: This is the central evidence for the paper’s context-dependent acquisition claim: Hybrid leads early, followed by TANN and EPIC later in the trajectory.

fig:reduction

Caption: Cohen's $ $ with a fraction of the pool labeled, TabICL as the backbone, relative to the fully labeled pool, on the eight datasets with pools of 1,358 to 7,000 rows (mean over datasets of the means over ten seeds). The dotted line marks 95\% of the full-context performance.

Why It Matters: It visualizes how much full-context performance can be recovered with only a small fraction of a large labeling pool.

🔮 Conclusion

Across twenty OpenML classification benchmarks and two frozen tabular foundation models, the paper finds that acquisition quality is not stationary over the labeling trajectory. Diversity is most useful in the cold start, while target-aware prediction-oriented rules become stronger after the context contains enough information for the model’s predictions to guide selection.

EPIC provides the strongest final-budget results and achieves 80.3% and 87.0% of full-context Cohen’s κ with 5% and 10% of large pools labeled. TANN is a simpler and cheaper approximation that is statistically close to EPIC in the reported pairwise analysis.

The practical takeaway is to treat acquisition as a schedule-sensitive problem: early coverage and later target relevance should be considered together. The paper’s own decisive next test is whether a model-driven switch between these behaviors can outperform either fixed family.

🛠️ Future Research Improvements

The most direct extension is an adaptive acquisition schedule that starts with diversity and switches to TANN or EPIC once the context becomes informative. The paper does not yet establish how to detect that transition from model behavior, so this remains an open empirical question.

EPIC should be extended to regression and evaluated on multiple regression domains rather than relying on the single indicative weather task. A continuous-label estimator must replace exact enumeration of hypothetical classes while preserving a meaningful predictive-information objective.

Joint batch acquisition could reduce redundancy among simultaneously selected rows. This would address the current greedy independent-selection approximation, although it would increase scoring cost.

Further work should stress-test target-aware acquisition under target shift, incomplete target visibility, changing deployment populations, and annotation costs that vary by candidate.

🏭 Potential Industry Use Scenarios

The method is most credible for workflows with a large unlabeled table, expensive expert labels, and a known or sampleable deployment population. Examples include prioritizing medical measurements, fraud investigations, customer-response labeling, software-defect triage, or quality-control inspections, provided the target inputs can be identified before annotation.

TANN is a practical first deployment candidate because it uses target uncertainty and nearest-neighbor structure without the repeated hypothetical-label evaluations required by EPIC. EPIC is better suited to offline annotation planning when the additional runtime is justified by higher label efficiency.

The reported transfer results suggest that an organization could acquire labels with a foundation-model-driven rule and later train a conventional gradient-boosted model on the same labels. This is evidence of cross-model usefulness, not a guarantee for a particular production domain.

Operational adoption should include safeguards for class imbalance, target drift, representation quality, label noise, and the cost of acquiring labels for rows selected by a model rather than by domain experts.

💬 Critical Analysis

The strongest aspect of the paper is the separation of three acquisition questions that are often conflated: when to use diversity, where to place labels relative to target inputs, and how to estimate a candidate label’s effect on target predictions. The two-model replication and cross-model label-transfer analysis make the central interpretation more robust than a single-backbone comparison.

The main empirical claim is directionally consistent but not uniformly decisive. The transition appears across benchmarks and models, yet individual dataset effects vary, and the paper notes that many comparisons are significant only at the 10% level. EPIC’s advantage over TANN is small in the pairwise results, which weakens the case for paying EPIC’s much higher runtime unless its information criterion is needed for a particular application.

The transductive assumption is important: the method optimizes labels for visible target inputs, so performance may degrade when deployment inputs are unavailable, unrepresentative, or nonstationary. The benchmark protocol also uses fixed budgets and OpenML classification datasets, leaving questions about noisy labels, nonuniform annotation costs, evolving pools, and production-scale latency.

For builders, the most defensible implementation path is to start with TANN or a hybrid schedule, measure final κ and label cost on a representative target set, and add EPIC only if its extra computation produces a material reduction in annotation effort. A successful adaptive schedule would be the clearest evidence that the paper’s temporal interpretation can become a deployable policy.