IMRouter: Imbalance-aware Temporal Pattern Routing for LLM-based Time Series Forecasting

📌 Highlights

IMRouter targets a specific failure mode in LLM-based forecasting: temporal pattern imbalance causes common local dynamics to dominate the shared optimization signal, leaving less frequent dynamics with higher errors. The paper combines sparse expert routing with routing-aware cross-modal alignment and reports improvements on long-term, short-term, few-shot, pattern-wise, and ablation evaluations.

The strongest evidence is internally consistent with the proposed mechanism: the full model reaches 0.171 MSE and 0.263 MAE on ECL and 0.423 MSE and 0.285 MAE on Traffic in the ablation table, while the pattern-wise Traffic analysis reports the lowest MSE across all six clusters and an approximate Cluster 6 MSE of 0.78.

• Core claim: frequency imbalance in recurring temporal patterns biases both temporal encoders and semantic prototype alignment.
• Main method: PATR uses gradient-conflict and density-aware routing; RGSA preserves routing information during cross-modal alignment.
• Reported long-term evaluation covers ETTh1, ETTh2, ETTm1, ETTm2, Weather, ECL, and Traffic.
• Reported short-term evaluation on M4 gives sMAPE 11.77, MASE 1.568, and OWA 0.844.
• Key caveat: transfer beyond the GPT-2 backbone and the reported datasets is not established.

🎯 Introduction

The paper frames time-series forecasting as important for website traffic prediction, cloud workload management, online service demand, traffic monitoring, and resource management. LLM-based forecasters commonly use a represent-then-align paradigm: a temporal encoder extracts representations, an alignment module maps them toward text prototypes derived from a frozen LLM vocabulary, and the system is trained with forecasting loss.

The paper argues that existing methods focus mainly on the modality gap between continuous time series and discrete text while overlooking the unequal frequency of local temporal patterns. Using TS2Vec representations and HDBSCAN clustering for analysis, the authors identify skewed cluster sizes and generally higher errors for less frequent clusters. IMRouter therefore aims to reduce frequency-induced bias in temporal representation learning and semantic alignment without manually assigned pattern labels.

🔬 Methodology

The forecasting backbone normalizes each channel, divides it into sliding-window patches, and maps the patches into temporal tokens. Temporal tokens are aligned with compressed text prototypes through cross-attention and then processed by a frozen LLM. The paper's key motivation is expressed by Eq. (5): the aggregate loss weights each pattern-specific loss by $\frac{|D_j|}{|D|}$, and Eq. (7) consequently weights its gradient contribution by the same empirical frequency.

PATR replaces a single shared temporal transformation with sparse mixture-of-experts routing. Eq. (8) computes patch-level assignment probabilities and Eq. (9) combines the selected experts. After an initial warm-up, the method penalizes incompatible assignments using the positive negative-cosine conflict score in Eq. (10). It estimates local density from normalized shared-router gradients and applies inverse-density weighting so sparsely represented optimization directions receive greater relative influence.

RGSA applies cross-attention to the PATR representations and transfers routing information to the aligned representation. The paper formulates this as mutual-information maximization and implements it with routing-guided semantic distillation: Eq. (19) minimizes KL divergence between the stop-gradient router distribution and a predictor inferred from the aligned representation. The complete training objective in Eq. (20) combines prediction, load balancing, density-weighted gradient conflict, and semantic distillation; RGSA is activated midway through training to avoid unstable early routing supervision.

📊 Experiments

For long-term forecasting, the paper evaluates ETTh1, ETTh2, ETTm1, ETTm2, Weather, Electricity (ECL), and Traffic with input length 96 and prediction horizons 96, 192, 336, and 720. It reports MSE and MAE, where lower values are better, against TimeSup, FSCA, CALF, Time-LLM, GPT4TS, PatchTST, iTransformer, and TimesNet. The reported aggregate comparison is provided in tab:long_term_forecasting: IMRouter has 0.171 MSE and 0.263 MAE on ECL, and 0.423 MSE and 0.285 MAE on Traffic.

For short-term forecasting, the M4 benchmark contains 100,000 univariate series across Yearly, Quarterly, Monthly, Weekly, Daily, and Hourly frequencies. The paper reports sMAPE, MASE, and OWA, with lower values better, averaged across the Yearly, Quarterly, Monthly, and Other subsets. IMRouter reports 11.77 sMAPE, 1.568 MASE, and 0.844 OWA, compared with FSCA's 11.828, 1.580, and 0.850.

The few-shot setting trains on 10% of the available samples and averages over horizons 96, 192, 336, and 720 on ETTm1, ETTm2, ETTh1, and ETTh2. IMRouter ranks first or second on six of eight reported metrics and reaches 0.498 MSE and 0.457 MAE on ETTm1 and 0.292 MSE and 0.330 MAE on ETTm2. The ablation table shows progressive gains from Vanilla MoE to PATR, RGSA, density weighting, and full IMRouter; on ECL, full IMRouter improves from 0.177 to 0.171 MSE over PATR w/o Density + RGSA, while on Traffic it improves from 0.430 to 0.423.

The pattern-wise Traffic experiment reports the lowest IMRouter MSE across all six frequency-ranked clusters, with stronger improvements on less frequent clusters; the paper gives an approximate Cluster 6 MSE of 0.78 versus 0.85–0.93 for competing methods. Sensitivity experiments on ECL and Traffic find k=2 best for the number of selected experts and expert dimension 256 best among the tested settings.

fig:pic1

Caption: Illustration of temporal pattern diversity in a time series. Frequently recurring peak-shaped patterns coexist with less frequent, smoother local patterns (highlighted by red dashed boxes), demonstrating the heterogeneity of temporal dynamics.

Why It Matters: It motivates the central problem by showing why a single average forecasting objective must handle multiple local temporal regimes.

fig:pic2

Caption: (a) Frequency distribution of temporal pattern clusters identified from a representative channel in the Traffic dataset, ranked by cluster size in descending order. (b) Cluster-wise forecasting errors (MSE) of three existing methods on the same clusters, showing generally higher errors for less frequent patterns. (c) Dataset-level TPI statistics across Electricity, Weather, and Traffic. Gini measures the inequality of cluster sizes, while Top-20 \% Coverage denotes the proportion of samples covered by the largest 20 \% of clusters.

Why It Matters: It provides the paper's main empirical diagnosis: pattern frequencies are skewed and less frequent clusters generally have higher forecasting error.

fig:pic3

Caption: Illustration of IMRouter

Why It Matters: It shows how PATR, RGSA, the frozen LLM, and the auxiliary optimization objectives fit together.

fig:patter-wise

Caption: Pattern-wise forecasting performance of IMRouter and three baseline methods on the Traffic dataset.

Why It Matters: It directly evaluates the claimed mechanism by comparing errors across clusters ordered by occurrence frequency.

sensitity

Caption: Hyperparameter sensitivity results on the number of top-k experts and the dimension of experts.

Why It Matters: It tests how routing sparsity and expert capacity affect forecasting error on ECL and Traffic.

🔮 Conclusion

The paper concludes that temporal pattern imbalance is a meaningful source of bias in LLM-based time-series forecasting. IMRouter addresses it at two linked points in the pipeline: PATR reduces optimization interference through gradient-guided expert routing, and RGSA preserves the resulting pattern-specific information during cross-modal semantic alignment.

Across the reported benchmarks, the method improves average forecasting metrics and shows especially clear benefits for underrepresented Traffic patterns. The evidence supports the value of treating temporal frequency imbalance as distinct from abrupt changes, non-stationarity, and extreme events.

🛠️ Future Research Improvements

The main body does not provide a detailed future-work agenda, so the following are evidence-grounded research questions rather than stated paper claims. The next evaluation should test whether the method remains effective with other frozen LLM backbones, since the reported comparisons standardize LLM-based models on GPT-2.

The pattern-wise conclusions could be strengthened by replacing or complementing TS2Vec-plus-HDBSCAN diagnostics with alternative representations, clustering procedures, or directly measured recurring regimes. Broader few-shot curves, explicit compute and latency accounting, and tests across more operational domains would clarify deployment trade-offs.

🏭 Potential Industry Use Scenarios

The paper's stated application areas map naturally to workload forecasting for web services and cloud resources, where frequent weekday patterns may coexist with less frequent weekend or special-event cycles. More reliable rare-pattern forecasts could support capacity planning, autoscaling, traffic routing, and service-level protection.

The benchmark coverage also suggests energy-demand forecasting, weather-sensitive operations, transportation-flow prediction, and other multivariate monitoring systems as plausible use cases. A production implementation would need to validate routing stability, memory overhead, latency, and performance under distribution shifts not evaluated in the supplied main body.

💬 Critical Analysis

The strongest aspect of IMRouter is that its mechanism is tied to an explicit optimization diagnosis rather than only adding capacity: Eq. (5) identifies frequency-weighted gradients, the ablation study separates vanilla expert routing from gradient-guided routing and density weighting, and the pattern-wise Traffic experiment directly examines errors by frequency. The complementary PATR and RGSA design also addresses both representation learning and cross-modal alignment.

The main uncertainty is external validity. The experiments use a CALF-based implementation, channel-wise routing, a GPT-2 backbone, selected public benchmarks, and a narrow sensitivity study. The supplied evidence does not establish the computational cost of gradient-based routing, the stability of density estimates under changing batch composition, or the behavior of the method when temporal patterns evolve over time.

Reproducibility is helped by the reported Python 3.10 and PyTorch 2.1.3 environment, Adam optimization, fixed random seed 2021, hyperparameter search spaces, two RTX 4090 GPUs, and early-stopping settings. A builder should nevertheless verify the exact warm-up and RGSA activation schedules, auxiliary-loss coefficients, pair-sampling strategy, and expert capacity constraints before treating the reported numbers as directly reproducible.