Reasoning over Non-Parametric Memory in Multilingual LLMs: A Cognitive-Based Analysis of Relational Knowledge

📌 Highlights

Rel-Bloom isolates evidence-conditioned relational reasoning by placing the relevant fact directly in the prompt, removing retrieval quality as a confound. Its controlled design crosses 30 Wikidata relations, five Bloom-inspired levels, and three veracity conditions to produce 450 questions evaluated across six languages and 43 models.

The strongest reported failure is the Apply-to-Analyze drop in open-weight models: the Large Granite bucket loses 47.2 percentage points, while Qwen3.5 loses 7.2 points and neither Gemini model shows the reported drop. Evaluate is a separate bottleneck: Gemini 3.1 Pro averages 99.5% on the other four levels but 60.7% on Evaluate.

Counterfactual contexts are generally harder than Factual contexts, while Neutralized contexts can improve higher-level performance in some model regimes. Item-level and relation-level difficulty rankings remain positively aligned across languages.

The main caveat is that level differences may partly reflect distractor construction, and one source triple per relation limits generalization beyond the selected relation instances.

• Evidence following is not captured by factual accuracy alone.
• Apply-to-Analyze and Evaluate expose different failure patterns.
• Cross-language difficulty structure is more stable than aggregate accuracy.
• The benchmark is diagnostic rather than an end-to-end RAG evaluation.

🎯 Introduction

RAG systems require models to reason over facts supplied in context, but a correct answer may arise from memorized entity knowledge rather than the retrieved evidence. Conversely, a model may ignore a stated relation when it conflicts with a salient association about the named entity. Existing end-to-end RAG evaluations mix this reasoning problem with retrieval quality, making it difficult to determine whether the model actually follows the supplied relation.

The paper addresses this gap with Rel-Bloom, a multilingual diagnostic that holds the source relation fixed while varying cognitive operation, veracity, entity identity, and language. It omits Create because open-ended generation is incompatible with the controlled multiple-choice format. The objective is to determine whether Bloom-inspired difficulty orderings, knowledge-conflict effects, and item or relation difficulty structures are stable across model families and languages.

• Task: reason over relational facts supplied in context.
• Prior limitation: aggregate or end-to-end RAG accuracy confounds evidence use with retrieval and memorized entity knowledge.
• Paper-level objective: measure relational evidence following under matched cognitive and knowledge conditions.

🔬 Methodology

The benchmark begins with candidate relations from Wikidata5M, stratified into Low, Medium, and High usage tiers using MediaWiki property-frequency counts. The authors manually select 10 relations per tier and one representative factual triple (h, p, t) per relation, prioritizing semantic clarity, domain diversity, and non-overlapping entities. GPT-5 assists with fact verbalization and question generation, after which questions, answer options, and keys are manually reviewed for factual correctness, relation preservation, answer uniqueness, and distractor plausibility.

Each source relation is instantiated at Remember, Understand, Apply, Analyze, and Evaluate. Factual items retain real entity names and the extracted relation; Counterfactual items alter exactly one argument while preserving coarse entity type and relation context; Neutralized items replace entity mentions with typed placeholders. The resulting 450 items are evaluated in Arabic, Chinese, English, Hindi, Persian, and Spanish under hybrid cross-lingual and fully translated settings.

The statistical design uses ten pre-specified hypotheses. Matched contrasts for scale, Bloom level, veracity, and neutralization are aggregated within the 30 source relations and evaluated with one-sided relation-cluster sign-flip tests and 95% relation-cluster bootstrap intervals. Usage-frequency effects use a 5 percentage-point equivalence margin; cross-language stability uses item-level Spearman correlations and relation-identity permutation tests.

• Model coverage: 41 open-weight models from seven families, analyzed as eight series, plus two Gemini models.
• Evaluation coverage: 226,800 predictions across six languages and two multilingual settings.
• The benchmark preserves matched relations across level and veracity comparisons.

📊 Experiments

The central benchmark is Rel-Bloom, constructed from Wikidata5M candidates and 30 manually curated Wikidata relations. It contains 90 questions per Bloom level, with 30 questions per veracity condition within each level and 10 questions per usage tier. The evaluation covers Arabic, Chinese, English, Hindi, Persian, and Spanish in hybrid cross-lingual and fully translated settings, using 41 open-weight models from seven families, eight open-weight series, and two Gemini models.

Scaling generally improves accuracy. In both multilingual settings, Medium significantly exceeds Small in all eight open-weight series and Large exceeds Medium in seven; DeepSeek is the exception to the latter transition. Qwen2.5 Instruct gains 23.8 percentage points from Small to Medium and 6.1 points from Medium to Large. Diminishing returns hold in five of eight series in the fully translated setting and six of eight in the hybrid setting.

Bloom-level results are non-monotonic across task types. Apply exceeds Analyze across all eight open-weight series in both settings, but the magnitude varies from a 47.2-point drop for Large Granite to a 7.2-point drop for Qwen3.5. Neither Gemini model shows this Apply-to-Analyze drop. Evaluate remains difficult for every open-weight series and both Gemini models; Gemini 3.1 Pro averages 99.5% on the other four levels but 60.7% on Evaluate.

Counterfactual is generally the weakest veracity condition, but the complete Neutralized > Factual > Counterfactual ordering is statistically supported in only a few size buckets and no series as a whole. Neutralized-over-Factual gains at higher levels recur in several open-weight regimes but weaken or reverse in stronger series. Usage-frequency equivalence is inconclusive: no open-weight series satisfies the stronger interval-based criterion. Item-level and relation-level difficulty stability is consistently positive across series, size buckets, and both Gemini models.

• Primary metric: multiple-choice accuracy, reported separately by level, condition, model scale, language, and setting.
• Cross-language stability: mean of 15 pairwise item-level Spearman correlations and relation-level permutation tests.
• Most consistently supported hypotheses: H9 and H10 cross-language stability, H3 Apply > Analyze for open-weight models, H8 Evaluate bottleneck, and Small-to-Medium scaling.

Figure 1

Caption: Figure 1: Overview of REL-BLOOM. The same 30 curated Wikidata source relations (10 per relation usage frequency tier) are each instantiated at all five Bloom-inspired levels (shown as a pyramid) and under all three veracity conditions, yielding 30 × 5 × 3 = 450 English items; this full crossing allows level and condition effects to be compared on matched relations. Items are then evaluated in six languages under two multilingual settings. The 43 models comprise 41 open-weight models from seven model families, analyzed as eight model series because Qwen2.5 Base and Qwen2.5 Instruct are treated separately, plus two Gemini models.

Why It Matters: The figure makes the benchmark's matched-factor design explicit: the same relations are reused across cognitive levels and veracity conditions before multilingual evaluation, enabling controlled comparisons.

🔮 Conclusion

The paper's results indicate that reliable reasoning over supplied relational evidence is not implied by general model capability or high factual accuracy. A model can perform well on direct extraction and schema application yet fail when it must reject a near-miss relation or identify the defining justification for a relation. Counterfactual and Neutralized variants reveal whether entity names assist reasoning or introduce competing associations.

For RAG evaluation, the practical recommendation is to report accuracy by operation, include counterfactual and entity-neutralized context variants, and measure multilingual behavior directly even when cross-language difficulty rankings appear stable. These practices expose failures that aggregate factual scores can conceal.

• Evaluate evidence following separately from retrieval quality.
• Treat Analyze and Evaluate as distinct diagnostic tasks.
• Use multilingual item and relation difficulty checks alongside language-specific accuracy.

🛠️ Future Research Improvements

The most direct extension is to add multiple source triples per relation, increasing relation-cluster power and testing whether observed difficulty patterns transfer across entities and facts. Future datasets should also vary distractor plausibility within a fixed Bloom level, especially for Evaluate, to separate definitional reasoning from simply choosing among harder alternatives.

Further work should connect the diagnostic to realistic RAG pipelines with noisy retrieval, multi-hop context, and open-ended generation. It should also measure memorization or entity familiarity directly, preregister the hypothesis tests, and examine more checkpoints in the memory-competition analysis.

• Increase triple coverage per relation.
• Control distractor plausibility within each task level.
• Add noisy retrieval and open-ended generation.
• Measure entity familiarity and memorization directly.
• Replicate error analyses across more checkpoints and model families.

🏭 Potential Industry Use Scenarios

Rel-Bloom-style tests can serve as a pre-deployment diagnostic for enterprise RAG systems that answer questions over retrieved knowledge bases. A production evaluation can compare factual, counterfactual, and entity-neutralized passages to determine whether a model follows retrieved relations or defaults to familiar entity associations.

The benchmark structure is also useful for multilingual quality assurance. Teams can track accuracy separately for relation extraction, paraphrase understanding, schema application, near-miss discrimination, and definitional justification across supported languages. This can reveal whether a model's apparent multilingual reliability comes from stable reasoning or from uneven language-specific performance.

For monitoring, the representative memory-competition errors suggest a targeted audit category: cases where a named entity activates a salient but irrelevant attribute. Such tests could support regression suites for knowledge assistants, search-grounded agents, and domain systems that must respect updated or counterfactual records.

• RAG grounding and evidence-following audits.
• Multilingual model release and regression testing.
• Knowledge-base update and conflict-resolution evaluation.
• Targeted monitoring for entity-associated memory interference.

💬 Critical Analysis

The benchmark's strongest design choice is its matched crossing of relation, task level, and veracity condition. Reusing the same source relation makes comparisons more interpretable than evaluations that change the underlying knowledge item at each difficulty level. The multilingual analysis also goes beyond aggregate accuracy by testing whether item and relation difficulty rankings are preserved across languages.

The central interpretive risk is construct validity. Analyze and Evaluate differ not only in nominal cognitive operation but also in distractor structure: Analyze distractors are near-miss relations that can be rejected against the fact, whereas Evaluate distractors are plausible statements that are not definitional. The persistent Evaluate gap may therefore reflect item design, distractor plausibility, or a reasoning limitation; the current evidence does not isolate these explanations.

The conclusions about memory competition are appropriately cautious. The reported errors are consistent with entity-specific associations overriding the requested relation, but they do not establish an internal causal mechanism. Similarly, the cross-language correlations may partly reflect shared English-origin templates and the single-triple-per-relation design. Rel-Bloom is therefore valuable as a controlled diagnostic, but its results should be treated as evidence about the evaluated relation instances and task constructions rather than as a complete theory of multilingual reasoning.

• Strength: tightly matched experimental factors support interpretable contrasts.
• Strength: multilingual stability is assessed at both item and relation levels.
• Caveat: task construction and distractor plausibility may explain part of the level gaps.
• Caveat: observed memory-competition errors are behavioral evidence, not mechanistic proof.
• Caveat: one triple per relation limits external validity.