Tracking the Limits of Knowledge Propagation:
How LLMs Fail at Multi-Step Reasoning with Conflicting Knowledge

A common solution for mitigating outdated or incorrect information in Large Language Models (LLMs) is to provide updated facts in-context or through knowledge editing. However, these methods introduce knowledge conflicts when the knowledge update fails to overwrite the model's parametric knowledge, which propagate to faulty reasoning. Current benchmarks for this problem largely focus only on single knowledge updates and fact recall without evaluating how these updates affect downstream reasoning. In this work, we introduce Track (Testing Reasoning Amid Conflicting Knowledge), a new benchmark for studying how LLMs propagate new knowledge through multi-step reasoning when it conflicts with the model's initial parametric knowledge. Spanning three reasoning-intensive scenarios (WIKI, CODE, and MATH), Track introduces multiple, realistic conflicts to mirror real-world complexity. Our results reveal that providing updated facts to models for reasoning can worsen performance compared to providing no updated facts at all, and that this performance degradation exacerbates as more updated facts are provided. We show this failure stems from both inability to faithfully integrate updated facts, and also flawed reasoning even when knowledge is integrated. Track provides a rigorous new benchmark to measure and guide future progress on propagating conflicting knowledge in multi-step reasoning.

We introduce Track to evaluate how LLMs propagate conflicting knowledge through multi-step reasoning. Unlike prior benchmarks, Track introduces multiple, realistic knowledge conflicts across diverse, reasoning-intensive domains to mirror real-world complexity and test true knowledge propagation. It uses a two-stage framework: ① Knowledge Probing first identifies each model's specific knowledge gaps, then ② Knowledge Injection provides conflicting facts at inference time to test whether the model can propagate them through reasoning.

Key Takeaways:

1. LLMs exhibit polarized knowledge confidence. They either know facts with high certainty or are clearly unaware.
2. Providing correct but conflicting facts yields limited gains and can even hurt performance.
3. Failures stem from both inability to integrate facts faithfully and flawed reasoning even when integration succeeds.

In the knowledge probing stage, we query each model on atomic facts derived from the benchmark questions, generating 10 responses per fact. We define Knowledge Confidence (KConf) as the proportion of correct responses, and classify a fact as known if the correct answer is the most frequent response.

In the knowledge injection stage, we test how models reason when given their own missing facts as conflicting knowledge. Methods include Base Model (no injection), Append (in-context facts), FT-CK (fine-tuning on conflicting knowledge), MeLLo (retrieval-augmented reasoning), and Append-T (extended thinking, for Qwen-3 and o4-mini).

We also vary Knowledge Aggregation Scope (KAS): the number of atomic facts provided at injection time. At KAS=1, only the one relevant fact is given. At KAS=10, 100, or 500, increasing numbers of irrelevant facts are mixed in, simulating a shared update environment where the model must find what is relevant.

To understand where failures come from, we analyze them at two levels. At the fact level, we track each atomic fact through the reasoning chain and record whether it was entailed (used correctly), directly failed (the first step needing it went wrong), or led to error propagation (later steps failed as a result of an earlier mistake).

At the answer level, we compare three metrics: AP (the rate of correct answers regardless of whether provided facts were used), FKE (the rate of cases where all required facts were correctly used in reasoning), and HP (the rate of correct final answers that also used all facts faithfully). The gap between AP and FKE reflects unfaithful integration; the gap between FKE and HP reflects flawed reasoning even after correct integration.