Causal reasoning from time-series data

Our US macro causal world model provided a reference for our evaluation, which found that LLMs often missed the effect of the evidence. In one scenario, weak wage growth sharply reduced the calculated risk of persistent inflation. LLM estimates stayed close to the probability obtained by ignoring the wage evidence.

Why causal reasoning is important

Causal reasoning estimates how an action changes an outcome and how new observations should change a forecast. It can also estimate what would have happened under a different past action or condition.

For example, an analyst assessing inflation risk needs to know how wage data changes the probability of persistent inflation. The answer can affect portfolio choices and risk limits. A security team choosing which vulnerability to fix first needs to estimate how each fix would change the risk of a breach.

The causal world model is fitted to observed data

Our US macro causal world model links wages, prices, and other economic measures, with delayed effects and uncertainty fitted to historical and forward-dated data. Figure 1 shows the model. We use it to generate economic histories and compute answers to causal questions.

US macro causal world model showing the causal links between economic measures. — *Figure 1. US macro causal world model. The arrows show the modelled causal links between economic measures.*

We assess how well the model captures economic behavior by checking its forecasts against observed outcomes. In our published macro forecast comparison, PolyBridge had lower forecast error than TimesFM-3 on six of seven indicators and lower error than public reference forecasts on six of seven. Forecasts used only information available at each forecast date. These results support using the causal world model as an environment for economic reasoning tests. Within that environment, we can calculate reference answers.

We gave LLMs histories generated by the US macro causal world model and told them to treat that world as complete and self-contained. We calculated reference answers from the same histories and compared LLM estimates with a baseline that ignores the relevant evidence.

LLM causal reasoning results

Here are three examples of tests we ran:

1. Conditioning on wage evidence

New wage data can change an analyst's assessment of inflation risk. We asked whether core inflation would remain above its February level two months later, given weak wage growth. Using the wage evidence substantially lowered the calculated probability. LLM estimates were close to what we would get by ignoring the wage evidence altogether. They put the probability above half (55%–67%), compared with the 26% calculated using the evidence.

2. Intervention on wage growth

Assessing an action requires estimating how changing one economic driver affects another. We tested this by setting wage growth to a fixed low level in the causal world model, replacing its usual causes, and asking how inflation risk changed. LLMs understated the reduction in risk but some estimates stayed close to the baseline that ignores the intervention.

3. Counterfactual energy inflation

Explaining a past inflation surge requires estimating how much of it came from energy prices. We asked what inflation would have been in the same cases if the earlier energy-price surge had been removed. High inflation became very unlikely in the causal world model. LLM estimates varied widely, with some near or above the baseline that ignores the change.

Figure 2 summarises the results. Median scores for using observed evidence were below the baseline in each of the five tested setups. Intervention and counterfactual scores varied. Claude Code + Opus 5 did better on those tasks, though its median score for using observed evidence stayed below the baseline.

Surprisingly, giving LLMs the map of causal links did not improve scores overall. More sampling also did not resolve the failures.

Median causal skill scores for five LLM setups on conditioning, intervention, and counterfactual questions. — Figure 2. Median causal skill for conditioning, intervention, and counterfactual questions across 750 responses. A score of 0 matches the baseline that ignores the relevant causal evidence; negative scores indicate larger errors, and 100 indicates no error. The ± values show 90% response-sampling margins.

Model and agent teams can contact [email protected] to discuss a sealed evaluation set, or held-out benchmark.