LLM Consistency Checks Reveal Macro Fallacy in Persona Prompting

What changed

Researchers have proposed a novel evaluation framework to assess the statistical self-consistency of Large Language Models (LLMs), focusing on in-context learning (ICL) and persona prompting. The core idea is to treat LLM outputs as estimates of conditional distributions and check if they adhere to fundamental probabilistic laws, specifically the law of total probability. This law states that prior-weighted conditional distributions, when aggregated over a valid partition of a population, should yield the same marginal distribution as a direct estimate of the population.

The evaluation protocol uses binary trees to recursively partition a population into increasingly fine-grained subpopulations. LLMs are prompted with verbalized descriptions of these subpopulations, and their responses are aggregated back to form population-level estimates. These reconstructed aggregates are then compared against direct population-level estimates from the same LLM and, where available, against human reference data. The study applied this method across various problem domains and state-of-the-art LLMs.

A key finding is the identification of a pattern termed the 'macro fallacy.' This phenomenon occurs when estimates reconstructed from more detailed subpopulation responses are systematically more accurate and better aligned with human reference data than direct population-level estimates. This effect was observed to be robust across different tree structures, estimation tasks, and LLMs. The research suggests that while models may hold relevant knowledge about specific subpopulations, they often fail to reliably propagate this knowledge into their aggregate estimates.

Why it matters for builders

This research introduces a critical lens for evaluating LLM behavior beyond simple accuracy metrics. The discovery of the 'macro fallacy' implies that LLMs might not be performing true conditional inference as commonly assumed. For builders, this means that direct prompts asking for aggregate information might yield less reliable results than prompts that break down the query into constituent parts and then aggregate the responses. Understanding this limitation is crucial for developing applications that rely on probabilistic reasoning or require consistent aggregation of information.

Practical impact

Builders should consider implementing strategies to mitigate the macro fallacy. This could involve designing prompts that explicitly guide the model to consider subpopulation characteristics before providing an aggregate answer, or employing techniques that implicitly encourage better aggregation. The research indicates that implicit prompting can partially recover the accuracy lost due to the macro fallacy, suggesting that prompt engineering plays a significant role. Developers might also explore post-processing techniques to aggregate LLM outputs from fine-grained queries, potentially leading to more robust and accurate results than direct, high-level prompts. Further investigation into the specific mechanisms causing this fallacy and how to best address them through model fine-tuning or advanced prompting techniques is warranted.

Caveats and source limits

The study is presented as a research paper on arXiv, and its findings are based on the specific evaluation framework and models tested. The paper does not provide specific benchmark numbers for the tested models or detail the exact datasets used for human reference statistics. While the macro fallacy is described as persistent, the extent to which it affects all LLMs and all types of tasks remains an open question. The research suggests partial recovery through implicit prompting, but the optimal methods for achieving this are not fully detailed. The findings are primarily theoretical and methodological, and practical implementation guidance is limited to general observations about prompt engineering.