Learning When to Trust via Selective Context Preference Optimization

What changed

Researchers have introduced a new benchmark called MIST designed to evaluate how well language models can selectively trust external signals. This benchmark presents reasoning tasks under four conditions: clean, misleading, correct-context, and irrelevant-context. Alongside MIST, they propose SC2W, a metric that measures how often a misleading signal causes a model to produce an incorrect answer when it would otherwise be correct. The study observes that this susceptibility to misleading context is a universal issue across current models. To address this, they developed SCOPE, an optimization method that mines failures from clean-correct/misleading-wrong scenarios. SCOPE optimizes a Direct Preference Optimization (DPO) objective using preference pairs that are equally balanced across all four context conditions, rather than focusing solely on misleading items.

Why it matters for builders

This research is significant for AI builders as it tackles a fundamental challenge in deploying language models: their reliability when presented with external information. Models that are overly resistant to context become less useful, while those easily swayed by misleading signals are untrustworthy. The MIST benchmark and SCOPE method provide tools to train and evaluate models that strike a better balance, making them more robust and dependable in real-world applications where context is crucial.

Practical impact

The SCOPE method has demonstrated a substantial reduction in the SC2W metric on popular open-source models. Importantly, this improvement in handling misleading context does not come at the expense of accuracy when the provided context is clean, correct, or irrelevant. This means builders can potentially deploy models that are more resilient to misinformation without sacrificing their general performance on tasks requiring accurate information processing.

Caveats and source limits

The provided source is a research paper abstract and excerpt, detailing the proposed benchmark, metric, and optimization method. Specific details on the implementation of SCOPE, the exact performance gains across a wider range of models, or comprehensive comparisons with other existing methods are not fully elaborated in the excerpt. The project page and GitHub repository mentioned in the metadata may contain further implementation details.