VLMs May Not Globally Enhance Human Alignment Over LLMs in Reading

What changed

This research investigates whether the multimodal training history of Vision-Language Models (VLMs) enhances their text representations to be more human-like during natural reading, compared to traditional Large Language Models (LLMs). The study employed a strictly text-only setting to isolate the impact of multimodal pretraining from online visual input or cross-modal fusion. Matched pairs of LLMs and VLMs were evaluated against a human natural-reading dataset, which included whole-cortex fMRI responses and synchronized eye-tracking saccades. The core finding is that multimodal pretraining does not confer a uniform, global advantage in human alignment during natural reading. Instead, language-internal representations appear to be the key determinant for modeling human text processing. However, a VLM advantage was observed more selectively when sentences contained stronger visual semantic content, supported by converging evidence from both fMRI and eye-movement alignments. This suggests that multimodal pretraining contributes selectively rather than globally to human-like language representations during natural reading.

Key Contributions:

• Multimodal pretraining does not automatically yield a stable, global advantage in human alignment during natural reading.
• VLM advantages are content-dependent, emerging more reliably for sentences with higher "visualness strength."
• A novel research paradigm using matched model pairs as controlled experimental tools to untangle the role of multimodal experience in human-like representations.

Why it matters for builders

For AI builders focused on creating models that mimic human language processing, this study provides a critical perspective. It challenges the assumption that simply incorporating visual data into training pipelines will universally lead to more human-like text understanding. The findings underscore the enduring significance of robust language-internal representations, suggesting that advancements in this area may be as, if not more, impactful than multimodal integration for general text comprehension tasks. Builders should carefully consider the specific requirements of their applications, particularly whether visual semantics play a dominant role, when designing and training their models.

Practical impact

AI developers working on natural language processing tasks, especially those aiming for high alignment with human cognitive processes, should consider these findings when selecting or developing models. If the goal is general text comprehension, prioritizing models with strong language-internal representations might be more effective than solely relying on VLMs. However, for applications where visual semantics are integral to the text (e.g., image captioning evaluation, visual storytelling analysis), VLMs may offer a distinct advantage. The research provides a framework for evaluating this trade-off, encouraging a more nuanced approach to multimodal model integration. Builders might explore fine-tuning existing LLMs on text-heavy datasets or investigating specific VLM architectures that excel in visually rich contexts.

Caveats and source limits

The study's findings are based on a specific experimental setup comparing matched LLM and VLM pairs under text-only conditions. The conclusions regarding the global vs. selective advantage of VLMs are drawn from this controlled environment. The research does not explore scenarios where VLMs process visual input concurrently with text, nor does it delve into the specific architectures or training methodologies of the models used beyond them being "tightly matched." The dataset used for evaluation, while including fMRI and eye-tracking data, represents a specific type of human natural reading. Further research may be needed to generalize these findings across different modalities, tasks, and model types. The source is a research paper, and independent verification of its claims through benchmark results or broader community adoption is not yet available.