Activation-Based Active Learning for In-Context Learning: Challenges and Insights

Researchers explored whether transformer model activations could provide a fine-grained signal to optimize the selection of in-context examples for large language models. The study conducted a comprehensive analysis of MLP activation-based deep active learning methods applied to in-context learning, examining how different attention masking strategies impact active learning across various classification and generative datasets. The investigation utilized both Llama-3.2-3B and Qwen2.5-3B base models.

Contrary to the initial hypothesis, the findings indicated a negative correlation: MLP outputs, when analyzed through the lens of massive activations or the first four moments, did not correlate with example quality or task performance. The absolute Spearman correlation coefficient was at most 0.33 across all tested tasks and models, leading to the conclusion that such activation-based sampling should not be used for in-context learning. The authors hypothesize that this lack of correlation might be due to superposition, where models represent more features than their dimensionality allows, and suggest that methods like Sparse Autoencoders (SAEs) could be a promising future direction.