Research PapersRDR75
Causally Evaluating the Learnability of Formal Language Tasks
Researchers propose a new methodology for evaluating the learnability of tasks in language models, moving beyond standard correlational analysis. By using formal languages derived from probabilistic finite automata, they introduce the 'binning semiring' to causally control data frequency and measure learnability. This approach aims to address the inherent flaws in correlational evaluations, which can lead to incorrect conclusions.