What I Read: Language world models or surface statistics?