Talkie, a 13B LM trained exclusively on pre-1931 data

AI researchers (Nick Levine, David Duvenaud, Alec Radford) just released “talkie,” a 13B language model trained on 260B tokens of text from before 1931, so it basically talks like someone whose worldview is stuck around 1930. The point is to study how LLMs actually generalize vs just memorize, since this model wasn’t trained on the modern web. They trained it on old books, newspapers, scientific journals, patents, and other historical text, then test things like whether it can come up with ideas that were discovered later, forecast future events, or learn bits of Python from examples. Early results seem pretty interesting too, with the model doing surprisingly well on core language/numeracy tasks and showing early signs of learning simple Python despite not being pretrained on modern code.