Rotate Device
This calculator works best in landscape orientation for optimal button size and visibility.
You can still use it in portrait, but the experience will be better rotated. Tap anywhere to dismiss
This calculator works best in landscape orientation for optimal button size and visibility.
You can still use it in portrait, but the experience will be better rotated. Tap anywhere to dismiss
On this page, we invite you to experiment with the Surprisal Calculator WM 7±2, a tool developed to turn the flow of information in human language into melodies. According to surprisal theory, the more surprising a sentence is, the longer it takes for the human brain to understand. To see this in action, consider the two sentences below:
Even though the word cat appears in both sentences, it's far more surprising in the context of the second sentence. Odds are that the second sentence also took you longer to process.
This seemingly fuzzy notion of "surprisingness" can actually be quantified using a formula first developed by Claude Shannon, the father of information theory. An event's surprisal is calculated as the negative log probability of that event happening in its given context. Applied to language, each word is an event, and the context is the sentence it appears in. Large language models, like GPT, provide a convenient way to calculate word probabilities in context.
Coming back to our examples: in sentence 1, cat has a low surprisal value, because it appears in a sentence where talking about cats is very probable. In sentence 2, cat has a high surprisal value, because lawyers giving cats lawsuits is improbable.
Surprisal is a fascinating way to measure how the human brain handles unpredictability in language. However, surprisal numbers aren't very intuitive to understand on their own. With the Surprisal Calculator WM 7±2, you can now input any sentence and hear its surprisal values translated to pitch intervals: a low pitch indicates low surprisal, and a high pitch indicates high surprisal.
The decision to translate surprisal numbers to musical pitches was not arbitrary. A good, informative sentence is made from a blend of predictability (low surprisal) and spontaneity (high surprisal); the same could also be said of a good melody. We quickly bore of a sentence made only of repetitive, predictable words ("he said that he thought that she said that they think that…"), we also quickly bore of a melody made of the same note, repeated over and over again. The delicate balancing act of predictability and spontaneity that plays out in both language and music seems to hint at a shared cognitive architecture used in both forms of expression.
Type any sentence in, and press compose to hear its surprisal values.
Use the calculator buttons to adjust instrument type, the LLM used to calculate probability scores, and the scale used to match surprisal numbers to pitches.
Try typing the beginning of a sentence, then clicking keys on the keyboard. This will generate text with the surprisal values corresponding to the key pitch: experiment with what a very low surprisal or very high surprisal sentence sounds like. We suggest playing a common melody on the keyboard ("jingle bells" is a good candidate). Good melodies, which balance repetition with variety, tend to generate good sentences.
You can also use the numbers 1-9 on your keyboard to play notes in the selected scale.
This project is open source. For the purposes of this demo, we have only made one model available in the calculator, but you can download the source code and run it with whatever combination of models you would like!
wobblybits/surprisal (source code)
Testing the Predictions of Surprisal Theory in 11 Languages (Wilcox et al., TACL 2023)
Expectation-based syntactic comprehension (Levy, Cognition 2008)
A mathematical theory of communication (Shannon, 1948)
GPT2: The smallest (124M parameters) version of the official open-sourced GPT-2 models released from OpenAI.
DistilGPT2: An English-language model (88.2M parameters) pre-trained with the supervision of the smallest version of Generative Pre-trained Transformer 2 (GPT-2).
SmolLM: The smallest (135M parameters) of the three models in the SmolLM series of state-of-the-art small language models developed by the Hugging Face research team. These models are built on Cosmo-Corpus, a meticulously curated high-quality training dataset.
Nano Mistral: A small (170M parameters) Mistral model built for general web text completion at extremely low resource use.
Qwen 2.5: A compact yet powerful language model (494M parameters) trained on a subset of the Magpie 300k Database to answer a variety of questions
Flan T5: Google's super small (74M parameters)version of the Text-To-Text Transfer Transformer (T5) model, fine-tuned on more than 1000 additional tasks (and more languages!).
Calculator display font is Portfolio 6x8 from the Old School PC Fonts project
Calculator button font is Fredoka by Milena Brandão
Instrument icons by Ains and IconPack at Flaticon
Sounds generated with Tone.js
Cat sound effect by bfcmusic
Cat playing with knitting yarn adapted from icons created by kerismaker at Flaticon