Build to learn · character-level GPT
A small neural network trained on the poems of Vemana to write classical Telugu verse one character at a time. It runs here, in full, inside your browser.
Telugu text → characters → tiny GPT → generated Telugu
Many tokenizers behind multilingual AI systems are trained on data where English dominates. Their vocabularies end up shaped around English's structure first, with other scripts fitted in around it. That imbalance is usually inherited rather than examined.
For Telugu, the cost is concrete. A written syllable is assembled from several Unicode characters, and words grow long strings of grammatical endings. A vocabulary tuned for English fragments Telugu into far more pieces than it needs, which means more compute, higher cost, and less room in the model's context window for the same sentence.
This project reasons about that choice from the ground up: a language model trained from nothing, character by character, on a real Telugu corpus, small enough that every decision stays visible. The model itself is tiny and mostly rhymes nonsense. The point is understanding, concretely, why character-level fits here and what changes at a larger scale.
The training text is the Vemana Satakam: four-line moral verses attributed to the poet Vemana, each ending on the refrain విశ్వదాభిరామ వినర వేమ. It is public domain, transcribed on Telugu Wikisource across 34 alphabetically ordered pages. After scraping and cleaning, stripping markup, normalizing Unicode, removing invisible characters and duplicates, 1,362 verses remain, about 153,000 characters.
Vemana's verse count varies widely by edition, from the 700s to well over 2,000 in different print sources. This Wikisource compilation's own index claims 1,362, and that is exactly what cleaning produces.
The vocabulary is just 70 symbols: every Telugu letter and sign in the corpus, plus spaces and punctuation. Each appears thousands of times, giving the model plenty of evidence. Splitting on words would explode the vocabulary, since Telugu stacks endings into dozens of surface forms per stem. Splitting into sub-word fragments, the usual modern default, needs far more than 150,000 characters before its statistics mean anything.
Telugu is an abugida: what you read as one letter is usually built from several Unicode code points. A base consonant carries a built-in vowel sound, a vowel sign changes it, and a virama mark binds a second consonant underneath to form a conjunct. The model never sees a written letter. It sees the code points underneath it, one at a time:
One written letter, three separate predictions. The model has to learn from data alone that a virama sits between two consonants, and that a vowel sign follows its consonant.
Live · tokenizer
Type Telugu below to see the tokens underneath it.
Blue-outlined boxes are conjunct clusters. The small letters below each glyph are the individual tokens; the numbers below those are their token IDs.
Counting lines per verse, a handful came back with eight or nine lines instead of four. My first guess was a scraping bug: two verses stuck together.
Part of that was right. A few Wikisource pages had a mistyped closing tag, so my parser had merged consecutive verses. I fixed it, but five verses still weren't four lines. Reading them settled it: four are genuinely eight-line sīsa padyam, a classical metre with four long lines and a short four-line coda called an eththugīti. The fifth is a real three-line fragment, incomplete in the source. All five stay in the corpus.
One verse, id=190, is different. Its last line reads
విశ్వదాభిరామ రామరవు మ మమ where the refrain
విశ్వదాభిరామ వినర వేమ belongs. That's a
transcription defect, kept as an honest record rather than silently
patched. It is not the same thing as the spelling variation elsewhere
in the corpus, where the refrain legitimately appears as
వినర in some verses and
వినుర in others: real manuscript differences,
deliberately preserved. Manuscript variation is authentic text.
id=190 is garbled text.
That's the texture of real data work. "That looks like a bug," "that's how the thing actually works," and "that's broken but worth keeping" can all be true in the same dataset. The only way to tell them apart is to look at the actual rows.
It's a small GPT, the same architecture as the well-known ones, shrunk to about 818,000 numbers: four layers, four attention heads, 128-dimensional vectors. Here's the idea in plain terms.
Each of the 70 characters gets a list of 128 numbers, learned during training so that characters that behave similarly drift close together. A second set of 128-number lists encodes position: slot 5 in the sequence, slot 6, and so on. Attention has no built-in sense of order, so it needs to be told.
At every position the model produces a query, a key, and a value: what am I looking for, what do I offer, what do I pass along if attended to. It compares this position's query against every earlier position's key, turns the comparisons into weights that sum to one, and blends the values by those weights. Four heads run in parallel, stacked four layers deep, enough to route information like: the most recent virama was preceded by a consonant, so the next character is probably a consonant too.
The model only predicts the next character, so each position may look backward only. Every forward-looking comparison is blanked out before the weighting step. Without that mask the model would read the answer off the input and learn nothing useful for generation. After four layers, one small matrix turns each position's 128 numbers into 70 scores: one per possible next character.
The 1,362 verses split 90/10: 1,226 to train on, 136 held out and never shown. The split is by verse, not by character offset, since the corpus is alphabetically ordered and a character-offset split would dump every verse starting with later letters into the test set.
Loss, roughly how surprised the model is by the true next character, is measured on both sets every 250 steps. It starts near 4.25, the score of guessing blindly among 70 characters. Both curves fall together for a while, then part around step 2,000: training keeps dropping while held-out flattens and drifts back up. That gap is the model starting to memorize its 1,226 training verses instead of learning patterns that transfer. Expected, on a corpus this small. The useful part is seeing exactly when it starts.
3,000 steps, batches of 32 windows of 128 characters. Final scores: training 1.62, held-out 1.77. Roughly one-in-six uncertainty per character, down from one-in-70.
The model predicts one character, appends it, and feeds the longer string back in, repeatedly. Two controls shape the draw. Temperature scales the scores: below 1 it sharpens the model's favorites, above 1 it flattens them, giving rare characters a real chance. Top-k is blunter: it keeps only the k highest-scoring characters and samples among those.
Lower temperature doesn't mean safer text. At 0.2 the model collapsed into repeating కాలు కాలు కాలు, since near-zero temperature is close to always picking the single likeliest character, which loops. The most coherent verses came from the model's natural temperature of 1.0 with a small top-k trimming the tail. Pushing temperature to 1.4 broke spelling instead: stray vowel signs with no consonant to attach to.
Worth knowing: this is a tiny educational model trained on a small poetry corpus. It learns character and verse patterns. It does not understand Telugu or reproduce authentic Vemana poetry.
Live · generation
Pick a seed, or type your own Telugu, then generate.
Generation stops at the character limit above, not at a natural ending. It can stop mid-word or mid-refrain: the model has no notion of "finished," only "next character."
There is no server behind this page. It downloaded one 4.4 MB JSON file containing every model weight, exported from PyTorch as 32-bit numbers. The transformer forward pass and the sampling loop are reimplemented from scratch in plain JavaScript: about 250 lines, no libraries.
The fair question is whether that JavaScript is really the same model, or just a lookalike. I ran an identical 21-character input through both the PyTorch model and the JavaScript port and compared all 1,470 output numbers. The largest disagreement was 0.0000032, and both versions picked the identical next character at every position. That residue is float32 versus float64 rounding, not approximation.
A static page opens instantly for anyone with the link and costs nothing to host, which is what makes it shareable. The price is writing the forward pass twice, and proving the copies match with the parity check above.
I built this as a learning exercise: a small, fully inspectable reference for reasoning about tokenizer and architecture choices in Indic-language systems, instead of gesturing at a pretrained model's documentation. The Vemana Satakam poems are historically public domain; the specific transcription used here is వేమన పద్యాలు on Telugu Wikisource, licensed CC BY-SA 4.0. Code, training scripts, and reproduction steps are in the project repository, under MIT.
A few directions I'd take this next, not built yet:
tinygpt.js already returns per-head weights when asked.