tiny-llm

This project contains toy programs for simple LLMs.

The first program is tiny-net using a single neural network and a 30K parameter model.
In other words, it's a million times smaller than the normal 30B parameter models hobbyists
use.

It takes a classic neural-net, as described in textbooks, and shows clearly how we feed a token
in, then how we get a token out. In other words, textbooks introduce AI feeding images to the
neural-net when students these days probably want to learn about LLMs.

Training is done on Karpathy's "Tiny Shakespeare", a 1-megabyte file of Shakespeare text. It
looks something like this:

$ bin/tiny-net train net.bin corpus.txt
loaded existing model: net.bin
previous training: 2 runs, 2000000 updates, last-run average loss 2.5598, all-runs average loss 2.5743
training on 1115394 corpus bytes for up to 50000 one-pair SGD updates
press Ctrl-C to stop cleanly and save the model
step 1 / 50000, loss 1.9708, run average 1.9708
step 1000 / 50000, loss 5.8252, run average 2.5125
step 2000 / 50000, loss 1.7526, run average 2.5193
step 3000 / 50000, loss 2.9251, run average 2.5212
step 4000 / 50000, loss 4.3293, run average 2.5440

It measures loss, how accurate it is at predicting the next token. When loss stops
decreasing, then you've reached the point where more training isn't improving things.

Running inference is shown below, where the file prompt.txt contains simply the word hello,
and this toy LLM tries to predict things from here.

$ bin/tiny-net infer net.bin prompt.txt
hello 
TE:
thit aoushawh bil mag tl no RONI pl, atelausfir, trthes basbly.
Facay ef or, ce
He.
TEy!
thadst d asst?
PENCNT:
Buprathayooor, thiler mbisthis, ay tismepegherea l t pusckil f l prte d ctkeostr; Dofonkiljonk! y; rver.
O:

This is obviously gibberish, but gibberish that looks a little like Shakespeare,
such as the way that character names are ALLCAPS followed by a colon :.
This means training had an effect on the program.

I have two more versions of this, implementing an RNN and LSTM. Still very basic,
but we can see how each dramatically improves the output. This sets up our understanding
so that we can then start to address "attention" and "transformers".