This is a Scala port of Andrej Karpathy's llama2.c, a bare bones implementation to run inference of models with a Llama-like transformer-based LLM architecture.

The code expects tokenizer.bin and stories15M.bin in the current directory.

So far, there's nothing original to see here, just a port of the original code. I'm planning to do some experiments, to raise the abstraction layer to a more reasonable level (e.g. remove all of the while-loops) but I wanted to start with an almost verbatim port to get some intial performance numbers to see if it even makes sense to continue on the JVM.

There are currently two implementations:

• Llama2SimpleTransformer which is a direct port of the original C code
• Llama2TensorTransformer which uses a Tensor abstraction to make the code more readable

Current numbers on my AMD Ryzen 7 4800H laptop:

So, with a fast JVM, there's little slowdown compared to the original C code, even if the Scala code has not been optimized at all yet (aside from being written in a hardly bearable C-like fashion...).

MIT