llm_with_cpp

LLM inference with cpp.

Only for fun.

Referenced from llama2.c

Platform

• Ubuntu 22.04.3
• GCC 17

Usage

1. Clone this project.

git clone https://github.com/SuperJokerayo/llm_with_cpp.git

2. Install sentencepiece tokenizer.

sudo apt-get install cmake build-essential pkg-config libgoogle-perftools-dev

git clone https://github.com/google/sentencepiece.git ./third_party/sentencepiece/

3. Download LLM checkpoint from llama2.c repo. For example:

# download OG model which has 15M parameters
wget -P ./checkpoints/  https://huggingface.co/karpathy/tinyllamas/resolve/main/stories15M.bin

4. Compile this project with cmake tool and run.

mkdir build
cd build
cmake
make -j $nproc

then, the executable file is put in ./bin

and you can run with:

./bin/run

5. The config parameters are written in config.ini, and custom config is supported.

6. You can also run the shell script if dependencies are installed manually.

bash ./run.sh

License

Have a look at the LICENSE file for details.