Skip to content

Commit

Permalink
[Docs] ggml: add TriLM and BitNet documents in basic/README.md
Browse files Browse the repository at this point in the history
Signed-off-by: dm4 <[email protected]>
  • Loading branch information
dm4 committed Nov 14, 2024
1 parent 610f4c2 commit d75c738
Showing 1 changed file with 23 additions and 0 deletions.
23 changes: 23 additions & 0 deletions wasmedge-ggml/basic/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,3 +82,26 @@ $ wasmedge --dir .:. \
--nn-preload default:GGML:AUTO:grok-1-Q2_K-split-00001-of-00009.gguf \
wasmedge-ggml-basic.wasm default 'hello'
```

## TriLM & BitNet Models

After the following pull requests are merged, the `TriLM` and `BitNet` models will be supported by the `ggml` plugin with model type `TQ1_0` and `TQ2_0`:
- https://github.com/ggerganov/llama.cpp/pull/7931
- https://github.com/ggerganov/llama.cpp/pull/8151

### Get the Model

Download the `TriLM` model:

```bash
curl -LO https://huggingface.co/Green-Sky/TriLM_3.9B-GGUF/resolve/main/TriLM_3.9B_Unpacked-4.0B-TQ2_0.gguf
```

### Execute

```console
$ wasmedge --dir .:. \
--env n_predict=100 \
--nn-preload default:GGML:AUTO:TriLM_3.9B_Unpacked-4.0B-TQ2_0.gguf \
wasmedge-ggml-basic.wasm default
```

0 comments on commit d75c738

Please sign in to comment.