Skip to content

Commit

Permalink
docs: how to import models
Browse files Browse the repository at this point in the history
  • Loading branch information
0xSage committed Dec 15, 2023
1 parent 66cd181 commit 9567ab9
Showing 1 changed file with 85 additions and 4 deletions.
89 changes: 85 additions & 4 deletions docs/docs/guides/quickstart.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,90 @@
---
title: Quickstart
description: Jan is a ChatGPT-alternative that runs on your own computer, with a local API server.
keywords: [Jan AI, Jan, ChatGPT alternative, local AI, private AI, conversational AI, no-subscription fee, large language model ]
keywords:
[
Jan AI,
Jan,
ChatGPT alternative,
local AI,
private AI,
conversational AI,
no-subscription fee,
large language model,
]
---

- Write in the style of comics, explanation
- Similar to why's (poignant) Guide to Ruby
- https://en.wikipedia.org/wiki/Why%27s_(poignant)_Guide_to_Ruby
Jan is compatible with all GGUF models.

In this guide we will use our latest model, [Pandora](https://huggingface.co/janhq/pandora-v1-10.7b-GGUF), as an example.

## 1. Create a model folder

Navigate to `~/jan/models` folder on your computer.

In `App Settings`, go to `Advanced`, then `Open App Directory`.

```sh
# Windows
C:/Users/<your_user_name>/jan/models

# MacOS/Linux
jan/models
```

In the `models` folder, create a folder with the name of the model.

```sh
mkdir pandora-v1-q4
```

## 2. Create a model JSON

Jan follows a standardized model template, called a `model.json`. This allows for easy model configurations, exporting, and sharing.

```sh
cd pandora-v1-q4
touch model.json
```

The following is an example template for `model.json`

```
{
"source_url": "https://huggingface.co/janhq/pandora-v1-10.7b-GGUF/blob/main/pandora-v1-10.7b.Q4_K_M.gguf",
"id": "pandora-v1-10-7b-gguf",
"object": "model",
"name": "PandoraQ4",
"version": "1.0",
"description": "A helpful assistant",
"format": "gguf",
"settings": {
"ctx_len": 2048,
"prompt_template": "<|im_start|>system\n{system_message}<|im_end|>\n<|im_start|>user\n{prompt}<|im_end|>\n<|im_start|>assistant"
},
"parameters": {
"max_tokens": 2048
},
"metadata": {
"author": "Jan",
"tags": ["7B", "Finetuned"]
},
"engine": "nitro"
}
```

:::caution
Ensure the `source_url` is the link to download model

Ensure the `id` is the same with the new created folder

Ensure to choose right `prompt_template`
:::

# 3. Use your model

Restart the Jan application and look for the your new custom model in the Hub.

![image](https://hackmd.io/_uploads/HJLAqvwI6.png)

There you go. If you have any questions or request GGUF version of any models, please message us on [Jan Discord](https://discord.gg/Dt7MxDyNNZ).

0 comments on commit 9567ab9

Please sign in to comment.