diff --git a/docs/docs/examples/chatboxgpt.md b/docs/docs/examples/chatboxgpt.md new file mode 100644 index 000000000..20abfbf0c --- /dev/null +++ b/docs/docs/examples/chatboxgpt.md @@ -0,0 +1,78 @@ +--- +title: Nitro on browser +description: Nitro intergration guide for using on Web browser. +keywords: [Nitro, Google Chrome, browser, Jan, fast inference, inference server, local AI, large language model, OpenAI compatible, open source, llama] +--- + +This guide demonstrates how to use Nitro on Web browser like an copilot on Web browser. + +## Installation +We'll use [chatGPTBox](https://github.com/josStorer/chatGPTBox) for this setup. + +- For Chrome/Edge, download directly from the [Google Web Store](https://chromewebstore.google.com/detail/chatgptbox/eobbhoofkanlmddnplfhnmkfbnlhpbbo). +- For other browsers, refer to their [installation guide](https://github.com/josStorer/chatGPTBox/wiki/Install). + +## Using chatGPTBox with Nitro + +**1. Start Nitro server** + +Open your terminal and run: + +```bash title="Run Nitro" +nitro +``` + +**2. Download Model** + +Download the [Stealth 7B](https://huggingface.co/jan-hq/stealth-v1.3) model with these commands: + +```bash title="Get a model" +mkdir model && cd model +wget -O stealth-7b.gguf https://huggingface.co/janhq/stealth-v1.3-GGUF/resolve/main/stealth-v1.3.Q4_K_M.gguf +``` + +> Explore more GGUF models at [The Bloke](https://huggingface.co/TheBloke). + +**3. Run the Model** + +To load the model, use the following command: + +```bash title="Load model to the server" +curl http://localhost:3928/inferences/llamacpp/loadmodel \ + -H 'Content-Type: application/json' \ + -d '{ + "llama_model_path": "model/stealth-7b.gguf", + "ctx_len": 512, + "ngl": 100, + }' +``` + +**4. Configure chatGPTBox** + +Click the `chatGPTBox` icon in your browser to open the main menu. + +![Main menu chatGPTBox](img/chatgptbox_menu.png) + +In the `API Mode` section, switch to `Custom Model`. + +Change the name to `Stealth 7B` and set the host to `http://localhost:3928/v1/chat/completions` + +![Use chatGPTBox](img/chatgptbox_cus.png) + +Your browser is now equipped with a copilot to help you on daily basis. + +![Web browser with chatGPTBox](img/chatgptbox_use.png) + +For detailed guidance, visit the [chatGPTBox guide](https://github.com/josStorer/chatGPTBox/wiki/Guide) + +### Shortcut + +| Shortcut | Description | +|----------|---------------------------------------------| +| Ctrl+B | Activate chat dialog box on any page. | +| Alt+B | Summarize any page via the right-click menu. | +| Ctrl+Shift+H | Open an independent conversation page. | + +## Futher Usage + +For convenient usage, you can utilize [Jan](https://jan.ai/), as it is integrated with Nitro. \ No newline at end of file diff --git a/docs/docs/examples/img/chatgptbox_cus.png b/docs/docs/examples/img/chatgptbox_cus.png new file mode 100644 index 000000000..752b35645 Binary files /dev/null and b/docs/docs/examples/img/chatgptbox_cus.png differ diff --git a/docs/docs/examples/img/chatgptbox_menu.png b/docs/docs/examples/img/chatgptbox_menu.png new file mode 100644 index 000000000..4862c7ede Binary files /dev/null and b/docs/docs/examples/img/chatgptbox_menu.png differ diff --git a/docs/docs/examples/img/chatgptbox_use.png b/docs/docs/examples/img/chatgptbox_use.png new file mode 100644 index 000000000..10883e88a Binary files /dev/null and b/docs/docs/examples/img/chatgptbox_use.png differ diff --git a/docs/docs/examples/palchat.md b/docs/docs/examples/palchat.md index 598a18104..438f071bc 100644 --- a/docs/docs/examples/palchat.md +++ b/docs/docs/examples/palchat.md @@ -27,7 +27,7 @@ Use these commands to download and save the [Llama2 7B chat model](https://huggi ```bash title="Get a model" mkdir model && cd model -wget -O llama-2-7b-model.gguf https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGUF/resolve/main/llama-2-7b-chat.Q5_K_M.gguf?download=true +wget -O llama-2-7b-model.gguf https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGUF/resolve/main/llama-2-7b-chat.Q5_K_M.gguf ``` > For more GGUF model, please look at [The Bloke](https://huggingface.co/TheBloke). diff --git a/docs/docs/features/log.md b/docs/docs/features/log.md new file mode 100644 index 000000000..d0e4511ef --- /dev/null +++ b/docs/docs/features/log.md @@ -0,0 +1,25 @@ +--- +title: Logging +description: Enabling logging in Nitro. +keywords: [Nitro, log, Jan, fast inference, inference server, local AI, large language model, OpenAI compatible, open source, llama] +--- + + +## Enabling Logging + +Nitro's logging feature can be activated by specifying a log folder. This is crucial for monitoring and troubleshooting. + +## Setting Up Logging + +To configure logging, you need to specify the path to the log folder. Use the following command to set it up: + +```bash title="Config logging" {5} +curl http://localhost:3928/inferences/llamacpp/loadmodel \ + -H 'Content-Type: application/json' \ + -d '{ + "llama_model_path": "/path/to/your_model.gguf", + "llama_log_folder": "/path/to/log/folder/" + }' +``` + +> **Note:** Ensure the log folder exists before running this command. If the specified folder does not exist, logs will default to your current directory. \ No newline at end of file diff --git a/docs/docs/features/vision.md b/docs/docs/features/vision.md new file mode 100644 index 000000000..22659c78f --- /dev/null +++ b/docs/docs/features/vision.md @@ -0,0 +1,65 @@ +--- +title: Vision +description: Inference engine for vision, the same as OpenAI's +keywords: [Nitro, Jan, fast inference, inference server, local AI, large language model, OpenAI compatible, open source, llava, bakllava, vision] +--- + +## Load model +Just like loading the Chat model, for the vision model, you need two specific types: +- the `GGUF model` +- the `mmproj model`. + +You can load the model using: + +```bash title="Load Model" {3,4} +curl -X POST 'http://127.0.0.1:3928/inferences/llamacpp/loadmodel' -H 'Content-Type: application/json' -d '{ + "llama_model_path": "/path/to/gguf/model/", + "mmproj": "/path/to/mmproj/model/", + "ctx_len": 2048, + "ngl": 100, + "cont_batching": false, + "embedding": false, + "system_prompt": "", + "user_prompt": "\n### Instruction:\n", + "ai_prompt": "\n### Response:\n" + }' +``` + +Download the models here: +- [Llava Model](https://huggingface.co/jartine/llava-v1.5-7B-GGUF/tree/main): Large Language and Vision Assistant achieves SoTA on 11 benchmarks. +- [Bakllava Model](https://huggingface.co/mys/ggml_bakllava-1/tree/main) is a Mistral 7B base augmented with the LLaVA architecture. + +## Inference + +Nitro currently only works with images converted to base64 format. Use this [base64 converter](https://www.base64-image.de/) to prepare your images. + +To get the model's understanding of an image, do the following: + +```bash title="Inference" +curl http://127.0.0.1:3928/v1/chat/completions \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer $OPENAI_API_KEY" \ + -d '{ + "model": "gpt-4-vision-preview", + "messages": [ + { + "role": "user", + "content": [ + { + "type": "text", + "text": "What’s in this image?" + }, + { + "type": "image_url", + "image_url": { + "url": "" + } + } + ] + } + ], + "max_tokens": 300 + }' +``` + +If the base64 string is too long and causes errors, consider using [Postman](https://www.postman.com/) as an alternative. \ No newline at end of file diff --git a/docs/sidebars.js b/docs/sidebars.js index 182db2e94..70c757953 100644 --- a/docs/sidebars.js +++ b/docs/sidebars.js @@ -39,6 +39,7 @@ const sidebars = { items: [ "features/chat", "features/embed", + "features/vision" ], }, { @@ -51,6 +52,7 @@ const sidebars = { "features/load-unload", "features/warmup", "features/prompt", + "features/log" ], }, { @@ -64,7 +66,8 @@ const sidebars = { "examples/palchat", "examples/openai-node", "examples/openai-python", - "examples/colab" + "examples/colab", + "examples/chatboxgpt" ], }, // {