-
Notifications
You must be signed in to change notification settings - Fork 134
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #1704 from janhq/dev
sync dev to main 1.0.3-rc1
- Loading branch information
Showing
16 changed files
with
583 additions
and
173 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,207 @@ | ||
| --- | ||
| title: Proxy | ||
| description: Setting up Proxy | ||
| slug: "proxy" | ||
| --- | ||
|
|
||
| import Tabs from "@theme/Tabs"; | ||
| import TabItem from "@theme/TabItem"; | ||
|
|
||
| :::warning | ||
| 🚧 Cortex.cpp is currently under development. Our documentation outlines the intended behavior of Cortex, which may not yet be fully implemented in the codebase. | ||
| ::: | ||
|
|
||
| # Proxy Configuration Guide | ||
|
|
||
| This document describes how to configure proxy settings for Cortex to be able to work behind your proxy. | ||
|
|
||
| ## Command Line Interface (CLI) | ||
|
|
||
| ### Basic Usage | ||
|
|
||
| ```bash | ||
| cortex config [OPTIONS] [COMMAND] | ||
| ``` | ||
|
|
||
| ### Commands | ||
|
|
||
| - `status`: Display all current configurations | ||
|
|
||
| ```bash | ||
| cortex config status | ||
| ``` | ||
|
|
||
| Example Output: | ||
|
|
||
| ```bash | ||
| +-----------------------+------------------------+ | ||
| | Config name | Value | | ||
| +-----------------------+------------------------+ | ||
| | no_proxy | localhost,127.0.0.1 | | ||
| +-----------------------+------------------------+ | ||
| | proxy_password | | | ||
| +-----------------------+------------------------+ | ||
| | proxy_url | http://localhost:8080 | | ||
| +-----------------------+------------------------+ | ||
| | proxy_username | | | ||
| +-----------------------+------------------------+ | ||
| | verify_host_ssl | true | | ||
| +-----------------------+------------------------+ | ||
| | verify_peer_ssl | false | | ||
| +-----------------------+------------------------+ | ||
| | verify_proxy_host_ssl | true | | ||
| +-----------------------+------------------------+ | ||
| | verify_proxy_ssl | true | | ||
| +-----------------------+------------------------+ | ||
| ``` | ||
|
|
||
| ### Options | ||
|
|
||
| | Option | Description | Example | | ||
| | ----------------------------------- | --------------------------- | ------------------------------------------------- | | ||
| | `-h, --help` | Print help message and exit | | ||
| | `--proxy_url <proxy_url>` | Set the proxy URL | `cortex config --proxy_url http://localhost:8080` | | ||
| | `--proxy_username <proxy_username>` | Set the username for proxy | `cortex config --proxy_username my_username` | | ||
| | `--proxy_password <proxy_password>` | Set the password for proxy | `cortex config --proxy_password my_password` | | ||
| | `--no_proxy <no_proxy>` | Set the no_proxy list | `cortex config --no_proxy localhost,127.0.0.1` | | ||
| | `--verify_proxy_ssl [on/off]` | Verify proxy SSL | `cortex config --verify_proxy_ssl on` | | ||
| | `--verify_proxy_host_ssl [on/off]` | Verify proxy host SSL | `cortex config --verify_proxy_host_ssl on` | | ||
| | `--verify_peer_ssl [on/off]` | Verify peer SSL | `cortex config --verify_peer_ssl off` | | ||
| | `--verify_host_ssl [on/off]` | Verify host SSL | `cortex config --verify_host_ssl on` | | ||
|
|
||
| ## Proxy API Configuration | ||
|
|
||
| ### Endpoints | ||
|
|
||
| #### Get Current Configuration | ||
|
|
||
| ```http | ||
| GET /v1/configs | ||
| ``` | ||
|
|
||
| Retrieves the current configuration settings. | ||
|
|
||
| ##### Response | ||
|
|
||
| ```json | ||
| { | ||
| "allowed_origins": [ | ||
| "http://localhost:39281", | ||
| "http://127.0.0.1:39281", | ||
| "http://0.0.0.0:39281" | ||
| ], | ||
| "cors": true, | ||
| "no_proxy": "localhost,127.0.0.1", | ||
| "proxy_password": "", | ||
| "proxy_url": "http://localhost:8080", | ||
| "proxy_username": "", | ||
| "verify_host_ssl": true, | ||
| "verify_peer_ssl": false, | ||
| "verify_proxy_host_ssl": true, | ||
| "verify_proxy_ssl": true | ||
| } | ||
| ``` | ||
|
|
||
| #### Update Configuration | ||
|
|
||
| ```http | ||
| PATCH /v1/configs | ||
| ``` | ||
|
|
||
| Updates proxy configuration settings. | ||
|
|
||
| ##### Request Headers | ||
|
|
||
| ``` | ||
| Content-Type: application/json | ||
| ``` | ||
|
|
||
| ##### Request Body | ||
|
|
||
| ```json | ||
| { | ||
| "no_proxy": "localhost", | ||
| "proxy_url": "http://localhost:8080", | ||
| "proxy_username": "my_username", | ||
| "proxy_password": "my_password", | ||
| "verify_host_ssl": false, | ||
| "verify_peer_ssl": false, | ||
| "verify_proxy_host_ssl": false, | ||
| "verify_proxy_ssl": false | ||
| } | ||
| ``` | ||
|
|
||
| ##### Parameters | ||
|
|
||
| | Field | Type | Description | | ||
| | ----------------------- | ------- | -------------------------------------------------------------------------------------- | | ||
| | `no_proxy` | string | List of origins which request do not need to go through a proxy. Comma separated value | | ||
| | `proxy_url` | string | Proxy URL | | ||
| | `proxy_username` | string | Username for proxy authentication | | ||
| | `proxy_password` | string | Password for proxy authentication | | ||
| | `verify_host_ssl` | boolean | Verify host SSL | | ||
| | `verify_peer_ssl` | boolean | Verify peer SSL | | ||
| | `verify_proxy_host_ssl` | boolean | Verify proxy host SSL | | ||
| | `verify_proxy_ssl` | boolean | Verify proxy SSL | | ||
|
|
||
| ##### Response | ||
|
|
||
| ```json | ||
| { | ||
| "config": { | ||
| "allowed_origins": [ | ||
| "http://localhost:39281", | ||
| "http://127.0.0.1:39281", | ||
| "http://0.0.0.0:39281" | ||
| ], | ||
| "cors": true, | ||
| "no_proxy": "localhost", | ||
| "proxy_password": "my_password", | ||
| "proxy_url": "http://localhost:8080", | ||
| "proxy_username": "my_username", | ||
| "verify_host_ssl": false, | ||
| "verify_peer_ssl": false, | ||
| "verify_proxy_host_ssl": false, | ||
| "verify_proxy_ssl": false | ||
| }, | ||
| "message": "Configuration updated successfully" | ||
| } | ||
| ``` | ||
|
|
||
| ## Testing proxy configuration | ||
|
|
||
| You can test your proxy configuration using [mitmproxy](https://docs.mitmproxy.org/stable). This guide is written on macOS, but you can use it on any other platform. | ||
|
|
||
| ### Install mitmproxy | ||
|
|
||
| ```bash | ||
| brew install mitmproxy | ||
| ``` | ||
|
|
||
| ### Start mitmproxy | ||
|
|
||
| ```bash | ||
| mitmproxy --set stream_large_bodies=1m | ||
| ``` | ||
|
|
||
| mitmproxy will start on port `8080`. After mitmproxy started, you can adding options by pressing `O`. mitmproxy will display an option screen. You can check their document to learn more about mitmproxy. But let's take a simple option for now by setting the `proxyauth` for our local proxy. Inside the option screen, search for `proxyauth` and hit enter. Then, type `username:password` and hit enter again. You will see your newly added option is red-colored. | ||
|
|
||
| ### Configuring Cortex to use that proxy | ||
|
|
||
| Let's using CLI to configure Cortex to use that proxy. | ||
|
|
||
| ```bash | ||
| cortex config --proxy_url http://localhost:8080 --proxy_username username --proxy_password password | ||
| ``` | ||
|
|
||
| ### Testing the proxy | ||
|
|
||
| Now, let's test the proxy. If you are setting the username and password correctly (same with `proxyauth` in mitmproxy), you will see the request in mitmproxy. For example, command `cortex pull tinyllama` should be successfully and returns a list of selectable models. Also, you will see your request in mitmproxy CLI screen. | ||
|
|
||
| Let's try to use a wrong authentication for your proxy. | ||
|
|
||
| ```bash | ||
| cortex config --proxy_password wrong_pw | ||
| ``` | ||
|
|
||
| Now, let's test the proxy again. You will see the request is failed and returns an error. |
Oops, something went wrong.