From 84e97a1c9808319ef9efd440fd4ee191a3393920 Mon Sep 17 00:00:00 2001 From: Mike Bird <63524998+MikeBirdTech@users.noreply.github.com> Date: Mon, 30 Sep 2024 15:03:18 -0400 Subject: [PATCH 1/4] Added 'Jan Computer Control' Jupyter notebook to demonstrate controlling a computer using Jan's chat interface with Open Interpreter. --- examples/jan_computer_control.ipynb | 73 +++++++++++++++++++++++++++++ 1 file changed, 73 insertions(+) create mode 100644 examples/jan_computer_control.ipynb diff --git a/examples/jan_computer_control.ipynb b/examples/jan_computer_control.ipynb new file mode 100644 index 0000000000..99375affcc --- /dev/null +++ b/examples/jan_computer_control.ipynb @@ -0,0 +1,73 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Jan Computer Control\n", + "\n", + "We love Jan as an A.I. inference server. It also has a chat interface to chat with LLMs. But did you know that you can use this same chat interface as a computer control interface? Read on!\n", + "\n", + "[View on YouTube](https://www.youtube.com/watch?v=1l3B0AzbbjQ)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Install and set up Jan\n", + "\n", + "https://jan.ai/" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Install Open Interpreter\n", + "\n", + "https://docs.openinterpreter.com/getting-started/introduction" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Run the Open Interpreter OpenAI-compatible server.\n", + "\n", + "`interpreter --server`\n", + "\n", + "Add flags to set the `--model`, `--context_window`, or any other [setting](https://docs.openinterpreter.com/settings/all-settings) you want" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Edit Jan's OpenAI settings to point to the local server.\n", + "\n", + "Settings => OpenAI => Chat Competion endpoint `http://127.0.0.1:8000/openai/chat/completions`.\n", + "\n", + "Jan has a requirement to set a dummy OpenAI API key." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Go to the chat window to start a new thread.\n", + "\n", + "Set `Model` to an OpenAI model. \n", + "\n", + "Start controlling your computer!" + ] + } + ], + "metadata": { + "language_info": { + "name": "python" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} From 33ca1ebc1e097b0b36a2903cc087d7727bbbe132 Mon Sep 17 00:00:00 2001 From: Mike Bird <63524998+MikeBirdTech@users.noreply.github.com> Date: Thu, 3 Oct 2024 12:16:28 -0400 Subject: [PATCH 2/4] Update Jan's chat window instructions and add new notebook on using Open Interpreter to talk to your database. --- examples/jan_computer_control.ipynb | 2 +- examples/talk_to_your_database.ipynb | 156 +++++++++++++++++++++++++++ 2 files changed, 157 insertions(+), 1 deletion(-) create mode 100644 examples/talk_to_your_database.ipynb diff --git a/examples/jan_computer_control.ipynb b/examples/jan_computer_control.ipynb index 99375affcc..2af81278bf 100644 --- a/examples/jan_computer_control.ipynb +++ b/examples/jan_computer_control.ipynb @@ -55,7 +55,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Go to the chat window to start a new thread.\n", + "Go to Jan's chat window to start a new thread.\n", "\n", "Set `Model` to an OpenAI model. \n", "\n", diff --git a/examples/talk_to_your_database.ipynb b/examples/talk_to_your_database.ipynb new file mode 100644 index 0000000000..805292ec37 --- /dev/null +++ b/examples/talk_to_your_database.ipynb @@ -0,0 +1,156 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Use Open Interpreter to talk to your database" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "> Note: Open Interpreter should ideally be limited to read-only actions on your database. If write operations are necessary, use a copy of your data to protect against unexpected changes from the AI model. " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "vscode": { + "languageId": "plaintext" + } + }, + "outputs": [], + "source": [ + "Set up a profile with:\n", + "- Database credentials\n", + "- Connection string\n", + "\n", + "Here is an example for a PostgreSQL database:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from interpreter import interpreter\n", + "import os\n", + "\n", + "# Use environment variables for database connection or update defaults with your credentials\n", + "db_user = os.environ.get(\"DB_USER\", \"user\")\n", + "db_host = os.environ.get(\"DB_HOST\", \"localhost\")\n", + "db_port = os.environ.get(\"DB_PORT\", \"5432\")\n", + "db_name = os.environ.get(\"DB_NAME\", \"demo_database\")\n", + "db_password = os.environ.get(\"DB_PASSWORD\", \"\")\n", + "\n", + "# Construct connection string with optional password\n", + "if db_password and db_password.strip():\n", + " connection_string = (\n", + " f\"postgresql://{db_user}:{db_password}@{db_host}:{db_port}/{db_name}\"\n", + " )\n", + "else:\n", + " connection_string = f\"postgresql://{db_user}@{db_host}:{db_port}/{db_name}\"" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Configure your instance of Open Interpreter.\n", + "\n", + "\n", + "This example uses a local model served by Ollama but you can use a hosted model:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# LLM settings\n", + "interpreter.llm.model = \"ollama/llama3.1\"\n", + "interpreter.llm.supports_functions = False\n", + "interpreter.llm.execution_instructions = False\n", + "interpreter.llm.max_tokens = 1000\n", + "interpreter.llm.context_window = 7000\n", + "interpreter.llm.load() \n", + "\n", + "# Computer settings\n", + "interpreter.computer.import_computer_api = False\n", + "\n", + "# Misc settings\n", + "interpreter.auto_run = False\n", + "interpreter.offline = True" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Set the custom instructions to maximize performance." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Custom Instructions\n", + "interpreter.custom_instructions = f\"\"\"\n", + " You are a SQL master and are the oracle of database knowledge. You are obsessed with SQL. You only want to discuss SQL. SQL is life.\n", + " Recap the plan before answering the user's query.\n", + " You will connect to a PostgreSQL database, with the connection string {connection_string}.\n", + " Remember to only query the {db_name} database.\n", + " Execute valid SQL commands to satisfy the user's query.\n", + " Write all code in a full Python script. When you have to re-write code, redo the entire script.\n", + " Execute the script to get the answer for the user's query.\n", + " **YOU CAN EXECUTE SQL COMMANDS IN A PYTHON SCRIPT.***\n", + " Get the schema of '{db_name}' before writing any other SQL commands. It is important to know the tables. This will let you know what commands are correct.\n", + " Only use real column names.\n", + " ***You ARE fully capable of executing SQL commands.***\n", + " Be VERY clear about the answer to the user's query. They don't understand technical jargon so make it very clear and direct.\n", + " You should respond in a very concise way.\n", + " You can do it, I believe in you.\n", + " \"\"\"" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "Save the profile in the `profiles` directory.\n", + "\n", + "Once you are happy with your profile, test it on a test table/database. \n", + "\n", + "Run the following in your terminal:\n", + "\n", + "`interpreter --profile `\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Iterate on the profile until you are happy with the performance. \n", + "\n", + "Remember to use the right LLM for the job. Larger models tend to have better reasoning.\n", + "\n", + "If you want to share your profile with the community, please open a PR." + ] + } + ], + "metadata": { + "language_info": { + "name": "python" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} From 89822854b252adc28992e1a9ff1a14311087a949 Mon Sep 17 00:00:00 2001 From: Mike Bird <63524998+MikeBirdTech@users.noreply.github.com> Date: Tue, 8 Oct 2024 14:44:09 -0400 Subject: [PATCH 3/4] Created a custom tool notebook that demonstrates how to add a custom tool to an Open Interpreter instance, including configuring the instance and defining a custom tool using Python. --- examples/custom_tool.ipynb | 127 +++++++++++++++++++++++++++++++++++++ 1 file changed, 127 insertions(+) create mode 100644 examples/custom_tool.ipynb diff --git a/examples/custom_tool.ipynb b/examples/custom_tool.ipynb new file mode 100644 index 0000000000..2b956d4a48 --- /dev/null +++ b/examples/custom_tool.ipynb @@ -0,0 +1,127 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Add a Custom Tool to your Instance\n", + "\n", + "You can create custom tools for your instance of Open Interpreter. This is extremely helpful for adding new functionality in a reliable way.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "First, create a profile and configure your instance:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Configure Open Interpreter\n", + "from interpreter import interpreter\n", + "\n", + "interpreter.llm.model = \"claude-3-5-sonnet-20240620\"\n", + "interpreter.computer.import_computer_api = True\n", + "interpreter.llm.supports_functions = True\n", + "interpreter.llm.supports_vision = True\n", + "interpreter.llm.context_window = 100000\n", + "interpreter.llm.max_tokens = 4096" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Then you will define your custom tool by writing valid Python code within a comment. This example is for searching the AWS documentation using Perplexity:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "custom_tool = \"\"\"\n", + "\n", + "import os\n", + "import requests\n", + "\n", + "def search_aws_docs(query):\n", + "\n", + " url = \"https://api.perplexity.ai/chat/completions\"\n", + "\n", + " payload = {\n", + " \"model\": \"llama-3.1-sonar-small-128k-online\",\n", + " \"messages\": [\n", + " {\n", + " \"role\": \"system\",\n", + " \"content\": \"Be precise and concise.\"\n", + " },\n", + " {\n", + " \"role\": \"user\",\n", + " \"content\": query\n", + " }\n", + " ],\n", + " \"temperature\": 0.2,\n", + " \"top_p\": 0.9,\n", + " \"return_citations\": True,\n", + " \"search_domain_filter\": [\"docs.aws.amazon.com\"],\n", + " \"return_images\": False,\n", + " \"return_related_questions\": False,\n", + " #\"search_recency_filter\": \"month\",\n", + " \"top_k\": 0,\n", + " \"stream\": False,\n", + " \"presence_penalty\": 0,\n", + " \"frequency_penalty\": 1\n", + " }\n", + " headers = {\n", + " \"Authorization\": f\"Bearer {os.environ.get('PPLX_API_KEY')}\",\n", + " \"Content-Type\": \"application/json\"\n", + " }\n", + "\n", + " response = requests.request(\"POST\", url, json=payload, headers=headers)\n", + "\n", + " print(response.text)\n", + "\n", + " return response.text\n", + "\n", + "\"\"\"\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Finally, you add the tool to the OI instance's computer:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "interpreter.computer.run(\"python\", custom_tool)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "> Note: You can define and set multiple tools in a single instance." + ] + } + ], + "metadata": { + "language_info": { + "name": "python" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} From 97d37a5593cf42082f42576d3badf8f3bf76f74d Mon Sep 17 00:00:00 2001 From: Mike Bird <63524998+MikeBirdTech@users.noreply.github.com> Date: Wed, 9 Oct 2024 12:03:28 -0400 Subject: [PATCH 4/4] Added Jupyter notebook example for organizing photos with Open Interpreter using a vision model. --- examples/organize_photos.ipynb | 118 +++++++++++++++++++++++++++++++++ 1 file changed, 118 insertions(+) create mode 100644 examples/organize_photos.ipynb diff --git a/examples/organize_photos.ipynb b/examples/organize_photos.ipynb new file mode 100644 index 0000000000..e2b2cea841 --- /dev/null +++ b/examples/organize_photos.ipynb @@ -0,0 +1,118 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Organize your photos with Open Interpreter" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "You can use Open Interpreter with a vision model to organize photos based on their contents. This is limited by the ability of the LLM as well as the organization of the directories storing photos. " + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "> Note: It is always recommended to back up photos and files on a regular basis. Most models are intelligent enough to know the difference between `move` and `delete` but on rare occasions, files can be deleted during some operations. It is important to test on duplicated photos and to keep an eye on code written by an LLM." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Create a profile. This example uses GPT-4o but you can use any vision model." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "\"\"\"\n", + "This is an Open Interpreter profile to organize a directory of photos. \n", + "\"\"\"\n", + "\n", + "from interpreter import interpreter\n", + "\n", + "\n", + "# LLM settings\n", + "interpreter.llm.model = \"gpt-4o\"\n", + "#interpreter.llm.model = \"ollama/codestral\"\n", + "interpreter.llm.supports_vision = True\n", + "interpreter.llm.execution_instructions = False\n", + "interpreter.llm.max_tokens = 1000\n", + "interpreter.llm.context_window = 7000\n", + "interpreter.llm.load() # Loads Ollama models\n", + "\n", + "# Computer settings\n", + "interpreter.computer.import_computer_api = True\n", + "\n", + "# Misc settings\n", + "interpreter.auto_run = False\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The following custom instruction is intended for a directory containing one sub-directory of unorganized photos and multiple empty sub-directories with names for the intended organization. Please update the custom instructions to match your use-case. This will take some trial and error, depending on the model used." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Custom Instructions\n", + "interpreter.custom_instructions=f\"\"\"\n", + " Recap the plan before answering the user's query!\n", + " Your job is to organize photos. You love organizing photos.\n", + " You will be given a parent directory with sub-directories. \n", + " One sub-directory will be of unorganized photos.\n", + " The other sub-directories will be categories that you move the photos in to.\n", + " Remember the sub-directories's names because they will be the categories for organizing.\n", + " It is extremely important because these are the only options for where you move the photos.\n", + " Loop through every photo in the unorganized photos directory. \n", + " Skip over non-photo files by checking for common photo extensions (.jpg, .jpeg, .png, etc).\n", + " In this loop you will determine the description of each image one at a time. \n", + " Use `computer.vision.query()` to get a description of the image.\n", + " `computer.vision.query()` takes a `path=` argument to know which photo to describe. \n", + " Print out the description so you can get the full context.\n", + " Determine which sub-directory the photo should go in to.\n", + " Every photo needs to go into one of the sub-directories.\n", + " Make sure you actually move the photo. \n", + " Your task is done when every photo in the unorganized photos directory has been moved to another directory. \n", + " **Confirm that the unorganized photos directory has no more photos**.\n", + " \"\"\"\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Save the profile with a descriptive name. Then run interpreter with:\n", + "\n", + "`interpreter --profile `\n", + "\n", + "Then ask it to organize the directory:\n", + "\n", + "`Please organize this directory: /path/to/directory`" + ] + } + ], + "metadata": { + "language_info": { + "name": "python" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +}