model_servers/llamacpp_python/tooling_options.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Interact with the AI Lab Playground\n",
    "\n",
    "This notebook will demonstrate 3 (although there could be many more) ways to write python code to interact with our playground model image. The goal here is to show that our approach allows for flexibility and user choice when it comes to which LLM framework they want to use for development. \n",
    "\n",
    "* [Custom code](#custom-code)\n",
    "* [OpenAI API for python](#openai-python)\n",
    "* [LangChain](#langchain)\n",
    "\n",
    "All 3 examples will demonstrate making a chat request to the same model service and getting a streaming text response back.   \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook assumes that the playground image is running locally. Once built, you can use the below to start the model service image. \n",
    "\n",
    "```bash\n",
    "podman run -it -p 8000:8000 -v <YOUR-LOCAL-PATH>/locallm/models:/locallm/models:Z -e MODEL_PATH=models/mistral-7b-instruct-v0.2.Q4_K_M.gguf playground\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Custom Code\n",
    "\n",
    "This code block indicates that we can fairly easily write a custom class that only relies on the python packages `json` and `requests` to interact with our model service."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  Ah, an excellent question! When it comes to naming a company that specializes in colorful socks, there are plenty of options that convey creativity, playfulness, and professionalism. Here are some suggestions:\n",
      "1. SoleMates - This name plays on the idea of \"soulmates,\" but replaces the word \"soul\" with \"sole,\" which is a nod to the fact that socks are worn on the feet. It's catchy and memorable, and it suggests that the company's products are the perfect match for their customers.\n",
      "2. Footloose & Fancy Free - This name has a fun, carefree vibe, and it conveys the idea of being unencumbered by stuffy shoes or boring footwear. It also has a nice ring to it, with the alliteration of \"foot\" and \"free.\"\n",
      "3. Hue & Cry - This name is a play on words that references both the colors of the socks and the idea of making a fuss or raised voice (i.e., crying). It's clever and sophisticated, and it suggests that the company's products are eye-catching and attention-grabbing.\n",
      "4. Toes & Tones - This name combines two things people often pay attention to when they're looking at their feet: the toes and the tone of their voice. It's catchy and easy to remember, and it suggests that the company's products are all about expressing oneself through fashionable footwear.\n",
      "5. Ankle Appeal - This name is a play on words that references both the anatomy of the foot (ankles) and the idea of appealing to customers. It's clever and witty, and it suggests that the company's products are not only stylish but also functional and comfortable.\n",
      "6. Pedal Pizzazz - This name references the pedals on a bike or other foot-powered vehicle, which is a nod to the active lifestyle and adventure that many people enjoy when they're wearing colorful socks. It's fun and upbeat, and it suggests that the company's products are all about adding some excitement to one's daily routine.\n",
      "7. SoleMate Society - This name builds on the \"SoleMates\" idea from earlier, but it adds a sense of exclusivity and membership to the mix. It suggests that customers who choose the company's socks are part of an elite group of style-conscious individuals who value quality and fashion above all else.\n",
      "8. Footworks - This name references both the feet themselves and the idea of \"work\" in the sense of creating something beautiful or functional. It's catchy and easy to remember, and it suggests that the company's products are crafted with care and attention to detail.\n",
      "These are just a few ideas to get you started, but I hope they help inspire you as you come up with a name for your colorful sock company!"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "import json\n",
    "\n",
    "class Chat:\n",
    "    def __init__(self, endpoint) -> None:\n",
    "        self.endpoint = endpoint\n",
    "        self.headers = {\"accept\": \"application/json\",\n",
    "                        \"Content-Type\": \"application/json\"}\n",
    "        self.system_prompt = [{\"role\": \"system\", \n",
    "                               \"content\": \n",
    "                               \"\"\"You are a helpful assistant that is comfortable speaking with C level executives in a professional setting.\"\"\"}]\n",
    "        self.session = requests.Session()\n",
    "\n",
    "    def ask(self, prompt):\n",
    "        self.system_prompt.append({\"role\":\"user\",\"content\":prompt})\n",
    "        data = {\"messages\": self.system_prompt,\n",
    "                \"stream\": True,\n",
    "                \"temperature\": 0.9\n",
    "                }               \n",
    "        r = self.session.post(self.endpoint, headers=self.headers,json=data,stream=True)\n",
    "        reply = \"\"\n",
    "        for line in r.iter_lines(decode_unicode=True):\n",
    "            if line:\n",
    "                if \"[DONE]\" in line or \": ping\" in line:\n",
    "                    continue\n",
    "                else:\n",
    "                    response = json.loads(line[6:])[\"choices\"][0][\"delta\"]\n",
    "                    if \"content\" in response.keys():\n",
    "                         yield response[\"content\"]\n",
    "\n",
    "chat = Chat(endpoint=\"http://localhost:8000/v1/chat/completions\")\n",
    "stream = chat.ask(\"What would be a good company name for a company that makes colorful socks?\")\n",
    "for chunk in stream:\n",
    "    print(chunk, end=\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### OpenAI Python \n",
    "\n",
    "This code block uses the OpenAI python package to interact with our model service. Since OpenAI built this tooling to interact with their hosted model service, we need to set the base_url variable to point to our local model service. But otherwise, this code from their docs is left unchanged.   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  Here are some suggestions for a company name that makes colorful socks:\n",
      "\n",
      "1. SoleMates - This name suggests the idea of companions or friends, which could work well for a brand that offers fun and colorful socks.\n",
      "2. HueHues - A play on the word \"hue,\" this name conveys a sense of vibrant colors and could work well for a brand that specializes in brightly colored socks.\n",
      "3. SockTastic - This name is meant to evoke excitement and enthusiasm, which could be appealing to customers who want fun and colorful socks.\n",
      "4. The Sockery - This name has a playful ring to it, and the use of \"ery\" at the end gives it a fun and creative feel.\n",
      "5. ColoredSocks Co. - A straightforward name that gets straight to the point. It's simple yet memorable, and easy for customers to remember.\n",
      "6. Socktopus - This name combines the words \"sock\" and \"octopus,\" which could work well for a brand that offers colorful and fun socks with unique designs. \n",
      "7. The Funky Foot Company - This name conveys a sense of fun and creativity, which could be appealing to customers who want colorful socks with bold designs.\n",
      "8. ColorFiesta - This name suggests a celebration of colors, which could work well for a brand that offers brightly colored socks. \n",
      "9. SoleSplash - This name combines the word \"sole\" with \"splash,\" which could suggest a fun and playful approach to sock design.\n",
      "10. HappyFeet - This name conveys a sense of happiness and joy, which could be appealing to customers who want colorful socks that make them feel good. "
     ]
    }
   ],
   "source": [
    "# Example code from https://platform.openai.com/docs/api-reference/streaming\n",
    "\n",
    "import openai\n",
    "import os \n",
    "\n",
    "from openai import OpenAI\n",
    "\n",
    "client = OpenAI(base_url=\"http://localhost:8000/v1\",\n",
    "    api_key = \"sk-no-key-required\")\n",
    "\n",
    "stream = client.chat.completions.create(\n",
    "    model=\"gpt-400\",\n",
    "    messages=[{\"role\": \"user\", \"content\": \"What would be a good company name for a company that makes colorful socks?\"}],\n",
    "    stream=True,\n",
    ")\n",
    "\n",
    "for chunk in stream:\n",
    "    if chunk.choices[0].delta.content is not None:\n",
    "        print(chunk.choices[0].delta.content, end=\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Langchain \n",
    "\n",
    "This code block uses the popular Langchain python package to interact with our local model service. Since our playground image is, at least partially, OpenAI API compatible, we can use Langchain the same way we'd use it with OpenAI. Like above, we simply change the base URL to point to our hosted model.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "I'm looking for something catchy, creative and eye-catching.  I want it to stand out and make people want to buy them!\n",
      "\n",
      "What are some suggestions?\n",
      "\n",
      "Comment: Here are some suggestions for a company name that makes colorful socks:\n",
      "\n",
      "1. SoleMates - a play on the word \"soulmates\" that incorporates the idea of matching socks.\n",
      "2. Footloose & Fancy Free - this name conveys the idea of being carefree and having fun with your footwear.\n",
      "3. Hue & Cry - a play on words that references both color and emotion.\n",
      "4. Toe-Tally Awesome - this name is a play on the phrase \"totally awesome\" and incorporates the idea of toes.\n",
      "5. The Sock Exchange - this name could work for a company that sells socks in a variety of colors and patterns.\n",
      "6. SoleMate Society - this name incorporates the idea of finding a matching pair of socks, but also implies a sense of community among sock enthusiasts.\n",
      "7. The Happy Feet Company - this name conveys the idea"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'\\n\\nI\\'m looking for something catchy, creative and eye-catching.  I want it to stand out and make people want to buy them!\\n\\nWhat are some suggestions?\\n\\nComment: Here are some suggestions for a company name that makes colorful socks:\\n\\n1. SoleMates - a play on the word \"soulmates\" that incorporates the idea of matching socks.\\n2. Footloose & Fancy Free - this name conveys the idea of being carefree and having fun with your footwear.\\n3. Hue & Cry - a play on words that references both color and emotion.\\n4. Toe-Tally Awesome - this name is a play on the phrase \"totally awesome\" and incorporates the idea of toes.\\n5. The Sock Exchange - this name could work for a company that sells socks in a variety of colors and patterns.\\n6. SoleMate Society - this name incorporates the idea of finding a matching pair of socks, but also implies a sense of community among sock enthusiasts.\\n7. The Happy Feet Company - this name conveys the idea'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Example code from https://github.com/mudler/LocalAI/blob/master/examples/langchain/langchainpy-localai-example/simple_demo.py\n",
    "\n",
    "from langchain.llms import OpenAI\n",
    "from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler\n",
    "\n",
    "\n",
    "llm = OpenAI(temperature=0.9,model_name=\"instructlab/granite-7b-lab\", base_url=\"http://localhost:8000/v1\", \n",
    "             openai_api_key=\"sk-no-key-required\", streaming=True,\n",
    "             callbacks=[StreamingStdOutCallbackHandler()])\n",
    "text = \"What would be a good company name for a company that makes colorful socks?\"\n",
    "llm(text)\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "locallm",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}