OpenAI_Finetuning_Distill_GPT_4_to_GPT_3_5.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q3fIJM8ebasA"
      },
      "source": [
        "# Fine Tuning GPT-3.5-Turbo\n",
        "\n",
        "In this notebook, we walk through an example of fine-tuning gpt-3.5-turbo.\n",
        "\n",
        "Specifically, we attempt to distill GPT-4's knowledge, by generating training data with GPT-4 to then fine-tune GPT-3.5.\n",
        "\n",
        "All training data is generated using two different sections of our index data, creating both a training and evalution set.\n",
        "\n",
        "Evaluation is done using the `ragas` library, which we will detail later on."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1QdMJRtqbasD"
      },
      "outputs": [],
      "source": [
        "!pip install llama-index pypdf sentence-transformers ragas"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "iHpR_evRbasE"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "import openai\n",
        "\n",
        "os.environ[\"OPENAI_API_KEY\"] = \"EMPTY\"\n",
        "openai.api_key = os.environ[\"OPENAI_API_KEY\"]\n",
        "openai.base_path = \"http://localhost:1234/v1\"\n",
        "openai.api_base = \"http://localhost:1234/v1\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rOw2gMDubasE"
      },
      "source": [
        "## Data Setup\n",
        "\n",
        "Here, we first load the text file that we will use to generate training data."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_ZeplTKibasF"
      },
      "outputs": [],
      "source": [
        "file = 'theme-notes.txt'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aSz-0Es5basF"
      },
      "source": [
        "The next step is generating a training and eval dataset.\n",
        "\n",
        "We will generate 40 questions on different sections of the text file we downloaded.\n",
        "\n",
        "We can use GPT-3.5 on the eval questions to get our baseline performance.\n",
        "\n",
        "Then, we will use GPT-4 on the train questions to generate our training data. The training data will be collected with out `OpenAIFineTuningHandler`.\n",
        "\n",
        "This step is entirely optional if you don't want to spend the time/tokens -- the eval and training questions are also provided in this folder, as well as the training data!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "09r5X8TYbasG"
      },
      "source": [
        "### Train Generation"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from llama_index import SimpleDirectoryReader, ServiceContext, Document\n",
        "from llama_index.llms import OpenAI\n",
        "from llama_index.evaluation import DatasetGenerator\n",
        "import random\n",
        "from llama_index.text_splitter import SentenceSplitter\n",
        "import tiktoken\n",
        "from llama_index.utils import globals_helper\n",
        "from langchain.text_splitter import (\n",
        "    NLTKTextSplitter,\n",
        "    SpacyTextSplitter,\n",
        "    RecursiveCharacterTextSplitter,\n",
        ")"
      ],
      "metadata": {
        "id": "3AQqYCpmA0_x"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "We need to split a given text file up into separate documents so that we can create a training and evaluation set."
      ],
      "metadata": {
        "id": "GBcXg0PNJEsb"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# replace \"data\" with the FOLDER containing your data\n",
        "document = SimpleDirectoryReader(\"data\").load_data()[0]\n",
        "text_splitter_default = SentenceSplitter(\n",
        "  separator=\" \",\n",
        "  chunk_size=1024,\n",
        "  chunk_overlap=20,\n",
        "  paragraph_separator=\"\\n\\n\\n\\n\",\n",
        "  secondary_chunking_regex=\"[^,.;。]+[,.;。]?\",\n",
        "  tokenizer=tiktoken.encoding_for_model(\"gpt-3.5-turbo\").encode\n",
        ")\n",
        "text_chunks = text_splitter_default.split_text(document.text)\n",
        "doc_chunks = [Document(text=t) for t in text_chunks]\n",
        "tokenizer = globals_helper.tokenizer\n",
        "with open(\"splitting_1.txt\", \"w\") as f:\n",
        "    for idx, doc in enumerate(doc_chunks):\n",
        "        f.write(\n",
        "            \"\\n-------\\n\\n{}. Size: {} tokens\\n\".format(idx, len(tokenizer(doc.text)))\n",
        "            + doc.text\n",
        "        )"
      ],
      "metadata": {
        "id": "itCh6OOZAjOb"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "for idx, doc in enumerate(doc_chunks):\n",
        "  with open(f'/content/documents/{idx}.txt', 'w') as file:\n",
        "    file.write(doc.text)"
      ],
      "metadata": {
        "id": "SN-7TouXFoVc"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_pKTsTzwbasG"
      },
      "outputs": [],
      "source": [
        "documents = SimpleDirectoryReader(\n",
        "    input_dir= '/content/documents'\n",
        ").load_data()\n",
        "\n",
        "# Shuffle the documents\n",
        "random.seed(42)\n",
        "random.shuffle(documents)\n",
        "\n",
        "gpt_35_context = ServiceContext.from_defaults(\n",
        "    llm=OpenAI(model=\"gpt-3.5-turbo\", temperature=0.3)\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Icni9CGqbasH"
      },
      "outputs": [],
      "source": [
        "question_gen_query = (\n",
        "    \"You are a Teacher/ Professor. Your task is to setup \"\n",
        "    \"a quiz/examination. Using the provided context from a \"\n",
        "    \"series of themes found in biblical passages, formulate \"\n",
        "    \"a single question that captures an important fact from the \"\n",
        "    \"context. Restrict the question to the context information provided.\"\n",
        ")\n",
        "\n",
        "dataset_generator = DatasetGenerator.from_documents(\n",
        "    documents[:50],\n",
        "    question_gen_query=question_gen_query,\n",
        "    service_context=gpt_35_context,\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "l9s_4vqSbasH"
      },
      "outputs": [],
      "source": [
        "# NOTE: this may take some time. Go grab a coffee! (or tea if you have taste buds)\n",
        "questions = dataset_generator.generate_questions_from_nodes(num=40)\n",
        "print(\"Generated \", len(questions), \" questions\")"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "questions[:5]"
      ],
      "metadata": {
        "id": "eeNR2nlDBFOh"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hh_geSX8basH"
      },
      "outputs": [],
      "source": [
        "with open(\"train_questions.txt\", \"w\") as f:\n",
        "    for question in questions:\n",
        "        f.write(question + \"\\n\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "61mF7guLbasH"
      },
      "source": [
        "### Eval Generation\n",
        "\n",
        "Now, lets generate questions on a completely different set of documents, in order to create our eval dataset."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ivceNT09basH"
      },
      "outputs": [],
      "source": [
        "dataset_generator = DatasetGenerator.from_documents(\n",
        "    documents[\n",
        "        50:\n",
        "    ],  # since we generated ~1 question for 40 documents, we can skip the first 40\n",
        "    question_gen_query=question_gen_query,\n",
        "    service_context=gpt_35_context,\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pSAEy5wpbasI"
      },
      "outputs": [],
      "source": [
        "# NOTE: this may take some time. Go grab a coffee! (or tea if you have tastebuds)\n",
        "questions = dataset_generator.generate_questions_from_nodes(num=40)\n",
        "print(\"Generated \", len(questions), \" questions\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "HUGBJyc7basI"
      },
      "outputs": [],
      "source": [
        "with open(\"eval_questions.txt\", \"w\") as f:\n",
        "    for question in questions:\n",
        "        f.write(question + \"\\n\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JDSd7AIybasI"
      },
      "source": [
        "## Initial Eval with GPT-3.5-Turbo Query Engine\n",
        "\n",
        "For this eval, we will be using the [`ragas` evaluation library](https://github.com/explodinggradients/ragas).\n",
        "\n",
        "Ragas has a ton of evaluation metrics for RAG pipelines, and you can read about them [here](https://github.com/explodinggradients/ragas/blob/main/docs/metrics.md).\n",
        "\n",
        "For this notebook, we will be using the following two metrics\n",
        "\n",
        "- `answer_relevancy` - This measures how relevant is the generated answer to the prompt. If the generated answer is incomplete or contains redundant information the score will be low. This is quantified by working out the chance of an LLM generating the given question using the generated answer. Values range (0,1), higher the better.\n",
        "- `faithfulness` - This measures the factual consistency of the generated answer against the given context. This is done using a multi step paradigm that includes creation of statements from the generated answer followed by verifying each of these statements against the context. The answer is scaled to (0,1) range. Higher the better."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Cy-X-JhrbasI"
      },
      "outputs": [],
      "source": [
        "questions = []\n",
        "with open(\"eval_questions.txt\", \"r\") as f:\n",
        "    for line in f:\n",
        "        questions.append(line.strip())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "u8ZFQr_sbasI"
      },
      "outputs": [],
      "source": [
        "from llama_index import VectorStoreIndex\n",
        "\n",
        "# limit the context window to 2048 tokens so that refine is used\n",
        "gpt_35_context = ServiceContext.from_defaults(\n",
        "    llm=OpenAI(model=\"gpt-3.5-turbo\", temperature=0.3), context_window=2048\n",
        ")\n",
        "\n",
        "index = VectorStoreIndex.from_documents(documents, service_context=gpt_35_context)\n",
        "\n",
        "query_engine = index.as_query_engine(similarity_top_k=2)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Z58fuJQ_basI"
      },
      "outputs": [],
      "source": [
        "contexts = []\n",
        "answers = []\n",
        "\n",
        "for question in questions:\n",
        "    response = query_engine.query(question)\n",
        "    contexts.append([x.node.get_content() for x in response.source_nodes])\n",
        "    answers.append(str(response))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Y0TiVi6cbasI"
      },
      "outputs": [],
      "source": [
        "from datasets import Dataset\n",
        "from ragas import evaluate\n",
        "from ragas.metrics import answer_relevancy, faithfulness\n",
        "\n",
        "ds = Dataset.from_dict(\n",
        "    {\n",
        "        \"question\": questions,\n",
        "        \"answer\": answers,\n",
        "        \"contexts\": contexts,\n",
        "    }\n",
        ")\n",
        "\n",
        "result = evaluate(ds, [answer_relevancy, faithfulness])\n",
        "print(result)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q_Xb9m1dbasI"
      },
      "source": [
        "## GPT-4 to Collect Training Data\n",
        "\n",
        "Here, we use GPT-4 and the `OpenAIFineTuningHandler` to collect data that we want to train on."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "W68qggawbasI"
      },
      "outputs": [],
      "source": [
        "from llama_index import ServiceContext\n",
        "from llama_index.llms import OpenAI\n",
        "from llama_index.callbacks import OpenAIFineTuningHandler\n",
        "from llama_index.callbacks import CallbackManager\n",
        "\n",
        "finetuning_handler = OpenAIFineTuningHandler()\n",
        "callback_manager = CallbackManager([finetuning_handler])\n",
        "\n",
        "gpt_4_context = ServiceContext.from_defaults(\n",
        "    llm=OpenAI(model=\"gpt-4\", temperature=0.3),\n",
        "    context_window=2048,  # limit the context window artifically to test refine process\n",
        "    callback_manager=callback_manager,\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "rMsapDGjbasI"
      },
      "outputs": [],
      "source": [
        "questions = []\n",
        "with open(\"train_questions.txt\", \"r\") as f:\n",
        "    for line in f:\n",
        "        questions.append(line.strip())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "y7AWjp5UbasI"
      },
      "outputs": [],
      "source": [
        "from llama_index import VectorStoreIndex\n",
        "\n",
        "index = VectorStoreIndex.from_documents(documents, service_context=gpt_4_context)\n",
        "\n",
        "query_engine = index.as_query_engine(similarity_top_k=2)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pXRazK3obasJ"
      },
      "outputs": [],
      "source": [
        "for question in questions:\n",
        "    response = query_engine.query(question)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PghSrk_rbasJ"
      },
      "source": [
        "## Create Fine-Tuning Data\n",
        "\n",
        "Fine-Tuning data must be written as a list of messages in a `.jsonl` file. Using the finetuning-handler, we can easily write the messages to a `.jsonl` file."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "r6suUlT5basJ"
      },
      "outputs": [],
      "source": [
        "finetuning_handler.save_finetuning_events(\"finetuning_events.jsonl\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C-LUuoCnbasJ"
      },
      "source": [
        "## Launch Fine-Tuning Job"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install \"transformers==4.31.0\" \"datasets==2.13.0\" \"peft==0.4.0\" \"accelerate==0.21.0\" \"bitsandbytes==0.40.2\" \"trl==0.4.7\" \"safetensors>=0.3.1\" --upgrade"
      ],
      "metadata": {
        "id": "LcQeL_nv9ru2"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from datasets import load_dataset\n",
        "from random import randrange\n",
        "\n",
        "dataset = load_dataset(\"json\", data_files=\"/content/finetuning_events.jsonl\",download_mode=\"force_redownload\", split=\"train\")\n",
        "\n",
        "print(f\"dataset size: {len(dataset)}\")\n",
        "print(dataset)"
      ],
      "metadata": {
        "id": "xa87A8E-9ryR"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def format_sample(sample):\n",
        "  return f'''\n",
        "  ## Instruction:\n",
        "  {sample['messages'][0]['content']}\n",
        "\n",
        "  ## Context:\n",
        "  {sample['messages'][1]['content']}\n",
        "\n",
        "  ## Answer:\n",
        "  {sample['messages'][2]['content']}\n",
        "  '''"
      ],
      "metadata": {
        "id": "3ov5cvYl9r2t"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "for prompt in dataset:\n",
        "    print(format_sample(prompt))"
      ],
      "metadata": {
        "id": "g997wT119r6E"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig\n",
        "\n",
        "use_flash_attention = False\n",
        "\n",
        "model_id = \"NousResearch/Llama-2-7b-hf\"\n",
        "\n",
        "bnb_config = BitsAndBytesConfig(\n",
        "    load_in_4bit=True,\n",
        "    bnb_4bit_use_double_quant=True,\n",
        "    bnb_4bit_quant_type=\"nf4\",\n",
        "    bnb_4bit_compute_dtype=torch.bfloat16\n",
        ")\n",
        "\n",
        "model = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=bnb_config, use_cache=False, device_map=\"auto\")\n",
        "model.config.pretraining_tp = 1\n",
        "\n",
        "tokenizer = AutoTokenizer.from_pretrained(model_id)\n",
        "tokenizer.pad_token = tokenizer.eos_token\n",
        "tokenizer.padding_side = \"right\""
      ],
      "metadata": {
        "id": "41x__C-O-mwI"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from peft import LoraConfig, prepare_model_for_kbit_training, get_peft_model\n",
        "\n",
        "# LoRA config based on QLoRA paper\n",
        "peft_config = LoraConfig(\n",
        "        lora_alpha=16,\n",
        "        lora_dropout=0.1,\n",
        "        r=64,\n",
        "        bias=\"none\",\n",
        "        task_type=\"CAUSAL_LM\",\n",
        ")\n",
        "\n",
        "\n",
        "# prepare model for training\n",
        "model = prepare_model_for_kbit_training(model)"
      ],
      "metadata": {
        "id": "kLHKsMXU9r8u"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from transformers import TrainingArguments\n",
        "\n",
        "args = TrainingArguments(\n",
        "    output_dir=\"llama-7-int4-dolly\",\n",
        "    num_train_epochs=3,\n",
        "    per_device_train_batch_size=6 if use_flash_attention else 4,\n",
        "    gradient_accumulation_steps=2,\n",
        "    gradient_checkpointing=True,\n",
        "    optim=\"paged_adamw_32bit\",\n",
        "    logging_steps=10,\n",
        "    save_strategy=\"epoch\",\n",
        "    learning_rate=2e-4,\n",
        "    bf16=True,\n",
        "    fp16=False,\n",
        "    tf32=True,\n",
        "    max_grad_norm=0.3,\n",
        "    warmup_ratio=0.03,\n",
        "    lr_scheduler_type=\"constant\",\n",
        "    disable_tqdm=False,  # disable tqdm since with packing values are in correct\n",
        ")\n",
        "\n",
        "model = get_peft_model(model, peft_config)"
      ],
      "metadata": {
        "id": "iJZqyQwU9sAE"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from trl import SFTTrainer\n",
        "\n",
        "max_seq_length = 2048 # max sequence length for model and packing of the dataset\n",
        "\n",
        "trainer = SFTTrainer(\n",
        "    model=model,\n",
        "    train_dataset=dataset,\n",
        "    peft_config=peft_config,\n",
        "    max_seq_length=max_seq_length,\n",
        "    tokenizer=tokenizer,\n",
        "    packing=True,\n",
        "    formatting_func=format_sample,\n",
        "    args=args,\n",
        ")"
      ],
      "metadata": {
        "id": "H_g5OCRw9sEN"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "trainer.train()\n",
        "trainer.save_model()"
      ],
      "metadata": {
        "id": "oU5SUrGo9sJM"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xUa5lgrHbasJ"
      },
      "source": [
        "## Evaluation\n",
        "\n",
        "After some time, your model will be done training!\n",
        "\n",
        "The next step is running our fine-tuned model on our eval dataset again to measure any performance increase."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7u9ANJ6DbasJ"
      },
      "outputs": [],
      "source": [
        "ft_model_name = \"ft:gpt-3.5-turbo-0613:...\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "WWdNY97cbasJ"
      },
      "outputs": [],
      "source": [
        "from llama_index import ServiceContext\n",
        "from llama_index.llms import OpenAI\n",
        "from llama_index.callbacks import OpenAIFineTuningHandler\n",
        "from llama_index.callbacks import CallbackManager\n",
        "\n",
        "\n",
        "ft_context = ServiceContext.from_defaults(\n",
        "    llm=OpenAI(model=ft_model_name, temperature=0.3),\n",
        "    context_window=2048,  # limit the context window artifically to test refine process\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7JZGB0OGbasJ"
      },
      "outputs": [],
      "source": [
        "questions = []\n",
        "with open(\"eval_questions.txt\", \"r\") as f:\n",
        "    for line in f:\n",
        "        questions.append(line.strip())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hrisqSVObasJ"
      },
      "outputs": [],
      "source": [
        "from llama_index import VectorStoreIndex\n",
        "\n",
        "index = VectorStoreIndex.from_documents(documents, service_context=ft_context)\n",
        "\n",
        "query_engine = index.as_query_engine(similarity_top_k=2)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "brXL6emGbasJ"
      },
      "outputs": [],
      "source": [
        "contexts = []\n",
        "answers = []\n",
        "\n",
        "for question in questions:\n",
        "    response = query_engine.query(question)\n",
        "    contexts.append([x.node.get_content() for x in response.source_nodes])\n",
        "    answers.append(str(response))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "YjN9Ib9zbasJ"
      },
      "outputs": [],
      "source": [
        "from datasets import Dataset\n",
        "from ragas import evaluate\n",
        "from ragas.metrics import answer_relevancy, faithfulness\n",
        "\n",
        "ds = Dataset.from_dict(\n",
        "    {\n",
        "        \"question\": questions,\n",
        "        \"answer\": answers,\n",
        "        \"contexts\": contexts,\n",
        "    }\n",
        ")\n",
        "\n",
        "result = evaluate(ds, [answer_relevancy, faithfulness])\n",
        "print(result)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-1gmOSA6basJ"
      },
      "source": [
        "## Exploring Differences\n",
        "\n",
        "Let's quickly compare the differences in responses, to demonstrate that fine tuning did indeed change something."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "DnQiq3DtbasJ"
      },
      "outputs": [],
      "source": [
        "from llama_index import VectorStoreIndex\n",
        "\n",
        "index = VectorStoreIndex.from_documents(documents)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "fq2yze5NbasJ"
      },
      "outputs": [],
      "source": [
        "questions = []\n",
        "with open(\"eval_questions.txt\", \"r\") as f:\n",
        "    for line in f:\n",
        "        questions.append(line.strip())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2WO6d_bkbasO"
      },
      "outputs": [],
      "source": [
        "print(questions[12])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BHOpP41dbasP"
      },
      "source": [
        "### Original"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "eAJMXbIabasP"
      },
      "outputs": [],
      "source": [
        "from llama_index.response.notebook_utils import display_response\n",
        "from llama_index import ServiceContext\n",
        "from llama_index.llms import OpenAI\n",
        "\n",
        "\n",
        "gpt_35_context = ServiceContext.from_defaults(\n",
        "    llm=OpenAI(model=\"gpt-3.5-turbo\", temperature=0.3),\n",
        "    context_window=2048,  # limit the context window artifically to test refine process\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "UOKiQwoBbasP"
      },
      "outputs": [],
      "source": [
        "query_engine = index.as_query_engine(service_context=gpt_35_context)\n",
        "\n",
        "response = query_engine.query(questions[12])\n",
        "\n",
        "display_response(response)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K5hFKXrUbasP"
      },
      "source": [
        "### Fine-Tuned"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bjpnem67basP"
      },
      "outputs": [],
      "source": [
        "from llama_index import ServiceContext\n",
        "from llama_index.llms import OpenAI\n",
        "\n",
        "\n",
        "ft_context = ServiceContext.from_defaults(\n",
        "    llm=OpenAI(model=ft_model_name, temperature=0.3),\n",
        "    context_window=2048,  # limit the context window artifically to test refine process\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NJFKZ_KQbasP"
      },
      "outputs": [],
      "source": [
        "query_engine = index.as_query_engine(service_context=ft_context)\n",
        "\n",
        "response = query_engine.query(questions[12])\n",
        "\n",
        "display_response(response)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LIjED_8kbasP"
      },
      "source": [
        "As we can see, the fine-tuned model provides a more thorough response! This lines up with the increased faithfullness score from ragas, since the answer is more representative of the retrieved context."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EerKZ5OibasP"
      },
      "source": [
        "## Conclusion\n",
        "\n",
        "So, in conclusion, finetuning with only ~61 questions actually helped improve our eval scores!\n",
        "\n",
        "**answer_relevancy: 0.9778 -> 0.9758**\n",
        "\n",
        "The answer relenvancy appears to be basically unchanged, between models.\n",
        "\n",
        "**faithfulness: 0.7638 -> 0.8088**\n",
        "\n",
        "The faithfulness appears to have been improved! This mains the anwers given better fuffil the original question that was asked."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "gxlG1lTubasP"
      },
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.12"
    },
    "orig_nbformat": 4,
    "colab": {
      "provenance": [],
      "collapsed_sections": [
        "JDSd7AIybasI",
        "Q_Xb9m1dbasI"
      ],
      "machine_shape": "hm",
      "gpuType": "A100"
    },
    "accelerator": "GPU"
  },
  "nbformat": 4,
  "nbformat_minor": 0
}