DataEngineering_Final_Project.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyOJPSvifCPz9hJ1mo4BSCkR",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/HestC/DACSS_690a/blob/main/DataEngineering_Final_Project.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Final project for DACSS 690A using the PredictIT API as the data source"
      ],
      "metadata": {
        "id": "bbExlUDmcN1a"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Install necessary libraries"
      ],
      "metadata": {
        "id": "v-a-I66PAF0o"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "\n",
        "# install necessary libraries\n",
        "from google.colab import drive\n",
        "import nbformat\n",
        "\n",
        "\n",
        "# mount Google Drive for saving files\n",
        "#drive.mount('/content/drive')\n",
        "\n",
        "\n",
        "\n",
        "!pip install requests pandas --quiet # quiets the noise\n",
        "\n",
        "!pip install sqlalchemy --quiet\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "import requests      # for making api requests and testing the endpoints\n",
        "import pandas as pd  # handles and process the data into a structured format\n",
        "import json"
      ],
      "metadata": {
        "id": "z7jhju1-ckuJ"
      },
      "execution_count": 32,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "The Predictit data is a platform that provides real-time prediction market data pertaining to current politcal events. Users can place wagers as to the outcome of these events. The data format is JSON which is structured and hierarchal."
      ],
      "metadata": {
        "id": "SnFaROS44oxq"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Extraction Phase**"
      ],
      "metadata": {
        "id": "phttGHsS_4Nn"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Python is used to extract the data and for API responses."
      ],
      "metadata": {
        "id": "jQVd7WVsGghk"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import requests\n",
        "\n",
        "# testing the endpoints for viability\n",
        "\n",
        "api_url = \"https://www.predictit.org/api/marketdata/all/\"\n",
        "response = requests.get(api_url)\n",
        "\n",
        "if response.status_code == 200:\n",
        "    print(\"Endpoint is working!\")\n",
        "else:\n",
        "    print(f\"Failed to connect. Status code: {response.status_code}\")\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0DYpNOuAf6g_",
        "outputId": "d8dff8ee-bd93-412e-fb76-294f1c419e0a"
      },
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Endpoint is working!\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import requests\n",
        "\n",
        "response = requests.get(\"https://www.predictit.org/api/marketdata/all/\")\n",
        "if response.status_code == 200:\n",
        "    print(\"API works!\")\n",
        "    print(response.json())  # display JSON data\n",
        "else:\n",
        "    print(\"Failed to fetch data\")\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "FKZHbFhi5ijK",
        "outputId": "0675c135-0bd8-4f84-b24e-95960ccc8838"
      },
      "execution_count": 34,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "API works!\n",
            "{'markets': [{'id': 7136, 'name': 'Will Joe Biden resign during his first term?', 'shortName': 'Will Biden resign during his first term?', 'image': 'https://az620379.vo.msecnd.net/images/Markets/5f1bcacc-8162-4152-ab4a-475c68e7100c.jpg', 'url': 'https://www.predictit.org/markets/detail/7136/Will-Joe-Biden-resign-during-his-first-term', 'contracts': [{'id': 25232, 'dateEnd': '2025-01-20T11:59:00', 'image': 'https://az620379.vo.msecnd.net/images/Contracts/small_5f1bcacc-8162-4152-ab4a-475c68e7100c.jpg', 'name': 'Will Joe Biden resign during his first term?', 'shortName': 'Will Biden resign during his first term?', 'status': 'Open', 'lastTradePrice': 0.05, 'bestBuyYesCost': 0.06, 'bestBuyNoCost': 0.95, 'bestSellYesCost': 0.05, 'bestSellNoCost': 0.94, 'lastClosePrice': 0.06, 'displayOrder': 0}], 'timeStamp': '2024-11-30T12:48:59.6248751', 'status': 'Open'}, {'id': 7419, 'name': 'Will Kamala Harris be the 47th US president?', 'shortName': 'Harris the 47th US president?', 'image': 'https://az620379.vo.msecnd.net/images/Markets/42e624ca-be5f-4c3e-8939-86bb42b0314d.jpg', 'url': 'https://www.predictit.org/markets/detail/7419/Will-Kamala-Harris-be-the-47th-US-president', 'contracts': [{'id': 27224, 'dateEnd': 'NA', 'image': 'https://az620379.vo.msecnd.net/images/Contracts/small_42e624ca-be5f-4c3e-8939-86bb42b0314d.jpg', 'name': 'Will Kamala Harris be the 47th US president?', 'shortName': 'Harris the 47th US president?', 'status': 'Open', 'lastTradePrice': 0.07, 'bestBuyYesCost': 0.07, 'bestBuyNoCost': 0.94, 'bestSellYesCost': 0.06, 'bestSellNoCost': 0.93, 'lastClosePrice': 0.07, 'displayOrder': 0}], 'timeStamp': '2024-11-30T12:48:59.6248751', 'status': 'Open'}, {'id': 7456, 'name': 'Who will win the 2024 US presidential election?', 'shortName': '2024 presidential election winner?', 'image': 'https://az620379.vo.msecnd.net/images/Markets/5a94f1fa-23ba-4bc4-ba25-a3c016449b75.png', 'url': 'https://www.predictit.org/markets/detail/7456/Who-will-win-the-2024-US-presidential-election', 'contracts': [{'id': 27485, 'dateEnd': 'NA', 'image': 'https://az620379.vo.msecnd.net/images/Contracts/small_3c10062b-0942-4fc3-9ace-bb6abdb790ff.jpg', 'name': 'Donald Trump', 'shortName': 'Trump', 'status': 'Open', 'lastTradePrice': 0.97, 'bestBuyYesCost': 0.99, 'bestBuyNoCost': 0.03, 'bestSellYesCost': 0.97, 'bestSellNoCost': 0.01, 'lastClosePrice': 0.98, 'displayOrder': 0}, {'id': 27487, 'dateEnd': 'NA', 'image': 'https://az620379.vo.msecnd.net/images/Contracts/small_42e624ca-be5f-4c3e-8939-86bb42b0314d.jpg', 'name': 'Kamala Harris', 'shortName': 'Harris', 'status': 'Open', 'lastTradePrice': 0.02, 'bestBuyYesCost': 0.02, 'bestBuyNoCost': 0.99, 'bestSellYesCost': 0.01, 'bestSellNoCost': 0.98, 'lastClosePrice': 0.01, 'displayOrder': 0}, {'id': 27484, 'dateEnd': 'NA', 'image': 'https://az620379.vo.msecnd.net/images/Contracts/small_1d8cb077-b0ce-456e-aadc-57fc4d88dcd7.png', 'name': 'Ron DeSantis', 'shortName': 'DeSantis', 'status': 'Open', 'lastTradePrice': 0.01, 'bestBuyYesCost': 0.01, 'bestBuyNoCost': None, 'bestSellYesCost': None, 'bestSellNoCost': 0.99, 'lastClosePrice': 0.01, 'displayOrder': 0}, {'id': 27486, 'dateEnd': 'NA', 'image': 'https://az620379.vo.msecnd.net/images/Contracts/small_4c64890a-503c-454c-b02e-732319f1459f.jpg', 'name': 'Joe Biden', 'shortName': 'Biden', 'status': 'Open', 'lastTradePrice': 0.01, 'bestBuyYesCost': 0.01, 'bestBuyNoCost': None, 'bestSellYesCost': None, 'bestSellNoCost': 0.99, 'lastClosePrice': 0.01, 'displayOrder': 0}, {'id': 28624, 'dateEnd': 'NA', 'image': 'https://az620379.vo.msecnd.net/images/Contracts/small_87412471-a5b6-4b74-bcc4-685106982248.jpg', 'name': 'Pete Buttigieg', 'shortName': 'Buttigieg', 'status': 'Open', 'lastTradePrice': 0.01, 'bestBuyYesCost': 0.01, 'bestBuyNoCost': None, 'bestSellYesCost': None, 'bestSellNoCost': 0.99, 'lastClosePrice': 0.01, 'displayOrder': 0}, {'id': 31066, 'dateEnd': 'NA', 'image': 'https://az620379.vo.msecnd.net/images/Contracts/small_a898b196-9a0f-454e-99fe-36e21c62f343.jpg', 'name': 'Gavin Newsom', 'shortName': 'Newsom', 'status': 'Open', 'lastTradePrice': 0.01, 'bestBuyYesCost': 0.01, 'bestBuyNoCost': None, 'bestSellYesCost': None, 'bestSellNoCost': 0.99, 'lastClosePrice': 0.01, 'displayOrder': 0}, {'id': 31294, 'dateEnd': 'NA', 'image': 'https://az620379.vo.msecnd.net/images/Contracts/small_88c5c8fe-3175-4907-9375-0f63071d1aab.png', 'name': 'Vivek Ramaswamy', 'shortName': 'Ramaswamy', 'status': 'Open', 'lastTradePrice': 0.01, 'bestBuyYesCost': 0.01, 'bestBuyNoCost': None, 'bestSellYesCost': None, 'bestSellNoCost': 0.99, 'lastClosePrice': 0.01, 'displayOrder': 0}, {'id': 31295, 'dateEnd': 'NA', 'image': 'https://az620379.vo.msecnd.net/images/Contracts/small_958885c8-c04c-4ad3-b64e-fe33e9af83cc.jpg', 'name': 'Tim Scott', 'shortName': 'Scott', 'status': 'Open', 'lastTradePrice': 0.01, 'bestBuyYesCost': 0.01, 'bestBuyNoCost': None, 'bestSellYesCost': None, 'bestSellNoCost': 0.99, 'lastClosePrice': 0.01, 'displayOrder': 0}, {'id': 31296, 'dateEnd': 'NA', 'image': 'https://az620379.vo.msecnd.net/images/Contracts/small_afcbd56b-265a-4ae8-a2b6-693899b708a9.png', 'name': 'Robert Kennedy Jr.', 'shortName': 'RFK Jr.', 'status': 'Open', 'lastTradePrice': 0.01, 'bestBuyYesCost': 0.01, 'bestBuyNoCost': None, 'bestSellYesCost': None, 'bestSellNoCost': 0.99, 'lastClosePrice': 0.01, 'displayOrder': 0}, {'id': 31389, 'dateEnd': 'NA', 'image': 'https://az620379.vo.msecnd.net/images/Contracts/small_20003de6-fa6b-4fb5-82be-aba2bc8c02f9.jpg', 'name': 'Nikki Haley', 'shortName': 'Haley', 'status': 'Open', 'lastTradePrice': 0.01, 'bestBuyYesCost': 0.01, 'bestBuyNoCost': None, 'bestSellYesCost': None, 'bestSellNoCost': 0.99, 'lastClosePrice': 0.01, 'displayOrder': 0}], 'timeStamp': '2024-11-30T12:48:59.6248751', 'status': 'Open'}]}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import requests\n",
        "\n",
        "# Define the endpoint\n",
        "url = \"https://www.predictit.org/api/marketdata/all/\"\n",
        "\n",
        "# fetch data\n",
        "response = requests.get(url)\n",
        "data = response.json()  # parse the JSON response\n",
        "\n",
        "# frint the first market to verify structure\n",
        "print(data['markets'][0])\n",
        "\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "C8uDWa5K6XHc",
        "outputId": "30dd35bf-5702-4bdb-ab1c-586aa45feef6"
      },
      "execution_count": 35,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'id': 7136, 'name': 'Will Joe Biden resign during his first term?', 'shortName': 'Will Biden resign during his first term?', 'image': 'https://az620379.vo.msecnd.net/images/Markets/5f1bcacc-8162-4152-ab4a-475c68e7100c.jpg', 'url': 'https://www.predictit.org/markets/detail/7136/Will-Joe-Biden-resign-during-his-first-term', 'contracts': [{'id': 25232, 'dateEnd': '2025-01-20T11:59:00', 'image': 'https://az620379.vo.msecnd.net/images/Contracts/small_5f1bcacc-8162-4152-ab4a-475c68e7100c.jpg', 'name': 'Will Joe Biden resign during his first term?', 'shortName': 'Will Biden resign during his first term?', 'status': 'Open', 'lastTradePrice': 0.05, 'bestBuyYesCost': 0.06, 'bestBuyNoCost': 0.95, 'bestSellYesCost': 0.05, 'bestSellNoCost': 0.94, 'lastClosePrice': 0.06, 'displayOrder': 0}], 'timeStamp': '2024-11-30T12:48:59.6248751', 'status': 'Open'}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import requests  # import the library for sending HTTP requests\n",
        "\n",
        "# define the API endpoint\n",
        "url = \"https://www.predictit.org/api/marketdata/all/\"\n",
        "\n",
        "# send a GET request to the API\n",
        "response = requests.get(url)\n",
        "\n",
        "# parse the JSON response\n",
        "# the 'indent' parameter is not applicable here; it is used in json.dumps for pretty printing.\n",
        "data = response.json()\n",
        "\n",
        "\n",
        "# print the parsed data\n",
        "import json\n",
        "print(json.dumps(data, indent=4))  # pretty-print the JSON response for readability\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "flPxFD12gkSJ",
        "outputId": "09407500-767e-48a6-9e22-264740ac4f5b"
      },
      "execution_count": 36,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{\n",
            "    \"markets\": [\n",
            "        {\n",
            "            \"id\": 7136,\n",
            "            \"name\": \"Will Joe Biden resign during his first term?\",\n",
            "            \"shortName\": \"Will Biden resign during his first term?\",\n",
            "            \"image\": \"https://az620379.vo.msecnd.net/images/Markets/5f1bcacc-8162-4152-ab4a-475c68e7100c.jpg\",\n",
            "            \"url\": \"https://www.predictit.org/markets/detail/7136/Will-Joe-Biden-resign-during-his-first-term\",\n",
            "            \"contracts\": [\n",
            "                {\n",
            "                    \"id\": 25232,\n",
            "                    \"dateEnd\": \"2025-01-20T11:59:00\",\n",
            "                    \"image\": \"https://az620379.vo.msecnd.net/images/Contracts/small_5f1bcacc-8162-4152-ab4a-475c68e7100c.jpg\",\n",
            "                    \"name\": \"Will Joe Biden resign during his first term?\",\n",
            "                    \"shortName\": \"Will Biden resign during his first term?\",\n",
            "                    \"status\": \"Open\",\n",
            "                    \"lastTradePrice\": 0.05,\n",
            "                    \"bestBuyYesCost\": 0.06,\n",
            "                    \"bestBuyNoCost\": 0.95,\n",
            "                    \"bestSellYesCost\": 0.05,\n",
            "                    \"bestSellNoCost\": 0.94,\n",
            "                    \"lastClosePrice\": 0.06,\n",
            "                    \"displayOrder\": 0\n",
            "                }\n",
            "            ],\n",
            "            \"timeStamp\": \"2024-11-30T12:48:59.6248751\",\n",
            "            \"status\": \"Open\"\n",
            "        },\n",
            "        {\n",
            "            \"id\": 7419,\n",
            "            \"name\": \"Will Kamala Harris be the 47th US president?\",\n",
            "            \"shortName\": \"Harris the 47th US president?\",\n",
            "            \"image\": \"https://az620379.vo.msecnd.net/images/Markets/42e624ca-be5f-4c3e-8939-86bb42b0314d.jpg\",\n",
            "            \"url\": \"https://www.predictit.org/markets/detail/7419/Will-Kamala-Harris-be-the-47th-US-president\",\n",
            "            \"contracts\": [\n",
            "                {\n",
            "                    \"id\": 27224,\n",
            "                    \"dateEnd\": \"NA\",\n",
            "                    \"image\": \"https://az620379.vo.msecnd.net/images/Contracts/small_42e624ca-be5f-4c3e-8939-86bb42b0314d.jpg\",\n",
            "                    \"name\": \"Will Kamala Harris be the 47th US president?\",\n",
            "                    \"shortName\": \"Harris the 47th US president?\",\n",
            "                    \"status\": \"Open\",\n",
            "                    \"lastTradePrice\": 0.07,\n",
            "                    \"bestBuyYesCost\": 0.07,\n",
            "                    \"bestBuyNoCost\": 0.94,\n",
            "                    \"bestSellYesCost\": 0.06,\n",
            "                    \"bestSellNoCost\": 0.93,\n",
            "                    \"lastClosePrice\": 0.07,\n",
            "                    \"displayOrder\": 0\n",
            "                }\n",
            "            ],\n",
            "            \"timeStamp\": \"2024-11-30T12:48:59.6248751\",\n",
            "            \"status\": \"Open\"\n",
            "        },\n",
            "        {\n",
            "            \"id\": 7456,\n",
            "            \"name\": \"Who will win the 2024 US presidential election?\",\n",
            "            \"shortName\": \"2024 presidential election winner?\",\n",
            "            \"image\": \"https://az620379.vo.msecnd.net/images/Markets/5a94f1fa-23ba-4bc4-ba25-a3c016449b75.png\",\n",
            "            \"url\": \"https://www.predictit.org/markets/detail/7456/Who-will-win-the-2024-US-presidential-election\",\n",
            "            \"contracts\": [\n",
            "                {\n",
            "                    \"id\": 27485,\n",
            "                    \"dateEnd\": \"NA\",\n",
            "                    \"image\": \"https://az620379.vo.msecnd.net/images/Contracts/small_3c10062b-0942-4fc3-9ace-bb6abdb790ff.jpg\",\n",
            "                    \"name\": \"Donald Trump\",\n",
            "                    \"shortName\": \"Trump\",\n",
            "                    \"status\": \"Open\",\n",
            "                    \"lastTradePrice\": 0.97,\n",
            "                    \"bestBuyYesCost\": 0.99,\n",
            "                    \"bestBuyNoCost\": 0.03,\n",
            "                    \"bestSellYesCost\": 0.97,\n",
            "                    \"bestSellNoCost\": 0.01,\n",
            "                    \"lastClosePrice\": 0.98,\n",
            "                    \"displayOrder\": 0\n",
            "                },\n",
            "                {\n",
            "                    \"id\": 27487,\n",
            "                    \"dateEnd\": \"NA\",\n",
            "                    \"image\": \"https://az620379.vo.msecnd.net/images/Contracts/small_42e624ca-be5f-4c3e-8939-86bb42b0314d.jpg\",\n",
            "                    \"name\": \"Kamala Harris\",\n",
            "                    \"shortName\": \"Harris\",\n",
            "                    \"status\": \"Open\",\n",
            "                    \"lastTradePrice\": 0.02,\n",
            "                    \"bestBuyYesCost\": 0.02,\n",
            "                    \"bestBuyNoCost\": 0.99,\n",
            "                    \"bestSellYesCost\": 0.01,\n",
            "                    \"bestSellNoCost\": 0.98,\n",
            "                    \"lastClosePrice\": 0.01,\n",
            "                    \"displayOrder\": 0\n",
            "                },\n",
            "                {\n",
            "                    \"id\": 27484,\n",
            "                    \"dateEnd\": \"NA\",\n",
            "                    \"image\": \"https://az620379.vo.msecnd.net/images/Contracts/small_1d8cb077-b0ce-456e-aadc-57fc4d88dcd7.png\",\n",
            "                    \"name\": \"Ron DeSantis\",\n",
            "                    \"shortName\": \"DeSantis\",\n",
            "                    \"status\": \"Open\",\n",
            "                    \"lastTradePrice\": 0.01,\n",
            "                    \"bestBuyYesCost\": 0.01,\n",
            "                    \"bestBuyNoCost\": null,\n",
            "                    \"bestSellYesCost\": null,\n",
            "                    \"bestSellNoCost\": 0.99,\n",
            "                    \"lastClosePrice\": 0.01,\n",
            "                    \"displayOrder\": 0\n",
            "                },\n",
            "                {\n",
            "                    \"id\": 27486,\n",
            "                    \"dateEnd\": \"NA\",\n",
            "                    \"image\": \"https://az620379.vo.msecnd.net/images/Contracts/small_4c64890a-503c-454c-b02e-732319f1459f.jpg\",\n",
            "                    \"name\": \"Joe Biden\",\n",
            "                    \"shortName\": \"Biden\",\n",
            "                    \"status\": \"Open\",\n",
            "                    \"lastTradePrice\": 0.01,\n",
            "                    \"bestBuyYesCost\": 0.01,\n",
            "                    \"bestBuyNoCost\": null,\n",
            "                    \"bestSellYesCost\": null,\n",
            "                    \"bestSellNoCost\": 0.99,\n",
            "                    \"lastClosePrice\": 0.01,\n",
            "                    \"displayOrder\": 0\n",
            "                },\n",
            "                {\n",
            "                    \"id\": 28624,\n",
            "                    \"dateEnd\": \"NA\",\n",
            "                    \"image\": \"https://az620379.vo.msecnd.net/images/Contracts/small_87412471-a5b6-4b74-bcc4-685106982248.jpg\",\n",
            "                    \"name\": \"Pete Buttigieg\",\n",
            "                    \"shortName\": \"Buttigieg\",\n",
            "                    \"status\": \"Open\",\n",
            "                    \"lastTradePrice\": 0.01,\n",
            "                    \"bestBuyYesCost\": 0.01,\n",
            "                    \"bestBuyNoCost\": null,\n",
            "                    \"bestSellYesCost\": null,\n",
            "                    \"bestSellNoCost\": 0.99,\n",
            "                    \"lastClosePrice\": 0.01,\n",
            "                    \"displayOrder\": 0\n",
            "                },\n",
            "                {\n",
            "                    \"id\": 31066,\n",
            "                    \"dateEnd\": \"NA\",\n",
            "                    \"image\": \"https://az620379.vo.msecnd.net/images/Contracts/small_a898b196-9a0f-454e-99fe-36e21c62f343.jpg\",\n",
            "                    \"name\": \"Gavin Newsom\",\n",
            "                    \"shortName\": \"Newsom\",\n",
            "                    \"status\": \"Open\",\n",
            "                    \"lastTradePrice\": 0.01,\n",
            "                    \"bestBuyYesCost\": 0.01,\n",
            "                    \"bestBuyNoCost\": null,\n",
            "                    \"bestSellYesCost\": null,\n",
            "                    \"bestSellNoCost\": 0.99,\n",
            "                    \"lastClosePrice\": 0.01,\n",
            "                    \"displayOrder\": 0\n",
            "                },\n",
            "                {\n",
            "                    \"id\": 31294,\n",
            "                    \"dateEnd\": \"NA\",\n",
            "                    \"image\": \"https://az620379.vo.msecnd.net/images/Contracts/small_88c5c8fe-3175-4907-9375-0f63071d1aab.png\",\n",
            "                    \"name\": \"Vivek Ramaswamy\",\n",
            "                    \"shortName\": \"Ramaswamy\",\n",
            "                    \"status\": \"Open\",\n",
            "                    \"lastTradePrice\": 0.01,\n",
            "                    \"bestBuyYesCost\": 0.01,\n",
            "                    \"bestBuyNoCost\": null,\n",
            "                    \"bestSellYesCost\": null,\n",
            "                    \"bestSellNoCost\": 0.99,\n",
            "                    \"lastClosePrice\": 0.01,\n",
            "                    \"displayOrder\": 0\n",
            "                },\n",
            "                {\n",
            "                    \"id\": 31295,\n",
            "                    \"dateEnd\": \"NA\",\n",
            "                    \"image\": \"https://az620379.vo.msecnd.net/images/Contracts/small_958885c8-c04c-4ad3-b64e-fe33e9af83cc.jpg\",\n",
            "                    \"name\": \"Tim Scott\",\n",
            "                    \"shortName\": \"Scott\",\n",
            "                    \"status\": \"Open\",\n",
            "                    \"lastTradePrice\": 0.01,\n",
            "                    \"bestBuyYesCost\": 0.01,\n",
            "                    \"bestBuyNoCost\": null,\n",
            "                    \"bestSellYesCost\": null,\n",
            "                    \"bestSellNoCost\": 0.99,\n",
            "                    \"lastClosePrice\": 0.01,\n",
            "                    \"displayOrder\": 0\n",
            "                },\n",
            "                {\n",
            "                    \"id\": 31296,\n",
            "                    \"dateEnd\": \"NA\",\n",
            "                    \"image\": \"https://az620379.vo.msecnd.net/images/Contracts/small_afcbd56b-265a-4ae8-a2b6-693899b708a9.png\",\n",
            "                    \"name\": \"Robert Kennedy Jr.\",\n",
            "                    \"shortName\": \"RFK Jr.\",\n",
            "                    \"status\": \"Open\",\n",
            "                    \"lastTradePrice\": 0.01,\n",
            "                    \"bestBuyYesCost\": 0.01,\n",
            "                    \"bestBuyNoCost\": null,\n",
            "                    \"bestSellYesCost\": null,\n",
            "                    \"bestSellNoCost\": 0.99,\n",
            "                    \"lastClosePrice\": 0.01,\n",
            "                    \"displayOrder\": 0\n",
            "                },\n",
            "                {\n",
            "                    \"id\": 31389,\n",
            "                    \"dateEnd\": \"NA\",\n",
            "                    \"image\": \"https://az620379.vo.msecnd.net/images/Contracts/small_20003de6-fa6b-4fb5-82be-aba2bc8c02f9.jpg\",\n",
            "                    \"name\": \"Nikki Haley\",\n",
            "                    \"shortName\": \"Haley\",\n",
            "                    \"status\": \"Open\",\n",
            "                    \"lastTradePrice\": 0.01,\n",
            "                    \"bestBuyYesCost\": 0.01,\n",
            "                    \"bestBuyNoCost\": null,\n",
            "                    \"bestSellYesCost\": null,\n",
            "                    \"bestSellNoCost\": 0.99,\n",
            "                    \"lastClosePrice\": 0.01,\n",
            "                    \"displayOrder\": 0\n",
            "                }\n",
            "            ],\n",
            "            \"timeStamp\": \"2024-11-30T12:48:59.6248751\",\n",
            "            \"status\": \"Open\"\n",
            "        }\n",
            "    ]\n",
            "}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#meta data\n",
        "\n",
        "\n",
        "source_metadata = {\n",
        "    'source_url': url,\n",
        "    'extraction_time': data['markets'][0].get('timestamp', None) if 'markets' in data and data['markets'] else None,\n",
        "    'total_markets': len(data['markets']) if 'markets' in data else 0\n",
        "}\n",
        "print(\"Source Metadata:\", source_metadata)\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9TIxlAN5yGFu",
        "outputId": "ff0b6a39-0eb3-4e04-c127-cd51b1df9131"
      },
      "execution_count": 37,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Source Metadata: {'source_url': 'https://www.predictit.org/api/marketdata/all/', 'extraction_time': None, 'total_markets': 3}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import json\n",
        "print(json.dumps(data, indent=4))\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_FaKTeLbyrDE",
        "outputId": "6d92da40-9e97-439a-a954-c004d37c6d7f"
      },
      "execution_count": 38,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{\n",
            "    \"markets\": [\n",
            "        {\n",
            "            \"id\": 7136,\n",
            "            \"name\": \"Will Joe Biden resign during his first term?\",\n",
            "            \"shortName\": \"Will Biden resign during his first term?\",\n",
            "            \"image\": \"https://az620379.vo.msecnd.net/images/Markets/5f1bcacc-8162-4152-ab4a-475c68e7100c.jpg\",\n",
            "            \"url\": \"https://www.predictit.org/markets/detail/7136/Will-Joe-Biden-resign-during-his-first-term\",\n",
            "            \"contracts\": [\n",
            "                {\n",
            "                    \"id\": 25232,\n",
            "                    \"dateEnd\": \"2025-01-20T11:59:00\",\n",
            "                    \"image\": \"https://az620379.vo.msecnd.net/images/Contracts/small_5f1bcacc-8162-4152-ab4a-475c68e7100c.jpg\",\n",
            "                    \"name\": \"Will Joe Biden resign during his first term?\",\n",
            "                    \"shortName\": \"Will Biden resign during his first term?\",\n",
            "                    \"status\": \"Open\",\n",
            "                    \"lastTradePrice\": 0.05,\n",
            "                    \"bestBuyYesCost\": 0.06,\n",
            "                    \"bestBuyNoCost\": 0.95,\n",
            "                    \"bestSellYesCost\": 0.05,\n",
            "                    \"bestSellNoCost\": 0.94,\n",
            "                    \"lastClosePrice\": 0.06,\n",
            "                    \"displayOrder\": 0\n",
            "                }\n",
            "            ],\n",
            "            \"timeStamp\": \"2024-11-30T12:48:59.6248751\",\n",
            "            \"status\": \"Open\"\n",
            "        },\n",
            "        {\n",
            "            \"id\": 7419,\n",
            "            \"name\": \"Will Kamala Harris be the 47th US president?\",\n",
            "            \"shortName\": \"Harris the 47th US president?\",\n",
            "            \"image\": \"https://az620379.vo.msecnd.net/images/Markets/42e624ca-be5f-4c3e-8939-86bb42b0314d.jpg\",\n",
            "            \"url\": \"https://www.predictit.org/markets/detail/7419/Will-Kamala-Harris-be-the-47th-US-president\",\n",
            "            \"contracts\": [\n",
            "                {\n",
            "                    \"id\": 27224,\n",
            "                    \"dateEnd\": \"NA\",\n",
            "                    \"image\": \"https://az620379.vo.msecnd.net/images/Contracts/small_42e624ca-be5f-4c3e-8939-86bb42b0314d.jpg\",\n",
            "                    \"name\": \"Will Kamala Harris be the 47th US president?\",\n",
            "                    \"shortName\": \"Harris the 47th US president?\",\n",
            "                    \"status\": \"Open\",\n",
            "                    \"lastTradePrice\": 0.07,\n",
            "                    \"bestBuyYesCost\": 0.07,\n",
            "                    \"bestBuyNoCost\": 0.94,\n",
            "                    \"bestSellYesCost\": 0.06,\n",
            "                    \"bestSellNoCost\": 0.93,\n",
            "                    \"lastClosePrice\": 0.07,\n",
            "                    \"displayOrder\": 0\n",
            "                }\n",
            "            ],\n",
            "            \"timeStamp\": \"2024-11-30T12:48:59.6248751\",\n",
            "            \"status\": \"Open\"\n",
            "        },\n",
            "        {\n",
            "            \"id\": 7456,\n",
            "            \"name\": \"Who will win the 2024 US presidential election?\",\n",
            "            \"shortName\": \"2024 presidential election winner?\",\n",
            "            \"image\": \"https://az620379.vo.msecnd.net/images/Markets/5a94f1fa-23ba-4bc4-ba25-a3c016449b75.png\",\n",
            "            \"url\": \"https://www.predictit.org/markets/detail/7456/Who-will-win-the-2024-US-presidential-election\",\n",
            "            \"contracts\": [\n",
            "                {\n",
            "                    \"id\": 27485,\n",
            "                    \"dateEnd\": \"NA\",\n",
            "                    \"image\": \"https://az620379.vo.msecnd.net/images/Contracts/small_3c10062b-0942-4fc3-9ace-bb6abdb790ff.jpg\",\n",
            "                    \"name\": \"Donald Trump\",\n",
            "                    \"shortName\": \"Trump\",\n",
            "                    \"status\": \"Open\",\n",
            "                    \"lastTradePrice\": 0.97,\n",
            "                    \"bestBuyYesCost\": 0.99,\n",
            "                    \"bestBuyNoCost\": 0.03,\n",
            "                    \"bestSellYesCost\": 0.97,\n",
            "                    \"bestSellNoCost\": 0.01,\n",
            "                    \"lastClosePrice\": 0.98,\n",
            "                    \"displayOrder\": 0\n",
            "                },\n",
            "                {\n",
            "                    \"id\": 27487,\n",
            "                    \"dateEnd\": \"NA\",\n",
            "                    \"image\": \"https://az620379.vo.msecnd.net/images/Contracts/small_42e624ca-be5f-4c3e-8939-86bb42b0314d.jpg\",\n",
            "                    \"name\": \"Kamala Harris\",\n",
            "                    \"shortName\": \"Harris\",\n",
            "                    \"status\": \"Open\",\n",
            "                    \"lastTradePrice\": 0.02,\n",
            "                    \"bestBuyYesCost\": 0.02,\n",
            "                    \"bestBuyNoCost\": 0.99,\n",
            "                    \"bestSellYesCost\": 0.01,\n",
            "                    \"bestSellNoCost\": 0.98,\n",
            "                    \"lastClosePrice\": 0.01,\n",
            "                    \"displayOrder\": 0\n",
            "                },\n",
            "                {\n",
            "                    \"id\": 27484,\n",
            "                    \"dateEnd\": \"NA\",\n",
            "                    \"image\": \"https://az620379.vo.msecnd.net/images/Contracts/small_1d8cb077-b0ce-456e-aadc-57fc4d88dcd7.png\",\n",
            "                    \"name\": \"Ron DeSantis\",\n",
            "                    \"shortName\": \"DeSantis\",\n",
            "                    \"status\": \"Open\",\n",
            "                    \"lastTradePrice\": 0.01,\n",
            "                    \"bestBuyYesCost\": 0.01,\n",
            "                    \"bestBuyNoCost\": null,\n",
            "                    \"bestSellYesCost\": null,\n",
            "                    \"bestSellNoCost\": 0.99,\n",
            "                    \"lastClosePrice\": 0.01,\n",
            "                    \"displayOrder\": 0\n",
            "                },\n",
            "                {\n",
            "                    \"id\": 27486,\n",
            "                    \"dateEnd\": \"NA\",\n",
            "                    \"image\": \"https://az620379.vo.msecnd.net/images/Contracts/small_4c64890a-503c-454c-b02e-732319f1459f.jpg\",\n",
            "                    \"name\": \"Joe Biden\",\n",
            "                    \"shortName\": \"Biden\",\n",
            "                    \"status\": \"Open\",\n",
            "                    \"lastTradePrice\": 0.01,\n",
            "                    \"bestBuyYesCost\": 0.01,\n",
            "                    \"bestBuyNoCost\": null,\n",
            "                    \"bestSellYesCost\": null,\n",
            "                    \"bestSellNoCost\": 0.99,\n",
            "                    \"lastClosePrice\": 0.01,\n",
            "                    \"displayOrder\": 0\n",
            "                },\n",
            "                {\n",
            "                    \"id\": 28624,\n",
            "                    \"dateEnd\": \"NA\",\n",
            "                    \"image\": \"https://az620379.vo.msecnd.net/images/Contracts/small_87412471-a5b6-4b74-bcc4-685106982248.jpg\",\n",
            "                    \"name\": \"Pete Buttigieg\",\n",
            "                    \"shortName\": \"Buttigieg\",\n",
            "                    \"status\": \"Open\",\n",
            "                    \"lastTradePrice\": 0.01,\n",
            "                    \"bestBuyYesCost\": 0.01,\n",
            "                    \"bestBuyNoCost\": null,\n",
            "                    \"bestSellYesCost\": null,\n",
            "                    \"bestSellNoCost\": 0.99,\n",
            "                    \"lastClosePrice\": 0.01,\n",
            "                    \"displayOrder\": 0\n",
            "                },\n",
            "                {\n",
            "                    \"id\": 31066,\n",
            "                    \"dateEnd\": \"NA\",\n",
            "                    \"image\": \"https://az620379.vo.msecnd.net/images/Contracts/small_a898b196-9a0f-454e-99fe-36e21c62f343.jpg\",\n",
            "                    \"name\": \"Gavin Newsom\",\n",
            "                    \"shortName\": \"Newsom\",\n",
            "                    \"status\": \"Open\",\n",
            "                    \"lastTradePrice\": 0.01,\n",
            "                    \"bestBuyYesCost\": 0.01,\n",
            "                    \"bestBuyNoCost\": null,\n",
            "                    \"bestSellYesCost\": null,\n",
            "                    \"bestSellNoCost\": 0.99,\n",
            "                    \"lastClosePrice\": 0.01,\n",
            "                    \"displayOrder\": 0\n",
            "                },\n",
            "                {\n",
            "                    \"id\": 31294,\n",
            "                    \"dateEnd\": \"NA\",\n",
            "                    \"image\": \"https://az620379.vo.msecnd.net/images/Contracts/small_88c5c8fe-3175-4907-9375-0f63071d1aab.png\",\n",
            "                    \"name\": \"Vivek Ramaswamy\",\n",
            "                    \"shortName\": \"Ramaswamy\",\n",
            "                    \"status\": \"Open\",\n",
            "                    \"lastTradePrice\": 0.01,\n",
            "                    \"bestBuyYesCost\": 0.01,\n",
            "                    \"bestBuyNoCost\": null,\n",
            "                    \"bestSellYesCost\": null,\n",
            "                    \"bestSellNoCost\": 0.99,\n",
            "                    \"lastClosePrice\": 0.01,\n",
            "                    \"displayOrder\": 0\n",
            "                },\n",
            "                {\n",
            "                    \"id\": 31295,\n",
            "                    \"dateEnd\": \"NA\",\n",
            "                    \"image\": \"https://az620379.vo.msecnd.net/images/Contracts/small_958885c8-c04c-4ad3-b64e-fe33e9af83cc.jpg\",\n",
            "                    \"name\": \"Tim Scott\",\n",
            "                    \"shortName\": \"Scott\",\n",
            "                    \"status\": \"Open\",\n",
            "                    \"lastTradePrice\": 0.01,\n",
            "                    \"bestBuyYesCost\": 0.01,\n",
            "                    \"bestBuyNoCost\": null,\n",
            "                    \"bestSellYesCost\": null,\n",
            "                    \"bestSellNoCost\": 0.99,\n",
            "                    \"lastClosePrice\": 0.01,\n",
            "                    \"displayOrder\": 0\n",
            "                },\n",
            "                {\n",
            "                    \"id\": 31296,\n",
            "                    \"dateEnd\": \"NA\",\n",
            "                    \"image\": \"https://az620379.vo.msecnd.net/images/Contracts/small_afcbd56b-265a-4ae8-a2b6-693899b708a9.png\",\n",
            "                    \"name\": \"Robert Kennedy Jr.\",\n",
            "                    \"shortName\": \"RFK Jr.\",\n",
            "                    \"status\": \"Open\",\n",
            "                    \"lastTradePrice\": 0.01,\n",
            "                    \"bestBuyYesCost\": 0.01,\n",
            "                    \"bestBuyNoCost\": null,\n",
            "                    \"bestSellYesCost\": null,\n",
            "                    \"bestSellNoCost\": 0.99,\n",
            "                    \"lastClosePrice\": 0.01,\n",
            "                    \"displayOrder\": 0\n",
            "                },\n",
            "                {\n",
            "                    \"id\": 31389,\n",
            "                    \"dateEnd\": \"NA\",\n",
            "                    \"image\": \"https://az620379.vo.msecnd.net/images/Contracts/small_20003de6-fa6b-4fb5-82be-aba2bc8c02f9.jpg\",\n",
            "                    \"name\": \"Nikki Haley\",\n",
            "                    \"shortName\": \"Haley\",\n",
            "                    \"status\": \"Open\",\n",
            "                    \"lastTradePrice\": 0.01,\n",
            "                    \"bestBuyYesCost\": 0.01,\n",
            "                    \"bestBuyNoCost\": null,\n",
            "                    \"bestSellYesCost\": null,\n",
            "                    \"bestSellNoCost\": 0.99,\n",
            "                    \"lastClosePrice\": 0.01,\n",
            "                    \"displayOrder\": 0\n",
            "                }\n",
            "            ],\n",
            "            \"timeStamp\": \"2024-11-30T12:48:59.6248751\",\n",
            "            \"status\": \"Open\"\n",
            "        }\n",
            "    ]\n",
            "}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import requests  # for sending HTTP requests\n",
        "import json  # for pretty-printing JSON\n",
        "\n",
        "# Define the API endpoint\n",
        "url = \"https://www.predictit.org/api/marketdata/all/\"\n",
        "\n",
        "# Send a GET request to the API\n",
        "response = requests.get(url)\n",
        "\n",
        "# Parse the JSON response\n",
        "data = response.json()\n",
        "\n",
        "# Extract metadata\n",
        "source_metadata = {\n",
        "    'source_url': url,\n",
        "    'extraction_time': data['markets'][0].get('timestamp', None) if 'markets' in data and data['markets'] else None,\n",
        "    'total_markets': len(data['markets']) if 'markets' in data else 0,\n",
        "    'first_market_name': data['markets'][0].get('name', 'Unknown') if 'markets' in data and data['markets'] else None,\n",
        "    'first_market_id': data['markets'][0].get('id', 'Unknown') if 'markets' in data and data['markets'] else None,\n",
        "}\n",
        "\n",
        "# pretty-print the metadata\n",
        "print(\"Source Metadata:\")\n",
        "print(json.dumps(source_metadata, indent=4))\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZHxbKbqIzYk6",
        "outputId": "ca68d87d-9716-4128-8897-e0d476316243"
      },
      "execution_count": 40,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Source Metadata:\n",
            "{\n",
            "    \"source_url\": \"https://www.predictit.org/api/marketdata/all/\",\n",
            "    \"extraction_time\": null,\n",
            "    \"total_markets\": 3,\n",
            "    \"first_market_name\": \"Will Joe Biden resign during his first term?\",\n",
            "    \"first_market_id\": 7136\n",
            "}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Normalize JSON into a data frame for future processing\n",
        "\n",
        "\n",
        "# Here I use Pandas\n",
        "\n",
        "\n",
        "import pandas as pd\n",
        "\n",
        "import requests\n",
        "import pandas as pd\n",
        "from sqlalchemy import create_engine\n",
        "\n",
        "# flatten markets data\n",
        "markets = data['markets']\n",
        "markets_df = pd.json_normalize(markets)\n",
        "\n",
        "# flatten contracts within each market\n",
        "contracts = [contract for market in markets for contract in market['contracts']]\n",
        "contracts_df = pd.json_normalize(contracts)\n",
        "\n",
        "print(markets_df.head())  # verify structure\n",
        "print(contracts_df.head())  # verify contracts data\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6nXHcypo6oaM",
        "outputId": "7e96dd5e-8876-434c-d525-a0d06c1f68b8"
      },
      "execution_count": 41,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "     id                                             name  \\\n",
            "0  7136     Will Joe Biden resign during his first term?   \n",
            "1  7419     Will Kamala Harris be the 47th US president?   \n",
            "2  7456  Who will win the 2024 US presidential election?   \n",
            "\n",
            "                                  shortName  \\\n",
            "0  Will Biden resign during his first term?   \n",
            "1             Harris the 47th US president?   \n",
            "2        2024 presidential election winner?   \n",
            "\n",
            "                                               image  \\\n",
            "0  https://az620379.vo.msecnd.net/images/Markets/...   \n",
            "1  https://az620379.vo.msecnd.net/images/Markets/...   \n",
            "2  https://az620379.vo.msecnd.net/images/Markets/...   \n",
            "\n",
            "                                                 url  \\\n",
            "0  https://www.predictit.org/markets/detail/7136/...   \n",
            "1  https://www.predictit.org/markets/detail/7419/...   \n",
            "2  https://www.predictit.org/markets/detail/7456/...   \n",
            "\n",
            "                                           contracts  \\\n",
            "0  [{'id': 25232, 'dateEnd': '2025-01-20T11:59:00...   \n",
            "1  [{'id': 27224, 'dateEnd': 'NA', 'image': 'http...   \n",
            "2  [{'id': 27485, 'dateEnd': 'NA', 'image': 'http...   \n",
            "\n",
            "                     timeStamp status  \n",
            "0  2024-11-30T12:50:00.0625191   Open  \n",
            "1  2024-11-30T12:50:00.0625191   Open  \n",
            "2  2024-11-30T12:50:00.0625191   Open  \n",
            "      id              dateEnd  \\\n",
            "0  25232  2025-01-20T11:59:00   \n",
            "1  27224                   NA   \n",
            "2  27485                   NA   \n",
            "3  27487                   NA   \n",
            "4  27484                   NA   \n",
            "\n",
            "                                               image  \\\n",
            "0  https://az620379.vo.msecnd.net/images/Contract...   \n",
            "1  https://az620379.vo.msecnd.net/images/Contract...   \n",
            "2  https://az620379.vo.msecnd.net/images/Contract...   \n",
            "3  https://az620379.vo.msecnd.net/images/Contract...   \n",
            "4  https://az620379.vo.msecnd.net/images/Contract...   \n",
            "\n",
            "                                           name  \\\n",
            "0  Will Joe Biden resign during his first term?   \n",
            "1  Will Kamala Harris be the 47th US president?   \n",
            "2                                  Donald Trump   \n",
            "3                                 Kamala Harris   \n",
            "4                                  Ron DeSantis   \n",
            "\n",
            "                                  shortName status  lastTradePrice  \\\n",
            "0  Will Biden resign during his first term?   Open            0.05   \n",
            "1             Harris the 47th US president?   Open            0.07   \n",
            "2                                     Trump   Open            0.97   \n",
            "3                                    Harris   Open            0.02   \n",
            "4                                  DeSantis   Open            0.01   \n",
            "\n",
            "   bestBuyYesCost  bestBuyNoCost  bestSellYesCost  bestSellNoCost  \\\n",
            "0            0.06           0.95             0.05            0.94   \n",
            "1            0.07           0.94             0.06            0.93   \n",
            "2            0.99           0.03             0.97            0.01   \n",
            "3            0.02           0.99             0.01            0.98   \n",
            "4            0.01            NaN              NaN            0.99   \n",
            "\n",
            "   lastClosePrice  displayOrder  \n",
            "0            0.06             0  \n",
            "1            0.07             0  \n",
            "2            0.98             0  \n",
            "3            0.01             0  \n",
            "4            0.01             0  \n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Here I encapsulate the API call in a function for reusability and better organization. Error handling ensures that issues like network failures or API downtime are reported clearly and easily, for ease of trouble shooting.\n",
        "    "
      ],
      "metadata": {
        "id": "qPuDnhf_5l8V"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import requests\n",
        "\n",
        "\n",
        "\n",
        "#function that fetches predictit data\n",
        "\n",
        "def fetch_predictit_data():\n",
        "    \"\"\"\n",
        "    Fetch data from the PredictIt API and return it as JSON.\n",
        "\n",
        "    Returns:\n",
        "        dict: JSON response from the PredictIt API containing market data.\n",
        "    \"\"\"\n",
        "    url = \"https://www.predictit.org/api/marketdata/all/\"\n",
        "    try:\n",
        "        # Send GET request to API\n",
        "        response = requests.get(url)\n",
        "        response.raise_for_status()  # Raise error for bad HTTP response\n",
        "        print(\"Data fetched successfully!\")\n",
        "        return response.json()\n",
        "    except requests.exceptions.RequestException as e:\n",
        "        print(f\"Error fetching data: {e}\")\n",
        "        return None\n",
        "\n",
        "# test function\n",
        "raw_data = fetch_predictit_data()\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8CBDuEAt0lQ5",
        "outputId": "656409ea-c1c7-436d-c235-ed2ccc5ede0e"
      },
      "execution_count": 42,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Data fetched successfully!\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Here I parse and structure the raw JSON data and structure it as a dataframe. This is done to isolate parsing logic in a separate function for modularity,moreover a dataframe is in tabular format which is ideal for analysis and storage."
      ],
      "metadata": {
        "id": "e1vwbk8P9Hk0"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Ingestion phase\n",
        "\n",
        "\n",
        "import pandas as pd\n",
        "\n",
        "def parse_predictit_data(raw_data):\n",
        "    \"\"\"\n",
        "    Parse raw JSON data from PredictIt API and convert it to a structured DataFrame.\n",
        "\n",
        "    Args:\n",
        "        raw_data (dict): Raw JSON response from PredictIt API.\n",
        "\n",
        "    Returns:\n",
        "        pd.DataFrame: Parsed data as a Pandas DataFrame.\n",
        "    \"\"\"\n",
        "    if not raw_data:\n",
        "        print(\"No data to parse.\")\n",
        "        return None\n",
        "\n",
        "    # Initialize an empty list to store parsed data\n",
        "    parsed_data = []\n",
        "\n",
        "    # Extract relevant fields from each market\n",
        "    for market in raw_data.get(\"markets\", []):\n",
        "        market_name = market.get(\"name\")\n",
        "        for contract in market.get(\"contracts\", []):\n",
        "            parsed_data.append({\n",
        "                \"Market\": market_name,\n",
        "                \"Contract\": contract.get(\"name\"),\n",
        "                \"LastTradePrice\": contract.get(\"lastTradePrice\"),\n",
        "                \"BestBuyYesCost\": contract.get(\"bestBuyYesCost\"),\n",
        "                \"BestBuyNoCost\": contract.get(\"bestBuyNoCost\")\n",
        "            })\n",
        "\n",
        "    # convert the list of dictionaries into a Pandas DataFrame\n",
        "    return pd.DataFrame(parsed_data)\n",
        "\n",
        "# parse raw data\n",
        "df = parse_predictit_data(raw_data)\n",
        "print(df.head())  # display the first few rows\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QbjS_OFV1Ad1",
        "outputId": "ec351295-7e1d-4718-dc2e-a1b4b3ef5708"
      },
      "execution_count": 43,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "                                            Market  \\\n",
            "0     Will Joe Biden resign during his first term?   \n",
            "1     Will Kamala Harris be the 47th US president?   \n",
            "2  Who will win the 2024 US presidential election?   \n",
            "3  Who will win the 2024 US presidential election?   \n",
            "4  Who will win the 2024 US presidential election?   \n",
            "\n",
            "                                       Contract  LastTradePrice  \\\n",
            "0  Will Joe Biden resign during his first term?            0.05   \n",
            "1  Will Kamala Harris be the 47th US president?            0.07   \n",
            "2                                  Donald Trump            0.97   \n",
            "3                                 Kamala Harris            0.02   \n",
            "4                                  Ron DeSantis            0.01   \n",
            "\n",
            "   BestBuyYesCost  BestBuyNoCost  \n",
            "0            0.06           0.95  \n",
            "1            0.07           0.94  \n",
            "2            0.99           0.03  \n",
            "3            0.02           0.99  \n",
            "4            0.01            NaN  \n"
          ]
        }
      ]
    }
  ]
}