DEBS_2022_GC_GROUP_4.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "# DEBS 2022 Grand Challenge \n",
    "## Real-time Market Analytics for Rapid Deployment and High Accessiblity using PySpark and Docker\n",
    "\n",
    "Group 4: Suyeon Wang, JaeKyeong Kim, Sejin Chun\n",
    "\n",
    "This code is also available at [github](https://github.com/developsu/debs2022gc)"
   ],
   "metadata": {
    "id": "q2yQkGXjW4-_"
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Before executing codes,\n",
    "- upload the **challenger.proto** file \n",
    "- login with your google account \n",
    "- give your authorization to google drive for drive mount\n",
    "- set up your token key\n",
    "- set up the **folder dir** to store streams of csv files retrieved from challenger system\n",
    "- Last, move **Runtime**-> Click [**Run all**] (Ctrl + F9)\n"
   ],
   "metadata": {
    "id": "H-VTfwPQhPxT"
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "**Dataset License.** Sebastian Frischbier, Jawad Tahir, Christoph Doblander, Arne Hormann, Ruben Mayer. \n",
    "The DEBS 2021 grand challenge: Detecting Trading Trends in Real Financial Tick Data. \n",
    "Proceedings of the 16th ACM International Conference on Distributed and Event-based Systems (DEBS'22) "
   ],
   "metadata": {
    "id": "7MAhuvALjdEW"
   }
  },
  {
   "cell_type": "code",
   "source": [
    "TOKEN_KEY = 'sdpiwsqcjhlimgruqdhyckrgwjwlawvh'\n",
    "folderDir = \"stream\"\n",
    "benchmark_type = \"Evaluation\""
   ],
   "metadata": {
    "id": "Y1JQtlNzicxt"
   },
   "execution_count": 1,
   "outputs": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "import os\n",
    "os.mkdir(folderDir)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Installation"
   ],
   "metadata": {
    "id": "1XwN7FTGWurw"
   }
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "s_-CtfhLP1KY",
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "outputId": "f5c7a43e-6337-464a-d15c-cc435ca78fc9"
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Reading package lists... Done\n",
      "Building dependency tree       \n",
      "Reading state information... Done\n",
      "The following additional packages will be installed:\n",
      "  openjdk-8-jre-headless\n",
      "Suggested packages:\n",
      "  openjdk-8-demo openjdk-8-source libnss-mdns fonts-dejavu-extra\n",
      "  fonts-ipafont-gothic fonts-ipafont-mincho fonts-wqy-microhei\n",
      "  fonts-wqy-zenhei fonts-indic\n",
      "The following NEW packages will be installed:\n",
      "  openjdk-8-jdk-headless openjdk-8-jre-headless\n",
      "0 upgraded, 2 newly installed, 0 to remove and 39 not upgraded.\n",
      "Need to get 36.5 MB of archives.\n",
      "After this operation, 143 MB of additional disk space will be used.\n",
      "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 openjdk-8-jre-headless amd64 8u312-b07-0ubuntu1~18.04 [28.2 MB]\n",
      "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 openjdk-8-jdk-headless amd64 8u312-b07-0ubuntu1~18.04 [8,298 kB]\n",
      "Fetched 36.5 MB in 5s (7,216 kB/s)\n",
      "Selecting previously unselected package openjdk-8-jre-headless:amd64.\n",
      "(Reading database ... 155455 files and directories currently installed.)\n",
      "Preparing to unpack .../openjdk-8-jre-headless_8u312-b07-0ubuntu1~18.04_amd64.deb ...\n",
      "Unpacking openjdk-8-jre-headless:amd64 (8u312-b07-0ubuntu1~18.04) ...\n",
      "Selecting previously unselected package openjdk-8-jdk-headless:amd64.\n",
      "Preparing to unpack .../openjdk-8-jdk-headless_8u312-b07-0ubuntu1~18.04_amd64.deb ...\n",
      "Unpacking openjdk-8-jdk-headless:amd64 (8u312-b07-0ubuntu1~18.04) ...\n",
      "Setting up openjdk-8-jre-headless:amd64 (8u312-b07-0ubuntu1~18.04) ...\n",
      "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/orbd to provide /usr/bin/orbd (orbd) in auto mode\n",
      "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/servertool to provide /usr/bin/servertool (servertool) in auto mode\n",
      "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/tnameserv to provide /usr/bin/tnameserv (tnameserv) in auto mode\n",
      "Setting up openjdk-8-jdk-headless:amd64 (8u312-b07-0ubuntu1~18.04) ...\n",
      "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/idlj to provide /usr/bin/idlj (idlj) in auto mode\n",
      "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/wsimport to provide /usr/bin/wsimport (wsimport) in auto mode\n",
      "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/jsadebugd to provide /usr/bin/jsadebugd (jsadebugd) in auto mode\n",
      "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/native2ascii to provide /usr/bin/native2ascii (native2ascii) in auto mode\n",
      "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/javah to provide /usr/bin/javah (javah) in auto mode\n",
      "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/hsdb to provide /usr/bin/hsdb (hsdb) in auto mode\n",
      "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/clhsdb to provide /usr/bin/clhsdb (clhsdb) in auto mode\n",
      "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/xjc to provide /usr/bin/xjc (xjc) in auto mode\n",
      "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/schemagen to provide /usr/bin/schemagen (schemagen) in auto mode\n",
      "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/extcheck to provide /usr/bin/extcheck (extcheck) in auto mode\n",
      "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/jhat to provide /usr/bin/jhat (jhat) in auto mode\n",
      "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/wsgen to provide /usr/bin/wsgen (wsgen) in auto mode\n"
     ]
    }
   ],
   "source": [
    "!apt-get install openjdk-8-jdk-headless # jdk"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "downloads and unzips the apache spark "
   ],
   "metadata": {
    "id": "FELudBvJXLAq"
   }
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "kz1bc3uXOWiU",
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "outputId": "3cf644c2-954c-4b4e-ff4a-15e75b1f04ac"
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "spark-3.1.2-bin-hadoop3.2/\n",
      "spark-3.1.2-bin-hadoop3.2/R/\n",
      "spark-3.1.2-bin-hadoop3.2/R/lib/\n",
      "spark-3.1.2-bin-hadoop3.2/R/lib/sparkr.zip\n",
      "spark-3.1.2-bin-hadoop3.2/R/lib/SparkR/\n",
      "spark-3.1.2-bin-hadoop3.2/R/lib/SparkR/worker/\n",
      "spark-3.1.2-bin-hadoop3.2/R/lib/SparkR/worker/worker.R\n",
      "spark-3.1.2-bin-hadoop3.2/R/lib/SparkR/worker/daemon.R\n",
      "spark-3.1.2-bin-hadoop3.2/R/lib/SparkR/tests/\n",
      "spark-3.1.2-bin-hadoop3.2/R/lib/SparkR/tests/testthat/\n",
      "spark-3.1.2-bin-hadoop3.2/R/lib/SparkR/tests/testthat/test_basic.R\n",
      "spark-3.1.2-bin-hadoop3.2/R/lib/SparkR/profile/\n",
      "spark-3.1.2-bin-hadoop3.2/R/lib/SparkR/profile/shell.R\n",
      "spark-3.1.2-bin-hadoop3.2/R/lib/SparkR/profile/general.R\n",
      "spark-3.1.2-bin-hadoop3.2/R/lib/SparkR/doc/\n",
      "spark-3.1.2-bin-hadoop3.2/R/lib/SparkR/doc/sparkr-vignettes.html\n",
      "spark-3.1.2-bin-hadoop3.2/R/lib/SparkR/doc/sparkr-vignettes.Rmd\n",
      "spark-3.1.2-bin-hadoop3.2/R/lib/SparkR/doc/sparkr-vignettes.R\n",
      "spark-3.1.2-bin-hadoop3.2/R/lib/SparkR/doc/index.html\n",
      "spark-3.1.2-bin-hadoop3.2/R/lib/SparkR/R/\n",
      "spark-3.1.2-bin-hadoop3.2/R/lib/SparkR/R/SparkR\n",
      "spark-3.1.2-bin-hadoop3.2/R/lib/SparkR/R/SparkR.rdx\n",
      "spark-3.1.2-bin-hadoop3.2/R/lib/SparkR/R/SparkR.rdb\n",
      "spark-3.1.2-bin-hadoop3.2/R/lib/SparkR/Meta/\n",
      "spark-3.1.2-bin-hadoop3.2/R/lib/SparkR/Meta/features.rds\n",
      "spark-3.1.2-bin-hadoop3.2/R/lib/SparkR/Meta/package.rds\n",
      "spark-3.1.2-bin-hadoop3.2/R/lib/SparkR/Meta/nsInfo.rds\n",
      "spark-3.1.2-bin-hadoop3.2/R/lib/SparkR/Meta/vignette.rds\n",
      "spark-3.1.2-bin-hadoop3.2/R/lib/SparkR/Meta/Rd.rds\n",
      "spark-3.1.2-bin-hadoop3.2/R/lib/SparkR/Meta/links.rds\n",
      "spark-3.1.2-bin-hadoop3.2/R/lib/SparkR/Meta/hsearch.rds\n",
      "spark-3.1.2-bin-hadoop3.2/R/lib/SparkR/DESCRIPTION\n",
      "spark-3.1.2-bin-hadoop3.2/R/lib/SparkR/NAMESPACE\n",
      "spark-3.1.2-bin-hadoop3.2/R/lib/SparkR/html/\n",
      "spark-3.1.2-bin-hadoop3.2/R/lib/SparkR/html/R.css\n",
      "spark-3.1.2-bin-hadoop3.2/R/lib/SparkR/html/00Index.html\n",
      "spark-3.1.2-bin-hadoop3.2/R/lib/SparkR/INDEX\n",
      "spark-3.1.2-bin-hadoop3.2/R/lib/SparkR/help/\n",
      "spark-3.1.2-bin-hadoop3.2/R/lib/SparkR/help/aliases.rds\n",
      "spark-3.1.2-bin-hadoop3.2/R/lib/SparkR/help/AnIndex\n",
      "spark-3.1.2-bin-hadoop3.2/R/lib/SparkR/help/SparkR.rdx\n",
      "spark-3.1.2-bin-hadoop3.2/R/lib/SparkR/help/SparkR.rdb\n",
      "spark-3.1.2-bin-hadoop3.2/R/lib/SparkR/help/paths.rds\n",
      "spark-3.1.2-bin-hadoop3.2/sbin/\n",
      "spark-3.1.2-bin-hadoop3.2/sbin/workers.sh\n",
      "spark-3.1.2-bin-hadoop3.2/sbin/stop-workers.sh\n",
      "spark-3.1.2-bin-hadoop3.2/sbin/stop-worker.sh\n",
      "spark-3.1.2-bin-hadoop3.2/sbin/stop-thriftserver.sh\n",
      "spark-3.1.2-bin-hadoop3.2/sbin/stop-slaves.sh\n",
      "spark-3.1.2-bin-hadoop3.2/sbin/stop-slave.sh\n",
      "spark-3.1.2-bin-hadoop3.2/sbin/stop-mesos-shuffle-service.sh\n",
      "spark-3.1.2-bin-hadoop3.2/sbin/stop-mesos-dispatcher.sh\n",
      "spark-3.1.2-bin-hadoop3.2/sbin/stop-master.sh\n",
      "spark-3.1.2-bin-hadoop3.2/sbin/stop-history-server.sh\n",
      "spark-3.1.2-bin-hadoop3.2/sbin/stop-all.sh\n",
      "spark-3.1.2-bin-hadoop3.2/sbin/start-workers.sh\n",
      "spark-3.1.2-bin-hadoop3.2/sbin/start-worker.sh\n",
      "spark-3.1.2-bin-hadoop3.2/sbin/start-thriftserver.sh\n",
      "spark-3.1.2-bin-hadoop3.2/sbin/start-slaves.sh\n",
      "spark-3.1.2-bin-hadoop3.2/sbin/start-slave.sh\n",
      "spark-3.1.2-bin-hadoop3.2/sbin/start-mesos-shuffle-service.sh\n",
      "spark-3.1.2-bin-hadoop3.2/sbin/start-mesos-dispatcher.sh\n",
      "spark-3.1.2-bin-hadoop3.2/sbin/start-master.sh\n",
      "spark-3.1.2-bin-hadoop3.2/sbin/start-history-server.sh\n",
      "spark-3.1.2-bin-hadoop3.2/sbin/start-all.sh\n",
      "spark-3.1.2-bin-hadoop3.2/sbin/spark-daemons.sh\n",
      "spark-3.1.2-bin-hadoop3.2/sbin/spark-daemon.sh\n",
      "spark-3.1.2-bin-hadoop3.2/sbin/slaves.sh\n",
      "spark-3.1.2-bin-hadoop3.2/sbin/decommission-worker.sh\n",
      "spark-3.1.2-bin-hadoop3.2/sbin/decommission-slave.sh\n",
      "spark-3.1.2-bin-hadoop3.2/sbin/spark-config.sh\n",
      "spark-3.1.2-bin-hadoop3.2/python/\n",
      "spark-3.1.2-bin-hadoop3.2/python/dist/\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark.egg-info/\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark.egg-info/SOURCES.txt\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark.egg-info/top_level.txt\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark.egg-info/requires.txt\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark.egg-info/dependency_links.txt\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark.egg-info/PKG-INFO\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/python/\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/python/pyspark/\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/python/pyspark/shell.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/__pycache__/\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/__pycache__/install.cpython-38.pyc\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/mllib/\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/mllib/evaluation.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/mllib/common.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/mllib/common.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/mllib/clustering.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/mllib/classification.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/mllib/_typing.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/mllib/__init__.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/mllib/util.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/mllib/util.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/mllib/tree.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/mllib/tree.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/mllib/tests/\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/mllib/tests/test_util.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/mllib/tests/test_streaming_algorithms.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/mllib/tests/test_stat.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/mllib/tests/test_feature.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/mllib/tests/test_algorithms.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/mllib/tests/__init__.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/mllib/tests/test_linalg.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/mllib/regression.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/mllib/clustering.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/mllib/classification.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/mllib/stat/\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/mllib/stat/_statistics.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/mllib/stat/__init__.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/mllib/stat/__init__.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/mllib/stat/KernelDensity.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/mllib/stat/KernelDensity.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/mllib/stat/test.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/mllib/stat/test.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/mllib/stat/distribution.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/mllib/stat/distribution.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/mllib/stat/_statistics.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/mllib/regression.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/mllib/recommendation.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/mllib/recommendation.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/mllib/random.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/mllib/random.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/mllib/linalg/\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/mllib/linalg/distributed.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/mllib/linalg/__init__.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/mllib/linalg/__init__.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/mllib/linalg/distributed.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/mllib/fpm.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/mllib/fpm.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/mllib/feature.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/mllib/feature.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/mllib/evaluation.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/wrapper.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/wrapper.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/util.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/util.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/tree.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/tests/\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/tests/test_wrapper.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/tests/test_util.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/tests/test_stat.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/tests/test_pipeline.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/tests/test_persistence.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/tests/test_linalg.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/tests/test_image.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/tests/test_feature.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/tests/test_algorithms.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/tests/__init__.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/tests/test_tuning.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/tests/test_training_summary.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/tests/test_param.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/tests/test_evaluation.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/tests/test_base.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/stat.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/stat.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/regression.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/recommendation.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/pipeline.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/pipeline.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/param/\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/param/shared.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/param/shared.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/param/_shared_params_code_gen.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/param/_shared_params_code_gen.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/param/__init__.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/param/__init__.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/linalg/\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/linalg/__init__.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/linalg/__init__.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/image.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/image.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/functions.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/fpm.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/feature.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/evaluation.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/common.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/common.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/clustering.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/classification.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/base.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/_typing.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/__init__.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/tuning.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/tuning.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/tree.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/regression.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/recommendation.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/functions.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/fpm.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/feature.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/evaluation.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/clustering.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/classification.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/ml/base.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/join.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/java_gateway.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/install.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/find_spark_home.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/files.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/files.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/daemon.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/context.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/context.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/conf.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/conf.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/cloudpickle/\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/cloudpickle/compat.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/cloudpickle/cloudpickle_fast.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/cloudpickle/cloudpickle.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/cloudpickle/__init__.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/broadcast.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/broadcast.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/accumulators.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/accumulators.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/_typing.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/_globals.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/__init__.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/__init__.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/window.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/window.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/utils.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/udf.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/types.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/types.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/tests/\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/tests/test_utils.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/tests/test_udf.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/tests/test_types.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/tests/test_streaming.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/tests/test_session.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/tests/test_serde.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/tests/test_readwriter.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/tests/test_pandas_udf_window.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/tests/test_pandas_udf_typehints.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/tests/test_pandas_udf_scalar.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/tests/test_pandas_udf_grouped_agg.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/tests/test_pandas_udf.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/tests/test_pandas_map.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/tests/test_pandas_cogrouped_map.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/tests/test_datasources.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/tests/test_context.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/tests/test_conf.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/tests/test_catalog.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/tests/__init__.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/tests/test_pandas_grouped_map.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/tests/test_group.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/tests/test_functions.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/tests/test_dataframe.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/tests/test_column.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/tests/test_arrow.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/streaming.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/session.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/readwriter.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/pandas/\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/pandas/utils.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/pandas/typehints.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/pandas/serializers.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/pandas/map_ops.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/pandas/map_ops.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/pandas/group_ops.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/pandas/group_ops.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/pandas/functions.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/pandas/functions.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/pandas/conversion.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/pandas/_typing/\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/pandas/_typing/protocols/\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/pandas/_typing/protocols/series.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/pandas/_typing/protocols/frame.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/pandas/_typing/protocols/__init__.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/pandas/_typing/__init__.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/pandas/__init__.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/pandas/types.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/pandas/conversion.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/group.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/group.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/dataframe.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/context.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/conf.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/conf.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/column.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/catalog.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/avro/\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/avro/__init__.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/avro/functions.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/avro/functions.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/_typing.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/__init__.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/__init__.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/udf.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/streaming.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/session.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/readwriter.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/functions.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/functions.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/dataframe.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/context.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/column.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/sql/catalog.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/shuffle.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/shell.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/serializers.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/resultiterable.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/resultiterable.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/resource/\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/resource/tests/\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/resource/tests/test_resources.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/resource/tests/__init__.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/resource/requests.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/resource/requests.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/resource/information.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/resource/information.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/resource/__init__.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/resource/profile.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/resource/profile.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/rddsampler.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/rdd.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/py.typed\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/profiler.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/profiler.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/worker.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/version.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/taskcontext.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/rdd.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/version.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/util.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/traceback_utils.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/tests/\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/tests/test_worker.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/tests/test_util.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/tests/test_taskcontext.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/tests/test_shuffle.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/tests/test_serializers.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/tests/test_readwrite.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/tests/test_rddbarrier.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/tests/test_rdd.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/tests/test_profiler.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/tests/test_pin_thread.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/tests/test_join.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/tests/test_install_spark.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/tests/test_daemon.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/tests/test_context.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/tests/test_conf.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/tests/test_broadcast.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/tests/test_appsubmit.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/tests/__init__.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/testing/\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/testing/streamingutils.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/testing/sqlutils.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/testing/mlutils.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/testing/mllibutils.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/testing/__init__.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/testing/utils.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/taskcontext.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/streaming/\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/streaming/util.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/streaming/tests/\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/streaming/tests/test_listener.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/streaming/tests/test_kinesis.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/streaming/tests/test_dstream.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/streaming/tests/test_context.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/streaming/tests/__init__.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/streaming/listener.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/streaming/listener.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/streaming/kinesis.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/streaming/kinesis.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/streaming/dstream.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/streaming/context.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/streaming/context.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/streaming/__init__.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/streaming/dstream.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/storagelevel.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/storagelevel.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/status.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/status.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/statcounter.pyi\n",
      "spark-3.1.2-bin-hadoop3.2/python/pyspark/statcounter.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/pylintrc\n",
      "spark-3.1.2-bin-hadoop3.2/python/lib/\n",
      "spark-3.1.2-bin-hadoop3.2/python/lib/pyspark.zip\n",
      "spark-3.1.2-bin-hadoop3.2/python/lib/PY4J_LICENSE.txt\n",
      "spark-3.1.2-bin-hadoop3.2/python/lib/py4j-0.10.9-src.zip\n",
      "spark-3.1.2-bin-hadoop3.2/python/docs/\n",
      "spark-3.1.2-bin-hadoop3.2/python/docs/source/\n",
      "spark-3.1.2-bin-hadoop3.2/python/docs/source/user_guide/\n",
      "spark-3.1.2-bin-hadoop3.2/python/docs/source/user_guide/python_packaging.rst\n",
      "spark-3.1.2-bin-hadoop3.2/python/docs/source/user_guide/index.rst\n",
      "spark-3.1.2-bin-hadoop3.2/python/docs/source/user_guide/arrow_pandas.rst\n",
      "spark-3.1.2-bin-hadoop3.2/python/docs/source/reference/\n",
      "spark-3.1.2-bin-hadoop3.2/python/docs/source/reference/pyspark.streaming.rst\n",
      "spark-3.1.2-bin-hadoop3.2/python/docs/source/reference/pyspark.ss.rst\n",
      "spark-3.1.2-bin-hadoop3.2/python/docs/source/reference/pyspark.rst\n",
      "spark-3.1.2-bin-hadoop3.2/python/docs/source/reference/pyspark.resource.rst\n",
      "spark-3.1.2-bin-hadoop3.2/python/docs/source/reference/pyspark.mllib.rst\n",
      "spark-3.1.2-bin-hadoop3.2/python/docs/source/reference/index.rst\n",
      "spark-3.1.2-bin-hadoop3.2/python/docs/source/reference/pyspark.sql.rst\n",
      "spark-3.1.2-bin-hadoop3.2/python/docs/source/reference/pyspark.ml.rst\n",
      "spark-3.1.2-bin-hadoop3.2/python/docs/source/migration_guide/\n",
      "spark-3.1.2-bin-hadoop3.2/python/docs/source/migration_guide/pyspark_2.3_to_2.4.rst\n",
      "spark-3.1.2-bin-hadoop3.2/python/docs/source/migration_guide/pyspark_2.3.0_to_2.3.1_above.rst\n",
      "spark-3.1.2-bin-hadoop3.2/python/docs/source/migration_guide/pyspark_2.2_to_2.3.rst\n",
      "spark-3.1.2-bin-hadoop3.2/python/docs/source/migration_guide/pyspark_1.4_to_1.5.rst\n",
      "spark-3.1.2-bin-hadoop3.2/python/docs/source/migration_guide/pyspark_1.0_1.2_to_1.3.rst\n",
      "spark-3.1.2-bin-hadoop3.2/python/docs/source/migration_guide/pyspark_2.4_to_3.0.rst\n",
      "spark-3.1.2-bin-hadoop3.2/python/docs/source/migration_guide/index.rst\n",
      "spark-3.1.2-bin-hadoop3.2/python/docs/source/getting_started/\n",
      "spark-3.1.2-bin-hadoop3.2/python/docs/source/getting_started/quickstart.ipynb\n",
      "spark-3.1.2-bin-hadoop3.2/python/docs/source/getting_started/index.rst\n",
      "spark-3.1.2-bin-hadoop3.2/python/docs/source/getting_started/install.rst\n",
      "spark-3.1.2-bin-hadoop3.2/python/docs/source/development/\n",
      "spark-3.1.2-bin-hadoop3.2/python/docs/source/development/testing.rst\n",
      "spark-3.1.2-bin-hadoop3.2/python/docs/source/development/setting_ide.rst\n",
      "spark-3.1.2-bin-hadoop3.2/python/docs/source/development/index.rst\n",
      "spark-3.1.2-bin-hadoop3.2/python/docs/source/development/debugging.rst\n",
      "spark-3.1.2-bin-hadoop3.2/python/docs/source/development/contributing.rst\n",
      "spark-3.1.2-bin-hadoop3.2/python/docs/source/_templates/\n",
      "spark-3.1.2-bin-hadoop3.2/python/docs/source/_templates/autosummary/\n",
      "spark-3.1.2-bin-hadoop3.2/python/docs/source/_templates/autosummary/class_with_docs.rst\n",
      "spark-3.1.2-bin-hadoop3.2/python/docs/source/_templates/autosummary/class.rst\n",
      "spark-3.1.2-bin-hadoop3.2/python/docs/source/_static/\n",
      "spark-3.1.2-bin-hadoop3.2/python/docs/source/_static/css/\n",
      "spark-3.1.2-bin-hadoop3.2/python/docs/source/_static/css/pyspark.css\n",
      "spark-3.1.2-bin-hadoop3.2/python/docs/source/_static/copybutton.js\n",
      "spark-3.1.2-bin-hadoop3.2/python/docs/source/index.rst\n",
      "spark-3.1.2-bin-hadoop3.2/python/docs/source/conf.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/docs/make.bat\n",
      "spark-3.1.2-bin-hadoop3.2/python/docs/make2.bat\n",
      "spark-3.1.2-bin-hadoop3.2/python/docs/Makefile\n",
      "spark-3.1.2-bin-hadoop3.2/python/README.md\n",
      "spark-3.1.2-bin-hadoop3.2/python/MANIFEST.in\n",
      "spark-3.1.2-bin-hadoop3.2/python/.gitignore\n",
      "spark-3.1.2-bin-hadoop3.2/python/.coveragerc\n",
      "spark-3.1.2-bin-hadoop3.2/python/setup.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/run-tests.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/run-tests-with-coverage\n",
      "spark-3.1.2-bin-hadoop3.2/python/mypy.ini\n",
      "spark-3.1.2-bin-hadoop3.2/python/test_support/\n",
      "spark-3.1.2-bin-hadoop3.2/python/test_support/userlibrary.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/test_support/userlib-0.1.zip\n",
      "spark-3.1.2-bin-hadoop3.2/python/test_support/sql/\n",
      "spark-3.1.2-bin-hadoop3.2/python/test_support/sql/text-test.txt\n",
      "spark-3.1.2-bin-hadoop3.2/python/test_support/sql/streaming/\n",
      "spark-3.1.2-bin-hadoop3.2/python/test_support/sql/streaming/text-test.txt\n",
      "spark-3.1.2-bin-hadoop3.2/python/test_support/sql/people_array_utf16le.json\n",
      "spark-3.1.2-bin-hadoop3.2/python/test_support/sql/people_array.json\n",
      "spark-3.1.2-bin-hadoop3.2/python/test_support/sql/people1.json\n",
      "spark-3.1.2-bin-hadoop3.2/python/test_support/sql/people.json\n",
      "spark-3.1.2-bin-hadoop3.2/python/test_support/sql/parquet_partitioned/\n",
      "spark-3.1.2-bin-hadoop3.2/python/test_support/sql/parquet_partitioned/year=2015/\n",
      "spark-3.1.2-bin-hadoop3.2/python/test_support/sql/parquet_partitioned/year=2015/month=9/\n",
      "spark-3.1.2-bin-hadoop3.2/python/test_support/sql/parquet_partitioned/year=2015/month=9/day=1/\n",
      "spark-3.1.2-bin-hadoop3.2/python/test_support/sql/parquet_partitioned/year=2015/month=9/day=1/part-r-00007.gz.parquet\n",
      "spark-3.1.2-bin-hadoop3.2/python/test_support/sql/parquet_partitioned/year=2015/month=9/day=1/.part-r-00007.gz.parquet.crc\n",
      "spark-3.1.2-bin-hadoop3.2/python/test_support/sql/parquet_partitioned/year=2015/month=10/\n",
      "spark-3.1.2-bin-hadoop3.2/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=26/\n",
      "spark-3.1.2-bin-hadoop3.2/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=26/part-r-00005.gz.parquet\n",
      "spark-3.1.2-bin-hadoop3.2/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=26/.part-r-00005.gz.parquet.crc\n",
      "spark-3.1.2-bin-hadoop3.2/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=25/\n",
      "spark-3.1.2-bin-hadoop3.2/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=25/part-r-00004.gz.parquet\n",
      "spark-3.1.2-bin-hadoop3.2/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=25/part-r-00002.gz.parquet\n",
      "spark-3.1.2-bin-hadoop3.2/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=25/.part-r-00004.gz.parquet.crc\n",
      "spark-3.1.2-bin-hadoop3.2/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=25/.part-r-00002.gz.parquet.crc\n",
      "spark-3.1.2-bin-hadoop3.2/python/test_support/sql/parquet_partitioned/year=2014/\n",
      "spark-3.1.2-bin-hadoop3.2/python/test_support/sql/parquet_partitioned/year=2014/month=9/\n",
      "spark-3.1.2-bin-hadoop3.2/python/test_support/sql/parquet_partitioned/year=2014/month=9/day=1/\n",
      "spark-3.1.2-bin-hadoop3.2/python/test_support/sql/parquet_partitioned/year=2014/month=9/day=1/part-r-00008.gz.parquet\n",
      "spark-3.1.2-bin-hadoop3.2/python/test_support/sql/parquet_partitioned/year=2014/month=9/day=1/.part-r-00008.gz.parquet.crc\n",
      "spark-3.1.2-bin-hadoop3.2/python/test_support/sql/parquet_partitioned/_metadata\n",
      "spark-3.1.2-bin-hadoop3.2/python/test_support/sql/parquet_partitioned/_common_metadata\n",
      "spark-3.1.2-bin-hadoop3.2/python/test_support/sql/parquet_partitioned/_SUCCESS\n",
      "spark-3.1.2-bin-hadoop3.2/python/test_support/sql/orc_partitioned/\n",
      "spark-3.1.2-bin-hadoop3.2/python/test_support/sql/orc_partitioned/b=1/\n",
      "spark-3.1.2-bin-hadoop3.2/python/test_support/sql/orc_partitioned/b=1/c=1/\n",
      "spark-3.1.2-bin-hadoop3.2/python/test_support/sql/orc_partitioned/b=1/c=1/part-r-00000-829af031-b970-49d6-ad39-30460a0be2c8.orc\n",
      "spark-3.1.2-bin-hadoop3.2/python/test_support/sql/orc_partitioned/b=1/c=1/.part-r-00000-829af031-b970-49d6-ad39-30460a0be2c8.orc.crc\n",
      "spark-3.1.2-bin-hadoop3.2/python/test_support/sql/orc_partitioned/b=0/\n",
      "spark-3.1.2-bin-hadoop3.2/python/test_support/sql/orc_partitioned/b=0/c=0/\n",
      "spark-3.1.2-bin-hadoop3.2/python/test_support/sql/orc_partitioned/b=0/c=0/part-r-00000-829af031-b970-49d6-ad39-30460a0be2c8.orc\n",
      "spark-3.1.2-bin-hadoop3.2/python/test_support/sql/orc_partitioned/b=0/c=0/.part-r-00000-829af031-b970-49d6-ad39-30460a0be2c8.orc.crc\n",
      "spark-3.1.2-bin-hadoop3.2/python/test_support/sql/orc_partitioned/_SUCCESS\n",
      "spark-3.1.2-bin-hadoop3.2/python/test_support/sql/ages_newlines.csv\n",
      "spark-3.1.2-bin-hadoop3.2/python/test_support/sql/ages.csv\n",
      "spark-3.1.2-bin-hadoop3.2/python/test_support/hello/\n",
      "spark-3.1.2-bin-hadoop3.2/python/test_support/hello/sub_hello/\n",
      "spark-3.1.2-bin-hadoop3.2/python/test_support/hello/sub_hello/sub_hello.txt\n",
      "spark-3.1.2-bin-hadoop3.2/python/test_support/hello/hello.txt\n",
      "spark-3.1.2-bin-hadoop3.2/python/test_support/SimpleHTTPServer.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/test_coverage/\n",
      "spark-3.1.2-bin-hadoop3.2/python/test_coverage/sitecustomize.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/test_coverage/coverage_daemon.py\n",
      "spark-3.1.2-bin-hadoop3.2/python/test_coverage/conf/\n",
      "spark-3.1.2-bin-hadoop3.2/python/test_coverage/conf/spark-defaults.conf\n",
      "spark-3.1.2-bin-hadoop3.2/python/setup.cfg\n",
      "spark-3.1.2-bin-hadoop3.2/python/run-tests\n",
      "spark-3.1.2-bin-hadoop3.2/bin/\n",
      "spark-3.1.2-bin-hadoop3.2/bin/sparkR2.cmd\n",
      "spark-3.1.2-bin-hadoop3.2/bin/sparkR.cmd\n",
      "spark-3.1.2-bin-hadoop3.2/bin/sparkR\n",
      "spark-3.1.2-bin-hadoop3.2/bin/spark-submit2.cmd\n",
      "spark-3.1.2-bin-hadoop3.2/bin/spark-submit.cmd\n",
      "spark-3.1.2-bin-hadoop3.2/bin/spark-submit\n",
      "spark-3.1.2-bin-hadoop3.2/bin/spark-sql2.cmd\n",
      "spark-3.1.2-bin-hadoop3.2/bin/spark-sql.cmd\n",
      "spark-3.1.2-bin-hadoop3.2/bin/spark-sql\n",
      "spark-3.1.2-bin-hadoop3.2/bin/spark-shell2.cmd\n",
      "spark-3.1.2-bin-hadoop3.2/bin/spark-shell.cmd\n",
      "spark-3.1.2-bin-hadoop3.2/bin/spark-shell\n",
      "spark-3.1.2-bin-hadoop3.2/bin/spark-class2.cmd\n",
      "spark-3.1.2-bin-hadoop3.2/bin/spark-class.cmd\n",
      "spark-3.1.2-bin-hadoop3.2/bin/spark-class\n",
      "spark-3.1.2-bin-hadoop3.2/bin/run-example.cmd\n",
      "spark-3.1.2-bin-hadoop3.2/bin/run-example\n",
      "spark-3.1.2-bin-hadoop3.2/bin/pyspark.cmd\n",
      "spark-3.1.2-bin-hadoop3.2/bin/load-spark-env.sh\n",
      "spark-3.1.2-bin-hadoop3.2/bin/load-spark-env.cmd\n",
      "spark-3.1.2-bin-hadoop3.2/bin/find-spark-home.cmd\n",
      "spark-3.1.2-bin-hadoop3.2/bin/find-spark-home\n",
      "spark-3.1.2-bin-hadoop3.2/bin/docker-image-tool.sh\n",
      "spark-3.1.2-bin-hadoop3.2/bin/beeline.cmd\n",
      "spark-3.1.2-bin-hadoop3.2/bin/beeline\n",
      "spark-3.1.2-bin-hadoop3.2/bin/pyspark2.cmd\n",
      "spark-3.1.2-bin-hadoop3.2/bin/pyspark\n",
      "spark-3.1.2-bin-hadoop3.2/README.md\n",
      "spark-3.1.2-bin-hadoop3.2/conf/\n",
      "spark-3.1.2-bin-hadoop3.2/conf/workers.template\n",
      "spark-3.1.2-bin-hadoop3.2/conf/spark-env.sh.template\n",
      "spark-3.1.2-bin-hadoop3.2/conf/spark-defaults.conf.template\n",
      "spark-3.1.2-bin-hadoop3.2/conf/metrics.properties.template\n",
      "spark-3.1.2-bin-hadoop3.2/conf/log4j.properties.template\n",
      "spark-3.1.2-bin-hadoop3.2/conf/fairscheduler.xml.template\n",
      "spark-3.1.2-bin-hadoop3.2/data/\n",
      "spark-3.1.2-bin-hadoop3.2/data/streaming/\n",
      "spark-3.1.2-bin-hadoop3.2/data/streaming/AFINN-111.txt\n",
      "spark-3.1.2-bin-hadoop3.2/data/mllib/\n",
      "spark-3.1.2-bin-hadoop3.2/data/mllib/streaming_kmeans_data_test.txt\n",
      "spark-3.1.2-bin-hadoop3.2/data/mllib/sample_svm_data.txt\n",
      "spark-3.1.2-bin-hadoop3.2/data/mllib/sample_multiclass_classification_data.txt\n",
      "spark-3.1.2-bin-hadoop3.2/data/mllib/sample_movielens_data.txt\n",
      "spark-3.1.2-bin-hadoop3.2/data/mllib/sample_linear_regression_data.txt\n",
      "spark-3.1.2-bin-hadoop3.2/data/mllib/sample_libsvm_data.txt\n",
      "spark-3.1.2-bin-hadoop3.2/data/mllib/sample_lda_libsvm_data.txt\n",
      "spark-3.1.2-bin-hadoop3.2/data/mllib/sample_lda_data.txt\n",
      "spark-3.1.2-bin-hadoop3.2/data/mllib/sample_kmeans_data.txt\n",
      "spark-3.1.2-bin-hadoop3.2/data/mllib/sample_isotonic_regression_libsvm_data.txt\n",
      "spark-3.1.2-bin-hadoop3.2/data/mllib/sample_fpgrowth.txt\n",
      "spark-3.1.2-bin-hadoop3.2/data/mllib/sample_binary_classification_data.txt\n",
      "spark-3.1.2-bin-hadoop3.2/data/mllib/ridge-data/\n",
      "spark-3.1.2-bin-hadoop3.2/data/mllib/ridge-data/lpsa.data\n",
      "spark-3.1.2-bin-hadoop3.2/data/mllib/pic_data.txt\n",
      "spark-3.1.2-bin-hadoop3.2/data/mllib/pagerank_data.txt\n",
      "spark-3.1.2-bin-hadoop3.2/data/mllib/kmeans_data.txt\n",
      "spark-3.1.2-bin-hadoop3.2/data/mllib/iris_libsvm.txt\n",
      "spark-3.1.2-bin-hadoop3.2/data/mllib/images/\n",
      "spark-3.1.2-bin-hadoop3.2/data/mllib/images/partitioned/\n",
      "spark-3.1.2-bin-hadoop3.2/data/mllib/images/partitioned/cls=multichannel/\n",
      "spark-3.1.2-bin-hadoop3.2/data/mllib/images/partitioned/cls=multichannel/date=2018-02/\n",
      "spark-3.1.2-bin-hadoop3.2/data/mllib/images/partitioned/cls=multichannel/date=2018-02/grayscale.jpg\n",
      "spark-3.1.2-bin-hadoop3.2/data/mllib/images/partitioned/cls=multichannel/date=2018-02/chr30.4.184.jpg\n",
      "spark-3.1.2-bin-hadoop3.2/data/mllib/images/partitioned/cls=multichannel/date=2018-01/\n",
      "spark-3.1.2-bin-hadoop3.2/data/mllib/images/partitioned/cls=multichannel/date=2018-01/BGRA_alpha_60.png\n",
      "spark-3.1.2-bin-hadoop3.2/data/mllib/images/partitioned/cls=multichannel/date=2018-01/BGRA.png\n",
      "spark-3.1.2-bin-hadoop3.2/data/mllib/images/partitioned/cls=kittens/\n",
      "spark-3.1.2-bin-hadoop3.2/data/mllib/images/partitioned/cls=kittens/date=2018-02/\n",
      "spark-3.1.2-bin-hadoop3.2/data/mllib/images/partitioned/cls=kittens/date=2018-02/DP802813.jpg\n",
      "spark-3.1.2-bin-hadoop3.2/data/mllib/images/partitioned/cls=kittens/date=2018-02/DP153539.jpg\n",
      "spark-3.1.2-bin-hadoop3.2/data/mllib/images/partitioned/cls=kittens/date=2018-02/54893.jpg\n",
      "spark-3.1.2-bin-hadoop3.2/data/mllib/images/partitioned/cls=kittens/date=2018-01/\n",
      "spark-3.1.2-bin-hadoop3.2/data/mllib/images/partitioned/cls=kittens/date=2018-01/not-image.txt\n",
      "spark-3.1.2-bin-hadoop3.2/data/mllib/images/partitioned/cls=kittens/date=2018-01/29.5.a_b_EGDP022204.jpg\n",
      "spark-3.1.2-bin-hadoop3.2/data/mllib/images/origin/\n",
      "spark-3.1.2-bin-hadoop3.2/data/mllib/images/origin/multi-channel/\n",
      "spark-3.1.2-bin-hadoop3.2/data/mllib/images/origin/multi-channel/grayscale.jpg\n",
      "spark-3.1.2-bin-hadoop3.2/data/mllib/images/origin/multi-channel/chr30.4.184.jpg\n",
      "spark-3.1.2-bin-hadoop3.2/data/mllib/images/origin/multi-channel/BGRA_alpha_60.png\n",
      "spark-3.1.2-bin-hadoop3.2/data/mllib/images/origin/multi-channel/BGRA.png\n",
      "spark-3.1.2-bin-hadoop3.2/data/mllib/images/origin/license.txt\n",
      "spark-3.1.2-bin-hadoop3.2/data/mllib/images/origin/kittens/\n",
      "spark-3.1.2-bin-hadoop3.2/data/mllib/images/origin/kittens/not-image.txt\n",
      "spark-3.1.2-bin-hadoop3.2/data/mllib/images/origin/kittens/DP802813.jpg\n",
      "spark-3.1.2-bin-hadoop3.2/data/mllib/images/origin/kittens/DP153539.jpg\n",
      "spark-3.1.2-bin-hadoop3.2/data/mllib/images/origin/kittens/54893.jpg\n",
      "spark-3.1.2-bin-hadoop3.2/data/mllib/images/origin/kittens/29.5.a_b_EGDP022204.jpg\n",
      "spark-3.1.2-bin-hadoop3.2/data/mllib/images/license.txt\n",
      "spark-3.1.2-bin-hadoop3.2/data/mllib/gmm_data.txt\n",
      "spark-3.1.2-bin-hadoop3.2/data/mllib/als/\n",
      "spark-3.1.2-bin-hadoop3.2/data/mllib/als/test.data\n",
      "spark-3.1.2-bin-hadoop3.2/data/mllib/als/sample_movielens_ratings.txt\n",
      "spark-3.1.2-bin-hadoop3.2/data/graphx/\n",
      "spark-3.1.2-bin-hadoop3.2/data/graphx/users.txt\n",
      "spark-3.1.2-bin-hadoop3.2/data/graphx/followers.txt\n",
      "spark-3.1.2-bin-hadoop3.2/NOTICE\n",
      "spark-3.1.2-bin-hadoop3.2/licenses/\n",
      "spark-3.1.2-bin-hadoop3.2/licenses/LICENSE-zstd.txt\n",
      "spark-3.1.2-bin-hadoop3.2/licenses/LICENSE-zstd-jni.txt\n",
      "spark-3.1.2-bin-hadoop3.2/licenses/LICENSE-xmlenc.txt\n",
      "spark-3.1.2-bin-hadoop3.2/licenses/LICENSE-vis-timeline.txt\n",
      "spark-3.1.2-bin-hadoop3.2/licenses/LICENSE-spire.txt\n",
      "spark-3.1.2-bin-hadoop3.2/licenses/LICENSE-sorttable.js.txt\n",
      "spark-3.1.2-bin-hadoop3.2/licenses/LICENSE-slf4j.txt\n",
      "spark-3.1.2-bin-hadoop3.2/licenses/LICENSE-scopt.txt\n",
      "spark-3.1.2-bin-hadoop3.2/licenses/LICENSE-scala.txt\n",
      "spark-3.1.2-bin-hadoop3.2/licenses/LICENSE-sbt-launch-lib.txt\n",
      "spark-3.1.2-bin-hadoop3.2/licenses/LICENSE-respond.txt\n",
      "spark-3.1.2-bin-hadoop3.2/licenses/LICENSE-reflectasm.txt\n",
      "spark-3.1.2-bin-hadoop3.2/licenses/LICENSE-re2j.txt\n",
      "spark-3.1.2-bin-hadoop3.2/licenses/LICENSE-pyrolite.txt\n",
      "spark-3.1.2-bin-hadoop3.2/licenses/LICENSE-py4j.txt\n",
      "spark-3.1.2-bin-hadoop3.2/licenses/LICENSE-protobuf.txt\n",
      "spark-3.1.2-bin-hadoop3.2/licenses/LICENSE-pmml-model.txt\n",
      "spark-3.1.2-bin-hadoop3.2/licenses/LICENSE-paranamer.txt\n",
      "spark-3.1.2-bin-hadoop3.2/licenses/LICENSE-netlib.txt\n",
      "spark-3.1.2-bin-hadoop3.2/licenses/LICENSE-mustache.txt\n",
      "spark-3.1.2-bin-hadoop3.2/licenses/LICENSE-modernizr.txt\n",
      "spark-3.1.2-bin-hadoop3.2/licenses/LICENSE-minlog.txt\n",
      "spark-3.1.2-bin-hadoop3.2/licenses/LICENSE-matchMedia-polyfill.txt\n",
      "spark-3.1.2-bin-hadoop3.2/licenses/LICENSE-machinist.txt\n",
      "spark-3.1.2-bin-hadoop3.2/licenses/LICENSE-leveldbjni.txt\n",
      "spark-3.1.2-bin-hadoop3.2/licenses/LICENSE-kryo.txt\n",
      "spark-3.1.2-bin-hadoop3.2/licenses/LICENSE-jsp-api.txt\n",
      "spark-3.1.2-bin-hadoop3.2/licenses/LICENSE-json-formatter.txt\n",
      "spark-3.1.2-bin-hadoop3.2/licenses/LICENSE-jquery.txt\n",
      "spark-3.1.2-bin-hadoop3.2/licenses/LICENSE-join.txt\n",
      "spark-3.1.2-bin-hadoop3.2/licenses/LICENSE-jodd.txt\n",
      "spark-3.1.2-bin-hadoop3.2/licenses/LICENSE-jline.txt\n",
      "spark-3.1.2-bin-hadoop3.2/licenses/LICENSE-jaxb-runtime.txt\n",
      "spark-3.1.2-bin-hadoop3.2/licenses/LICENSE-javolution.txt\n",
      "spark-3.1.2-bin-hadoop3.2/licenses/LICENSE-javax-transaction-transaction-api.txt\n",
      "spark-3.1.2-bin-hadoop3.2/licenses/LICENSE-javassist.html\n",
      "spark-3.1.2-bin-hadoop3.2/licenses/LICENSE-janino.txt\n",
      "spark-3.1.2-bin-hadoop3.2/licenses/LICENSE-jakarta.xml.bind-api.txt\n",
      "spark-3.1.2-bin-hadoop3.2/licenses/LICENSE-jakarta.activation-api.txt\n",
      "spark-3.1.2-bin-hadoop3.2/licenses/LICENSE-jakarta-ws-rs-api\n",
      "spark-3.1.2-bin-hadoop3.2/licenses/LICENSE-jakarta-annotation-api\n",
      "spark-3.1.2-bin-hadoop3.2/licenses/LICENSE-istack-commons-runtime.txt\n",
      "spark-3.1.2-bin-hadoop3.2/licenses/LICENSE-graphlib-dot.txt\n",
      "spark-3.1.2-bin-hadoop3.2/licenses/LICENSE-f2j.txt\n",
      "spark-3.1.2-bin-hadoop3.2/licenses/LICENSE-dnsjava.txt\n",
      "spark-3.1.2-bin-hadoop3.2/licenses/LICENSE-datatables.txt\n",
      "spark-3.1.2-bin-hadoop3.2/licenses/LICENSE-dagre-d3.txt\n",
      "spark-3.1.2-bin-hadoop3.2/licenses/LICENSE-d3.min.js.txt\n",
      "spark-3.1.2-bin-hadoop3.2/licenses/LICENSE-cloudpickle.txt\n",
      "spark-3.1.2-bin-hadoop3.2/licenses/LICENSE-bootstrap.txt\n",
      "spark-3.1.2-bin-hadoop3.2/licenses/LICENSE-automaton.txt\n",
      "spark-3.1.2-bin-hadoop3.2/licenses/LICENSE-arpack.txt\n",
      "spark-3.1.2-bin-hadoop3.2/licenses/LICENSE-antlr.txt\n",
      "spark-3.1.2-bin-hadoop3.2/licenses/LICENSE-JTransforms.txt\n",
      "spark-3.1.2-bin-hadoop3.2/licenses/LICENSE-JLargeArrays.txt\n",
      "spark-3.1.2-bin-hadoop3.2/licenses/LICENSE-CC0.txt\n",
      "spark-3.1.2-bin-hadoop3.2/licenses/LICENSE-AnchorJS.txt\n",
      "spark-3.1.2-bin-hadoop3.2/LICENSE\n",
      "spark-3.1.2-bin-hadoop3.2/examples/\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/PrefixSpanExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/PowerIterationClusteringExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/PMMLModelExportExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/PCAOnSourceVectorExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/PCAOnRowMatrixExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/NormalizerExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/NaiveBayesExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/MultivariateSummarizer.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/MulticlassMetricsExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/MultiLabelMetricsExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/MovieLensALS.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/LogisticRegressionWithLBFGSExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/LatentDirichletAllocationExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/LDAExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/LBFGSExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/KernelDensityEstimationExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/KMeansExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/IsotonicRegressionExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/HypothesisTestingKolmogorovSmirnovTestExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/HypothesisTestingExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/GradientBoostingRegressionExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/GradientBoostingClassificationExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/GradientBoostedTreesRunner.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/GaussianMixtureExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/FPGrowthExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/ElementwiseProductExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/DenseKMeans.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/DecisionTreeRunner.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/DecisionTreeRegressionExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/DecisionTreeClassificationExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/CosineSimilarity.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/CorrelationsExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/Correlations.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/ChiSqSelectorExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/BisectingKMeansExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/BinaryClassificationMetricsExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/BinaryClassification.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/AssociationRulesExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/AbstractParams.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/Word2VecExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/TallSkinnySVD.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/TallSkinnyPCA.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/TFIDFExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/SummaryStatisticsExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/StreamingTestExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/StreamingLogisticRegression.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/StreamingLinearRegressionExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/StreamingKMeansExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/StratifiedSamplingExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/StandardScalerExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/SparseNaiveBayes.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/SimpleFPGrowth.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/SampledRDDs.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/SVMWithSGDExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/SVDExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/RecommendationExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/RankingMetricsExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/RandomRDDGeneration.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/RandomForestRegressionExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/mllib/RandomForestClassificationExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/Word2VecExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/VectorSlicerExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/VectorSizeHintExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/VectorIndexerExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/VectorAssemblerExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/VarianceThresholdSelectorExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/UnivariateFeatureSelectorExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/UnaryTransformerExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/TokenizerExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/TfIdfExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/SummarizerExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/StringIndexerExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/StopWordsRemoverExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/StandardScalerExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/SQLTransformerExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/RobustScalerExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/RandomForestRegressorExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/RandomForestExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/RandomForestClassifierExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/RFormulaExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/QuantileDiscretizerExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/PrefixSpanExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/PowerIterationClusteringExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/PolynomialExpansionExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/PipelineExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/PCAExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/OneVsRestExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/OneHotEncoderExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/NormalizerExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/NaiveBayesExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/NGramExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/MultilayerPerceptronClassifierExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/MulticlassLogisticRegressionWithElasticNetExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/ModelSelectionViaTrainValidationSplitExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/ModelSelectionViaCrossValidationExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/MinMaxScalerExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/MinHashLSHExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/MaxAbsScalerExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/LogisticRegressionWithElasticNetExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/LogisticRegressionSummaryExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/LogisticRegressionExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/LinearSVCExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/LinearRegressionWithElasticNetExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/LinearRegressionExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/LDAExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/KMeansExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/IsotonicRegressionExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/InteractionExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/IndexToStringExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/ImputerExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/GradientBoostedTreeRegressorExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/GradientBoostedTreeClassifierExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/GeneralizedLinearRegressionExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/GaussianMixtureExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/GBTExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/FeatureHasherExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/FPGrowthExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/FMRegressorExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/FMClassifierExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/EstimatorTransformerParamExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/ElementwiseProductExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/DeveloperApiExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeRegressionExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeClassificationExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/DataFrameExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/DCTExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/CountVectorizerExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/CorrelationExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/ChiSquareTestExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/ChiSqSelectorExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/BucketizerExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/BucketedRandomProjectionLSHExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/BisectingKMeansExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/BinarizerExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/ALSExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ml/AFTSurvivalRegressionExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/graphx/\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/graphx/TriangleCountingExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/graphx/SynthBenchmark.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/graphx/SSSPExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/graphx/PageRankExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/graphx/LiveJournalPageRank.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/graphx/ConnectedComponentsExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/graphx/ComprehensiveExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/graphx/Analytics.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/graphx/AggregateMessagesExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/SparkTC.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/SparkRemoteFileTest.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/SparkPi.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/SparkPageRank.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/SparkLR.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/SparkKMeans.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/SparkHdfsLR.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/SparkALS.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/SkewedGroupByTest.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/SimpleSkewedGroupByTest.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/MultiBroadcastTest.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/LogQuery.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/LocalPi.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/LocalLR.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/LocalFileLR.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/LocalALS.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/HdfsTest.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/GroupByTest.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/ExceptionHandlingTest.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/LocalKMeans.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/DriverSubmissionTest.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/DFSReadWriteTest.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/BroadcastTest.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/AccumulatorMetricsTest.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/streaming/\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/streaming/clickstream/\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/streaming/clickstream/PageViewStream.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/streaming/clickstream/PageViewGenerator.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/streaming/StreamingExamples.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/streaming/StatefulNetworkWordCount.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/streaming/SqlNetworkWordCount.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/streaming/RecoverableNetworkWordCount.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/streaming/RawNetworkGrep.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/streaming/QueueStream.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/streaming/NetworkWordCount.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/streaming/HdfsWordCount.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/streaming/DirectKerberizedKafkaWordCount.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/streaming/DirectKafkaWordCount.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/streaming/CustomReceiver.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/sql/\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/sql/streaming/\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/sql/streaming/StructuredSessionization.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/sql/streaming/StructuredNetworkWordCountWindowed.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/sql/streaming/StructuredNetworkWordCount.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/sql/streaming/StructuredKerberizedKafkaWordCount.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/sql/streaming/StructuredKafkaWordCount.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/sql/hive/\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/sql/hive/SparkHiveExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/sql/UserDefinedUntypedAggregation.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/sql/UserDefinedTypedAggregation.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/sql/UserDefinedScalar.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/sql/SparkSQLExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/sql/RDDRelation.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/sql/SimpleTypedAggregator.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/sql/SQLDataSourceExample.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/pythonconverters/\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scala/org/apache/spark/examples/pythonconverters/AvroConverters.scala\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/resources/\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/resources/users.parquet\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/resources/users.orc\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/resources/users.avro\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/resources/user.avsc\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/resources/people.txt\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/resources/people.json\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/resources/people.csv\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/resources/kv1.txt\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/resources/full_user.avsc\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/resources/employees.json\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/resources/dir1/\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/resources/dir1/file3.json\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/resources/dir1/file1.parquet\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/resources/dir1/dir2/\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/resources/dir1/dir2/file2.parquet\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/r/\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/r/streaming/\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/r/streaming/structured_network_wordcount.R\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/r/ml/\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/r/ml/svmLinear.R\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/r/ml/survreg.R\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/r/ml/randomForest.R\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/r/ml/prefixSpan.R\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/r/ml/powerIterationClustering.R\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/r/ml/naiveBayes.R\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/r/ml/mlp.R\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/r/ml/ml.R\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/r/ml/logit.R\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/r/ml/lm_with_elastic_net.R\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/r/ml/lda.R\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/r/ml/kstest.R\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/r/ml/kmeans.R\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/r/ml/isoreg.R\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/r/ml/glm.R\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/r/ml/gbt.R\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/r/ml/gaussianMixture.R\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/r/ml/fpm.R\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/r/ml/fmRegressor.R\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/r/ml/fmClassifier.R\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/r/ml/decisionTree.R\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/r/ml/bisectingKmeans.R\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/r/ml/als.R\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/r/dataframe.R\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/r/data-manipulation.R\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/r/RSparkSQLExample.R\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/wordcount.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/transitive_closure.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/streaming/\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/streaming/stateful_network_wordcount.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/streaming/sql_network_wordcount.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/streaming/recoverable_network_wordcount.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/streaming/queue_stream.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/streaming/network_wordjoinsentiments.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/streaming/network_wordcount.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/streaming/hdfs_wordcount.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/status_api_demo.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/sql/\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/sql/streaming/\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/sql/streaming/structured_network_wordcount_windowed.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/sql/streaming/structured_network_wordcount.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/sql/streaming/structured_kafka_wordcount.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/sql/hive.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/sql/basic.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/sql/arrow.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/sql/datasource.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/sort.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/pi.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/parquet_inputformat.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/pagerank.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/mllib/\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/mllib/word2vec_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/mllib/word2vec.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/mllib/tf_idf_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/mllib/svm_with_sgd_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/mllib/svd_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/mllib/summary_statistics_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/mllib/streaming_linear_regression_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/mllib/streaming_k_means_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/mllib/stratified_sampling_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/mllib/standard_scaler_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/mllib/sampled_rdds.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/mllib/regression_metrics_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/mllib/recommendation_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/mllib/ranking_metrics_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/mllib/random_rdd_generation.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/mllib/random_forest_regression_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/mllib/random_forest_classification_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/mllib/power_iteration_clustering_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/mllib/pca_rowmatrix_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/mllib/normalizer_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/mllib/naive_bayes_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/mllib/multi_label_metrics_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/mllib/multi_class_metrics_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/mllib/logistic_regression_with_lbfgs_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/mllib/logistic_regression.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/mllib/linear_regression_with_sgd_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/mllib/latent_dirichlet_allocation_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/mllib/kmeans.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/mllib/kernel_density_estimation_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/mllib/k_means_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/mllib/isotonic_regression_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/mllib/hypothesis_testing_kolmogorov_smirnov_test_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/mllib/hypothesis_testing_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/mllib/gradient_boosting_regression_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/mllib/gradient_boosting_classification_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/mllib/gaussian_mixture_model.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/mllib/gaussian_mixture_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/mllib/fpgrowth_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/mllib/elementwise_product_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/mllib/decision_tree_regression_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/mllib/decision_tree_classification_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/mllib/correlations_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/mllib/correlations.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/mllib/bisecting_k_means_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/mllib/binary_classification_metrics_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/word2vec_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/vector_slicer_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/vector_size_hint_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/vector_indexer_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/vector_assembler_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/variance_threshold_selector_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/univariate_feature_selector_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/train_validation_split.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/tokenizer_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/tf_idf_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/summarizer_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/string_indexer_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/stopwords_remover_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/standard_scaler_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/sql_transformer.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/robust_scaler_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/rformula_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/random_forest_regressor_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/random_forest_classifier_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/quantile_discretizer_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/prefixspan_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/power_iteration_clustering_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/polynomial_expansion_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/pipeline_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/pca_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/onehot_encoder_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/one_vs_rest_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/normalizer_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/naive_bayes_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/n_gram_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/multilayer_perceptron_classification.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/multiclass_logistic_regression_with_elastic_net.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/min_max_scaler_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/min_hash_lsh_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/max_abs_scaler_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/logistic_regression_with_elastic_net.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/logistic_regression_summary_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/linearsvc.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/linear_regression_with_elastic_net.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/lda_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/kmeans_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/isotonic_regression_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/interaction_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/index_to_string_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/imputer_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/gradient_boosted_tree_regressor_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/gradient_boosted_tree_classifier_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/generalized_linear_regression_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/gaussian_mixture_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/fpgrowth_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/fm_regressor_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/fm_classifier_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/feature_hasher_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/estimator_transformer_param_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/elementwise_product_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/decision_tree_regression_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/decision_tree_classification_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/dct_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/dataframe_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/cross_validator.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/count_vectorizer_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/correlation_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/chisq_selector_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/chi_square_test_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/bucketizer_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/bucketed_random_projection_lsh_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/bisecting_k_means_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/binarizer_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/als_example.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/ml/aft_survival_regression.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/logistic_regression.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/kmeans.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/avro_inputformat.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/python/als.py\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scripts/\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/scripts/getGpusResources.sh\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/streaming/\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/streaming/JavaStatefulNetworkWordCount.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/streaming/JavaSqlNetworkWordCount.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/streaming/JavaRecoverableNetworkWordCount.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/streaming/JavaRecord.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/streaming/JavaQueueStream.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/streaming/JavaNetworkWordCount.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/streaming/JavaDirectKerberizedKafkaWordCount.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/streaming/JavaDirectKafkaWordCount.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/streaming/JavaCustomReceiver.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/sql/\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/sql/streaming/\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/sql/streaming/JavaStructuredSessionization.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/sql/streaming/JavaStructuredNetworkWordCountWindowed.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/sql/streaming/JavaStructuredNetworkWordCount.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/sql/streaming/JavaStructuredKerberizedKafkaWordCount.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/sql/streaming/JavaStructuredKafkaWordCount.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/sql/hive/\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/sql/hive/JavaSparkHiveExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/sql/JavaUserDefinedUntypedAggregation.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/sql/JavaUserDefinedTypedAggregation.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/sql/JavaUserDefinedScalar.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/sql/JavaSparkSQLExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/sql/JavaSQLDataSourceExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/mllib/\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/mllib/JavaSummaryStatisticsExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/mllib/JavaStreamingTestExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/mllib/JavaStratifiedSamplingExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/mllib/JavaSimpleFPGrowth.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/mllib/JavaSVMWithSGDExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/mllib/JavaSVDExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/mllib/JavaRecommendationExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/mllib/JavaRankingMetricsExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/mllib/JavaRandomForestRegressionExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/mllib/JavaRandomForestClassificationExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/mllib/JavaPrefixSpanExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/mllib/JavaPowerIterationClusteringExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/mllib/JavaPCAExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/mllib/JavaNaiveBayesExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/mllib/JavaMulticlassClassificationMetricsExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/mllib/JavaMultiLabelClassificationMetricsExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/mllib/JavaLogisticRegressionWithLBFGSExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/mllib/JavaLatentDirichletAllocationExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/mllib/JavaLBFGSExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/mllib/JavaKernelDensityEstimationExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/mllib/JavaKMeansExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/mllib/JavaIsotonicRegressionExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/mllib/JavaHypothesisTestingKolmogorovSmirnovTestExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/mllib/JavaHypothesisTestingExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/mllib/JavaGradientBoostingRegressionExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/mllib/JavaGradientBoostingClassificationExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/mllib/JavaGaussianMixtureExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/mllib/JavaElementwiseProductExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/mllib/JavaDecisionTreeRegressionExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/mllib/JavaDecisionTreeClassificationExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/mllib/JavaCorrelationsExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/mllib/JavaChiSqSelectorExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/mllib/JavaBisectingKMeansExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/mllib/JavaBinaryClassificationMetricsExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/mllib/JavaAssociationRulesExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/mllib/JavaALS.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaWord2VecExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaVectorSlicerExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaVectorSizeHintExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaVectorIndexerExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaVectorAssemblerExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaVarianceThresholdSelectorExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaUnivariateFeatureSelectorExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaTfIdfExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaSummarizerExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaStringIndexerExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaStopWordsRemoverExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaStandardScalerExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaSQLTransformerExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaRobustScalerExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaRandomForestRegressorExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaRandomForestClassifierExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaRFormulaExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaQuantileDiscretizerExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaPrefixSpanExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaPowerIterationClusteringExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaPolynomialExpansionExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaPipelineExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaPCAExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaOneVsRestExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaOneHotEncoderExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaNormalizerExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaNaiveBayesExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaNGramExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaMultilayerPerceptronClassifierExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaMulticlassLogisticRegressionWithElasticNetExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaModelSelectionViaTrainValidationSplitExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaModelSelectionViaCrossValidationExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaMinMaxScalerExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaMinHashLSHExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaMaxAbsScalerExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaLogisticRegressionWithElasticNetExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaLogisticRegressionSummaryExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaLinearSVCExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaLinearRegressionWithElasticNetExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaLabeledDocument.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaLDAExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaKMeansExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaIsotonicRegressionExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaInteractionExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaIndexToStringExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaImputerExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaGradientBoostedTreeRegressorExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaGradientBoostedTreeClassifierExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaGeneralizedLinearRegressionExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaGaussianMixtureExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaFeatureHasherExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaFPGrowthExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaFMRegressorExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaFMClassifierExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaEstimatorTransformerParamExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaElementwiseProductExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaDocument.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaDecisionTreeRegressionExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaDecisionTreeClassificationExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaDCTExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaCountVectorizerExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaCorrelationExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaChiSquareTestExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaChiSqSelectorExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaBucketizerExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaBucketedRandomProjectionLSHExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaBisectingKMeansExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaBinarizerExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaALSExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaAFTSurvivalRegressionExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/ml/JavaTokenizerExample.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/JavaWordCount.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/JavaTC.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/JavaStatusTrackerDemo.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/JavaSparkPi.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/JavaPageRank.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/JavaLogQuery.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/src/main/java/org/apache/spark/examples/JavaHdfsLR.java\n",
      "spark-3.1.2-bin-hadoop3.2/examples/jars/\n",
      "spark-3.1.2-bin-hadoop3.2/examples/jars/spark-examples_2.12-3.1.2.jar\n",
      "spark-3.1.2-bin-hadoop3.2/examples/jars/scopt_2.12-3.7.1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/kubernetes/\n",
      "spark-3.1.2-bin-hadoop3.2/kubernetes/tests/\n",
      "spark-3.1.2-bin-hadoop3.2/kubernetes/tests/worker_memory_check.py\n",
      "spark-3.1.2-bin-hadoop3.2/kubernetes/tests/python_executable_check.py\n",
      "spark-3.1.2-bin-hadoop3.2/kubernetes/tests/pyfiles.py\n",
      "spark-3.1.2-bin-hadoop3.2/kubernetes/tests/py_container_checks.py\n",
      "spark-3.1.2-bin-hadoop3.2/kubernetes/tests/decommissioning_cleanup.py\n",
      "spark-3.1.2-bin-hadoop3.2/kubernetes/tests/decommissioning.py\n",
      "spark-3.1.2-bin-hadoop3.2/kubernetes/tests/autoscale.py\n",
      "spark-3.1.2-bin-hadoop3.2/kubernetes/dockerfiles/\n",
      "spark-3.1.2-bin-hadoop3.2/kubernetes/dockerfiles/spark/\n",
      "spark-3.1.2-bin-hadoop3.2/kubernetes/dockerfiles/spark/bindings/\n",
      "spark-3.1.2-bin-hadoop3.2/kubernetes/dockerfiles/spark/bindings/python/\n",
      "spark-3.1.2-bin-hadoop3.2/kubernetes/dockerfiles/spark/bindings/python/Dockerfile\n",
      "spark-3.1.2-bin-hadoop3.2/kubernetes/dockerfiles/spark/bindings/R/\n",
      "spark-3.1.2-bin-hadoop3.2/kubernetes/dockerfiles/spark/bindings/R/Dockerfile\n",
      "spark-3.1.2-bin-hadoop3.2/kubernetes/dockerfiles/spark/Dockerfile\n",
      "spark-3.1.2-bin-hadoop3.2/kubernetes/dockerfiles/spark/entrypoint.sh\n",
      "spark-3.1.2-bin-hadoop3.2/kubernetes/dockerfiles/spark/decom.sh\n",
      "spark-3.1.2-bin-hadoop3.2/yarn/\n",
      "spark-3.1.2-bin-hadoop3.2/yarn/spark-3.1.2-yarn-shuffle.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/\n",
      "spark-3.1.2-bin-hadoop3.2/jars/zstd-jni-1.4.8-1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/zookeeper-3.4.14.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/zjsonpatch-0.3.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/xz-1.5.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/xbean-asm7-shaded-4.15.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/woodstox-core-5.0.3.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/velocity-1.5.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/univocity-parsers-2.9.1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/transaction-api-1.1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/token-provider-1.0.1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/threeten-extra-1.5.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/super-csv-2.2.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/stream-2.9.6.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/stax2-api-3.1.4.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/stax-api-1.0.1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/spire_2.12-0.17.0-M1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/spire-util_2.12-0.17.0-M1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/spire-platform_2.12-0.17.0-M1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/spire-macros_2.12-0.17.0-M1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/spark-yarn_2.12-3.1.2.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/spark-unsafe_2.12-3.1.2.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/spark-tags_2.12-3.1.2.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/spark-tags_2.12-3.1.2-tests.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/spark-streaming_2.12-3.1.2.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/spark-sql_2.12-3.1.2.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/spark-sketch_2.12-3.1.2.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/spark-repl_2.12-3.1.2.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/spark-network-shuffle_2.12-3.1.2.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/spark-network-common_2.12-3.1.2.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/spark-mllib_2.12-3.1.2.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/spark-mllib-local_2.12-3.1.2.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/spark-mesos_2.12-3.1.2.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/spark-launcher_2.12-3.1.2.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/spark-kvstore_2.12-3.1.2.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/spark-kubernetes_2.12-3.1.2.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/spark-hive_2.12-3.1.2.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/spark-hive-thriftserver_2.12-3.1.2.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/spark-graphx_2.12-3.1.2.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/spark-core_2.12-3.1.2.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/spark-catalyst_2.12-3.1.2.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/snappy-java-1.1.8.2.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/snakeyaml-1.24.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/slf4j-log4j12-1.7.30.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/slf4j-api-1.7.30.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/shims-0.9.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/shapeless_2.12-2.3.3.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/scala-xml_2.12-1.2.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/scala-reflect-2.12.10.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/scala-parser-combinators_2.12-1.1.2.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/scala-library-2.12.10.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/scala-compiler-2.12.10.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/scala-collection-compat_2.12-2.1.1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/re2j-1.1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/pyrolite-4.30.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/py4j-0.10.9.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/protobuf-java-2.5.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/parquet-jackson-1.10.1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/parquet-hadoop-1.10.1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/parquet-format-2.4.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/parquet-encoding-1.10.1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/parquet-common-1.10.1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/parquet-column-1.10.1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/paranamer-2.8.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/osgi-resource-locator-1.0.3.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/oro-2.0.8.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/orc-shims-1.5.12.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/orc-mapreduce-1.5.12.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/orc-core-1.5.12.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/opencsv-2.3.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/okio-1.14.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/okhttp-3.12.12.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/okhttp-2.7.5.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/objenesis-2.6.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/nimbus-jose-jwt-4.41.1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/netty-all-4.1.51.Final.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/minlog-1.3.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/metrics-jvm-4.1.1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/metrics-json-4.1.1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/metrics-jmx-4.1.1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/metrics-graphite-4.1.1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/metrics-core-4.1.1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/mesos-1.4.0-shaded-protobuf.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/macro-compat_2.12-1.1.1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/machinist_2.12-0.6.8.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/lz4-java-1.7.1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/logging-interceptor-3.12.12.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/log4j-1.2.17.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/libthrift-0.12.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/libfb303-0.9.3.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/leveldbjni-all-1.8.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/kubernetes-model-storageclass-4.12.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/kubernetes-model-settings-4.12.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/kubernetes-model-scheduling-4.12.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/kubernetes-model-rbac-4.12.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/kubernetes-model-policy-4.12.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/kubernetes-model-networking-4.12.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/kubernetes-model-metrics-4.12.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/kubernetes-model-extensions-4.12.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/kubernetes-model-events-4.12.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/kubernetes-model-discovery-4.12.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/kubernetes-model-core-4.12.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/kubernetes-model-coordination-4.12.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/kubernetes-model-common-4.12.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/kubernetes-model-certificates-4.12.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/kubernetes-model-batch-4.12.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/kubernetes-model-autoscaling-4.12.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/kubernetes-model-apps-4.12.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/kubernetes-model-apiextensions-4.12.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/kubernetes-model-admissionregistration-4.12.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/kubernetes-client-4.12.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/kryo-shaded-4.0.2.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/kerby-xdr-1.0.1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/kerby-util-1.0.1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/kerby-pkix-1.0.1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/kerby-config-1.0.1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/kerby-asn1-1.0.1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/kerb-util-1.0.1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/kerb-simplekdc-1.0.1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/kerb-server-1.0.1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/kerb-identity-1.0.1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/kerb-crypto-1.0.1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/kerb-core-1.0.1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/kerb-common-1.0.1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/kerb-client-1.0.1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/kerb-admin-1.0.1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/jul-to-slf4j-1.7.30.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/jta-1.1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/jsr305-3.0.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/jsp-api-2.1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/json4s-scalap_2.12-3.7.0-M5.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/json4s-jackson_2.12-3.7.0-M5.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/json4s-core_2.12-3.7.0-M5.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/json4s-ast_2.12-3.7.0-M5.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/json-smart-2.3.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/json-1.8.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/jpam-1.1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/jodd-core-3.5.2.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/joda-time-2.10.5.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/jline-2.14.6.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/jersey-server-2.30.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/jersey-media-jaxb-2.30.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/jersey-hk2-2.30.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/jersey-container-servlet-core-2.30.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/jersey-container-servlet-2.30.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/jersey-common-2.30.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/jersey-client-2.30.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/jdo-api-3.0.1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/jcl-over-slf4j-1.7.30.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/jcip-annotations-1.0-1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/jaxb-runtime-2.3.2.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/jaxb-api-2.2.11.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/javolution-5.5.1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/javax.jdo-3.2.0-m3.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/javax.inject-1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/javassist-3.25.0-GA.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/janino-3.0.16.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/jakarta.xml.bind-api-2.3.2.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/jakarta.ws.rs-api-2.1.6.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/jakarta.validation-api-2.0.2.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/jakarta.servlet-api-4.0.3.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/jakarta.inject-2.6.1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/jakarta.annotation-api-1.3.5.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/jakarta.activation-api-1.2.1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/jackson-module-scala_2.12-2.10.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/jackson-module-paranamer-2.10.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/jackson-module-jaxb-annotations-2.10.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/jackson-mapper-asl-1.9.13.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/jackson-jaxrs-json-provider-2.9.5.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/jackson-jaxrs-base-2.9.5.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/jackson-datatype-jsr310-2.11.2.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/jackson-dataformat-yaml-2.10.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/jackson-databind-2.10.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/jackson-core-asl-1.9.13.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/jackson-core-2.10.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/jackson-annotations-2.10.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/ivy-2.4.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/istack-commons-runtime-3.0.8.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/httpcore-4.4.12.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/httpclient-4.5.6.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/htrace-core4-4.1.0-incubating.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/hk2-utils-2.6.1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/hk2-locator-2.6.1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/hk2-api-2.6.1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/hive-vector-code-gen-2.3.7.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/hive-storage-api-2.7.2.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/hive-shims-scheduler-2.3.7.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/hive-shims-common-2.3.7.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/hive-shims-2.3.7.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/hive-shims-0.23-2.3.7.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/hive-service-rpc-3.1.2.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/hive-serde-2.3.7.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/hive-metastore-2.3.7.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/hive-llap-common-2.3.7.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/hive-jdbc-2.3.7.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/hive-exec-2.3.7-core.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/hive-common-2.3.7.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/hive-cli-2.3.7.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/hive-beeline-2.3.7.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/hadoop-yarn-server-web-proxy-3.2.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/hadoop-yarn-server-common-3.2.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/hadoop-yarn-registry-3.2.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/hadoop-yarn-common-3.2.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/hadoop-yarn-client-3.2.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/hadoop-yarn-api-3.2.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/hadoop-mapreduce-client-jobclient-3.2.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/hadoop-mapreduce-client-core-3.2.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/hadoop-mapreduce-client-common-3.2.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/hadoop-hdfs-client-3.2.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/hadoop-common-3.2.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/hadoop-client-3.2.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/hadoop-auth-3.2.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/hadoop-annotations-3.2.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/guice-servlet-4.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/guice-4.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/guava-14.0.1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/gson-2.2.4.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/geronimo-jcache_1.0_spec-1.0-alpha-1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/generex-1.0.2.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/flatbuffers-java-1.9.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/ehcache-3.3.1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/dnsjava-2.1.7.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/derby-10.12.1.1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/datanucleus-rdbms-4.1.19.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/datanucleus-core-4.1.17.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/datanucleus-api-jdo-4.2.4.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/curator-recipes-2.13.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/curator-framework-2.13.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/curator-client-2.13.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/core-1.1.2.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/compress-lzf-1.0.3.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/commons-text-1.6.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/commons-pool-1.5.4.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/commons-net-3.1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/commons-math3-3.4.1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/commons-logging-1.1.3.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/commons-lang3-3.10.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/commons-lang-2.6.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/commons-io-2.5.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/commons-httpclient-3.1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/commons-dbcp-1.4.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/commons-daemon-1.0.13.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/commons-crypto-1.1.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/commons-configuration2-2.1.1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/commons-compress-1.20.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/commons-compiler-3.0.16.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/commons-collections-3.2.2.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/commons-codec-1.10.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/commons-cli-1.2.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/commons-beanutils-1.9.4.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/chill_2.12-0.9.5.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/chill-java-0.9.5.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/cats-kernel_2.12-2.0.0-M4.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/breeze_2.12-1.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/breeze-macros_2.12-1.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/bonecp-0.8.0.RELEASE.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/avro-mapred-1.8.2-hadoop2.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/avro-ipc-1.8.2.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/avro-1.8.2.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/automaton-1.11-8.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/audience-annotations-0.5.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/arrow-vector-2.0.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/arrow-memory-netty-2.0.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/arrow-memory-core-2.0.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/arrow-format-2.0.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/arpack_combined_all-0.1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/aopalliance-repackaged-2.6.1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/aopalliance-1.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/antlr4-runtime-4.8-1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/antlr-runtime-3.5.2.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/algebra_2.12-2.0.0-M2.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/aircompressor-0.10.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/activation-1.1.1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/accessors-smart-1.2.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/ST4-4.0.4.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/RoaringBitmap-0.9.0.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/JTransforms-3.1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/JLargeArrays-1.5.jar\n",
      "spark-3.1.2-bin-hadoop3.2/jars/HikariCP-2.5.1.jar\n",
      "spark-3.1.2-bin-hadoop3.2/RELEASE\n"
     ]
    }
   ],
   "source": [
    "!wget -q -nc https://mirror.navercorp.com/apache/spark/spark-3.1.2/spark-3.1.2-bin-hadoop3.2.tgz\n",
    "!tar -xvf spark-3.1.2-bin-hadoop3.2.tgz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "v32a9GGiPGT8"
   },
   "outputs": [],
   "source": [
    "os.environ[\"JAVA_HOME\"] = \"/usr/lib/jvm/java-8-openjdk-amd64\"\n",
    "os.environ[\"SPARK_HOME\"] = \"/content/spark-3.1.2-bin-hadoop3.2\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "pa44iu_CxPjS",
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "outputId": "e2e6c32c-7caa-4c01-ba0d-c7bd3391c5da"
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Collecting findspark\n",
      "  Downloading findspark-2.0.1-py2.py3-none-any.whl (4.4 kB)\n",
      "Installing collected packages: findspark\n",
      "Successfully installed findspark-2.0.1\n"
     ]
    }
   ],
   "source": [
    "!pip install findspark"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "id": "u9RJepGQP--0"
   },
   "outputs": [],
   "source": [
    "import findspark\n",
    "findspark.init()"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "mounts google drive for the usage of cloud storage"
   ],
   "metadata": {
    "id": "lMKasmAyXYCJ"
   }
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "id": "LXOq9YpBRuwl",
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "outputId": "d754e67e-05b5-40b3-eb87-b20c7245131c"
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Mounted at /content/drive\n"
     ]
    }
   ],
   "source": [
    "from google.colab import drive\n",
    "drive.mount('/content/drive')"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "installs gRPC library in Python"
   ],
   "metadata": {
    "id": "oDq6zJaQXokS"
   }
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "id": "HGeRJVofRGz2",
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "outputId": "8140e695-2d9a-457c-a740-1515d1eeb599"
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Requirement already satisfied: grpcio in /usr/local/lib/python3.7/dist-packages (1.44.0)\n",
      "Requirement already satisfied: six>=1.5.2 in /usr/local/lib/python3.7/dist-packages (from grpcio) (1.15.0)\n"
     ]
    }
   ],
   "source": [
    "!pip install grpcio"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "setting up libraries"
   ],
   "metadata": {
    "id": "olXuBTd9XvVr"
   }
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "id": "GkPL5u-dQCWi"
   },
   "outputs": [],
   "source": [
    "import pyspark\n",
    "import pandas\n",
    "import pyspark\n",
    "from pyspark.sql.functions import *\n",
    "from pyspark.sql.window import Window\n",
    "from pyspark.sql.types import *\n",
    "from pyspark.sql import *\n",
    "from tqdm import tqdm\n",
    "import ast\n",
    "import sys\n",
    "import os\n",
    "import time\n",
    "from datetime import datetime\n",
    "from google.protobuf.timestamp_pb2 import Timestamp\n",
    "import csv\n",
    "import pandas as pd\n",
    "# from dependencies.spark import start_spark\n",
    "from pyspark.sql.functions import pandas_udf, PandasUDFType\n",
    "# If grpc is missing: pip install grpcio\n",
    "import grpc\n",
    "from google.protobuf import empty_pb2\n",
    "# If the classes below are missing, generate them:\n",
    "# You need to install the grpcio-tools to generate the stubs: pip install grpcio-tools\n",
    "# python -m grpc_tools.protoc -I . --python_out=. --grpc_python_out=. challenger.proto"
   ]
  },
  {
   "cell_type": "code",
   "source": [
    "!wget -q -nc https://raw.githubusercontent.com//developsu/debs2022gc/main/challenger_pb2.py\n",
    "!wget -q -nc https://raw.githubusercontent.com//developsu/debs2022gc/main/challenger_pb2_grpc.py"
   ],
   "metadata": {
    "id": "DDMkQ-diiYSX",
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "outputId": "0ff8f2e7-7a13-4baf-a76b-e2fc185d22ba"
   },
   "execution_count": 11,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "/usr/bin/python3: Error while finding module specification for 'grpc_tools.protoc' (ModuleNotFoundError: No module named 'grpc_tools')\n"
     ]
    }
   ]
  },
  {
   "cell_type": "code",
   "source": [
    "import challenger_pb2 as ch\n",
    "import challenger_pb2_grpc as api"
   ],
   "metadata": {
    "id": "axvhJh2WiZZn"
   },
   "execution_count": 12,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zahs-5p8VBjN"
   },
   "source": [
    "## Core functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gHNn3_QUVJ7m"
   },
   "source": [
    "This function reads the stream of csv files from remote benchmark system"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "id": "4NQhqRIpU_92"
   },
   "outputs": [],
   "source": [
    "\"\"\"## Core functions\n",
    "This function reads the stream of csv files from remote benchmark system\n",
    "\"\"\"\n",
    "def read_stream(folderDir):\n",
    "    \"\"\"\n",
    "    reads Streaming data and sets up the schema \n",
    "    :param spark session\n",
    "    :return: data stream\n",
    "    \"\"\"\n",
    "    global spark\n",
    "    stream_df = spark.readStream.format(\"csv\") \\\n",
    "        .option(\"header\", \"true\") \\\n",
    "        .schema(get_schema()) \\\n",
    "        .load(path=folderDir)\n",
    "    \n",
    "    return stream_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "d1Hs2aqlZhSR"
   },
   "source": [
    "This function represents the schema of stock market data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "id": "EZ81h7egZoea"
   },
   "outputs": [],
   "source": [
    "\"\"\"This function represents the schema of stock market data\"\"\"\n",
    "def get_schema():\n",
    "    \"\"\"\n",
    "    return schema for DEBS 2022\n",
    "    :return: StructType\n",
    "    \"\"\"\n",
    "    return StructType([\n",
    "        StructField(\"id\", LongType(), True),\n",
    "        StructField(\"symbol\", StringType(), True),\n",
    "        StructField(\"Sectype\", StringType(), True),\n",
    "        StructField(\"lasttradeprice\", DoubleType(), True),\n",
    "        StructField(\"lastTrade\", TimestampType(), True),\n",
    "        StructField(\"seconds\", LongType(), True),\n",
    "        StructField(\"nanos\", LongType(), True)\n",
    "    ])"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Calc EMA"
   ],
   "metadata": {
    "id": "_jMphIvgL9hq"
   }
  },
  {
   "cell_type": "code",
   "source": [
    "\"\"\"The following functions calculate EMA38 and EMA100 indicators, respectively.\"\"\"\n",
    "def calcEMA_func(symbol, lastTradePrice, prev_ema, days):\n",
    "  current_EMA = ( lastTradePrice * (2 / (days + 1)) ) + ( prev_ema  * (1 - 2 / (days + 1)) )\n",
    "  return current_EMA"
   ],
   "metadata": {
    "id": "zAj9u62oIRFa"
   },
   "execution_count": 15,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "### EMA_Comp"
   ],
   "metadata": {
    "id": "N2jgMDsxFe7j"
   }
  },
  {
   "cell_type": "code",
   "source": [
    "\"\"\"This function determines the choice of buy, sell, or stay(=hold).\"\"\"\n",
    "def EMA_Comp(nEMA38_MINUS_EMA100, pEMA38_MINUS_EMA100):\n",
    "    minus_prev = pEMA38_MINUS_EMA100\n",
    "    minus_now = nEMA38_MINUS_EMA100\n",
    "    if minus_now * minus_prev >= 0:\n",
    "      return \"Stay\"\n",
    "    else:\n",
    "      if minus_now > minus_prev:\n",
    "          return \"Buy\"\n",
    "\n",
    "    return \"Sell\""
   ],
   "metadata": {
    "id": "NxHE3GnHEsD9"
   },
   "execution_count": 16,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DySgqMITVlYP"
   },
   "source": [
    "## transformation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "id": "UuIGVy41VpQe"
   },
   "outputs": [],
   "source": [
    "def transform_calcEMA(stream):\n",
    "    # global spark, prevEMA\n",
    "    \n",
    "    stream_df = stream.select(\"symbol\", \"Sectype\", \"lastTrade\", \"lasttradeprice\", \"seconds\", \"nanos\").withWatermark(\"lastTrade\", \"5 minutes\") \\\n",
    "        .groupBy(window(\"lastTrade\", \"5 minutes\", \"5 minutes\"), col(\"symbol\")) \\\n",
    "        .agg(lit(last(\"lasttradeprice\")).alias(\"lasttradeprice\"), lit(last(\"Sectype\")).alias(\"Sectype\"), lit(last(\"seconds\")).alias(\"seconds\"), lit(last(\"nanos\")).alias(\"nanos\"))\\\n",
    "        .orderBy(\"window\")\n",
    "\n",
    "    stream_df = stream_df.withColumn(\"lastWindowTime\", to_timestamp(stream_df.window.end, \"yyyy-MM-dd HH:mm:ss\"))\n",
    "    \n",
    "    return stream_df"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "## results of Query 1"
   ],
   "metadata": {
    "id": "kdLo1NurZ89e"
   }
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "id": "CL-2dZK7V67Q"
   },
   "outputs": [],
   "source": [
    "def send_to_message_query1(lookup_symbols):\n",
    "  global prevEMA\n",
    "  result_Q1 = ch.ResultQ1()\n",
    "  list_indicators = list()\n",
    "\n",
    "  # serialize\n",
    "  for s in lookup_symbols: # search for symbols required\n",
    "    if s in prevEMA.index: # if prevEMA tables has information about previous EMAs.\n",
    "        row = prevEMA.loc[s]\n",
    "        indicator = ch.Indicator(symbol = row['symbol'], \\\n",
    "                                ema_38 = row['EMA38'], \\\n",
    "                                ema_100 =  row['EMA100'] )\n",
    "        \n",
    "    else: # else , send with 0s\n",
    "        indicator = ch.Indicator(symbol = s, \\\n",
    "                                ema_38 = 0, \\\n",
    "                                ema_100 =  0 )\n",
    "    list_indicators.append(indicator)\n",
    "  \n",
    "  result_Q1.indicators.extend(list_indicators)\n",
    "  \n",
    "  return result_Q1.indicators"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "x_3itZYkWFmG"
   },
   "source": [
    "## results of Query 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "id": "eBhyD9dsWFsq"
   },
   "outputs": [],
   "source": [
    "def send_to_message_query2(lookup_symbols):\n",
    "    global crossover\n",
    "    crossover = crossover[crossover['BuyOrSell'] != 'Stay']\n",
    "    tail_crossover = crossover.groupby(['symbol']).tail(3)\n",
    "    tail_crossover =tail_crossover.reset_index(drop=True)\n",
    "    \n",
    "    result_Q2 = ch.ResultQ2()\n",
    "    list_crossover = list()\n",
    "\n",
    "    # serialize\n",
    "    for i in range(len(tail_crossover)): # search for symbols required\n",
    "      cross_over_event = ch.CrossoverEvent()\n",
    "      row = tail_crossover.loc[i,:]\n",
    "      cross_over_event.symbol = row[\"symbol\"]\n",
    "\n",
    "      if row[\"BuyOrSell\"] == \"Buy\":\n",
    "        cross_over_event.signal_type = ch.CrossoverEvent.SignalType.Buy\n",
    "      elif row[\"BuyOrSell\"] == \"Sell\":\n",
    "        cross_over_event.signal_type = ch.CrossoverEvent.SignalType.Sell\n",
    "\n",
    "      if row[\"Sectype\"] == \"E\" or row[\"Sectype\"] == \"0\":\n",
    "        cross_over_event.security_type = ch.SecurityType.Equity\n",
    "      else:\n",
    "        cross_over_event.security_type = ch.SecurityType.Index\n",
    "      \n",
    "      cross_over_event.ts.seconds = row['seconds']\n",
    "      cross_over_event.ts.nanos = row['nanos']\n",
    "      list_crossover.append(cross_over_event)\n",
    "    \n",
    "    result_Q2.crossover_events.extend(list_crossover)\n",
    "    return result_Q2.crossover_events"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "p7Vv86C2WOc9"
   },
   "source": [
    "## Streamed file processing\n",
    "This funtions saves the stream of stock market data as csv files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "id": "cw5Uj-_NWOji"
   },
   "outputs": [],
   "source": [
    "\"\"\"## Streamed file processing\n",
    "This funtions saves the stream of stock market data as csv files.\n",
    "\"\"\"\n",
    "def make_CSV_file(batch_no, events):\n",
    "    fieldnames = [\"id\", \"symbol\", \"Sectype\", \"lasttradeprice\", \"lastTrade\", \"seconds\", \"nanos\"]\n",
    "    event_list_dict = []\n",
    "    for i in range(len(events)):\n",
    "            temp_dict = {}\n",
    "            temp_dict['id'] = i\n",
    "            temp_dict['symbol'] = events[i].symbol\n",
    "            temp_dict['Sectype'] = events[i].security_type\n",
    "            temp_dict['lasttradeprice'] = events[i].last_trade_price\n",
    "            temp_dict['lastTrade'] = datetime.fromtimestamp(events[i].last_trade.seconds).strftime(\"%Y-%m-%d %H:%M:%S\")\n",
    "            temp_dict['seconds'] = events[i].last_trade.seconds\n",
    "            temp_dict['nanos'] = events[i].last_trade.nanos\n",
    "            event_list_dict.append(temp_dict)\n",
    "\n",
    "    with open(folderDir + f'/stock_stream-{batch_no}.csv', 'w+', encoding='UTF8', newline='') as f:\n",
    "        writer = csv.DictWriter(f, fieldnames=fieldnames)\n",
    "        writer.writeheader()\n",
    "        writer.writerows(event_list_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Main Program"
   ],
   "metadata": {
    "id": "PmGAhnWybZll"
   }
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "EZ5C2u3Y40wa"
   },
   "source": [
    "setup global variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "id": "-da9ryaZ3zbH"
   },
   "outputs": [],
   "source": [
    "global spark, sc\n",
    "global calcEMA, detectCrossover\n",
    "global prevEMA\n",
    "global prevCrossover, crossover"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "B7UjOywr44W3"
   },
   "source": [
    "### initialize spark"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "id": "tuqsSJge3Z6Q"
   },
   "outputs": [],
   "source": [
    "# start Spark application and get Spark session, logger and config\n",
    "spark = SparkSession.builder.master(\"local[*]\") \\\n",
    "        .config(\"spark.executor.instances\", \"2\") \\\n",
    "        .config(\"spark.executor.cores\", \"2\") \\\n",
    "        .config('spark.ui.port', '4050') \\\n",
    "        .config(\"spark.driver.memory\", \"20g\")\\\n",
    "        .config(\"spark.executor.memory\", \"20g\")\\\n",
    "        .config(\"spark.dynamicAllocation.enabled\", \"true\") \\\n",
    "        .appName(\"Spark for DEBS 2020 GC\").getOrCreate()\n",
    "spark.conf.set(\"spark.sql.execution.arrow.enabled\", \"true\")\n",
    "spark.conf.set(\"spark.sql.execution.arrow.pyspark.enabled\", \"false\")\n",
    "\n",
    "sc = spark.sparkContext"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rpO7Uo5p6Val"
   },
   "source": [
    "### Define streaming input sources"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "id": "nQzk_Qz_3hIu"
   },
   "outputs": [],
   "source": [
    "# execute ETL pipeline\n",
    "stream = read_stream(folderDir)"
   ]
  },
  {
   "cell_type": "code",
   "source": [
    "prevEMA = pd.DataFrame({'symbol': pd.Series(dtype='str'),\n",
    "                   'EMA38': pd.Series(dtype='float'),\n",
    "                   'EMA100': pd.Series(dtype='float')})\n",
    "\n",
    "prevEMA.set_index('symbol')"
   ],
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 81
    },
    "id": "9wekN3fppUf8",
    "outputId": "1a6c4ab9-ea5a-4472-f5a4-1f2a36a43824"
   },
   "execution_count": 24,
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [EMA38, EMA100]\n",
       "Index: []"
      ],
      "text/html": [
       "\n",
       "  <div id=\"df-6ecc6f94-38de-42f9-b42c-fa9c49709851\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>EMA38</th>\n",
       "      <th>EMA100</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>symbol</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-6ecc6f94-38de-42f9-b42c-fa9c49709851')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-6ecc6f94-38de-42f9-b42c-fa9c49709851 button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-6ecc6f94-38de-42f9-b42c-fa9c49709851');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ]
     },
     "metadata": {},
     "execution_count": 24
    }
   ]
  },
  {
   "cell_type": "code",
   "source": [
    "prevEMA = prevEMA.iloc[0:0]\n",
    "prevEMA"
   ],
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 49
    },
    "id": "XI1Cxp3uuJ8W",
    "outputId": "bc382ead-8742-43cc-e550-db9335760870"
   },
   "execution_count": 25,
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [symbol, EMA38, EMA100]\n",
       "Index: []"
      ],
      "text/html": [
       "\n",
       "  <div id=\"df-0daec49b-68de-4a0f-93d4-312e39a948ee\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>symbol</th>\n",
       "      <th>EMA38</th>\n",
       "      <th>EMA100</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-0daec49b-68de-4a0f-93d4-312e39a948ee')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-0daec49b-68de-4a0f-93d4-312e39a948ee button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-0daec49b-68de-4a0f-93d4-312e39a948ee');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ]
     },
     "metadata": {},
     "execution_count": 25
    }
   ]
  },
  {
   "cell_type": "code",
   "source": [
    "crossover = pd.DataFrame({'symbol': pd.Series(dtype='str'),\n",
    "                   'lastWindowTime': pd.Series(dtype='timedelta64[ns]'),\n",
    "                   'EMA38': pd.Series(dtype='float'),\n",
    "                   'EMA100': pd.Series(dtype='float'),\n",
    "                   'Sectype': pd.Series(dtype='str'),\n",
    "                   'BuyOrSell': pd.Series(dtype='str'),\n",
    "                   'seconds': pd.Series(dtype='int'),\n",
    "                   'nanos': pd.Series(dtype='int')})\n"
   ],
   "metadata": {
    "id": "-7RNVWOqA22B"
   },
   "execution_count": 26,
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": [
    "prevCrossover = pd.DataFrame({'symbol': pd.Series(dtype='str'),\n",
    "                   'Diff': pd.Series(dtype='float')})\n",
    "\n",
    "prevCrossover.set_index('symbol')"
   ],
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 81
    },
    "id": "rsT1jGoktJew",
    "outputId": "cd896c07-9f71-44a4-aa7a-f4336c3ffc49"
   },
   "execution_count": 27,
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [Diff]\n",
       "Index: []"
      ],
      "text/html": [
       "\n",
       "  <div id=\"df-1949fd03-5aec-4f69-864c-a837cf78b8f2\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Diff</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>symbol</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-1949fd03-5aec-4f69-864c-a837cf78b8f2')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-1949fd03-5aec-4f69-864c-a837cf78b8f2 button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-1949fd03-5aec-4f69-864c-a837cf78b8f2');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ]
     },
     "metadata": {},
     "execution_count": 27
    }
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "GolCKziT4Gs5"
   },
   "source": [
    "### 2. Transform streaming data and define output sink&mode\n",
    "queries 1 and 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "id": "ANL8koMg3rrc"
   },
   "outputs": [],
   "source": [
    "calcEMA = transform_calcEMA(stream)\n",
    "\n",
    "writer1 =  calcEMA.writeStream.format(\"memory\") \\\n",
    "    .outputMode(\"complete\") \\\n",
    "    .option(\"truncate\", False) \\\n",
    "    .queryName(\"EMA\") # table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "AH-eDO9Yh1vb"
   },
   "source": [
    "### Start the queries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "id": "Oc6JTwrUQM2E"
   },
   "outputs": [],
   "source": [
    "streamingQuery1 = writer1.start()"
   ]
  },
  {
   "cell_type": "code",
   "source": [
    "def execute_streamingQuery1(df_a):\n",
    "  global prevEMA, crossover\n",
    "  for idx, row in tqdm(df_a.iterrows(), total = df_a.shape[0], disable=True):\n",
    "    symbol = row['symbol']\n",
    "    if symbol in prevEMA.index:\n",
    "      existing_row = prevEMA.loc[symbol]\n",
    "      new_EMA38 = calcEMA_func(symbol, row['lasttradeprice'], existing_row['EMA38'], 38 )\n",
    "      new_EMA100 = calcEMA_func(symbol, row['lasttradeprice'], existing_row['EMA100'], 100 )\n",
    "      \n",
    "      prevEMA.loc[symbol] = { 'symbol': symbol, 'EMA38': new_EMA38, 'EMA100': new_EMA100}\n",
    "\n",
    "      new_row = { 'symbol': symbol, 'lastWindowTime': row['lastWindowTime'],'EMA38': new_EMA38, 'EMA100': new_EMA100, 'Sectype': row['Sectype'], 'BuyOrSell' : 'Stay',  'seconds': row['seconds'], 'nanos': row['nanos'] }\n",
    "      crossover = crossover.append(pd.Series(new_row, index=crossover.columns, name=symbol))\n",
    "      \n",
    "    else: \n",
    "      new_EMA38 = calcEMA_func(symbol, row['lasttradeprice'], 0.0, 38 )\n",
    "      new_EMA100 = calcEMA_func(symbol, row['lasttradeprice'], 0.0, 100 )\n",
    "      new_row = { 'symbol': symbol, 'lastWindowTime': row['lastWindowTime'],'EMA38': new_EMA38, 'EMA100': new_EMA100, 'Sectype': row['Sectype'], 'BuyOrSell' : 'Stay', 'seconds': row['seconds'], 'nanos': row['nanos'] }\n",
    "      crossover = crossover.append(pd.Series(new_row, index=crossover.columns, name=symbol))\n",
    "\n",
    "      new_row = { 'symbol': symbol, 'EMA38': new_EMA38, 'EMA100': new_EMA100}\n",
    "      prevEMA = prevEMA.append(pd.Series(new_row, index=prevEMA.columns, name=symbol))\n",
    "      "
   ],
   "metadata": {
    "id": "TMGozrn3iybv"
   },
   "execution_count": 31,
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": [
    "def execute_streamingQuery2():\n",
    "  global crossover, prevCrossover\n",
    "  for idx, row in tqdm(crossover.iterrows(), total = crossover.shape[0], disable=True):\n",
    "    symbol = row['symbol']\n",
    "    if symbol not in prevCrossover.index:\n",
    "      new_row = { 'symbol': symbol, 'Diff': row['EMA38'] - row['EMA100']}\n",
    "      prevCrossover = prevCrossover.append(pd.Series(new_row, index=prevCrossover.columns, name=symbol))\n",
    "    \n",
    "    existing_row = prevCrossover.loc[symbol]\n",
    "    BuyOrSell = EMA_Comp(row['EMA38']-row[\"EMA100\"], existing_row['Diff'])\n",
    "    cond = (crossover.symbol == symbol) & (crossover.lastWindowTime == row['lastWindowTime'])\n",
    "    crossover.loc[cond, ('BuyOrSell')] = BuyOrSell\n",
    "    prevCrossover.loc[symbol] = { 'symbol': symbol, 'Diff':row['EMA38']-row[\"EMA100\"]}\n",
    "   "
   ],
   "metadata": {
    "id": "Dw2W0DzVtcqy"
   },
   "execution_count": 32,
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": [
    "def reset_dataframe(lookup_symbols):\n",
    "  global prevEMA, crossover, prevCrossover\n",
    "  for s in range(len(lookup_symbols)):\n",
    "    spec = prevEMA['symbol'].str.contains(lookup_symbols[s]).index\n",
    "    if lookup_symbols[s] in prevEMA.index:\n",
    "      prevEMA.drop(spec,inplace = True)\n",
    "\n",
    "    spec = crossover['symbol'].str.contains(lookup_symbols[s]).index\n",
    "    if lookup_symbols[s] in crossover.index:\n",
    "      crossover.drop(spec,inplace = True)\n",
    "      \n",
    "    spec = prevCrossover['symbol'].str.contains(lookup_symbols[s]).index\n",
    "    if lookup_symbols[s] in prevCrossover.index:\n",
    "      prevCrossover.drop(spec,inplace = True)"
   ],
   "metadata": {
    "id": "daYopwblQp9c"
   },
   "execution_count": 33,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gE2mhwMu4caH"
   },
   "source": [
    "## Challenger system"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "id": "xADk23ta4Usa",
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 835
    },
    "outputId": "0ba0073b-6838-427b-ee3e-286d14353de3"
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "batch seq_id 0\n",
      "batch seq_id 1\n",
      "batch seq_id 2\n",
      "batch seq_id 3\n",
      "batch seq_id 4\n",
      "batch seq_id 5\n",
      "batch seq_id 6\n",
      "batch seq_id 7\n",
      "batch seq_id 8\n",
      "batch seq_id 9\n",
      "batch seq_id 10\n",
      "batch seq_id 11\n",
      "batch seq_id 12\n",
      "batch seq_id 13\n",
      "batch seq_id 14\n",
      "batch seq_id 15\n",
      "batch seq_id 16\n",
      "batch seq_id 17\n",
      "batch seq_id 18\n",
      "batch seq_id 19\n",
      "batch seq_id 20\n",
      "batch seq_id 21\n",
      "batch seq_id 22\n"
     ]
    },
    {
     "output_type": "error",
     "ename": "KeyboardInterrupt",
     "evalue": "ignored",
     "traceback": [
      "\u001B[0;31m---------------------------------------------------------------------------\u001B[0m",
      "\u001B[0;31mKeyboardInterrupt\u001B[0m                         Traceback (most recent call last)",
      "\u001B[0;32m<ipython-input-34-85604c29770e>\u001B[0m in \u001B[0;36m<module>\u001B[0;34m()\u001B[0m\n\u001B[1;32m     33\u001B[0m         \u001B[0mlastTrades\u001B[0m \u001B[0;34m=\u001B[0m \u001B[0mspark\u001B[0m\u001B[0;34m.\u001B[0m\u001B[0msql\u001B[0m\u001B[0;34m(\u001B[0m\u001B[0;34m\"select * from EMA\"\u001B[0m\u001B[0;34m)\u001B[0m\u001B[0;34m.\u001B[0m\u001B[0mfilter\u001B[0m\u001B[0;34m(\u001B[0m\u001B[0mcol\u001B[0m\u001B[0;34m(\u001B[0m\u001B[0;34m'symbol'\u001B[0m\u001B[0;34m)\u001B[0m\u001B[0;34m.\u001B[0m\u001B[0misin\u001B[0m\u001B[0;34m(\u001B[0m\u001B[0mlookup_symbols\u001B[0m\u001B[0;34m)\u001B[0m\u001B[0;34m)\u001B[0m\u001B[0;34m.\u001B[0m\u001B[0mtoPandas\u001B[0m\u001B[0;34m(\u001B[0m\u001B[0;34m)\u001B[0m\u001B[0;34m\u001B[0m\u001B[0;34m\u001B[0m\u001B[0m\n\u001B[1;32m     34\u001B[0m \u001B[0;34m\u001B[0m\u001B[0m\n\u001B[0;32m---> 35\u001B[0;31m         \u001B[0mexecute_streamingQuery1\u001B[0m\u001B[0;34m(\u001B[0m\u001B[0mlastTrades\u001B[0m\u001B[0;34m)\u001B[0m\u001B[0;34m\u001B[0m\u001B[0;34m\u001B[0m\u001B[0m\n\u001B[0m\u001B[1;32m     36\u001B[0m \u001B[0;34m\u001B[0m\u001B[0m\n\u001B[1;32m     37\u001B[0m         \u001B[0;31m# 3. send the result of query 1 to the challenger system\u001B[0m\u001B[0;34m\u001B[0m\u001B[0;34m\u001B[0m\u001B[0;34m\u001B[0m\u001B[0m\n",
      "\u001B[0;32m<ipython-input-31-604d432f5266>\u001B[0m in \u001B[0;36mexecute_streamingQuery1\u001B[0;34m(df_a)\u001B[0m\n\u001B[1;32m     17\u001B[0m       \u001B[0mnew_EMA100\u001B[0m \u001B[0;34m=\u001B[0m \u001B[0mcalcEMA_func\u001B[0m\u001B[0;34m(\u001B[0m\u001B[0msymbol\u001B[0m\u001B[0;34m,\u001B[0m \u001B[0mrow\u001B[0m\u001B[0;34m[\u001B[0m\u001B[0;34m'lasttradeprice'\u001B[0m\u001B[0;34m]\u001B[0m\u001B[0;34m,\u001B[0m \u001B[0;36m0.0\u001B[0m\u001B[0;34m,\u001B[0m \u001B[0;36m100\u001B[0m \u001B[0;34m)\u001B[0m\u001B[0;34m\u001B[0m\u001B[0;34m\u001B[0m\u001B[0m\n\u001B[1;32m     18\u001B[0m       \u001B[0mnew_row\u001B[0m \u001B[0;34m=\u001B[0m \u001B[0;34m{\u001B[0m \u001B[0;34m'symbol'\u001B[0m\u001B[0;34m:\u001B[0m \u001B[0msymbol\u001B[0m\u001B[0;34m,\u001B[0m \u001B[0;34m'lastWindowTime'\u001B[0m\u001B[0;34m:\u001B[0m \u001B[0mrow\u001B[0m\u001B[0;34m[\u001B[0m\u001B[0;34m'lastWindowTime'\u001B[0m\u001B[0;34m]\u001B[0m\u001B[0;34m,\u001B[0m\u001B[0;34m'EMA38'\u001B[0m\u001B[0;34m:\u001B[0m \u001B[0mnew_EMA38\u001B[0m\u001B[0;34m,\u001B[0m \u001B[0;34m'EMA100'\u001B[0m\u001B[0;34m:\u001B[0m \u001B[0mnew_EMA100\u001B[0m\u001B[0;34m,\u001B[0m \u001B[0;34m'Sectype'\u001B[0m\u001B[0;34m:\u001B[0m \u001B[0mrow\u001B[0m\u001B[0;34m[\u001B[0m\u001B[0;34m'Sectype'\u001B[0m\u001B[0;34m]\u001B[0m\u001B[0;34m,\u001B[0m \u001B[0;34m'BuyOrSell'\u001B[0m \u001B[0;34m:\u001B[0m \u001B[0;34m'Stay'\u001B[0m\u001B[0;34m,\u001B[0m \u001B[0;34m'seconds'\u001B[0m\u001B[0;34m:\u001B[0m \u001B[0mrow\u001B[0m\u001B[0;34m[\u001B[0m\u001B[0;34m'seconds'\u001B[0m\u001B[0;34m]\u001B[0m\u001B[0;34m,\u001B[0m \u001B[0;34m'nanos'\u001B[0m\u001B[0;34m:\u001B[0m \u001B[0mrow\u001B[0m\u001B[0;34m[\u001B[0m\u001B[0;34m'nanos'\u001B[0m\u001B[0;34m]\u001B[0m \u001B[0;34m}\u001B[0m\u001B[0;34m\u001B[0m\u001B[0;34m\u001B[0m\u001B[0m\n\u001B[0;32m---> 19\u001B[0;31m       \u001B[0mcrossover\u001B[0m \u001B[0;34m=\u001B[0m \u001B[0mcrossover\u001B[0m\u001B[0;34m.\u001B[0m\u001B[0mappend\u001B[0m\u001B[0;34m(\u001B[0m\u001B[0mpd\u001B[0m\u001B[0;34m.\u001B[0m\u001B[0mSeries\u001B[0m\u001B[0;34m(\u001B[0m\u001B[0mnew_row\u001B[0m\u001B[0;34m,\u001B[0m \u001B[0mindex\u001B[0m\u001B[0;34m=\u001B[0m\u001B[0mcrossover\u001B[0m\u001B[0;34m.\u001B[0m\u001B[0mcolumns\u001B[0m\u001B[0;34m,\u001B[0m \u001B[0mname\u001B[0m\u001B[0;34m=\u001B[0m\u001B[0msymbol\u001B[0m\u001B[0;34m)\u001B[0m\u001B[0;34m)\u001B[0m\u001B[0;34m\u001B[0m\u001B[0;34m\u001B[0m\u001B[0m\n\u001B[0m\u001B[1;32m     20\u001B[0m \u001B[0;34m\u001B[0m\u001B[0m\n\u001B[1;32m     21\u001B[0m       \u001B[0mnew_row\u001B[0m \u001B[0;34m=\u001B[0m \u001B[0;34m{\u001B[0m \u001B[0;34m'symbol'\u001B[0m\u001B[0;34m:\u001B[0m \u001B[0msymbol\u001B[0m\u001B[0;34m,\u001B[0m \u001B[0;34m'EMA38'\u001B[0m\u001B[0;34m:\u001B[0m \u001B[0mnew_EMA38\u001B[0m\u001B[0;34m,\u001B[0m \u001B[0;34m'EMA100'\u001B[0m\u001B[0;34m:\u001B[0m \u001B[0mnew_EMA100\u001B[0m\u001B[0;34m}\u001B[0m\u001B[0;34m\u001B[0m\u001B[0;34m\u001B[0m\u001B[0m\n",
      "\u001B[0;32m/usr/local/lib/python3.7/dist-packages/pandas/core/frame.py\u001B[0m in \u001B[0;36mappend\u001B[0;34m(self, other, ignore_index, verify_integrity, sort)\u001B[0m\n\u001B[1;32m   8967\u001B[0m                 \u001B[0mignore_index\u001B[0m\u001B[0;34m=\u001B[0m\u001B[0mignore_index\u001B[0m\u001B[0;34m,\u001B[0m\u001B[0;34m\u001B[0m\u001B[0;34m\u001B[0m\u001B[0m\n\u001B[1;32m   8968\u001B[0m                 \u001B[0mverify_integrity\u001B[0m\u001B[0;34m=\u001B[0m\u001B[0mverify_integrity\u001B[0m\u001B[0;34m,\u001B[0m\u001B[0;34m\u001B[0m\u001B[0;34m\u001B[0m\u001B[0m\n\u001B[0;32m-> 8969\u001B[0;31m                 \u001B[0msort\u001B[0m\u001B[0;34m=\u001B[0m\u001B[0msort\u001B[0m\u001B[0;34m,\u001B[0m\u001B[0;34m\u001B[0m\u001B[0;34m\u001B[0m\u001B[0m\n\u001B[0m\u001B[1;32m   8970\u001B[0m             )\n\u001B[1;32m   8971\u001B[0m         ).__finalize__(self, method=\"append\")\n",
      "\u001B[0;32m/usr/local/lib/python3.7/dist-packages/pandas/util/_decorators.py\u001B[0m in \u001B[0;36mwrapper\u001B[0;34m(*args, **kwargs)\u001B[0m\n\u001B[1;32m    309\u001B[0m                     \u001B[0mstacklevel\u001B[0m\u001B[0;34m=\u001B[0m\u001B[0mstacklevel\u001B[0m\u001B[0;34m,\u001B[0m\u001B[0;34m\u001B[0m\u001B[0;34m\u001B[0m\u001B[0m\n\u001B[1;32m    310\u001B[0m                 )\n\u001B[0;32m--> 311\u001B[0;31m             \u001B[0;32mreturn\u001B[0m \u001B[0mfunc\u001B[0m\u001B[0;34m(\u001B[0m\u001B[0;34m*\u001B[0m\u001B[0margs\u001B[0m\u001B[0;34m,\u001B[0m \u001B[0;34m**\u001B[0m\u001B[0mkwargs\u001B[0m\u001B[0;34m)\u001B[0m\u001B[0;34m\u001B[0m\u001B[0;34m\u001B[0m\u001B[0m\n\u001B[0m\u001B[1;32m    312\u001B[0m \u001B[0;34m\u001B[0m\u001B[0m\n\u001B[1;32m    313\u001B[0m         \u001B[0;32mreturn\u001B[0m \u001B[0mwrapper\u001B[0m\u001B[0;34m\u001B[0m\u001B[0;34m\u001B[0m\u001B[0m\n",
      "\u001B[0;32m/usr/local/lib/python3.7/dist-packages/pandas/core/reshape/concat.py\u001B[0m in \u001B[0;36mconcat\u001B[0;34m(objs, axis, join, ignore_index, keys, levels, names, verify_integrity, sort, copy)\u001B[0m\n\u001B[1;32m    305\u001B[0m     )\n\u001B[1;32m    306\u001B[0m \u001B[0;34m\u001B[0m\u001B[0m\n\u001B[0;32m--> 307\u001B[0;31m     \u001B[0;32mreturn\u001B[0m \u001B[0mop\u001B[0m\u001B[0;34m.\u001B[0m\u001B[0mget_result\u001B[0m\u001B[0;34m(\u001B[0m\u001B[0;34m)\u001B[0m\u001B[0;34m\u001B[0m\u001B[0;34m\u001B[0m\u001B[0m\n\u001B[0m\u001B[1;32m    308\u001B[0m \u001B[0;34m\u001B[0m\u001B[0m\n\u001B[1;32m    309\u001B[0m \u001B[0;34m\u001B[0m\u001B[0m\n",
      "\u001B[0;32m/usr/local/lib/python3.7/dist-packages/pandas/core/reshape/concat.py\u001B[0m in \u001B[0;36mget_result\u001B[0;34m(self)\u001B[0m\n\u001B[1;32m    531\u001B[0m \u001B[0;34m\u001B[0m\u001B[0m\n\u001B[1;32m    532\u001B[0m             new_data = concatenate_managers(\n\u001B[0;32m--> 533\u001B[0;31m                 \u001B[0mmgrs_indexers\u001B[0m\u001B[0;34m,\u001B[0m \u001B[0mself\u001B[0m\u001B[0;34m.\u001B[0m\u001B[0mnew_axes\u001B[0m\u001B[0;34m,\u001B[0m \u001B[0mconcat_axis\u001B[0m\u001B[0;34m=\u001B[0m\u001B[0mself\u001B[0m\u001B[0;34m.\u001B[0m\u001B[0mbm_axis\u001B[0m\u001B[0;34m,\u001B[0m \u001B[0mcopy\u001B[0m\u001B[0;34m=\u001B[0m\u001B[0mself\u001B[0m\u001B[0;34m.\u001B[0m\u001B[0mcopy\u001B[0m\u001B[0;34m\u001B[0m\u001B[0;34m\u001B[0m\u001B[0m\n\u001B[0m\u001B[1;32m    534\u001B[0m             )\n\u001B[1;32m    535\u001B[0m             \u001B[0;32mif\u001B[0m \u001B[0;32mnot\u001B[0m \u001B[0mself\u001B[0m\u001B[0;34m.\u001B[0m\u001B[0mcopy\u001B[0m\u001B[0;34m:\u001B[0m\u001B[0;34m\u001B[0m\u001B[0;34m\u001B[0m\u001B[0m\n",
      "\u001B[0;32m/usr/local/lib/python3.7/dist-packages/pandas/core/internals/concat.py\u001B[0m in \u001B[0;36mconcatenate_managers\u001B[0;34m(mgrs_indexers, axes, concat_axis, copy)\u001B[0m\n\u001B[1;32m    207\u001B[0m                 \u001B[0mvalues\u001B[0m \u001B[0;34m=\u001B[0m \u001B[0mvalues\u001B[0m\u001B[0;34m.\u001B[0m\u001B[0mview\u001B[0m\u001B[0;34m(\u001B[0m\u001B[0;34m)\u001B[0m\u001B[0;34m\u001B[0m\u001B[0;34m\u001B[0m\u001B[0m\n\u001B[1;32m    208\u001B[0m             \u001B[0mfastpath\u001B[0m \u001B[0;34m=\u001B[0m \u001B[0;32mTrue\u001B[0m\u001B[0;34m\u001B[0m\u001B[0;34m\u001B[0m\u001B[0m\n\u001B[0;32m--> 209\u001B[0;31m         \u001B[0;32melif\u001B[0m \u001B[0m_is_uniform_join_units\u001B[0m\u001B[0;34m(\u001B[0m\u001B[0mjoin_units\u001B[0m\u001B[0;34m)\u001B[0m\u001B[0;34m:\u001B[0m\u001B[0;34m\u001B[0m\u001B[0;34m\u001B[0m\u001B[0m\n\u001B[0m\u001B[1;32m    210\u001B[0m             \u001B[0mvals\u001B[0m \u001B[0;34m=\u001B[0m \u001B[0;34m[\u001B[0m\u001B[0mju\u001B[0m\u001B[0;34m.\u001B[0m\u001B[0mblock\u001B[0m\u001B[0;34m.\u001B[0m\u001B[0mvalues\u001B[0m \u001B[0;32mfor\u001B[0m \u001B[0mju\u001B[0m \u001B[0;32min\u001B[0m \u001B[0mjoin_units\u001B[0m\u001B[0;34m]\u001B[0m\u001B[0;34m\u001B[0m\u001B[0;34m\u001B[0m\u001B[0m\n\u001B[1;32m    211\u001B[0m \u001B[0;34m\u001B[0m\u001B[0m\n",
      "\u001B[0;32m/usr/local/lib/python3.7/dist-packages/pandas/core/internals/concat.py\u001B[0m in \u001B[0;36m_is_uniform_join_units\u001B[0;34m(join_units)\u001B[0m\n\u001B[1;32m    605\u001B[0m         \u001B[0;31m# no blocks that would get missing values (can lead to type upcasts)\u001B[0m\u001B[0;34m\u001B[0m\u001B[0;34m\u001B[0m\u001B[0;34m\u001B[0m\u001B[0m\n\u001B[1;32m    606\u001B[0m         \u001B[0;31m# unless we're an extension dtype.\u001B[0m\u001B[0;34m\u001B[0m\u001B[0;34m\u001B[0m\u001B[0;34m\u001B[0m\u001B[0m\n\u001B[0;32m--> 607\u001B[0;31m         \u001B[0mall\u001B[0m\u001B[0;34m(\u001B[0m\u001B[0;32mnot\u001B[0m \u001B[0mju\u001B[0m\u001B[0;34m.\u001B[0m\u001B[0mis_na\u001B[0m \u001B[0;32mor\u001B[0m \u001B[0mju\u001B[0m\u001B[0;34m.\u001B[0m\u001B[0mblock\u001B[0m\u001B[0;34m.\u001B[0m\u001B[0mis_extension\u001B[0m \u001B[0;32mfor\u001B[0m \u001B[0mju\u001B[0m \u001B[0;32min\u001B[0m \u001B[0mjoin_units\u001B[0m\u001B[0;34m)\u001B[0m\u001B[0;34m\u001B[0m\u001B[0;34m\u001B[0m\u001B[0m\n\u001B[0m\u001B[1;32m    608\u001B[0m         \u001B[0;32mand\u001B[0m\u001B[0;34m\u001B[0m\u001B[0;34m\u001B[0m\u001B[0m\n\u001B[1;32m    609\u001B[0m         \u001B[0;31m# no blocks with indexers (as then the dimensions do not fit)\u001B[0m\u001B[0;34m\u001B[0m\u001B[0;34m\u001B[0m\u001B[0;34m\u001B[0m\u001B[0m\n",
      "\u001B[0;32m/usr/local/lib/python3.7/dist-packages/pandas/core/internals/concat.py\u001B[0m in \u001B[0;36m<genexpr>\u001B[0;34m(.0)\u001B[0m\n\u001B[1;32m    605\u001B[0m         \u001B[0;31m# no blocks that would get missing values (can lead to type upcasts)\u001B[0m\u001B[0;34m\u001B[0m\u001B[0;34m\u001B[0m\u001B[0;34m\u001B[0m\u001B[0m\n\u001B[1;32m    606\u001B[0m         \u001B[0;31m# unless we're an extension dtype.\u001B[0m\u001B[0;34m\u001B[0m\u001B[0;34m\u001B[0m\u001B[0;34m\u001B[0m\u001B[0m\n\u001B[0;32m--> 607\u001B[0;31m         \u001B[0mall\u001B[0m\u001B[0;34m(\u001B[0m\u001B[0;32mnot\u001B[0m \u001B[0mju\u001B[0m\u001B[0;34m.\u001B[0m\u001B[0mis_na\u001B[0m \u001B[0;32mor\u001B[0m \u001B[0mju\u001B[0m\u001B[0;34m.\u001B[0m\u001B[0mblock\u001B[0m\u001B[0;34m.\u001B[0m\u001B[0mis_extension\u001B[0m \u001B[0;32mfor\u001B[0m \u001B[0mju\u001B[0m \u001B[0;32min\u001B[0m \u001B[0mjoin_units\u001B[0m\u001B[0;34m)\u001B[0m\u001B[0;34m\u001B[0m\u001B[0;34m\u001B[0m\u001B[0m\n\u001B[0m\u001B[1;32m    608\u001B[0m         \u001B[0;32mand\u001B[0m\u001B[0;34m\u001B[0m\u001B[0;34m\u001B[0m\u001B[0m\n\u001B[1;32m    609\u001B[0m         \u001B[0;31m# no blocks with indexers (as then the dimensions do not fit)\u001B[0m\u001B[0;34m\u001B[0m\u001B[0;34m\u001B[0m\u001B[0;34m\u001B[0m\u001B[0m\n",
      "\u001B[0;32m/usr/local/lib/python3.7/dist-packages/pandas/_libs/properties.pyx\u001B[0m in \u001B[0;36mpandas._libs.properties.CachedProperty.__get__\u001B[0;34m()\u001B[0m\n",
      "\u001B[0;32m/usr/local/lib/python3.7/dist-packages/pandas/core/internals/concat.py\u001B[0m in \u001B[0;36mis_na\u001B[0;34m(self)\u001B[0m\n\u001B[1;32m    379\u001B[0m         \u001B[0;32mreturn\u001B[0m \u001B[0mis_valid_na_for_dtype\u001B[0m\u001B[0;34m(\u001B[0m\u001B[0mna_value\u001B[0m\u001B[0;34m,\u001B[0m \u001B[0mdtype\u001B[0m\u001B[0;34m)\u001B[0m\u001B[0;34m\u001B[0m\u001B[0;34m\u001B[0m\u001B[0m\n\u001B[1;32m    380\u001B[0m \u001B[0;34m\u001B[0m\u001B[0m\n\u001B[0;32m--> 381\u001B[0;31m     \u001B[0;34m@\u001B[0m\u001B[0mcache_readonly\u001B[0m\u001B[0;34m\u001B[0m\u001B[0;34m\u001B[0m\u001B[0m\n\u001B[0m\u001B[1;32m    382\u001B[0m     \u001B[0;32mdef\u001B[0m \u001B[0mis_na\u001B[0m\u001B[0;34m(\u001B[0m\u001B[0mself\u001B[0m\u001B[0;34m)\u001B[0m \u001B[0;34m->\u001B[0m \u001B[0mbool\u001B[0m\u001B[0;34m:\u001B[0m\u001B[0;34m\u001B[0m\u001B[0;34m\u001B[0m\u001B[0m\n\u001B[1;32m    383\u001B[0m         \u001B[0;32mif\u001B[0m \u001B[0mself\u001B[0m\u001B[0;34m.\u001B[0m\u001B[0mblock\u001B[0m \u001B[0;32mis\u001B[0m \u001B[0;32mNone\u001B[0m\u001B[0;34m:\u001B[0m\u001B[0;34m\u001B[0m\u001B[0;34m\u001B[0m\u001B[0m\n",
      "\u001B[0;31mKeyboardInterrupt\u001B[0m: "
     ]
    }
   ],
   "source": [
    "# %%timeit\n",
    "# setting the option for GRPC\n",
    "op = [('grpc.max_send_message_length', 100 * 1024 * 1024),\n",
    "      ('grpc.max_receive_message_length', 1000 * 1024 * 1024)]\n",
    "\n",
    "# Submit results with benchmark\n",
    "with grpc.insecure_channel('challenge.msrg.in.tum.de:5023', options=op) as channel:\n",
    "    stub = api.ChallengerStub(channel)\n",
    "\n",
    "    # Create a new Benchmark\n",
    "    benchmarkconfiguration = ch.BenchmarkConfiguration(token= TOKEN_KEY,\n",
    "                                                        benchmark_name=\"Dept. Computer Engineering, DongA Univ.\",\n",
    "                                                        # This name is used here: https://challenge.msrg.in.tum.de/benchmarks/\n",
    "                                                        benchmark_type=benchmark_type,\n",
    "                                                        # Test or Evaluation, Evaluation will be available end of January. Test can be used to start implementing\n",
    "                                                        queries=[ch.Query.Q1, ch.Query.Q2])\n",
    "    benchmark = stub.createNewBenchmark(benchmarkconfiguration)\n",
    "\n",
    "    stub.startBenchmark(benchmark)\n",
    "    event_count = 0\n",
    "\n",
    "    while True:\n",
    "        batch = stub.nextBatch(benchmark)\n",
    "        event_count = event_count + len(batch.events)\n",
    "\n",
    "        lookup_symbols = list(batch.lookup_symbols)\n",
    "        print(\"batch seq_id\", batch.seq_id)\n",
    "        \n",
    "        # 1. every batch data is stored as csv files in the specific folder\n",
    "        make_CSV_file(batch.seq_id, batch.events)\n",
    "        \n",
    "        # 2. update the previous EMA table\n",
    "        lastTrades = spark.sql(\"select * from EMA\").filter(col('symbol').isin(lookup_symbols)).toPandas()\n",
    "        \n",
    "        execute_streamingQuery1(lastTrades)\n",
    "        \n",
    "        # 3. send the result of query 1 to the challenger system\n",
    "        resultQ1 = ch.ResultQ1(\n",
    "            benchmark_id=benchmark.id,  # The id of the benchmark\n",
    "            batch_seq_id=batch.seq_id,  # The sequence id of the batch\n",
    "            indicators= send_to_message_query1(lookup_symbols))  # Query1을 계산하여 challenge 서버에 보낼 message 생성\n",
    "        stub.resultQ1(resultQ1)  # send the result of query 1 back\n",
    "        \n",
    "        # 4. update the previous crossover table\n",
    "        execute_streamingQuery2()\n",
    "\n",
    "        # 5. send the result of query 2 to the challenger system\n",
    "        resultQ2 = ch.ResultQ2(\n",
    "            benchmark_id=benchmark.id,  # The id of the benchmark\n",
    "            batch_seq_id=batch.seq_id,  # The sequence id of the batch\n",
    "            crossover_events= send_to_message_query2(lookup_symbols)) # 마지막 3개 크로스오버를 계산하여 challenge 서버에 보낼 message 생성\n",
    "        \n",
    "        stub.resultQ2(resultQ2)  # submit the results of Q2\n",
    "\n",
    "        # prevEMA, Crossover 테이블 리셋\n",
    "        reset_dataframe(lookup_symbols)\n",
    "        \n",
    "        # Step 4 - once the last event is received, stop the clock\n",
    "        # See the statistics within ~5min here: https://challenge.msrg.in.tum.de/benchmarks/\n",
    "        if batch.last:\n",
    "            print(f\"received last batch, total batches: {event_count}\")\n",
    "            stub.endBenchmark(benchmark)\n",
    "            break\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    ""
   ],
   "metadata": {
    "id": "ZaG-38YZFTMO"
   }
  },
  {
   "cell_type": "code",
   "source": [
    "print(f\"Completed.\")\n",
    "spark.stop()"
   ],
   "metadata": {
    "id": "QmgGqCDeNII1"
   },
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Spark UI"
   ],
   "metadata": {
    "id": "IyVBBRwTKGw0"
   }
  },
  {
   "cell_type": "code",
   "source": [
    "!pip install pyngrok\n",
    "from pyngrok import ngrok"
   ],
   "metadata": {
    "id": "FOaU1xwTKGZJ"
   },
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": [
    "NGROK_AUTH_TOKEN = \"20Ayd0ezBx9EMWrfVGhPdif1skf_2h2MfcootBnk2WTBJf6GQ\"\n",
    "ngrok.set_auth_token(NGROK_AUTH_TOKEN)"
   ],
   "metadata": {
    "id": "OPNTs_VmKIjB"
   },
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": [
    "public_url = ngrok.connect(4050, \"http\")\n",
    "public_url"
   ],
   "metadata": {
    "id": "54RpxR7YKJyR"
   },
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XSjELz-Eild_"
   },
   "source": [
    "await terminiation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "rQKw3aWF4X7J"
   },
   "outputs": [],
   "source": [
    "streamingQuery1.awaitTermination()\n",
    "streamingQuery2.awaitTermination()\n",
    "\n",
    "spark.stop()"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "machine_shape": "hm",
   "name": "DEBS 2022 GC: GROUP 4",
   "provenance": [],
   "background_execution": "on",
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  },
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}