process_mining.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep Learning Toolkit for Splunk - Process Mining with PM4Py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook contains a barebone example workflow how to work on custom containerized code that seamlessly runs in Splunk Enterprise and interfaces with the Deep Learning Toolkit for Splunk."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stage 0 - import libraries\n",
    "At stage 0 we define all imports necessary to run our subsequent code depending on various libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "deletable": false,
    "name": "dltk_import"
   },
   "outputs": [],
   "source": [
    "import json\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import pm4py\n",
    "from pm4py.objects.log.util import dataframe_utils\n",
    "from pm4py.objects.conversion.log import converter as log_converter\n",
    "from pm4py.algo.discovery.alpha import algorithm as alpha_miner\n",
    "from pm4py.algo.discovery.inductive import algorithm as inductive_miner\n",
    "from pm4py.algo.discovery.dfg import algorithm as dfg_discovery\n",
    "from pm4py.visualization.petrinet import visualizer as pn_visualizer\n",
    "\n",
    "# ...\n",
    "# global constants\n",
    "MODEL_DIRECTORY = \"/srv/app/model/data/\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numpy version: 1.18.1\n",
      "pandas version: 1.1.5\n",
      "pm4py version: 2.1.2\n"
     ]
    }
   ],
   "source": [
    "# THIS CELL IS NOT EXPORTED - free notebook cell for testing or development purposes\n",
    "print(\"numpy version: \" + np.__version__)\n",
    "print(\"pandas version: \" + pd.__version__)\n",
    "print(\"pm4py version: \" + pm4py.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stage 1 - get a data sample from Splunk\n",
    "In Splunk run a search to pipe a dataset into your notebook environment. Note: mode=stage is used in the | compute command to do this."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "index=_internal uri=* user=* <br>\n",
    "| stats count by _time uri user <br>\n",
    "| eval start_timestamp=strftime(_time, \"%Y%m%dT%H%M%S\") <br>\n",
    "| rename uri as case:concept:name user as concept:name <br>\n",
    "| eval time:timestamp = start_timestamp<br>\n",
    "| fit MLTKContainer algo=process_mining mode=stage case:concept:name,concept:name,start_timestamp,time:timestamp into process_mining<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After you run this search your data set sample is available as a csv inside the container to develop your model. The name is taken from the model_name value or set to \"default\" if no model_name is present. This step is intended to work with a subset of your data to create your custom model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "deletable": false,
    "name": "dltk_stage"
   },
   "outputs": [],
   "source": [
    "# this cell is not executed and should only be used for staging data into the notebook environment to have it accessible in this notebook\n",
    "def stage(name):\n",
    "    with open(\"data/\"+name+\".csv\", 'r') as f:\n",
    "        df = pd.read_csv(f)\n",
    "    with open(\"data/\"+name+\".json\", 'r') as f:\n",
    "        param = json.load(f)\n",
    "    return df, param"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'options': {'params': {'algo': 'process_mining', 'mode': 'stage'}, 'args': ['case:concept:name', 'concept:name', 'start_timestamp', 'time:timestamp'], 'feature_variables': ['case:concept:name', 'concept:name', 'start_timestamp', 'time:timestamp'], 'model_name': 'process_mining', 'algo_name': 'MLTKContainer', 'mlspl_limits': {'disabled': False, 'handle_new_cat': 'default', 'max_distinct_cat_values': '10000', 'max_distinct_cat_values_for_classifiers': '10000', 'max_distinct_cat_values_for_scoring': '10000', 'max_fit_time': '6000', 'max_inputs': '100000000', 'max_memory_usage_mb': '4000', 'max_model_size_mb': '150', 'max_score_time': '6000', 'streaming_apply': '0', 'use_sampling': '1'}, 'kfold_cv': None}, 'feature_variables': ['case:concept:name', 'concept:name', 'start_timestamp', 'time:timestamp']}\n"
     ]
    }
   ],
   "source": [
    "# THIS CELL IS NOT EXPORTED - free notebook cell for testing or development purposes\n",
    "df, param = stage(\"process_mining\")\n",
    "print(param)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>case:concept:name</th>\n",
       "      <th>concept:name</th>\n",
       "      <th>start_timestamp</th>\n",
       "      <th>time:timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>/services/server/info</td>\n",
       "      <td>-</td>\n",
       "      <td>20201221T105423</td>\n",
       "      <td>20201221T105423</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>/favicon.ico</td>\n",
       "      <td>-</td>\n",
       "      <td>20201221T105423</td>\n",
       "      <td>20201221T105423</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>/robots.txt</td>\n",
       "      <td>-</td>\n",
       "      <td>20201221T105423</td>\n",
       "      <td>20201221T105423</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>/servicesNS/nobody/splunk_instrumentation/tele...</td>\n",
       "      <td>splunk-system-user</td>\n",
       "      <td>20201221T105426</td>\n",
       "      <td>20201221T105426</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>/services/server/info</td>\n",
       "      <td>splunk-system-user</td>\n",
       "      <td>20201221T105426</td>\n",
       "      <td>20201221T105426</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2204</th>\n",
       "      <td>/en-GB/splunkd/__raw/services/search/shelper?o...</td>\n",
       "      <td>admin</td>\n",
       "      <td>20201221T110413</td>\n",
       "      <td>20201221T110413</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2205</th>\n",
       "      <td>/en-GB/splunkd/__raw/services/search/shelper?o...</td>\n",
       "      <td>admin</td>\n",
       "      <td>20201221T110413</td>\n",
       "      <td>20201221T110413</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2206</th>\n",
       "      <td>/en-GB/splunkd/__raw/services/search/shelper?o...</td>\n",
       "      <td>admin</td>\n",
       "      <td>20201221T110413</td>\n",
       "      <td>20201221T110413</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2207</th>\n",
       "      <td>/en-GB/splunkd/__raw/services/search/shelper?o...</td>\n",
       "      <td>admin</td>\n",
       "      <td>20201221T110413</td>\n",
       "      <td>20201221T110413</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2208</th>\n",
       "      <td>/en-GB/splunkd/__raw/services/search/shelper?o...</td>\n",
       "      <td>admin</td>\n",
       "      <td>20201221T110413</td>\n",
       "      <td>20201221T110413</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2209 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      case:concept:name        concept:name  \\\n",
       "0                                 /services/server/info                   -   \n",
       "1                                          /favicon.ico                   -   \n",
       "2                                           /robots.txt                   -   \n",
       "3     /servicesNS/nobody/splunk_instrumentation/tele...  splunk-system-user   \n",
       "4                                 /services/server/info  splunk-system-user   \n",
       "...                                                 ...                 ...   \n",
       "2204  /en-GB/splunkd/__raw/services/search/shelper?o...               admin   \n",
       "2205  /en-GB/splunkd/__raw/services/search/shelper?o...               admin   \n",
       "2206  /en-GB/splunkd/__raw/services/search/shelper?o...               admin   \n",
       "2207  /en-GB/splunkd/__raw/services/search/shelper?o...               admin   \n",
       "2208  /en-GB/splunkd/__raw/services/search/shelper?o...               admin   \n",
       "\n",
       "      start_timestamp   time:timestamp  \n",
       "0     20201221T105423  20201221T105423  \n",
       "1     20201221T105423  20201221T105423  \n",
       "2     20201221T105423  20201221T105423  \n",
       "3     20201221T105426  20201221T105426  \n",
       "4     20201221T105426  20201221T105426  \n",
       "...               ...              ...  \n",
       "2204  20201221T110413  20201221T110413  \n",
       "2205  20201221T110413  20201221T110413  \n",
       "2206  20201221T110413  20201221T110413  \n",
       "2207  20201221T110413  20201221T110413  \n",
       "2208  20201221T110413  20201221T110413  \n",
       "\n",
       "[2209 rows x 4 columns]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stage 2 - create and initialize a model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "deletable": false,
    "name": "dltk_init"
   },
   "outputs": [],
   "source": [
    "# initialize your model\n",
    "# available inputs: data and parameters\n",
    "# returns the model object which will be used as a reference to call fit, apply and summary subsequently\n",
    "def init(df,param):\n",
    "    model = {}\n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{}\n"
     ]
    }
   ],
   "source": [
    "# THIS CELL IS NOT EXPORTED - free notebook cell for testing or development purposes\n",
    "model = init(df,param)\n",
    "print(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stage 3 - fit the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "deletable": false,
    "name": "dltk_fit"
   },
   "outputs": [],
   "source": [
    "# train your model\n",
    "# returns a fit info json object and may modify the model object\n",
    "def fit(model,df,param):\n",
    "    info = {}\n",
    "    return info"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{}\n"
     ]
    }
   ],
   "source": [
    "# THIS CELL IS NOT EXPORTED - free notebook cell for testing or development purposes\n",
    "print(fit(model,df,param))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stage 4 - apply the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "deletable": false,
    "name": "dltk_apply"
   },
   "outputs": [],
   "source": [
    "# apply your model\n",
    "# returns the calculated results\n",
    "def pngfile_to_base64(filepath):\n",
    "    import base64\n",
    "    import io\n",
    "    with open(filepath, 'rb') as file:\n",
    "        pic_hash = base64.b64encode(file.read())\n",
    "    return str(pic_hash)\n",
    "\n",
    "def apply(model,df,param):\n",
    "    # convert dataframe to pm4py compatible event_log object\n",
    "    log_csv = dataframe_utils.convert_timestamp_columns_in_df(df)\n",
    "    log_csv = log_csv.sort_values('start_timestamp')\n",
    "    event_logs = log_converter.apply(log_csv, variant=log_converter.Variants.TO_EVENT_LOG)\n",
    "\n",
    "    # apply dfg discovery\n",
    "    dfg, start_activities, end_activities = pm4py.discover_dfg(event_logs)\n",
    "    #dfg = dfg_discovery.apply(event_logs, variant=dfg_discovery.Variants.PERFORMANCE)\n",
    "    temp_viz_file = 'dfg.png'\n",
    "    pm4py.save_vis_dfg(dfg, start_activities, end_activities, temp_viz_file, log=None)\n",
    "    model['plot_pairplot'] = pngfile_to_base64(temp_viz_file)\n",
    "        \n",
    "    # apply inductive miner for petri net retrival\n",
    "    net, initial_marking, final_marking = inductive_miner.apply(event_logs)\n",
    "    temp_viz_file = 'petrinet.png'\n",
    "    pm4py.save_vis_petri_net(net, initial_marking, final_marking, temp_viz_file)\n",
    "    model['plot_matrix'] = pngfile_to_base64(temp_viz_file)\n",
    "\n",
    "    # return a dot graphviz compatible description to feed the process diagram custom visualization on a splunk dashboard\n",
    "    gviz = pn_visualizer.apply(net, initial_marking, final_marking, variant=pn_visualizer.Variants.FREQUENCY, log=log_csv)    \n",
    "    return pd.DataFrame([str(gviz)], columns=['dot'])\n",
    "    \n",
    "    # Other options:\n",
    "    # discover process tree\n",
    "    #process_tree = pm4py.discover_tree_inductive(event_logs)\n",
    "    #pm4py.save_vis_process_tree(process_tree, 'processtree.png')\n",
    "\n",
    "    # add frequency information\n",
    "    #dfg_frequency = dfg_discovery.apply(event_logs, variant=dfg_discovery.Variants.FREQUENCY)\n",
    "    \n",
    "    # get the raw arcs, e.g. of the petri net\n",
    "    #return pd.DataFrame(list(net.arcs))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>dot</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>digraph \"imdf_net_1608545197.571939\" {\\n\\tgrap...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                 dot\n",
       "0  digraph \"imdf_net_1608545197.571939\" {\\n\\tgrap..."
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# THIS CELL IS NOT EXPORTED - free notebook cell for testing or development purposes\n",
    "result = apply(model,df,param)\n",
    "result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stage 5 - save the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "deletable": false,
    "name": "dltk_save"
   },
   "outputs": [],
   "source": [
    "# save model to name in expected convention \"algo_<model_name>\"\n",
    "def save(model,name):\n",
    "    #with open(MODEL_DIRECTORY + name + \".json\", 'w') as file:\n",
    "    #    json.dump(model, file)\n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#saved_model = save(model,'algo_barebone_model')\n",
    "#saved_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stage 6 - load the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "name": "dltk_load"
   },
   "outputs": [],
   "source": [
    "# load model from name in expected convention \"algo_<model_name>\"\n",
    "def load(name):\n",
    "    model = {}\n",
    "    #with open(MODEL_DIRECTORY + name + \".json\", 'r') as file:\n",
    "    #    model = json.load(file)\n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#loaded_model = load('algo_barebone_model')\n",
    "#loaded_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stage 7 - provide a summary of the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "deletable": false,
    "name": "dltk_summary"
   },
   "outputs": [],
   "source": [
    "# return a model summary\n",
    "def summary(model=None):\n",
    "    returns = {\"version\": {\"pm4py\": pm4py.__version__} }\n",
    "    return returns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'version': {'pm4py': '2.0.1.3'}}"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "summary(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## End of Stages\n",
    "All subsequent cells are not tagged and can be used for further freeform code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}