ML-0-preprocess-outliners.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Поиск выбросов (outliers)"
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Выброс** — результат измерения, который выделяется из общей выборки. Выбросы появляются из-за ошибок в данных и сильно выделяющихся среди остальных значений. Их нужно искать и удалять, иначе данные будут подстраиваться под них и давать неверный результат.\n",
        "\n",
        "**Как искать выбросы:**\n",
        "\n",
        "- **Поиск и удаление по порогу**:  \n",
        "Способ нахождения выбросов, основанный на здравом смысле и методе пристального взгляда. Посмотреть данные и поискать невозможные и нелогичные значения, например, в столбце «высота здания» стоит 30 км.\n",
        "- **Поиск с помощью статистики**:  \n",
        "Поиск с помощью здравого смысла трудно автоматизировать, поэтому эффективнее статистические методы. Например, поиск по отклонению или по распределению IQR (interquartile range, межквартильное расстояние).\n",
        "- **Поиск с помощью DBSCAN**:  \n",
        "Поиск с помощью методов кластеризации, один из таких методов — DBSCAN, который может автоматически находить шум в данных.\n",
        "\n",
        "Разные способы поиска выбросов могут давать разные результаты. Не всегда выделяющееся значение будет являться выбросом, возможно, у нас пока просто не хватает данных."
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "from math import radians, sin, cos, asin, sqrt"
      ],
      "outputs": [],
      "execution_count": 1,
      "metadata": {
        "collapsed": false,
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "execution": {
          "iopub.status.busy": "2020-03-02T06:45:24.554Z",
          "iopub.execute_input": "2020-03-02T06:45:24.563Z",
          "iopub.status.idle": "2020-03-02T06:45:24.985Z",
          "shell.execute_reply": "2020-03-02T06:45:24.993Z"
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%matplotlib inline\n",
        "\n",
        "vis_data = pd.read_csv(\"./data/train.csv\", \n",
        "                       encoding = 'ISO-8859-1', \n",
        "                       low_memory = False)\n",
        "vis_data = vis_data.drop(['violation_zip_code', 'clean_up_cost'], axis=1)\n",
        "latlons = pd.read_csv(\"./data/latlons.csv\")\n",
        "vis_data = pd.concat([vis_data, latlons], axis=1)\n",
        "\n",
        "def haversine(lon1, lat1, lon2, lat2):\n",
        "    lon1, lat1, lon2, lat2 = map(radians, [lon1, lat1, lon2, lat2])\n",
        "    dlon = lon2 - lon1\n",
        "    dlat = lat2 - lat1\n",
        "    a = sin(dlat/2)**2 + cos(lat1) * cos(lat2) * sin(dlon/2) ** 2\n",
        "    c = 2 * asin(sqrt(a))\n",
        "    earth_radius = 6371 \n",
        "    return c * earth_radius\n",
        "\n",
        "vis_data['distance'] = vis_data.apply(lambda row: \n",
        "                                      haversine(lon1 = -83.045753, \n",
        "                                                lat1 = 42.331429,\n",
        "                                                lon2 = row['lon'], \n",
        "                                                lat2 = row['lat']), \n",
        "                                      axis=1)"
      ],
      "outputs": [],
      "execution_count": 1,
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Поиск и удаление по порогу\n",
        "\n",
        "Часто оказывается так, что при погружении в предметную область вы начинаете понимать, какие значения данные признаки могут принимать, а какие - не могут. Например, если нужно заплатить штраф, полезно будет проверить, нет ли в выборке значений штрафов, меньших, чем `0`:"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "np.sum(vis_data.fine_amount < 0)"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 2,
          "data": {
            "text/plain": [
              "0"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 2,
      "metadata": {
        "scrolled": true
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Кроме этого, интересно посмотреть максимальное значение штрафа:"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "max_fine = np.max(vis_data.fine_amount)\n",
        "max_fine"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 3,
          "data": {
            "text/plain": [
              "10000.0"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 3,
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "И сколько раз и как часто такой штраф встречается в данных:"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "max_fine_count = np.sum(vis_data.fine_amount == max_fine)\n",
        "max_fine_count, max_fine_count/vis_data.shape[0]"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 4,
          "data": {
            "text/plain": [
              "(357, 0.0014262542647799094)"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 4,
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "vis_data.fine_amount.hist();"
      ],
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": [
              "iVBORw0KGgoAAAANSUhEUgAAAY0AAAD8CAYAAACLrvgBAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAElBJREFUeJzt3X+s3fVdx/Hn23aw2o21HXJT28ZCbIw4Iis30Dlj7oaWgsZiAkkJsR3D1CCLm5JocX9Uh0s2I5spTrYqlbIwGLLNNqxYm46bxWRjLQ5pGXS9YxUurXSsrKMsulXf/nE+lx0up/d+es7tvvfH85F8c77n/f18v5/v53wvfd3vj3OJzESSpBo/1fQOSJKmDkNDklTN0JAkVTM0JEnVDA1JUjVDQ5JUzdCQJFUzNCRJ1QwNSVK12U3vwEQ799xzc+nSpV2t+8orrzB37tyJ3aFJzjHPDI55ZuhlzI899tiLmfkz47WbdqGxdOlS9u7d29W6g4ODDAwMTOwOTXKOeWZwzDNDL2OOiP+saeflKUlSNUNDklTN0JAkVTM0JEnVDA1JUjVDQ5JUzdCQJFUzNCRJ1QwNSVK1afeN8F7se/4479nwxUb6PvSR32ykX0k6HZ5pSJKqGRqSpGqGhiSpmqEhSapmaEiSqhkakqRqhoYkqZqhIUmqZmhIkqoZGpKkaoaGJKmaoSFJqmZoSJKqGRqSpGqGhiSpmqEhSapmaEiSqhkakqRqhoYkqZqhIUmqZmhIkqoZGpKkaoaGJKmaoSFJqjZuaETEkoh4JCKeiognI+L9pb4gInZFxMHyOr/UIyI2RcRQRDwREcvbtrWutD8YEeva6pdExL6yzqaIiLH6kCQ1o+ZM4yRwS2b+IrACuDkiLgQ2ALszcxmwu7wHuBJYVqb1wJ3QCgBgI3AZcCmwsS0E7ixtR9ZbVeqn6kOS1IBxQyMzj2Tmv5f5l4GngEXAamBrabYVuLrMrwbuyZavAvMiYiFwBbArM49l5kvALmBVWXZOZn4lMxO4Z9S2OvUhSWrA7NNpHBFLgbcDjwJ9mXkEWsESEeeVZouA59pWGy61serDHeqM0cfo/VpP60yFvr4+BgcHT2dYr+qbA7dcdLKrdXvV7T736sSJE4313RTHPDM45jOjOjQi4k3A54APZOb3y22Hjk071LKLerXM3AxsBujv78+BgYHTWf1Vd9y7jdv3nVaOTphD1w800u/g4CDdfl5TlWOeGRzzmVH19FREvIFWYNybmZ8v5RfKpSXK69FSHwaWtK2+GDg8Tn1xh/pYfUiSGlDz9FQAdwFPZebH2hZtB0aegFoHbGurry1PUa0AjpdLTDuBlRExv9wAXwnsLMtejogVpa+1o7bVqQ9JUgNqrsW8E/hdYF9EPF5qfwZ8BHggIm4EngWuLct2AFcBQ8APgBsAMvNYRNwG7CntPpSZx8r8TcDdwBzg4TIxRh+SpAaMGxqZ+W90vu8AcHmH9gncfIptbQG2dKjvBd7Wof7dTn1IkprhN8IlSdUMDUlSNUNDklTN0JAkVTM0JEnVDA1JUjVDQ5JUzdCQJFUzNCRJ1QwNSVI1Q0OSVM3QkCRVMzQkSdUMDUlSNUNDklTN0JAkVTM0JEnVDA1JUjVDQ5JUzdCQJFUzNCRJ1QwNSVI1Q0OSVM3QkCRVMzQkSdUMDUlSNUNDklTN0JAkVTM0JEnVDA1JUjVDQ5JUzdCQJFUzNCRJ1QwNSVI1Q0OSVG3c0IiILRFxNCL2t9X+PCKej4jHy3RV27JbI2IoIg5ExBVt9VWlNhQRG9rq50fEoxFxMCI+GxFnlfrZ5f1QWb50ogYtSepOzZnG3cCqDvWPZ+bFZdoBEBEXAmuAXyrr/F1EzIqIWcAngCuBC4HrSluAj5ZtLQNeAm4s9RuBlzLz54GPl3aSpAaNGxqZ+WXgWOX2VgP3Z+b/ZOa3gSHg0jINZeYzmflD4H5gdUQE8G7gwbL+VuDqtm1tLfMPApeX9pKkhvRyT+N9EfFEuXw1v9QWAc+1tRkutVPV3wp8LzNPjqq/Zltl+fHSXpLUkNldrncncBuQ5fV24L1ApzOBpHM45RjtGWfZa0TEemA9QF9fH4ODg2Ps+qn1zYFbLjo5fsMzoNt97tWJEyca67spjnlmcMxnRlehkZkvjMxHxN8DD5W3w8CStqaLgcNlvlP9RWBeRMwuZxPt7Ue2NRwRs4G3cIrLZJm5GdgM0N/fnwMDA90Mizvu3cbt+7rN0d4cun6gkX4HBwfp9vOaqhzzzOCYz4yuLk9FxMK2t78DjDxZtR1YU558Oh9YBnwN2AMsK09KnUXrZvn2zEzgEeCasv46YFvbttaV+WuAL5X2kqSGjPtrdUTcBwwA50bEMLARGIiIi2ldLjoE/D5AZj4ZEQ8A3wBOAjdn5v+W7bwP2AnMArZk5pOliz8F7o+IvwS+DtxV6ncBn46IIVpnGGt6Hq0kqSfjhkZmXtehfFeH2kj7DwMf7lDfAezoUH+G1tNVo+v/DVw73v5Jkn5y/Ea4JKmaoSFJqmZoSJKqGRqSpGqGhiSpmqEhSapmaEiSqhkakqRqhoYkqZqhIUmqZmhIkqoZGpKkaoaGJKmaoSFJqmZoSJKqGRqSpGqGhiSpmqEhSapmaEiSqhkakqRqhoYkqZqhIUmqZmhIkqoZGpKkaoaGJKmaoSFJqmZoSJKqGRqSpGqGhiSpmqEhSapmaEiSqhkakqRqhoYkqZqhIUmqZmhIkqoZGpKkauOGRkRsiYijEbG/rbYgInZFxMHyOr/UIyI2RcRQRDwREcvb1llX2h+MiHVt9UsiYl9ZZ1NExFh9SJKaU3OmcTewalRtA7A7M5cBu8t7gCuBZWVaD9wJrQAANgKXAZcCG9tC4M7SdmS9VeP0IUlqyLihkZlfBo6NKq8Gtpb5rcDVbfV7suWrwLyIWAhcAezKzGOZ+RKwC1hVlp2TmV/JzATuGbWtTn1IkhrS7T2Nvsw8AlBezyv1RcBzbe2GS22s+nCH+lh9SJIaMnuCtxcdatlF/fQ6jVhP6xIXfX19DA4Onu4mAOibA7dcdLKrdXvV7T736sSJE4313RTHPDM45jOj29B4ISIWZuaRconpaKkPA0va2i0GDpf6wKj6YKkv7tB+rD5eJzM3A5sB+vv7c2Bg4FRNx3THvdu4fd9E52idQ9cPNNLv4OAg3X5eU5Vjnhkc85nR7eWp7cDIE1DrgG1t9bXlKaoVwPFyaWknsDIi5pcb4CuBnWXZyxGxojw1tXbUtjr1IUlqyLi/VkfEfbTOEs6NiGFaT0F9BHggIm4EngWuLc13AFcBQ8APgBsAMvNYRNwG7CntPpSZIzfXb6L1hNYc4OEyMUYfkqSGjBsamXndKRZd3qFtAjefYjtbgC0d6nuBt3Wof7dTH5Kk5viNcElSNUNDklTN0JAkVTM0JEnVDA1JUjVDQ5JUzdCQJFUzNCRJ1QwNSVI1Q0OSVM3QkCRVMzQkSdUMDUlSNUNDklTN0JAkVTM0JEnVDA1JUjVDQ5JUzdCQJFUzNCRJ1QwNSVI1Q0OSVM3QkCRVMzQkSdUMDUlSNUNDklTN0JAkVTM0JEnVDA1JUjVDQ5JUzdCQJFUzNCRJ1QwNSVI1Q0OSVM3QkCRVMzQkSdV6Co2IOBQR+yLi8YjYW2oLImJXRBwsr/NLPSJiU0QMRcQTEbG8bTvrSvuDEbGurX5J2f5QWTd62V9JUm8m4kzjXZl5cWb2l/cbgN2ZuQzYXd4DXAksK9N64E5ohQywEbgMuBTYOBI0pc36tvVWTcD+SpK6dCYuT60Gtpb5rcDVbfV7suWrwLyIWAhcAezKzGOZ+RKwC1hVlp2TmV/JzATuaduWJKkBs3tcP4F/jYgEPpWZm4G+zDwCkJlHIuK80nYR8FzbusOlNlZ9uEP9dSJiPa0zEvr6+hgcHOxqMH1z4JaLTna1bq+63edenThxorG+m+KYZwbHfGb0GhrvzMzDJRh2RcTTY7TtdD8iu6i/vtgKq80A/f39OTAwMOZOn8od927j9n29fiTdOXT9QCP9Dg4O0u3nNVU55pnBMZ8ZPV2eyszD5fUo8AVa9yReKJeWKK9HS/NhYEnb6ouBw+PUF3eoS5Ia0nVoRMTciHjzyDywEtgPbAdGnoBaB2wr89uBteUpqhXA8XIZayewMiLmlxvgK4GdZdnLEbGiPDW1tm1bkqQG9HItpg/4QnkKdjbwmcz8l4jYAzwQETcCzwLXlvY7gKuAIeAHwA0AmXksIm4D9pR2H8rMY2X+JuBuYA7wcJkkSQ3pOjQy8xnglzvUvwtc3qGewM2n2NYWYEuH+l7gbd3uoyRpYvmNcElSNUNDklTN0JAkVTM0JEnVDA1JUjVDQ5JUzdCQJFUzNCRJ1QwNSVI1Q0OSVM3QkCRVMzQkSdUMDUlSNUNDklTN0JAkVTM0JEnVDA1JUjVDQ5JUzdCQJFUzNCRJ1QwNSVI1Q0OSVM3QkCRVMzQkSdUMDUlSNUNDklTN0JAkVTM0JEnVDA1JUjVDQ5JUzdCQJFWb3fQOqGXphi820u/dq+Y20q+kqckzDUlSNc801IimzqzAsyupF55pSJKqTfrQiIhVEXEgIoYiYkPT+yNJM9mkvjwVEbOATwC/AQwDeyJie2Z+o9k9mz72PX+c9zR4qUjS1DLZzzQuBYYy85nM/CFwP7C64X2SpBlrUp9pAIuA59reDwOXNbQvkjSu6f6Qx2QPjehQy9c1ilgPrC9vT0TEgS77Oxd4sct1p6Q/nIFjftdHZ96YmYHHmRk45h5/tn+uptFkD41hYEnb+8XA4dGNMnMzsLnXziJib2b297qdqcQxzwyOeWb4SYx5st/T2AMsi4jzI+IsYA2wveF9kqQZa1KfaWTmyYh4H7ATmAVsycwnG94tSZqxJnVoAGTmDmDHT6i7ni9xTUGOeWZwzDPDGR9zZL7uvrIkSR1N9nsakqRJxNAopsufK4mIJRHxSEQ8FRFPRsT7S31BROyKiIPldX6pR0RsKuN+IiKWt21rXWl/MCLWNTWmWhExKyK+HhEPlffnR8SjZf8/Wx6mICLOLu+HyvKlbdu4tdQPRMQVzYykTkTMi4gHI+LpcrzfMd2Pc0T8Ufm53h8R90XEG6fbcY6ILRFxNCL2t9Um7LhGxCURsa+ssykiOn214dQyc8ZPtG6yfwu4ADgL+A/gwqb3q8uxLASWl/k3A98ELgT+CthQ6huAj5b5q4CHaX0nZgXwaKkvAJ4pr/PL/PymxzfO2P8Y+AzwUHn/ALCmzH8SuKnM/wHwyTK/Bvhsmb+wHPuzgfPLz8Sspsc1xni3Ar9X5s8C5k3n40zry77fBua0Hd/3TLfjDPwasBzY31absOMKfA14R1nnYeDK09q/pj+gyTCVD3Bn2/tbgVub3q8JGts2Wn+76wCwsNQWAgfK/KeA69raHyjLrwM+1VZ/TbvJNtH6Ds9u4N3AQ+U/iBeB2aOPMa2n8d5R5meXdjH6uLe3m2wTcE75BzRG1aftcebHfyFiQTluDwFXTMfjDCwdFRoTclzLsqfb6q9pVzN5eaql058rWdTQvkyYcjr+duBRoC8zjwCU1/NKs1ONfap9Jn8D/Anwf+X9W4HvZebJ8r59/18dW1l+vLSfSmO+APgO8I/lktw/RMRcpvFxzszngb8GngWO0DpujzG9j/OIiTqui8r86Ho1Q6Ol6s+VTCUR8Sbgc8AHMvP7YzXtUMsx6pNORPwWcDQzH2svd2ia4yybMmOm9ZvzcuDOzHw78AqtyxanMuXHXK7jr6Z1SelngbnAlR2aTqfjPJ7THWPPYzc0Wqr+XMlUERFvoBUY92bm50v5hYhYWJYvBI6W+qnGPpU+k3cCvx0Rh2j9JeR30zrzmBcRI99Fat//V8dWlr8FOMbUGvMwMJyZj5b3D9IKkel8nH8d+HZmficzfwR8HvgVpvdxHjFRx3W4zI+uVzM0WqbNnyspT0LcBTyVmR9rW7QdGHmCYh2tex0j9bXlKYwVwPFy+rsTWBkR88tveCtLbdLJzFszc3FmLqV17L6UmdcDjwDXlGajxzzyWVxT2meprylP3ZwPLKN103DSycz/Ap6LiF8opcuBbzCNjzOty1IrIuKny8/5yJin7XFuMyHHtSx7OSJWlM9wbdu26jR9w2eyTLSeQvgmrScpPtj0/vQwjl+ldbr5BPB4ma6idS13N3CwvC4o7YPW/+jqW8A+oL9tW+8Fhsp0Q9Njqxz/AD9+euoCWv8YDAH/BJxd6m8s74fK8gva1v9g+SwOcJpPlTQw1ouBveVY/zOtp2Sm9XEG/gJ4GtgPfJrWE1DT6jgD99G6Z/MjWmcGN07kcQX6y+f3LeBvGfUwxXiT3wiXJFXz8pQkqZqhIUmqZmhIkqoZGpKkaoaGJKmaoSFJqmZoSJKqGRqSpGr/D9vwkR7VLG5eAAAAAElFTkSuQmCC\n"
            ],
            "text/plain": [
              "<matplotlib.figure.Figure at 0x106109eb8>"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 5,
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "Очень большое количество данных сосредоточено около 0. Посмотрим, сколько значений штрафов в выборке равно 0:"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "vis_data[vis_data.fine_amount == 0].shape[0]"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 6,
          "data": {
            "text/plain": [
              "195"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 6,
      "metadata": {
        "scrolled": true
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Возможно, те значения, где размер штрафа равен 0, являются выбросами, если инспектор забыл добавить размер штрафа или ошибся при вводе.  \n",
        "В таком случае эти значения следует убрать из выборки. В то же время, сложно однозначно сделать такое решение, не зная всей информации о данных."
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "Теперь посмотрим на данные поближе:"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "vis_data.fine_amount[vis_data.fine_amount < 2000].hist(bins=100);"
      ],
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": [
              "iVBORw0KGgoAAAANSUhEUgAAAY0AAAD8CAYAAACLrvgBAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAGhRJREFUeJzt3X+QXfV93vH3U23AMgQkGbOjkdRIjjduZZQ00o5Q6tqzthxpIQ4iremI0USLo44mrnDtVpmwlGmVsc0MNCXETG0yiqUieQiCEHvQRKKyRuaW6QxgfrMIjLUIFRbJUmwJmTW26Tqf/nG+iw/L3d3v3ivu3QPPa+bOPedzvufczz0L++z5ca8UEZiZmeX4J+1uwMzMqsOhYWZm2RwaZmaWzaFhZmbZHBpmZpbNoWFmZtkcGmZmls2hYWZm2RwaZmaWraPdDZxpF1xwQSxcuLChdX/yk59wzjnnnNmG3kZV6rdKvUK1+q1Sr1CtfqvUKzTX76OPPvrDiHj/pAMj4h31WLZsWTTqvvvua3jddqhSv1XqNaJa/Vap14hq9VulXiOa6xd4JDJ+x/r0lJmZZXNomJlZNoeGmZllc2iYmVk2h4aZmWVzaJiZWTaHhpmZZZs0NCRtl3RC0tN1lv2JpJB0QZqXpFskDUp6StLS0tg+SYfSo69UXyZpIK1ziySl+hxJ+9P4/ZJmn5m3bGZmjco50rgN6B1blLQA+F3gxVL5EqArPTYCt6axc4AtwMXAcmBLKQRuTWNH1xt9rX7gQER0AQfSvJmZtdGkoRER9wMn6yy6GfhTIEq1NcDO9AHDB4FZkuYCq4H9EXEyIk4B+4HetOy8iHggfSJxJ3B5aVs70vSOUt2Ahf17GHj5NAv797S7FTN7F2nomoaky4CXI+LJMYvmAS+V5odSbaL6UJ06QGdEHANIzxc20quZmZ05U/7CQknvBa4DVtVbXKcWDdSn2tNGilNcdHZ2UqvVproJAIaHhxtet9U2Lxmhc2bxXIWeq7RvoVr9VqlXqFa/VeoVWtNvI99y++vAIuDJdM16PvCYpOUURwoLSmPnA0dTvWdMvZbq8+uMBzguaW5EHEunsU6M11BEbAW2AnR3d0dPT894QydUq9VodN1Wu6p/D5uXjHDTQAdH1vW0u51JVWnfQrX6rVKvUK1+q9QrtKbfKZ+eioiBiLgwIhZGxEKKX/xLI+IHwG5gfbqLagVwOp1a2geskjQ7XQBfBexLy16VtCLdNbUeuCe91G5g9C6rvlLdzMzaJOeW2zuAB4APSRqStGGC4XuBw8Ag8NfAvweIiJPAl4CH0+OLqQbwWeDraZ3ngXtT/QbgdyUdorhL64apvTUzMzvTJj09FRFXTrJ8YWk6gE3jjNsObK9TfwS4qE79R8DKyfozM7PW8SfCzcwsm0PDzMyyOTTMzCybQ8PMzLI5NMzMLJtDw8zMsjk0zMwsm0PDzMyyOTTMzCybQ8PMzLI5NMzMLJtDw8zMsjk0zMwsm0PDzMyyOTTMzCybQ8PMzLI5NMzMLJtDw8zMsjk0zMwsm0PDzMyyTRoakrZLOiHp6VLtzyV9T9JTkr4laVZp2bWSBiU9J2l1qd6baoOS+kv1RZIeknRI0p2Szkr1s9P8YFq+8Ey9aTMza0zOkcZtQO+Y2n7gooj4TeD7wLUAkhYDa4EPp3W+JmmGpBnAV4FLgMXAlWkswI3AzRHRBZwCNqT6BuBURHwQuDmNMzOzNpo0NCLifuDkmNq3I2IkzT4IzE/Ta4BdEfHziHgBGASWp8dgRByOiNeBXcAaSQI+Adyd1t8BXF7a1o40fTewMo03M7M2ORPXNP4IuDdNzwNeKi0bSrXx6u8DXikF0Gj9TdtKy0+n8WZm1iYdzaws6TpgBLh9tFRnWFA/nGKC8RNtq14fG4GNAJ2dndRqtfGbnsDw8HDD67ba5iUjdM4snqvQc5X2LVSr3yr1CtXqt0q9Qmv6bTg0JPUBnwJWRsToL/MhYEFp2HzgaJquV/8hMEtSRzqaKI8f3daQpA7gfMacJhsVEVuBrQDd3d3R09PT0Huq1Wo0um6rXdW/h81LRrhpoIMj63ra3c6kqrRvoVr9VqlXqFa/VeoVWtNvQ6enJPUC1wCXRcRrpUW7gbXpzqdFQBfwXeBhoCvdKXUWxcXy3Sls7gM+ndbvA+4pbasvTX8a+E4pnMzMrA0mPdKQdAfQA1wgaQjYQnG31NnA/nRt+sGI+OOIOCjpLuAZitNWmyLiF2k7VwP7gBnA9og4mF7iGmCXpC8DjwPbUn0b8A1JgxRHGGvPwPs1M7MmTBoaEXFlnfK2OrXR8dcD19ep7wX21qkfpri7amz9Z8AVk/VnZmat40+Em5lZNoeGmZllc2iYmVk2h4aZmWVzaJiZWTaHhpmZZXNomJlZNoeGmZllc2iYmVk2h4aZmWVzaJiZWTaHhpmZZXNomJlZNoeGmZllc2iYmVk2h4aZmWVzaJiZWTaHhpmZZXNomJlZNoeGmZllc2iYmVm2SUND0nZJJyQ9XarNkbRf0qH0PDvVJekWSYOSnpK0tLROXxp/SFJfqb5M0kBa5xZJmug1zMysfXKONG4DesfU+oEDEdEFHEjzAJcAXemxEbgVigAAtgAXA8uBLaUQuDWNHV2vd5LXMDOzNpk0NCLifuDkmPIaYEea3gFcXqrvjMKDwCxJc4HVwP6IOBkRp4D9QG9adl5EPBARAewcs616r2FmZm3S0eB6nRFxDCAijkm6MNXnAS+Vxg2l2kT1oTr1iV7jLSRtpDhaobOzk1qt1tCbGh4ebnjdVtu8ZITOmcVzFXqu0r6FavVbpV6hWv1WqVdoTb+NhsZ4VKcWDdSnJCK2AlsBuru7o6enZ6qbAKBWq9Houq12Vf8eNi8Z4aaBDo6s62l3O5Oq0r6FavVbpV6hWv1WqVdoTb+N3j11PJ1aIj2fSPUhYEFp3Hzg6CT1+XXqE72GmZm1SaOhsRsYvQOqD7inVF+f7qJaAZxOp5j2AaskzU4XwFcB+9KyVyWtSHdNrR+zrXqvYWZmbTLp6SlJdwA9wAWShijugroBuEvSBuBF4Io0fC9wKTAIvAZ8BiAiTkr6EvBwGvfFiBi9uP5Ziju0ZgL3pgcTvIaZmbXJpKEREVeOs2hlnbEBbBpnO9uB7XXqjwAX1an/qN5rmJlZ+/gT4WZmls2hYWZm2RwaZmaWzaFhZmbZHBpmZpbNoWFmZtkcGmZmls2hYWZm2RwaZmaWzaFhZmbZHBpmZpbNoWFmZtkcGmZmls2hYWZm2RwaZmaWzaFhZmbZHBpmZpbNoWFmZtkcGmZmls2hYWZm2ZoKDUn/UdJBSU9LukPSeyQtkvSQpEOS7pR0Vhp7dpofTMsXlrZzbao/J2l1qd6baoOS+pvp1czMmtdwaEiaB/wHoDsiLgJmAGuBG4GbI6ILOAVsSKtsAE5FxAeBm9M4JC1O630Y6AW+JmmGpBnAV4FLgMXAlWmsmZm1SbOnpzqAmZI6gPcCx4BPAHen5TuAy9P0mjRPWr5SklJ9V0T8PCJeAAaB5ekxGBGHI+J1YFcaa2ZmbdJwaETEy8B/B16kCIvTwKPAKxExkoYNAfPS9DzgpbTuSBr/vnJ9zDrj1c3MrE06Gl1R0myKv/wXAa8Af0txKmmsGF1lnGXj1esFWtSpIWkjsBGgs7OTWq02UevjGh4ebnjdVtu8ZITOmcVzFXqu0r6FavVbpV6hWv1WqVdoTb8NhwbwSeCFiPgHAEnfBP4lMEtSRzqamA8cTeOHgAXAUDqddT5wslQfVV5nvPqbRMRWYCtAd3d39PT0NPSGarUaja7balf172HzkhFuGujgyLqedrczqSrtW6hWv1XqFarVb5V6hdb028w1jReBFZLem65NrASeAe4DPp3G9AH3pOndaZ60/DsREam+Nt1dtQjoAr4LPAx0pbuxzqK4WL67iX7NzKxJDR9pRMRDku4GHgNGgMcp/trfA+yS9OVU25ZW2QZ8Q9IgxRHG2rSdg5LuogicEWBTRPwCQNLVwD6KO7O2R8TBRvs1M7PmNXN6iojYAmwZUz5McefT2LE/A64YZzvXA9fXqe8F9jbTo5mZnTn+RLiZmWVzaJiZWTaHhpmZZXNomJlZNoeGmZllc2iYmVk2h4aZmWVzaJiZWTaHhpmZZXNomJlZNoeGmZllc2iYmVk2h4aZmWVzaJiZWTaHhpmZZXNomJlZNoeGmZllc2iYmVk2h4aZmWVzaJiZWbamQkPSLEl3S/qepGcl/Y6kOZL2SzqUnmensZJ0i6RBSU9JWlraTl8af0hSX6m+TNJAWucWSWqmXzMza06zRxpfAf5XRPwz4LeAZ4F+4EBEdAEH0jzAJUBXemwEbgWQNAfYAlwMLAe2jAZNGrOxtF5vk/2amVkTGg4NSecBHwO2AUTE6xHxCrAG2JGG7QAuT9NrgJ1ReBCYJWkusBrYHxEnI+IUsB/oTcvOi4gHIiKAnaVtmZlZGzRzpPEB4B+A/ynpcUlfl3QO0BkRxwDS84Vp/DzgpdL6Q6k2UX2oTt3MzNqko8l1lwKfi4iHJH2FX56Kqqfe9YhooP7WDUsbKU5j0dnZSa1Wm6CN8Q0PDze8bqttXjJC58ziuQo9V2nfQrX6rVKvUK1+q9QrtKbfZkJjCBiKiIfS/N0UoXFc0tyIOJZOMZ0ojV9QWn8+cDTVe8bUa6k+v874t4iIrcBWgO7u7ujp6ak3bFK1Wo1G1221q/r3sHnJCDcNdHBkXU+725lUlfYtVKvfKvUK1eq3Sr1Ca/pt+PRURPwAeEnSh1JpJfAMsBsYvQOqD7gnTe8G1qe7qFYAp9Ppq33AKkmz0wXwVcC+tOxVSSvSXVPrS9syM7M2aOZIA+BzwO2SzgIOA5+hCKK7JG0AXgSuSGP3ApcCg8BraSwRcVLSl4CH07gvRsTJNP1Z4DZgJnBvepiZWZs0FRoR8QTQXWfRyjpjA9g0zna2A9vr1B8BLmqmRzMzO3P8iXAzM8vm0DAzs2wODTMzy+bQMDOzbA4NMzPL5tAwM7NsDg0zM8vm0DAzs2wODTMzy+bQMDOzbA4NMzPL5tAwM7NsDg0zM8vm0DAzs2wODTMzy+bQMDOzbA4NMzPL5tAwM7NsDg0zM8vm0DAzs2xNh4akGZIel/T3aX6RpIckHZJ0p6SzUv3sND+Yli8sbePaVH9O0upSvTfVBiX1N9urmZk150wcaXweeLY0fyNwc0R0AaeADam+ATgVER8Ebk7jkLQYWAt8GOgFvpaCaAbwVeASYDFwZRprZmZt0lRoSJoP/B7w9TQv4BPA3WnIDuDyNL0mzZOWr0zj1wC7IuLnEfECMAgsT4/BiDgcEa8Du9JYMzNrk2aPNP4S+FPgH9P8+4BXImIkzQ8B89L0POAlgLT8dBr/Rn3MOuPVzcysTToaXVHSp4ATEfGopJ7Rcp2hMcmy8er1Ai3q1JC0EdgI0NnZSa1WG7/xCQwPDze8bqttXjJC58ziuQo9V2nfQrX6rVKvUK1+q9QrtKbfhkMD+AhwmaRLgfcA51EcecyS1JGOJuYDR9P4IWABMCSpAzgfOFmqjyqvM179TSJiK7AVoLu7O3p6ehp6Q7VajUbXbbWr+veweckINw10cGRdT7vbmVSV9i1Uq98q9QrV6rdKvUJr+m349FREXBsR8yNiIcWF7O9ExDrgPuDTaVgfcE+a3p3mScu/ExGR6mvT3VWLgC7gu8DDQFe6G+us9Bq7G+3XzMya18yRxniuAXZJ+jLwOLAt1bcB35A0SHGEsRYgIg5Kugt4BhgBNkXELwAkXQ3sA2YA2yPi4NvQr5mZZTojoRERNaCWpg9T3Pk0dszPgCvGWf964Po69b3A3jPRo5mZNc+fCDczs2wODTMzy+bQMDOzbA4NMzPL5tAwM7NsDg0zM8vm0DAzs2wODTMzy+bQMDOzbA4NMzPL5tAwM7NsDg0zM8v2dnzL7TvOwv49b0wfueH32tiJmVl7+UjDzMyyOTTMzCybQ8PMzLI5NMzMLJtDw8zMsjk0zMwsm0PDzMyyOTTMzCxbw6EhaYGk+yQ9K+mgpM+n+hxJ+yUdSs+zU12SbpE0KOkpSUtL2+pL4w9J6ivVl0kaSOvcIknNvFkzM2tOM0caI8DmiPjnwApgk6TFQD9wICK6gANpHuASoCs9NgK3QhEywBbgYmA5sGU0aNKYjaX1epvo9221sH/PGw8zs3eqhkMjIo5FxGNp+lXgWWAesAbYkYbtAC5P02uAnVF4EJglaS6wGtgfEScj4hSwH+hNy86LiAciIoCdpW2ZmVkbqPh93ORGpIXA/cBFwIsRMau07FREzJb098ANEfF/Uv0AcA3QA7wnIr6c6v8F+ClQS+M/meofBa6JiE/Vef2NFEckdHZ2Ltu1a1dD72N4eJhzzz33LfWBl0+/Mb1k3vl1180ZcyYNvHyazplw/Keteb1mjbdvp6sq9VulXqFa/VapV2iu349//OOPRkT3ZOOa/sJCSecCfwd8ISJ+PMFlh3oLooH6W4sRW4GtAN3d3dHT0zNJ1/XVajXqrXtV+QsL19Xfds6YM+mq/j1sXjLCTQMdLXm9Zo23b6erKvVbpV6hWv1WqVdoTb9NhYakX6EIjNsj4pupfFzS3Ig4lk4xnUj1IWBBafX5wNFU7xlTr6X6/Drj3/H8rbpmNl01c/eUgG3AsxHxF6VFu4HRO6D6gHtK9fXpLqoVwOmIOAbsA1ZJmp0ugK8C9qVlr0pakV5rfWlbZmbWBs0caXwE+ENgQNITqfafgRuAuyRtAF4ErkjL9gKXAoPAa8BnACLipKQvAQ+ncV+MiJNp+rPAbcBM4N70MDOzNmk4NNIF7fEuYKysMz6ATeNsazuwvU79EYqL62ZmNg34E+FmZpbNoWFmZtkcGmZmls2hYWZm2RwaZmaWzaFhZmbZHBpmZpat6e+eMpvI6FeibF4y8qbvijGzavKRhpmZZXNomJlZNoeGmZllc2iYmVk2h4ZZxQ28fNr/Pr21jEPDzMyy+ZbbdzD/C4Bmdqb5SMPMzLI5NMzMLJtDw8zMsjk0zMwsm0PDzMyyTfu7pyT1Al8BZgBfj4gbWvG6vufdzOytpvWRhqQZwFeBS4DFwJWSFre3KzOzd6/pfqSxHBiMiMMAknYBa4Bn2trVFIw9YvHnJcysyqZ7aMwDXirNDwEXt6kXe4crB/xtvee0sRNrN/87MONTRLS7h3FJugJYHRH/Ls3/IbA8Ij43ZtxGYGOa/RDwXIMveQHwwwbXbYcq9VulXqFa/VapV6hWv1XqFZrr99ci4v2TDZruRxpDwILS/Hzg6NhBEbEV2Nrsi0l6JCK6m91Oq1Sp3yr1CtXqt0q9QrX6rVKv0Jp+p/WFcOBhoEvSIklnAWuB3W3uyczsXWtaH2lExIikq4F9FLfcbo+Ig21uy8zsXWtahwZAROwF9rbo5Zo+xdViVeq3Sr1CtfqtUq9QrX6r1Cu0oN9pfSHczMyml+l+TcPMzKYRh0YiqVfSc5IGJfVPg34WSLpP0rOSDkr6fKr/maSXJT2RHpeW1rk29f+cpNVt6PmIpIHU1yOpNkfSfkmH0vPsVJekW1K/T0la2sI+P1Taf09I+rGkL0ynfStpu6QTkp4u1aa8LyX1pfGHJPW1sNc/l/S91M+3JM1K9YWSflrax39VWmdZ+u9nML0ftbDfKf/sW/E7Y5xe7yz1eUTSE6nemn0bEe/6B8VF9ueBDwBnAU8Ci9vc01xgaZr+VeD7FF+l8mfAn9QZvzj1fTawKL2fGS3u+QhwwZjafwP603Q/cGOavhS4FxCwAniojT/7HwC/Np32LfAxYCnwdKP7EpgDHE7Ps9P07Bb1ugroSNM3lnpdWB43ZjvfBX4nvY97gUtauG+n9LNv1e+Mer2OWX4T8F9buW99pFF44+tKIuJ1YPTrStomIo5FxGNp+lXgWYpPyI9nDbArIn4eES8AgxTvq93WADvS9A7g8lJ9ZxQeBGZJmtuG/lYCz0fE/51gTMv3bUTcD5ys08dU9uVqYH9EnIyIU8B+oLcVvUbEtyNiJM0+SPEZq3Glfs+LiAei+C23k1++v7e93wmM97Nvye+MiXpNRwv/Frhjom2c6X3r0CjU+7qSiX5Bt5SkhcBvAw+l0tXpsH/76CkKpsd7CODbkh5V8Sl9gM6IOAZFEAIXpvp06BeKz/6U/6ebrvsWpr4vp0vff0Tx1+2oRZIel/S/JX001eZR9DeqHb1O5Wc/HfbtR4HjEXGoVHvb961Do1Dv/N60uK1M0rnA3wFfiIgfA7cCvw78C+AYxeEpTI/38JGIWErxrcSbJH1sgrFt71fFB0YvA/42labzvp3IeP21vW9J1wEjwO2pdAz4pxHx28B/Av5G0nm0v9ep/uzb3S/Albz5D56W7FuHRiHr60paTdKvUATG7RHxTYCIOB4Rv4iIfwT+ml+eJmn7e4iIo+n5BPCt1Nvx0dNO6flEGt72finC7bGIOA7Te98mU92Xbe07XXj/FLAunRYhneb5UZp+lOK6wG+kXsunsFraawM/+3bv2w7gXwN3jtZatW8dGoVp93Ul6XzlNuDZiPiLUr183v8PgNG7KnYDayWdLWkR0EVx8atV/Z4j6VdHpykuhD6d+hq9a6cPuKfU7/p0588K4PToqZcWetNfatN135ZMdV/uA1ZJmp1Ot6xKtbedin887Rrgsoh4rVR/v4p/JwdJH6DYl4dTv69KWpH+219fen+t6HeqP/t2/874JPC9iHjjtFPL9u2Zvtpf1QfFHSjfp0jn66ZBP/+K4hDyKeCJ9LgU+AYwkOq7gbmlda5L/T/H23TnyQT9foDiDpIngYOj+xB4H3AAOJSe56S6KP6BrefT++lucb/vBX4EnF+qTZt9SxFmx4D/R/GX4oZG9iXF9YTB9PhMC3sdpDjnP/rf7l+lsf8m/ffxJPAY8Pul7XRT/LJ+HvgfpA8ft6jfKf/sW/E7o16vqX4b8MdjxrZk3/oT4WZmls2np8zMLJtDw8zMsjk0zMwsm0PDzMyyOTTMzCybQ8PMzLI5NMzMLJtDw8zMsv1/tEOLPZado7QAAAAASUVORK5CYII=\n"
            ],
            "text/plain": [
              "<matplotlib.figure.Figure at 0x12974ecc0>"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 7,
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "Непохоже, что здесь есть выбросы (кроме нулевых штрафов), но можно категоризировать данные, заменив все значения больше 2000 на переменную `>2000` и сделав все остальные значения категориальными."
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Поиск по отклонению (z-score)\n",
        "[Источник](http://colingorrie.github.io/outlier-detection.html)\n",
        "\n",
        "Есть еще один способ поиска выбросов, не требующий настолько же пристального взгляда на данные, что и предыдущий метод.  \n",
        "Он основан на поиске значения, на которое отклоняется данная точка от среднего. Делается это с помощью z-score, то есть, расстояния (измеряемого в количестве стандартных отклонений) от среднего значения точек.  \n",
        "Для начала посмотрим на распределение данных для признака `distance`, который мы посчитали ранее:"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "import warnings"
      ],
      "outputs": [],
      "execution_count": 8,
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "vis_data.distance.hist(bins=100);"
      ],
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": [
              "iVBORw0KGgoAAAANSUhEUgAAAYcAAAD8CAYAAACcjGjIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAEypJREFUeJzt3W2MnXWZx/Hvta08LD6UB5k0bbODoS9AWREnUMO+GMEtBYzlBSQQshTTZBKDCSYkbtlNlvhAAi9WDBsl2ywNxbgiqxIaqFubwslmE3loBSm1sh2xK7NtaNyWymjErV774vynOfZ/pnN6Wntm7n4/ycm57+v+3/f5Xzrwm/thDpGZSJLU6c8GPQFJ0uxjOEiSKoaDJKliOEiSKoaDJKliOEiSKoaDJKliOEiSKoaDJKkyf9AT6Nd5552Xw8PDfe3761//mrPOOuvETmgWaGpf0NzemtoX2NtstG3btl9m5vt7GTtnw2F4eJitW7f2tW+r1WJ0dPTETmgWaGpf0NzemtoX2NtsFBH/3etYLytJkiqGgySpYjhIkiqGgySpYjhIkiqGgySpYjhIkiqGgySpYjhIkipz9i+k/xSG1zx9eHn3fdcPcCaSNFieOUiSKoaDJKliOEiSKoaDJKliOEiSKoaDJKnSUzhExO6I2B4RL0fE1lI7JyI2R8Su8n52qUdEPBgR4xHxSkRc1nGcVWX8rohY1VH/aDn+eNk3TnSjkqTeHcuZw8cz89LMHCnra4AtmbkU2FLWAa4FlpbXGPAQtMMEuAe4ArgcuGcqUMqYsY79VvTdkSTpuB3PZaWVwPqyvB64oaP+aLY9ByyIiIXANcDmzNyfmQeAzcCKsu29mfnDzEzg0Y5jSZIGoNdwSOAHEbEtIsZKbSgz9wKU9/NLfRHwRse+E6V2tPpEl7okaUB6/fqMKzNzT0ScD2yOiJ8eZWy3+wXZR70+cDuYxgCGhoZotVpHnfR0Jicnu+571yWHDi/3e+xBmq6vJmhqb03tC+xtruspHDJzT3nfFxFP0L5n8GZELMzMveXS0L4yfAJY0rH7YmBPqY8eUW+V+uIu47vNYy2wFmBkZCRHR0e7DZtRq9Wi2763d3630q39HXuQpuurCZraW1P7Anub62a8rBQRZ0XEe6aWgeXAq8AGYOqJo1XAk2V5A3BbeWppGXCwXHbaBCyPiLPLjejlwKay7e2IWFaeUrqt41iSpAHo5cxhCHiiPF06H/jXzPz3iHgReDwiVgO/AG4q4zcC1wHjwG+ATwNk5v6I+BLwYhn3xczcX5Y/AzwCnAl8v7wkSQMyYzhk5uvAh7vU/xe4uks9gTumOdY6YF2X+lbgQz3MV5J0EvgX0pKkiuEgSaoYDpKkiuEgSaoYDpKkiuEgSaoYDpKkiuEgSaoYDpKkiuEgSaoYDpKkiuEgSaoYDpKkiuEgSaoYDpKkiuEgSaoYDpKkiuEgSaoYDpKkiuEgSaoYDpKkiuEgSarMH/QEBm14zdODnoIkzTqeOUiSKoaDJKliOEiSKoaDJKliOEiSKoaDJKliOEiSKj2HQ0TMi4iXIuKpsn5BRDwfEbsi4tsRcVqpn17Wx8v24Y5j3F3qr0XENR31FaU2HhFrTlx7kqR+HMuZw53Azo71+4EHMnMpcABYXeqrgQOZeSHwQBlHRFwM3Ax8EFgBfL0Ezjzga8C1wMXALWWsJGlAegqHiFgMXA/8S1kP4CrgO2XIeuCGsryyrFO2X13GrwQey8x3MvPnwDhweXmNZ+brmfk74LEyVpI0IL2eOXwV+Dzwh7J+LvBWZh4q6xPAorK8CHgDoGw/WMYfrh+xz3R1SdKAzPjdShHxSWBfZm6LiNGpcpehOcO26erdAiq71IiIMWAMYGhoiFarNf3Ej2JycvLwvnddcqjrmH6PPUidfTVNU3tral9gb3NdL1+8dyXwqYi4DjgDeC/tM4kFETG/nB0sBvaU8RPAEmAiIuYD7wP2d9SndO4zXf2PZOZaYC3AyMhIjo6O9jD9WqvVYmrf26f54r3dt/Z37EHq7KtpmtpbU/sCe5vrZryslJl3Z+bizBymfUP5mcy8FXgWuLEMWwU8WZY3lHXK9mcyM0v95vI00wXAUuAF4EVgaXn66bTyGRtOSHeSpL4cz1d2/y3wWER8GXgJeLjUHwa+ERHjtM8YbgbIzB0R8TjwE+AQcEdm/h4gIj4LbALmAesyc8dxzEuSdJyOKRwyswW0yvLrtJ80OnLMb4Gbptn/XuDeLvWNwMZjmYsk6U/Hv5CWJFUMB0lSxXCQJFUMB0lSxXCQJFUMB0lSxXCQJFUMB0lSxXCQJFUMB0lSxXCQJFUMB0lSxXCQJFUMB0lSxXCQJFUMB0lSxXCQJFUMB0lSxXCQJFUMB0lSxXCQJFUMB0lSxXCQJFUMB0lSxXCQJFUMB0lSxXCQJFUMB0lSxXCQJFUMB0lSxXCQJFVmDIeIOCMiXoiIH0fEjoj4QqlfEBHPR8SuiPh2RJxW6qeX9fGyfbjjWHeX+msRcU1HfUWpjUfEmhPfpiTpWPRy5vAOcFVmfhi4FFgREcuA+4EHMnMpcABYXcavBg5k5oXAA2UcEXExcDPwQWAF8PWImBcR84CvAdcCFwO3lLGSpAGZMRyybbKsvqu8ErgK+E6prwduKMsryzpl+9UREaX+WGa+k5k/B8aBy8trPDNfz8zfAY+VsZKkAZnfy6Dy2/024ELav+X/DHgrMw+VIRPAorK8CHgDIDMPRcRB4NxSf67jsJ37vHFE/Ypp5jEGjAEMDQ3RarV6mX5lcnLy8L53XXKo65h+jz1InX01TVN7a2pfYG9zXU/hkJm/By6NiAXAE8BF3YaV95hm23T1bmcv2aVGZq4F1gKMjIzk6Ojo0Sc+jVarxdS+t695uuuY3bf2d+xB6uyraZraW1P7Anub647paaXMfAtoAcuABRExFS6LgT1leQJYAlC2vw/Y31k/Yp/p6pKkAenlaaX3lzMGIuJM4BPATuBZ4MYybBXwZFneUNYp25/JzCz1m8vTTBcAS4EXgBeBpeXpp9No37TecCKakyT1p5fLSguB9eW+w58Bj2fmUxHxE+CxiPgy8BLwcBn/MPCNiBinfcZwM0Bm7oiIx4GfAIeAO8rlKiLis8AmYB6wLjN3nLAOJUnHbMZwyMxXgI90qb9O+0mjI+u/BW6a5lj3Avd2qW8ENvYwX0nSSeBfSEuSKoaDJKliOEiSKoaDJKliOEiSKoaDJKliOEiSKoaDJKliOEiSKoaDJKliOEiSKoaDJKliOEiSKoaDJKliOEiSKoaDJKliOEiSKoaDJKliOEiSKoaDJKliOEiSKoaDJKliOEiSKoaDJKliOEiSKoaDJKliOEiSKoaDJKliOEiSKoaDJKkyYzhExJKIeDYidkbEjoi4s9TPiYjNEbGrvJ9d6hERD0bEeES8EhGXdRxrVRm/KyJWddQ/GhHbyz4PRkT8KZqVJPWmlzOHQ8BdmXkRsAy4IyIuBtYAWzJzKbClrANcCywtrzHgIWiHCXAPcAVwOXDPVKCUMWMd+604/tYkSf2aMRwyc29m/qgsvw3sBBYBK4H1Zdh64IayvBJ4NNueAxZExELgGmBzZu7PzAPAZmBF2fbezPxhZibwaMexJEkDMP9YBkfEMPAR4HlgKDP3QjtAIuL8MmwR8EbHbhOldrT6RJd6t88fo32GwdDQEK1W61imf9jk5OThfe+65FDXMf0ee5A6+2qapvbW1L7A3ua6nsMhIt4NfBf4XGb+6ii3BbptyD7qdTFzLbAWYGRkJEdHR2eYdXetVoupfW9f83TXMbtv7e/Yg9TZV9M0tbem9gX2Ntf19LRSRLyLdjB8MzO/V8pvlktClPd9pT4BLOnYfTGwZ4b64i51SdKA9PK0UgAPAzsz8ysdmzYAU08crQKe7KjfVp5aWgYcLJefNgHLI+LsciN6ObCpbHs7IpaVz7qt41iSpAHo5bLSlcDfANsj4uVS+zvgPuDxiFgN/AK4qWzbCFwHjAO/AT4NkJn7I+JLwItl3Bczc39Z/gzwCHAm8P3ykiQNyIzhkJn/Sff7AgBXdxmfwB3THGsdsK5LfSvwoZnmIkk6OfwLaUlSxXCQJFUMB0lSxXCQJFUMB0lSxXCQJFUMB0lSxXCQJFUMB0lSxXCQJFUMB0lSxXCQJFUMB0lSxXCQJFUMB0lSxXCQJFUMB0lSpZf/TOgpaXjN04eXd993/QBnIkknn2cOkqSK4SBJqhgOkqSK4SBJqhgOkqSK4SBJqhgOkqSK4SBJqhgOkqSK4SBJqhgOkqTKKfndStv/5yC3d3x3kiTpj8145hAR6yJiX0S82lE7JyI2R8Su8n52qUdEPBgR4xHxSkRc1rHPqjJ+V0Ss6qh/NCK2l30ejIg40U1Kko5NL5eVHgFWHFFbA2zJzKXAlrIOcC2wtLzGgIegHSbAPcAVwOXAPVOBUsaMdex35GdJkk6yGcMhM/8D2H9EeSWwviyvB27oqD+abc8BCyJiIXANsDkz92fmAWAzsKJse29m/jAzE3i041iSpAHp94b0UGbuBSjv55f6IuCNjnETpXa0+kSXuiRpgE70Delu9wuyj3r3g0eM0b4ExdDQEK1Wq48pwtCZcNclh3oe3+/nnGyTk5NzZq7Hqqm9NbUvsLe5rt9weDMiFmbm3nJpaF+pTwBLOsYtBvaU+ugR9VapL+4yvqvMXAusBRgZGcnR0dHphh7VP33zSf5xe++t7761v8852VqtFv3+bzLbNbW3pvYF9jbX9XtZaQMw9cTRKuDJjvpt5amlZcDBctlpE7A8Is4uN6KXA5vKtrcjYll5Sum2jmNJkgZkxl+fI+JbtH/rPy8iJmg/dXQf8HhErAZ+AdxUhm8ErgPGgd8AnwbIzP0R8SXgxTLui5k5dZP7M7SfiDoT+H55SZIGaMZwyMxbptl0dZexCdwxzXHWAeu61LcCH5ppHpKkk8evz5AkVQwHSVLFcJAkVQwHSVLFcJAkVQwHSVLFcJAkVQwHSVLFcJAkVQwHSVLFcJAkVQwHSVLFcJAkVQwHSVLFcJAkVQwHSVLFcJAkVQwHSVLFcJAkVQwHSVLFcJAkVQwHSVLFcJAkVQwHSVLFcJAkVQwHSVLFcJAkVQwHSVLFcJAkVQwHSVJl1oRDRKyIiNciYjwi1gx6PpJ0KpsV4RAR84CvAdcCFwO3RMTFg52VJJ26ZkU4AJcD45n5emb+DngMWDngOUnSKWv+oCdQLALe6FifAK4Y0Fwqw2uePry8+77rBzgTSTo5Zks4RJdaVoMixoCxsjoZEa/1+XnnAb/sZ8e4v89PPDn67msOaGpvTe0L7G02+oteB86WcJgAlnSsLwb2HDkoM9cCa4/3wyJia2aOHO9xZpum9gXN7a2pfYG9zXWz5Z7Di8DSiLggIk4DbgY2DHhOknTKmhVnDpl5KCI+C2wC5gHrMnPHgKclSaesWREOAJm5Edh4kj7uuC9NzVJN7Qua21tT+wJ7m9Mis7rvK0k6xc2Wew6SpFnklAqHuf4VHRGxLiL2RcSrHbVzImJzROwq72eXekTEg6XXVyLissHN/OgiYklEPBsROyNiR0TcWepN6O2MiHghIn5cevtCqV8QEc+X3r5dHsQgIk4v6+Nl+/Ag5z+TiJgXES9FxFNlvSl97Y6I7RHxckRsLbU5//N4LE6ZcGjIV3Q8Aqw4orYG2JKZS4EtZR3afS4trzHgoZM0x34cAu7KzIuAZcAd5f+bJvT2DnBVZn4YuBRYERHLgPuBB0pvB4DVZfxq4EBmXgg8UMbNZncCOzvWm9IXwMcz89KOR1ab8PPYu8w8JV7Ax4BNHet3A3cPel599DEMvNqx/hqwsCwvBF4ry/8M3NJt3Gx/AU8Cf9203oA/B35E+6//fwnML/XDP5u0n9j7WFmeX8bFoOc+TT+Laf9L8irgKdp/zDrn+ypz3A2cd0StUT+PM71OmTMHun9Fx6IBzeVEGsrMvQDl/fxSn5P9lssNHwGepyG9lUsvLwP7gM3Az4C3MvNQGdI5/8O9le0HgXNP7ox79lXg88Afyvq5NKMvaH9Dww8iYlv5ZgZoyM9jr2bNo6wnQU9f0dEgc67fiHg38F3gc5n5q4huLbSHdqnN2t4y8/fApRGxAHgCuKjbsPI+J3qLiE8C+zJzW0SMTpW7DJ1TfXW4MjP3RMT5wOaI+OlRxs613npyKp059PQVHXPQmxGxEKC87yv1OdVvRLyLdjB8MzO/V8qN6G1KZr4FtGjfV1kQEVO/nHXO/3BvZfv7gP0nd6Y9uRL4VETspv0tylfRPpOY630BkJl7yvs+2oF+OQ37eZzJqRQOTf2Kjg3AqrK8ivb1+qn6beVJimXAwalT4tkm2qcIDwM7M/MrHZua0Nv7yxkDEXEm8AnaN3CfBW4sw47sbarnG4FnslzInk0y8+7MXJyZw7T/WXomM29ljvcFEBFnRcR7ppaB5cCrNODn8ZgM+qbHyXwB1wH/Rfua798Pej59zP9bwF7g/2j/trKa9nXbLcCu8n5OGRu0n876GbAdGBn0/I/S11/RPg1/BXi5vK5rSG9/CbxUensV+IdS/wDwAjAO/BtweqmfUdbHy/YPDLqHHnocBZ5qSl+lhx+X146pf1c04efxWF7+hbQkqXIqXVaSJPXIcJAkVQwHSVLFcJAkVQwHSVLFcJAkVQwHSVLFcJAkVf4fGHv5thE7EW4AAAAASUVORK5CYII=\n"
            ],
            "text/plain": [
              "<matplotlib.figure.Figure at 0x129780a90>"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 9,
      "metadata": {
        "scrolled": true
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Попытаемся \"на глаз\" определить порог, после которого мы считаем выбросы. Применим этот порог и посмотрим на количество \"выбросов\":"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "vis_data.distance[vis_data.distance > 30].hist();"
      ],
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": [
              "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD8CAYAAABn919SAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAEchJREFUeJzt3X+MZXdZx/H3h7ZQ7GBbaJms28bB0BAIK8VOakn9Y6b8sFIDJSkJTYMl1owmQGps1FUTBZGkJEqNCTFubGn/QAbkR0q2VWhKR0JiirOwdLeupAVX7LayqbQrQ0x15fGPOYvjMtt75s69c7vfeb+Sk3vP93zPvc+TTj57eu6596SqkCSd/p436QIkSaNhoEtSIwx0SWqEgS5JjTDQJakRBrokNcJAl6RGGOiS1AgDXZIaceZWvtkFF1xQMzMzW/mWW+b73/8+55xzzqTLGDv7bMt26LOFHvft2/dkVV04aN6WBvrMzAzLy8tb+ZZbZmlpibm5uUmXMXb22Zbt0GcLPSb5lz7zPOUiSY0w0CWpEQa6JDXCQJekRhjoktQIA12SGmGgS1IjDHRJaoSBLkmN2NJvim7GzO57Jvbeh2+9ZmLvLUl9eYQuSY0w0CWpEQa6JDXCQJekRhjoktSI3oGe5IwkX0uyt1t/WZIHkzyS5BNJnj++MiVJg2zkCP1m4NCa9Q8Bt1XVJcBTwE2jLEyStDG9Aj3JRcA1wF926wGuAj7VTbkLuHYcBUqS+ul7hP6nwG8BP+jWXwI8XVXHu/XHgJ0jrk2StAEDvyma5BeBo1W1L8ncieF1ptYp9l8AFgCmp6dZWloaqtBbdh0fPGlM+tS8srIydG+nE/tsy3boczv0eEKfr/5fCbwlyZuBs4EfZ/WI/bwkZ3ZH6RcBj6+3c1XtAfYAzM7O1rA3a33XJL/6f8PcwDkt3Ii2D/tsy3boczv0eMLAUy5V9TtVdVFVzQDvAL5YVTcADwDXddNuBO4eW5WSpIE2cx36bwO/keRRVs+p3z6akiRJw9jQry1W1RKw1D3/FnD56EuSJA3Db4pKUiMMdElqhIEuSY0w0CWpEQa6JDXCQJekRhjoktQIA12SGmGgS1IjDHRJaoSBLkmNMNAlqREGuiQ1wkCXpEYY6JLUCANdkhoxMNCTnJ3kK0m+nuThJO/vxu9M8s9J9nfLpeMvV5J0Kn3uWPQMcFVVrSQ5C/hykr/ptv1mVX1qfOVJkvoaGOhVVcBKt3pWt9Q4i5IkbVxW83rApOQMYB/wcuAjVfXbSe4EXsfqEfz9wO6qemadfReABYDp6enLFhcXhyr0wJFjQ+03Crt2njtwzsrKClNTU1tQzWTZZ1u2Q58t9Dg/P7+vqmYHzesV6D+cnJwHfBZ4L/DvwL8Bzwf2AN+sqj98tv1nZ2dreXm59/utNbP7nqH2G4XDt14zcM7S0hJzc3PjL2bC7LMt26HPFnpM0ivQN3SVS1U9DSwBV1fVE7XqGeCjwOVDVSpJGok+V7lc2B2Zk+SFwBuAf0qyoxsLcC1wcJyFSpKeXZ+rXHYAd3Xn0Z8HfLKq9ib5YpILgQD7gV8bY52SpAH6XOXyEPDadcavGktFkqSh+E1RSWqEgS5JjTDQJakRBrokNcJAl6RGGOiS1AgDXZIaYaBLUiMMdElqhIEuSY0w0CWpEQa6JDXCQJekRhjoktQIA12SGtHnjkVnJ/lKkq8neTjJ+7vxlyV5MMkjST6R5PnjL1eSdCp9jtCfAa6qqtcAlwJXJ7kC+BBwW1VdAjwF3DS+MiVJgwwM9O5G0Cvd6lndUsBVwKe68btYva+oJGlCep1DT3JGkv3AUeA+4JvA01V1vJvyGLBzPCVKkvpIVfWfnJwHfBb4feCjVfXybvxi4N6q2rXOPgvAAsD09PRli4uLQxV64MixofYbhV07zx04Z2VlhampqS2oZrLssy3boc8Wepyfn99XVbOD5g28SfRaVfV0kiXgCuC8JGd2R+kXAY+fYp89wB6A2dnZmpub28hb/tC7dt8z1H6jcPiGuYFzlpaWGLa304l9tmU79Lkdejyhz1UuF3ZH5iR5IfAG4BDwAHBdN+1G4O5xFSlJGqzPEfoO4K4kZ7D6D8Anq2pvkn8EFpP8EfA14PYx1ilJGmBgoFfVQ8Br1xn/FnD5OIqSJG2c3xSVpEYY6JLUCANdkhphoEtSIwx0SWqEgS5JjTDQJakRBrokNcJAl6RGGOiS1AgDXZIaYaBLUiMMdElqhIEuSY0w0CWpEQa6JDWizy3oLk7yQJJDSR5OcnM3/r4kR5Ls75Y3j79cSdKp9LkF3XHglqr6apIXAfuS3Ndtu62q/nh85UmS+upzC7ongCe6599LcgjYOe7CJEkbs6Fz6ElmWL2/6IPd0HuSPJTkjiTnj7g2SdIGpKr6TUymgL8DPlhVn0kyDTwJFPABYEdV/fI6+y0ACwDT09OXLS4uDlXogSPHhtpvFHbtPHfgnJWVFaampragmsmyz7Zshz5b6HF+fn5fVc0Omtcr0JOcBewFPl9VH15n+wywt6pe/WyvMzs7W8vLywPfbz0zu+8Zar9ROHzrNQPnLC0tMTc3N/5iJsw+27Id+myhxyS9Ar3PVS4BbgcOrQ3zJDvWTHsbcHCYQiVJo9HnKpcrgXcCB5Ls78Z+F7g+yaWsnnI5DPzqWCqUJPXS5yqXLwNZZ9O9oy9HkjQsvykqSY0w0CWpEQa6JDXCQJekRhjoktQIA12SGmGgS1IjDHRJaoSBLkmNMNAlqREGuiQ1wkCXpEYY6JLUCANdkhphoEtSIwx0SWpEn1vQXZzkgSSHkjyc5OZu/MVJ7kvySPd4/vjLlSSdSp8j9OPALVX1SuAK4N1JXgXsBu6vqkuA+7t1SdKEDAz0qnqiqr7aPf8ecAjYCbwVuKubdhdw7biKlCQNlqrqPzmZAb4EvBr4dlWdt2bbU1X1I6ddkiwACwDT09OXLS4uDlXogSPHhtpvFHbtPHfgnJWVFaampragmsmyz7Zshz5b6HF+fn5fVc0Omtc70JNMAX8HfLCqPpPk6T6Bvtbs7GwtLy/3er+Tzey+Z6j9RuHwrdcMnLO0tMTc3Nz4i5kw+2zLduizhR6T9Ar0Xle5JDkL+DTwsar6TDf8nSQ7uu07gKPDFitJ2rw+V7kEuB04VFUfXrPpc8CN3fMbgbtHX54kqa8ze8y5EngncCDJ/m7sd4FbgU8muQn4NvD28ZQoSepjYKBX1ZeBnGLz60dbjiRpWH5TVJIaYaBLUiMMdElqhIEuSY0w0CWpEQa6JDXCQJekRhjoktQIA12SGmGgS1IjDHRJaoSBLkmNMNAlqREGuiQ1wkCXpEb0uWPRHUmOJjm4Zux9SY4k2d8tbx5vmZKkQfocod8JXL3O+G1VdWm33DvasiRJGzUw0KvqS8B3t6AWSdImbOYc+nuSPNSdkjl/ZBVJkoaSqho8KZkB9lbVq7v1aeBJoIAPADuq6pdPse8CsAAwPT192eLi4lCFHjhybKj9RmHXznMHzllZWWFqamoLqpks+2zLduizhR7n5+f3VdXsoHlDBXrfbSebnZ2t5eXlge+3npnd9wy13ygcvvWagXOWlpaYm5sbfzETZp9t2Q59ttBjkl6BPtQplyQ71qy+DTh4qrmSpK1x5qAJST4OzAEXJHkM+ANgLsmlrJ5yOQz86hhrlCT1MDDQq+r6dYZvH0MtkqRN8JuiktQIA12SGmGgS1IjDHRJaoSBLkmNMNAlqREGuiQ1wkCXpEYY6JLUCANdkhphoEtSIwx0SWqEgS5JjTDQJakRBrokNcJAl6RGDAz0JHckOZrk4JqxFye5L8kj3eP54y1TkjRInyP0O4GrTxrbDdxfVZcA93frkqQJGhjoVfUl4LsnDb8VuKt7fhdw7YjrkiRtUKpq8KRkBthbVa/u1p+uqvPWbH+qqtY97ZJkAVgAmJ6evmxxcXGoQg8cOTbUfqOwa+e5A+esrKwwNTW1BdVMln22ZTv02UKP8/Pz+6pqdtC8gTeJ3qyq2gPsAZidna25ubmhXuddu+8ZYVUbc/iGuYFzlpaWGLa304l9tmU79Lkdejxh2KtcvpNkB0D3eHR0JUmShjFsoH8OuLF7fiNw92jKkSQNq89lix8H/h54RZLHktwE3Aq8MckjwBu7dUnSBA08h15V159i0+tHXIskaRP8pqgkNcJAl6RGGOiS1AgDXZIaYaBLUiMMdElqhIEuSY0w0CWpEQa6JDXCQJekRhjoktQIA12SGmGgS1IjDHRJaoSBLkmN2NQ9RZMcBr4H/A9wvM9NTCVJ4zGKm0TPV9WTI3gdSdImeMpFkhqx2UAv4AtJ9iVZGEVBkqThpKqG3zn5iap6PMlLgfuA91bVl06aswAsAExPT1+2uLg41HsdOHJs6Dq3wvQL4Tv/OdrX3LXz3NG+4AisrKwwNTU16TLGzj7b0UKP8/Pz+/p8RrmpQP9/L5S8D1ipqj8+1ZzZ2dlaXl4e6vVndt8zZGVb45Zdx/mTA6P4SOL/HL71mpG+3igsLS0xNzc36TLGzj7b0UKPSXoF+tCnXJKck+RFJ54DbwIODvt6kqTN2cwh5TTw2SQnXuevqupvR1KVJGnDhg70qvoW8JoR1iJJ2gQvW5SkRhjoktQIA12SGmGgS1IjDHRJaoSBLkmNMNAlqREGuiQ1wkCXpEYY6JLUiNH+PKC0SZP6Vc3n4i9bShvlEbokNcJAl6RGGOiS1AgDXZIa4YeiEj/6Yewtu47zri36gNYPZLfOJG9luRX/nTd1hJ7k6iTfSPJokt2jKkqStHGbuafoGcBHgF8AXgVcn+RVoypMkrQxmzlCvxx4tKq+VVX/BSwCbx1NWZKkjdpMoO8E/nXN+mPdmCRpAlJVw+2YvB34+ar6lW79ncDlVfXek+YtAAvd6iuAbwxf7nPaBcCTky5iC9hnW7ZDny30+JNVdeGgSZu5yuUx4OI16xcBj588qar2AHs28T6nhSTLVTU76TrGzT7bsh363A49nrCZUy7/AFyS5GVJng+8A/jcaMqSJG3U0EfoVXU8yXuAzwNnAHdU1cMjq0yStCGb+mJRVd0L3DuiWk53zZ9W6thnW7ZDn9uhR2ATH4pKkp5b/C0XSWqEgd5TkjuSHE1ycM3Yi5Pcl+SR7vH8bjxJ/qz7SYSHkvzM5CrvL8nFSR5IcijJw0lu7sZb6/PsJF9J8vWuz/d34y9L8mDX5ye6D/tJ8oJu/dFu+8wk69+oJGck+VqSvd16c30mOZzkQJL9SZa7sab+bvsw0Pu7E7j6pLHdwP1VdQlwf7cOqz+HcEm3LAB/vkU1btZx4JaqeiVwBfDu7uccWuvzGeCqqnoNcClwdZIrgA8Bt3V9PgXc1M2/CXiqql4O3NbNO53cDBxas95qn/NVdemaSxRb+7sdrKpcei7ADHBwzfo3gB3d8x3AN7rnfwFcv96802kB7gbe2HKfwI8BXwV+ltUvn5zZjb8O+Hz3/PPA67rnZ3bzMunae/Z3EathdhWwF0ijfR4GLjhprNm/21MtHqFvznRVPQHQPb60Gz/tfxah+9/t1wIP0mCf3WmI/cBR4D7gm8DTVXW8m7K2lx/22W0/Brxkayse2p8CvwX8oFt/CW32WcAXkuzrvp0ODf7dDuLvoY9H1hk7bS4nSjIFfBr49ar6j2S9dlanrjN2WvRZVf8DXJrkPOCzwCvXm9Y9npZ9JvlF4GhV7Usyd2J4namndZ+dK6vq8SQvBe5L8k/PMvd07vNZeYS+Od9JsgOgezzajff6WYTnoiRnsRrmH6uqz3TDzfV5QlU9DSyx+pnBeUlOHOSs7eWHfXbbzwW+u7WVDuVK4C1JDrP6a6hXsXrE3lqfVNXj3eNRVv+BvpyG/25PxUDfnM8BN3bPb2T1nPOJ8V/qPk2/Ajh24n/9nsuyeih+O3Coqj68ZlNrfV7YHZmT5IXAG1j90PAB4Lpu2sl9nuj/OuCL1Z18fS6rqt+pqouqaobVn+b4YlXdQGN9JjknyYtOPAfeBByksb/bXiZ9Ev90WYCPA08A/83qv/A3sXp+8X7gke7xxd3csHrzj28CB4DZSdffs8efY/V/PR8C9nfLmxvs86eBr3V9HgR+vxv/KeArwKPAXwMv6MbP7tYf7bb/1KR7GKLnOWBvi312/Xy9Wx4Gfq8bb+rvts/iN0UlqRGecpGkRhjoktQIA12SGmGgS1IjDHRJaoSBLkmNMNAlqREGuiQ14n8Bpujd5EkASr0AAAAASUVORK5CYII=\n"
            ],
            "text/plain": [
              "<matplotlib.figure.Figure at 0x1173608d0>"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 27,
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "def outliers_z_score(ys, threshold=3):\n",
        "    mean_y = np.mean(ys)\n",
        "    std_y = np.std(ys)\n",
        "    z_scores = [(y - mean_y) / std_y for y in ys]\n",
        "    return np.where(np.abs(z_scores) > threshold)[0]"
      ],
      "outputs": [],
      "execution_count": 11,
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "with warnings.catch_warnings():\n",
        "    warnings.simplefilter('ignore')\n",
        "    o = outliers_z_score(vis_data.distance)"
      ],
      "outputs": [],
      "execution_count": 12,
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "len(o)"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 13,
          "data": {
            "text/plain": [
              "55"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 13,
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "Значение по умолчанию для функции поиска выбросов равно `3`. Это означает, что мы считаем выбросами все, что находится на расстоянии больше 3 стандартных отклонений от среднего. Если мы подвигаем этот параметр в большую или меньшую сторону, мы найдем меньше или больше выбросов."
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "## IQR (interquartile range, межквартильное расстояние)\n",
        "\n",
        "Один из способов - использовать перцентили (в данном случае квартили) для определения, где находятся выбросы. Для начала, посмотрим на `boxplot` для признака `distance`. Данный вид графика делит данные на 4 части. Та часть данных, которая отделена слева и справа двумя вертикальными линиями (находится внутри коробки) содержит 50% данных, так как левая линия обозначает первый квартиль (25-й перцентиль), а правая линия обозначает третий квартиль (75-й перцентиль).  \n",
        "Большой вертикальной линией выделена медиана:"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "import seaborn as sns"
      ],
      "outputs": [],
      "execution_count": 14,
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "sns.boxplot(vis_data.distance, color='yellow');"
      ],
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": [
              "iVBORw0KGgoAAAANSUhEUgAAAWQAAAEKCAYAAAAl5S8KAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAADuxJREFUeJzt3X9s3PV9x/HX2zVJGgKF/GhlzTHmGIxG00I3q9B2k0JJKlrNgUpGuIpE0FDQoiK1E16JVa3apEkEzVqrThtavR+ANM1VwtoCG22dNLQVldo6hFA3lBDOJKRYS8ChWYgIeH7vj+/nbmf7krMd393b5+dDOvnue59v7vvJffP0N9/kvjZ3FwCg/prqvQEAgAxBBoAgCDIABEGQASAIggwAQRBkAAiCIANAEAQZAIIgyAAQRPNsBq9evdrb29urtCkA0Jj279//hruvqTRuVkFub2/X0NDQ3LcKABYhMzs6k3GcsgCAIAgyAARBkAEgCIIMAEEQZAAIgiADQBAEGQCCIMgAEARBBoAgCDIABEGQASAIggwAQRBkAAiCIANAEAQZAIIgyAAQBEEGgCAIMgAEQZABIIhZ/Uy9+dTf369nn31WktTS0qJcLqdt27bVa3MAoO7qFuR8Pq9Tp97QkiUTeuWVE/XaDAAIo25BlqQlSybU2vpOPTcBAMLgHDIABEGQASAIggwAQRBkAAiCIANAEAQZAIIgyAAQBEEGgCAIMgAEQZABIAiCDABBEGQACIIgA0AQBBkAgiDIABAEQQaAIAgyAARBkAEgCIIMAEEQZAAIgiADQBAEGQCCIMgAEARBBoAgCDIABEGQASAIggwAQRBkAAiCIANAEAQZAIIgyAAQBEEGgCAIMgAEQZABIAiCDABBEGQACIIgA0AQBBkAgiDIABAEQQaAIAgyAARRkyD39/erv7+/auMBoBE01+JF8vl8VccDQCPglAUABEGQASAIggwAQRBkAAiCIANAEAQZAIIgyAAQBEEGgCAIMgAEQZABIAiCDABBEGQACIIgA0AQBBkAgiDIABAEQQaAIAgyAARBkAEgCIIMAEEQZAAIgiADQBAEGQCCIMgAEARBBoAgCDIABEGQASAIggwAQRBkAAiCIANAEAQZAIIgyAAQBEEGgCAIMgAEQZABIAiCDABBEGQACIIgA0AQBBkAgggZ5OHhYQ0PD6uzs1OdnZ3asmWLOjs7dffddxeXFW5PP/20urq61NnZqYMHD0qSDhw4oNtuu634uGDXrl3F9e677z6NjIxox44dOnXqlMbGxnT//fdr+/bt2rx586R18/m87rzzTo2MjBSXjY2NFdctp9w6mF+V3oNqjcPiUsv9ImSQpzp9+rQk6c0335z23MMPP6xz585Jknbu3ClJeuihhzQxMVF8XPDYY48V7x89elR9fX06dOiQBgYGNDAwoMOHD+v48eNy90nr9vX16ezZs+rr6ysuGxgYKK5bTrl1ML8qvQfVGofFpZb7Rbggd3Z2zmq8uxfvnzlzRrt379bbb79dfFw40t21a9e0dY8dOyZ31+DgoAYHByc9V1g3n8/rtddeK44fGRnR2NiY9u7dK3fXnj17pn3nLLcO5lel96Ba47C41Hq/qEmQR0dHlc/n1dvbW7zl83m9955Jkk6eXFJ8/mI9+uijkx4XjnRLj46nGh8f1/j4+LTlO3funHaE29fXp4GBAU1MTEiSJiYmpn3nLLcO5lel96Ba47C41Hq/qBhkM7vXzIbMbOjkyZNV3ZhqOHPmTMUxpUfZU9ctHOkWHDt2TM8880wx4OPj49q3b9+kMeXWwfyq9B5UaxwWl1rvFxWD7O7fcPcOd+9Ys2bNnF6kpaVFuVxODz74YPGWy+V0ySVZCNesebf4/HxbsWJFxTFmdt51165dO2lZW1ubNmzYoObmZklSc3Ozbr755kljyq2D+VXpPajWOCwutd4vwp1Dvlhbt26d9HjHjh2SpLvuuuu86zQ3Nxd/06eu29PTM2lZT0+Puru71dSU/dY1NTWpu7t72pgLPcbFq/QeVGscFpda7xfhgvzkk0/Oanzp0e2KFSvU1dWlSy+9tPh4/fr1kqQ77rhj2rptbW0yM23atEmbNm2a9Fxh3VwuVzzibWtr09VXX62VK1fqlltukZlp48aNuvLKKyetW24dzK9K70G1xmFxqfV+ES7I5Vx++eWSpFWrVk17bvv27Vq6dKmk/z8afuCBB9TU1FR8XFB6lHzVVVepp6dH69atU3d3t7q7u3XdddeptbVVZjZp3Z6eHi1fvnzSkW53d3dx3XLKrYP5Vek9qNY4LC613C/sfP+gVU5HR4cPDQ3N+kUK/3ui9Bxxb2+vjhw5qNbWdyRJy5bdWHy+3HgAWKjMbL+7d1QatyCOkAFgMSDIABAEQQaAIAgyAARBkAEgCIIMAEEQZAAIgiADQBAEGQCCIMgAEARBBoAgCDIABEGQASAIggwAQRBkAAiCIANAEAQZAIIgyAAQBEEGgCAIMgAEQZABIAiCDABBEGQACIIgA0AQBBkAgiDIABAEQQaAIAgyAARBkAEgCIIMAEEQZAAIgiADQBAEGQCCIMgAEARBBoAgCDIABEGQASAIggwAQTTX4kVyuVxVxwNAI6hJkLdt21bV8QDQCDhlAQBBEGQACIIgA0AQBBkAgiDIABAEQQaAIAgyAARBkAEgCIIMAEEQZAAIgiADQBAEGQCCIMgAEARBBoAgCDIABEGQASAIggwAQRBkAAiCIANAEAQZAIIgyAAQBEEGgCAIMgAEQZABIAiCDABBEGQACIIgA0AQBBkAgiDIABAEQQaAIAgyAARBkAEgCIIMAEEQZAAIgiADQBAEGQCCIMgAEARBBoAgCDIABEGQASCIugb53XebdPz4Mv3618vquRkAEEJzvV44l8tpdHRUktTS0qJcLlevTQGAEMzdZzy4o6PDh4aGqrg5ANB4zGy/u3dUGsc5ZAAIgiADQBAEGQCCIMgAEARBBoAgCDIABEGQASAIggwAQRBkAAiCIANAEAQZAIIgyAAQBEEGgCAIMgAEQZABIAiCDABBEGQACIIgA0AQBBkAgiDIABDErH7IqZmdlHR0jq+1WtIbc1w3skadl8TcFqJGnZe0sOd2lbuvqTRoVkG+GGY2NJOfurrQNOq8JOa2EDXqvKTGnlsBpywAIAiCDABB1DLI36jha9VSo85LYm4LUaPOS2rsuUmq4TlkAMCFccoCAIKoepDN7FYze8nMjpjZjmq/3nwzs38xsxNmNlyybKWZDZrZy+nrlWm5mdnX01xfMLPfr9+WX5iZrTWzfWb2opn90sy+kJY3wtyWmdnPzOxgmttfpeVXm9lP09y+aWZL0vKl6fGR9Hx7Pbe/EjN7n5kdMLOn0uNGmderZvYLM3vezIbSsgW/P85GVYNsZu+T9PeSPi1pnaTPmdm6ar5mFTwi6dYpy3ZI2uvu10ramx5L2TyvTbd7JT1co22ci3FJ97v7hyXdJOnz6b1phLmdk/RJd18v6QZJt5rZTZIekvTVNLdTku5J4++RdMrdf1vSV9O4yL4g6cWSx40yL0m62d1vKPnvbY2wP86cu1ftJuljkr5X8rhXUm81X7NK82iXNFzy+CVJLel+i6SX0v1/lPS5cuOi3yR9R9KmRpubpOWSnpN0o7IPFTSn5cV9U9L3JH0s3W9O46ze236e+bQqC9MnJT0lyRphXmkbX5W0esqyhtofK92qfcrityS9VvL4eFq20H3I3UclKX39YFq+IOeb/ir7EUk/VYPMLf21/nlJJyQNSnpF0lvuPp6GlG5/cW7p+d9IWlXbLZ6xr0n6kqSJ9HiVGmNekuSSvm9m+83s3rSsIfbHmWqu8q9vZZY18n/rWHDzNbMVkh6X9EV3P21WbgrZ0DLLws7N3f9X0g1mdoWkb0n6cLlh6euCmJuZ/bGkE+6+38w2FBaXGbqg5lXiE+7+upl9UNKgmf3qAmMX2txmpNpHyMclrS153Crp9Sq/Zi38t5m1SFL6eiItX1DzNbNLlMX439z9P9Lihphbgbu/JekZZefJrzCzwkFI6fYX55ae/4Cksdpu6Yx8QtJmM3tV0oCy0xZf08KflyTJ3V9PX08o+yb6UTXY/lhJtYP8c0nXpn8FXiKpW9ITVX7NWnhC0tZ0f6uy86+F5XelfwG+SdJvCn/disayQ+F/lvSiu/9tyVONMLc16chYZvZ+SRuV/SPYPkldadjUuRXm3CXpB55OTEbi7r3u3uru7cr+LP3A3bdogc9LkszsUjO7rHBf0qckDasB9sdZqcGJ+s9IOqzsHN6X633SfA7b/++SRiW9p+y78j3KzsPtlfRy+royjTVl/6vkFUm/kNRR7+2/wLz+UNlf8V6Q9Hy6faZB5vZ7kg6kuQ1L+kpanpP0M0lHJO2StDQtX5YeH0nP5+o9hxnMcYOkpxplXmkOB9Ptl4VWNML+OJsbn9QDgCD4pB4ABEGQASAIggwAQRBkAAiCIANAENX+pB5QkZn9paQzki6X9CN333OecbdLOuzuh2q4eUDNcISMMNz9K+eLcXK7sqsGAg2JIKMuzOzLll0ne4+k30nLHjGzrnR/p5kdSte67TOzj0vaLOlv0vVyrzGzbWb283Td48fNbHnJr/N1M/uJmeULv2Z67kvpmrsHzWxnWnaNmX03XdTmx2Z2fc1/QwBxygJ1YGZ/oOyjvx9Rtg8+J2l/yfMrJX1W0vXu7mZ2hbu/ZWZPKPt02u407i1370/3/1rZpyj/Lv0yLco+jXi9so/Z7jazTys7yr7R3c+m15Gyn9X2p+7+spndKOkflF0nAqgpgox6+CNJ33L3s5KUQlvqtKR3JP2Tmf2nsuv+lvO7KcRXSFqh7Pq/Bd929wlJh8zsQ2nZRkn/Wnhddx9LV7v7uKRdJVe6W3pRswPmiCCjXs77mX13Hzezj0q6RdmR9H0qf8T6iKTb3f2gmd2t7PoOBedK7lvJ16mv26TsesI3zGbjgWrgHDLq4UeSPmtm709X+OosfTIdtX7A3f9L0heV/RgmSfofSZeVDL1M0mi6jOiWGbzu9yX9Scm55pXuflrSiJndkZaZma2/iLkBc0aQUXPu/pykbyq7wtzjkn48Zchlkp4ysxck/VDSn6XlA5L+3LIf8HmNpL9Q9lNOBiVd6GLmhdf9rrLzyUPpp4n0pKe2SLrHzApXGrvtIqYHzBlXewOAIDhCBoAgCDIABEGQASAIggwAQRBkAAiCIANAEAQZAIIgyAAQxP8BynOVb3ZmfK8AAAAASUVORK5CYII=\n"
            ],
            "text/plain": [
              "<matplotlib.figure.Figure at 0x1120c4630>"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 15,
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "Метод межквартильного расстояния выделяет как выбросы все значения, находящиеся за пределами коробки.  \n",
        "Посмотрим, сколько выбросов он найдет:"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "def outliers_iqr(ys):\n",
        "    quartile_1, quartile_3 = np.percentile(ys, [25, 75])\n",
        "    iqr = quartile_3 - quartile_1\n",
        "    lower_bound = quartile_1 - (iqr * 1.5)\n",
        "    upper_bound = quartile_3 + (iqr * 1.5)\n",
        "    return np.where((ys > upper_bound) | (ys < lower_bound))[0]"
      ],
      "outputs": [],
      "execution_count": 16,
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "Данный метод чувствителен к пропускам, поэтому избавимся от них:"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "o = outliers_iqr(vis_data.distance.dropna())"
      ],
      "outputs": [],
      "execution_count": 17,
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "len(o)"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 18,
          "data": {
            "text/plain": [
              "134"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 18,
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Поиск по распределению\n",
        "[Источник](http://aqibsaeed.github.io/2016-07-17-anomaly-detection/)\n",
        "\n",
        "Еще один способ найти выбросы - построить нормальное распределение поверх данных, посчитать вероятность находиться в данном распределении для каждого значения. Самые невероятные значения будем считать выбросами."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "from scipy.stats import norm\n",
        "\n",
        "def estimate_gaussian(dataset):\n",
        "    mu = np.mean(dataset, axis=0)\n",
        "    sigma = np.cov(dataset.T)\n",
        "    return mu, sigma\n",
        "    \n",
        "def get_gaussian(mu, sigma):\n",
        "    distribution = norm(mu, sigma)\n",
        "    return distribution\n",
        "\n",
        "def get_probs(distribution, dataset):\n",
        "    return distribution.pdf(dataset)"
      ],
      "outputs": [],
      "execution_count": 19,
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "mu, sigma = estimate_gaussian(vis_data.distance.dropna())\n",
        "distribution = get_gaussian(mu, sigma)\n",
        "probabilities = get_probs(distribution, vis_data.distance.dropna())"
      ],
      "outputs": [],
      "execution_count": 21,
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "Мы построили нормальное распределение на предоставленных данных и посчитали вероятность появления данной точки при данном распределении для каждого значения. Теперь выберем разумное значение вероятности, ниже которой будем считать точку выбросом. Например, `0.014`:"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "o_1 = np.where(probabilities < 0.014)[0]"
      ],
      "outputs": [],
      "execution_count": 22,
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "len(o_1)"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 23,
          "data": {
            "text/plain": [
              "58"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 23,
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Проблемы с выбросами\n",
        "\n",
        "Несмотря на то, что выбросы \"портят\" общую картину и заставляют статистики в данных (например, среднее) выглядеть неправильно, иногда они очень важны:\n",
        "\n",
        " - Если мы занимаемся поиском аномалий (Anomaly Detection), выбросы являются нашими зависимыми переменными, которые необходимо найти, а не значениями, которые нужно выбросить. Поиск аномалий выходит за рамки курса\n",
        " - Как можно было понять из методов выше, нет однозначного способа понять, что перед нами выброс. Разные методы возвращают разные точки в качестве выбросов, и эти точки пересекаются лишь частично. Кроме этого, разные параметры дают разные результаты и эти параметры необходимо крутить руками. Из-за этого найденные выбросы является крайне субъективным результатом"
      ],
      "metadata": {}
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "name": "python",
      "version": "3.7.2",
      "mimetype": "text/x-python",
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "pygments_lexer": "ipython3",
      "nbconvert_exporter": "python",
      "file_extension": ".py"
    },
    "nteract": {
      "version": "0.22.0"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 2
}