03_tpot.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook we'll attempt to use TPOT (auto-ml) to discover interesting modeling techniques."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tpot import TPOTRegressor\n",
    "from sklearn.model_selection import train_test_split\n",
    "import stars\n",
    "from astropy.io import ascii\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import multiprocessing\n",
    "\n",
    "#if __name__ == '__main__':\n",
    "#    multiprocessing.set_start_method('forkserver')\n",
    "\n",
    "sl = stars.StarLoader('data/mastarall-v3_1_1-v1_7_7.fits', 'data/mastar-combspec-v3_1_1-v1_7_7-lsfpercent99.5.fits')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initial experimenation\n",
    "\n",
    "First, I copy-pasted tutorial code for TPOT. It ran for 2h and crashed after 3 generations (out of default 100). It thought ExtraTreesRegressor was best:\n",
    "\n",
    "```python\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.ensemble import ExtraTreesRegressor\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# NOTE: Make sure that the outcome column is labeled 'target' in the data file\n",
    "tpot_data = pd.read_csv('PATH/TO/DATA/FILE', sep='COLUMN_SEPARATOR', dtype=np.float64)\n",
    "features = tpot_data.drop('target', axis=1)\n",
    "training_features, testing_features, training_target, testing_target = \\\n",
    "            train_test_split(features, tpot_data['target'], random_state=1)\n",
    "\n",
    "# Average CV score on the training set was: -137065.8257885053\n",
    "exported_pipeline = ExtraTreesRegressor(bootstrap=False, max_features=0.1, min_samples_leaf=1, min_samples_split=4, n_estimators=100)\n",
    "# Fix random state in exported estimator\n",
    "if hasattr(exported_pipeline, 'random_state'):\n",
    "    setattr(exported_pipeline, 'random_state', 1)\n",
    "\n",
    "exported_pipeline.fit(training_features, training_target)\n",
    "results = exported_pipeline.predict(testing_features)\n",
    "```\n",
    "\n",
    "Then I exported the data to `data/goodt.csv` and ran from CLI:\n",
    "\n",
    "```bash\n",
    "tpot data/goodt.csv -is , -target INPUT_TEFF -mode regression -o -njobs -1 -s 1 -v 2\n",
    "```\n",
    "\n",
    "And I got the following error almost immediately:\n",
    "\n",
    "```\n",
    "joblib.externals.loky.process_executor.TerminatedWorkerError: A worker process managed by the executor was unexpectedly terminated. This could be caused by a segmentation fault while calling the function or by an excessive memory usage causing the Operating System to kill the worker.\n",
    "\n",
    "The exit codes of the workers are {SIGKILL(-9)}\n",
    "```\n",
    "\n",
    "Next, I tried running TPOT by limiting config to XGBoost, just to compare what comes out. I also used just 4 cores this time and restricted algorithms, generations and population severely so that I can see a full TPOT run in action. There is no specific reason I chose XGBoost (I wanted to pick just one).\n",
    "\n",
    "```\n",
    "tpot data/goodt.csv -is , -target INPUT_TEFF -mode regression -o tpot_exported_pipeline2.py -njobs 4 -s 1 -cf tpot_checkpoints -v 3 -config tpot_xgboost.py -g 5 -p 5 -cv 5\n",
    "```\n",
    "\n",
    "Checkpoint was twice saved to tpot_checkpoints successfully and the main execution loop took 30 minutes (without optimisation step, around 44s/pipeline). The score after 3 generations was much worse to compared with the ExtraTreesRegressor, so I killed off the TPOT \"optimisation\" step. Overall best result was:\n",
    "\n",
    "```python\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.pipeline import make_pipeline, make_union\n",
    "from tpot.builtins import StackingEstimator\n",
    "from xgboost import XGBRegressor\n",
    "from tpot.export_utils import set_param_recursive\n",
    "from sklearn.preprocessing import FunctionTransformer\n",
    "from copy import copy\n",
    "\n",
    "# NOTE: Make sure that the outcome column is labeled 'target' in the data file\n",
    "tpot_data = pd.read_csv('PATH/TO/DATA/FILE', sep='COLUMN_SEPARATOR', dtype=np.float64)\n",
    "features = tpot_data.drop('target', axis=1)\n",
    "training_features, testing_features, training_target, testing_target = \\\n",
    "            train_test_split(features, tpot_data['target'], random_state=1)\n",
    "\n",
    "# Average CV score on the training set was: -22709696.973815482\n",
    "exported_pipeline = make_pipeline(\n",
    "    make_union(\n",
    "        FunctionTransformer(copy),\n",
    "        FunctionTransformer(copy)\n",
    "    ),\n",
    "    XGBRegressor(learning_rate=0.001, max_depth=7, min_child_weight=7, n_estimators=100, n_jobs=1, objective=\"reg:squarederror\", subsample=0.15000000000000002, verbosity=0)\n",
    ")\n",
    "# Fix random state for all the steps in exported pipeline\n",
    "set_param_recursive(exported_pipeline.steps, 'random_state', 1)\n",
    "\n",
    "exported_pipeline.fit(training_features, training_target)\n",
    "results = exported_pipeline.predict(testing_features)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "As a next step, I decided to go back to notebook-based execution, but keep the severe restrictions of generation, population and algorithm. Also, I used the forkserver trick hoping that it will prevent crashes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "goodt = sl.stars[sl.stars['INPUT_TEFF']>0]\n",
    "X_train, X_test, y_train, y_test = train_test_split(goodt['FLUX_CORR'], goodt['INPUT_TEFF'], train_size=0.9, test_size=0.1, random_state=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 operators have been imported by TPOT.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "de6209803d0d42359fbbe9adff18c2e8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Optimization Progress:   0%|          | 0/30 [00:00<?, ?pipeline/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Skipped pipeline #1 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #3 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #5 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #7 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #10 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #12 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #14 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #16 due to time out. Continuing to the next pipeline.\n",
      "\n",
      "Generation 1 - Current Pareto front scores:\n",
      "\n",
      "-1\t-175279.31687513887\tExtraTreesRegressor(input_matrix, ExtraTreesRegressor__bootstrap=True, ExtraTreesRegressor__max_features=0.45, ExtraTreesRegressor__min_samples_leaf=10, ExtraTreesRegressor__min_samples_split=13, ExtraTreesRegressor__n_estimators=100)\n",
      "Saving periodic pipeline from pareto front to tpot_checkpoints/pipeline_gen_1_idx_0_2022.05.26_10-40-59.py\n",
      "Skipped pipeline #20 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #23 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #25 due to time out. Continuing to the next pipeline.\n",
      "\n",
      "Generation 2 - Current Pareto front scores:\n",
      "\n",
      "-1\t-170583.51871409133\tExtraTreesRegressor(input_matrix, ExtraTreesRegressor__bootstrap=True, ExtraTreesRegressor__max_features=0.45, ExtraTreesRegressor__min_samples_leaf=9, ExtraTreesRegressor__min_samples_split=13, ExtraTreesRegressor__n_estimators=100)\n",
      "Saving periodic pipeline from pareto front to tpot_checkpoints/pipeline_gen_2_idx_0_2022.05.26_10-50-01.py\n",
      "Pipeline encountered that has previously been evaluated during the optimization process. Using the score from the previous evaluation.\n",
      "Skipped pipeline #31 due to time out. Continuing to the next pipeline.\n",
      "\n",
      "Generation 3 - Current Pareto front scores:\n",
      "\n",
      "-1\t-155932.94160968057\tExtraTreesRegressor(input_matrix, ExtraTreesRegressor__bootstrap=True, ExtraTreesRegressor__max_features=0.55, ExtraTreesRegressor__min_samples_leaf=3, ExtraTreesRegressor__min_samples_split=20, ExtraTreesRegressor__n_estimators=100)\n",
      "Saving periodic pipeline from pareto front to tpot_checkpoints/pipeline_gen_3_idx_0_2022.05.26_10-55-04.py\n",
      "Pipeline encountered that has previously been evaluated during the optimization process. Using the score from the previous evaluation.\n",
      "Skipped pipeline #34 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #36 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #38 due to time out. Continuing to the next pipeline.\n",
      "\n",
      "Generation 4 - Current Pareto front scores:\n",
      "\n",
      "-1\t-155932.94160968057\tExtraTreesRegressor(input_matrix, ExtraTreesRegressor__bootstrap=True, ExtraTreesRegressor__max_features=0.55, ExtraTreesRegressor__min_samples_leaf=3, ExtraTreesRegressor__min_samples_split=20, ExtraTreesRegressor__n_estimators=100)\n",
      "Periodic pipeline was not saved, probably saved before...\n",
      "Skipped pipeline #41 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #43 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #45 due to time out. Continuing to the next pipeline.\n",
      "\n",
      "Generation 5 - Current Pareto front scores:\n",
      "\n",
      "-1\t-155932.94160968057\tExtraTreesRegressor(input_matrix, ExtraTreesRegressor__bootstrap=True, ExtraTreesRegressor__max_features=0.55, ExtraTreesRegressor__min_samples_leaf=3, ExtraTreesRegressor__min_samples_split=20, ExtraTreesRegressor__n_estimators=100)\n",
      "Periodic pipeline was not saved, probably saved before...\n",
      "-123561.13159\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/x/opt/miniconda3/envs/mastar/lib/python3.10/site-packages/sklearn/metrics/_scorer.py:765: FutureWarning: sklearn.metrics.SCORERS is deprecated and will be removed in v1.3. Please use sklearn.metrics.get_scorer_names to get a list of available scorers and sklearn.metrics.get_metric to get scorer.\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "# Customising operators http://epistasislab.github.io/tpot/using/#customizing-tpots-operators-and-parameters\n",
    "# Defaults from here https://github.com/EpistasisLab/tpot/tree/master/tpot/config\n",
    "tpot_config = {\n",
    "    'sklearn.ensemble.ExtraTreesRegressor': {\n",
    "        'n_estimators': [100],\n",
    "        'max_features': np.arange(0.05, 1.01, 0.05),\n",
    "        'min_samples_split': range(2, 21),\n",
    "        'min_samples_leaf': range(1, 21),\n",
    "        'bootstrap': [True, False]\n",
    "    },\n",
    "}\n",
    "\n",
    "# http://epistasislab.github.io/tpot/using/#crashfreeze-issue-with-n_jobs-1-under-osx-or-linux\n",
    "if __name__ == '__main__':\n",
    "    pipeline_optimizer = TPOTRegressor(random_state=1, verbosity=3, n_jobs=4, generations=5, population_size=5,\n",
    "        periodic_checkpoint_folder='tpot_checkpoints', cv=5, config_dict=tpot_config, scoring='neg_mean_squared_error')\n",
    "    pipeline_optimizer.fit(X_train, y_train)\n",
    "    print(pipeline_optimizer.score(X_test, y_test))\n",
    "    pipeline_optimizer.export('tpot_exported_pipeline.py')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Main execution loop here took 27 minutes (around 44s/pipeline) and the following code. The worsening of the result could be due to different test split.\n",
    "\n",
    "```python\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.ensemble import ExtraTreesRegressor\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# NOTE: Make sure that the outcome column is labeled 'target' in the data file\n",
    "tpot_data = pd.read_csv('PATH/TO/DATA/FILE', sep='COLUMN_SEPARATOR', dtype=np.float64)\n",
    "features = tpot_data.drop('target', axis=1)\n",
    "training_features, testing_features, training_target, testing_target = \\\n",
    "            train_test_split(features, tpot_data['target'], random_state=1)\n",
    "\n",
    "# Average CV score on the training set was: -155932.94160968057\n",
    "exported_pipeline = ExtraTreesRegressor(bootstrap=True, max_features=0.55, min_samples_leaf=3, min_samples_split=20, n_estimators=100)\n",
    "# Fix random state in exported estimator\n",
    "if hasattr(exported_pipeline, 'random_state'):\n",
    "    setattr(exported_pipeline, 'random_state', 1)\n",
    "\n",
    "exported_pipeline.fit(training_features, training_target)\n",
    "results = exported_pipeline.predict(testing_features)\n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/x/opt/miniconda3/envs/mastar/lib/python3.10/site-packages/xgboost/compat.py:36: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.\n",
      "  from pandas import MultiIndex, Int64Index\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "30 operators have been imported by TPOT.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7d9a97541429479ab9dde55963fadb69",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Optimization Progress:   0%|          | 0/420 [00:00<?, ?pipeline/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/x/opt/miniconda3/envs/mastar/lib/python3.10/site-packages/xgboost/compat.py:36: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.\n",
      "  from pandas import MultiIndex, Int64Index\n",
      "/Users/x/opt/miniconda3/envs/mastar/lib/python3.10/site-packages/xgboost/compat.py:36: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.\n",
      "  from pandas import MultiIndex, Int64Index\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Skipped pipeline #8 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #10 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #16 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #18 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #20 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #22 due to time out. Continuing to the next pipeline.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/x/opt/miniconda3/envs/mastar/lib/python3.10/site-packages/xgboost/compat.py:36: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.\n",
      "  from pandas import MultiIndex, Int64Index\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Skipped pipeline #31 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #34 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #36 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #38 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #40 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #42 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #47 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #51 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #53 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #55 due to time out. Continuing to the next pipeline.\n",
      "\n",
      "Generation 1 - Current Pareto front scores:\n",
      "\n",
      "-1\t-138587.751580905\tExtraTreesRegressor(input_matrix, ExtraTreesRegressor__bootstrap=False, ExtraTreesRegressor__max_features=0.5, ExtraTreesRegressor__min_samples_leaf=7, ExtraTreesRegressor__min_samples_split=7, ExtraTreesRegressor__n_estimators=100)\n",
      "\n",
      "-2\t-62062.1953125\tLassoLarsCV(Normalizer(input_matrix, Normalizer__norm=l2), LassoLarsCV__normalize=False)\n",
      "Saving periodic pipeline from pareto front to tpot_checkpoints/pipeline_gen_1_idx_0_2022.05.26_21-57-35.py\n",
      "Saving periodic pipeline from pareto front to tpot_checkpoints/pipeline_gen_1_idx_1_2022.05.26_21-57-35.py\n",
      "_pre_test decorator: _random_mutation_operator: num_test=0 Found array with 0 feature(s) (shape=(50, 0)) while a minimum of 1 is required by ExtraTreesRegressor..\n",
      "_pre_test decorator: _random_mutation_operator: num_test=0 Unsupported set of arguments: The combination of penalty='l2' and loss='epsilon_insensitive' are not supported when dual=False, Parameters: penalty='l2', loss='epsilon_insensitive', dual=False.\n",
      "Pipeline encountered that has previously been evaluated during the optimization process. Using the score from the previous evaluation.\n",
      "Skipped pipeline #69 due to time out. Continuing to the next pipeline.\n",
      "\n",
      "Generation 2 - Current Pareto front scores:\n",
      "\n",
      "-1\t-136001.67007935382\tExtraTreesRegressor(input_matrix, ExtraTreesRegressor__bootstrap=True, ExtraTreesRegressor__max_features=0.6, ExtraTreesRegressor__min_samples_leaf=2, ExtraTreesRegressor__min_samples_split=8, ExtraTreesRegressor__n_estimators=100)\n",
      "\n",
      "-2\t-62062.1953125\tLassoLarsCV(Normalizer(input_matrix, Normalizer__norm=l2), LassoLarsCV__normalize=False)\n",
      "Saving periodic pipeline from pareto front to tpot_checkpoints/pipeline_gen_2_idx_0_2022.05.26_22-17-35.py\n",
      "Periodic pipeline was not saved, probably saved before...\n",
      "_pre_test decorator: _random_mutation_operator: num_test=0 Expected n_neighbors <= n_samples,  but n_samples = 50, n_neighbors = 68.\n",
      "Skipped pipeline #83 due to time out. Continuing to the next pipeline.\n",
      "\n",
      "Generation 3 - Current Pareto front scores:\n",
      "\n",
      "-1\t-136001.67007935382\tExtraTreesRegressor(input_matrix, ExtraTreesRegressor__bootstrap=True, ExtraTreesRegressor__max_features=0.6, ExtraTreesRegressor__min_samples_leaf=2, ExtraTreesRegressor__min_samples_split=8, ExtraTreesRegressor__n_estimators=100)\n",
      "\n",
      "-2\t-62062.1953125\tLassoLarsCV(Normalizer(input_matrix, Normalizer__norm=l2), LassoLarsCV__normalize=False)\n",
      "Periodic pipeline was not saved, probably saved before...\n",
      "Periodic pipeline was not saved, probably saved before...\n",
      "_pre_test decorator: _random_mutation_operator: num_test=0 Found array with 0 feature(s) (shape=(50, 0)) while a minimum of 1 is required by ExtraTreesRegressor..\n",
      "Pipeline encountered that has previously been evaluated during the optimization process. Using the score from the previous evaluation.\n",
      "Skipped pipeline #102 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #108 due to time out. Continuing to the next pipeline.\n",
      "\n",
      "Generation 4 - Current Pareto front scores:\n",
      "\n",
      "-1\t-136001.67007935382\tExtraTreesRegressor(input_matrix, ExtraTreesRegressor__bootstrap=True, ExtraTreesRegressor__max_features=0.6, ExtraTreesRegressor__min_samples_leaf=2, ExtraTreesRegressor__min_samples_split=8, ExtraTreesRegressor__n_estimators=100)\n",
      "\n",
      "-2\t-62062.1953125\tLassoLarsCV(Normalizer(input_matrix, Normalizer__norm=l2), LassoLarsCV__normalize=False)\n",
      "Periodic pipeline was not saved, probably saved before...\n",
      "Periodic pipeline was not saved, probably saved before...\n",
      "_pre_test decorator: _random_mutation_operator: num_test=0 Automatic alpha grid generation is not supported for l1_ratio=0. Please supply a grid by providing your estimator with the appropriate `alphas=` argument..\n",
      "Skipped pipeline #124 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #130 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #135 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #141 due to time out. Continuing to the next pipeline.\n",
      "\n",
      "Generation 5 - Current Pareto front scores:\n",
      "\n",
      "-1\t-136001.67007935382\tExtraTreesRegressor(input_matrix, ExtraTreesRegressor__bootstrap=True, ExtraTreesRegressor__max_features=0.6, ExtraTreesRegressor__min_samples_leaf=2, ExtraTreesRegressor__min_samples_split=8, ExtraTreesRegressor__n_estimators=100)\n",
      "\n",
      "-2\t-62062.1953125\tLassoLarsCV(Normalizer(input_matrix, Normalizer__norm=l2), LassoLarsCV__normalize=False)\n",
      "Periodic pipeline was not saved, probably saved before...\n",
      "Periodic pipeline was not saved, probably saved before...\n",
      "_pre_test decorator: _random_mutation_operator: num_test=0 Found array with 0 feature(s) (shape=(50, 0)) while a minimum of 1 is required by LassoLarsCV..\n",
      "-57328.5974554\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/x/opt/miniconda3/envs/mastar/lib/python3.10/site-packages/sklearn/metrics/_scorer.py:765: FutureWarning: sklearn.metrics.SCORERS is deprecated and will be removed in v1.3. Please use sklearn.metrics.get_scorer_names to get a list of available scorers and sklearn.metrics.get_metric to get scorer.\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "if __name__ == '__main__':\n",
    "    multiprocessing.set_start_method('forkserver')\n",
    "    pipeline_optimizer = TPOTRegressor(random_state=1, verbosity=3, n_jobs=4, generations=20, population_size=20,\n",
    "        periodic_checkpoint_folder='tpot_checkpoints', cv=5, scoring='neg_mean_squared_error', max_eval_time_mins=10)\n",
    "    pipeline_optimizer.fit(X_train, y_train)\n",
    "    print(pipeline_optimizer.score(X_test, y_test))\n",
    "    pipeline_optimizer.export('tpot_exported_pipeline.py')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The runs just kept crashing on the same \"process was killed\" error mentioned earlier. The longest run I was able to extract was 192 minutes. Maybe we need to run non-parallel.\n",
    "\n",
    "The only special result different than ExtraTreesRegressort was LassoLarsCV:\n",
    "\n",
    "```python\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.linear_model import LassoLarsCV\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.pipeline import make_pipeline\n",
    "from sklearn.preprocessing import Normalizer\n",
    "from tpot.export_utils import set_param_recursive\n",
    "\n",
    "# NOTE: Make sure that the outcome column is labeled 'target' in the data file\n",
    "tpot_data = pd.read_csv('PATH/TO/DATA/FILE', sep='COLUMN_SEPARATOR', dtype=np.float64)\n",
    "features = tpot_data.drop('target', axis=1)\n",
    "training_features, testing_features, training_target, testing_target = \\\n",
    "            train_test_split(features, tpot_data['target'], random_state=1)\n",
    "\n",
    "# Average CV score on the training set was: -62062.1953125\n",
    "exported_pipeline = make_pipeline(\n",
    "    Normalizer(norm=\"l2\"),\n",
    "    LassoLarsCV(normalize=False)\n",
    ")\n",
    "# Fix random state for all the steps in exported pipeline\n",
    "set_param_recursive(exported_pipeline.steps, 'random_state', 1)\n",
    "\n",
    "exported_pipeline.fit(training_features, training_target)\n",
    "results = exported_pipeline.predict(testing_features)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "19 operators have been imported by TPOT.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "06cf8b5ce96442208006dad97f474f83",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Optimization Progress:   0%|          | 0/420 [00:00<?, ?pipeline/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Skipped pipeline #10 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #12 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #14 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #17 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #19 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #21 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #23 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #25 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #27 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #29 due to time out. Continuing to the next pipeline.\n",
      "_pre_test decorator: _random_mutation_operator: num_test=0 Found array with 0 feature(s) (shape=(50, 0)) while a minimum of 1 is required by LassoLarsCV..\n",
      "_pre_test decorator: _random_mutation_operator: num_test=0 manhattan was provided as affinity. Ward can only work with euclidean distances..\n",
      "_pre_test decorator: _random_mutation_operator: num_test=0 Found array with 0 feature(s) (shape=(50, 0)) while a minimum of 1 is required by LassoLarsCV..\n",
      "Pipeline encountered that has previously been evaluated during the optimization process. Using the score from the previous evaluation.\n",
      "Pipeline encountered that has previously been evaluated during the optimization process. Using the score from the previous evaluation.\n",
      "Skipped pipeline #40 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #42 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #44 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #46 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #48 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #50 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #53 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #55 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #57 due to time out. Continuing to the next pipeline.\n",
      "\n",
      "Generation 1 - Current Pareto front scores:\n",
      "\n",
      "-1\t-1112434.225\tLassoLarsCV(CombineDFs(input_matrix, input_matrix), LassoLarsCV__normalize=True)\n",
      "Saving periodic pipeline from pareto front to tpot_checkpoints/pipeline_gen_1_idx_0_2022.05.27_11-39-57.py\n",
      "_pre_test decorator: _random_mutation_operator: num_test=0 Found array with 0 feature(s) (shape=(50, 0)) while a minimum of 1 is required by LassoLarsCV..\n",
      "Pipeline encountered that has previously been evaluated during the optimization process. Using the score from the previous evaluation.\n",
      "Pipeline encountered that has previously been evaluated during the optimization process. Using the score from the previous evaluation.\n",
      "Skipped pipeline #71 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #73 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #75 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #77 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #79 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #82 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #85 due to time out. Continuing to the next pipeline.\n",
      "\n",
      "Generation 2 - Current Pareto front scores:\n",
      "\n",
      "-1\t-1076020.5125\tLassoLarsCV(CombineDFs(input_matrix, input_matrix), LassoLarsCV__normalize=False)\n",
      "Saving periodic pipeline from pareto front to tpot_checkpoints/pipeline_gen_2_idx_0_2022.05.27_13-09-37.py\n",
      "_pre_test decorator: _random_mutation_operator: num_test=0 Found array with 0 feature(s) (shape=(50, 0)) while a minimum of 1 is required by LassoLarsCV..\n",
      "Pipeline encountered that has previously been evaluated during the optimization process. Using the score from the previous evaluation.\n",
      "Pipeline encountered that has previously been evaluated during the optimization process. Using the score from the previous evaluation.\n",
      "Skipped pipeline #97 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #99 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #104 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #106 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #108 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #111 due to time out. Continuing to the next pipeline.\n",
      "\n",
      "Generation 3 - Current Pareto front scores:\n",
      "\n",
      "-1\t-1076020.5125\tLassoLarsCV(CombineDFs(input_matrix, input_matrix), LassoLarsCV__normalize=False)\n",
      "Periodic pipeline was not saved, probably saved before...\n",
      "_pre_test decorator: _random_mutation_operator: num_test=0 Found array with 0 feature(s) (shape=(50, 0)) while a minimum of 1 is required by LassoLarsCV..\n",
      "_pre_test decorator: _random_mutation_operator: num_test=0 Found array with 0 feature(s) (shape=(50, 0)) while a minimum of 1 is required by LassoLarsCV..\n",
      "_pre_test decorator: _random_mutation_operator: num_test=0 Found array with 0 feature(s) (shape=(50, 0)) while a minimum of 1 is required by LassoLarsCV..\n",
      "Pipeline encountered that has previously been evaluated during the optimization process. Using the score from the previous evaluation.\n",
      "Pipeline encountered that has previously been evaluated during the optimization process. Using the score from the previous evaluation.\n",
      "Pipeline encountered that has previously been evaluated during the optimization process. Using the score from the previous evaluation.\n",
      "Pipeline encountered that has previously been evaluated during the optimization process. Using the score from the previous evaluation.\n",
      "Skipped pipeline #126 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #128 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #131 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #133 due to time out. Continuing to the next pipeline.\n",
      "\n",
      "Generation 4 - Current Pareto front scores:\n",
      "\n",
      "-1\t-1076020.5125\tLassoLarsCV(CombineDFs(input_matrix, input_matrix), LassoLarsCV__normalize=False)\n",
      "Periodic pipeline was not saved, probably saved before...\n",
      "_pre_test decorator: _random_mutation_operator: num_test=0 Found array with 0 feature(s) (shape=(50, 0)) while a minimum of 1 is required by LassoLarsCV..\n",
      "_pre_test decorator: _random_mutation_operator: num_test=0 cosine was provided as affinity. Ward can only work with euclidean distances..\n",
      "Pipeline encountered that has previously been evaluated during the optimization process. Using the score from the previous evaluation.\n",
      "Pipeline encountered that has previously been evaluated during the optimization process. Using the score from the previous evaluation.\n",
      "Skipped pipeline #148 due to time out. Continuing to the next pipeline.\n",
      "Skipped pipeline #150 due to time out. Continuing to the next pipeline.\n"
     ]
    },
    {
     "ename": "",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31mCanceled future for execute_request message before replies were done"
     ]
    },
    {
     "ename": "",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31mThe Kernel crashed while executing code in the the current cell or a previous cell. Please review the code in the cell(s) to identify a possible cause of the failure. Click <a href='https://aka.ms/vscodeJupyterKernelCrash'>here</a> for more info. View Jupyter <a href='command:jupyter.viewOutput'>log</a> for further details."
     ]
    }
   ],
   "source": [
    "tpot_config = {\n",
    "    'sklearn.linear_model.LassoLarsCV': {\n",
    "        'normalize': [True, False]\n",
    "    },\n",
    "    # Preprocessors\n",
    "    'sklearn.preprocessing.Binarizer': {\n",
    "        'threshold': np.arange(0.0, 1.01, 0.05)\n",
    "    },\n",
    "\n",
    "    'sklearn.decomposition.FastICA': {\n",
    "        'tol': np.arange(0.0, 1.01, 0.05)\n",
    "    },\n",
    "\n",
    "    'sklearn.cluster.FeatureAgglomeration': {\n",
    "        'linkage': ['ward', 'complete', 'average'],\n",
    "        'affinity': ['euclidean', 'l1', 'l2', 'manhattan', 'cosine']\n",
    "    },\n",
    "\n",
    "    'sklearn.preprocessing.MaxAbsScaler': {\n",
    "    },\n",
    "\n",
    "    'sklearn.preprocessing.MinMaxScaler': {\n",
    "    },\n",
    "\n",
    "    'sklearn.preprocessing.Normalizer': {\n",
    "        'norm': ['l1', 'l2', 'max']\n",
    "    },\n",
    "\n",
    "    'sklearn.kernel_approximation.Nystroem': {\n",
    "        'kernel': ['rbf', 'cosine', 'chi2', 'laplacian', 'polynomial', 'poly', 'linear', 'additive_chi2', 'sigmoid'],\n",
    "        'gamma': np.arange(0.0, 1.01, 0.05),\n",
    "        'n_components': range(1, 11)\n",
    "    },\n",
    "\n",
    "    'sklearn.decomposition.PCA': {\n",
    "        'svd_solver': ['randomized'],\n",
    "        'iterated_power': range(1, 11)\n",
    "    },\n",
    "\n",
    "    'sklearn.preprocessing.PolynomialFeatures': {\n",
    "        'degree': [2],\n",
    "        'include_bias': [False],\n",
    "        'interaction_only': [False]\n",
    "    },\n",
    "\n",
    "    'sklearn.kernel_approximation.RBFSampler': {\n",
    "        'gamma': np.arange(0.0, 1.01, 0.05)\n",
    "    },\n",
    "\n",
    "    'sklearn.preprocessing.RobustScaler': {\n",
    "    },\n",
    "\n",
    "    'sklearn.preprocessing.StandardScaler': {\n",
    "    },\n",
    "\n",
    "    'tpot.builtins.ZeroCount': {\n",
    "    },\n",
    "\n",
    "    'tpot.builtins.OneHotEncoder': {\n",
    "        'minimum_fraction': [0.05, 0.1, 0.15, 0.2, 0.25],\n",
    "        'sparse': [False],\n",
    "        'threshold': [10]\n",
    "    },\n",
    "\n",
    "\n",
    "    # Selectors\n",
    "    'sklearn.feature_selection.SelectFwe': {\n",
    "        'alpha': np.arange(0, 0.05, 0.001),\n",
    "        'score_func': {\n",
    "            'sklearn.feature_selection.f_regression': None\n",
    "        }\n",
    "    },\n",
    "\n",
    "    'sklearn.feature_selection.SelectPercentile': {\n",
    "        'percentile': range(1, 100),\n",
    "        'score_func': {\n",
    "            'sklearn.feature_selection.f_regression': None\n",
    "        }\n",
    "    },\n",
    "\n",
    "    'sklearn.feature_selection.VarianceThreshold': {\n",
    "        'threshold': [0.0001, 0.0005, 0.001, 0.005, 0.01, 0.05, 0.1, 0.2]\n",
    "    },\n",
    "\n",
    "    'sklearn.feature_selection.SelectFromModel': {\n",
    "        'threshold': np.arange(0, 1.01, 0.05),\n",
    "        'estimator': {\n",
    "            'sklearn.ensemble.ExtraTreesRegressor': {\n",
    "                'n_estimators': [100],\n",
    "                'max_features': np.arange(0.05, 1.01, 0.05)\n",
    "            }\n",
    "        }\n",
    "    }\n",
    "}\n",
    "\n",
    "pipeline_optimizer = TPOTRegressor(random_state=1, verbosity=3, generations=20, population_size=20,\n",
    "    periodic_checkpoint_folder='tpot_checkpoints', cv=5, scoring='neg_mean_squared_error', max_eval_time_mins=10, config_dict=tpot_config)\n",
    "pipeline_optimizer.fit(X_train, y_train)\n",
    "print(pipeline_optimizer.score(X_test, y_test))\n",
    "pipeline_optimizer.export('tpot_exported_pipeline.py')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Kernel crashed after ~4h hours:\n",
    "\n",
    "```\n",
    "Canceled future for execute_request message before replies were done\n",
    "The Kernel crashed while executing code in the the current cell or a previous cell. Please review the code in the cell(s) to identify a possible cause of the failure. Click here for more info. View Jupyter log for further details.\n",
    "```\n",
    "\n",
    "At this stage I gave up - the longest execution I could get running locally was around 5 hours, and the results seemed to always get stuck on the same pipelines, not improve, and also individual models were timing out (I'm guessing this could have something to do with the amount of features being fed in - spectrum of 4k points.\n",
    "\n",
    "The best solution found was:\n",
    "\n",
    "```python\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.linear_model import LassoLarsCV\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.pipeline import make_pipeline\n",
    "from sklearn.preprocessing import Normalizer\n",
    "from tpot.export_utils import set_param_recursive\n",
    "\n",
    "# NOTE: Make sure that the outcome column is labeled 'target' in the data file\n",
    "tpot_data = pd.read_csv('PATH/TO/DATA/FILE', sep='COLUMN_SEPARATOR', dtype=np.float64)\n",
    "features = tpot_data.drop('target', axis=1)\n",
    "training_features, testing_features, training_target, testing_target = \\\n",
    "            train_test_split(features, tpot_data['target'], random_state=1)\n",
    "\n",
    "# Average CV score on the training set was: -62062.1953125\n",
    "exported_pipeline = make_pipeline(\n",
    "    Normalizer(norm=\"l2\"),\n",
    "    LassoLarsCV(normalize=False)\n",
    ")\n",
    "# Fix random state for all the steps in exported pipeline\n",
    "set_param_recursive(exported_pipeline.steps, 'random_state', 1)\n",
    "\n",
    "exported_pipeline.fit(training_features, training_target)\n",
    "results = exported_pipeline.predict(testing_features)\n",
    "```\n",
    "\n",
    "Also see the `tpot_checkpoints` directory for best solutions found in each generation."
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "7e66e1a829a2a82f0cb6aeacf2ea5f44cd8944c8a18096b63b985eee8a9e9eb6"
  },
  "kernelspec": {
   "display_name": "Python 3.10.4 ('mastar')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}