Machine Learning Guide.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "<font size=\"4\">\n",
    "    \n",
    "# A practical guide to machine learning materials (with python)\n",
    "\n",
    "This guide is supposed to act as a general outline for simple machine learning. I have included a \n",
    "supplimentary python file (**SPF**), found [here](https://github.com/kaaiian/Houston_ML_in_MSE_workshop/blob/master/machine_learn_elastic_properties.py), to get you started with the coding aspects. \n",
    "\n",
    "This workflow should be sufficient to build simple models to predict materials properties based on the composition. \n",
    "\n",
    "If you are completely new to coding, you can follow [this link](https://www.youtube.com/watch?v=5mDYijMfSzs) (a good video I found on youtube) to get a brief walkthrough on how to download python using the anaconda distribution on a windows computer. [This link](https://www.youtube.com/watch?v=WhBg-mC0ChQ&t) goes through the supplimentary python file (**SPF**) line-by-line and explains in detail how the code is being used and why. \n",
    "\n",
    "-----\n",
    "\n",
    "## Find some materials data\n",
    "\n",
    "In order to do some machine learning, we need materials data. \n",
    "\n",
    "Some easy places to get data include: \n",
    "\n",
    "- [Citrination](https://citrination.com). You will need to make an account. Probably want to use API's to get a useful amount of data.\n",
    "\n",
    "- [Materials Project](https://materialsproject.org/). Same deal. You will need to make an account. Again, you will probably want to use API's to get a useful amount of data. <br>\n",
    "\n",
    "- [Aflow](http://aflowlib.org/). Just need to search for the property you want. Data is retruned in html table. This can easily be copied into excel.\n",
    "\n",
    "- [Matminer](https://hackingmaterials.github.io/matminer/). \"Matminer is a Python library for data mining the properties of materials\". A small bit of python will give you quick access to Citrination and Materials Project data as well as select data  from individual publications.\n",
    "\n",
    "- [Literature](https://scholar.google.com/).\n",
    "\n",
    "For this guide, we can use data obtained from the [Aflow](http://aflowlib.org/). This [link](https://youtu.be/ptLzE0o6sGs) showns me using aflow to get elastic data and saving it to a CSV file.\n",
    "\n",
    "(*If you follow the video, don't just grab the first thousand. You will likely want to return 3000 compounds to make sure you get everything! There are a total of 2382 ternary-compounds with elastic data*)\n",
    "\n",
    "## Clean the data\n",
    "\n",
    "Once we have data, we want to make sure that the data is clean and usable. This means we need to know how we want to use the data. \n",
    "\n",
    "In our case, we will be using the composition to try to predict some materials properties. \n",
    "\n",
    "For this reason, we will want to make sure our data is formated to clearly have a composition and its associated properties. The following table gives an example of what clean data could look like. Using elastic properties as our target for learning, we might get something like this.\n",
    "\n",
    "| formula | Bulk Modulus (GPa) |\n",
    "|:---|:---|\n",
    "| Ag10S8Sb2 | 35.2 |\n",
    "| Ag1Al1S2 | 71.6 |\n",
    "|... | ... |\n",
    "| Sn6V6Y1 | 112.3 |\n",
    "\n",
    "(*If you grabbed the aflow data following the youtube link. You will notice the 'ENTRY ' column has the formula as well as the unique aflow id. in square-brackets. I remove this using lines 41-77 in the **SPF**.*)\n",
    "\n",
    "## Evaluate the data for issues.\n",
    "\n",
    "---\n",
    "- Are there duplicates? \n",
    "- What is the distribution? \n",
    "- Are there distinct chemical groups? \n",
    "- Are there useful auxilary properties?\n",
    "---\n",
    "\n",
    "### Duplicates?\n",
    "\n",
    "One we have the data, we will want to make sure there are no problems with the data. A first thing to check is the existance of duplicate instances of data. For us, that means looking for duplicate compositions in our dataset, we will want to devise a plan to address or remove duplicates. Note, sometimes duplicate formulae can have very different properties if they are in different environments (E.g. high vs. low temp/pressure, Different crystal/micro structure, thin-film vs. bulk, etc.).\n",
    "\n",
    "Common ways to handle duplicate instances of data:\n",
    "- Drop all but 1 at random\n",
    "    - a simple approach that is useful if values are not too dissimilar\n",
    "- Take the mean value\n",
    "    - the same idea, but feels a bit better\n",
    "- Take the median value\n",
    "    - proably a better approach, but can be ill-defined for even numbers\n",
    "- Distringuish between duplicates\n",
    "    - add additional features that encodes extra information such as structure, pressure, temp, etc.\n",
    "\n",
    "### Distribution?\n",
    "\n",
    "After taking care of duplicate values, we can look at the distribution of the data to make sure that there is no funny business going on. However, before we look at the distribution it is often useful to prepare our \"engineering senses\". \n",
    "\n",
    "Some questions to ask before looking at the data:\n",
    "- What do I expect this distribution to look like?\n",
    "- Do I expect there to be negative values?\n",
    "- What materials do I expect to be at the extremes?\n",
    "- What would unreasonable values look like?\n",
    "- Can I expect to see extreme outliers?\n",
    "- Does it make more sense to predict the log(property) (think electrical conductivity)\n",
    "\n",
    "Here are my (non-expert) oppinions on bulk modulus\n",
    "- What do I expect this distribution to look like?\n",
    "    - Log normal-ish is pretty common\n",
    "- Do I expect there to be negative values?\n",
    "    - That doesn't make sense to me given the definition of bulk modulus\n",
    "- What materials do I expect to be at the extremes?\n",
    "    - Maybe formulae with boron, nitorgen or carbon\n",
    "- What would unreasonable values look like?\n",
    "    - Diamond is really hard and has a bulk modulus of ~550 (googled that. lol). Values in that range are probably worth investigating.\n",
    "- Can I expect to see extreme outliers\n",
    "    - My gut feeling: \"DFT is less common for mundane compositions. So we should probably see couple.\" (might be very wrong)\n",
    "\n",
    "### Distinct Chemical Groups?\n",
    "\n",
    "If we are being hardcore, we can try to partition our data into distinct chemical groups that exists (for a good paper on why and how to do this, click [here](https://pubs.rsc.org/en/content/articlelanding/2018/me/c8me00012c)). We will skip this, but say it might be something to try.\n",
    "\n",
    "### Auxilary Information?\n",
    "\n",
    "This is related to working with duplicates. Often times there is more to our data than just \"composition\". We should stop here and consider: \n",
    "1. Do we have properties besides just the composition? and\n",
    "2. What tool we want to make and how do we plan to use it? \n",
    "\n",
    "If we want to make a model that can predict our property given nothing but composition.... than our machine learning algorithm needs to only be trained on composition.\n",
    "\n",
    "If we want to identify possible [superhard materials](https://pubs.acs.org/doi/abs/10.1021/jacs.8b02717) (shout out to the Brgoch group) from the pearson crystal database, than we can include materials properties associated with the structure.\n",
    "\n",
    "(*If you want to add auxilary properties, these should be added to the feature vector befor any scaling steps.*)\n",
    "\n",
    "## Featurize the data (machine readable representation of formula)\n",
    "\n",
    "Once we really know what we are dealing with, we need to get our data into a machine learnable format. That is to say, machine learning algorithms require a vector representation of the data (making this vector is the 'featurization' step). But how do we vectorize materials data? There are a lot of ways, [matminer](https://hackingmaterials.github.io/matminer/featurizer_summary.html) has a table of techinques and is a place to start\n",
    "\n",
    "For now, lets look at our data and consier what we are working with.\n",
    "\n",
    "| formula | Bulk Modulus (GPa) |\n",
    "|:---|:---|\n",
    "| Ag10S8Sb2 | 35.2 |\n",
    "| Ag1Al1S2 | 71.6 |\n",
    "|... | ... |\n",
    "| Sn6V6Y1 | 112.3 |\n",
    "\n",
    "It looks like we want to correlate a chemical formula to bulk modulus. We want to create a vector that can represent any composition. Perhaps the simplest way to do this is to make a vector where each component represents an element ([cool paper here](https://www.nature.com/articles/s41598-018-35934-y)).\n",
    "\n",
    "That vector would look something like this:\n",
    "\n",
    "$v_{formula} = $ (Ag, Al, B, ..., O, ..., Zr)\n",
    "\n",
    "Then, if we wanted to conver Al$_2$O$_3$, it would be as simple as putting a \"2\" in the aluminum dimension, a \"3\" in the oxygen dimension, with \"0\" everywhere else.\n",
    "\n",
    "$v_{Al_2O_3} = $ (0, 2, 0, ..., 3, ..., 0)\n",
    "\n",
    "This is a simple approach that illustrates the idea. More elegantly, we can do featurization using a composition-based feature vector ([read more here](https://www.nature.com/articles/npjcompumats201628)).\n",
    " \n",
    "\n",
    "## Split the data into training set (for building the model) and test set (for ensuring out-of-sample performance)\n",
    "\n",
    "If are going to model the data, it is best practice to split our data into a training and test set. We will use the training set to optimize and fit the model, and the test set as a final indicator of model performance. Because most machine learning models have a number of tunable parameters, we will want to split the training set into train & validation parts. This can be done in a way similar to the train test split. Or it can be done using a cross-validation scheme. The following illustrates the a typical **train-test split followed by 3-fold cross-validation**:\n",
    "\n",
    "![train-test-validation](https://upload.wikimedia.org/wikipedia/commons/thumb/1/12/Train-Test-Validation.png/640px-Train-Test-Validation.png)\n",
    "\n",
    "## Consider scaling the features\n",
    "\n",
    "Once we have our features. We will probably want to scale them. \n",
    "\n",
    "Most machine learning algorithms rely on gradient descent to find the optimal model fit. \n",
    "\n",
    "If features have drastically different values, issues can arrise with during the search for our optimum. \n",
    "\n",
    "**Inser image of scaled vs non-scaled for gradient boosting\n",
    "\n",
    "Algorithms that will most likely require gradient descent:\n",
    "- Linear Regression, \n",
    "- Ridge Regression, \n",
    "- Lasso,\n",
    "- Support Vector Machines, \n",
    "- Neural Networks, \n",
    "- Logistic regression, etc.\n",
    "\n",
    "Algorithms that do not require gradient descent (usually decision tree based):\n",
    "- Decision Tree, \n",
    "- Random Forest,\n",
    "- Gradient Tree Boosting, etc.\n",
    "\n",
    "## Select desired algorithms\n",
    "\n",
    "At this point, we can decide on the algorithm we want to use.\n",
    "\n",
    "Here are some general characteristics for a few popular algorithms\n",
    "- random forest (rf): \n",
    "    - usually works\n",
    "    - doesn't need any feature scaling\n",
    "    - little effort to get good results\n",
    "    - scales pretty good with data\n",
    "    - can learn complex non-linear relationships\n",
    "\n",
    "- support vector machine (svm): (regression=(SVR), classification=(SVC))\n",
    "    - requires detailed parameter tuning\n",
    "    - models are slow to train with large amounts of data ( $\\approx 10000$)\n",
    "    - resulting models are generally very good\n",
    "    - can learn complex non-linear relationships\n",
    "    \n",
    "- linear regression (lr):\n",
    "    - a simple approach, has some interpretability\n",
    "    - regularization can be added if overfitting\n",
    "        - L1 regularization = Lasso\n",
    "        - L2 regularization = Ridge Regression\n",
    "    - extermely fast. Works with huge amounts of data\n",
    "    - very limited model complexity (often underfits)\n",
    "\n",
    "- neural network (nn):\n",
    "    - more difficult to implement (reasonably sized models require additional software)\n",
    "    - requires detailed parameter tuning\n",
    "    - scales well with large amounts of data\n",
    "    - networks can be designed to learn arbitrarily complex relationships\n",
    "        - particularly useful for image data\n",
    "    - usually considered the most accurate\n",
    "\n",
    "A safe bet is to try many or all of them! Lets go simple and use a support vector regression (SVR).\n",
    "\n",
    "## Optimize algorithm parameters\n",
    "\n",
    "Regardless of the algorithm we choose, we will want to optimize the model parameters to get the best performance possible. Trying a bunch of different parameter combinations is a simple way to do this. We can do just that using a 'grid search'. Once a gride search is complete, we can judge whether our best parameters seem optimal. If so, we can make a final model and check our performance test set. If not, we can repeat the grid search changing the range of parameters we look at. \n",
    "\n",
    "(*When using ridge, lasso, svm, or nn models, it is a general rule of thumb to search logarithmic space for most parameters.*)\n",
    "\n",
    "## Check performance on the test set\n",
    "\n",
    "Once we are satisfied with our performance on the training set, we can move to the test set. The test set should have similar, or slightly worse, performance to the training data. This is the final step and should be done after all optimization has taken place and we are sure about our model and parameter choices.\n",
    "\n",
    "To check our performance, we will take the optimal parameters from the grid search and train a new model with all of the training data. Once a model is trained with all the data, we can use it to predict on the test data (Do not forget, if you scaled the training data, you need to also scale the test data in the same way)! Once we have predictions on the test data we can calculate an R$^2$ score, and RMSE and we can also plot the results. If your model is succesful, we can take it into the \"real world\" and get started predicting!\n",
    "\n",
    "## Make predictions on future compounds\n",
    "\n",
    "Once we have a fully trained and validated model, we of course want to use it. Here are some steps to make sure a new formula is ready to be put through our model to generate a prediction.\n",
    "\n",
    "New prediction checklist:\n",
    "1. Convert the composition/formula into a feature vector\n",
    "2. If you added auxilary information, add that to the feature vector\n",
    "    - the meaning of each vector component must match the vectors from the training set)\n",
    "3. Use scaling (calulated from the training) set on the new feature vector\n",
    "4. Input prepared vector into the model to generate prediction!\n",
    "\n",
    "\n",
    "</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}