Taller Clase 1.ipynb

{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"Taller Clase 1.ipynb","version":"0.3.2","provenance":[],"collapsed_sections":[]},"kernelspec":{"name":"python3","display_name":"Python 3"}},"cells":[{"cell_type":"markdown","metadata":{"id":"ZfXL9kavj4M_","colab_type":"text"},"source":["# Artificial Intelligence Workshop Day 1\n","\n","By Juan-Pablo Silva (https://github.com/juanpablos) jpsilva@dcc.uchile.cl, Alexandre Bergel (abergel@dcc.uchile.cl) and Alejandro Infante (ainfante@dcc.uchile.cl)."]},{"cell_type":"markdown","metadata":{"id":"R7H90BX5Z9A4","colab_type":"text"},"source":["## Pytorch Basics\n","\n","We will be importing our basic packages. Pytorch, Numpy (for some auxiliary functions) and matplotlib (to plot our results)."]},{"cell_type":"code","metadata":{"id":"a6ht3uFbW3So","colab_type":"code","colab":{}},"source":["import numpy as np\n","import torch\n","import math\n","from matplotlib import pyplot as plt"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"JHzY5spLaHLD","colab_type":"text"},"source":["### Tensors\n","\n","Tensors are the basic structure in Pytorch, and work just like numpy arrays. If you are not familiar with numpy arrays, think of matlab's matrices.\n","\n","A tensor is just a N-dimensional matrix. A vector is 1 dimensional, a matrix is 2 dimensional."]},{"cell_type":"code","metadata":{"id":"vHMxmBsZoxMz","colab_type":"code","outputId":"8fb95546-cf59-4d1c-f7bb-4c14afb9ec2e","executionInfo":{"status":"ok","timestamp":1565053654584,"user_tz":240,"elapsed":2530,"user":{"displayName":"Alexandre Bergel","photoUrl":"https://lh4.googleusercontent.com/-H7nD27bt7tg/AAAAAAAAAAI/AAAAAAAAAIA/M0i2TiMptd4/s64/photo.jpg","userId":"02563535896661050156"}},"colab":{"base_uri":"https://localhost:8080/","height":85}},"source":["# We create tensors with torch.tensor(...)\n","tensor_1 = torch.tensor(10)\n","print(\"Printing a tensor: \", tensor_1)\n","print(f\"You can ask the shape of a tensor with .shape or .size(): {tensor_1.shape}\")\n","print(f\"Ask for a tensor data type with .dtype: {tensor_1.dtype}\")\n","print(f\"Single element tensors can have their value retrieved with .item(): {tensor_1.item()}\")"],"execution_count":0,"outputs":[{"output_type":"stream","text":["Printing a tensor:  tensor(10)\n","You can ask the shape of a tensor with .shape or .size(): torch.Size([])\n","Ask for a tensor data type with .dtype: torch.int64\n","Single element tensors can have their value retrieved with .item(): 10\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"lEoqqkkkW-7j","colab_type":"code","outputId":"a77d2be1-6194-40f2-8e00-0c3c02bb8501","executionInfo":{"status":"ok","timestamp":1565053654585,"user_tz":240,"elapsed":2501,"user":{"displayName":"Alexandre Bergel","photoUrl":"https://lh4.googleusercontent.com/-H7nD27bt7tg/AAAAAAAAAAI/AAAAAAAAAIA/M0i2TiMptd4/s64/photo.jpg","userId":"02563535896661050156"}},"colab":{"base_uri":"https://localhost:8080/","height":68}},"source":["# We can define an tensor from a sequence, for example a list\n","tensor_1 = torch.tensor([1,2,3,4,5,6])\n","# The ones(shape) constructor let us define a tensor with all ones\n","tensor_2 = torch.ones(6)\n","\n","# Lets print our new tensors\n","print(tensor_1, tensor_2)\n","# There are their shapes\n","print(tensor_1.shape, tensor_2.shape)\n","\n","# But their types are different?\n","print(tensor_1.dtype, tensor_2.dtype)"],"execution_count":0,"outputs":[{"output_type":"stream","text":["tensor([1, 2, 3, 4, 5, 6]) tensor([1., 1., 1., 1., 1., 1.])\n","torch.Size([6]) torch.Size([6])\n","torch.int64 torch.float32\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"mHPoFljbfjwd","colab_type":"text"},"source":["The types of our new tensors are different. Pytorch does not allow any type of computation between different types of data. In this case we have that tensor_1 es an interger tensor, and that tensor_2 is a float tensor. Since we are more interested in floating points, lets convert tensor_1 to a FloatTensor."]},{"cell_type":"code","metadata":{"id":"kB1EQmtpXM_V","colab_type":"code","outputId":"0be31e18-2bfb-4511-eaae-1c632ee03927","executionInfo":{"status":"ok","timestamp":1565053654586,"user_tz":240,"elapsed":2466,"user":{"displayName":"Alexandre Bergel","photoUrl":"https://lh4.googleusercontent.com/-H7nD27bt7tg/AAAAAAAAAAI/AAAAAAAAAIA/M0i2TiMptd4/s64/photo.jpg","userId":"02563535896661050156"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["# Here we make tensor_1 into a FloatTensor.\n","tensor_1 = tensor_1.type(torch.FloatTensor)\n","# We could also have defined it like that in the beggining with the dtype parameter\n","# tensor_1 = torch.tensor([1,2,3,4,5,6], dtype=torch.float)\n","print(tensor_1.dtype)"],"execution_count":0,"outputs":[{"output_type":"stream","text":["torch.float32\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"Cv5z_O9Z3V4c","colab_type":"text"},"source":["We can do simple operations with the tensors, just like matrices in matlab and numpy arrays."]},{"cell_type":"code","metadata":{"id":"WuJ9ase3XYg9","colab_type":"code","outputId":"589bc215-f51d-4949-e921-3242bf7e4318","executionInfo":{"status":"ok","timestamp":1565053654587,"user_tz":240,"elapsed":2432,"user":{"displayName":"Alexandre Bergel","photoUrl":"https://lh4.googleusercontent.com/-H7nD27bt7tg/AAAAAAAAAAI/AAAAAAAAAIA/M0i2TiMptd4/s64/photo.jpg","userId":"02563535896661050156"}},"colab":{"base_uri":"https://localhost:8080/","height":68}},"source":["# Element-wise addition\n","print(tensor_1 + tensor_2)\n","# Element-wise multiplication\n","print(tensor_1 * tensor_2)\n","# Dot product, scalar product\n","print(tensor_1 @ tensor_2)"],"execution_count":0,"outputs":[{"output_type":"stream","text":["tensor([2., 3., 4., 5., 6., 7.])\n","tensor([1., 2., 3., 4., 5., 6.])\n","tensor(21.)\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"0GXAsc8i4a2C","colab_type":"text"},"source":["Many times we would like to change a tensor's dimension. To do this we can use the .view() method.\n","In this method we pass the shape we want the tensor to be. If we pass a -1 value in a dimension, pytorch can infer the value based on the remaining elements.\n","You could use the .reshape() method with the same parameters and get the same result. They are not the same method internally, but most of the time you would want to use view. You can look at the Pytorch documentation for more details on this."]},{"cell_type":"code","metadata":{"id":"pSyYzZGmXa2q","colab_type":"code","outputId":"95ad5a6e-85a4-4af9-9cd1-12a95178973e","executionInfo":{"status":"ok","timestamp":1565053654588,"user_tz":240,"elapsed":2399,"user":{"displayName":"Alexandre Bergel","photoUrl":"https://lh4.googleusercontent.com/-H7nD27bt7tg/AAAAAAAAAAI/AAAAAAAAAIA/M0i2TiMptd4/s64/photo.jpg","userId":"02563535896661050156"}},"colab":{"base_uri":"https://localhost:8080/","height":136}},"source":["# Let's create a matrix.\n","# Create an array of 9 elements, from 1 to 9\n","tensor_1 = torch.tensor(list(range(1,10)), dtype=torch.float)\n","print(tensor_1)\n","\n","# Right now our tensor is 1x9\n","# We can resize it to be a 3x3 matrix with .view()\n","# We ask for a tensor with 3 rows and let pytorch infer how many columns (3 in this case)\n","# because we have 9 elements, 3xX=9 => X=3\n","tensor_1 = tensor_1.view(3,-1)\n","\n","# We create a 3x3 tensor with random values.\n","tensor_2 = torch.randn(3,3)\n","\n","# You can see the results here. Both tensors have the same shape.\n","print(tensor_1, tensor_1.shape)\n","print(tensor_2, tensor_2.shape)"],"execution_count":0,"outputs":[{"output_type":"stream","text":["tensor([1., 2., 3., 4., 5., 6., 7., 8., 9.])\n","tensor([[1., 2., 3.],\n","        [4., 5., 6.],\n","        [7., 8., 9.]]) torch.Size([3, 3])\n","tensor([[ 1.9460, -1.8746,  0.2707],\n","        [-0.7172, -1.7842, -2.1367],\n","        [-2.7372, -1.2388, -1.7430]]) torch.Size([3, 3])\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"YeJt4CSG6TZE","colab_type":"text"},"source":["Same as before, when we showed operations with one dimensional tensors (arrays), here we work with matrices."]},{"cell_type":"code","metadata":{"id":"fGguI-dBX81u","colab_type":"code","outputId":"35474789-d02f-46bf-94a1-bb90e602338f","executionInfo":{"status":"ok","timestamp":1565053654589,"user_tz":240,"elapsed":2365,"user":{"displayName":"Alexandre Bergel","photoUrl":"https://lh4.googleusercontent.com/-H7nD27bt7tg/AAAAAAAAAAI/AAAAAAAAAIA/M0i2TiMptd4/s64/photo.jpg","userId":"02563535896661050156"}},"colab":{"base_uri":"https://localhost:8080/","height":170}},"source":["# Element-wise addition\n","print(tensor_1 + tensor_2)\n","# Element-wise multiplication\n","print(tensor_1 * tensor_2)\n","# Dot product\n","print(tensor_1 @ tensor_2) # matrix multiplication"],"execution_count":0,"outputs":[{"output_type":"stream","text":["tensor([[2.9460, 0.1254, 3.2707],\n","        [3.2828, 3.2158, 3.8633],\n","        [4.2628, 6.7612, 7.2570]])\n","tensor([[  1.9460,  -3.7491,   0.8121],\n","        [ -2.8687,  -8.9208, -12.8200],\n","        [-19.1604,  -9.9101, -15.6871]])\n","tensor([[ -7.7000,  -9.1592,  -9.2317],\n","        [-12.2252, -23.8516, -20.0586],\n","        [-16.7504, -38.5441, -30.8855]])\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"4yWodyLN_B5E","colab_type":"text"},"source":["Clearly, as we know, there are conditions for the matrix multiplication. Let A be a MxN matrix, and B a KxL matrix. We can calculate the dot product of A and B only if N=K, and the result will be a MxL size matrix.\n","For the element-wise operations, the size of the tensors must be the same."]},{"cell_type":"code","metadata":{"id":"xoB86pTnYU-O","colab_type":"code","outputId":"0d9c330c-1256-44fb-bef6-a8258e9d229d","executionInfo":{"status":"ok","timestamp":1565053654590,"user_tz":240,"elapsed":2334,"user":{"displayName":"Alexandre Bergel","photoUrl":"https://lh4.googleusercontent.com/-H7nD27bt7tg/AAAAAAAAAAI/AAAAAAAAAIA/M0i2TiMptd4/s64/photo.jpg","userId":"02563535896661050156"}},"colab":{"base_uri":"https://localhost:8080/","height":102}},"source":["# We have a 2x5 matrix here\n","tensor_1 = torch.tensor(list(range(1,11)), dtype=torch.float).view(2,-1)\n","print(tensor_1)\n","# and a 5x9 matrix here\n","tensor_2 = torch.randn(5,9)\n","\n","print(f\"tensor_1: {tensor_1.shape}\")\n","print(f\"tensor_2: {tensor_2.shape}\")\n","\n","# print(tensor_1 + tensor_2) not the same size\n","# print(tensor_1 * tensor_2) not the same size\n","# the resulting shape is a 2x9 matrix\n","print((tensor_1 @ tensor_2).shape)"],"execution_count":0,"outputs":[{"output_type":"stream","text":["tensor([[ 1.,  2.,  3.,  4.,  5.],\n","        [ 6.,  7.,  8.,  9., 10.]])\n","tensor_1: torch.Size([2, 5])\n","tensor_2: torch.Size([5, 9])\n","torch.Size([2, 9])\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"usMh8AQgaim4","colab_type":"text"},"source":["### Gradients\n","\n","A gradient is just the multi-variable generalization of the derivative. In the case of a matrix, we end up calculating the Jacobian matrix.\n","\n","Pytorch takes care of all the derivation for us, this is what is called an _automatic differentiation engine_.\n","\n","We will start with a simple example:\n","\n","$\n","y= x ^3 \\\\\n","\\frac{\\partial y}{\\partial x} = \\frac{\\partial x^3}{\\partial x} = 3x^2\n","$"]},{"cell_type":"code","metadata":{"id":"f2bWnVsEak1O","colab_type":"code","outputId":"33804662-83c1-4da9-9420-304a80152427","executionInfo":{"status":"ok","timestamp":1565053654590,"user_tz":240,"elapsed":2304,"user":{"displayName":"Alexandre Bergel","photoUrl":"https://lh4.googleusercontent.com/-H7nD27bt7tg/AAAAAAAAAAI/AAAAAAAAAIA/M0i2TiMptd4/s64/photo.jpg","userId":"02563535896661050156"}},"colab":{"base_uri":"https://localhost:8080/","height":68}},"source":["# We define a single value tensor\n","tensor_1 = torch.tensor(10, requires_grad=True, dtype=torch.float)\n","# cube it\n","tensor_2 = tensor_1 ** 3\n","print(f\"Lets call this x, x={tensor_1}\")\n","print(f\"Lets call this y(x)=x^3, y(10)={tensor_2}\")\n","\n","# We have the ecuation y = x ^ 3\n","# and its derivative is dy = 3x^2\n","# Lets let Pytorch take care of the gradient\n","tensor_2.backward()\n","\n","# We know that the answer should be 3x^2 with x=10 so\n","# x=10 => dy=300\n","# Pytorch thinks the same!\n","print(f\"dy/dx|x=10={tensor_1.grad}\")"],"execution_count":0,"outputs":[{"output_type":"stream","text":["Lets call this x, x=10.0\n","Lets call this y(x)=x^3, y(10)=1000.0\n","dy/dx|x=10=300.0\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"2jjebfZKQKpL","colab_type":"text"},"source":["Okay, maybe that was too easy. Lets try with matrices now.\n","\n","Let $x$ and $y$ be our variables.\n","\n","$\n","x = \n","\\begin{pmatrix}\n","1 & 2 & 3 \\\\\n","4 & 5 & 6 \\\\\n","7 & 8 & 9\n","\\end{pmatrix}\\\\\n","y = \n","\\begin{pmatrix}\n","2 & 3 & 4 \\\\\n","3 & 4 & 5 \\\\\n","4 & 5 & 6\n","\\end{pmatrix}\n","$\n","\n","The function we want to calculate is: $f(x, y) = sum((x \\cdot y)^2)$. We have $x$ that is a $3x3$ matrix, and $y$ that is another $3x3$ matrix. The reason that we are using the $sum(\\cdot)$ function is because there is a slight complication in _machine learning_: all we do is optimize. The traditional way to optimize is for a single variable, multi-objective optimization very hard. If we would not sum the result of the multiplication we would be entering a multi-objective optimization, so by summing it we output a scalar. It could be the sum, max, mean, etc. It does not need to be the sum function, but we will stick with that.\n","\n","Because the derivative of the sum is the sum of the derivative, we extract the sum.\n","Lest's calculate the gradient with respect to $x$ and $y$.\n","\n","$\n","\\begin{align}\n","f(x, y) & =  \\sum(x \\cdot y)^2 \\\\\n","\\frac{\\partial f(x, y)}{\\partial x} & = \\frac{\\partial\\sum(x \\cdot y)^2}{\\partial x} \\\\\n","& = \\sum\\frac{\\partial(x \\cdot y)^2}{\\partial x}\n","\\end{align}\n","$\n","\n","For easier comprehension, lets remove the sum:\n","\n","$\n","\\begin{align}\n","& = \\frac{\\partial(x \\cdot y)^2}{\\partial x} \\\\\n","& = 2x \\cdot y\\frac{\\partial(x \\cdot y)}{\\partial x} \\\\\n","& = 2x \\cdot y \\cdot y\n","\\end{align}\n","$\n","\n","Then for with respect to y we have:\n","\n","$\n","\\begin{align}\n","& = \\frac{\\partial(x \\cdot y)^2}{\\partial y} \\\\\n","& = 2 \\frac{\\partial(x \\cdot y)}{\\partial y} \\cdot x \\cdot y \\\\\n","& = 2x^T \\cdot x \\cdot y\n","\\end{align}\n","$\n","\n","Remember that we are working with matrices, here the order in which they are multiplied matters.\n","Let's replace our variables with the known values.\n","\n","$\n","\\begin{align}\n","\\frac{\\partial f(x, y)}{\\partial x} & = 2x \\cdot y \\cdot y \\\\\n","    &= 2\n","\\begin{pmatrix}\n","1 & 2 & 3 \\\\\n","4 & 5 & 6 \\\\\n","7 & 8 & 9\n","\\end{pmatrix}\n","*\n","\\begin{pmatrix}\n","2 & 3 & 4 \\\\\n","3 & 4 & 5 \\\\\n","4 & 5 & 6\n","\\end{pmatrix}\n","*\n","\\begin{pmatrix}\n","2 & 3 & 4 \\\\\n","3 & 4 & 5 \\\\\n","4 & 5 & 6\n","\\end{pmatrix}\n","\\\\\n","& = \n","\\begin{pmatrix}\n","492 & 648 & 804 \\\\\n","1176 & 1548 & 1920 \\\\\n","1860 & 2448 & 3036\n","\\end{pmatrix}\n","\\\\\n","\\frac{\\partial f(x, y)}{\\partial y} & = 2x^T \\cdot x \\cdot y \\\\\n","&= 2\n","\\begin{pmatrix}\n","1 & 2 & 3 \\\\\n","4 & 5 & 6 \\\\\n","7 & 8 & 9\n","\\end{pmatrix}^T\n","*\n","\\begin{pmatrix}\n","1 & 2 & 3 \\\\\n","4 & 5 & 6 \\\\\n","7 & 8 & 9\n","\\end{pmatrix}\n","*\n","\\begin{pmatrix}\n","2 & 3 & 4 \\\\\n","3 & 4 & 5 \\\\\n","4 & 5 & 6\n","\\end{pmatrix}\n","\\\\\n","     &= 2\n","\\begin{pmatrix}\n","1 & 4 & 7 \\\\\n","2 & 5 & 8 \\\\\n","3 & 6 & 9\n","\\end{pmatrix}\n","*\n","\\begin{pmatrix}\n","1 & 2 & 3 \\\\\n","4 & 5 & 6 \\\\\n","7 & 8 & 9\n","\\end{pmatrix}\n","*\n","\\begin{pmatrix}\n","2 & 3 & 4 \\\\\n","3 & 4 & 5 \\\\\n","4 & 5 & 6\n","\\end{pmatrix}\n","\\\\\n","& = \\begin{pmatrix}\n","1452 & 1920 & 2388 \\\\\n","1734 & 2292 & 2850 \\\\\n","2016 & 2664 & 3312\n","\\end{pmatrix}\n","\\end{align}\n","$\n","\n","Now lets check with Pytorch if we get the same result."]},{"cell_type":"code","metadata":{"id":"8vAIQpeULM4X","colab_type":"code","outputId":"8474c80e-a40f-4cba-b139-4ef22fc4de2b","executionInfo":{"status":"ok","timestamp":1565053654591,"user_tz":240,"elapsed":2271,"user":{"displayName":"Alexandre Bergel","photoUrl":"https://lh4.googleusercontent.com/-H7nD27bt7tg/AAAAAAAAAAI/AAAAAAAAAIA/M0i2TiMptd4/s64/photo.jpg","userId":"02563535896661050156"}},"colab":{"base_uri":"https://localhost:8080/","height":153}},"source":["tensor_1 = torch.tensor([[1, 2, 3], [4, 5, 6],[7,8,9]], requires_grad=True, dtype=torch.float)\n","tensor_2 = torch.tensor([[2,3,4], [3,4,5], [4,5,6]], requires_grad=True, dtype=torch.float)\n","tensor_4 = ((tensor_1 @ tensor_2)**2).sum()\n","\n","tensor_4.backward()\n","print(f\"The gradient for tensor_1 is:\\n {tensor_1.grad}\")\n","print(f\"The gradient for tensor_1 is:\\n {tensor_2.grad}\")"],"execution_count":0,"outputs":[{"output_type":"stream","text":["The gradient for tensor_1 is:\n"," tensor([[ 492.,  648.,  804.],\n","        [1176., 1548., 1920.],\n","        [1860., 2448., 3036.]])\n","The gradient for tensor_1 is:\n"," tensor([[1452., 1920., 2388.],\n","        [1734., 2292., 2850.],\n","        [2016., 2664., 3312.]])\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"hpJAr12_Ysp2","colab_type":"text"},"source":["Okay, so now that we have a basic understanding about how we calculate the gradients, we won't be calculating them by hand anymore in the workshop. Instead, we will just use the _automatic differentiation_ provided by Pytorch."]},{"cell_type":"markdown","metadata":{"id":"KdDnEROxgZi4","colab_type":"text"},"source":["## Stochastic Gradient Descent (SGD)\n","\n","The Gradient Descent, or Stochastic Gradient Descent, is an optimization algorithm for updating _machine learning_ model parameters. There are multiple other updating algorithms (namely optimizers), but SGD is a simpler one that is easy to implement.\n","\n","For this optimizer we need a learning rate, usually a small float number. This will be a constant (in SGD at least) that will reflect on how abruptly we will be updating the network parameters. The bigger it is, the more we change in each iteration, the smaller, the more subtle the change."]},{"cell_type":"code","metadata":{"id":"g9DTP2qKdr0O","colab_type":"code","outputId":"c5c2a4c8-2b58-4814-fc14-3ef8ad8274c5","executionInfo":{"status":"ok","timestamp":1565053654592,"user_tz":240,"elapsed":2239,"user":{"displayName":"Alexandre Bergel","photoUrl":"https://lh4.googleusercontent.com/-H7nD27bt7tg/AAAAAAAAAAI/AAAAAAAAAIA/M0i2TiMptd4/s64/photo.jpg","userId":"02563535896661050156"}},"colab":{"base_uri":"https://localhost:8080/","height":136}},"source":["# Lets set the learning rate to 0.5\n","lr = 0.5\n","# We generate 2 random tensors.\n","# One 1x2\n","_input = torch.randn(1,2, dtype=torch.float)\n","# and the other 2x1\n","tensor_2 = torch.randn(2,1)\n","# we need to explicitly tell pytorch that we want this tensor to hace its gradients calculated\n","tensor_2.requires_grad_()\n","\n","print(f\"Our tensor before the update:\\n {tensor_2}\")\n","\n","# Multiply the input tensor with our middle tensor\n","y = _input @ tensor_2\n","print(\"The output {}\".format(y))\n","\n","# calculate the gradients\n","y.backward()\n","\n","# this is just an indicator to pytorch to not listen to the following operations\n","# if we dont use this, the gradient could change unexpectedly\n","with torch.no_grad():\n","    # update the tensor with the learning rate and its gradient\n","    tensor_2 -= tensor_2.grad * lr\n","    # set the gradient to zero, we dont want this to accumulate\n","    tensor_2.grad.zero_()\n","    \n","    \n","print(f\"Our tensor after the update:\\n {tensor_2}\")"],"execution_count":0,"outputs":[{"output_type":"stream","text":["Our tensor before the update:\n"," tensor([[0.6099],\n","        [0.5049]], requires_grad=True)\n","The output tensor([[0.4886]], grad_fn=<MmBackward>)\n","Our tensor after the update:\n"," tensor([[0.4669],\n","        [0.1937]], requires_grad=True)\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"d2QZVpbPkP0j","colab_type":"text"},"source":["### Exercise: train AND logic gate\n","\n","In the following exercise you will be implementing a neural network capable of representing the AND logic gate. This is an ilustrative example for you to get familiar with Pytorch."]},{"cell_type":"code","metadata":{"id":"9SDJVpi0un5L","colab_type":"code","colab":{}},"source":["# We defined some functions for you.\n","# The sigmoid activation function\n","def sigmoid(z):\n","    return 1 / (1 + torch.exp(-z))\n","   \n","# A loss function. We want to minimize this.\n","# This measures the distance between the network's output, and out desired output.\n","def MSE_loss(output, target):\n","    return ((output - target)**2).mean()\n","    \n","# Here we select 1000x2 random boolean values\n","dataset = torch.from_numpy(np.random.choice(a=[0, 1], size=(1000, 2)))\n","# and convert them to FloatTensors so we can work with them\n","x_train_and = dataset.type(torch.FloatTensor)\n","# Here we calculate our desired outputs for each of the inputs.\n","y_train_and = (dataset[:,0] & dataset[:,1]).type(torch.FloatTensor).unsqueeze(1)\n","\n","# samples could be useful if you train in batches\n","samples, _ = x_train_and.shape\n","# how many times the network will go over the whole dataset\n","epochs = 100\n","# if you use batches, the batch size\n","batch_size = 10\n","# the learning rate\n","lr = 0.1"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"ZOwn-2SDamEr","colab_type":"text"},"source":["Remember the formula for the neural networks: $$f_{\\theta, b}(x) = \\theta \\cdot x + b$$\n","$f_{\\theta, b}$ is commonly writen only as $f_{\\theta}$, $\\theta$ represents the networks parameters. In this case we represent $\\theta$ as the network weights and $b$ as the bias. For the function $f$ representing the neural network, we need to find the parameters $\\theta$ and $b$ such that the network gives the correct outputs."]},{"cell_type":"code","metadata":{"id":"uEs__41rhp0u","colab_type":"code","outputId":"89875a88-3d5a-4aa9-963f-1ae343d194a0","executionInfo":{"status":"error","timestamp":1565053654617,"user_tz":240,"elapsed":2219,"user":{"displayName":"Alexandre Bergel","photoUrl":"https://lh4.googleusercontent.com/-H7nD27bt7tg/AAAAAAAAAAI/AAAAAAAAAIA/M0i2TiMptd4/s64/photo.jpg","userId":"02563535896661050156"}},"colab":{"base_uri":"https://localhost:8080/","height":232}},"source":["# we want to keep track of the loss for each epoch, so we can visualize it later\n","# append your loss values to this list once per epoch\n","loss_values = []\n","\n","# -------- Learn AND ---------\n","weights = torch.randn(2, 1)\n","weights.requires_grad_()\n","bias = torch.randn(1)\n","bias.requires_grad_()\n","\n","def model(xb):\n","    # Define how the neural network works\n","    return \"define me!\"\n","\n","for epoch in range(epochs):\n","    loss_epoch = []\n","    for i in range((samples - 1) // batch_size + 1):\n","        start_i = i * batch_size\n","        end_i = start_i + batch_size        \n","        xb = x_train_and[start_i:end_i]\n","        yb = y_train_and[start_i:end_i]\n","        \n","        # something should be done here with our inputs!\n","        #####################################\n","        \n","        loss_epoch.append(loss.item())\n","\n","        # We need to calculate the gradients and update the parameters!\n","        ##################\n","            \n","    loss_values.append(np.mean(loss_epoch))\n","    \n","    if epoch % 10 == 0:\n","            print(f'Epoch [{epoch}/{epochs}] Loss: {loss_values[-1]:.4f}')\n","    \n","\n","# ------- END Learn AND -------\n","print(\"Ready\")"],"execution_count":0,"outputs":[{"output_type":"error","ename":"NameError","evalue":"ignored","traceback":["\u001b[0;31m---------------------------------------------------------------------------\u001b[0m","\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)","\u001b[0;32m<ipython-input-39-0be262ab76a1>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     22\u001b[0m         \u001b[0;31m#####################################\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     23\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 24\u001b[0;31m         \u001b[0mloss_epoch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mloss\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mitem\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     25\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     26\u001b[0m         \u001b[0;31m# We need to calculate the gradients and update the parameters!\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;31mNameError\u001b[0m: name 'loss' is not defined"]}]},{"cell_type":"code","metadata":{"id":"WXSx3f67piY4","colab_type":"code","colab":{}},"source":["# Okay! Let's see how the network's loss (or error) changed over time\n","plt.plot(loss_values)\n","\n","# Here we ask the network what it thinks there values should be\n","print(model(torch.tensor([[0,0],[0,1],[1,0],[1,1]], dtype=torch.float)))\n","# they look good!"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"bvB2-fihs1b_","colab_type":"text"},"source":["## More Pytorch\n","\n","Iterating over the dataset manually, setting the batch size and slicing the data, manually updating the parameters of the model, implementing the loss and activation functions... Pytorch have us covered here!"]},{"cell_type":"markdown","metadata":{"id":"ScF6-6bJtUDf","colab_type":"text"},"source":["First we need to import some more packages"]},{"cell_type":"code","metadata":{"id":"wzdb5DDzsgxn","colab_type":"code","colab":{}},"source":["# There is a bunch of optimizers here, aside from SGD\n","from torch import optim\n","# Package containing lots of functions to build neural networks\n","from torch import nn\n","# To load and manage our datasets. Very useful when we have huge datasets\n","from torch.utils.data import DataLoader,TensorDataset"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"AWBiOxLeucbj","colab_type":"text"},"source":["First, lets introduce the _TensorDataset_ and _DataLoader_. The _TensorDataset_ is just an object that will be storing our data so we don't have to slice it ourselves. The _DataLoader_ allow us to iterate over our data without we worring about it. It also implements parallelism, so if your dataset is too big, or you do some type of preprocessing _on the fly_ you can set multiple workers to load it at the same time."]},{"cell_type":"code","metadata":{"id":"mkuHvXVdufi3","colab_type":"code","colab":{}},"source":["# Create the dataset\n","train_ds_and = TensorDataset(x_train_and, y_train_and)\n","# Use the dataset as a data source for the dataloader\n","# Here we also pass the batch size to allow for mini-batches.\n","train_dl_and = DataLoader(train_ds_and, batch_size=batch_size, shuffle=True)"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"LiLLKZs5uLQF","colab_type":"text"},"source":["Leaving the network hanging around in our code is really messy and will surely raise some problems later. We can create a class to contain it, and this way add more layers and configurations when needed with not much effort."]},{"cell_type":"code","metadata":{"id":"VczSBNX3tYkH","colab_type":"code","colab":{}},"source":["# We just need to extend from the nn.Module class and implement the forward method\n","class Network(nn.Module):\n","    def __init__(self):\n","        # initialize the super class\n","        super().__init__()\n","        # create a single layex with 2 inputs and 1 output\n","        self.layer_1 = nn.Linear(2, 1)\n","        \n","    def forward(self, xb):\n","        # apply the input to the layer\n","        o = self.layer_1(xb)\n","        # apply an activation function\n","        o = torch.sigmoid(o)\n","        # we are done with the forward pass!\n","        return o\n","    "],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"0rWS53EGuOFD","colab_type":"text"},"source":["Our loss function is also implemented in Pytorch, lets just use it here."]},{"cell_type":"code","metadata":{"id":"NE_Hj4S2uSWY","colab_type":"code","colab":{}},"source":["loss_func = nn.MSELoss()"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"cq4W_vwVuT7u","colab_type":"text"},"source":["For us the use SGD, we need to tell Pytorch which parameters we want to be updated by the optimizer. In our case, we want Pytorch to take care of every parameter so we don't have to worry, but if some day you need to, you could input only some parameters here."]},{"cell_type":"code","metadata":{"id":"ulXIpLp1vKrV","colab_type":"code","colab":{}},"source":["# Create a network object. This is just the class we just implemented\n","model = Network()\n","# Create the SGD optimized and pass the network parameters\n","opt = optim.SGD(model.parameters(), lr=lr)"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"vmCZ6d9cvoOQ","colab_type":"text"},"source":["To train it is best to create a function, generally called _fit_, so everything is tidy and we know where to look in case we want to change anything."]},{"cell_type":"code","metadata":{"id":"MvuPCjrrvpu7","colab_type":"code","colab":{}},"source":["# The fit function needs the model we want to train, the optimizer we are using,\n","# the number of epochs we want it to run, the data we want it to train on,\n","# and the loss function we want to minimize.\n","def fit(model, optim, n_epochs, data, loss_func):\n","    # we will be storing out epoch loss here\n","    acc_loss = []\n","    \n","    # do n_epochs many epochs\n","    for epoch in range(n_epochs):\n","        # epoch loss stores the loss per mini-batch. We will be averaging this later\n","        epoch_loss = []\n","        # iterate over the dataset\n","        for xb, yb in data:\n","            # make a prediction. Here the network (our model) is calling the *forward* method\n","            pred = model(xb)\n","            # the loss compare how close we are with our predictions to the real value\n","            loss = loss_func(pred, yb)\n","\n","            # mini-batch loss\n","            epoch_loss.append(loss.item())\n","            # calculate the gradients\n","            loss.backward()\n","            # make the SGD do its thing.\n","            # this will update all the parameters we passed in the constructor\n","            # just like we manually updated then before\n","            opt.step()\n","            # never forget to reset the gradients!\n","            opt.zero_grad()\n","            \n","        # average the mini-batch loses and append them to the epoch loss\n","        acc_loss.append(np.mean(epoch_loss))\n","        \n","        if epoch % 10 == 0:\n","            print(f'Epoch [{epoch}/{epochs}] Loss: {acc_loss[-1]:.4f}')\n","        \n","    return acc_loss"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"6FPRmNZDwu1q","colab_type":"code","colab":{}},"source":["# Here we call the training function.\n","# The function returns a list will all the historic losses per epoch\n","# so we can visualize it and check if the model is performing as expected\n","loss_values = fit(model=model, optim=opt, n_epochs=100, data=train_dl_and, loss_func=loss_func)\n","\n","# plot the losses\n","plt.plot(loss_values)\n","# check if the network really learnt the logic gate\n","print(model(torch.tensor([[0,0],[0,1],[1,0],[1,1]], dtype=torch.float)))"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"Q5ITKUEnxOsT","colab_type":"text"},"source":["### Exercise: train the XOR logic gate\n","\n","In this exercise you will be implementing a neural network that will be able to represent the XOR logic gate. Once you have implemented the network, modify the number of layers, learning rate and number of neurons to see if it affects the learning speed of the network, how the accuracy change over time, etc.\n","\n","Hint: remember the logic gate tables and where AND and OR differs from XOR."]},{"cell_type":"code","metadata":{"id":"R6MN113rxF0z","colab_type":"code","colab":{}},"source":["# data\n","dataset = torch.from_numpy(np.random.choice(a=[0, 1], size=(1000, 2)))\n","x_train_xor = dataset.type(torch.FloatTensor)\n","# xor calculation\n","y_train_xor = (dataset[:,0] ^ dataset[:,1]).type(torch.FloatTensor).unsqueeze(1)\n","\n","epochs = 500\n","batch_size = 10\n","lr = 0.1"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"v3orF1etxgim","colab_type":"code","colab":{}},"source":["# Add your epoch losses here\n","loss_values = []\n","\n","\n","# -------- Learn XOR ---------\n","train_ds_xor = TensorDataset(x_train_xor, y_train_xor)\n","train_dl_xor = DataLoader(train_ds_xor, batch_size=batch_size, shuffle=True)\n","\n","class Network(nn.Module):\n","    def __init__(self):\n","        super().__init__()\n","        # define the layers!\n","        #########################\n","        \n","    def forward(self, xb):\n","        # how the layers are going to be composed?\n","        ########################\n","        o = xb\n","        return o\n","    \n","# Remember you need a loss function, an optimizer and to instantiate the network\n","########################\n","\n","# complete this function as we saw before\n","def fit(model, optim, n_epochs, data, loss_func):  \n","    # you can ignore the logging function, but it helps debug!\n","    ########################\n","        \n","# general fit parameters\n","fit(model=model, optim=opt, n_epochs=epochs, data=train_dl_xor, loss_func=loss_func)\n","\n","# ------- END Learn XOR -------\n","print(\"Ready\")"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"bCRPLL99zF_N","colab_type":"code","colab":{}},"source":["# Lets see how it went!\n","plt.plot(loss_values)\n","# Check the outputs\n","print(model(torch.tensor([[0,0],[0,1],[1,0],[1,1]], dtype=torch.float)))"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"D2TCp4Tz0Y6Q","colab_type":"text"},"source":["### Exercise: train a polinomial function\n","\n","In this exercise you will need a bit more things to take into account. We will be introducing a data scaler. This will take your data, and scale it to a defined range, generally between 0 and 1.\n","\n","Why do you think we need to scale our data? Try to remove that lines and feed the unscaled data to the network. Does it work? What happens? Can you answer why is that the case?"]},{"cell_type":"code","metadata":{"id":"E0faFNe-7meD","colab_type":"code","colab":{}},"source":["# import the scaler from sklearn\n","from sklearn.preprocessing import MinMaxScaler"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"Gtt1n6t8zeHM","colab_type":"code","colab":{}},"source":["# Define your polinomial function here\n","def polinomio(x):\n","    return 10*x**5 + x + 5\n","\n","# generate the data points\n","step = 0.01\n","x_train_poli_pre = np.arange(-50, 50 + step, step).reshape(-1, 1)\n","# create the real values\n","y_train_poli_pre = polinomio(x_train_poli_pre)\n","\n","# scale the data\n","scaler_x = MinMaxScaler((0, 1)).fit(x_train_poli_pre)\n","scaler_y = MinMaxScaler((0, 1)).fit(y_train_poli_pre)\n","\n","# create tensors to feed to the neural network\n","x_train_poli = torch.tensor(scaler_x.transform(x_train_poli_pre), dtype=torch.float)\n","y_train_poli = torch.tensor(scaler_y.transform(y_train_poli_pre), dtype=torch.float)"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"uJx5S_3D1nWq","colab_type":"code","colab":{}},"source":["# -------- Learn poli ---------\n","epochs = 1000\n","batch_size = 100\n","lr = 0.1\n","loss_values = []\n","train_ds_poli = TensorDataset(x_train_poli, y_train_poli)\n","train_dl_poli = DataLoader(train_ds_poli, batch_size=batch_size, shuffle=True)\n","\n","class Network(nn.Module):\n","    def __init__(self):\n","        super().__init__()\n","        # Try with different layers\n","        ########################\n","        \n","    def forward(self, xb):\n","        # how the layers are going to be composed?\n","        ########################\n","        o = xb\n","        return o\n","    \n","# Remember you need a loss function, an optimizer and to instantiate the network\n","\n","# complete this function as we saw before\n","def fit(model, optim, n_epochs, data, loss_func):   \n","    # you can ignore the logging function, but it helps debug!\n","        \n","# general fit parameters\n","fit(model=model, optim=opt, n_epochs=epochs, data=train_dl_poli, loss_func=loss_func)\n","\n","# -------- END Learn poli ---------\n","print(\"Ready\")"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"BodVfnFt2kia","colab_type":"code","colab":{}},"source":["# Lets compare our results with the real ones!\n","plt.plot(x_train_poli.view(-1).detach().numpy(), y_train_poli.view(-1).detach().numpy())\n","plt.plot(x_train_poli.view(-1).detach().numpy(), model(x_train_poli).view(-1).detach().numpy())"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"NI3dJUrG3pvD","colab_type":"code","colab":{}},"source":["# And how the training went\n","plt.plot(loss_values)"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"M7FxF7PZdSpc","colab_type":"text"},"source":["## Classification\n","\n","Uo to this point we have only been training on regression tasks. This means that we have a number we want to reach.\n","There is another type of task called _classification_. Here our network does not output a single number, but many of them. To be precise, it outputs how many classes we have. For example, if we have Cat and Dog pictures, and we would like to classify them, we would have 2 classes, so the network will have 2 outputs.\n","\n","We introduce here the one-hot encoding, a very useful technique to represent nominal data. In the above example of Cat and Dog pictures, we have 2 possible classifications for an image, it is Cat or it is Dog. We could represent this as a single output value where values lower that 0.5 are Cat and higher are Dogs. But what if we had 10 different classes? We could set intervals of 0.1 to represent each of the classes, but this start to get confusing and in practice does not work very well.\n","The one-hot encoding solves this. If we have 2 possible classes, we create a 2 value vector [v1, v2]. If Our network thinks the picture is of a Cat, we would like v1 to be 1, and v2 to be 0. Likewise, if it thinks it is of a Dog, we want v1 to be 0 and v1 to be 1. This way we can encode how many different classes we want."]},{"cell_type":"markdown","metadata":{"id":"I8UnrUfOkJU-","colab_type":"text"},"source":["### Iris dataset\n","\n","The iris dataset is a well known multi-variate dataset that consists of 150 samples of 4 different measures of 3 type of flowers. We will be using it to train a neural network to be able to predict which type of flower a datapoint is, based on the measurements and characteristics."]},{"cell_type":"code","metadata":{"id":"92gGNfjmdWi3","colab_type":"code","colab":{}},"source":["# First, lets import the data set from sklearn\n","from sklearn.datasets import load_iris\n","# We also want this very useful auxiliary function\n","# we use it to split out data in training and testing sets\n","# this way we can check if our model is capable of generalizing\n","# what we showed it\n","from sklearn.model_selection import train_test_split"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"O7-Tuv0qeF9_","colab_type":"code","colab":{}},"source":["# Load the dataset\n","iris = load_iris()\n","# here we have the datapoints\n","iris_data = iris[\"data\"].astype('float32')\n","# here we have the type of flower each datapoint is\n","iris_target = iris[\"target\"]\n","\n","# split our data into training and testing sets\n","X_train_raw, X_test_raw, y_train, y_test = train_test_split(iris_data, iris_target, test_size=0.33, random_state=42)\n","\n","# we scale our input data for it not to explode\n","# we should ONLY scale the data on the TRAINING data\n","# it is cheating if you also use the testing data\n","scaler_x = MinMaxScaler((0, 1)).fit(X_train_raw)\n","# scale the training and testing data with what is seen on the training data only\n","X_train, X_test = scaler_x.transform(X_train_raw), scaler_x.transform(X_test_raw)\n","\n","# get the tensors to feed to the network\n","X_train, X_test, y_train, y_test = map(torch.from_numpy, [X_train, X_test, y_train, y_test])"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"7ygXZOEqeJnu","colab_type":"code","colab":{}},"source":["# Same as before, create the network class\n","class Network(torch.nn.Module):\n","    def __init__(self):\n","        super().__init__()\n","        # Here we will be using 2 layers a 4x20 and a 20x3.\n","        # We have 4 inputs and 3 outputs\n","        self.layer_1 = torch.nn.Linear(4, 20)\n","        self.layer_2 = torch.nn.Linear(20, 3)\n","    def forward(self,xb):\n","        # apply the layers\n","        o = self.layer_1(xb)\n","        # relu is an activation function defined as max(0, xb).\n","        # basically clips all negative values\n","        o = torch.relu(o)\n","        o = self.layer_2(o)\n","        return o"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"QsgOT-pQe5Dl","colab_type":"code","colab":{}},"source":["# Same hiperparameters as always\n","epochs = 1500\n","batch_size = 100\n","lr = 0.1\n","loss_values = []\n","\n","# the dataset and dataloader\n","train_ds_iris = TensorDataset(X_train, y_train)\n","train_dl_iris = DataLoader(train_ds_iris, batch_size=batch_size, shuffle=True)\n","\n","# For classification problems we use a different loss function called\n","# Cross Entropy. This is also known as the negative log likelihood (nll).\n","# The Cross Entropy is indeed the nll with a softmax layer applied at the end.\n","# All these confusing names are needed so we optimize for a single class in the output\n","# meaning, decrease the values for the classes that are not correct, and increase\n","# the correct values.\n","loss_func = nn.CrossEntropyLoss()\n","model = Network()\n","# the optimizer is the same\n","opt = optim.SGD(model.parameters(), lr=lr)\n","def fit(model, optim, n_epochs, data, loss_func):    \n","    for epoch in range(n_epochs):\n","        epoch_loss = []\n","        for xb, yb in data:\n","            pred = model(xb)\n","            loss = loss_func(pred, yb)\n","\n","            epoch_loss.append(loss.item())\n","            loss.backward()\n","            opt.step()\n","            opt.zero_grad()\n","            \n","        loss_values.append(np.mean(epoch_loss))\n","        \n","        if epoch % 50 == 0:\n","            print(f'Epoch [{epoch}/{epochs}] Loss: {loss_values[-1]:.4f}')\n","        \n","fit(model=model, optim=opt, n_epochs=epochs, data=train_dl_iris, loss_func=loss_func)\n","\n","print(\"Ready\")"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"xCRVlbwwe7vd","colab_type":"code","colab":{}},"source":["# plot the loss\n","plt.plot(loss_values)"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"PUXdNnBy2Xwn","colab_type":"code","colab":{}},"source":["# This cool function let us create a good basic report with\n","# our model's accuracy\n","from sklearn.metrics import classification_report"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"IrLZdJ3wswhL","colab_type":"code","colab":{}},"source":["# Our network have 3 outputs, but we only care about the highest value.\n","# The highest value means that the network thinks that is the correct class\n","# Because we used one-hot enconding, the index of the highest value is the\n","# predicted class. We use the max function to get the indices for each sample\n","_, y_test_pred = torch.max(model(X_test), dim=1)\n","\n","# lets generate the report!\n","print(classification_report(y_test, y_test_pred, target_names=iris[\"target_names\"]))"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"kso3D-U1kSDN","colab_type":"text"},"source":["### Exercise: train on the MNIST dataset\n","\n","The MNIST dataset is a classic dataset almost always used to teach neural networks as it presents an interesting task to solve.\n","\n","The MNIST dataset is a set of hand-written numbers, from 0 to 9. As you know, everyone writes in a different way, but us humans are, almost all the time, able to differentiate from one number from another. Here we will be teaching a neural network to do the same and check how it performs.\n","\n","The dataset consists of 70.000 samples, each of a 28x28 matrix. We will be using 60.000 samples to train our network, and the other 10.000 will be used for testing.\n","Since the numbers are in grey scale, we can just have a 28x28 matrix showing in each value how back the cell is. If the numbers had colours, we would have to take another approach to solve the problem, using Convolutional Neural Networks.\n","\n","Before we start, we must note that we do not know how to handle a 28x28 input matrix, we just know how to handle vectors. What we can do to workaround this is reshaping the 28x28 matrix in a 784 element vector. Now he have everything to start training.\n","\n","Okay, so let's begin."]},{"cell_type":"code","metadata":{"id":"bSpS0lTUs2dC","colab_type":"code","colab":{}},"source":["# First lets import the dataset with a simple call\n","from torchvision import datasets\n","# This package will let us make some modifications to out data\n","import torchvision"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"8ZisS8brtpW7","colab_type":"code","colab":{}},"source":["# Download and load the dataset\n","train_data = datasets.MNIST('data', train=True, download=True)\n","test_data = datasets.MNIST('data', train=False, download=True)\n","\n","# The dataset comes in a Byte format, so we convert it to Float\n","X_train, y_train = train_data.data.type(torch.FloatTensor), train_data.targets\n","X_test, y_test = test_data.data.type(torch.FloatTensor), test_data.targets\n","\n","# The dataset comes in a (Samples x rows x columns) manner. We do not know how to\n","# handle the (rows x columns) input, so we just reshape it to a\n","# rows*columns long vector. In this case we will have a (Samples x (rows*columns))\n","# input tensor, being the first dimension the number of datapoints, and the second\n","# the input dimension.\n","# We use the handy -1 notation to keep the first dimension untouched, and\n","# reshape the rest of the matrix to a vector.\n","X_train, X_test = X_train.view(X_train.shape[0], -1), X_test.view(X_test.shape[0], -1)"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"6dwsNQwLwVLO","colab_type":"code","colab":{}},"source":["# Let's take a look at some example points from the dataset, just to see how they look.\n","fig = plt.figure(figsize=(6, 4))\n","for i, index in enumerate(np.random.randint(0, X_train.shape[0], size=6)):\n","    ax = fig.add_subplot(2, 3, i+1, xticks=[], yticks=[])\n","    ax.imshow(X_train[index].reshape(28, 28), cmap='gray', interpolation='none')\n","    ax.set_title(\"Number: {}\".format(y_train[index]))"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"Re7mO7NXsQMp","colab_type":"code","colab":{}},"source":["# Time to train. Modify these parameters accordingly\n","import math\n","epochs = 20\n","batch_size = 10000 # best to be a big number to make it faster\n","lr = 0.1"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"ToZ-0zm62EHC","colab_type":"code","colab":{}},"source":["# Add your epoch losses here\n","loss_values = []\n","\n","\n","# -------- Learn MNIST ---------\n","train_ds_mnist = TensorDataset(X_train, y_train)\n","test_ds_mnist = TensorDataset(X_test, y_test)\n","train_dl_mnist = DataLoader(train_ds_mnist, batch_size=batch_size, shuffle=True)\n","test_dl_mnist = DataLoader(test_ds_mnist, batch_size=batch_size, shuffle=True)\n","\n","class Network(nn.Module):\n","    def __init__(self):\n","        super().__init__()\n","        # define the network!\n","        # with one layer it already works\n","        # remember we have a 28x28 input matrix (but we only know how to work with vectors!)\n","        # and we have a 10 possible outputs\n","        # so... ?\n","        \n","    def forward(self, xb):\n","        # define how the network works\n","        ###################\n","        o = xb\n","        return o\n","    \n","    \n","# Remember you need a loss function, an optimizer and to instantiate the network\n","# in this case the loss funtion can't be the MSE (why?), so which could be?\n","\n","# complete this function as we saw before\n","def fit(model, optim, n_epochs, data, loss_func):   \n","    # you can ignore the logging function, but it helps debug!\n","        \n","# general fit parameters\n","fit(model=model, optim=opt, n_epochs=epochs, data=train_dl_mnist, loss_func=loss_func)\n","\n","# -------- END Learn MNIST ---------\n","print(\"Ready\")"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"WxMnfmGAgX5B","colab_type":"code","colab":{}},"source":["# Lets plot our loss function!\n","plt.plot(loss_values)\n","\n","# Make the predictions over the test set\n","# Remember we have not show these datapoints to the network\n","_, y_test_pred = torch.max(model(X_test), dim=1)\n","\n","# Check how our network performs for each of the labels\n","print(classification_report(y_test, y_test_pred))"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"VP7VF7ZPC5sc","colab_type":"text"},"source":["We can see that the model is not perfect, as it fail to recognize several samples. This number let us know that the network still have room for improvement, but we still have around a 90% success rate, that seems quite good.\n","\n","Let's take a look at some predictions from our network. We will be putting correct label for each sample in parentheses. Keep running this cell until you see some red labels. That means the network failed to identify correctly the number in the picture. Do you find that number hard to identify? Do you think it is a valid mistake?"]},{"cell_type":"code","metadata":{"id":"TlJ6BL9sgiwP","colab_type":"code","colab":{}},"source":["n = 20\n","fig = plt.figure(figsize=(25, 4))\n","for i, index in enumerate(np.random.randint(0, X_test.shape[0], size=n)):\n","    ax = fig.add_subplot(2, n/2, i+1, xticks=[], yticks=[])\n","    ax.imshow(X_test[index].reshape(28, 28), cmap='gray', interpolation='none')\n","    ax.set_title(f\"pred: {y_test_pred[index]} (real: {y_test[index]})\", \n","                 color=(\"green\" if y_test_pred[index]==y_test[index] else \"red\"))"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"eGx1j70hD29R","colab_type":"text"},"source":["### Now is time to test it on your own!\n","\n","All the data in the MNIST dataset is provided. We can see some cool predictions and all but... does it really work?\n","Let's try something, let's draw some numbers ourselves, upload them to this notebook, and see if the network can identify then correctly!"]},{"cell_type":"code","metadata":{"id":"70wUNnSdd5kT","colab_type":"code","colab":{}},"source":["# Import this package to upload files to the notebook\n","from google.colab import files"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"4QqoHgm3EZnF","colab_type":"text"},"source":["To draw our numbers, we could take a pencil and draw it in a papel sheet, then take a photo of it, tranfer it to the computer, resize it, and upload it here. But that takes too much time, so let's use an online image editor.\n","\n","We recommend tested with https://www.favicon-generator.org/image-editor/ and it worked pretty well. Here are the instructions to generate your own number:\n","1. https://www.favicon-generator.org/image-editor/\n","2. Resize the canvas clicking on Image > Size... > enter a 28 x 28 value.\n","3. Now the canvas is quite small, so zoom in until you are confortable.\n","4. Click the Paint bucket and select the color black. Then click on the canvas to paint it black. We want to paint the whole background black.\n","5. Now select the Brush option and select the white color. Adjust the size to around 4.\n","6. Draw a number\n","7. Export your number with File > Save as... > and save it with jpg extension.\n","\n","We will be using an easy and fast method to load out images our model. For that reason, follow the following instructions to generate a folder structure suitable for the method we will be using:\n","1. Create a folder called 'temp'. It can be any name you want, but you will have to modify the name in the following cells if you do use other name.\n","2. Create sub folders with the numbers you want to upload. For example, if I drew a 5 and 8, I will create a subfolder called 5, and other called 8.\n","3. Move the drawn numbers to their correspoding folder.\n","4. Compress the 'temp' folder into a zip file. If you are in windows, you can just right click the 'temp' folder > Send to > Compressed (zipped) folder. In MacOS, right click the folder > compress items. In linux just use the zip command `zip -r temp.zip temp`.\n","5. Run the next cell to upload the zipped folder and unzip it automatically into the notebook."]},{"cell_type":"code","metadata":{"id":"d94kOGhdDjYi","colab_type":"code","colab":{}},"source":["# Navigate and upload the zipped temo folder\n","files.upload()\n","# unzips\n","!unzip temp.zip "],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"8K6u9gS-JXIT","colab_type":"text"},"source":["We use ImageFolder to load and make automatic transformations to the images. In particular, the images are saved in RGB, that is a 3-channel format. This means that for each value in the matrix, we will have 3 color values. We don't know how to handle that, so we will be converting them into grey scale on a single channel. Then, because our original data ranges from 0 to 255, and the grey scale ranges from 0 to 1, we need to multiply our converted image by 255.\n","In this case we could have also scaled our original data to fit the rango 0 to 1, but that was not the case. Do you think that would have been an improvement to the network's training?\n","Finally because our data is a 28x28 matrix, we reshape it to a single 784 vector, with is the reshape(-1) instruction."]},{"cell_type":"code","metadata":{"id":"maQl7hjRFfGw","colab_type":"code","colab":{}},"source":["# Load the dataset based on a folder structure\n","user_made = torchvision.datasets.ImageFolder(\n","        root=\"temp/\",\n","        transform=torchvision.transforms.Compose([\n","            torchvision.transforms.Grayscale(num_output_channels=1),\n","            torchvision.transforms.ToTensor(),\n","            torchvision.transforms.Lambda(lambda x: x * 255),\n","            torchvision.transforms.Lambda(lambda x: x.reshape(-1))\n","        ]) \n","    )\n","# Ignore this. Just some limitations on Pytorchs end...\n","user_made.target_transform = lambda _id: int(list(user_made.class_to_idx.keys())[list(user_made.class_to_idx.values()).index(_id)])\n","\n","# Create a loader to speed and simplify the process\n","user_data = DataLoader(user_made, batch_size=len(user_made.samples), shuffle=False)\n","# Just get all the data, but in the format we expect\n","user_x, user_y = next(iter(user_data))\n","\n","# make predictions on our numbers!\n","_, y_user_pred = torch.max(model(user_x), dim=1)"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"b-aegLzqK_Xp","colab_type":"text"},"source":["Let's check if the network is capable of correctly identify the numbers we draw!"]},{"cell_type":"code","metadata":{"id":"bK-wxN2KHHnG","colab_type":"code","colab":{}},"source":["n = len(y_user_pred)\n","fig = plt.figure(figsize=(25, 2 * math.ceil(n/10)))\n","for i, user_tensor in enumerate(user_x):\n","    ax = fig.add_subplot(math.ceil(n/10), n/math.ceil(n/10), i+1, xticks=[], yticks=[])\n","    ax.imshow(user_tensor.reshape(28, 28), cmap='gray', interpolation='none')\n","    ax.set_title(f\"pred: {y_user_pred[i]} (real: {user_y[i]})\", \n","                 color=(\"green\" if y_user_pred[i]==user_y[i] else \"red\"))"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"B6HPTqk6LFQw","colab_type":"text"},"source":["----"]},{"cell_type":"markdown","metadata":{"id":"Ltd7El1nLDFK","colab_type":"text"},"source":["That was for the day 1 of this workshop. If you want more details on what we have seen here are some resources:\n","* Pytorch documentation: https://pytorch.org/docs/stable/index.html\n","* An introduction to Deep Learning with Pytorch: https://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html\n","* A tutorial on how autograd works: https://towardsdatascience.com/pytorch-autograd-understanding-the-heart-of-pytorchs-magic-2686cd94ec95\n","* More details on backpropagation and gradient calculus: http://cs231n.stanford.edu/handouts/derivatives.pdf\n","* More on matrix differentiation: https://www.comp.nus.edu.sg/~cs5240/lecture/matrix-differentiation.pdf\n","* The matrix calculus you neef for deep learning: https://explained.ai/matrix-calculus/index.html"]},{"cell_type":"code","metadata":{"id":"TTq-LBrxOUX7","colab_type":"code","colab":{}},"source":[""],"execution_count":0,"outputs":[]}]}