05 Model Architectures.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Model Architectures: Part 1\n",
    "\n",
    "Neon supports the ability to build more complex models than just a linear list of layers. In this series of notebooks, you will implement several models and understand how data should be passed when a model may have multiple inputs/outputs.\n",
    "\n",
    "## Tree Models\n",
    "\n",
    "Neon supports models with a main trunk that includes branch points to leaf nodes. In this scenario, the models takes a single input but produces multiple outputs that can be matched against multiple targets. For example, consider the below topology:\n",
    "\n",
    "```\n",
    "cost1      cost3\n",
    "  |          /\n",
    " m_l4      b2_l2\n",
    "  |        /\n",
    "  | ___b2_l1\n",
    "  |/\n",
    " m_l3       cost2\n",
    "  |          /\n",
    " m_l2      b1_l2\n",
    "  |        /\n",
    "  | ___b1_l1\n",
    "  |/\n",
    "  |\n",
    " m_l1\n",
    "  |\n",
    "  |\n",
    " data\n",
    " ```\n",
    " \n",
    "Suppose we wanted to apply this model to the MNIST dataset. The MNIST data iterator returns, for each minibatch, a tuple of tensors `(X, Y)`. Since there are multiple outputs, the single target labels `Y` are used to match against all these outputs. Alternatively, we could write a custom iterator that yields for each minibatch, a nested tuple `(X, (Y1, Y2, Y3))`. Then, each target label will mapped to its respective output layer. \n",
    " \n",
    "We will guide you through implementing such a branching model. We first import all the needed ingredients:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from neon.callbacks.callbacks import Callbacks\n",
    "from neon.initializers import Gaussian\n",
    "from neon.layers import GeneralizedCost, Affine, Multicost, SingleOutputTree\n",
    "from neon.models import Model\n",
    "from neon.optimizers import GradientDescentMomentum\n",
    "from neon.transforms import Rectlin, Logistic, Softmax\n",
    "from neon.transforms import CrossEntropyBinary, CrossEntropyMulti, Misclassification\n",
    "from neon.backends import gen_backend"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also set up the backend and load the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "be = gen_backend(batch_size=128)\n",
    "\n",
    "from neon.data import MNIST\n",
    "\n",
    "mnist = MNIST(path='data/')\n",
    "train_set = mnist.train_iter\n",
    "valid_set = mnist.valid_iter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now its your turn! Set up the branch nodes and layer structure above. Some tips:\n",
    "- Use `Affine` layers. \n",
    "- You can choose your hidden unit sizes, just make sure that the three final output layers have 10 units for the 10 categories in the MNIST dataset. \n",
    "- The three final output layers should also use Softmax activation functions to ensure that the probability sums to 1. \n",
    "\n",
    "As a reminder, to define a single layer, we need a weight initialization and an activation function:\n",
    "\n",
    "```\n",
    "\n",
    "# define a layers\n",
    "layer1 = Affine(nout=100, init=Gaussian(0.01), activation=Rectlin())\n",
    "\n",
    "# alternative, you can take advantage of common parameters by constructing\n",
    "# a dictionary:\n",
    "normrelu = dict(init=init_norm, activation=Rectlin())\n",
    "\n",
    "# pass the dictionary to the layers as keyword arguments using the ** syntax.\n",
    "layer1 = Affine(nout=100, **normrelu)\n",
    "layer2 = Affine(nout=10, **normrelu)\n",
    "\n",
    "```\n",
    "\n",
    "To set up a simple Tree:\n",
    "\n",
    "```\n",
    "# define a branch node\n",
    "b1 = BranchNode(name=\"b1\")\n",
    "\n",
    "# define the main trunk\n",
    "path1 = [layer1, b1, layer2]\n",
    "\n",
    "# define the branch\n",
    "path2 = [b1, layer3]\n",
    "\n",
    "# build the model as a Tree\n",
    "# alphas are the weights given to the branches of Tree during backpropagation.\n",
    "model = Model(layers=SingleOutputTree([path1, path2], alphas = [1, 1]))\n",
    "```\n",
    "\n",
    "We have included below skeleton of the code for you to fill out to build the model above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from neon.layers import BranchNode\n",
    "BranchNode.instances = dict()\n",
    "\n",
    "# define common parameters as dictionary (see above)\n",
    "init_norm = Gaussian(loc=0.0, scale=0.01)\n",
    "\n",
    "normrelu = dict(init=init_norm, activation=Rectlin())\n",
    "normsigm = dict(init=init_norm, activation=Logistic(shortcut=True))\n",
    "normsoft = dict(init=init_norm, activation=Softmax())\n",
    "\n",
    "# define your branch nodes\n",
    "# branch nodes need to have a unique name\n",
    "b1 = BranchNode(name=\"b1\")\n",
    "b2 = BranchNode(name=\"b2\")\n",
    "\n",
    "# define the main trunk (cost1 above)\n",
    "..\n",
    "\n",
    "# define the branch (cost2)\n",
    "...\n",
    "\n",
    "# define the branch (cost3)\n",
    "...\n",
    "\n",
    "# build the model as a SingleOutputTree\n",
    "..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's fit our model! First, set up multiple costs for each of the three branches using `MultiCost`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cost = Multicost(costs=[GeneralizedCost(costfunc=CrossEntropyMulti()),\n",
    "                        GeneralizedCost(costfunc=CrossEntropyMulti()),\n",
    "                        GeneralizedCost(costfunc=CrossEntropyMulti())])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To test that your model was constructed properly, we first initialize the model with a dataset (so that it configures the layer shapes appropriately) and a cost, then print the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "model.initialize(train_set, cost)\n",
    "print model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, we set up the remaining components and run fit!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# setup optimizer\n",
    "optimizer = GradientDescentMomentum(0.1, momentum_coef=0.9)\n",
    "\n",
    "# setup standard fit callbacks\n",
    "callbacks = Callbacks(model, eval_set=valid_set, eval_freq=1)\n",
    "model.fit(train_set, optimizer=optimizer, num_epochs=20, cost=cost, callbacks=callbacks)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Try to adjust the model architecture and hyperparameters to reach a CrossEntropyLoss of 0.16 or below. When re-running the model, we suggest restarting the ipython kernel: `[Kernel -> Restart & Run All]`"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}