13_other_work.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<center>\n",
    "<h2>Practical advice</h2>\n",
    "<h3>Making sure everything actually works!</h3>\n",
    "</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### How to apply Imitation Learning?###\n",
    "\n",
    "For any task, define:\n",
    "- Transition\n",
    "- Loss function\n",
    "- Expert policy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Is the expert policy working as intended? ###\n",
    "\n",
    "Perform full rollins with $\\pi^{\\star}$.<ul><li>Resulting sequences should have $L(S_{final}, \\mathbf{y}) = 0$.</li><li class=\"fragment\" data-fragment-index=\"3\">Or close to 0, if happy with suboptimal $\\pi^{\\star}$.</li></ul>\n",
    "\n",
    "<center>\n",
    "<img src=\"images/stateTransitExpert.png\">\n",
    "</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "If expert policy takes too long to calculate during rollout:\n",
    "- Consider suboptimal alternatives!\n",
    "- More IL iterations could be more helpful."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Are the roll-outs working as intended? ###\n",
    "\n",
    "Ensure costs obtained through rollouts are sensible.\n",
    "- Actions returned by optimal $\\pi^{\\star}$ should have low cost.\n",
    "- Equally optimal actions should be costed closely.\n",
    "\n",
    "<center>\n",
    "<img src=\"images/determReachStatesScored.png\" width=\"60%\">\n",
    "</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Is Imitation Learning working? ###\n",
    "\n",
    "Examine time-steps before and after application of IL.\n",
    "\n",
    "<center>\n",
    "<img src=\"images/toBeAnimated/NLG_LOLS.png\" width=\"80%\">\n",
    "</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "If loss rises:\n",
    "- Adjust transition system.\n",
    "- Make sure features (latent representations) describe the state and actions adequately."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Avoid cycles ###\n",
    "\n",
    "Need to prevent cycles between state transitions!\n",
    "<br>\n",
    "<br>\n",
    "<span style=\"font-variant: small-caps;\">... -> Swap($e_i$, $e_j$) -> Swap($e_j$, $e_i$) -> ...</span></ul>\n",
    "\n",
    "Cycles can be controlled in either:\n",
    "- Transition system, or\n",
    "- loss function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### Are the actions learnable? ###\n",
    "\n",
    "Make sure that the features (latent representations) can describe them adequately."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Include features that consider the previous actions.\n",
    "- To avoid repeating or undoing previous actions.\n",
    "- To learn action chains."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Identify problems in your task ###\n",
    "\n",
    "Suboptimal expert policy?\n",
    "- Use rollouts that mix expert and classifier."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Errors in rollin introducing noise to the feature vectors?\n",
    "- Try sequence correction."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Variance or irrelevant errors in rollouts affect costing?\n",
    "- Try focused costing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Too many actions to rollout?\n",
    "- Try partial or targeted exploration."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Irrelevant errors in rollouts affect costing?\n",
    "- Try focused costing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Noisy training instances?\n",
    "- Try $a$-bound noise reduction."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### Work using imitation learning in EACL 2017 ###\n",
    "\n",
    "<b>Tackling Error Propagation through Reinforcement Learning:\n",
    "A Case of Greedy Dependency Parsing</b>\n",
    "<br>\n",
    "Minh Lê and Antske Fokkens\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Summary ###\n",
    "\n",
    "Discussed expert policy definitions:\n",
    "<ul><li>Static vs. dynamic vs. suboptimal policies.</li></ul>\n",
    "<br><br>\n",
    "<span class=\"fragment\" data-fragment-index=\"2\">Examined variations to exploration:</span>\n",
    "<ul><li class=\"fragment\" data-fragment-index=\"2\">Targeted and partial exploration.</li></ul>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Showed how to reduce noise in the transition sequence.\n",
    "<ul><li>Sequence correction and focused costing.</li></ul><br>\n",
    "<span class=\"fragment\" data-fragment-index=\"1\">And how to reduce noise for the classifier.</span>\n",
    "<ul><li class=\"fragment\" data-fragment-index=\"1\">$\\alpha$-bound.</li></ul>\n",
    "<br><br>\n",
    "<span class=\"fragment\" data-fragment-index=\"2\">Applying Imitation Learning on various NLP tasks.</span>\n",
    "<ul><li class=\"fragment\" data-fragment-index=\"2\">Transitions, loss functions, expert policies.</li>\n",
    "<li class=\"fragment\" data-fragment-index=\"3\">And improved on results!</li></ul>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<center>\n",
    "<h2>Thank you!</h2>\n",
    "<p style=\"text-align:center\"><small><a href=\"http://sheffieldnlp.github.io/ImitationLearningTutorialEACL2017/\">sheffieldnlp.github.io/ImitationLearningTutorialEACL2017/</a></small></p>\n",
    "</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Available code ### \n",
    "\n",
    "<b>Imitation Learning</b><br>\n",
    "Vowpal Wabbit Credit assignment compiler ([in Python](http://hunch.net/~vw))<br>\n",
    "V-DAgger ([in Scala](https://github.com/hopshackle/dagger-AMR)) ([in Python](https://github.com/sheffieldnlp/APEimitaion))<br> \n",
    "LOLS ([in Java](https://github.com/glampouras/JLOLS_NLG))\n",
    "<br> \n",
    "MIXER ([in Lua](https://research.fb.com/downloads/mixer/))\n",
    "<br><br>\n",
    "<b>Applications</b><br>\n",
    "Dependency parsing ([in Python](https://bitbucket.org/yoavgo/tacl2013dynamicoracles))<br>\n",
    "Natural language generation ([in Java](https://github.com/glampouras/JLOLS_NLG))<br>\n",
    "Semantic parsing for AMR ([in Scala](https://github.com/hopshackle/dagger-AMR))\n",
    "<br><br>\n",
    "<b>Cost-sensitive multiclass classification</b><br>\n",
    "Vowpal Wabbit cost-sensitive classifier ([in Python](http://hunch.net/~vw))<br>\n",
    "Adaptive Regularization of Weights ([in Python](https://github.com/andreasvlachos/arow_csc)) ([in Java](https://github.com/glampouras/JLOLS_NLG))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Annotated bibliography \n",
    "\n",
    "#### PhD theses\n",
    "\n",
    "[Hal Daumé III, 2006](http://www.umiacs.umd.edu/~hal/docs/daume06thesis.pdf): Practical Structured Learning Techniques for Natural Language Processing\n",
    "\n",
    "[Stéphane Ross, 2013](http://www.cs.cmu.edu/~sross1/publications/ross_phdthesis.pdf): Interactive Learning for Sequential Decisions and Predictions\n",
    "\n",
    "[Pieter Abbeel, 2008](http://www.cs.stanford.edu/~pabbeel/thesis/thesis.pdf): Apprenticeship Learning and Reinforcement Learning with Application to Robotic Control"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Papers\n",
    "\n",
    "[Abeel and Ng, 2004](http://ai.stanford.edu/~ang/papers/icml04-apprentice.pdf): Apprenticeship Learning via Inverse Reinforcement Learning *(inverse reinforcement learning)*\n",
    "\n",
    "[Viera and Eisner, 2016](http://timvieira.github.io/doc/2016-tacl-pruning.pdf): Learning to Prune: Exploring the Frontier of Fast and Accurate Parsing *(LOLS with random expert=RL, changeprop)*\n",
    "\n",
    "[Goldberg and Nivre, 2012](http://www.aclweb.org/anthology/C12-1059): A Dynamic Oracle for Arc-Eager Dependency Parsing *(DAgger for dependency parsing)*\n",
    "\n",
    "[Ballesteros et al., 2016](https://arxiv.org/pdf/1603.03793.pdf): Training with Exploration Improves a Greedy Stack LSTM Parser *(DAgger for LSTM-based dependency parsing)*\n",
    "\n",
    "[Clark and Manning, 2015](http://cs.stanford.edu/people/kevclark/resources/clark-manning-acl15-entity.pdf): Entity-Centric Coreference Resolution with Model Stacking"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Papers\n",
    "\n",
    "[Lampouras and Vlachos 2016](https://aclweb.org/anthology/C/C16/C16-1105.pdf): Imitation learning for language generation from unaligned data *(LoLS for natural language generation, sequence correction)*.\n",
    "\n",
    "[Goodman et al. 2016](http://aclweb.org/anthology/P16-1001): Noise reduction and targeted exploration in imitation learning for\n",
    "Abstract Meaning Representation parsing *(V-DAgger for semantic parsing, targeted exploration)*.\n",
    "\n",
    "[Vlachos and Craven, 2011](http://www.aclweb.org/anthology/W/W11/W11-0307.pdf): Search-based Structured Prediction applied to Biomedical Event Extraction *(SEARN for biomedical event extraction, focused costing)*\n",
    "\n",
    "[Berant and Liang, 2015](https://www.transacl.org/ojs/index.php/tacl/article/view/646/160): Imitation Learning of Agenda-based Semantic Parsers\n",
    "\n",
    "[Ranzato et al., 2016](https://arxiv.org/pdf/1511.06732.pdf): Sequence Level Training with Recurrent Neural Networks *(Imitation learning for RNNs, learns a cost estimator instead of using roll-outs)*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Papers\n",
    "\n",
    "[Daumé III et al., 2009](http://hunch.net/~jl/projects/reductions/searn/searn.pdf): Search-based structured prediction *(SEARN algorithm)*\n",
    "\n",
    "[Daumé III, 2009](http://www.umiacs.umd.edu/~hal/docs/daume09unsearn.pdf): Unsupervised Search-based Structured Prediction *(Unsupervised structured prediction with SEARN)*\n",
    "\n",
    "[Chang et al., 2015](https://arxiv.org/pdf/1502.02206.pdf): Learning to search better than your teacher *(LoLS algorithm, connection with RL)*\n",
    "\n",
    "[Ho and Ermon, 2016](https://arxiv.org/abs/1606.03476): Generative Adversarial Imitation Learning *(Connection with adversarial training)*\n",
    "\n",
    "[He et al., 2012](https://papers.nips.cc/paper/4545-imitation-learning-by-coaching.pdf): Imitation Learning by Coaching *(coaching for DAgger)*"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  },
  "livereveal": {
   "height": 768,
   "start_slideshow_at": "selected",
   "theme": "solarized",
   "transition": "slide",
   "width": 1024
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}