Paper Collection of Neural Text Generation (NTG)

Neural Text Generation refers to a kind of methods that mainly attempt to use NN as function approximators to mimic the underlying distribution of (natural) languages. The most important applications of the conditional version of this topic include Neural Machine Translation (NMT), neural image captioning and dialogue system (chatbot). However, the researches of NTG usually refer to those focus on the unconditional problem, that is to really learn the latent distribution of the target language (instead of a transformation mapping from source form to target form).

This repository presents a collection of previous research papers of Neural Text Generation (NTG), as well as a taxonomy constructed according to publication time, method paradigm or paper type.

Taxonomy of Papers

Survey and Theoretical Analysis

• Neural Text Generation: Past, Present and Beyond
• How (not) to Train your Generative Model: Scheduled Sampling, Likelihood, Adversary?

Metrics, Toolbox and Dataset

• BLEU: a method for automatic evaluation of machine translation
• METEOR: An automatic metric for MT evaluation with improved correlation with human judgments
• Perplexity—a measure of the difficulty of speech recognition tasks
• NLL_oracle, NLL_test, SelfBLEU, Texygen Texygen: A Benchmarking Platform for Text Generation Models

Online-available Course

• Introduction to Reinforcement Learning by David Silver

Research Paper

• Architechture
  • NNLM A neural probabilistic language model
  • RNNLM Recurrent neural network based language model
  • LSTM Long short-term memory
  • GRU Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
  • SRU Training RNNs as Fast as CNNs
  • Hierarchical Softmax Classes for fast maximum entropy training
  • Feudal-like Language Model Long Text Generation via Adversarial Training with Leaked Information
• Training Algorithm / Models
  • Likelihood Based
    • Maximum Likelihood Estimation / Teacher Forcing A learning algorithm for continually running fully recurrent neural networks
    • Scheduled Sampling Scheduled sampling for sequence prediction with recurrent neural networks
  • Static-target Reinforcement Learning
    • Policy Gradient Policy gradient methods for reinforcement learning with function approximation
      • PG-BLEU: Use Policy Gradient to optimize BLEU.
  • Adversarial Methods
    • Adversary as a Regularization
      • Professor Forcing Professor forcing: A new algorithm for training recurrent networks
    • Direct
      • SeqGAN SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient
      • MaliGAN Maximum-likelihood augmented discrete generative adversarial networks
      • RankGAN Adversarial ranking for language generation
      • ScratchGAN - Big Version of SeqGAN gets rid of pre-training! Training Language GANs from Scratch
      • RelGAN RelGAN: Relational Generative Adversarial Networks for Text Generation
    • Adversarial Feature Matching
      • TextGAN Adversarial feature matching for text generation
    • Denoise Sequence-to-sequence Learning
      • MaskGAN MaskGAN: Better Text Generation via Filling in the _
    • Reparametrized Sampling
      • GSGAN Gans for sequences of discrete elements with the gumbel-softmax distribution
    • Learning to Exploit Leaked Information
      • LeakGAN Long Text Generation via Adversarial Training with Leaked Information
    • Smoothing-N-Rediscretization
      • WGAN-GP, GAN-GP Adversarial generation of natural language
  • Cooperative Methods
    • CoT CoT: Cooperative Training for Generative Modeling of Discrete Data