Chapter9_Conclusions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<< [第八章：生成模型深度学习](Chapter8_Generative_deep_learning.ipynb) || [目录](index.md)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 第九章：总结\n",
    "\n",
    "> You have almost reached the end of this book. In this last chapter, we focus on\n",
    "summarizing and reviewing core concepts, while expanding your horizons beyond the\n",
    "relatively basic notions you have learned so far. Understanding deep learning and AI is a\n",
    "journey, and finishing this book is merely the first step of it. I want to make sure that you\n",
    "realize this, and that you are properly equipped for taking the next steps of this journey\n",
    "on your own.\n",
    "\n",
    "你已经基本上到达本书的结尾了。在最后这个章节中，我们专注于总结和回顾核心概念，同时在你已经掌握相对基础的内容上扩展视野。理解深度学习和AI是一个旅程，完成并理解本书仅仅是旅程的第一步。作者希望能够让你确定这点，并且本书能够为你装备好行李，使你有能力展开你自己后续的旅程。\n",
    "\n",
    "> This chapter is structured in four sections:\n",
    "\n",
    "> - A bird’s eye view of what you should take away from this book. This should refresh your\n",
    "memory on some of the concepts you have previously learned.\n",
    "- An overview of some key limitations of deep learning. To use a tool appropriately, you\n",
    "should not only understand what it can do, but also be aware of what it won’t do.\n",
    "- Speculative thoughts about the future evolution of the fields of deep learning, machine\n",
    "learning and AI. This should be especially interesting to you if you would like to get into\n",
    "fundamental research.\n",
    "- A short list of resources and strategies for learning further about AI and staying up to\n",
    "date with new advances.\n",
    "\n",
    "本章分为四个小节：\n",
    "\n",
    "- 鸟瞰整本书你需要打包到脑袋的内容。本节能刷新你的记忆，让你回顾之前学过的那些重要概念。\n",
    "- 深度学习的一些关键局限性的综述。要正确使用工具的话，你不仅需要理解它能做什么，还需要知道它不能做什么。\n",
    "- 深度学习、机器学习和AI领域未来的一些推测性的想法。如果你希望进入这个领域的基础性研究方向的话，这应该很合胃口。\n",
    "- 一些资源和策略，帮助你能继续在AI深入学习及保持与时俱进。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 9.1 回顾核心概念\n",
    "\n",
    "> This sections aims at briefly synthesizing the key take-aways from this book. If you ever\n",
    "need a quick refresher to help you recall what you’ve learned in this book, you can just\n",
    "read these few pages.\n",
    "\n",
    "本小节主要为了帮助你综合回顾本书前面章节所有需要理解的内容。如果以后你需要一个快速对整本书的复习，你可以只读下面这几页。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 9.1.1 AI各种不同的实现方式\n",
    "\n",
    "> First of all, deep learning is not synonymous with AI, nor even with machine learning. AI\n",
    "is an ancient and very broad field that can generally be defined as \"all attempts to\n",
    "automate cognitive processes\". The automation of thought. This can range from the very\n",
    "basic, like an Excel spreadsheet, to the very advanced, like a humanoid robot able to walk\n",
    "and talk.\n",
    "\n",
    "首先深度学习不是AI的同义词，甚至不是机器学习的同义词。AI是一个古远而非常广泛的领域，可以被泛化的定义为“一切自动认知过程的尝试”。也就是思维的自动化。这个领域范围非常广泛，可以从非常基础的Excel表格，到非常高级的能够行走和交谈的类人型机器人。\n",
    "\n",
    "> Machine learning is a specific subfield of AI that aims at automatically developing\n",
    "programs (called \"models\") purely from exposure to training data. This process of turning\n",
    "data into a program is called \"learning\". Albeit machine learning has been around for a\n",
    "long time, it only started taking off in the 1990s.\n",
    "\n",
    "机器学习是AI中一个特定的子领域，目标是能够单纯从训练数据中获得自动发展得到的程序（称为“模型”）。将数据转换成程序的过程叫做“学习”。尽管机器学习已经存在了很长一段时间，但是它直到1990年代才开始真正获得发展。\n",
    "\n",
    "> Deep learning is one of many branches of machine learning, where the models are\n",
    "long chains of geometric functions, chained one after the other. These operations are\n",
    "structured into modules called \"layers\": deep learning models are typically just stacks of\n",
    "layers, or more generally graphs of layers. These layers are parametrized by \"weights\",\n",
    "which are the parameters that are learned during training. The \"knowledge\" of a model is\n",
    "stored in its weights, and process of \"learning\" consists in finding good values for these\n",
    "weights.\n",
    "\n",
    "深度学习是机器学习中的一个分支，它其中的模型是一长串的几何函数的组合，一个接着另一个。这些操作被封装到被称为“层”的模块当中：深度学习模型最基础的结构就是层次的堆叠，也可能是层次组成的图。这些层次具有“权重”参数，这些参数就是训练过程中进行学习的内容。模型的“知识”存储在它的权重当中，“学习”的过程其实就是找到这些权重最合适的值的过程。\n",
    "\n",
    "> Even though deep learning is just one approach to machine learning among many, it\n",
    "isn’t on an equal footing with the others. Deep learning is a breakout success. Here’s\n",
    "why.\n",
    "\n",
    "虽然深度学习只是机器学习的一个分支，但是它目前具有不同于其他机器学习方法的地位。深度学习目前获得了突破性的成功。下面来说说原因。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 9.1.2 深度学习在机器学习中特别的原因\n",
    "\n",
    "> In the span of just a few years, deep learning has achieved tremendous breakthroughs\n",
    "across a wide range of tasks that have been historically perceived as extremely difficult\n",
    "for computers, especially in machine perception: extracting useful information from\n",
    "images, videos, sound, and more. Given sufficient training data (in particular, training\n",
    "data appropriately labeled by humans), it is possible to extract from perceptual data\n",
    "almost anything that a human could extract. Hence it is sometimes said that deep learning\n",
    "has \"solved perception\", albeit that is only true for a fairly narrow definition of\n",
    "\"perception\".\n",
    "\n",
    "过去的几年中，深度学习在广泛的任务领域中获得极大的突破，很多曾经被认为是对于计算机极端困难的问题都获得了解决，特别是在机器感知当中：从图像、视频和声音等数据中提取有用的信息。只要提供足够的训练数据（确切的说，被人类正确标记过的数据），它几乎能完成任何提取感知数据的工作，就像人类一样。因此有一种说法是深度学习已经“解决了感知”问题，尽管这只是“感知”定义的一小部分。\n",
    "\n",
    "> Due to its unprecedented technical successes, deep learning has single-handedly\n",
    "brought about the third and by far the largest \"AI summer\", a period of intense interest,\n",
    "investment, and hype, in the field of AI. As this book is being written, we are in the\n",
    "middle of it. Whether this period will end in the near future, and what happens after it\n",
    "ends, is a topic of debate. One thing is certain: in stark contrast with previous AI\n",
    "summers, deep learning has been providing enormous business value to a number of\n",
    "large technology companies, enabling human-level speech recognition, smart assistants,\n",
    "human-level image classification, vastly improved machine translation, and more. The\n",
    "hype may (and likely will) recess, but the sustained economic and technological impact\n",
    "of deep learning will remain. In that sense, deep learning could be analogous to the\n",
    "Internet: it may be overly hyped up for a few years, but in the longer term it will still be a\n",
    "major revolution that will transform our economy and our lives.\n",
    "\n",
    "基于深度学习获得了空前的技术成功，它凭借一己之力就带来了第三次也是迄今为止最大的一次“AI热潮”，也就是在AI领域中有一段时期能够获得大量的关注和投资，当然也包括炒作。当本书写作之时，我们正处于这个热潮之中。这个热潮是否会结束，结束之后会发生什么，这仍然是一个有争议的话题。有一件事情是确定的：与之前的AI热潮鲜明对比的是，深度学习已经给很多大型科技公司带来了商业价值，成为语音识别、智能助手、图像分类、巨大改进的机器翻译等应用的关键支撑技术。这其中的炒作也许（非常可能）会停歇，但是深度学习带来的经济和科技冲击会延续。在这层含义上，深度学习就像互联网一样：它可能会被过度炒作一段时间，但是在长期上看，它将成为改变我们经济和生活的革命性技术。\n",
    "\n",
    "> I am particularly optimistic about deep learning, because, even if we were to make no\n",
    "further technological progress in the next decade, simply deploying existing algorithms to\n",
    "every problem where they are applicable would already be a game-changer for most\n",
    "industries. Deep learning is nothing short of a revolution. And as it happens, progress is\n",
    "currently happening at an incredibly fast rate—due to an exponential investment in\n",
    "resources and headcount. From where I stand, the future looks bright. It does seem,\n",
    "however, that short-term expectations are somewhat over-optimistic. Deploying deep\n",
    "learning to the full extent of its potential will take well over a decade.\n",
    "\n",
    "作者对深度学习特别乐观，因为即使我们下一个十年无法在技术上获得进展，只有将现有算法应用到所有可以应用深度学习的工业领域就足以改变世界。深度学习无疑就是一场技术革命。并且当热潮到来时，整个趋势发展的速度令人惊异，因为其中对资源和人力的指数性投入。因此在作者看来，未来是光明的。虽然某种程度上短期内可能存在过分乐观的预期，但是将深度学习应用到所有潜在的领域会是下一个十年的必然趋势。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 8.1.3 如何看待深度学习技术\n",
    "\n",
    "> The most surprising thing about deep learning is how simple it is. Ten years ago, no one\n",
    "expected that we would achieve such amazing results on machine perception problems by\n",
    "using simple parametric models trained with gradient descent. Now, it turns out that all\n",
    "you need is sufficiently large parametric models trained with gradient descent on\n",
    "sufficiently many examples. As Feynman once said about the universe, \"It’s not\n",
    "complicated, it’s just a lot of it\" .\n",
    "\n",
    "关于深度学习最令人惊奇的一点是它非常简单。十年前，没有人能够预期我们能够使用简单的参数化模型经过梯度下降训练之后，在机器感知上获得如此了不起的成就。现在人们发现，所需要的只是足够大的参数化模型，然后在上面用足够量的数据进行梯度下降训练。正如当年费曼评价宇宙：“宇宙不是太复杂了，它只是太多内容了”。\n",
    "\n",
    "> In deep learning, everything is a vector, i.e. everything is a point in a geometric space\n",
    ". Model inputs (it could be text, images, etc) and targets are first \"vectorized\", i.e. turned\n",
    "into some initial input vector space and target vector space. Each layer in a deep learning\n",
    "model operates one simple geometric transformation on the data that goes through it.\n",
    "Together, the chain of layers of the model forms one very complex geometric\n",
    "transformation, broken down into a series of simple ones. This complex transformation\n",
    "attempts to maps the input space to the target space, one point at a time. This\n",
    "transformation is parametrized by the weights of the layers, which are iteratively updated\n",
    "based on how well the model is currently performing. A key characteristic of this\n",
    "geometric transformation is that it must be differentiable , which is required in order for\n",
    "us to be able to learn its parameters via gradient descent. Intuitively, this means that the\n",
    "geometric morphing from inputs to outputs must be smooth and continuous—a\n",
    "significant constraint.\n",
    "\n",
    "在深度学习中，所有东西都是一个向量，也就是说，任何事物都是几何空间中的一个点。模型输入（可能是文本、图像等）和目标首先是“向量化”的，或者说将输入和目标转换到它们的向量空间中。深度学习中每个层次知识对输入的数据进行简单的几何运算。将这些串联的层次合起来就能获得非常复杂的几何运算，我们将复杂的打散成一系列简单的结构。这个复杂的转换试图将输入空间映射到目标空间，一次一个点。整个转换操作都是有层次上面的参数化权重值控制的，这些权重值会根据模型目前的性能情况不断进行更新。这里面几何转换运算的关键一点在于它必须是可微的，这样我们才能使用梯度下降来学习它的参数。更直观的说这意味着这种从输入到输出的几何运算必须是平滑和连续的，这是一个重要的限制。\n",
    "\n",
    "> The whole process of applying this complex geometric transformation to the input\n",
    "data can be visualized in 3D by imagining a person trying to a paper ball: the\n",
    "crumpled paper ball is the manifold of the input data that the model starts with. Each\n",
    "movement operated by the person on the paper ball is similar to a simple geometric\n",
    "transformation operated by one layer. The full uncrumpling gesture sequence is the\n",
    "complex transformation of the entire model. Deep learning models are mathematical\n",
    "machines for uncrumpling complicated manifolds of high-dimensional data.\n",
    "\n",
    "上面这个复杂的几何转换完整的过程可以在一个3D场景中进行想象，某个人在展开手中的纸团：揉成一团的纸团就是输入数据的流形，也是模型接受的输入数据。那个人每次尝试展开纸团的操作就像一个层次对数据进行的简单几何转换。展开纸团的所有过程是一系列的操作，正如整个模型最终合成的复杂几何转换。深度学习模型就是这样一种将高维度数据流形进行展开的数学机器。\n",
    "\n",
    "> That’s the magic of deep learning: turning meaning into vectors, into geometric\n",
    "spaces, then incrementally learning complex geometric transformations that map one\n",
    "space to another. All you need are spaces of sufficiently high dimensionality in order to\n",
    "capture the full scope of the relationships found in the original data.\n",
    "\n",
    "这就是深度学习的全部奥秘：将内涵转为向量，转为几何空间，然后渐进式的学习从一个空间到另一个空间的复杂集合转换过程。在其中你需要的知识一个足够高维度的空间能够捕捉原始数据中的全部可能的关系。\n",
    "\n",
    "> The whole thing hinges on one single core idea: that meaning is derived from the\n",
    "pairwise relationship between things (between words in a language, between pixels in an\n",
    "image, etc) and that these relationships can be captured by a distance function . But do\n",
    "note that whether or not the brain implements meaning via geometric spaces is an entirely\n",
    "separate question. Vector spaces are very efficient to work with from a computational\n",
    "standpoint, but different data structures for intelligence can easily be envisioned, in\n",
    "particular graphs. In fact, \"neural networks\" initially emerged from the very idea of using\n",
    "graphs as a way to encode meaning, which is why they are named \"neural networks\", and\n",
    "the surrounding field of research used to be called \"connectionism\". Nowadays the name\n",
    "\"neural network\" is still around purely for historical reasons—it is an extremely\n",
    "misleading name since they are in fact neither neural nor networks. In particular, they\n",
    "have hardly anything to do with the brain. A more appropriate name could have been\n",
    "\"layered representations learning\" or \"hierarchical representations learning\". Or maybe\n",
    "even \"deep differentiable models\" or \"chained geometric transforms\", in order to\n",
    "emphasize the fact that continuous geometric space manipulation is at their core.\n",
    "\n",
    "这里的原理取决于一个核心概念：也就是内涵是从事物之间的相互关系上获得的（语言中的词与词之间，图像中的像素之间等等），并且这些关系可以使用距离函数进行描述。不过要牢记我们的大脑是否通过几何距离来实现认知是完全不同的一个问题。在计算角度来看向量空间是非常高效的，但是也很容易设想到其他的数据结构也能产生智能，比方说图。事实上“神经网络”最早的时候就是希望使用图结构来编码意识，这也是它为什么叫做神经网络的原因，并且相关领域的研究曾经被叫做“联接学”。时至今日我们仍然使用神经网络这个名称仅仅是因为历史原因，因为目前的深度学习既不是神经也不是网络，这个名词具有一定的误导性。特别是它与我们的大脑基本没有任何联系。一个更加合适的名字可能会是“层表现学习”或者“层次化表现学习”，或者甚至可以叫“深度微分模型”和“串联几何转换”，用来准确描述这个技术的核心是连续的几何空间操作。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 8.1.4 技术突破的关键因素\n",
    "\n",
    "> The technological revolution that is currently unfolding did not start with any single\n",
    "breakthrough invention. Rather, like any other revolution, it is the product of a vast\n",
    "accumulation of enabling factors—slow at first, then all of a sudden. In the case of deep\n",
    "learning, we can point out the following key factors:\n",
    "\n",
    "> - Incremental algorithmic innovations, first spread over two decades (starting with\n",
    "backpropagation), then happening increasingly faster as more research effort was poured\n",
    "into deep learning after 2012.\n",
    "- The availability of large amounts of perceptual data, a requirement in order to realize that\n",
    "\"sufficiently large models trained on sufficiently large data are all you need\". This is in\n",
    "turn a by-product of the rise of the consumer Internet and Moore’s law applied to storage\n",
    "media.\n",
    "- The availability of fast, highly parallel computation hardware at a low price, especially\n",
    "the GPUs produced by NVIDIA—first gaming GPUs, then chips designed from the\n",
    "ground up for deep learning, as CEO Jensen Huang took note early on of the deep\n",
    "learning boom and decided to bet the company’s future on it.\n",
    "- A complex stack of software layers that makes this computational power available to\n",
    "humans: the CUDA language, frameworks like TensorFlow that do automatic\n",
    "differentiation, and finally Keras that makes deep learning accessible to most people.\n",
    "\n",
    "我们正在经历的这个技术革命并不是由于某个突破性的发明导致的。就像其他的技术革命一样，它是多种因素累积的成果，一开始很缓慢，然后突然就全局突破了。在深度学习中，我们可以看到下面这些关键因素：\n",
    "\n",
    "- 不断的算法创新，一开始在两个年代中慢慢发展（开始于反向传播），然后在2012年之后由于大量研究投入发生了急速的发展。\n",
    "- 大量可用的感知数据，这是实现“在足够量的数据上训练足够大的模型”的其中一个关键。当然这得力于互联网的发展以及将存储介质的摩尔定律生效。\n",
    "- 有能力进行高速且高度并行化计算的便宜硬件设备，特别是NVIDIA的GPU，一开始为了游戏设计，后来开始专门为深度学习进行设计，NVIDIA的CEO黄仁勋早在深度学习热潮开始爆发的时候就决定将公司的未来都赌在上面。\n",
    "- 还需要一个复杂的软件架构能够让普通程序员容易使用这种计算能力：CUDA语言，TensorFlow这样的框架进行自动微分，还有Keras使得大部分人都能简单的进行深度学习实验。\n",
    "\n",
    "> In the future, deep learning will not only be used by specialists—researchers, grad\n",
    "students, and expensive engineers with an academic profile. Rather, it will be a tool in the\n",
    "toolbox of every developer, much like web technology today. Everyone needs to build\n",
    "intelligent apps: much like every business today needs a website, every product will need\n",
    "to intelligently make sense of user-generated data. Making this future happen requires us\n",
    "to build tools that make deep learning radically easy to use, that make deep learning\n",
    "accessible to anyone with basic coding abilities. Keras is the first major step in that\n",
    "direction.\n",
    "\n",
    "在未来，深度学习不应该仅仅称为专家、研究人员、毕业生和具有学术资历的高级工程的专用技术，它将会成为每个开发者的一个工具，就像现在的Web技术一样。每个人都需要构建智能化的应用，正如现在每个行业都需要网站，每个产品都需要智能来理解用户产生的数据。要达到这样的未来，需要我们在深度学习上构建一些非常容易使用的工具，使得深度学习能够被任何具有基础编程能力的人掌握。Keras是在这个方向上走出的第一步。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 9.1.5 通用机器学习工作流\n",
    "\n",
    "> Having access to an extremely powerful tool for creating models that map any input\n",
    "space to any target space is great, but the difficult part of the machine learning workflow\n",
    "is often everything that comes before designing and training such models (and, for\n",
    "production models, what comes after as well). Understanding the problem domain so as\n",
    "to be able to determine what to attempt to predict, given what data, and how to measure\n",
    "success, is a prerequisite for any successful application of machine learning, and it isn’t\n",
    "something that advanced tools like Keras or TensorFlow will help you with. As a\n",
    "reminder, here’s a quick summary of the typically machine learning workflow as\n",
    "described in Chapter 4:\n",
    "\n",
    "> - First, define the problem: what data is available, and what are you trying to predict? Will\n",
    "you need to collect more data, to hire people to manually label a dataset?\n",
    "- Identify a way to reliably measure success on your goal. For simple tasks, this may be\n",
    "prediction accuracy, but in many cases it will require sophisticated domain-specific\n",
    "metrics.\n",
    "- Prepare the validation process that you will use to evaluate your models. In particular,\n",
    "you should define a training set, validation set, and test set. Your validation and test set\n",
    "labels should not \"leak\" into your training data: for instance, with temporal prediction,\n",
    "the validation and test data should be posterior to the training data.\n",
    "- Vectorize your data, by turning it into vectors and preprocessing it in a way that makes it\n",
    "more easily approachable by a neural network (normalization, etc).\n",
    "- Develop a first model that beats a trivial common-sense baseline—thus demonstrating\n",
    "that machine learning can work on your problem. This might not always be the case!\n",
    "- Gradually refine your model architecture by tuning hyperparameters and adding\n",
    "regularization. Make changes based on your performance on the validation data only, not\n",
    "the test data nor the training data. Remember that you should manage to get your model\n",
    "to overfit (thus identifying a model capacity level that is above that you need), and only\n",
    "then start adding regularization or start downsizing your model.\n",
    "- Mind \"validation set overfitting\" when turning hyperparameters, i.e. the fact that your\n",
    "hyperparameters might end up being over-specialized to your validation set. Avoiding\n",
    "this is precisely the purpose of having a separate test set!\n",
    "\n",
    "具有能够创建从任何输入空间映射到任何目标空间的模型的工具当然很好，但是机器学习工作流中最困难的部分通常在设计和训练模型之前（当然对于生产模型来说，还有训练之后的问题）。对问题领域的理解对于设定预测的目标是至关重要的，需要什么样的数据，如何衡量成功，也都是机器学习应用的前提条件，这些都不是像Keras或者TensorFlow这样的高级工具能够帮你解决的。下面快速概要复习一下我们在第四章中介绍过的机器学习工作流：\n",
    "\n",
    "- 首先定义问题：可用的数据有哪些，你希望预测的目标是什么？是否需要收集更多数据，是否需要雇佣人来手动标记数据集？\n",
    "- 定义一个方法能够可靠的衡量你的目标是否成功。对于简单人物来说，这可以是预测准确率，但是很多情况下，你都需要复杂的领域相关指标。\n",
    "- 准备验证过程用于训练时衡量模型的性能。准确的说，你需要定义一个训练集、一个验证集和一个测试集。你的验证和测试集的标记不能“泄露”到你的训练数据中：例如，对于时序预测，验证数据和测试数据应该在时间上出现在训练数据之后。\n",
    "- 向量化你的数据，将数据转换成向量并且使用某种技巧对数据进行预处理使得它们更容易被神经网络处理（标准化等）。\n",
    "- 开发你的第一个模型，尝试用它击败最简单的常识性基线指标，这能够证明机器学习技术能够应用在你的问题上。记住并不是所有的问题都可能的。\n",
    "- 不断调整你的模型结构，可以通过调整超参数或者增加正则化。针对模型在验证数据上的表现来进行调整，而不是测试数据或者训练数据。记住你应该特意让模型出现过拟合（这样能够说明模型容量已经满足你的需要），这之后才开始增加正则化或者开始缩减模型容量。\n",
    "- 必须注意调整超参数时可能出现的“验证集过拟合”情况，也就是经过多次调整后你的超参数已经过于适合验证集了。这也是我们需要一个独立的测试集的根本原因。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 9.1.6 关键网络结构\n",
    "\n",
    "> The three families of network architectures that you should be familiar with are\n",
    "densely-connected networks, convolutional networks, and recurrent networks. Each type\n",
    "of network is meant for a specific input modality: a network architecture (dense,\n",
    "convolutional, recurrent) encodes assumptions about the structure of the data, a\n",
    "hypothesis space within which the search for a good model will proceed. Whether or not\n",
    "a given architecture will work on a given problem depends entirely on the match between\n",
    "the structure of the data and the assumptions of the network architecture.\n",
    "\n",
    "有三个网络结构家族是你应该熟悉的，它们是全连接网络、卷积网络和循环网络。每种类型的网络对应着特定输入形态：一个网络结构（全连接、卷积、循环）已经内涵了对数据结构的假设，也就是模型能够在其中搜索的假设空间。一种特定的结构是否能够在给定问题上工作，取决于数据的结构与网络结构假设之间是否匹配。\n",
    "\n",
    "> These different network types can be easily combined to achieve larger multi-modal\n",
    "networks, much like one would combine together Lego bricks. In a way, deep learning\n",
    "layers are Lego for information processing. Here is a quick overview of the mapping\n",
    "between input modalities and appropriate network architecture:\n",
    "\n",
    "> - Vector data: Densely-connected network ( Dense layers).\n",
    "- Image data: 2D convnets.\n",
    "- Sound data (e.g. waveform): Either 1D convnets (preferred) or RNNs.\n",
    "- Text data: Either 1D convnets (preferred) or RNN.\n",
    "- Timeseries data: Either RNNs (preferred) or 1D convnets.\n",
    "- Other types of sequence data: Either RNNs or 1D convnets. Prefer RNNs if data ordering\n",
    "is strongly meaningful (e.g. for timeseries, but not for text).\n",
    "- Video data: Either 3D convnets (if you need to capture motion effects) or a combination\n",
    "of a frame-level convnet for feature extraction followed by either a RNN or a 1D convnet\n",
    "to process the resulting sequences.\n",
    "- Volumetric data: 3D convnets.\n",
    "\n",
    "这些不同的网络类型能够组合成大型的多模态网络，就像我们可以用乐高积木组合成复杂形状一样。在某种程度上深度学习就是信息处理领域的乐高积木。下面列出了输入模态与合适的网络结构之间的对应表：\n",
    "\n",
    "- 向量数据：全连接网络（全连接层）\n",
    "- 图像数据：2D卷积网络\n",
    "- 声音数据（如声音波形）：1D卷积网络（推荐）或RNN\n",
    "- 文本数据：1D卷积网络（推荐）或RNN\n",
    "- 时序数据：RNN（推荐）或1D卷积网络\n",
    "- 其他类型的序列数据：RNN或1D卷积网络。推荐RNN如果数据顺序具有强烈含义（如时序数据，不是文本数据）\n",
    "- 视频数据：3D卷积网络（如果你需要捕捉动作效果）或使用一个对视频帧进行特征提取的卷积网络跟着一个RNN或者1D卷积网络对序列化结果进行处理的组合网络\n",
    "- 体数据：3D卷积网络\n",
    "\n",
    "> Now, let’s quickly review the specificities of each network architecture.\n",
    "\n",
    "下面让我们快速回顾一下每个网络结构的特性。\n",
    "\n",
    "#### 全连接网络\n",
    "\n",
    "> Densely-connected networks are just a stack of Dense layers, meant to process vector\n",
    "data (i.e. batches of vectors). They assume no specific structure in the input features: they\n",
    "are called \"densely-connected\" because the units of a Dense layer are \"connected\" to\n",
    "every other unit; the layer will attempt to map relationships between any two input\n",
    "features, which is unlike a 2D convolution layer, for instance, which only looks at local\n",
    "relationships.\n",
    "\n",
    "全连接网络是全连接层的堆叠，用来处理向量数据（批量的向量）。这种网络假设输入特征没有特定的结构：它们之所以被称为“全连接”是因为全连接层中的每个单元都“连接”到其他所有的单元，这样的层次会尝试映射任意输入特征之间的关系，不像其他的网络例如2D卷积层，只会观察局部的关系。\n",
    "\n",
    "> Densely-connected networks are most commonly used for categorical data (e.g.\n",
    "where the input features are lists of attributes), for instance the Boston Housing dataset\n",
    "we covered in Chapter 3. They are also used as the final classification or regression stage\n",
    "of most networks. For instance, the convnets we covered in Chapter 5 typically ended\n",
    "with one or two Dense layers, and so did our recurrent networks from Chapter 6.\n",
    "\n",
    "全连接网络通常用在分类数据上（例如输入的特征是属性的列表），例如我们在第三章中使用的波斯顿房价数据集。它们也会被用作大多数分类或者回归任务中最后阶段的层次。例如我们在第五章中介绍的卷积网络基本上都会以一个或两个全连接层结束，第六章中的循环网络也是一样。\n",
    "\n",
    "> Remember: to perform binary classification , end your stack of layers with a Dense\n",
    "layer with a single unit and a sigmoid activation, and use binary_crossentropy as\n",
    "loss. Your targets should be either 0 or 1.\n",
    "\n",
    "需要注意的是，要进行二分分类，网络的最高层应该是一个全连接层，只有一个单元并且使用`sigmoid`激活，整个网络使用`binary_crossentropy`作为损失函数。那么获得的目标应该是0或者1。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.layers import Dense\n",
    "from tensorflow.keras.models import Sequential\n",
    "\n",
    "num_input_features = 128\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Dense(32, activation='relu', input_shape=(num_input_features,)))\n",
    "model.add(Dense(32, activation='relu'))\n",
    "model.add(Dense(1, activation='sigmoid'))\n",
    "\n",
    "model.compile(optimizer='rmsprop', loss='binary_crossentropy')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> To perform single-label categorical classification (where each sample has exactly\n",
    "one class, no more), end your stack of layers with a Dense layer with a number of units\n",
    "equal to the number of classes, and a softmax activation. If your targets are one-hot\n",
    "encoded, use categorical_crossentropy as loss; if they are integers, use\n",
    "sparse_categorical_crossentropy .\n",
    "\n",
    "要进行单标签数据分类（也就是每个样本只有一个分类标签），网络的最高层应该是一个全连接层，使用与分类数量相等的单元数量，使用`softmax`激活。如果你的目标是`one-hot`编码过的，使用`categorical_crossentropy`作为损失函数；如果目标是整数，使用`sparse_categorical_crossentropy`损失函数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_classes = 10\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Dense(32, activation='relu', input_shape=(num_input_features,)))\n",
    "model.add(Dense(32, activation='relu'))\n",
    "model.add(Dense(num_classes, activation='softmax'))\n",
    "\n",
    "model.compile(optimizer='rmsprop', loss='categorical_crossentropy')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> To perform multi-label categorical classification (where each sample can have\n",
    "several classes), end your stack of layers with a Dense layer with a number of units equal\n",
    "to the number of classes and a sigmoid activation, and use binary_crossentropy as\n",
    "loss. Your targets should be one-hot encoded.\n",
    "\n",
    "要进行多标签数据分类（每个样本可能有多个分类），最高层用全连接，单元数量等于分类数量，使用`sigmoid`作为激活函数，使用`binary_crossentropy`作为损失函数。你的目标必须是`one-hot`编码的。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "model.add(Dense(32, activation='relu', input_shape=(num_input_features,)))\n",
    "model.add(Dense(32, activation='relu'))\n",
    "model.add(Dense(num_classes, activation='sigmoid'))\n",
    "\n",
    "model.compile(optimizer='rmsprop', loss='binary_crossentropy')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> To perform regression towards a vector of continuous values, end your stack of layers\n",
    "with a Dense layer with a number of units equal to the number of values you are trying to\n",
    "predict (often a single one, like the price of a house), and no activation. There are several\n",
    "losses that can be used for regression, most commonly mean_squared_error (MSE) and\n",
    "mean_absolute_error (MAE).\n",
    "\n",
    "要进行回归任务预测一个连续值的向量，最高层用全连接，单元数量等于你需要预测的数值个数（通常是一个值，例如房价），不适用激活函数。有许多可能的损失函数选择，最常用的包括`mean_squared_error`（MSE）和`mean_absolute_error(MAE)`。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_values = 1\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Dense(32, activation='relu', input_shape=(num_input_features,)))\n",
    "model.add(Dense(32, activation='relu'))\n",
    "model.add(Dense(num_values))\n",
    "\n",
    "model.compile(optimizer='rmsprop', loss='mse')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 卷积网络\n",
    "\n",
    "> Convolution layers look at spatially local patterns, by applying a same geometric\n",
    "transformation to different spatial locations (\"patches\") in an input tensor. This results in\n",
    "representations that are translation-invariant , making convolution layers highly\n",
    "data-efficient and modular. This idea is applicable to spaces of any dimensionality: 1D\n",
    "(sequences), 2D (images), 3D (volumes), etc. You can use the Conv1D layer to process\n",
    "sequences (especially text; it doesn’t work as well on timeseries that often do not follow\n",
    "the translation invariance assumption), the Conv2D layer to process images, and the\n",
    "Conv3D layers to process volumes.\n",
    "\n",
    "卷积层关注的是空间局部模式，通过对输入张量不同空间位置（“patches”）进行相同的几何转换。这体现在结果的转换不变性上，这让卷积层能非常高效的处理数据及模块化。这个原理能够应用在任何维度上：1D（序列），2D（图像），3D（体数据）等。你可以使用`Conv1D`层来处理序列（特别是文本，不太适合时间序列因为不符合转换不变性假设），使用`Conv2D`层来处理图像，使用`Conv3D`层来处理体数据。\n",
    "\n",
    "> Convnets, or convolutional networks, consist of stacks of convolution and\n",
    "max-pooling layers. The pooling layers allow to spatially downsample the data, which is\n",
    "required to keep feature maps to a reasonable size as the number of features grows, and\n",
    "to allow subsequent convolution layers to \"see\" a greater spatial extent of the inputs.\n",
    "Convnets are often ended with either a Flatten operation or a global pooling layer,\n",
    "turning spatial feature maps into vectors, followed by Dense layers to achieve\n",
    "classification or regression.\n",
    "\n",
    "卷积网络一般都是由堆叠的卷积层和最大池化层构成。池化层能够在空间范围对数据进行下采样，这样就能在特征数量增加的时候仍然保持特征地图在一个可以接受的大小范围内，还能够让后续的卷积层能够看到输入数据中更加广泛的范围。卷积网络通常最后会加上一个平铺层或者全局池化层，将空间特征地图转换成向量，再接着一个全连接层来进行分类或回归。\n",
    "\n",
    "> Note that it is highly likely that regular convolutions will soon be mostly (or\n",
    "completely) replaced by an equivalent but faster and even representationally efficient\n",
    "alternative, the depthwise separable convolution ( SeparableConv2D layer). This is true\n",
    "for 3D, 2D or 1D inputs. When building a new network from scratch, using depthwise\n",
    "separable convolutions is definitely the way to go. The SeparableConv2D layer can be\n",
    "used as a drop-in replacement for Conv2D , resulting in a smaller and faster network that\n",
    "will also perform better on its task.\n",
    "\n",
    "请注意目前的普通卷积不久之后很有可能大部分（或者全部）被深度可分卷积（`SeparableConv2D`）取代，这是一个相同功能但是更快甚至更加具有表现力的卷积技术。对于3D、2D或者1D数据都是如此。当从头构建新网络是，使用深度可分卷积绝对是最佳的选择。`SeparableConv2D`可以用来作为`Conv2D`的替代，并且获得一个更小更快的网络，在任务中输出更好的结果。\n",
    "\n",
    "> Here is what a typical image classification network would look like (categorical\n",
    "classification, in this case):\n",
    "\n",
    "下面是一个典型的图像分类网络的结构："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.layers import SeparableConv2D, MaxPooling2D, GlobalAveragePooling2D\n",
    "\n",
    "height = 128\n",
    "width = 128\n",
    "channels = 3\n",
    "\n",
    "model = Sequential()\n",
    "\n",
    "model.add(SeparableConv2D(32, 2, activation='relu', input_shape=(height, width, channels)))\n",
    "model.add(SeparableConv2D(64, 3, activation='relu'))\n",
    "model.add(MaxPooling2D(2))\n",
    "\n",
    "model.add(SeparableConv2D(64, 2, activation='relu'))\n",
    "model.add(SeparableConv2D(128, 3, activation='relu'))\n",
    "model.add(MaxPooling2D(2))\n",
    "\n",
    "model.add(SeparableConv2D(64, 2, activation='relu'))\n",
    "model.add(SeparableConv2D(128, 3, activation='relu'))\n",
    "model.add(GlobalAveragePooling2D())\n",
    "\n",
    "model.add(Dense(32, activation='relu'))\n",
    "model.add(Dense(num_classes, activation='softmax'))\n",
    "\n",
    "model.compile(optimizer='rmsprop', loss='categorical_crossentropy')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### RNN\n",
    "\n",
    "> Recurrent neural networks (RNNs) work by processing sequences of inputs one timestep\n",
    "at a time, and maintaining a \"state\" throughout (a state is typically just a vector, or set of\n",
    "vectors, i.e. a point in a geometric space of states). They should be used preferentially\n",
    "over 1D convnets in the case of sequences where patterns of interest are not invariant by\n",
    "temporal translation (for instance, timeseries data where the recent past if more important\n",
    "than the distant past).\n",
    "\n",
    "循环神经网络通过对序列数据每次处理一个时间步长来处理输入，并且在整个过程中保持着“状态”（状态实际上就是一个序列，或者一组向量，也就是状态几何空间中的点）。RNN应该在序列数据在时间维度上不能保持转换不变形的情况下优先于1D卷积网络来使用（例如，时间数据中邻近过去的信息比远古信息要重要的多的情况下）。\n",
    "\n",
    "> There are three RNN layers available in Keras: SimpleRNN , GRU and LSTM . For most\n",
    "practical purposes, you should be using either GRU or LSTM . LSTM is the more powerful of\n",
    "the two, but also the most expensive; you can see GRU as a simpler and cheaper\n",
    "alternative to it.\n",
    "\n",
    "Keras中有三种可以直接使用的RNN：`SimpleRNN`，`GRU`和`LSTM`。在实践中，你应该使用`GRU`或者`LSTM`。`LSTM`是其中最强大的一个，但是几乎也是最昂贵的，你可以将`GRU`看做是简单和更便宜的选择。\n",
    "\n",
    "> In order to stack multiple RNN layers on top of each other, each layer prior to the last\n",
    "one should return the full sequence of its outputs (to each input timestep will correspond\n",
    "an output timestep); if you are not stacking any further RNN layers, then it is common to\n",
    "only return the last output, which contains information about the entire sequence.\n",
    "\n",
    "为了能够将多个RNN层互相堆叠，每个前面的层次都必须输出完整的序列（对于每个输入时间步长都有相应的输出时间步长），如果你不需要堆叠后续的RNN层，那么就只需要返回最后的输出，也就是只需要包含整个序列的整体信息即可。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 单个RNN层做二分分类\n",
    "from tensorflow.keras.layers import LSTM, Dense\n",
    "\n",
    "num_classes = 2\n",
    "num_timesteps = 500\n",
    "num_features = 32\n",
    "\n",
    "model = Sequential()\n",
    "model.add(LSTM(32, input_shape=(num_timesteps, num_features)))\n",
    "model.add(Dense(num_classes, activation='sigmoid'))\n",
    "model.compile(optimizer='rmsprop', loss='binary_crossentropy')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 堆叠RNN层做二分分类\n",
    "\n",
    "model = Sequential()\n",
    "model.add(LSTM(32, return_sequences=True,\n",
    "input_shape=(num_timesteps, num_features)))\n",
    "model.add(LSTM(32, return_sequences=True))\n",
    "model.add(LSTM(32))\n",
    "model.add(Dense(num_classes, activation='sigmoid'))\n",
    "model.compile(optimizer='rmsprop', loss='binary_crossentropy')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 9.1.7 概率空间\n",
    "\n",
    "> What will you build with deep learning? Remember, building deep learning models is\n",
    "like playing with Lego bricks: layers can be plugged together to map essentially anything\n",
    "to anything, given that you have appropriate training data available and that the mapping\n",
    "is achievable via a continuous geometric transformation of reasonable complexity. The\n",
    "space of possibilities is infinite. Here are a few examples to inspire you to think beyond\n",
    "the basic classification or regression tasks that have been traditionally the bread and\n",
    "butter of machine learning.\n",
    "\n",
    "你应该如何构建深度学习模型？请记住，构建深度学习模型就像是堆乐高积木：层次被互相堆叠镶嵌到一起，来将一样事物映射到另一样事物，通过合适的训练数据和足够复杂的连续几何变换可以完成基本所有任务。概率空间是无限的。下面有一些例子，能够让你突破思维，从传统的机器学习中的面包和黄油也就是基础的分类和回归任务中想的更多。\n",
    "\n",
    "> We sort our suggested applications by input and output modalities. Do note that quite\n",
    "a few of them are stretching the limits of what is possible—while a model could be\n",
    "trained on these tasks in every case, in some cases such a model would probably not\n",
    "generalize very far from its training data. In the next couple sections, we address how\n",
    "these limitations could be lifted in the future.\n",
    "\n",
    "我们根据这些应用的输入输出模态来排序。请记住这里面的许多任务基本上没有什么局限性，因此你可以在任何场景中训练模型来完成任务，但是在某些场景中模型可能无法利用训练数据进行良好的泛化。在后面的小节当中，我们会讨论未来这些领域可能获得的突破。\n",
    "\n",
    "#### 将向量数据映射到向量数据\n",
    "\n",
    "> - Predictive healthcare: mapping patient medical records to predictions of patient outcome.\n",
    "- Behavioral targeting: mapping a set of website attributes with data on how long a user\n",
    "will spend on the website.\n",
    "- Product quality control: apping a set of attributes relative to an instance of a\n",
    "manufactured product with the probability that the product will fail by next year.\n",
    "\n",
    "- 预测健康情况：将病人的医疗记录数据映射到病人健康情况的预测上。\n",
    "- 行为分析：将一个网站的属性数据映射到用户可能在网站停留时间的预测上。\n",
    "- 产品质量控制：将某个产品的相关属性数据映射到下一个年度该产品的不良率预测上。\n",
    "\n",
    "#### 将图像数据映射到向量数据\n",
    "\n",
    "> - Doctor assistant: mapping slides of medical images with a prediction about the presence\n",
    "of a tumor.\n",
    "- Self-driving: mapping car dashcam video frames to steering wheel angle commands.\n",
    "- Board game AI: mapping Go or Chess boards to to next player move.\n",
    "- Diet helper: mapping pictures of a dish with its calorie count.\n",
    "- Age prediction: mapping selfies to the age of the person.\n",
    "\n",
    "- 辅助诊断：将一系列的医学图像映射到某个肿瘤诊断预测上。\n",
    "- 自动驾驶：将行车记录仪摄像头的图像数据映射到驾驶方向盘的控制数据上。\n",
    "- 棋类游戏AI：将围棋或国际象棋棋盘图像数据映射到下一步落子的数据上。\n",
    "- 节食助手：将餐盘食物的图像映射到卡路里数量上。\n",
    "- 年龄判断：将自拍照映射到人的年龄数据上。\n",
    "\n",
    "#### 将时序数据映射到向量数据\n",
    "\n",
    "> - Weather prediction: mapping timeseries of weather data in a grid of locations of weather\n",
    "data the following week at a specific location.\n",
    "- Brain-computer interfaces: mapping timeseries of magnetoencephalogram (MEG) data to\n",
    "computer commands.\n",
    "- Behavioral targeting: mapping timeseries of user interactions on a website with the\n",
    "probability that a user will buy something.\n",
    "\n",
    "- 天气预测：将某个地区的历史天气数据映射到该地区未来几天的天气预测数据上。\n",
    "- 脑机接口：将脑磁图数据（MEG）映射到计算机指令数据上。\n",
    "- 行为预测：将用网站的户历史行为数据映射到该用户购买商品的可能性数据上。\n",
    "\n",
    "#### 将文本映射到文本\n",
    "\n",
    "> - Smart reply: mapping emails to possible one-line replies.\n",
    "- Question answering: mapping general knowledge questions to answers.\n",
    "- Summarization: mapping long articles to short summary of the article.\n",
    "\n",
    "- 智能回复：将电子邮件映射到可能的简短回复上。\n",
    "- 问答系统：将通用知识问题映射到答案上。\n",
    "- 概要：将长文章映射到简短的概要上。\n",
    "\n",
    "#### 将文本映射到图像\n",
    "\n",
    "> - Conditioned image generation: mapping short text descriptions to images matching the\n",
    "description.\n",
    "- Logo generation/selection: mapping the name and description of a company to the\n",
    "company’s logo.\n",
    "\n",
    "- 条件图像生成：将简短的文字描述映射到符合描述的图像上。\n",
    "- 图标生成/选择：将公司名称和描述映射到公司的图标上。\n",
    "\n",
    "#### 将图像映射到图像\n",
    "\n",
    "> - Super-resolution: mapping downsized images to higher-resolution versions of the same\n",
    "images.\n",
    "- Visual depth sensing: mapping images of indoor environments to maps of depth\n",
    "predictions.\n",
    "\n",
    "- 超分辨率：将缩小的图像映射到高分辨率的版本。\n",
    "- 视觉深度感知：将室内环境图像映射到图像深度地图。\n",
    "\n",
    "#### 将图像和文本映射到文本\n",
    "\n",
    "> - Visual QA: mapping images and natural language questions about the contents of the\n",
    "images, to natural language answers.\n",
    "\n",
    "- 可视化问答：将图像和有关图像的自然语言描述的问题映射到相应的答案上。\n",
    "\n",
    "#### 将视频和文本映射到文本\n",
    "\n",
    "> - Video QA: mapping short videos and natural language questions about the contents of the\n",
    "video, to natural language answers.\n",
    "\n",
    "- 视频问答：将短视频和自然语言描述的问题映射到相应的答案上。\n",
    "\n",
    "> Almost anything is possible—but not quite anything . Let’s see in the next section what\n",
    "you can’t do with deep learning.\n",
    "\n",
    "几乎任何任务都是可能的，但也就是几乎。下一节我们来看看哪些事情是无法使用深度学习完成的。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 9.2 深度学习的局限性\n",
    "\n",
    "> The space of applications that can be implemented with deep learning is nearly infinite.\n",
    "And yet, many more applications are completely out of reach for current deep learning\n",
    "techniques—even given vast amounts of human-annotated data. Say, for instance, that\n",
    "you could assemble a dataset of hundreds of thousands—even millions—of English\n",
    "language descriptions of the features of a software product, as written by a product\n",
    "manager, as well as the corresponding source code developed by a team of engineers to\n",
    "meet these requirements. Even with this data, you could not train a deep learning model\n",
    "to simply read a product description and generate the appropriate codebase. That’s just\n",
    "one example among many. In general, anything that requires reasoning—like\n",
    "programming, or applying the scientific method—long-term planning, and\n",
    "algorithmic-like data manipulation, is out of reach for deep learning models, no matter\n",
    "how much data you throw at them. Even learning a sorting algorithm with a deep neural\n",
    "network is tremendously difficult.\n",
    "\n",
    "深度学习的应用空间几乎是无限的。然而，还有很多的应用完全超过了目前深度学习技术的范围，即使在拥有了大量人工标注的数据的情况也是如此。例如你可以收集一个具有数十万甚至数百万的软件产品特征描述，它们是由产品经理写成的，你还可以收集这些产品相应的所有源代码，它们是有软件工程师团队依据产品需求写成的。即使你有了这些数据，你也无法训练一个深度学习模型来读取新的产品需求描述然后生成相应的代码库。这只是其中一个例子。一般的说，任何需要推理逻辑的事物，如编程，或者应用科学方法证明，以及代数式数据操作，都超出了深度学习模型的范围，无论你提供给模型多少数据。甚至在深度神经网络中训练一个排序算法都是极端困难的。\n",
    "\n",
    "> This is because a deep learning model is \"just\" a chain of simple, continuous\n",
    "geometric transformations mapping one vector space into another. All it can do is map\n",
    "one data manifold X into another manifold Y, assuming the existence of a learnable\n",
    "continuous transform from X to Y. So even though a deep learning model can be\n",
    "interpreted as a kind of program, inversely most programs cannot be expressed as deep\n",
    "learning models —for most tasks, either there exists no corresponding deep neural\n",
    "network that solves the task, or even if there exists one, it may not be learnable , i.e. the\n",
    "corresponding geometric transform may be far too complex, or there may not be\n",
    "appropriate data available to learn it.\n",
    "\n",
    "这是因为深度学习模型试试简单的连续几何变换的串联，将一个向量空间映射到另一个向量空间。它的所有工作就是将数据的流形X转换成另一个流形Y，前提是假设存在着这样的从X到Y的连续变换。因此虽然深度学习模型可以认为以一种程序，但是反过来大多数程序都无法使用深度学习模型表达。对于大多数这样的任务来说，要不就是根本不存在这样的深度学习模型，要不就是即使存在，它是不可学习的，也就是这样的几何变换可以过于复杂，或者不存在合适的数据能够训练它。\n",
    "\n",
    "> Scaling up current deep learning techniques by stacking more layers and using more\n",
    "training data can only superficially palliate some of these issues. It will not solve the\n",
    "more fundamental problem that deep learning models are very limited in what they can\n",
    "represent, and that most of the programs that one may wish to learn cannot be expressed\n",
    "as a continuous geometric morphing of a data manifold.\n",
    "\n",
    "通过堆叠更多的层次和使用更多的训练数据来扩展目前的深度学习技术，仅能表面上缓解这样的问题。它无法从根本上解决深度学习模型在表现形式上的局限性，且大多数这样的程序都无法表示成数据流形的连续集合变形。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 9.2.1 机器学习模型拟人化的风险\n",
    "\n",
    "> One very real risk with contemporary AI is that of misinterpreting what deep learning\n",
    "models do, and overestimating their abilities. A fundamental feature of the human mind\n",
    "is our \"theory of mind\", our tendency to project intentions, beliefs and knowledge on the\n",
    "things around us. Drawing a smiley face on a rock suddenly makes it \"happy\"—in our\n",
    "minds. Applied to deep learning, this means that when we are able to somewhat\n",
    "successfully train a model to generate captions to describe pictures, for instance, we are\n",
    "led to believe that the model \"understands\" the contents of the pictures, as well as the\n",
    "captions it generates. We then proceed to be very surprised when any slight departure\n",
    "from the sort of images present in the training data causes the model to start generating\n",
    "completely absurd captions.\n",
    "\n",
    "这是目前AI发展的一个真实风险，就是错误解读了深度学习模型的工作，高估了它们的能力。人类心智的基本特征在于我们的“心智理论”，我们惯于将目标、信仰和知识投射到我们周边的事物当中。在石头上画一张微笑的脸就会让我们心智上感到高兴。如果应用到深度学习上，这意味着我们在某种程度上训练了一个能够描述图像内容的模型，也就是说我们相信模型能够“理解”图像的内容和它产生的解读。然后我们就会惊奇的发现如果出现了与训练数据有细微偏离的图像时，模型开始产生完全荒唐的解读。\n",
    "\n",
    "![mis captions of dl](imgs/f9.1.jpg)\n",
    "\n",
    "图9-1 深度学习模型对图像解读的一个错误案例"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> In particular, this is highlighted by \"adversarial examples\", which are input samples\n",
    "to a deep learning network that are designed to trick the model into misclassifying them.\n",
    "You are already aware that it is possible to do gradient ascent in input space to generate\n",
    "inputs that maximize the activation of some convnet filter, for instance—this was the\n",
    "basis of the filter visualization technique we introduced in Chapter 5, as well as the Deep\n",
    "Dream algorithm from Chapter 8. Similarly, through gradient ascent, one can slightly\n",
    "modify an image in order to maximize the class prediction for a given class. By taking a\n",
    "picture of a panda and adding to it a \"gibbon\" gradient, we can get a neural network to\n",
    "classify this panda as a gibbon. This evidences both the brittleness of these models, and\n",
    "the deep difference between the input-to-output mapping that they operate and our own\n",
    "human perception.\n",
    "\n",
    "确切来说，这种数据就被称为“对抗样本”，表示那些输入到深度学习模型中用来误导模型预测结论的数据样本。读者已经了解了我们可以在卷积网络过滤器中通过梯度增强来最大化某些输入空间的模式，例如我们在第五章中使用的过滤器可视化技术，还有就是第八章中的Deep Dream算法。类似的，通过梯度增强，我们也可以稍微编辑一张图像，使得模型会最大化某个分类的预测值。拍摄一张熊猫的照片然后在上面增加“长臂猿”的梯度，我们就可以让神经网络将熊猫辨识为长臂猿。这既证明了这些模型的脆弱性，也证明了模型的输入输出映射和我们人类的认知之间的深层次差异。\n",
    "\n",
    "![adversarial example](imgs/f9.2.jpg)\n",
    "\n",
    "图9-2 对抗样本：一个肉眼无法感知的图像修改会导致模型完全颠覆性的预测错误"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> In short, deep learning models do not have any understanding of their input, at least\n",
    "not in any human sense. Our own understanding of images, sounds, and language, is\n",
    "grounded in our sensorimotor experience as humans. Machine learning models have no\n",
    "access to such experiences and thus cannot \"understand\" their inputs in any\n",
    "human-relatable way. By annotating large numbers of training examples to feed into our\n",
    "models, we get them to learn a geometric transform that maps data to human concepts on\n",
    "this specific set of examples, but this mapping is just a simplistic sketch of the original\n",
    "model in our minds, the one developed from our experience as embodied agents—it is\n",
    "like a dim image in a mirror.\n",
    "\n",
    "简要来说，深度学习模型完全不具有任何对数据数据的理解，至少不再任何人类的层次上。我们自身对图像、声音和语言的理解，是建立在人类感知活动经验之上的。机器学习模型不具有这样的经验，因此无法在人类的感知层次上“理解”它们的数据。通过将大量标注的数据输入到模型中进行训练，我们知识让模型学习到了如何按照人类的概念将这一部分特定样本通过几何变换映射到另外一种形式而已，但是这样的映射知识原始人类思维模型的草图，也就是发展了一个人类经验的代理方，就像镜像中的衣服模糊的图像。\n",
    "\n",
    "![dim mirror of human experience](imgs/f9.3.jpg)\n",
    "\n",
    "图9-3 目前机器学习模型：就像镜像中模糊的图像"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> As a machine learning practitioner, always be mindful of this, and never fall into the\n",
    "trap of believing that neural networks understand the task they perform—they don’t, at\n",
    "least not in a way that would make sense to us. They were trained on a different, far\n",
    "narrower task than the one we wanted to teach them: that of merely mapping training\n",
    "inputs to training targets, point by point. Show them anything that deviates from their\n",
    "training data, and they will break in the most absurd ways.\n",
    "\n",
    "作为一个机器学习实践者，必须永远牢记这一点，那就是永远不要相信神经网络能够理解它们执行的任务，它们不能，至少不是以我们的方式理解任务的。它们被训练成仅能在不同的狭义的任务上工作，而不是我们希望教会它们的范围：也就是它们将输入数据点对点的映射到目标。只要给他们展示任何偏离训练数据的内容，它们都会产生荒谬的错误。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 9.2.2 局部泛化 vs 极致泛化\n",
    "\n",
    "> There just seems to be fundamental differences between the straightforward geometric\n",
    "morphing from input to output that deep learning models do, and the way that humans\n",
    "think and learn. It isn’t just the fact that humans learn by themselves from embodied\n",
    "experience instead of being presented with explicit training examples. Aside from the\n",
    "different learning processes, there is a fundamental difference in the nature of the\n",
    "underlying representations.\n",
    "\n",
    "刚才看到深度学习模型直接将输入到输出进行几何变换的方式，与人类思考与学习方式的本质不同。除了人类是通过自身的经验学习而不是通过展示的特定训练样本学习这个区别之外，还有一个本质的区别，那就是底层表现形式的本质区别。\n",
    "\n",
    "> Humans are capable of far more than mapping immediate stimuli to immediate\n",
    "responses, like a deep net, or maybe an insect, would do. They maintain complex,\n",
    "abstract models of their current situation, of themselves, of other people, and can use\n",
    "these models to anticipate different possible futures and perform long-term planning.\n",
    "They are capable of merging together known concepts to represent something they have\n",
    "never experienced before—like picturing a horse wearing jeans, for instance, or\n",
    "imagining what they would do if they won the lottery. This ability to handle\n",
    "hypotheticals, to expand our mental model space far beyond what we can experience\n",
    "directly, in a word, to perform abstraction and reasoning , is arguably the defining\n",
    "characteristic of human cognition. I call it \"extreme generalization\": an ability to adapt to\n",
    "novel, never experienced before situations, using very little data or even no new data at\n",
    "all.\n",
    "\n",
    "人类的思维能力远远不止将即时刺激转换成即时反应，就是深度网络或者昆虫那样。人类能够保存对当前情况（包括自身、其他人等）复杂的抽象的模型，并且将这些模型用来预测不同可能的未来及采取长期的规划。人类能够将多个一致的概念整合起来用来表示那些他们之前没有体验过的事物，例如画出一只穿着牛仔服的马匹，或者想象如果赢了彩票他们会做什么。这种处理假想的能力，这种能够将思维模型空间扩展到超越直接体验的能力，简而言之，就是一种抽象和推理能力，是人类认知定义中的一项具有争议特征。作者将其称为“极致泛化”：这是一种能够仅仅使用少量数据甚至没有任何数据的情况下，空想那些从未经历过的情景的能力。\n",
    "\n",
    "> This stands in sharp contrast with what deep nets do, which I would call \"local\n",
    "generalization\": the mapping from inputs to outputs performed by deep nets quickly stops\n",
    "making sense if new inputs differ even slightly from what they saw at training time.\n",
    "Consider, for instance, the problem of learning the appropriate launch parameters to get a\n",
    "rocket to land on the moon. If you were to use a deep net for this task, whether training\n",
    "using supervised learning or reinforcement learning, you would need to feed it with\n",
    "thousands or even millions of launch trials, i.e. you would need to expose it to a dense\n",
    "sampling of the input space, in order to learn a reliable mapping from input space to\n",
    "output space. By contrast, humans can use their power of abstraction to come up with\n",
    "physical models—rocket science—and derive an exact solution that will get the rocket on\n",
    "the moon in just one or few trials. Similarly, if you developed a deep net controlling a\n",
    "human body, and wanted it to learn to safely navigate a city without getting hit by cars,\n",
    "the net would have to die many thousands of times in various situations until it could\n",
    "infer that cars and dangerous, and develop appropriate avoidance behaviors. Dropped\n",
    "into a new city, the net would have to relearn most of what it knows. On the other hand,\n",
    "humans are able to learn safe behaviors without having to die even once—again, thanks\n",
    "to their power of abstract modeling of hypothetical situations.\n",
    "\n",
    "这一点完全与深度网络没有丝毫相关性，深度网络的泛化能力，作者称为“局部泛化”：因为深度网络使用的从输入到输出的映射方式只要在对输入进行细微的改变的情况下就和快变得没有意义了，这个改变只要没有出现在训练中就会导致这种泛化失败。例如假设我们需要训练一个模型来提供火箭登录月球的相关参数。如果采用深度网络来进行这项任务的话，不论使用的是有监督学习还是强化学习，你都需要给模型提供上千甚至上百万次的模拟，也就是说你需要提供从输入空间到输出空间的一个稠密样本空间给模型。相反人类可以使用他们的抽象能力来建立物理模型，火箭科学，然后从一次获此少数几次的试验中推导出这些相关的参数。同样的，如果你需要开发一个深度网络来控制人体运动，希望它能帮助人体安全的穿越城市道路而避开车辆，这个网络需要首先在不同情况下死亡数千次，才能最终推断出什么是车辆以及危险，并且发展处合适的避险行为。然后将它放到一个新的城市中，网络有需要重新学习大多数的内容。相反人类可以在不用死亡的情况下就学习到什么是安全的行为，感谢我们具有这样的对假想环境的抽象模型能力。\n",
    "\n",
    "![local generalization vs extreme generalization](imgs/f9.4.jpg)\n",
    "\n",
    "图9-4 局部泛化 vs 极致泛化"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> In short, despite our progress on machine perception, we are still very far from\n",
    "human-level AI: our models can only perform local generalization , adapting to new\n",
    "situations that must stay very close from past data, while human cognition is capable of\n",
    "extreme generalization , quickly adapting to radically novel situations, or planning very\n",
    "for long-term future situations.\n",
    "\n",
    "简而言之，尽管我们在机器感知领域获得了巨大进展，我们距离人类层面的AI还有很长距离：我们的模型只能局部泛化，它们只能适应那些与过往数据非常接近的环境，而人类认知具有极致泛化能力，能够快速适应那些急剧变化的新环境，或者对长期未来情况做出规划。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 9.2.3 小结\n",
    "\n",
    "> Here’s what you should remember: the only real success of deep learning so far has been\n",
    "the ability to map space X to space Y using a continuous geometric transform, given\n",
    "large amounts of human-annotated data. Doing this well is a game-changer for essentially\n",
    "every industry, but it is still a very long way from human-level AI.\n",
    "\n",
    "本小节你应该记住：深度学习目前取得的成功仅限于使用连续几何变换将空间X映射到空间Y上，同时需要大量人类标记的数据。能将这件事做好可以改变每一个行业的现状，但是它距离人类级别的智能还有很长距离。\n",
    "\n",
    "> To lift some of these limitations and start competing with human brains, we need to\n",
    "move away from straightforward input-to-output mappings, and on to reasoning and\n",
    "abstraction . A likely appropriate substrate for abstract modeling of various situations and\n",
    "concepts is that of computer programs. We have said before that machine learning\n",
    "models could be defined as \"learnable programs\"; currently we can only learn programs\n",
    "that belong to a very narrow and specific subset of all possible programs. But what if we\n",
    "could learn any program, in a modular and reusable way? Let’s see in the next section\n",
    "what the road ahead may look like.\n",
    "\n",
    "要突破这些限制并且与人脑产生竞争的话，我们需要从直接的输入输出映射方式进化到推理和抽象的方式。不同环境和概念的抽象模型的一个可能实现就是计算机程序。我们前面说过机器学习模型可以被定义为“可学习的程序”；目前我们只能学习那些非常狭窄和特定情景下的程序。但如果我们能够以模块化和可重用的方式学习任何程序呢？下一节中我们将会看到未来的路会怎么走。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 9.3 深度学习的未来\n",
    "\n",
    "> This is a more speculative section aimed at opening up horizons to people who want to\n",
    "get into a research program or start doing independent research. Given what we know of\n",
    "how deep nets work, of their limitations, and of the current state of the research\n",
    "landscape, can we predict where things are headed in the medium term? Here are some\n",
    "purely personal thoughts. Note that I don’t have a crystal ball, so a lot of what I anticipate\n",
    "might fail to become reality. I am sharing these predictions not because I expect them to\n",
    "be proven completely right in the future, but because they are interesting and actionable\n",
    "in the present.\n",
    "\n",
    "这是一个推测的小结，目标读者是那些希望加入研究团队或者开始独立研究的人群，期望能够达到打开更大视野的目的。我们前面已经知道了深度学习网络的工作原理以及它们的局限性，我们了解了当前研究领域的状况之后，是否能够预测中期的未来场景？本小节是作者个人的一些思考。提醒读者的是，作者没有水晶球，因此很多预测的事物可能并不会出现。作者分享这些预测不是因为期望它们未来能被证明完全正确，而是因为它们在当前状况下很有趣并且具有可行性。\n",
    "\n",
    "> At a high-level, the main directions in which I see promise are:\n",
    "\n",
    "> - Models closer to general-purpose computer programs, built on top of far richer primitives\n",
    "than our current differentiable layers—this is how we will get to reasoning and\n",
    "abstraction , the fundamental weakness of current models.\n",
    "- New forms of learning that make the above possible—allowing models to move away\n",
    "from just differentiable transforms.\n",
    "- Models that require less involvement from human engineers—it shouldn’t be your job to\n",
    "tune knobs endlessly.\n",
    "- Greater, systematic reuse of previously learned features and architectures; meta-learning\n",
    "systems based on reusable and modular program subroutines.\n",
    "\n",
    "用较高的角度来说，作者看到未来的一些希望方向是：\n",
    "\n",
    "- 模型更加接近通用计算机程序，建立在更加丰富的原语之上，而不是目前仅仅的微分层次，这是我们获得推理和抽象能力的关键，也是目前模型的最根本弱点。\n",
    "- 新形式的学习方法的发明，使得上面变为可能，让模型能够摆脱仅仅是微分转换形式。\n",
    "- 模型需要更少的人类工程师介入，无穷无尽的超参数调整不应该是你的工作。\n",
    "- 更加广泛更加系统化的预训练特征和结构的重用，可以在这些可重用模块化的子程序之上建立元学习。\n",
    "\n",
    "> Additionally, do note that these considerations are not specific to the sort of\n",
    "supervised learning that has been the bread and butter of deep learning so far—rather,\n",
    "they are applicable to any form of machine learning, including unsupervised,\n",
    "self-supervised, and reinforcement learning. It is not fundamentally important where your\n",
    "labels come from or what your training loop looks like; these different branches of\n",
    "machine learning are just different facets of a same construct.\n",
    "\n",
    "需要额外强调的是，上面这些想法不仅仅针对有监督学习这种目前深度学习的基本形式，它们应该对于任何形式的机器学习都通用，包括无监督学习、自监督学习和强化学习。获得数据标记的方式和你的训练流程不是根本的因素，机器学习的这些不同分支知识同一个结构的不同剖面而已。\n",
    "\n",
    "> Let’s dive in.\n",
    "\n",
    "我们深入讨论一下。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 9.3.1 模型即程序\n",
    "\n",
    "> As we noted in the previous section, a necessary transformational development that we\n",
    "can expect in the field of machine learning is a move away from models that perform\n",
    "purely pattern recognition and can only achieve local generalization , towards models\n",
    "capable of abstraction and reasoning , that can achieve extreme generalization . Current\n",
    "AI programs that are capable of basic forms of reasoning are all hard-coded by human\n",
    "programmers: for instance, software that relies on search algorithms, graph manipulation,\n",
    "formal logic. In DeepMind’s AlphaGo, for example, most of the \"intelligence\" on display\n",
    "is designed and hard-coded by expert programmers (e.g. Monte-Carlo tree search);\n",
    "learning from data only happens in specialized submodules (value networks and policy\n",
    "networks). But in the future, such AI systems may well be fully learned, with no human\n",
    "involvement.\n",
    "\n",
    "正如前面所述，我们期望机器学习领域能够发生的革命性改变就是让模型能够具有抽象和推理的能力，而不仅仅是目前的单纯的模式识别获得的局部泛化能力，使得最终模型具有极致泛化的能力。当前具有基本推理能力的AI程序都是由人类程序员硬编码进程序中的：例如那些依赖搜索算法、图操作、形式逻辑的软件。比方说DeepMind的AlphaGo，大部分展现的“智能”都是人类专家程序员设计并硬编码在内的（如蒙特卡洛树搜索）；从数据中学习仅仅发生在特定的子模块当中（价值网络和策略网络）。在未来，这样的AI系统可以完全学习获得，不需要人类干预。\n",
    "\n",
    "> What could be the path to make this happen? Consider a well-known type of network:\n",
    "RNNs. Importantly, RNNs have slightly less limitations than feedforward networks. That\n",
    "is because RNNs are a bit more than a mere geometric transformation: they are geometric\n",
    "transformations repeatedly applied inside a for loop . The temporal for loop is itself\n",
    "hard-coded by human developers: it is a built-in assumption of the network. Naturally,\n",
    "RNNs are still extremely limited in what they can represent, primarily because each step\n",
    "they perform is still just a differentiable geometric transformation, and the way they carry\n",
    "information from step to step is via points in a continuous geometric space (state\n",
    "vectors). Now, imagine neural networks that would be \"augmented\" in a similar way with\n",
    "programming primitives such as for loops—but not just a single hard-coded for loop\n",
    "with a hard-coded geometric memory, rather, a large set of programming primitives that\n",
    "the model would be free to manipulate to expand its processing function, such as if\n",
    "branches, while statements, variable creation, disk storage for long-term memory,\n",
    "sorting operators, advanced datastructures like lists, graphs, and hashtables, and many\n",
    "more. The space of programs that such a network could represent would be far broader\n",
    "than what can be represented with current deep learning models, and some of these\n",
    "programs could achieve superior generalization power.\n",
    "\n",
    "实现这个目标的路径是什么？考虑一种熟知的网络类型RNN。RNN与其他前向网络有一个重要区别，那就是RNN具有超出几何变换的部分能力：它能够在一个循环中重复进行几何变换。循环的持续长度是人类开发人员硬编码在网络中的：这是网络内在的假设。当然RNN仍然在表现形式上受到了极大的限制，主要因为它进行的每一步操作仍然是一个可微分的几何变换，以及它通过连续空间中的点（状态向量）在每一步中传递信息的方式。现在让我们想象一下神经网络通过程序设计的原语例如循环这样的方法进行“扩增”，但不仅局限于在一个硬编码的几何空间中使用一个硬编码的for循环，而是能够应用极大范围的程序设计原语，模型能够在处理过程中自由的操控和扩展，例如if分支、while、创建变量、磁盘存储以实现长期记忆、排序操作、甚至更高级的数据结构如列表、图和哈希表等等。那么这个模型程序能够展现的空间将会大大超越目前的深度学习模型，这些程序一部分就能获得超越的泛化能力。\n",
    "\n",
    "> In a word, we will move away from having on one hand \"hard-coded algorithmic\n",
    "intelligence\" (handcrafted software) and on the other hand \"learned geometric\n",
    "intelligence\" (deep learning). We will have instead a blend of formal algorithmic\n",
    "modules that provide reasoning and abstraction capabilities, and geometric modules that\n",
    "provide informal intuition and pattern recognition capabilities. The whole system would\n",
    "be learned with little or no human involvement.\n",
    "\n",
    "简短来说我们将走出一手“硬编码算法智能”（手工实现软件）和另一手“学习到的几何智能”（深度学习）的境地。我们会获得具有推理和抽象能力的形式算法，以及提供非形式直觉和模式识别能力几何模块的混合体。整个系统能够在极少甚至无人工干预的情况下学习。\n",
    "\n",
    "> A related subfield of AI that I think may be about to take off in a big way is that of\n",
    "program synthesis , in particular neural program synthesis. Program synthesis consists in\n",
    "automatically generating simple programs, by using a search algorithm (possibly genetic\n",
    "search, as in genetic programming) to explore a large space of possible programs. The\n",
    "search stops when a program is found that matches the required specifications, often\n",
    "provided as a set of input-output pairs. As you can see, is it highly reminiscent of\n",
    "machine learning: given \"training data\" provided as input-output pairs, we find a\n",
    "\"program\" that matches inputs to outputs and can generalize to new inputs. The\n",
    "difference is that instead of learning parameter values in a hard-coded program (a neural\n",
    "network), we generate source code via a discrete search process.\n",
    "\n",
    "作者认为AI其一个将要获得巨大进展的相关子领域是程序合成，确切的说是神经程序合成。程序合成包括自动化生成简单程序，通过使用一个搜索算法（可能是遗传程序设计中的遗传搜索）来探索可能程序所在的巨大空间。搜索会在达到需要的指标之后停止，通常指标会以一组输入输出对的形式提供。你可以看到，这种做法很容易让你联想到机器学习：以输入输出对的方式给定“训练数据”，我们会找到一个“程序”能够将输入映射到输出并且对新的输入提供泛化能力。两者的区别在于机器学习是在一个硬编码程序中学习参数值（神经网络），而程序合成是在一个离散搜索过程中产生源代码。\n",
    "\n",
    "> I would definitely expect this subfield to see a wave of renewed interest in the next\n",
    "few years. In particular, I would expect the emergence of a crossover subfield in-between\n",
    "deep learning and program synthesis, where we would not quite be generating programs\n",
    "in a general-purpose language, but rather, where we would be generating neural networks\n",
    "(geometric data processing flows) augmented with a rich set of algorithmic primitives,\n",
    "such as for loops—and many others. This should be far more tractable and useful than\n",
    "directly generating source code, and it would dramatically expand the scope of problems\n",
    "that can be solved with machine learning—the space of programs that we can generate\n",
    "automatically given appropriate training data. A blend of symbolic AI and geometric AI.\n",
    "Contemporary RNNs can be seen as a prehistoric ancestor to such hybrid\n",
    "algorithmic-geometric models.\n",
    "\n",
    "作者迫切的期望这个子领域的突破能够在未来几年中延续机器学习的热潮。更具体来说作者急切希望看到两个领域的结合，深度学习和程序合成，这里我们并不是使用通用语言生成程序，而是使用一组丰富的算法原语来生成神经网络（几何数据处理流程），比方说循环和其他的程序结构。这比直接生成源代码要容易实现和有用，并且能够极大扩展机器学习能够自动解决问题的范围，范围能够扩大到给定合适的训练数据情况下生成程序的空间。符号AI和几何AI以及当前的RNN都可以被认为是这种混合算法-几何模型的远古祖先。\n",
    "\n",
    "![program synthesis](imgs/f9.5.jpg)\n",
    "\n",
    "图9-5 一个学习到的程序结合了几何（模式识别、感知）和算法原语（推理、搜索、记忆）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 9.3.2 超越反向传播和可微分层次\n",
    "\n",
    "> If machine learning models become more like programs, then they will mostly no longer\n",
    "be differentiable—certainly, these programs will still leverage continuous geometric\n",
    "layers as subroutines, which will be differentiable, but the model as a whole would not\n",
    "be. As a result, using backpropagation to adjust weight values in a fixed, hard-coded\n",
    "network, cannot be the method of choice for training models in the future—at least, it\n",
    "cannot be the whole story. We need to figure out to train non-differentiable systems\n",
    "efficiently. Current approaches include genetic algorithms, \"evolution strategies\", certain\n",
    "reinforcement learning methods, and ADMM (alternating direction method of\n",
    "multipliers). Naturally, gradient descent is not going anywhere—gradient information\n",
    "will always be useful for optimizing differentiable parametric functions. But our models\n",
    "will certainly become increasingly more ambitious than mere differentiable parametric\n",
    "functions, and thus their automatic development (the \"learning\" in \"machine learning\")\n",
    "will require more than backpropagation.\n",
    "\n",
    "如果机器学习模型变得更加接近程序，那么它们很有可能会变得不再可微，当然这些程序仍然使用连续几何层次作为子模块，这些子模块是可微分的，但是整体的模型就不再可微。因此使用反向传播在一个固定硬编码的网络中作为调整权重值的方式将不再适用，至少反向传播算法不能代表整体算法。我们需要找到一种高效训练非可微系统的新算法。目前的解决方案包括遗传算法，“进化策略”，某些强化学习方法和ADMM（交替方向乘子法）。当然梯度下降不会小时，在优化可微分参数函数是梯度信息永远是有用的。但是我们的模型将会超越可微参数函数范围，因此它们的自动进化过程（也就是“机器学习”中的学习）会需要除了反向传播以外更多的方式。\n",
    "\n",
    "> Besides, backpropagation is end-to-end, which is a great thing for learning good\n",
    "chained transformations, but is rather computationally inefficient since it doesn’t fully\n",
    "leverage the modularity of deep networks. To make something more efficient, there is\n",
    "one universal recipe: introduce modularity and hierarchy. So we can make backprop\n",
    "itself more efficient by introducing decoupled training modules with some\n",
    "synchronization mechanism between them, organized in a hierarchical fashion. This\n",
    "strategy is somewhat reflected in DeepMind’s recent work on \"synthetic gradients\". I\n",
    "would expect more more work along these lines in the near future.\n",
    "\n",
    "除此之外，反向传播是端对端的，因此很适合学习串行的信息，但是它又是非常计算低效的因为并没有完全利用深度网络的模块化性质。要使得一件事物更加高效，有一条亘古不变的原则：引入模块化和层次化。因此我们能够通过在反向传播之中引入一些同步化机制来解耦这些训练模块，将它们组织成一个更加层次化的结构。这个策略最近被DeepMind应用到成果中，叫做“合成梯度”。我们能够期待不久之后就能出现更多相关的前沿研究。\n",
    "\n",
    "> One can imagine a future where models that would be globally non-differentiable (but\n",
    "would feature differentiable parts) would be trained—grown—using an efficient search\n",
    "process that would not leverage gradients, while the differentiable parts would be trained\n",
    "even faster by taking advantage of gradients using some more efficient version of\n",
    "backpropagation.\n",
    "\n",
    "你可以设想未来模型将会是总体非可微的（不过会含有可微的部分），它们将会使用一个高效的搜索过程而不用梯度来进行训练，并且其中可微的部分也可以应用更加高效的反向传播算法进行更加快速的训练。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 9.3.3 自动化机器学习\n",
    "\n",
    "> In the future, model architectures will be learned, rather than handcrafted by\n",
    "engineer-artisans. Learning architectures automatically goes hand in hand with the use of\n",
    "richer sets of primitives and program-like machine learning models.\n",
    "\n",
    "在未来，模型的结构是从学习得到的，而不是有工程师手工雕琢出来的。自动学习结构会随着机器学习模型中使用更加丰富的程序设计原语而愈加成熟。\n",
    "\n",
    "> Currently, most of the job of a deep learning engineer consists in munging data with\n",
    "Python scripts, then lengthily tuning the architecture and hyperparameters of a deep\n",
    "network to get a working model—or even, to get to a state-of-the-art model, if the\n",
    "engineer is so ambitious. Needless to say, that is not an optimal setup. But AI can help\n",
    "there too. Unfortunately, the data munging part is tough to automate, since it often\n",
    "requires domain knowledge as well as a clear high-level understanding of what the\n",
    "engineer wants to achieve. Hyperparameter tuning, however, is a simple search\n",
    "procedure, and we already know what the engineer wants to achieve in this case: it is\n",
    "defined by the loss function of the network being tuned. It is already common practice to\n",
    "set up basic \"AutoML\" systems that will take care of most of the model knob tuning. I\n",
    "even set up my own years ago to win Kaggle competitions.\n",
    "\n",
    "深度学习工程师当前最大部分的工作包括使用Python脚本处理消化数据，然后通过一个漫长的超参数调整过程来更改模型的结构以获得一个满意的模型，如果工程师有强烈的野心的话，甚至调整以获得一个最先进的模型。不用过多解释，这并不是一个让人舒适的设定。不过AI也可以在这里提供帮助。很不幸的是，处理消化数据部分很难自动化，因为这通常需要领域知识以及对要达成目标的清晰和高层次的理解。然而超参数调整就是一个简单的搜索过程，此时我们已经明确知道目标是什么了：也就是由网络定义的损失函数锁定的目标。目前普遍都有了使用基本“AutoML”系统来进行模型参数调整的共识。作者在几年前赢得Kaggle竞赛时就已经开始配置自己的AutoML了。\n",
    "\n",
    "> At the most basic level, such a system would simply tune the number of layers in a\n",
    "stack, their order, and the number of units or filters in each layer. This is commonly done\n",
    "with libraries such as Hyperopt, which we discussed in Chapter 7. But we can also be far\n",
    "more ambitious, and attempt to learn an appropriate architecture from scratch, with as\n",
    "few constraints as possible. This is possible via reinforcement learning, for instance, or\n",
    "genetic algorithms.\n",
    "\n",
    "在最基本层面上，这种系统只会简单的调整堆叠的层次数量，层次的顺序和每个层次的单元或者过滤器个数。这通常可以使用类似`Hyperopt`这样的库来实现，我们在第七章中已经介绍过。这里我们还可以更加激进一下，尝试从0开始学习一个合适的模型结构，使用尽量少的约束条件。这可以通过强化学习或者遗传算法实现。\n",
    "\n",
    "> Another important AutoML direction is to learn model architecture jointly with\n",
    "model weights. Because training a new model from scratch every time we try a slightly\n",
    "different architecture is tremendously inefficient, a truly powerful AutoML system would\n",
    "manage to evolve architectures at the same time as the features of the model are being\n",
    "tuned via backprop on the training data. Such approaches are already starting to emerge\n",
    "as I am writing these lines.\n",
    "\n",
    "另一个AutoML的重要方向是将模型结构与模型权重联合起来进行学习。因此每次从0开始训练一个新的模型会是非常低效的做法，因此一个真正强大的AutoML系统应该能够在模型通过训练数据反向传播进行权重更新时就调整模型的结构。这样的尝试已经在作者写作本书时就开始了。\n",
    "\n",
    "> When this starts happening, the jobs of machine learning engineers will not\n",
    "disappear—rather, engineers will move higher up the value creation chain. They will start\n",
    "putting a lot more effort into crafting complex loss functions that truly reflect business\n",
    "goals, and understanding deeply how their models impact the digital ecosystems in which\n",
    "they are deployed (e.g. the users that consume the model’s predictions and generate the\n",
    "model’s training data) —problems that currently only the largest company can afford to\n",
    "consider.\n",
    "\n",
    "当上面说的发生时，机器学习工程师的工作不会消失，相反工程师会走进价值链的更高层。他们将开始将更多的经历花费在制作更加复杂的损失函数上，这些复杂的损失能够真实反映商业目标，并且能够更加深入的理解他们的模型在数字生态系统中带来的影响（如那些使用模型预测值和生成模型训练数据的用户），这是目前仅限于最大型的公司才有足够资源考虑的问题。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 9.3.4 终身学习和模块化重用\n",
    "\n",
    "> If models get more complex and are built on top of richer algorithmic primitives, then\n",
    "this increased complexity will require higher reuse between tasks, rather than training a\n",
    "new model from scratch every time we have a new task or a new dataset. Indeed, a lot\n",
    "datasets would not contain enough information to develop a new complex model from\n",
    "scratch, and it will become necessary to leverage information coming from previously\n",
    "encountered datasets. Much like you don’t learn English from scratch every time you\n",
    "open a new book—that would be impossible. Besides, training models from scratch on\n",
    "every new task is very inefficient due to the large overlap between the current tasks and\n",
    "previously encountered tasks.\n",
    "\n",
    "如果模型变得更加复杂，并且构建在更加丰富的算法原语基础之上，那么这种不断增加的复杂度会需要任务之间具有更高的可重用性，而不是每次我们有了一个新任务或新的数据集就要从头开始训练一个新的模型。事实上很多的数据集不会包含足够的信息来从0开始发展成一个新的复杂模型，因此它需要利用那些之前碰到过的数据集。这就像你不会每次阅读一本新书的时候都重新从头学习英语一样，这是不可能的。除此之外，每次从0开始训练模型是非常低效率的，因此当前任务和前面遇到的任务会存在大量重叠的部分。\n",
    "\n",
    "> Additionally, a remarkable observation that has been made repeatedly in recent years\n",
    "is that training a same model to do several loosely connected tasks at the same time\n",
    "results in a model that is better at each task . For instance, training a same neural machine\n",
    "translation model to cover both English-to-German translation and French-to-Italian\n",
    "translation will result in a model that is better at each language pair. Training an image\n",
    "classification model jointly with an image segmentation model, sharing the same\n",
    "convolutional base, results in a model that is better at both tasks. And so on. This is fairly\n",
    "intuitive: there is always some information overlap between these seemingly\n",
    "disconnected tasks, and the joint model has thus access to a greater amount of\n",
    "information about each individual task than a model trained on that specific task only.\n",
    "\n",
    "还有一点是，最近几年的研究多次发现，在多个松散联系的任务上同时训练一个相同的模型能够比各自独立训练同一个模型的效果号。例如，在英语-德语翻译任务和法语-意大利语翻译任务上训练同一个神经机器翻译模型，会比在每一对语言上独立训练的性能要高。在共享同一个卷积网络的图像分类模型和图像切割模型上进行联合训练，会比独立训练这两个模型的性能要高，等等。这是很符合直觉的：这些看起来不相关的任务之间一定存在这一些信息的重叠，因此联合模型能够接触到大量这样的信息，而单独对特定任务进行训练将不会获得这种优势。\n",
    "\n",
    "> What we currently do along the lines of model reuse across tasks is to leverage\n",
    "pre-trained weights for models that perform common functions, like visual feature\n",
    "extraction. You saw this in action in Chapter 5. In the future, I would expect a\n",
    "generalized version of this to be commonplace: we would not only leverage previously\n",
    "learned features (submodel weights), but also model architectures and training\n",
    "procedures. As models become more like programs, we would start reusing program\n",
    "subroutines , like the functions and classes found in human programming languages.\n",
    "\n",
    "我们目前使用跨任务间模型重用的方向是利用这些模型在实现通用功能上的那些预训练权重，比方说视觉特征提取。你在第五章中已经学习过了。未来作者会预期这种做法会编程一个更加普遍的泛化版本：我们不仅会利用预训练特征（子模块权重），还会利用模型结构和训练过程。因为模型将会变得更像程序，我们会开始重用模型的子模块，就像使用人类编程语言中的函数和类那样。\n",
    "\n",
    "> Think of the process of software development today: once an engineer solves a\n",
    "specific problem (HTTP queries in Python, for instance), they will package it as an\n",
    "abstract and reusable library. Engineers that face a similar problem in the future can\n",
    "simply search for existing libraries, download one and use it in their own project. In a\n",
    "similar way, in the future, meta-learning systems will be able to assemble new programs\n",
    "by sifting through a global library of high-level reusable blocks. When the system would\n",
    "find itself developing similar program subroutines for several different tasks, if would\n",
    "come up with an \"abstract\", reusable version of the subroutine and would store it in the\n",
    "global library. Such a process would implement the capability for abstraction , a\n",
    "necessary component for achieving \"extreme generalization\": a subroutine that is found\n",
    "to be useful across different tasks and domains can be said to \"abstract\" some aspect of\n",
    "problem-solving. This definition of \"abstraction\" is similar to the notion of abstraction in\n",
    "software engineering. These subroutines could be either geometric (deep learning\n",
    "modules with pre-trained representations) or algorithmic (closer to the libraries that\n",
    "contemporary software engineers manipulate).\n",
    "\n",
    "试想一下我们当今的软件开发过程：一旦工程师解决了一个特定问题（比方说Python中的HTTP请求），他们会将这部分功能抽象出来并且打包成一个可重用的库。后面遇到同样问题的工程师就可以搜索已经存在的库，下载它并且在自己的项目中使用它。同样的，未来的元学习系统能够从一个包含高层次可重用模块的公共库中筛选出那些需要的模块，然和组合成一个新的程序。当系统能够开发出针对特定任务的程序子模块时，它就能获得一个“抽象的”可重用的的子模块版本并且把它保存到公共库中。这个过程就能实现抽象的能力，这是达到“极致泛化”的一个必须条件：一个能够针对不同任务和领域都适用的子模块，就可以被称为解决问题方面的某个“抽象”。这里“抽象”的定义就类似于软件工程中的“抽象”定义。这些子模块可能是几何式的（带预训练参数的深度学习模块），也可能是算法式的（接近于当前软件工程中应用的代码库）。\n",
    "\n",
    "![meta learning](imgs/f9.6.jpg)\n",
    "\n",
    "图9-6 一个元学习系统能够很快的使用可重用的原语（几何式或算法式）来构建特定任务的模型，从而实现“极致泛化”"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 9.3.5 小结：远景\n",
    "\n",
    "> In short, here is my long-term vision for machine learning:\n",
    "\n",
    "> - Models will be more like programs, and will have capabilities that go far beyond the\n",
    "continuous geometric transformations of the input data that we currently work with.\n",
    "These programs will arguably be much closer to the abstract mental models that humans\n",
    "maintain about their surroundings and themselves, and they will be capable of stronger\n",
    "generalization due to their rich algorithmic nature.\n",
    "- In particular, models will blend algorithmic modules providing formal reasoning, search,\n",
    "and abstraction capabilities, with geometric modules providing informal intuition and\n",
    "pattern recognition capabilities. AlphaGo (a system that required a lot of manual software\n",
    "engineering and human-made design decisions) provides an early example of what such a\n",
    "blend between symbolic and geometric AI could look like.\n",
    "- They will be grown automatically rather than hard-coded by human engineers, using\n",
    "modular parts stored in a global library of reusable subroutines—a library evolved by\n",
    "learning high-performing models on thousands of previous tasks and datasets. As\n",
    "frequent problem-solving patterns are identified by the meta-learning system, they would\n",
    "be turned into a reusable subroutine—much like functions and classes in software\n",
    "engineering—and added to the global library. This achieves the capability for abstraction\n",
    ".\n",
    "- This global library and associated model-growing system will be able to achieve some\n",
    "form of human-like \"extreme generalization\": given a new task, a new situation, the\n",
    "system would be able to assemble a new working model appropriate for the task using\n",
    "very little data, thanks to 1) rich program-like primitives that generalize well and 2)\n",
    "extensive experience with similar tasks. In the same way that humans can learn to play a\n",
    "complex new video game using very little play time because they have experience with\n",
    "many previous games, and because the models derived from this previous experience are\n",
    "abstract and program-like, rather than a basic mapping between stimuli and action.\n",
    "- As such, this perpetually learning model-growing system could be interpreted as an\n",
    "AGI—an Artificial General Intelligence. But don’t expect any singularitarian robot\n",
    "apocalypse to ensue: that’s a pure fantasy, coming from a long series of profound\n",
    "misunderstandings of both intelligence and technology. This critique, however, does not\n",
    "belong in this book.\n",
    "\n",
    "简而言之，作者对机器学习的远景看法如下：\n",
    "\n",
    "- 模型会变得更像程序，它的能力会远超目前对输入数据的连续几何变换。这样的程序会具争议性的更加接近人类能够保留他们环境和自身的抽象心智模型，并且这种程序基于其更加丰富的算法本质能够实现更为强大的泛化。\n",
    "- 确切来说，模型会在几何模块提供的非形式直觉和模式识别能力的基础上，混合提供形式推理、搜索和抽象能力。AlphaGo（这是一个需要大量人工软件工程和人为设计的决策的系统）提供了这样一个符号结合几何AI的早期方式。\n",
    "- 这样的模型会自动进化而不是由人类工程师硬编码的，它们通过一个集中存储的可重用的模块化库来实现这一点，这是一个通过之前成千上万个任务和数据集训练演化得到的一个高性能的模型库。随着这些常见问题解决模式被元学习系统发现和使用，它们就自动被转换成了可重用的子模块，就像我们现在在软件工程中使用的函数和类一样，元学习系统会将成熟的子模块加入到公共库中。这就实现了抽象的能力。\n",
    "- 这个公共库和相关的模型进化系统能够在某种程度上实现类人的“极致泛化”能力：当遇到一个新的任务或新的环境时，系统能够组装一个新的模型，仅使用少量的数据就能在新的任务上具有良好性能。这是由于1）具有丰富的程序原语使得泛化良好，2）对相似任务已经具有丰富的经验。这就如同人类能够花费少量时间的情况下就学习玩一个电子游戏一样，因为他们以前就已经积累了玩游戏的经验了，并且这些经验形成的模型是抽象和程序化的，而不是简单的从刺激到反应的映射。\n",
    "- 因此这样的永久学习模型系统可以称为AGI（人工通用智能）。但是不要错误的认为这是机器人天启的奇点：这纯属科幻，由长期以来对智能和科技的误解形成的错误观点。与这个辩论的内容超出了本书的范畴。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 9.4 在这个高速发展领域与时俱进\n",
    "\n",
    "> As final parting words, I would like to give you some pointers on how to keep learning\n",
    "and updating your knowledge and skills after you’ve turned the last page. The field of\n",
    "modern deep learning, as we know it today, is only a few years old—despite a long, slow\n",
    "prehistory stretching back decades. With an exponential increase in financial resources\n",
    "and research headcount since 2013, the field as a whole has been moving at a frenetic\n",
    "pace. You cannot hope that what you’ve learned in this book will forever stay relevant, or\n",
    "that it will be all that will need for the rest of your career.\n",
    "\n",
    "作为最后的部分，当你阅读完本书后，作者会给出一些关于如何继续学习和更新你的知识和技术观点。我们今天知道的现代深度学习领域，只有数年的历史，尽管它的前身可以追溯到几十年前。随着2013年后资本和研究人力资源的指数化增长，整个领域都处于一种狂热的演进当中。你不能寄希望与本书介绍的内容永远保持先进，或者认为这就是你的事业后续所有需要获取的知识。\n",
    "\n",
    "> Thankfully, there are plenty of free online resources that you can leverage to stay up\n",
    "to date and expand your horizons. Here are a few.\n",
    "\n",
    "幸运的是网上有很多的免费资源，你可以用来保持知识更新和拓展视野。下面介绍其中的一些。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 9.4.1 在Kaggle上使用真实世界数据来练手\n",
    "\n",
    "> One very effective way to acquire real-world experience is to try your hand at machine\n",
    "learning competitions on kaggle.com . The only real way to learn is through practice and\n",
    "actual coding—that’s the philosophy of this book, and Kaggle competitions are the\n",
    "natural continuation of this. On Kaggle, you will find an array of constantly renewed data\n",
    "science competitions, a lot of them involving deep learning, prepare by companies\n",
    "interested in obtaining novel solutions to some of their most challenging machine\n",
    "learning problems. There are fairly large money prizes offered to top entrants.\n",
    "\n",
    "获取真实时间经验的一个非常有效的方式是在[kaggel.com](https://kaggle.com)的竞赛中练习使用机器学习。这是通过实践和编码来学习的最好方式，本书的写作哲学也是如此，而在Kaggle竞赛中时间就是这种哲学的延续。在Kaggle上，你会持续看到不断更新的数据科学竞赛，其中的许多都设计深度学习，它们是那些希望通过竞赛方式获得其具有挑战性机器学习问题优秀解决方案的公司提供的。这些竞赛对排名前列的参赛者都提供了不菲的奖金。\n",
    "\n",
    "> Most competitions are won using either the XGBoost library (for shallow machine\n",
    "learning) or Keras (for deep learning). So you will fit right in! By participating in a few\n",
    "competitions, maybe as part of a team, you will become more familiar with the practical\n",
    "side of some of the advanced best practices we have described in this book, especially\n",
    "hyperparameter tuning, avoiding validation set overfitting, and model ensembling.\n",
    "\n",
    "其中很多的竞赛的获奖者都使用了XGBoost库（浅机器学习领域）或者Keras（深度学习领域）。因此你已经具备了相关的能力。通过参与一些竞赛或者作为团队成员参赛，你会更加熟悉本书中介绍的高级最佳实践的知识，特别是超参数调整，避免验证集过拟合和模型组装。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 9.4.2 在Arxiv上阅读最新进展\n",
    "\n",
    "> Deep learning research, in contrast with some other scientific fields, takes places\n",
    "completely in the open. Papers are made publicly and freely accessible as soon as they\n",
    "are finalized, and a lot of related software gets open-sourced. arxiv.org (pronounced\n",
    "\"archive\"—that x stands for a greek chi ) is an open-access preprint server for Physics,\n",
    "Mathematics, and Computer Science research papers. It has become the de-facto way to\n",
    "stay up-to-date on the bleeding edge of machine learning and deep learning. The large\n",
    "majority of deep learning researchers will upload any paper they write to Arxiv shortly\n",
    "after completion. This allows them to \"plant a flag\" and claim a specific finding without\n",
    "waiting for a conference acceptance (which takes half a year), which is necessary given\n",
    "the very fast pace of research and the intense competition in the field. It also allows the\n",
    "field to move extremely fast: all new findings are immediately available for all to see and\n",
    "to build upon.\n",
    "\n",
    "深度学习研究，不像其他科学领域那样封闭，它是全开放的。论文写就之后就会公开并且提供免费获取，其中的大部分相关软件都会开源。[arxiv.org](https://arxiv.org)（读作“archive”，其中的x代表希腊的chi）是一个提供免费获取物理学、数学和计算机科学研究论文的网站。它已经成为保持站在机器学习和深度学习研究前沿的事实标准。大量的深度学习研究者会在完成论文后第一时间将其提交到Arxiv。这能够让他们首先“立一个旗帜”，在不需要获得会议论坛接受（可能需要半年左右时间）的情况下就宣称某个成果，在这个高速发展和剧烈竞争的研究领域，这是非常必需的。这种做法同样反哺了整个领域：使得发展更加迅速，因为所有新的发现都立即可以被所有人看到，从而在此基础上继续研究。\n",
    "\n",
    "> An important downside is that the sheer quantity of new papers getting posted every\n",
    "day on Arxiv makes it impossible to even skim through them all, and the fact that they\n",
    "are not peer-reviewed makes it difficult to identify those that are both important and\n",
    "high-quality. It is difficult, and getting increasingly more difficult, to find the signal in\n",
    "the noise. Currently, there’s isn’t a good solution to this problem. But some tools can\n",
    "help: an auxiliary website called arxiv-sanity.com can serve a recommendation engine\n",
    "for new papers and can help you keep track of new developments within a specific\n",
    "narrow vertical of deep learning. Additionally, you can use Google Scholar to keep track\n",
    "of the publications of your favorite authors.\n",
    "\n",
    "但是这也带来了一个很大的缺点，那就是每天在Arxiv上发表的新论文数量是如此之多，使得哪怕是匆匆浏览所有的论文也变得不可能，而且因为这个网站上并不存在真正的同行评议，导致无法从中筛选出那些重要和高质量的论文。从噪音中找到信号是非常困难的，而且正在变得越来越困难。目前针对这个问题不存在好的解决方案。不过有以下工具能提供一点帮助：有一个辅助网站[arxiv-sanity.com](https://arxiv-sanity.com)可以作为一个新论文的推荐引擎，可以帮助你在一个较小范围内获得深度学习相关的新进展。并且你可以使用[谷歌学术](https://scholar.google.com/)来追踪那些你喜爱的作者发表的新论文。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 9.4.3 探索Keras的生态\n",
    "\n",
    "> With about 150,000 users as of June 2017, and fast growing, Keras has a large ecosystem\n",
    "of tutorials, guides and related open-source projects.\n",
    "\n",
    "> - Your main reference for working with Keras itself is the online documentation at\n",
    "keras.io .\n",
    "- The Keras source code can be found at github.com/fchollet/keras .\n",
    "- You can ask for help and join deep learning discussions on the Keras Slack channel,\n",
    "kerasteam.slack.com .\n",
    "- The Keras blog, blog.keras.io , offers Keras tutorials and other deep learning-related\n",
    "articles.\n",
    "\n",
    "截止2017年六月，Keras拥有15万的用户，并且在告诉发展，它有着一个庞大的生态，包括教程、指引和相关的开源项目。\n",
    "\n",
    "- 使用Keras的时候你需要的最主要的参考是在[keras.io](http://keras.io)网站上的在线文档。\n",
    "- Keras的源代码可以在[Github仓库](https://github.com/fchollet/keras)上找到。\n",
    "- 你可以在Keras的Slack channel上提问和参与深度学习的讨论。[Slack channel地址](http://kerasteam.slack.com)。\n",
    "- [Keras的博客](https://blog.keras.io)提供了Keras的教程和其他深度学习相关文章。\n",
    "\n",
    "> Also, you can follow me on Twitter: @fchollet .\n",
    "\n",
    "还有你也能在推特上follow作者：@fchollet。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 9.5 结束语\n",
    "\n",
    "> That’s the end of Deep Learning with Python! I hope you have learned a thing or two\n",
    "about machine learning, deep learning, Keras, and maybe even cognition in general.\n",
    "Learning is a lifelong journey, especially in the field of AI, where we have far more\n",
    "unknowns on our hands than certitudes. So please go on learning, questioning, and\n",
    "researching. Never stop. Because for all the progress made so far, it seems like most of\n",
    "the fundamental questions in AI remain unanswered. Many have not even been properly\n",
    "asked yet.\n",
    "\n",
    "终于到了本书的结束了。作者希望你通过本书学习到了一些有关机器学习、深度学习、Keras、甚至是通用认知方面的知识。学习是终身旅程，尤其是在AI这个领域，未知远大于我们的已知。因此请继续学习、提问和研究。不要停下你的脚步。因为对于目前取得的所有进展而言，AI中大多数的基础问题仍未得到解决。甚至还有很多问题还没有被正确的提出。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<< [第八章：生成模型深度学习](Chapter8_Generative_deep_learning.ipynb) || [目录](index.md)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.7.6 64-bit ('base': conda)",
   "language": "python",
   "name": "python37664bitbaseconda70fb04b0bd9543d0a4d5588de79b26c5"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}