Chapter3_Getting_started_with_neural_networks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<< [第二章：开始之前：神经网络的数学知识](Chapter2_Mathematical_blocks_of_neural_networks.ipynb) || [目录](index.md) || [第四章：机器学习基础](Chapter4_Fundamentals_of_machine_learning.ipynb) >>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 第三章：进入神经网络\n",
    "\n",
    "> This chapter is designed to get you started with using neural networks to solve real\n",
    "problems. You will consolidate the knowledge you gained from our very first practical\n",
    "example, and you apply what you have learned to three new problems covering the three\n",
    "most common use cases of neural networks: binary classification, multi-class\n",
    "classification, and scalar regression.\n",
    "\n",
    "本章的主要目标是帮助你开始使用神经网络解决实际问题。你可以从中巩固我们前面上一章例子中学习到的知识，并且将学习到的内容应用到三个新的问题当中，这些问题涵盖了神经网络三个最常用的应用场景：二元分类，多类别分类和标量回归。\n",
    "\n",
    "> In this chapter, you will:\n",
    "\n",
    "> - Take a closer look at the core components of neural networks we introduced in our first\n",
    "example: layers, networks, objective functions and optimizers.\n",
    "- Get a quick introduction to Keras, the Python deep learning library which we will use\n",
    "throughout the book.\n",
    "- Set up a workstation for deep learning, with TensorFlow, Theano, Keras, and GPU\n",
    "support.\n",
    "- Dive into three introductory examples of how to use neural networks to solve real\n",
    "problems:\n",
    "    - classifying movie reviews into positive and negative ones (binary classification).\n",
    "    - classifying news wires by their topic (multi-class classification).\n",
    "    - estimating the price of a house given real estate data (regression).\n",
    "\n",
    "在本章中，你将会：\n",
    "\n",
    "- 近距离的观察我们在第一个例子中用到的神经网络的核心组件：层、网络、目标函数和优化器。\n",
    "- 获得Keras的快速入门介绍，它是贯穿本书我们用的的Python深度学习库。\n",
    "- 搭建一个深度学习工作站，使用TensorFlow、Theano、Keras和GPU支持。\n",
    "- 深入三个入门级的例子，介绍神经网络如何解决实际问题：\n",
    "    - 将影评分为正向和负向（二元分类）。\n",
    "    - 将新闻归类到它们的主题中（多类别分类）。\n",
    "    - 从给定的房价数据中预测房子的价格（回归）。\n",
    "\n",
    "> By the end of this chapter, you will already be able to use neural networks to solve\n",
    "simple machine problems such as classification or regression over vector data. You will\n",
    "then be ready to start building a more principled and theory-driven understanding of\n",
    "machine learning, in the next chapter.\n",
    "\n",
    "学习完本章内容后，你将会具备使用神经网络解决一些简单问题的能力，比方说在矢量数据上进行分类或回归。为下一章开始构建更主要的、更理论化的理解机器学习做好准备。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.1 神经网络剖析\n",
    "\n",
    "> As we saw in the previous chapters, training a neural network revolves around the\n",
    "following objects:\n",
    "\n",
    "> - Layers, which are combined into a network (or model ).\n",
    "- The input data and corresponding targets .\n",
    "- The loss function , which defines the feedback signal which is used for learning.\n",
    "- The optimizer , which determines how the learning proceeds.\n",
    "\n",
    "上一章我们看到，训练一个神经网络涉及下面这些对象：\n",
    "\n",
    "- 层，多个层可以组成一个网络（或模型）。\n",
    "- 输入数据和对应的目标值。\n",
    "- 损失函数，它定义进行学习的反馈信号。\n",
    "- 优化器，它指导学习发展的进程。\n",
    "\n",
    "> You can visualize their interaction in the following way: the network , composed of layers chained together, maps the input data into predictions . The loss function then\n",
    "compares these predictions to the targets , producing a loss value, a measure how well the\n",
    "predictions of the network match what was expected. The optimizer uses this loss value\n",
    "to update the weights of the network.\n",
    "\n",
    "你可以将上面几个对象之间的相互作用可视化出来：网络，由多个层串联构成，将输入数据映射到预测值。然后损失函数比较这些预测值和目标值，产生一个损失值，这是一个衡量网络预测能力的指标。优化器使用这个损失值来更新网络的权重。\n",
    "\n",
    "![network layers loss func optimizer](imgs/f3.1.jpg)\n",
    "\n",
    "图3-1 网络、层、损失函数和优化器之间的联系\n",
    "\n",
    "> Let’s take a closer look at layers, networks, loss functions and optimizers.\n",
    "\n",
    "本节我们详细的研究层、网络、损失函数和优化器。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1.1 层：深度学习的乐高积木\n",
    "\n",
    "> The fundamental data structure in neural networks is the \"layer\", to which you have\n",
    "already been introduced in the previous chapter. A layer is a data-processing module that\n",
    "takes as input one or more tensors, and outputs one or more tensors. Some layers are\n",
    "stateless, but more frequently layers have a state: the layer’s \"weights\", one or several\n",
    "tensors learned with stochastic gradient descent, and which together contain the\n",
    "\"knowledge\" of the network.\n",
    "\n",
    "神经网络中最基本的数据结构就是“层”，在上一章中我们已经初步接触过它。层是一个数据处理模块，它接受一个或多个张量作为输入，然后输出一个或多个张量。有些层是无状态的，但更普遍的情况是层是有状态的：层的“权重”，它们是一个或数个使用随机梯度下降学习到的张量，其中含有网络的“知识“。\n",
    "\n",
    "> Different layers are appropriate for different tensor formats and different types of data\n",
    "processing. For instance, simple vector data, stored in 2D tensors of shape (samples,\n",
    "features) , is often processed by \"fully-connected\" layers, also called\n",
    "\"densely-connected\" or \"dense\" layers (the Dense class in Keras). Sequence data, stored\n",
    "in 3D tensors of shape (samples, timesteps, features) , is typically processed by\n",
    "\"recurrent\" layers such as a LSTM layer. Image data, stored in 4D tensors, is usually\n",
    "processed by 2D convolution layers ( Conv2D ).\n",
    "\n",
    "不同的层适合不同的张量格式和不同类型的数据处理。例如简单的矢量数据，存储在形状为(样本, 特征)的2D张量当中，通常用“全连接”层进行处理，也被称为“紧密连接”或“紧密”层（在Keras中是Dense类）。序列数据，存储在形状为(样本, 序列号, 特征)的3D张量中，通常使用“循环”层如LSTM层进行处理。图像数据，存储在4D张量中，通常使用2D卷积层（Conv2D）进行处理。\n",
    "\n",
    "> You can think of layers as the Lego bricks of deep learning, a metaphor which is\n",
    "made explicit by frameworks like Keras. Building deep learning models in Keras is done\n",
    "by clipping together compatible layers to form useful data transformation pipelines. The\n",
    "notion of \"layer compatibility\" here refers specifically to the fact that every layer will\n",
    "only accept input tensors of a certain shape, and will return output tensors of a certain\n",
    "shape. Consider the following example:\n",
    "\n",
    "你可以将层想象成深度学习的乐高积木，这当然是一个比喻，但是类似Keras这样的框架把这个比喻具体化了。在Keras中构建深度学习模型就是通过将兼容的层次堆叠在一起形成有意义的数据转换管道完成的。这里的“层兼容性”具体的含义就是每个层只能接受某种特定形状的输入张量，并且输出某种特定形状的张量。考虑下面的例子："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras import layers\n",
    "\n",
    "# 一个带有32个输入单元的全连接层\n",
    "layer = layers.Dense(32, input_shape=(784,))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> We are creating a layer that will only accept as input 2D tensors where the first\n",
    "dimension is 784 (the zero-th dimension, the batch dimension, is unspecified and thus\n",
    "any value would be accepted). And this layer will return a tensor where the first\n",
    "dimension has been transformed to be 32 :\n",
    "\n",
    "我们上面创建了一个仅接受2D张量输入的层，输入的张量的第一个维度是784（第一个轴本来应该是批次轴，在这里没有设置，因此能够接受任何值）。这个层会输出一个张量，其第一个维度会被转换为32："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(None, 32)"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from tensorflow.keras.models import Sequential\n",
    "\n",
    "model = Sequential()\n",
    "model.add(layer)\n",
    "layer.output_shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Thus this layer can only be connected to an upstream that expects 32-dimensional\n",
    "vectors as its input. When using Keras you don’t have to worry about compatibility,\n",
    "because the layers that you add to your models are dynamically built to match the shape\n",
    "of the incoming layer. For instance, if you write the following:\n",
    "\n",
    "因此这个层只能够后续连接一个接收32维矢量输入的层。当使用Keras时，你不需要担心兼容性问题，因为加入网络的层会自动匹配上游输出的形状然后进行动态构建，例如下面的代码："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "model.add(layers.Dense(32, input_shape=(784,)))\n",
    "model.add(layers.Dense(32))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> The second layer did not receive an input shape argument—instead it automatically\n",
    "inferred its input shape as being the output shape of the layer that came before.\n",
    "\n",
    "第二层没有指定输入张量形状参数，因为它根据上面层次的输出形状自动推断出了输入形状。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1.2 模型：层的网络结构\n",
    "\n",
    "> A deep learning model is simply a directed acyclic graph of layers. The most common\n",
    "instance would be a linear stack of layers, mapping a single input to a single output.\n",
    "\n",
    "一个深度学习模型简单来说就是一个有向无环图（DAG）的层网络。其中最常见的就是一个线性堆叠的层栈，将一个输入映射成一个输出。\n",
    "\n",
    "> However, as you move forward, you will be exposed to a much broader variety of\n",
    "network topologies. Some common ones include:\n",
    "\n",
    "> - Two-branch networks\n",
    "- Multi-head networks\n",
    "- Inception blocks\n",
    "\n",
    "不过随着你慢慢深入之后，会碰到一些更加复杂的网络拓扑。其中常见的包括：\n",
    "\n",
    "- 双分支网络\n",
    "- 多头网络\n",
    "- 认知结构块\n",
    "\n",
    "> The topology of a network defines an hypothesis space . You may remember that in\n",
    "chapter one, we defined machine learning as \"searching for useful representations of\n",
    "some input data, within a pre-defined space of possibilities, using guidance from some\n",
    "feedback signal\". By choosing a network topology, you have constrained your \"space of\n",
    "possibilities\" (hypothesis space) to a specific series of tensor operations, mapping input data to output data. What you will then be \"searching\" for, is a good set of values for the\n",
    "weight tensors involved in these tensor operations.\n",
    "\n",
    "网络的拓扑定义了一个假设空间。你应该还记得我们在第一章中把机器学习定义为“对一些预设概率空间的输入数据，通过反馈信号指引，寻找有用的表现形式”。在选择网络拓扑时，你就已经限制了你的“概率空间”（假设空间），使得它只能应用特定的一些张量操作，将输入数据映射到输出数据。最终你将“找到”涉及假设空间及相应张量操作的一组较理想的值，拿来作为权重张量。\n",
    "\n",
    "> Picking the right network architecture is more an art than a science, and while there\n",
    "are some best practices and principles you can rely on, only practice can really help you\n",
    "become a proper neural network architect. The next few chapters will both teach you\n",
    "explicit principles for building neural networks, and will help you develop intuition as to\n",
    "what works or doesn’t work for specific problems.\n",
    "\n",
    "选择正确的网络结构更像是一门艺术而不是科学，而且虽然有一些最佳实践和原则你可以遵循，但是只有时间才能真正帮助你成为一个合格的神经网络架构师。下面一些章节既会告诉你关于构建神经网络的明确原则，也会帮助你建立对于特定问题选择较好方案的直觉。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1.3 损失函数和优化器：控制学习过程的关键\n",
    "\n",
    "> Once the network architecture is defined, we still have to pick two more things:\n",
    "\n",
    "> - The loss function (or objective function), the quantity that will be minimized during\n",
    "training. It represents a measure of success on the task at hand.\n",
    "- The optimizer, which determines how the network will be updated based on the loss\n",
    "function. It implements a specific variant of stochastic gradient descent.\n",
    "\n",
    "当网络结构定义好了之后，我们还需要再选择两样东西：\n",
    "\n",
    "- 损失函数（或者叫目标函数），这是一个会在学习过程中试图最小化的数值。它代表着当前任务成功程度的度量。\n",
    "- 优化器，它决定网络会如何依据损失函数更新它的参数。一般采用随机梯度下降方法的一个特定变体来实现。\n",
    "\n",
    "> A neural network that has multiple outputs may have multiple loss functions (one per\n",
    "output), however the gradient descent process must be based on a single scalar loss value,\n",
    "so what happens for multi-loss networks is that all losses are combined (via averaging)\n",
    "into a single scalar quantity.\n",
    "\n",
    "一个有多个输出的神经网络也可以有多个损失函数（每个输出对应一个），然而梯度下降过程必须基于一个单独的标量损失值，因此具有多个损失函数的网络会将全部的损失组合（通过平均）成一个单独的标量值。\n",
    "\n",
    "> Picking the right objective function for the right problem is extremely important: your\n",
    "network will take any shortcut it can to minimize it, so if the objective doesn’t fully\n",
    "correlate with actual success on the task at hand, your network will end up doing things\n",
    "you may not have wanted. Imagine a stupid omnipotent AI trained via stochastic gradient\n",
    "descent, with the poorly-chosen objective function of \"maximizing the average\n",
    "well-being of all humans alive\". To make its job easier, this AI might choose to kill all\n",
    "humans except a few, and focus on the well-being on the remaining ones—since average\n",
    "well-being is not affected by how many humans are left. That might not be what you\n",
    "intended! Just remember that all neural networks you build will be just as ruthless in\n",
    "lowering their loss function—so choose the objective wisely.\n",
    "\n",
    "选择正确的目标函数对于解决问题是至关重要的：你的网络会尝试任何途径去最小化它，因此如果目标不能完全与手头任务的成功关联，网络可能会停止在你不希望的状态上。我们可以设想这样一个场景，通过随机梯度下降生成了一个的愚昧却无所不能的人工智能，给它设定的目标函数是“最大化所有人类的平均幸福水平”。要使得这个任务更简单，这个AI可能会选择杀死大部分人类，只留下哪些幸福水平高的小部分人，因为平均幸福水平不会在意还剩下多少人类。这肯定不是你期望的结果！牢记你构建的神经网络唯一会做的事情就是粗暴地降低它们的损失函数。因此请明智的选择目标。\n",
    "\n",
    "> Thankfully, when it comes to common problems such as classification, regression, or\n",
    "sequence predictions, there are simple guidelines that you can follow to choose the right\n",
    "loss: for instance, you will use binary crossentropy for a two-class classification problem,\n",
    "categorical crossentropy for a many-class classification problem, mean squared error for\n",
    "a regression problem, CTC for a sequence learning problem... only when you are\n",
    "working on truly new research problems will you have to develop your own objective\n",
    "functions.\n",
    "\n",
    "所幸的是，常见的问题诸如分类、回归或者序列预测都有着简单的指引，你只需要遵循它们来选择正确的损失：例如，在二元分类问题中使用二分交叉熵，在多类别分类问题中使用类别交叉熵，在序列学习问题中使用连接时间分类CTC...只有当你在研究真正的全新问题时才需要开发你自己的目标函数。\n",
    "\n",
    "> In the next few chapters, we will detail explicitly which loss functions to pick, for a\n",
    "wide range of common tasks.\n",
    "\n",
    "后续章节中我们会详细介绍对于许多常见任务的损失函数的选择。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.2 Keras介绍\n",
    "\n",
    "> Throughout this book, all of our code examples use Keras. Keras is a deep learning\n",
    "framework for Python which provides a convenient way to define and train almost any\n",
    "kind of deep learning model. Keras was initially developed for researchers, aiming at\n",
    "enabling fast experimentation.\n",
    "\n",
    "贯穿本书所有的代码例子都是使用Keras的。Keras是一个Python深度学习框架，它提供了一个简便的方式来定义和训练几乎任何类型的深度学习模型。Keras最初是为研究人员开发的，目标是让快速实验变为可能。\n",
    "\n",
    "> Keras has the following key features:\n",
    "\n",
    "> - It allows the same code to run on CPU or on GPU, seamlessly.\n",
    "- It has a user-friendly API which makes it easy to quickly prototype deep learning models.\n",
    "- It has build-in support for convolutional networks (for computer vision), recurrent\n",
    "networks (for sequence processing), and any combination of both.\n",
    "- It supports arbitrary network architectures: multi-input or multi-output models, layer\n",
    "sharing, model sharing, etc. This means that Keras is appropriate for building essentially\n",
    "any deep learning model, from a generative adversarial network to a neural Turing\n",
    "machine.\n",
    "\n",
    "Keras具有以下的一些关键特性：\n",
    "\n",
    "- 它允许无缝的使用相同的代码在CPU和GPU之间切换运行。\n",
    "- 它有一套用户友好的API，能够简单快捷的建立深度学习模型原型。\n",
    "- 它有着內建的卷积网络（提供计算机视觉使用），循环网络（提供序列处理使用），以及两者的任意组合。\n",
    "- 它支持任意的网络结构：多输入或多输出模型，共享层，共享模型等。这意味着Keras适合用来构建几乎任何深度学习模型，从生成对抗网络到神经图灵机。\n",
    "\n",
    "> Keras is distributed under the permissive MIT license, which means that it can be\n",
    "freely used in commercial projects. It is compatible with any version of Python from 2.7\n",
    "to 3.6 (as of mid-2017). Its documentation is available at keras.io .\n",
    "\n",
    "Keras遵循宽松的MIT协议发布，意味着它能自由的在商业项目中使用。它也能兼容任何Python版本，从2.7到3.6(截止2017年中）。可以访问[keras官网](https://keras.io)获取它的文档。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Keras has well over a hundred of thousands of users, ranging from academic\n",
    "researchers and engineers at both startups and large companies, to graduate students and\n",
    "even hobbyists. Keras is used at Google, Netflix, Uber, CERN, Yelp, and at hundreds of\n",
    "startups working on a wide range of problems. Keras is also very a popular framework on\n",
    "Kaggle, the machine learning competition website, where almost every recent deep\n",
    "learning competition has been won using Keras models.\n",
    "\n",
    "Keras有远超上万的用户，范围涵盖从学术研究者和工程师，无论是初创企业还是大企业，到本科生和甚至技术爱好者。Keras在Google、Netflix、Uber、CERN、Yelp以及成百上千的初创企业中被使用，运用到非常广泛的问题上。Keras也是机器学习竞赛网站Kaggle上非常流行的框架，最近基本每一次的深度学习竞赛都有获奖者使用Keras模型。\n",
    "\n",
    "![DNN trend](imgs/f3.2.jpg)\n",
    "\n",
    "图3-2 Google搜索不同的深度学习框架趋势"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2.1 Keras，TensorFlow，Theano和CNTK\n",
    "\n",
    "> Keras is a model-level library, providing high-level building blocks for developing deep\n",
    "learning models. It does not handle itself low-level operations such as tensor\n",
    "manipulation and differentiation. Instead, it relies on a specialized, well-optimized tensor\n",
    "library to do so, serving as the \"backend engine\" of Keras. Rather than picking one single\n",
    "tensor library and making the implementation of Keras tied to that library, Keras handles\n",
    "the problem in a modular way, and several different backend engines can be plugged\n",
    "seamlessly into Keras. Currently, the three existing backend implementations are the\n",
    "TensorFlow backend, the Theano backend, and the CNTK backend. In the future, it is\n",
    "likely that Keras will be extended to work with even more deep learning execution\n",
    "engines.\n",
    "\n",
    "Keras是一个模型级的库，为开发深度学习模型提供高层的构建组件。它自身并不处理底层的运算，比如张量操作和微分。在底层它依赖一个特定的优化张量运算库来完成这些工作，这个库被称为Keras的“后端引擎”。但是Keras并没有将自身绑定在某一个特定的后端引擎上来实现，它采用了一种模块化的方式来解决这个问题，因此多个不同的后端引擎都能够无缝的对接到Keras上。目前，有三种后端引擎实现，分别是TensorFlow后端、Theano后端和CNTK后端。未来Keras还可能扩展到能支持更多的深度学习执行引擎上。\n",
    "\n",
    "> TensorFlow, CNTK, and Theano are some of the main platforms for deep learning\n",
    "today. Theano is developed by the MILA lab at Université de Montréal , while\n",
    "TensorFlow is developed by Google, and CNTK is developed by Microsoft. Any piece of\n",
    "code that you write with Keras can be run with any of these backends without having to\n",
    "change anything to the code: you can seamlessly switch between the two during\n",
    "development, which often proves useful, for instance if one of these backends proves to\n",
    "be faster for a specific task. By default, I would recommend using the TensorFlow\n",
    "backend for most of your deep learning needs.\n",
    "\n",
    "TensorFlow、CNTK和Theano是今天主要的深度学习平台。Theano是蒙特利尔大学MILA实验室开发的，TensorFlow是Google开发的，CNTK是微软开发的。你在Keras上写的任何一段代码都可以在上述三个后端中运行，而不需要修改任何代码：你可以在开发过程中无缝的切换后端，这种情况有时非常有用，例如在其中一个后端被证明在某个特性任务上比较快的情况下。默认情况下，作者建议在大多数深度学习任务中使用TensorFlow后端。\n",
    "\n",
    "> Via TensorFlow (or Theano, or CNTK), Keras is able to run on both CPU and GPU\n",
    "seamlessly. When running on CPU, TensorFlow is itself wrapping a low-level library for\n",
    "tensor operations called Eigen. On GPU, TensorFlow wraps a library of well-optimized\n",
    "deep learning operations called cuDNN, developed by NVIDIA.\n",
    "\n",
    "通过TensorFlow（或Theano或CNTK），Keras能够无缝的在CPU和GPU之间切换运行。当运行在CPU下时，TensorFlow自身封装了一个底层的张量运算库，称为Eigen。在GPU情况下，TensorFlow封装了一个非常优化的深度学习运算库，NVIDIA开发的cuDNN。\n",
    "\n",
    "![Keras tech stack](imgs/f3.3.jpg)\n",
    "\n",
    "图3-3 深度学习软件和硬件技术栈"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2.2 使用Keras开发：快速概述\n",
    "\n",
    "> You’ve already seen one example of a Keras model: our MNIST example. The typical\n",
    "Keras workflow looks just like our example:\n",
    "\n",
    "> - Define your training data: input tensors and target tensors.\n",
    "- Define a network of layers (or model ) that maps your inputs to your targets.\n",
    "- Configure the learning process by picking a loss function, an optimizer, and some metrics\n",
    "to monitor.\n",
    "- Iterate on your training data by calling the fit method of your model.\n",
    "\n",
    "前面你已经看到一个Keras模型的例子：MNIST。典型的Keras流程就像我们的例子那样：\n",
    "\n",
    "- 定义训练数据：输入张量和目标张量。\n",
    "- 定义层组成的网络（或模型），将输入映射成目标。\n",
    "- 通过选择损失函数、优化器和一些监控指标来配置学习过程。\n",
    "- 在你的模型上调用fit方法迭代训练你的数据。\n",
    "\n",
    "> There are two ways to define a model: using the Sequential class (only for linear\n",
    "stacks of layers, which is the most common network architecture by far), and the\n",
    "\"functional API\" (for directed acyclic graphs of layers, allowing to build completely\n",
    "arbitrary architectures).\n",
    "\n",
    "有两种方法定义一个模型：使用Sequential类（仅提供线性层堆叠，也是目前为止最常见的网络结构），和“函数式API”（提供有向无环图结构，允许构建完全任意的结构）。\n",
    "\n",
    "> As a refresher, here’s a two-layer model defined using the Sequential class (note\n",
    "that we are passing the expected shape of the input data to the first layer):\n",
    "\n",
    "作为复习，下面是我们上一章介绍使用Sequential构建的两层模型（注意我们在第一层中传递了期望的输入张量形状）："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras import models\n",
    "from tensorflow.keras import layers\n",
    "\n",
    "model = models.Sequential()\n",
    "model.add(layers.Dense(32, activation='relu', input_shape=(784,)))\n",
    "model.add(layers.Dense(10, activation='softmax'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> And here’s the same model defined using the functional API. With this API, you are\n",
    "manipulating the data tensor that the model processes, and applying layers to this tensor\n",
    "as if they were functions. A detailed guide to what you can with the functional API can\n",
    "be found in Chapter 7. Until Chapter 6, we will only be using the Sequential class in\n",
    "our code examples.\n",
    "\n",
    "下面是使用函数式API定义相同模型的方法。使用这些API，你控制的是模型处理的数据张量，然后将层应用到这些张量上，就好像它们是函数一样。在第七章中会有关于函数式API的详细指引。在那之前，我们都只会使用Sequential类。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_tensor = layers.Input(shape=(784,))\n",
    "x = layers.Dense(32, activation='relu')(input_tensor)\n",
    "output_tensor = layers.Dense(10, activation='softmax')(x)\n",
    "\n",
    "model = models.Model(inputs=input_tensor, outputs=output_tensor)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Once your model architecture is defined, it doesn’t matter whether you used a\n",
    "Sequential model or the functional API: all following steps are the same.\n",
    "\n",
    "一旦你的模型结构定义好了，无论你使用的是Sequential模型还是函数式API：后续的步骤都是相同的。\n",
    "\n",
    "> The learning process is configured at the \"compilation\" step, where you specify the\n",
    "optimizer and loss function(s) that the model should use, as well as the metrics you want\n",
    "to monitor during training. Here’s an example with a single loss function, by far the most\n",
    "common case:\n",
    "\n",
    "学习过程是在模型的“编译”步骤配置的，你在编译时应该指定模型需要使用的优化器和损失函数，还有训练过程中你希望监视的指标。下面例子使用了单个损失函数，也是目前最常见的情况："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras import optimizers\n",
    "\n",
    "model.compile(optimizer=optimizers.RMSprop(lr=0.001),\n",
    "              loss='mse',\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Lastly, the learning process itself consists of passing Numpy arrays of input data (and\n",
    "the corresponding target data) to the model via the fit() method, similarly to what you\n",
    "would do in Scikit-Learn or several other machine learning libraries:\n",
    "\n",
    "最后，学习过程本身只包括传递输入数据的Numpy数组（和相应的目标数据）到模型，这通过fit方法来实现，正如你在Scikit-Learn或其他一些机器学习库中的做法一样："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 本例中无训练数据集，因此下面的代码无法运行\n",
    "model.fit(input_tensor, target_tensor, batch_size=128, epochs=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Over the next few chapters, you will build a solid intuition as to what type of network\n",
    "architectures work for different kinds of problems, how to pick the right learning\n",
    "configuration, and how to tweak a model until it gives you the results you want to see.\n",
    "We’ll start with three basic examples in the next three sections: a two-class classification\n",
    "example, a many-class classification example, and a regression example.\n",
    "\n",
    "后面小节中，你会建立出哪种类型网络结构适合哪种类型问题，如何选择正确的学习参数，以及如何对模型进行调优以达到你希望的结果的直觉。我们后续三个小节会分别讲述三个基本例子：一个二元分类，一个多类别分类和一个回归的例子。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.3 搭建深度学习工作站\n",
    "\n",
    "### 3.3.1 初步考虑\n",
    "\n",
    "> Before you can get started developing deep learning applications, you need to set up your\n",
    "workstation. It is highly recommended, though not strictly necessary, to run deep\n",
    "learning code on a modern NVIDIA GPU. Some applications, in particular image\n",
    "processing with convolutional networks and sequence processing with recurrent neural\n",
    "networks, will be excruciatingly slow on CPU, even with a fast multi-core CPU. And\n",
    "even for applications that can realistically be run on CPU, you would generally observe a\n",
    "5x to 10x speedup by using a modern GPU. If you don’t want to install a GPU on your\n",
    "machine, you could alternatively consider running your experiments on a AWS EC2\n",
    "GPU instance, or on Google Cloud Platform. But note that that cloud GPU instances can\n",
    "get quite expensive over time.\n",
    "\n",
    "在你开始开发深度学习应用之前，你需要搭建你的工作站。虽然不是严格需要，但仍然高度推荐能够使用现代NVIDIA GPU来运行深度学习代码。一些应用，特别是使用卷积网络进行图像处理和使用循环神经网络处理序列数据，在CPU上运行都会慢得让人难以忍受，即便使用高速多核的CPU也是这样。而且那些能够在CPU上运行的应用，使用GPU时也能获得5倍到10倍的速度提升。如果你不打算在你的计算机上安装GPU，你也可以考虑使用云服务运行你的实验，比方说AWS EC2 GPU或者GCP。但需要注意的是，云端的GPU服务实例在长时间运行时可能会很昂贵。\n",
    "\n",
    "> Also, whether running locally or in the cloud, it is better for you to be using a Unix\n",
    "workstation. While it is technically possible to use Keras on Windows (all three Keras\n",
    "backends support Windows), we don’t recommend it. In the installation instructions we\n",
    "provide as an appendix to this book, we will consider an Ubuntu machine. If you are a\n",
    "Windows user, the simplest solution to get everything running is to set up an Ubuntu dual\n",
    "boot on your machine. It may seem like a hassle, but using Ubuntu will save you a lot of\n",
    "time and a lot of trouble in the long run.\n",
    "\n",
    "还有不管本地运行还是云端运行，你都应该使用Unix工作站。虽然在Windows上使用Keras在技术上是可行的（所有三种Keras的后端都支持Windows），但是我们不推荐使用。在本书附录中提供的安装指引中，我们推荐的是Ubuntu系统。如果你是一个Windows用户，最简单的解决方案是在你的机器上安装一个Ubuntu系统作为双启动。这样做乍一看很麻烦，但是使用Ubuntu能在你长时间运行过程中节省很多的时间和减少很多的麻烦。\n",
    "\n",
    "> Note that in order to use Keras, you need to install Theano, or CTNK, or TensorFlow\n",
    "(or all of them, if you want to be able to switch back and forth between the three\n",
    "backends). In this book, we will focus on TensorFlow, with some light instructions\n",
    "relative to Theano. We will not cover CNTK.\n",
    "\n",
    "还需要提醒的是，要使用Keras，你需要安装Theano或CNTK或TensorFlow（或者全部都安装，如果你希望能够在三种后端之间进行切换的话）。在本书中，我们会聚焦在TensorFlow，其中会涉及Theano的一些指引，但不会涉及CNTK。\n",
    "\n",
    "译者注：新版的TensorFlow已经内置了Keras了，因此实际上运行本书的例子只需要安装TensorFlow即可。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3.2 Jupyter notebooks：运行深度学习实验的推荐工具\n",
    "\n",
    "> Jupyter notebooks are a great way to run deep learning experiments, in particular the\n",
    "many code examples contained in this book. They are widely used in the data science and\n",
    "machine learning community. A Jupyter Notebook is a file generated by the Jupyter app,\n",
    "which you can edit in your browser. It mixes the ability to execute Python code, together\n",
    "with rich text editing capabilities for annotating what you are doing. A Notebook also\n",
    "allows you to break up long experiments into smaller \"cells\" which can be executed\n",
    "independently, which makes development interactive, and means that you don’t have to\n",
    "re-run all of your previous code in case something goes wrong late in an experiment.\n",
    "\n",
    "Jupyter notebooks是非常棒的深度学习实验环境，特别是对于本书里面的许多代码例子来说。它在数据科学和机器学习社区被广泛使用。一个Jupyter Notebook是由Jupyter应用程序产生的文件，它能在浏览器中进行编辑。它兼有在其中执行Python代码和富文本编辑的能力，方便在其中记录你的工作。一个Notebook还支持你将一个很长的实验拆散成为多个小的“单元”来独立执行，既能交互式的进行开发，也意味着当实验的最新步骤发生错误时，你不需要重新运行所有之前的代码。\n",
    "\n",
    "> You can find more information about Jupyter at jupyter.org .\n",
    "\n",
    "> We recommend using Jupyter notebooks to get started with Keras, albeit that is not a\n",
    "requirement: you could also be running standalone Python scripts, or you could be\n",
    "running code from within an IDE such as PyCharm. We are making all code examples in\n",
    "this book available as open-source Notebooks, downloadable online.\n",
    "\n",
    "更多Jupyter的内容可以在[Jupyter官网](https://jupyter.org)得到。\n",
    "\n",
    "我们推荐使用Jupyter notebooks来学习Keras，尽管这不是必须的：你也可以在独立的Python脚本中执行代码，或者在诸如PyCharm之类的IDE中执行。作者会将本书所有的代码例子以开源Notebooks的形式放在网上提供下载。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3.3 让Keras跑起来：两种选择\n",
    "\n",
    "> To get started in practice, we recommend one of the following two options:\n",
    "\n",
    "> - Use the official EC2 \"Deep Learning\" AMI and run Keras experiments as Jupyter\n",
    "notebooks on EC2.\n",
    "    - Do this preferably if you do not already have a GPU on your local machine.\n",
    "    - We provide a step by step guide in the appendix \"Running Jupyter notebooks on a EC2 GPU instance\".\n",
    "- Install everything from scratch on a local Unix workstation. You can then either run local Jupyter Notebooks or a regular Python codebase.\n",
    "    - Do this if you already have a high-end NVIDIA GPU.\n",
    "    - We provide a step by step guide in the appendix \"Installing Keras and its dependencies on Unix\".\n",
    "    \n",
    "要真正开始实践，我们推荐下面两种选择之一：\n",
    "\n",
    "- 在AWS上使用官方的EC2 \"Deep Learning\" AMI然后在EC2上使用Jupyter notebooks运行Keras实验。\n",
    "    - 如果你在本地计算机上没有安装GPU，这是推荐的选择。\n",
    "    - 我们在附录“在EC2 GPU实例上运行Jupyter notebooks”中提供了详细的教程。\n",
    "- 在本地Unix工作站上从0搭建所有东西。你可以使用运行本地的Jupyter notebooks或者运行常规的Python代码脚本。\n",
    "    - 如果你有一个高端的NVIDIA GPU的话，采用这个选择。\n",
    "    - 我们在附录“在Unix上安装Keras和它的依赖”中提供了详细的教程。\n",
    "\n",
    "> Let’s take a closer look at some of the compromises involved in picking one option\n",
    "over the other.\n",
    "\n",
    "我们来看看这两种选择的对比。\n",
    "\n",
    "译者注：事实上，对于初学者来说，现在还有一个更好的选择，那就是[Google的Colab](https://colab.research.google.com/)，一个基于Jupyter notebooks的在线工作站，并且提供了免费的GPU可以使用，是并不宽裕的初学者的最佳选择。当然大陆读者因为不可描述的原因，需要一把优秀的梯子。本翻译的所有notebooks都提供了相应的Colab链接可供使用。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3.4 云端运行深度学习任务的优缺点\n",
    "\n",
    "> If you don’t already have a GPU that you can use for deep learning (a recent, high-end\n",
    "NVIDIA GPU), then running deep learning experiments in the cloud is a simple,\n",
    "low-cost way for you to get started without having to buy any additional hardware. If you\n",
    "are using Jupyter notebooks, the experience of running in the cloud is no different from\n",
    "running locally. As of mid-2017, the cloud offering that makes it easiest to get started\n",
    "with deep learning is definitely AWS EC2. In appendix, we provide a step-by-step guide\n",
    "to start running Jupyter notebooks on a EC2 GPU instance.\n",
    "\n",
    "如果你还没有可以用来进行深度学习任务的GPU（目前来说唯一选择就是高端的NVIDIA GPU），那么在云端运行深度学习实验是用来入门的比较简单和省钱的方式，因为你无需购买任何额外的硬件。如果你在使用Jupyter notebooks，在云端运行的体验和本地运行并没有区别。在2017年中时候，云端提供深度学习能力让初学者容易入门的服务无疑是AWS EC2。在附录中提供了在EC2 GPU实例上运行Jupyter notebooks的指南。\n",
    "\n",
    "> However, if you are a heavy user of deep learning, this setup is not sustainable in the\n",
    "long term—or even past a few weeks. EC2 instances are expensive: the instance type we\n",
    "recommend in our appendix, the p2.xlarge instance, which will not provide you with\n",
    "much power, alreadys costs $0.90 per hour (as of mid-2017). Meanwhile, a solid consumer-class GPU will cost you somewhere between $1,000 and $1,500—a price that\n",
    "has been fairly stable over time, even as the specs of these GPUs keep improving. If you\n",
    "are serious about deep learning, you should set up a local workstation with one or more\n",
    "GPUs.\n",
    "\n",
    "然而，如果你是深度学习的重度用户，这不是一个长期选择，甚至超过一个星期就不合适了。EC2实例很贵：在附录中我们推荐的实例类型p2.xlarge实例，它并不能为你提供很强大的算力，却已经需要每小时0.9美元（在2017年中）。同时，一个实体消费级GPU需要花费大约1000-1500美元 - 即使新的GPU还在发展，这个价格已经基本稳定下来。如果你要投入深度学习，你应该搭建一个或多个GPU组成的工作站。\n",
    "\n",
    "> In short: EC2 is a great way to get started. You could follow the code examples in this\n",
    "book entirely on a EC2 GPU instance. But if you are going to be a power user of deep\n",
    "learning, then get your own GPUs.\n",
    "\n",
    "简单来说：EC2是一个入门的好地方。你可以在EC2 GPU实力上完成本书的所有代码例子。但是如果你要成为一个深度学习的超级用户，那么需要购买自己的GPU。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3.5 哪种GPU最合适？\n",
    "\n",
    "> If you are going to buy a GPU, which one should you choose? The first thing to note it\n",
    "that it would have to be a NVIDIA GPU. NVIDIA is the only graphics computing\n",
    "company to have heavily invested into deep learning so far, and modern deep learning\n",
    "frameworks can only run on NVIDIA cards.\n",
    "\n",
    "如果你打算购买GPU，那么应该选择哪个呢？首先要确保的它必须是一个NVIDIA GPU。NVIDIA是目前唯一大量投入在深度学习领域的图形计算公司，而且现在的深度学习框架只能在NVIDIA显卡上运行。\n",
    "\n",
    "> As of mid-2017, I would recommend the NVIDIA Titan Xp as the best card on the\n",
    "market for deep learning. For lower budgets, you might want to consider the GTX 1060.\n",
    "If you are reading these pages in 2018 or later, do take the time to look online for fresher\n",
    "recommendations, as new models come out every year.\n",
    "\n",
    "在2017年中，作者会推荐NVIDIA Titan Xp作为市面上深度学习使用的最佳显卡。如果预算不够，你可以考虑GTX 1060。如果你是在2018年之后读到本书，请花些时间搜索网上的推荐，因为每年都会推出新的显卡。\n",
    "\n",
    "> From this section onwards, we will assume that you have access to a machine with\n",
    "Keras and its dependencies installed—preferably with GPU support. Make sure you get\n",
    "this step done before you go any further. Go through our step-by-step guides provided in\n",
    "appendix, and look online if you need further help. There is no shortage of tutorials on\n",
    "how to install Keras and common deep learning dependencies.\n",
    "\n",
    "看完前面章节后，我们假设你已经有了一台设备安装好了Keras和它的依赖了，当然最好有GPU支持。请在继续之前确认这一步已经完成。你可以参考附录中的指南，或者在网上寻找更多帮助，有关安装Keras和通用深度学习依赖的教程在网上汗牛充栋。\n",
    "\n",
    "> We can now start diving into practical Keras examples.\n",
    "\n",
    "我们下面可以进入实际的Keras例子了。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.4 二元分类例子：影评分类\n",
    "\n",
    "> Two-class classification, or binary classification, may be the most widely applied kind of\n",
    "machine learning problem. In this example, we will learn to classify movie reviews into\n",
    "\"positive\" reviews and \"negative\" reviews, just based on the text content of the reviews.\n",
    "\n",
    "二元分类，或叫二分分类，可能是最广泛应用的机器学习问题。在这个例子中，我们会学习基于影评的文本内容将影评数据分成“积极”和“消极”两类。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.4.1 IMDB数据集\n",
    "\n",
    "> We’ll be working with \"IMDB dataset\", a set of 50,000 highly-polarized reviews from\n",
    "the Internet Movie Database. They are split into 25,000 reviews for training and 25,000\n",
    "reviews for testing, each set consisting in 50% negative and 50% positive reviews.\n",
    "\n",
    "我们会使用“IMDB数据集”，它包含着国际互联网电影数据库（IMDB）中50000个高度具有偏向性的评论。它们被分成25000个评论组成的训练集和25000个组成的测试集，每个数据集都由50%的消极评论和50%的积极评论组成。\n",
    "\n",
    "> Why do we have these two separate training and test sets? You should never test a\n",
    "machine learning model on the same data that you used to train it! Just because a model\n",
    "performs well on its training data doesn’t mean that it will perform well on data it has\n",
    "never seen, and what you actually care about is your model’s performance on new data\n",
    "(since you already know the labels of your training data—obviously you don not need\n",
    "your model to predict those). For instance, it is possible that your model could end up\n",
    "merely memorizing a mapping between your training samples and their targets—which\n",
    "would be completely useless for the task of predicting targets for data never seen before.\n",
    "We will go over this point in much more detail in the next chapter.\n",
    "\n",
    "为什么我们需要两个独立的训练和测试数据集呢？这里有一个基本原则，那就是永远不要使用与训练相同的数据来对模型进行测试！仅仅因为模型在训练数据上表现优异并不代表它能在没有见过的数据上也能很好的工作，而实际上你最终关心的是模型在新数据上的性能（因为你已经知道了训练数据上的标签，显然你不需要模型再去预测它们）。例如，你的模型可能仅仅是记住了训练样本和它们目标之间的映射关系。这对于预测未知数据的目标任务来说是完全没有作用的。我们会在下一章中更加深入的讨论这一点。\n",
    "\n",
    "> Just like the MNIST dataset, the IMDB dataset comes packaged with Keras. It has already been preprocessed: the reviews (sequences of words) have been turned into\n",
    "sequences of integers, where each integer stands for a specific word in a dictionary.\n",
    "\n",
    "就像MNIST数据集一样，IMDB数据集也內建在Keras中。它已经被预处理过：这些影评（单词的序列）已经被转换为整数的序列，其中的每个整数代表着字典中一个特定的单词。\n",
    "\n",
    "> The following code will load the dataset (when you run it for the first time, about\n",
    "80MB of data will be downloaded to your machine):\n",
    "\n",
    "下面的代码会装载数据集（当你第一次运行是，会下载大约80MB的数据）："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras import datasets\n",
    "\n",
    "(train_data, train_label), (test_data, test_label) = datasets.imdb.load_data(num_words=10000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> The argument num_words=10000 means that we will only keep the top 10,000 most\n",
    "frequently occurring words in the training data. Rare words will be discarded. This\n",
    "allows us to work with vector data of manageable size.\n",
    "\n",
    "参数`num_words=10000`表示我们仅保留训练数据中出现频率最高的头10000个单词。生僻单词会被抛弃。这可以让我们使用一个可控大小的矢量来进行实验。\n",
    "\n",
    "> The variables train_data and test_data are lists of reviews, each review being a\n",
    "list of word indices (encoding a sequence of words). train_labels and test_labels\n",
    "are lists of 0s and 1s, where 0 stands for \"negative\" and 1 stands for \"positive\":\n",
    "\n",
    "变量`train_data`和`test_data`是影评的列表，每个评论是一个单词序号的列表（编码成单词的序列）。`train_label`和`test_label`是一个由0和1组成的列表，这里的0代表消极，1代表积极："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1,\n",
       " 14,\n",
       " 22,\n",
       " 16,\n",
       " 43,\n",
       " 530,\n",
       " 973,\n",
       " 1622,\n",
       " 1385,\n",
       " 65,\n",
       " 458,\n",
       " 4468,\n",
       " 66,\n",
       " 3941,\n",
       " 4,\n",
       " 173,\n",
       " 36,\n",
       " 256,\n",
       " 5,\n",
       " 25,\n",
       " 100,\n",
       " 43,\n",
       " 838,\n",
       " 112,\n",
       " 50,\n",
       " 670,\n",
       " 2,\n",
       " 9,\n",
       " 35,\n",
       " 480,\n",
       " 284,\n",
       " 5,\n",
       " 150,\n",
       " 4,\n",
       " 172,\n",
       " 112,\n",
       " 167,\n",
       " 2,\n",
       " 336,\n",
       " 385,\n",
       " 39,\n",
       " 4,\n",
       " 172,\n",
       " 4536,\n",
       " 1111,\n",
       " 17,\n",
       " 546,\n",
       " 38,\n",
       " 13,\n",
       " 447,\n",
       " 4,\n",
       " 192,\n",
       " 50,\n",
       " 16,\n",
       " 6,\n",
       " 147,\n",
       " 2025,\n",
       " 19,\n",
       " 14,\n",
       " 22,\n",
       " 4,\n",
       " 1920,\n",
       " 4613,\n",
       " 469,\n",
       " 4,\n",
       " 22,\n",
       " 71,\n",
       " 87,\n",
       " 12,\n",
       " 16,\n",
       " 43,\n",
       " 530,\n",
       " 38,\n",
       " 76,\n",
       " 15,\n",
       " 13,\n",
       " 1247,\n",
       " 4,\n",
       " 22,\n",
       " 17,\n",
       " 515,\n",
       " 17,\n",
       " 12,\n",
       " 16,\n",
       " 626,\n",
       " 18,\n",
       " 2,\n",
       " 5,\n",
       " 62,\n",
       " 386,\n",
       " 12,\n",
       " 8,\n",
       " 316,\n",
       " 8,\n",
       " 106,\n",
       " 5,\n",
       " 4,\n",
       " 2223,\n",
       " 5244,\n",
       " 16,\n",
       " 480,\n",
       " 66,\n",
       " 3785,\n",
       " 33,\n",
       " 4,\n",
       " 130,\n",
       " 12,\n",
       " 16,\n",
       " 38,\n",
       " 619,\n",
       " 5,\n",
       " 25,\n",
       " 124,\n",
       " 51,\n",
       " 36,\n",
       " 135,\n",
       " 48,\n",
       " 25,\n",
       " 1415,\n",
       " 33,\n",
       " 6,\n",
       " 22,\n",
       " 12,\n",
       " 215,\n",
       " 28,\n",
       " 77,\n",
       " 52,\n",
       " 5,\n",
       " 14,\n",
       " 407,\n",
       " 16,\n",
       " 82,\n",
       " 2,\n",
       " 8,\n",
       " 4,\n",
       " 107,\n",
       " 117,\n",
       " 5952,\n",
       " 15,\n",
       " 256,\n",
       " 4,\n",
       " 2,\n",
       " 7,\n",
       " 3766,\n",
       " 5,\n",
       " 723,\n",
       " 36,\n",
       " 71,\n",
       " 43,\n",
       " 530,\n",
       " 476,\n",
       " 26,\n",
       " 400,\n",
       " 317,\n",
       " 46,\n",
       " 7,\n",
       " 4,\n",
       " 2,\n",
       " 1029,\n",
       " 13,\n",
       " 104,\n",
       " 88,\n",
       " 4,\n",
       " 381,\n",
       " 15,\n",
       " 297,\n",
       " 98,\n",
       " 32,\n",
       " 2071,\n",
       " 56,\n",
       " 26,\n",
       " 141,\n",
       " 6,\n",
       " 194,\n",
       " 7486,\n",
       " 18,\n",
       " 4,\n",
       " 226,\n",
       " 22,\n",
       " 21,\n",
       " 134,\n",
       " 476,\n",
       " 26,\n",
       " 480,\n",
       " 5,\n",
       " 144,\n",
       " 30,\n",
       " 5535,\n",
       " 18,\n",
       " 51,\n",
       " 36,\n",
       " 28,\n",
       " 224,\n",
       " 92,\n",
       " 25,\n",
       " 104,\n",
       " 4,\n",
       " 226,\n",
       " 65,\n",
       " 16,\n",
       " 38,\n",
       " 1334,\n",
       " 88,\n",
       " 12,\n",
       " 16,\n",
       " 283,\n",
       " 5,\n",
       " 16,\n",
       " 4472,\n",
       " 113,\n",
       " 103,\n",
       " 32,\n",
       " 15,\n",
       " 16,\n",
       " 5345,\n",
       " 19,\n",
       " 178,\n",
       " 32]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_label[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Since we restricted ourselves to the top 10,000 most frequent words, no word index\n",
    "will exceed 10,000:\n",
    "\n",
    "因为我们将训练数据限制在频率最高的10000个单词上，因此没有单词的序号会超过10000："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9999"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "max([max(sequence) for sequence in train_data])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> For kicks, here’s how you can quickly decode one of these reviews back to English\n",
    "words:\n",
    "\n",
    "为了更加清晰，下面是我们将一条评论解码回到英文单词的方法："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"? this film was just brilliant casting location scenery story direction everyone's really suited the part they played and you could just imagine being there robert ? is an amazing actor and now the same being director ? father came from the same scottish island as myself so i loved the fact there was a real connection with this film the witty remarks throughout the film were great it was just brilliant so much that i bought the film as soon as it was released for ? and would recommend it to everyone to watch and the fly fishing was amazing really cried at the end it was so sad and you know what they say if you cry at a film it must have been good and this definitely was also ? to the two little boy's that played the ? of norman and paul they were just brilliant children are often left out of the ? list i think because the stars that play them all grown up are such a big profile for the whole film but these children are amazing and should be praised for what they have done don't you think the whole story was so lovely because it was true and was someone's life after all that was shared with us all\""
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# word_index是将单词映射成整数的字典\n",
    "word_index = datasets.imdb.get_word_index()\n",
    "# 将其中的KV反过来，形成一个将整数映射回单词的字典\n",
    "reverse_word_index = dict([(value, key) for (key, value) in word_index.items()])\n",
    "# 解码评论内容，注意序号从3开始，因为0，1，2是保留的，分别代表“对齐”，“序列开始”和“未知”\n",
    "decoded_review = ' '.join([reverse_word_index.get(i - 3, '?') for i in train_data[0]])\n",
    "decoded_review"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.4.2 数据准备\n",
    "\n",
    "> We cannot feed lists of integers into a neural network. We have to turn our lists into\n",
    "tensors. There are two ways we could do that:\n",
    "\n",
    "> - We could pad our lists so that they all have the same length, and turn them into an integer\n",
    "tensor of shape (samples, word_indices) , then use as first layer in our network a\n",
    "layer capable of handling such integer tensors (the Embedding layer, which we will cover\n",
    "in detail later in the book).\n",
    "- We could one-hot-encode our lists to turn them into vectors of 0s and 1s. Concretely, this\n",
    "would mean for instance turning the sequence [3, 5] into a 10,000-dimensional vector\n",
    "that would be all-zeros except for indices 3 and 5, which would be ones. Then we could\n",
    "use as first layer in our network a Dense layer, capable of handling floating point vector\n",
    "data.\n",
    "\n",
    "我们不能直接将一个整数列表放入神经网络。我们将列表转换成张量。有两种方式来处理：\n",
    "\n",
    "- 我们对列表进行填充，让它们有相同的长度，然后将它转换成一个形状为(样本, 单词序号)的张量，然后在我们的网络第一层中应用一个能够处理这样的整数张量的层（被称为嵌入层，我们在本书后面会详细介绍）。\n",
    "- 我们也可以使用one-hot编码将列表编程一个仅含有0和1的矢量。具体来说，例如我们有一个序列`[3, 5]`，我们可以将它转换成一个10000维度的矢量，矢量上面除了序号3和5的位置为1外，其余都是0。这样我们就可以在网络中使用全连接层作为第一层，用来正确处理浮点矢量数据。\n",
    "\n",
    "> We will go with the latter solution. Let’s vectorize our data, which we will do\n",
    "manually for maximum clarity:\n",
    "\n",
    "下面我们采用第二种方法。将数据矢量化，这里为了清晰起见，我们采用手动实现："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "def vectorize_sequences(sequences, dimension=10000):\n",
    "    # 创建一个全0矩阵，形状是((len(sequences), dimension))\n",
    "    results = np.zeros((len(sequences), dimension))\n",
    "    for i, sequence in enumerate(sequences):\n",
    "        results[i, sequence] = 1. # 设置相应的位置为1\n",
    "    return results\n",
    "\n",
    "# 获得矢量化的训练数据\n",
    "x_train = vectorize_sequences(train_data)\n",
    "# 获得矢量化的测试数据\n",
    "x_test = vectorize_sequences(test_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0., 1., 1., ..., 0., 0., 0.])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 矢量化后训练数据看起来如下\n",
    "x_train[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> We should also vectorize our labels, which is straightforward:\n",
    "\n",
    "我们也需要矢量化标签，很直接的使用类型转换就行："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train = np.asarray(train_label).astype('float32')\n",
    "y_test = np.asarray(test_label).astype('float32')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Now our data is ready to be fed into a neural network.\n",
    "\n",
    "现在数据已经准备好了，可以输入到神经网络中了。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.4.3 构建我们的网络\n",
    "\n",
    "> Our input data is simply vectors, and our labels are scalars (1s and 0s): this is the easiest\n",
    "setup you will ever encounter. A type of network that performs well on such a problem\n",
    "would be a simple stack of fully-connected ( Dense ) layers with relu activations:\n",
    "Dense(16, activation='relu')\n",
    "\n",
    "输入数据是简单的矢量，标签是标量（1和0）：这是你将会遇到的最简单的情况。一种简单的网络能在这种情况下运行良好，那就是将全连接（Dense）层堆叠起来使用relu作为激活函数的网络：`Dense(16, activation='relu')`\n",
    "\n",
    "> The argument being passed to each Dense layer (16) is the number of \"hidden units\"\n",
    "of the layer. What’s a hidden unit? It’s a dimension in the representation space of the\n",
    "layer. You may remember from the previous chapter that each such Dense layer with a\n",
    "relu activation implements the following chain of tensor operations:\n",
    "\n",
    "传递给全连接层的参数（16）是该层当中“隐藏单元”的数量。什么是隐藏单元？它是该层表示空间的维度。你可能还记得前一章中我们介绍过这样使用relu激活函数的全连接层实现了下面的链式张量操作：\n",
    "$$output = relu(W \\cdot input + b)$$\n",
    "\n",
    "> Having 16 hidden units means that the weight matrix W will have shape\n",
    "(input_dimension, 16) , i.e. the dot product with W will project the input data onto a\n",
    "16-dimensional representation space (and then we would add the bias vector b and apply\n",
    "the relu operation). You can intuitively understand the dimensionality of your\n",
    "representation space as \"how much freedom you are allowing the network to have when\n",
    "learning internal representations\". Having more hidden units (a higher-dimensional\n",
    "representation space) allows your network to learn more complex representations, but it\n",
    "makes your network more computationally expensive and may lead to learning unwanted\n",
    "patterns (patterns that will improve performance on the training data but not on the test\n",
    "data).\n",
    "\n",
    "拥有16个隐藏单元表示这里的权重矩阵W的形状是(输入维度, 16)，也就是说，W和输入数据的点积其实就是W会将输入数据投射到一个16维的表示空间上（然后我们会加上偏差和应用relu操作）。你可以将表示空间的维度直观的理解为“当网络学习内部表示形式时有多少的自由度”。有更多隐藏单元（更高的表示空间维度）允许你的网络能够学习更加复杂的表示，但它会让网络更加消耗计算资源和可能导致学习到不需要的模式（这些不需要的模式仅会改善模型在训练数据上的性能但不会改善在测试数据上的性能）。\n",
    "\n",
    "> There are two key architecture decisions to be made about such stack of dense layers:\n",
    "\n",
    "> - How many layers to use.\n",
    "- How many \"hidden units\" to chose for each layer.\n",
    "\n",
    "对于全连接层的堆叠结构来说有两个关键的决定：\n",
    "\n",
    "- 要使用的层数。\n",
    "- 每一层要选择的“隐藏单元”。\n",
    "\n",
    "> In the next chapter, you will learn formal principles to guide you in making these\n",
    "choices. For the time being, you will have to trust us with the following architecture\n",
    "choice: two intermediate layers with 16 hidden units each, and a third layer which will\n",
    "output the scalar prediction regarding the sentiment of the current review. The\n",
    "intermediate layers will use relu as their \"activation function\", and the final layer will\n",
    "use a sigmoid activation so as to output a probability (a score between 0 and 1, indicating\n",
    "how likely the sample is to have the target \"1\", i.e. how likely the review is to be\n",
    "positive). A relu (rectified linear unit) is a function meant to zero-out negative values\n",
    "(see Figure 3.4) while a sigmoid \"squashes\" arbitrary values into the [0, 1] interval,\n",
    "(see Figure 3.5), thus outputting something that can be interpreted as a probability.\n",
    "\n",
    "在下一章中，你会学习到进行上述选择的一些正式原则。现在，你只需要清楚我们选择了以下的结构：两个中间层，每层有16个隐藏单元，第三层会根据当前评论的情绪输出标量预测结果。这两个中间层会使用relu作为它们的“激活函数”，最终层会使用sigmoid激活函数让它能够输出一个概率（0到1之间的分值，标志这个样本有多大概率为“1”，也就是这条评论有多大概率是积极的）。relu（整流线性单元）是一个函数会将所有负值变成0（参见图3-4），sigmoid将任意值压缩到`[0, 1]`区间（参见图3-5），因此可以输出概率。\n",
    "\n",
    "![relu](imgs/f3.4.jpg)\n",
    "\n",
    "图3-4 整流线性单元函数\n",
    "\n",
    "![sigmoid](imgs/f3.5.jpg)\n",
    "\n",
    "图3-5 sigmoid函数"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Here’s what our network looks like:\n",
    "\n",
    "下图是我们网络的结构：\n",
    "\n",
    "![arch of network](imgs/f3.6.jpg)\n",
    "\n",
    "图3-6 我们的3层网络结构"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> And here’s the Keras implementation, very similar to the MNIST example you saw\n",
    "previously:\n",
    "\n",
    "下面是Keras的实现，这些代码与前面看到的MNIST例子非常相似："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras import models\n",
    "from tensorflow.keras import layers\n",
    "\n",
    "model = models.Sequential()\n",
    "model.add(layers.Dense(16, activation='relu', input_shape=(10000,)))\n",
    "model.add(layers.Dense(16, activation='relu'))\n",
    "model.add(layers.Dense(1, activation='sigmoid'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 注：什么是激活函数，为什么我们需要它？\n",
    "\n",
    "> Without an activation function like relu (also called a non-linearity ), our\n",
    "Dense layer would consist of two linear operations, a dot product and an\n",
    "addition.\n",
    "output = dot(W, input) + b\n",
    "\n",
    "如果没有诸如relu这样的激活函数（也被成为非线性），我们的全连接层仅包括两个线性操作，一个点积和一个加法。\n",
    "$$output = W \\cdot input + b$$\n",
    "\n",
    "> So the layer could only learn linear transformations (affine\n",
    "transformations) of the input data, i.e. the hypothesis space of the layer\n",
    "would be the set of all possible linear transformations of the input data\n",
    "into a 16-dimensional space. Such an hypothesis space is too restricted,\n",
    "and wouldn’t benefit from multiple layers of representations, because a\n",
    "deep stack of linear layers would still implement a linear operation: adding\n",
    "more layers wouldn’t extend the hypothesis space.\n",
    "\n",
    "因此这个层只能学习到输入数据的线性变换（也叫仿射变换），也就是说层的16维假设空间只是一组输入数据的所有可能的线性变换的组合。这样的假设空间的局限性非常大，并且也无法因为层次的增加而改善，因为一个很深的线性层堆叠依旧智能只能线性操作：增加更多的层不能扩展假设空间。\n",
    "\n",
    "> Lastly, we need to pick a loss function and an optimizer. Since we are facing a binary\n",
    "classification problem and the output of our network is a probability (we end our network\n",
    "with a single-unit layer with a sigmoid activation), is it best to use the\n",
    "binary_crossentropy loss. It isn’t the only viable choice: you could use, for instance,\n",
    "mean_squared_error . But crossentropy is usually the best choice when you are dealing\n",
    "with models that output probabilities. Crossentropy is a quantity from the field of\n",
    "Information Theory, that measures the \"distance\" between probability distributions, or in\n",
    "our case, between the ground-truth distribution and our predictions.\n",
    "\n",
    "最后，我们需要选取损失函数和优化器。因为我们面对的是二元分类问题，且网络的输出是一个概率（我们网络最终是只有一个单元和使用sigmoid激活函数的层），因此最好使用`binary_crossentropy`损失函数。但这不是唯一可行的选择：你可以使用例如`mean_sqaure_err`。但是交叉熵通常是当你处理模型的输出是概率的情况下的最佳选择。交叉熵是信息论领域的一个量，它量度概率分布之间的“距离”，或者在我们这个例子中，就是真实分布和我们预测之间的距离。\n",
    "\n",
    "> Here’s the step where we configure our model with the rmsprop optimizer and the\n",
    "binary_crossentropy loss function. Note that we will also monitor accuracy during\n",
    "training.\n",
    "\n",
    "下面的步骤用来配置我们网络使用`rmsprop`优化器和`binary_crossentropy`损失函数。注意我们还配置了在训练过程中监视准确率。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(optimizer='rmsprop',\n",
    "              loss='binary_crossentropy',\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> We are passing our optimizer, loss function and metrics as strings, which is possible\n",
    "because rmsprop , binary_crossentropy and accuracy are packaged as part of Keras.\n",
    "Sometimes you may want to configure the parameters of your optimizer, or pass a\n",
    "custom loss function or metric function. This former can be done by passing an optimizer\n",
    "class instance as the optimizer argument:\n",
    "\n",
    "在这里我们使用字符串形式传递了优化器，损失函数和指标参数，这是因为`rmsprop`、`binary_crossentropy`和`accuracy`被打包成了Keras的一部分。有时你可能需要定义优化器的参数，或者传递一个自定义的损失函数或者指标函数。前者可以通过传递一个`optimizer`类的实例来作为优化器参数："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras import optimizers\n",
    "\n",
    "model.compile(optimizer=optimizers.RMSprop(lr=0.001),\n",
    "              loss='binary_crossentropy',\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> The latter can be done by passing function objects as the loss or metrics arguments:\n",
    "\n",
    "后者可以通过传递函数对象到loss或metrics参数上实现："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras import losses, metrics\n",
    "\n",
    "model.compile(optimizer=optimizers.RMSprop(lr=0.001),\n",
    "              loss=losses.binary_crossentropy,\n",
    "              metrics=[metrics.binary_accuracy])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.4.4 验证模型\n",
    "\n",
    "> In order to monitor during training the accuracy of the model on data that it has never\n",
    "seen before, we will create a \"validation set\" by setting apart 10,000 samples from the\n",
    "original training data:\n",
    "\n",
    "为了能够在训练过程中监视模型在它没见过的数据上的准确率，我们会创建一个“验证集”，通过从原始训练数据中抽取10000个样本来获得："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_val = x_train[:10000]\n",
    "partial_x_train = x_train[10000:]\n",
    "\n",
    "y_val = y_train[:10000]\n",
    "partial_y_train = y_train[10000:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> We will now train our model for 20 epochs (20 iterations over all samples in the\n",
    "x_train and y_train tensors), in mini-batches of 512 samples. At this same time we\n",
    "will monitor loss and accuracy on the 10,000 samples that we set apart. This is done by\n",
    "passing the validation data as the validation_data argument:\n",
    "\n",
    "我们现在开始对我们的模型进行20 epochs的训练（在训练集所有样本上进行20次的迭代），使用的批次大小是512个样本。与此同时我们会在刚才分出来的10000个样本的验证集上监视损失和准确率。这是通过将验证集传递到`validation_data`参数来实现的："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 15000 samples, validate on 10000 samples\n",
      "Epoch 1/20\n",
      "15000/15000 [==============================] - 5s 300us/sample - loss: 0.4483 - binary_accuracy: 0.8191 - val_loss: 0.3398 - val_binary_accuracy: 0.8695\n",
      "Epoch 2/20\n",
      "15000/15000 [==============================] - 1s 73us/sample - loss: 0.2454 - binary_accuracy: 0.9146 - val_loss: 0.2833 - val_binary_accuracy: 0.8881\n",
      "Epoch 3/20\n",
      "15000/15000 [==============================] - 1s 77us/sample - loss: 0.1764 - binary_accuracy: 0.9396 - val_loss: 0.2789 - val_binary_accuracy: 0.8877\n",
      "Epoch 4/20\n",
      "15000/15000 [==============================] - 2s 114us/sample - loss: 0.1349 - binary_accuracy: 0.9552 - val_loss: 0.2983 - val_binary_accuracy: 0.8837\n",
      "Epoch 5/20\n",
      "15000/15000 [==============================] - 1s 45us/sample - loss: 0.1083 - binary_accuracy: 0.9640 - val_loss: 0.3199 - val_binary_accuracy: 0.8836\n",
      "Epoch 6/20\n",
      "15000/15000 [==============================] - 1s 49us/sample - loss: 0.0822 - binary_accuracy: 0.9745 - val_loss: 0.3541 - val_binary_accuracy: 0.8784\n",
      "Epoch 7/20\n",
      "15000/15000 [==============================] - 1s 59us/sample - loss: 0.0621 - binary_accuracy: 0.9800 - val_loss: 0.4027 - val_binary_accuracy: 0.8743\n",
      "Epoch 8/20\n",
      "15000/15000 [==============================] - 1s 61us/sample - loss: 0.0438 - binary_accuracy: 0.9882 - val_loss: 0.4545 - val_binary_accuracy: 0.8711\n",
      "Epoch 9/20\n",
      "15000/15000 [==============================] - 1s 47us/sample - loss: 0.0337 - binary_accuracy: 0.9911 - val_loss: 0.4862 - val_binary_accuracy: 0.8723\n",
      "Epoch 10/20\n",
      "15000/15000 [==============================] - 1s 94us/sample - loss: 0.0214 - binary_accuracy: 0.9952 - val_loss: 0.5770 - val_binary_accuracy: 0.8665\n",
      "Epoch 11/20\n",
      "15000/15000 [==============================] - 1s 46us/sample - loss: 0.0154 - binary_accuracy: 0.9967 - val_loss: 0.6288 - val_binary_accuracy: 0.8646\n",
      "Epoch 12/20\n",
      "15000/15000 [==============================] - 1s 50us/sample - loss: 0.0093 - binary_accuracy: 0.9984 - val_loss: 0.6588 - val_binary_accuracy: 0.8693\n",
      "Epoch 13/20\n",
      "15000/15000 [==============================] - 1s 47us/sample - loss: 0.0067 - binary_accuracy: 0.9989 - val_loss: 0.7335 - val_binary_accuracy: 0.8643\n",
      "Epoch 14/20\n",
      "15000/15000 [==============================] - 1s 64us/sample - loss: 0.0050 - binary_accuracy: 0.9991 - val_loss: 0.7747 - val_binary_accuracy: 0.8658\n",
      "Epoch 15/20\n",
      "15000/15000 [==============================] - 1s 66us/sample - loss: 0.0029 - binary_accuracy: 0.9995 - val_loss: 0.8366 - val_binary_accuracy: 0.8645\n",
      "Epoch 16/20\n",
      "15000/15000 [==============================] - 1s 48us/sample - loss: 0.0015 - binary_accuracy: 0.9999 - val_loss: 0.9268 - val_binary_accuracy: 0.8630\n",
      "Epoch 17/20\n",
      "15000/15000 [==============================] - 1s 65us/sample - loss: 7.6299e-04 - binary_accuracy: 0.9999 - val_loss: 1.0181 - val_binary_accuracy: 0.8585\n",
      "Epoch 18/20\n",
      "15000/15000 [==============================] - 1s 50us/sample - loss: 6.6945e-04 - binary_accuracy: 0.9999 - val_loss: 1.0543 - val_binary_accuracy: 0.8631\n",
      "Epoch 19/20\n",
      "15000/15000 [==============================] - 2s 116us/sample - loss: 0.0015 - binary_accuracy: 0.9996 - val_loss: 1.1325 - val_binary_accuracy: 0.8611\n",
      "Epoch 20/20\n",
      "15000/15000 [==============================] - 1s 91us/sample - loss: 1.3431e-04 - binary_accuracy: 1.0000 - val_loss: 1.1499 - val_binary_accuracy: 0.8621\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(partial_x_train, partial_y_train,\n",
    "                    epochs=20, batch_size=256,\n",
    "                    validation_data=(x_val, y_val))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> On CPU, this will take less than two seconds per epoch—training is over in 20\n",
    "seconds. At the end of every epoch, there is a slight pause as the model computes its loss\n",
    "and accuracy on the 10,000 samples of the validation data.\n",
    "\n",
    "在CPU上这个训练每次epoch会花费少于两秒的时间 - 训练会在约20秒完成。在每个epoch的结束，会有一个轻微的停顿，因为模型需要在验证数据的10000个样本上计算它的损失和准确率。\n",
    "\n",
    "> Note that the call to model.fit() returns a History object. This object has a\n",
    "member history , which is a dictionary containing data about everything that happened\n",
    "during training. Let’s take a look at it:\n",
    "\n",
    "注意调用`model.fit()`会返回一个History对象。这个对象有一个成员history，这是一个包含着在训练过程中发生的几乎所有事情的字典。让我们来看一下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['loss', 'binary_accuracy', 'val_loss', 'val_binary_accuracy'])"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "history.history.keys()\n",
    "\n",
    "# 会输出：dict_keys(['loss', 'binary_accuracy', 'val_loss', 'val_binary_accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> It contains 4 entries: one per metric that was being monitored, during training and\n",
    "during validation. Let’s use Matplotlib to plot the training and validation loss side by\n",
    "side, as well as the training and validation accuracy:\n",
    "\n",
    "它包含四个项目：每个对应着一个在训练或验证过程中被监视的指标。让我们使用Matplotlib来绘制训练和验证损失，以及训练和验证准确率："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f47a4317950>"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3dd5hU5dnH8e8toEixATZAwAgoSF+wrF2jgAiIBZCIBCv2LsZECYa8UXnVkNgAgw1F36gEBcWAUhQ1oCIRhYAEcBUUUZp0uN8/nrMwrFvZnT0zO7/Pdc21M6fNvWdnzz1POc9j7o6IiGSuPeIOQERE4qVEICKS4ZQIREQynBKBiEiGUyIQEclwSgQiIhlOiUDKlJm9YWaXlPW2cTKzxWZ2RhKO62Z2RPT8cTP7XXG23Y336WNmb+1unIUc9xQzyynr40r5qxx3ABI/M1uX8LIasAnYFr2+0t1HF/dY7t4pGdtWdO5+VVkcx8waAv8Fqrj71ujYo4Fi/w0l8ygRCO5eI/e5mS0GLnP3SXm3M7PKuRcXEak4VDUkBcot+pvZHWa2HBhlZvub2etmtsLMfoye10vYZ4qZXRY972dm75rZ0Gjb/5pZp93ctpGZTTOztWY2ycweMbPnCoi7ODHea2bvRcd7y8xqJ6y/2MyWmNlKM7urkPNzjJktN7NKCcvONbM50fMOZva+ma0ys2Vm9lcz27OAYz1lZn9IeH1btM83ZtY/z7Znm9knZrbGzL4ys0EJq6dFP1eZ2TozOy733Cbsf7yZzTSz1dHP44t7bgpjZkdF+68ys7lm1jVhXWcz+zw65tdmdmu0vHb091llZj+Y2XQz03WpnOmES1EOBg4AGgBXED4zo6LXhwEbgL8Wsv8xwHygNnA/8KSZ2W5s+zzwL6AWMAi4uJD3LE6MFwG/Bg4E9gRyL0zNgMei4x8avV898uHuHwI/AaflOe7z0fNtwE3R73MccDpwdSFxE8XQMYrnl0BjIG/7xE9AX2A/4GxggJl1j9adFP3cz91ruPv7eY59ADAeGBb9bg8C482sVp7f4WfnpoiYqwCvAW9F+10HjDazptEmTxKqGWsCRwNvR8tvAXKAOsBBwG8AjXtTzpQIpCjbgXvcfZO7b3D3le7+sruvd/e1wBDg5EL2X+LuI9x9G/A0cAjhH77Y25rZYUB74G533+zu7wLjCnrDYsY4yt3/4+4bgJeA1tHy84HX3X2au28Cfhedg4K8APQGMLOaQOdoGe7+kbt/4O5b3X0x8EQ+ceTnwii+z9z9J0LiS/z9prj7v919u7vPid6vOMeFkDgWuPuzUVwvAPOAcxK2KejcFOZYoAbwp+hv9DbwOtG5AbYAzcxsH3f/0d0/Tlh+CNDA3be4+3TXAGjlTolAirLC3TfmvjCzamb2RFR1soZQFbFfYvVIHstzn7j7+uhpjRJueyjwQ8IygK8KCriYMS5PeL4+IaZDE48dXYhXFvRehG//PcxsL6AH8LG7L4niaBJVeyyP4vgjoXRQlF1iAJbk+f2OMbN3oqqv1cBVxTxu7rGX5Fm2BKib8Lqgc1NkzO6emDQTj3seIUkuMbOpZnZctPwBYCHwlpktMrOBxfs1pCwpEUhR8n47uwVoChzj7vuwsyqioOqesrAMOMDMqiUsq1/I9qWJcVnisaP3rFXQxu7+OeGC14ldq4UgVDHNAxpHcfxmd2IgVG8lep5QIqrv7vsCjycct6hv098QqswSHQZ8XYy4ijpu/Tz1+zuO6+4z3b0bodpoLKGkgbuvdfdb3P1woCtws5mdXspYpISUCKSkahLq3FdF9c33JPsNo2/Ys4BBZrZn9G3ynEJ2KU2Mfwe6mNkJUcPuYIr+P3keuIGQcP4vTxxrgHVmdiQwoJgxvAT0M7NmUSLKG39NQglpo5l1ICSgXCsIVVmHF3DsCUATM7vIzCqbWU+gGaEapzQ+JJQebjezKmZ2CuFvNCb6m/Uxs33dfQvhnGwHMLMuZnZE1Ba0mtCuUlhVnCSBEoGU1MPA3sD3wAfAm+X0vn0IDa4rgT8ALxLud8jPbsfo7nOBawgX92XAj4TGzMLk1tG/7e7fJyy/lXCRXguMiGIuTgxvRL/D24Rqk7fzbHI1MNjM1gJ3E327jvZdT2gTeS/qiXNsnmOvBLoQSk0rgduBLnniLjF330y48HcinPdHgb7uPi/a5GJgcVRFdhXh7wmhMXwSsA54H3jU3d8pTSxScqZ2GUlHZvYiMM/dk14iEanoVCKQtGBm7c3sF2a2R9S9shuhrllESkl3Fku6OBh4hdBwmwMMcPdP4g1JpGJIWtWQmf2NUBf5nbsfnc/6PsAdhN4Oawn/2J8mJRgRESlQMquGngI6FrL+v8DJ7t4CuBcYnsRYRESkAEmrGnL3aRZGQixo/YyElx9QwG38edWuXdsbNizwsCIiko+PPvroe3evk9+6VGkjuBR4ozgbNmzYkFmzZiU5HBGRisXM8t5RvkPsicDMTiUkghMK2eYKwoBnHHZY3pssRUSkNGLtPmpmLYGRQLfoRpd8uftwd89y96w6dfIt2YiIyG6KLRFEI0q+Alzs7v+JKw4RkUyXtKohM3sBOAWobWFe03uAKgDu/jjh1vhawKPRkPNb3T1rd95ry5Yt5OTksHHjxqI3llhVrVqVevXqUaVKlbhDEZFIMnsN9S5i/WXAZWXxXjk5OdSsWZOGDRtS8JwnEjd3Z+XKleTk5NCoUaO4wxGRSIUYYmLjxo3UqlVLSSDFmRm1atVSyU0kxVSIRAAoCaQJ/Z1EUk/s3UdFRKRgy5fD7Nnh0a4d/PKXZf8eFaZEEKeVK1fSunVrWrduzcEHH0zdunV3vN68eXOh+86aNYvrr7++yPc4/vjjyyTWKVOm0KVLlzI5loiUnW3b4Isv4IUX4I474Kyz4KCD4JBDoFMnuPNOmDw5Oe+dkSWC0aPhrrtg6VI47DAYMgT69Cl6v4LUqlWL2bNnAzBo0CBq1KjBrbfeumP91q1bqVw5/1OdlZVFVlbRnaVmzJhR5DYikh7WrYN//3vnN/3Zs8PrDRvC+ipV4OijoXNnaN06PFq1gv32S048GZcIRo+GK66A9dE06EuWhNdQumSQV79+/ahatSqffPIJ2dnZ9OrVixtuuIGNGzey9957M2rUKJo2bcqUKVMYOnQor7/+OoMGDWLp0qUsWrSIpUuXcuONN+4oLdSoUYN169YxZcoUBg0aRO3atfnss89o164dzz33HGbGhAkTuPnmm6levTrZ2dksWrSI118veAbCH374gf79+7No0SKqVavG8OHDadmyJVOnTuWGG24AQp3+tGnTWLduHT179mTNmjVs3bqVxx57jBNPPLHsTphIBbZ9OwwfDu+8Ey76CxZA7sDP++8fLvRXXbXzon/kkbDnnuUXX8Ylgrvu2pkEcq1fH5aXZSKA0K11xowZVKpUiTVr1jB9+nQqV67MpEmT+M1vfsPLL7/8s33mzZvHO++8w9q1a2natCkDBgz4WZ/7Tz75hLlz53LooYeSnZ3Ne++9R1ZWFldeeSXTpk2jUaNG9O5daO9dAO655x7atGnD2LFjefvtt+nbty+zZ89m6NChPPLII2RnZ7Nu3TqqVq3K8OHDOeuss7jrrrvYtm0b6/OeRBHJ108/wcUXw6uvQqNG0KYN/OpXOy/69epB3H0oMi4RLF1asuWlccEFF1CpUiUAVq9ezSWXXMKCBQswM7Zs2ZLvPmeffTZ77bUXe+21FwceeCDffvst9ertOjBrhw4ddixr3bo1ixcvpkaNGhx++OE7+uf37t2b4cMLH9n73Xff3ZGMTjvtNFauXMmaNWvIzs7m5ptvpk+fPvTo0YN69erRvn17+vfvz5YtW+jevTutW7cu1bkRyQQ5OdC1K3z6KTz0ENxwQ/wX/fxkXGNxQWPWJWMsu+rVq+94/rvf/Y5TTz2Vzz77jNdee63AvvR77bXXjueVKlVi69atu7VNaQwcOJCRI0eyYcMGsrOzmTdvHieddBLTpk2jbt269OvXj2eeeaZM31Okopk5Ezp0gIUL4bXX4MYbUzMJQAYmgiFDoFq1XZdVqxaWJ9Pq1aupW7cuAE899VSZH79p06YsWrSIxYsXA/Diiy8Wuc+JJ57I6NGjgdCbqHbt2uyzzz58+eWXtGjRgjvuuIP27dszb948lixZwkEHHcTll1/OZZddxscff1zmv4NIRfHSS3DSSbDXXjBjRmj0TWUZlwj69AmNNg0ahOzcoEF4XdbtA3ndfvvt3HnnnbRp06bMv8ED7L333jz66KN07NiRdu3aUbNmTfbdd99C9xk0aBAfffQRLVu2ZODAgTz99NMAPPzwwxx99NG0bNmSKlWq0KlTJ6ZMmUKrVq1o06YNL7744o7GZBHZyR0GD4aePUOf/w8/DL1/Ul3S5ixOlqysLM87Mc0XX3zBUUcdFVNEqWPdunXUqFEDd+eaa66hcePG3HTTTXGH9TP6e0lFtGEDXHppuA/g4othxIhQIkgVZvZRQQN7ZlyJoCIbMWIErVu3pnnz5qxevZorr7wy7pBEMsLy5XDqqSEJ/PGP8PTTqZUEipJxvYYqsptuuiklSwAiFdmnn8I558DKlfDyy9CjR9wRlZxKBCIiu2ncOMjODjeMTZ+enkkAlAhERErMHe6/H7p3h6OOgn/9C9q2jTuq3adEICJSAps3h0bhO+6ACy6AqVPh0EPjjqp0lAhERIrp++/hjDNg1Ci4++7QOJz3vqR0pERQBk499VQmTpy4y7KHH36YAQMGFLjPKaecQm432M6dO7Nq1aqfbTNo0CCGDh1a6HuPHTuWzz//fMfru+++m0mTJpUk/HxpuGqRXX3+ORxzTKgGev55+P3vYY8KcgWtIL9GvHr37s2YMWN2WTZmzJhiDfwGMGHCBPbbzfFl8yaCwYMHc8YZZ+zWsURkV9u3w9tvw0UXhTaAn36CKVOgmP/aaUOJoAycf/75jB8/fsckNIsXL+abb77hxBNPZMCAAWRlZdG8eXPuueeefPdv2LAh33//PQBDhgyhSZMmnHDCCcyfP3/HNiNGjKB9+/a0atWK8847j/Xr1zNjxgzGjRvHbbfdRuvWrfnyyy/p168ff//73wGYPHkybdq0oUWLFvTv359NmzbteL977rmHtm3b0qJFC+bNm1fo7/fDDz/QvXt3WrZsybHHHsucOXMAmDp16o4JeNq0acPatWtZtmwZJ510Eq1bt+boo49m+vTppTu5IjH4+usw7EzjxnD66TBhAlx2WRg/6Nhj446u7FW4+whuvDGM912WWreGhx8ueP0BBxxAhw4deOONN+jWrRtjxozhwgsvxMwYMmQIBxxwANu2beP0009nzpw5tGzZMt/jfPTRR4wZM4bZs2ezdetW2rZtS7t27QDo0aMHl19+OQC//e1vefLJJ7nuuuvo2rUrXbp04fzzz9/lWBs3bqRfv35MnjyZJk2a0LdvXx577DFuvPFGAGrXrs3HH3/Mo48+ytChQxk5cmSBv5+Gq5ZMsGULjB8PTz4ZLvzbt8Mpp4QhI3r0gL33jjvC5FGJoIwkVg8lVgu99NJLtG3bljZt2jB37txdqnHymj59Oueeey7VqlVjn332oWvXrjvWffbZZ5x44om0aNGC0aNHM3fu3ELjmT9/Po0aNaJJkyYAXHLJJUybNm3H+h5Rh+d27drtGKiuIO+++y4XX3wxkP9w1cOGDWPVqlVUrlyZ9u3bM2rUKAYNGsS///1vatasWeixReK2YAEMHAj168O558KsWaFH0IIFYSKZPn0qdhKAClgiKOybezJ169aNm266iY8//pj169fTrl07/vvf/zJ06FBmzpzJ/vvvT79+/Qocfroo/fr1Y+zYsbRq1YqnnnqKKVOmlCre3KGsSzOM9cCBAzn77LOZMGEC2dnZTJw4ccdw1ePHj6dfv37cfPPN9O3bt1SxipS19evDXcAjR8K0aVCpEpx9dugW2rkzFDCzbIWlEkEZqVGjBqeeeir9+/ffURpYs2YN1atXZ9999+Xbb7/ljTfeKPQYJ510EmPHjmXDhg2sXbuW1157bce6tWvXcsghh7Bly5YdQ0cD1KxZk7Vr1/7sWE2bNmXx4sUsXLgQgGeffZaTTz55t343DVctFcXHH8PVV4cJ4fv2DW0B//M/8NVX8I9/hElkMi0JQAUsEcSpd+/enHvuuTuqiHKHbT7yyCOpX78+2dnZhe7ftm1bevbsSatWrTjwwANp3779jnX33nsvxxxzDHXq1OGYY47ZcfHv1asXl19+OcOGDdvRSAxQtWpVRo0axQUXXMDWrVtp3749V1111W79XoMGDaJ///60bNmSatWq7TJc9TvvvMMee+xB8+bN6dSpE2PGjOGBBx6gSpUq1KhRQxPYSEpYtgzOOw/efx+qVoXzzw/f/k8+OXUniylPSRuG2sz+BnQBvnP3n43IbWYG/BnoDKwH+rl7kV8fNQx1+tPfS8rT11/Daaft/Pb/q1+FCeMzTVzDUD8FdCxkfSegcfS4AngsibGISAb66qvwrf+bb+DNN+G66zIzCRQlaYnA3acBPxSySTfgGQ8+APYzs0OSFY+IZJbFi0MSWLEC/vlPOOGEuCNKXXE2FtcFvkp4nRMt+xkzu8LMZpnZrBUrVuR7sHSbaS1T6e8k5eHLL0MS+PFHmDy5Yt4EVpbSoteQuw939yx3z6pTp87P1letWpWVK1fqIpPi3J2VK1dStWrVuEORCmzBgpAE1q0Lw0Nk5VsrLoni7DX0NVA/4XW9aFmJ1atXj5ycHAoqLUjqqFq1KvXq1Ys7DKmg5s0LDcNbtoSbwQq4iV/yiDMRjAOuNbMxwDHAandftjsHqlKlCo0aNSrT4EQkvcydG5KAWRgYrnnzuCNKH0lLBGb2AnAKUNvMcoB7gCoA7v44MIHQdXQhofvor5MVi4hUbHPmhMHhqlQJ1UFHHhl3ROklaYnA3QsdqNVDhf41yXp/EckMn3wSJoupVi0kgcaN444o/aRFY7GISH5mzgzVQTVqhCkjlQR2jxKBiKSlDz4IJYH99w9J4PDD444ofSkRiEjaee89OPNMOPDAkAQaNow7ovSmRCAiaWXqVDjrLDj00NA7qH79IneRIigRiEjamDwZOnWCBg1CEqib71gEUlJKBCKSFiZOhC5d4Igjws1iBx8cd0QVhxKBiKS811+Hbt3C/QFvvx3aBqTsKBGISEp7+eUwl3DLlqFqqHbtuCOqeJQIRCRlPf889OwJHTqEoaQPOCDuiComJQIRSUmjRoXZxE48MbQP7Ltv3BFVXEoEIpJyHn8c+veHX/4Sxo8Pdw5L8igRiEhKefhhGDAAzjkH/vGPMIaQJJcSgYikjD/9CW66Cc47D/7+d9AcRuVDiUBEYucOgwbBnXfCRRfBmDGw555xR5U54pyYRkQE95AA7rsPfv1rGDECKlWKO6rMokQgIrFxhxtvhGHDQrvAX/8Ke6ieotzplItILLZvDxf/YcNCMnjkESWBuOi0i0i527YtdA994olQLfTgg2GuYYmHqoZEpFxt2QJ9+4YG4d//Hn73OyWBuCkRiEi52bwZevWCV18NjcO33x53RAJKBCJSTjZuhPPPD3cK//nPcP31cUckudRGICJJ9+23YS6B8ePD8BFKAqlFiUBEksY9tAU0bw7Tp8PTT8OVV8YdleSlRCAiSbF8eRgqondv+MUv4JNPQiOxpB4lAhEpU+5hHoHmzWHCBLj/fnjvPWjWLO7IpCBJTQRm1tHM5pvZQjMbmM/6w8zsHTP7xMzmmFnnZMYjIsm1fDn06AF9+kCTJjB7Ntx2G1RWt5SUlrREYGaVgEeATkAzoLeZ5f1O8FvgJXdvA/QCHk1WPCKSPO4wenT41v/mmzB0KLz7bphjWFJfMksEHYCF7r7I3TcDY4BuebZxYJ/o+b7AN0mMR0SSYNky6N49zCZ25JGhFHDLLRo4Lp0kMxHUBb5KeJ0TLUs0CPiVmeUAE4Dr8juQmV1hZrPMbNaKFSuSEauIlJA7PPtsKAW89Rb87/+GnkFNm8YdmZRU3I3FvYGn3L0e0Bl41sx+FpO7D3f3LHfPqlOnTrkHKSK7+uYb6No19AJq1gw+/RRuvlmlgHSVzETwNVA/4XW9aFmiS4GXANz9faAqUDuJMYlIKbjDM8+EHkGTJoXB4qZNCw3Dkr6SmQhmAo3NrJGZ7UloDB6XZ5ulwOkAZnYUIRGo7kckBX39dZhH+JJLQiKYMydMK6lSQPpLWiJw963AtcBE4AtC76C5ZjbYzLpGm90CXG5mnwIvAP3c3ZMVk4iU3OLFodrnqKPg7bfD5PJTp0LjxnFHJmUlqb173X0CoRE4cdndCc8/B7KTGYOIlJw7zJgBDz0URgrdYw+44AIYPBiOOCLu6KSs6TYPEdlhyxZ4+eVQ9z9zJuy3X7gh7NproV69uKOTZFEiEBF+/DFMGv+Xv0BOTqj2eeSR0B5QvXrc0UmyKRGIZLAFC8LcAE89BT/9BKeeCo89Bp07a/7gTKJEIJJh3ENj70MPwWuvhXGALrooTCDfunXc0UkclAhEMsTmzWFugIceCsNA1K4Nv/0tXH01HHxw3NFJnJQIRDLAu++GeQFycsKdwCNGhBFC99477sgkFSgRiFRg7vDEE3DdddCoURgZ9MwzwSzuyCSVKBGIVFCbNoVunyNHhsbf0aNDd1CRvNQvQKQC+uYbOOWUkAR++9vQKKwkIAVRiUCkgpkxI8wVvHZtuDmsR4+4I5JUpxKBSAUyfHgoCVSvDh98oCQgxaNEIFIBbNoEV14ZHqefHoaHOProuKOSdKFEIJLmli0LdwQPHw533gmvvw777x93VJJO1EYgksbefz+0B6xeDS+9FEYIFSkplQhE0tTIkXDyyeGmsA8+UBKQ3adEIJJmNm+GAQPg8stDldDMmdCiRdxRSTpTIhBJI8uXw2mnweOPw+23w4QJcMABcUcl6U5tBCJp4sMPQ3fQVavC4HE9e8YdkVQUSgQiKSwnJ0wV+fLLMH06HHZYuGGsVau4I5OKRIlAJMUsXAivvBIeH34YljVvHoaKuP56qFUr3vik4lEiEImZO8ydG771v/IKzJkTlmdlwR//GKqDmjaNN0ap2JQIRGLgDrNmhQv/yy+HKSPNIDs7TBxz7rnQoEHcUUqmUCIQKSfbtoX6/dxv/l99BZUqhV5AN98M3btrpjCJhxKBSJJt3QpPPgmDB4fhoffaK0wOc++9cM456v4p8VMiEEkS99DP/7bb4Isv4IQT4MEHwyQxNWvGHZ3ITsW6oczMqpvZHtHzJmbW1cyqJDc0kfQ1ezaccQZ06RJKBK++CtOmhb7/SgKSaop7Z/E0oKqZ1QXeAi4GnipqJzPraGbzzWyhmQ0sYJsLzexzM5trZs8XN3CRVJSTA/36Qdu28OmnMGxY6BHUvbvmCZbUVdyqIXP39WZ2KfCou99vZrML3cGsEvAI8EsgB5hpZuPc/fOEbRoDdwLZ7v6jmR24e7+GSLzWroX77gtVP9u2heqgO+/U9JCSHopbIjAzOw7oA4yPllUqYp8OwEJ3X+Tum4ExQLc821wOPOLuPwK4+3fFjEckJWzdCk88AUccAUOGhG/+8+eHpKAkIOmiuIngRsI391fdfa6ZHQ68U8Q+dYGvEl7nRMsSNQGamNl7ZvaBmXXM70BmdoWZzTKzWStWrChmyCLJk9sQ3KoVXHUVNGkS7gJ+/nlo2DDu6ERKplhVQ+4+FZgKEDUaf+/u15fR+zcGTgHqAdPMrIW7r8rz/sOB4QBZWVleBu8rsttmz4Zbb4XJk0NJ4JVX1AYg6a24vYaeN7N9zKw68BnwuZndVsRuXwP1E17Xi5YlygHGufsWd/8v8B9CYhBJOV9/Db/+dWgInj17Z0PwuecqCUh6K27VUDN3XwN0B94AGhF6DhVmJtDYzBqZ2Z5AL2Bcnm3GEkoDmFltQlXRomLGJFJuJk4M1T/PPx9KAwsXwnXXwZ57xh2ZSOkVt9dQlei+ge7AX919i5kVWkXj7lvN7FpgIqFh+W9R+8JgYJa7j4vWnWlmnwPbgNvcfeVu/zYiSfD662Fe4KOOCvcDNGoUd0QiZau4ieAJYDHwKaEevwGwpqid3H0CMCHPsrsTnjtwc/QQSTmvvhpuAmvVKpQKNByEVETFqhpy92HuXtfdO3uwBDg1ybGJxOqll8KE8O3awaRJSgJScRW3sXhfM3swtwunmf0vUD3JsYnE5rnnoHdvOO44eOst2HffuCMSSZ7iNhb/DVgLXBg91gCjkhWUSJxGjYK+feHkk+HNNzU2kFR8xW0j+IW7n5fw+vdFDTEhko6eeCLcIHbmmaF9oFq1uCMSSb7ilgg2mNkJuS/MLBvYkJyQROLxl7+EJHD22fCPfygJSOYobongKuAZM8utKf0RuCQ5IYmUv6FDw0Bx3bvDiy/q/gDJLMXtNfSpu7cCWgIt3b0NcFpSIxMpJ3/8Y0gCF1wQegopCUimKW7VEADuvia6wxjU91/SnDsMGgR33QV9+oS7hqtouiXJQKWZqlKjq0jacoff/Ab+9KcwkczIkWEieZFMVJpEoFFAJS25h/GCHnwQrrgCHnsM9ihR2VikYik0EZjZWvK/4Buwd1IiEkmi7dvhhhvgr3+Fa68NI4hq5FDJdIUmAnfXrTRSYWzfHrqHjhgBt9wCDzygJCACJWwsFklX334b5hIYMSK0DSgJiOxUmjYCkZS2cmWYPWzMGJgyJZQIBg2Cu+9WEhBJpEQgFcqqVTB2bLgpbNKkMLl848ahFNCzJxx9dNwRiqQeJQJJe2vXwmuvhYv/m2/C5s3QoAHcfHO4+LdpoxKASGGUCCQtrV8P48eHi//48bBxI9StC9dcEy7+HTro4i9SXBnRWDx6NDRsGPqKN2wYXkv62bQpDAZ30UVw4IFw4YUwfTpceilMmwZLl4Z7A445RklApCQqfIlg9Ohw09D69eH1kiXhNYRhBST1bdwIf/5zuAt41aowU1ifPuGb/8kn645gkdKyMG1w+sjKyvJZs2YVe/uGDcPFP68GDWDx4jILS5Jg+3Z44YXQ0Lt0aRge+tpr4fTTNVCeAX8AABEGSURBVCaQSEmZ2UfunpXfugpfIli6tGTLJTVMnRqGgZg1KzT2jhoFp2m8W5GkqPBtBIcdVrLlEq/588OcAKecAsuXwzPPhGSgJCCSPBU+EQwZ8vOZpqpVC8sldaxYEap9mjeHt98OcwT85z9w8cUaEE4k2Sr8v1ifPjB8eGgTMAs/hw9XQ3Gq2LAB7rsPjjgCHn88NOQvXAh33gl7a1hDkXJR4dsIIFz0deFPLXkbgs85JySEo46KOzKRzJPUEoGZdTSz+Wa20MwGFrLdeWbmZpZvi7ZULFOnhr7+v/oV1K4dqoLGjVMSEIlL0hKBmVUCHgE6Ac2A3mbWLJ/tagI3AB8mKxZJDYkNwd9+C88+CzNnwqmnxh2ZSGZLZomgA7DQ3Re5+2ZgDNAtn+3uBe4DNiYxFonRV1+Fuv/EhuD580OJQA3BIvFL5r9hXeCrhNc50bIdzKwtUN/dxxd2IDO7wsxmmdmsFStWlH2kkhTffQc33RRG/3z6abj6ajUEi6Si2BqLzWwP4EGgX1HbuvtwYDiEO4uTG5mU1o8/wtChYViIDRvC5PB33x16bIlI6klmIvgaqJ/wul60LFdN4GhgioURwg4GxplZV3cv/hgSkjLWrQtzAD/wQBgTqFcv+P3voUmTuCMTkcIkMxHMBBqbWSNCAugFXJS70t1XA7VzX5vZFOBWJYH0s3EjPPFEqPv/7rvQFfTee6FVq7gjE5HiSFobgbtvBa4FJgJfAC+5+1wzG2xmXZP1vlJ+tmyBkSNDG8CNN4bZv95/P3QFVRIQSR9JbSNw9wnAhDzL7i5g21OSGYuUne3bwzzA99wTGn+PPTY0Bms8IJH0pM57UmzuYWKY1q3DndrVqoVv/zNmKAmIpDMlAinS2rXw1FPhbuDu3UObwJgx8MknoT1As4GJpLeMGGtISm7bNpg8OQwD/coroRvoEUfAk09C375QWZ8ckQojY/6dt2wJd7WedVbckaS2zz4LF//Ro+Gbb2C//eCSS8LF/9hj9e1fpCLKmETw9NNw+eXQpUvo696oUdwRpY5vvw0jgT7zTKjuqVwZOncOF/+zz4aqVeOOUESSKWMSwSWXwOrVMGgQNGsWhj++7bbMvcht3Bgaep95Bt58M1QFZWWFJNmrF9SpE3eEIlJeMqaxuEoVuOUWmDcPunYNQx60aAETJ8YdWflxh3ffDQPAHXww9OwJs2eHhDh3bhgJ9LrrlAREMk3GlAhy1a0LL74Il10G11wDHTvCeefBQw9B/fpF759O3EM///feC49Jk2DxYqhePfzOffuGIaErVYo7UhGJk7mn1xhuWVlZPmtW2YxCsWlTGBxtyJAwHPI994Q7ZKtUKZPDl7vNm0Md/7vv7rz4f/ddWLfffnDCCXDhhXDuuVCjRryxikj5MrOP3D3fyb8yOhHkWrwYbrgh1Jk3awaPPgonn1ymb5EUP/4YhnR4771w8f/Xv0LdP8Dhh0N2drj4Z2eH2b809r9I5iosEWRc1VB+GjYMd8y+9hpcf32oLunTJ5QWDj447ugC95Cwci/6770X6vXdQ9VO27Zw1VXhop+dDYccEnfEIpIuVCLIY/16+J//gfvvDz2K/vAHGDCg/G+g+u670Hib+Midk2effeC443Z+2+/QIdT7i4gURFVDu2HBArj2WnjrrTC2zqOPhotvMqxdCx99FC72//pX+LlkSVhnFqqrOnSA9u3Dhb95czXwikjJqGpoNzRuHPrXv/xyaEA+/vhQXXT00aGhtahHtWr518lv2gRz5ux60f/ii1DFA+FGt2OOCUmoQ4dQ5aOGXRFJJpUIimHdOhg8GP7yl52NscVRvXq4iNesufNi/vnnoXcPwIEHhm/5ud/2s7LUh19EkkNVQ2XEPXyjX7fu54+1a/NfnvjYsiXcxJZ78a9fX2P3iEj5UNVQGTELDchVq0Lt2kVvLyKSDtSzXEQkwykRFMPo0eFegz32CD9Hj447IhGRsqOqoSKMHh0GaVu/PrxesiS8htCLSEQk3alEUIS77tqZBHKtXx+Wi4hUBEoERVi6tGTLRUTSjRJBEQ47rGTLRUTSjRJBEYYMCXcJJ6pWLSwXEakIlAiK0KcPDB8ODRqE+wgaNAiv1VAsIhVFUhOBmXU0s/lmttDMBuaz/mYz+9zM5pjZZDNrkMx4dlefPmEI6O3bw08lARGpSJKWCMysEvAI0AloBvQ2s2Z5NvsEyHL3lsDfgfuTFY+IiOQvmSWCDsBCd1/k7puBMUC3xA3c/R13z+2c+QFQL4nxiIhIPpKZCOoCXyW8zomWFeRS4I0kxiMiIvlIiTuLzexXQBaQ70zBZnYFcAXAYeq3KSJSppJZIvgaqJ/wul60bBdmdgZwF9DV3TfldyB3H+7uWe6eVScNB+zXWEUiksqSWSKYCTQ2s0aEBNALuChxAzNrAzwBdHT375IYS2w0VpGIpLqklQjcfStwLTAR+AJ4yd3nmtlgM+sabfYAUAP4PzObbWbjkhVPXDRWkYikOs1QlmR77LFzPuJEZuG+BBGR8lDYDGW6szjJNFaRiKQ6JYIk01hFIpLqlAiSTGMViUiqUyIoB6Udq0jdT0UkmVLihjIpmLqfikiyqUSQ4tT9VESSTYkgxWmqTBFJNiWCFFcW3U/VxiAihVEiSHGl7X6a28awZEm4sS23jUHJQERyKRGkuNJ2P1Ubg4gURUNMVHAa4kJEQENMZDQNcSEiRVEiqODKaogLNTiLVFxKBBVcWQxxoQZnkYpNbQRSpIYNw8U/rwYNwpAZIpL61EYgpVIWN7WpakkkdSkRSJFK2+CsqiWR1KZEIEUqbYOz7mUQSW1KBFKk0jY4q2pJJLVpGGoplj59dn/Y68MOy7+xuaRVSxqKWyQ5VCKQpEuFqiWVKEQKpkQgSRd31ZIaq0UKp0Qg5aI003WWtteSShQihVMikJRX2qqlVChRKJFIKlMikJRX2qqluEsUSiSS8tw9rR7t2rVzkZJ47jn3atXcw2U4PKpVC8uLw2zXfXMfZsXbv0GD/Pdv0KB84s89RoMGIeYGDUq2byrsL6UHzPICrqtJvWgDHYH5wEJgYD7r9wJejNZ/CDQs6phKBLI7SnMhKu2FPN0TSdz75x4jnRNZ3Pu7x5QIgErAl8DhwJ7Ap0CzPNtcDTwePe8FvFjUcZUIpLyV9kKW7okk7v3jTkTpvn+uuBLBccDEhNd3Anfm2WYicFz0vDLwPdGIqAU9lAgkDqX5RpbuiSTu/eNOROm+f67CEkEyG4vrAl8lvM6JluW7jbtvBVYDtfIeyMyuMLNZZjZrxYoVSQpXpGCl6f5a2sbu0vaaKm1jedz7l7bXV6bvXxxp0WvI3Ye7e5a7Z9WpUyfucERKLJ0TSdz7x52I0n3/YimoqFDaB6oaEkkZcTdWxlm1lun75yKmNoLKwCKgETsbi5vn2eYadm0sfqmo4yoRiGSedE5kqbC/e+GJIKlTVZpZZ+BhQg+iv7n7EDMbHAU0zsyqAs8CbYAfgF7uvqiwY2qqShGRkitsqsqkDkPt7hOACXmW3Z3wfCNwQTJjEBGRwqVFY7GIiCSPEoGISIZTIhARyXBKBCIiGS6pvYaSwcxWAPnMgJsSahPuhUhVqR4fpH6Miq90FF/plCa+Bu6e7x25aZcIUpmZzSqoe1YqSPX4IPVjVHylo/hKJ1nxqWpIRCTDKRGIiGQ4JYKyNTzuAIqQ6vFB6seo+EpH8ZVOUuJTG4GISIZTiUBEJMMpEYiIZDglghIys/pm9o6ZfW5mc83shny2OcXMVpvZ7Ohxd37HSmKMi83s39F7/2yoVguGmdlCM5tjZm3LMbamCedltpmtMbMb82xT7ufPzP5mZt+Z2WcJyw4ws3+a2YLo5/4F7HtJtM0CM7ukHON7wMzmRX/DV81svwL2LfTzkMT4BpnZ1wl/x84F7NvRzOZHn8eB5RjfiwmxLTaz2QXsm9TzV9A1pVw/fwWNT61HgfMsHAK0jZ7XBP4DNMuzzSnA6zHGuBioXcj6zsAbgAHHAh/GFGclYDnhRpdYzx9wEtAW+Cxh2f3AwOj5QOC+fPY7gDDvxgHA/tHz/cspvjOBytHz+/KLrzifhyTGNwi4tRifgS+Bw9k5b0mz8ogvz/r/Be6O4/wVdE0pz8+fSgQl5O7L3P3j6Pla4At+PhdzqusGPOPBB8B+ZnZIDHGcDnzp7rHfKe7u0whzYiTqBjwdPX8a6J7PrmcB/3T3H9z9R+CfQMfyiM/d3/Iw1zfAB0C9sn7f4irg/BVHB2Chuy9y983AGMJ5L1OFxWdmBlwIvFDW71schVxTyu3zp0RQCmbWkDCpzof5rD7OzD41szfMrHm5BgYOvGVmH5nZFfmsrwt8lfA6h3iSWS8K/ueL8/zlOsjdl0XPlwMH5bNNqpzL/oRSXn6K+jwk07VR1dXfCqjaSIXzdyLwrbsvKGB9uZ2/PNeUcvv8KRHsJjOrAbwM3Ojua/Ks/phQ3dEK+AswtpzDO8Hd2wKdgGvM7KRyfv8imdmeQFfg//JZHff5+xkP5fCU7GttZncBW4HRBWwS1+fhMeAXQGtgGaH6JRX1pvDSQLmcv8KuKcn+/CkR7AYzq0L4g41291fyrnf3Ne6+Lno+AahiZrXLKz53/zr6+R3wKqH4nehroH7C63rRsvLUCfjY3b/NuyLu85fg29wqs+jnd/lsE+u5NLN+QBegT3Sx+JlifB6Swt2/dfdt7r4dGFHA+8Z9/ioDPYAXC9qmPM5fAdeUcvv8KRGUUFSf+CTwhbs/WMA2B0fbYWYdCOd5ZTnFV93MauY+JzQofpZns3FA36j30LHA6oQiaHkp8FtYnOcvj3FAbi+MS4B/5LPNROBMM9s/qvo4M1qWdGbWEbgd6Oru6wvYpjifh2TFl9judG4B7zsTaGxmjaJSYi/CeS8vZwDz3D0nv5Xlcf4KuaaU3+cvWS3hFfUBnEAoos0BZkePzsBVwFXRNtcCcwk9ID4Aji/H+A6P3vfTKIa7ouWJ8RnwCKG3xr+BrHI+h9UJF/Z9E5bFev4ISWkZsIVQz3opUAuYDCwAJgEHRNtmASMT9u0PLIwevy7H+BYS6odzP4ePR9seCkwo7PNQTvE9G32+5hAuaofkjS963ZnQU+bL8owvWv5U7ucuYdtyPX+FXFPK7fOnISZERDKcqoZERDKcEoGISIZTIhARyXBKBCIiGU6JQEQkwykRiETMbJvtOjJqmY2EaWYNE0e+FEklleMOQCSFbHD31nEHIVLeVCIQKUI0Hv390Zj0/zKzI6LlDc3s7WhQtclmdli0/CAL8wN8Gj2Ojw5VycxGRGPOv2Vme0fbXx+NRT/HzMbE9GtKBlMiENlp7zxVQz0T1q129xbAX4GHo2V/AZ5295aEAd+GRcuHAVM9DJrXlnBHKkBj4BF3bw6sAs6Llg8E2kTHuSpZv5xIQXRnsUjEzNa5e418li8GTnP3RdHgYMvdvZaZfU8YNmFLtHyZu9c2sxVAPXfflHCMhoRx4xtHr+8Aqrj7H8zsTWAdYZTVsR4NuCdSXlQiECkeL+B5SWxKeL6NnW10ZxPGfmoLzIxGxBQpN0oEIsXTM+Hn+9HzGYTRMgH6ANOj55OBAQBmVsnM9i3ooGa2B1Df3d8B7gD2BX5WKhFJJn3zENlpb9t1AvM33T23C+n+ZjaH8K2+d7TsOmCUmd0GrAB+HS2/ARhuZpcSvvkPIIx8mZ9KwHNRsjBgmLuvKrPfSKQY1EYgUoSojSDL3b+POxaRZFDVkIhIhlOJQEQkw6lEICKS4ZQIREQynBKBiEiGUyIQEclwSgQiIhnu/wEiglJJ2AsU1wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "acc = history.history['binary_accuracy']\n",
    "val_acc = history.history['val_binary_accuracy']\n",
    "loss = history.history['loss']\n",
    "val_loss = history.history['val_loss']\n",
    "epochs = range(1, len(acc) + 1)\n",
    "# \"bo\" 训练损失绘制成蓝色原点\n",
    "plt.plot(epochs, loss, 'bo', label='Training loss')\n",
    "# \"b\" 验证损失绘制成蓝色线\n",
    "plt.plot(epochs, val_loss, 'b', label='Validation loss')\n",
    "plt.title('Training and validation loss')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Loss')\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f47fc1d0610>"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEWCAYAAABxMXBSAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3deZxU1Z338c+XRRAhKouKtCzuy0QQWoy4oXHBJaBGo9hj3CbELYmZMcaMJjFGZ+LETHx4NCY47mLAjAYx0Wjc88Qk0iqg4tYiKIiKKIIim/yeP85tLIrqpovu6qqG7/v1uq+699ylzr1d3B9nuecqIjAzM2uqduXOgJmZtS0OHGZmVhQHDjMzK4oDh5mZFcWBw8zMiuLAYWZmRXHgsGaT9ICk01p623KSNEvSoSU4bkjaMZv/taQfNmXb9fieGkkPrW8+zRojP8excZL0cc5iF2AZ8Fm2/M2IGN/6uaockmYB/xIRD7fwcQPYKSLqWmpbSf2BN4COEbGyJfJp1pgO5c6AlUdEdK2fb+wmKamDb0ZWKfx7rAyuqrI1SBouaY6k70t6B7hZ0paS/iBpvqQPs/mqnH0el/Qv2fzpkv6fpKuzbd+QdOR6bjtA0pOSFkt6WNJ1ku5oIN9NyeNPJf01O95DknrmrD9V0mxJCyRd0sj12UfSO5La56QdJ2l6Nj9U0t8kLZQ0T9K1kjZp4Fi3SLoiZ/l72T5vSzozb9ujJT0naZGktyRdlrP6yexzoaSPJe1bf21z9h8maYqkj7LPYU29NkVe5+6Sbs7O4UNJk3LWjZI0NTuH1yWNyNLXqBaUdFn931lS/6zK7ixJbwKPZum/y/4OH2W/kT1y9t9U0i+yv+dH2W9sU0l/lPStvPOZLum4QudqDXPgsEK2AboD/YAxpN/JzdlyX+BT4NpG9t8HeAXoCfwXcKMkrce2dwJPAz2Ay4BTG/nOpuTxFOAMYCtgE+BCAEm7A9dnx982+74qCoiIfwCfAIfkHffObP4z4LvZ+ewLfBk4t5F8k+VhRJafw4CdgPz2lU+ArwNbAEcD50g6Nlt3YPa5RUR0jYi/5R27O/BHYGx2bv8N/FFSj7xzWOvaFLCu63w7qepzj+xYv8zyMBS4Dfhedg4HArMauh4FHATsBhyRLT9Auk5bAc8CuVWrVwNDgGGk3/FFwCrgVuCf6zeSNBDoQ7o2VoyI8LSRT6R/wIdm88OB5UDnRrYfBHyYs/w4qaoL4HSgLmddFyCAbYrZlnRTWgl0yVl/B3BHE8+pUB4vzVk+F/hTNv8jYELOus2ya3BoA8e+Argpm+9Guqn3a2DbC4Df5ywHsGM2fwtwRTZ/E/CznO12zt22wHGvAX6ZzffPtu2Qs/504P9l86cCT+ft/zfg9HVdm2KuM9CbdIPessB2v6nPb2O/v2z5svq/c865bd9IHrbIttmcFNg+BQYW2K4z8CGp3QhSgPlVa/972xAmlziskPkRsbR+QVIXSb/Jiv6LSFUjW+RW1+R5p34mIpZks12L3HZb4IOcNIC3GspwE/P4Ts78kpw8bZt77Ij4BFjQ0HeRShfHS+oEHA88GxGzs3zsnFXfvJPl4z9IpY91WSMPwOy889tH0mNZFdFHwNlNPG79sWfnpc0m/W+7XkPXZg3ruM7bkf5mHxbYdTvg9Sbmt5DV10ZSe0k/y6q7FvF5yaVnNnUu9F3Zb3oi8M+S2gGjSSUkK5IDhxWS39Xu34BdgH0i4gt8XjXSUPVTS5gHdJfUJSdtu0a2b04e5+UeO/vOHg1tHBEzSDfeI1mzmgpSldfLpP/VfgH49/XJA6nEletOYDKwXURsDvw657jr6hr5NqlqKVdfYG4T8pWvsev8FulvtkWB/d4CdmjgmJ+QSpv1timwTe45ngKMIlXnbU4qldTn4X1gaSPfdStQQ6pCXBJ51XrWNA4c1hTdSMX/hVl9+Y9L/YXZ/+BrgcskbSJpX+ArJcrj/wLHSNo/a8i+nHX/27gT+A7pxvm7vHwsAj6WtCtwThPzcBdwuqTds8CVn/9upP/NL83aC07JWTefVEW0fQPHvh/YWdIpkjpIOgnYHfhDE/OWn4+C1zki5pHaHn6VNaJ3lFQfWG4EzpD0ZUntJPXJrg/AVODkbPtq4IQm5GEZqVTYhVSqq8/DKlK1339L2jYrneyblQ7JAsUq4Be4tLHeHDisKa4BNiX9b+7vwJ9a6XtrSA3MC0jtChNJN4xC1juPEfEicB4pGMwj1YPPWcduvyU12D4aEe/npF9IuqkvBm7I8tyUPDyQncOjQF32metc4HJJi0ltMnfl7LsEuBL4q1Jvri/lHXsBcAyptLCA1Fh8TF6+m2pd1/lUYAWp1PUeqY2HiHia1Pj+S+Aj4Ak+LwX9kFRC+BD4CWuW4Aq5jVTimwvMyPKR60LgeWAK8AFwFWve624DvkhqM7P14AcArc2QNBF4OSJKXuKxDZekrwNjImL/cuelrXKJwyqWpL0l7ZBVbYwg1WtPWtd+Zg3JqgHPBcaVOy9tmQOHVbJtSF1FPyY9g3BORDxX1hxZmyXpCFJ70LusuzrMGuGqKjMzK4pLHGZmVpSNYpDDnj17Rv/+/cudDTOzNuWZZ555PyJ65advFIGjf//+1NbWljsbZmZtiqT8EQcAV1WZmVmRHDjMzKwoDhxmZlYUBw4zMyuKA4eZmRWlpIFD0k2S3pP0QgPrJWmspLrsFY6Dc9adJum1bDotJ32IpOezfcY28mY5M9tIjR8P/ftDu3bpc/z4de1RWfs3V8m/v5RviSINOT0YeKGB9UeRhmEW8CXgH1l6d2Bm9rllNr9ltu7pbFtl+x65rnwMGTIkzKzp7rgjol+/CCl93nFH29n/jjsiunSJgM+nLl2afoxy719/jHKdfy6gNgrduwsltuREeslKQ4HjN8DonOVXSK+fHA38Jn+7bN3LOelrbNfQ5MBhG5uN+cbbr9+a+9ZP/fq1jf3Lff65KjVw/AHYP2f5EaCaNJ5+7juQf5ilVQMP56QfAPyhgWOPIb0IqLZv377FXzGzNqrcN55y7y8V3l9qG/uX+/xzNRQ4NtjG8YgYFxHVEVHdq9daT8ybVbTm1FFfcgksWbJm2pIlKb0p3nyzuPRK279v/kt315FeafuX+/ybotyBYy5rvme5KktrLL2qQLpZRWnOjX/8eBgzBmbPTv9XnD07LTf1GOW+8ZR7/yuvhC5d1kzr0iWlt4X9y33+TVKoGNKSE41XVR3Nmo3jT2fp3YE3SA3jW2bz3bN1+Y3jR60rD27jsNbU1quKyt1GUe7G5XLvXwnnX49ytHGQ3ss8j/QO4jnAWcDZwNnZegHXAa+T3hFcnbPvmaR3L9cBZ+SkVwMvZPtcS/ZOkcYmBw5rTeWuo66EG0+592/rKuX8GwocG8WLnKqrq8Oj41pradcu3a7zSbBq1br3798/VU/l69cPZs1qWh7Gj09tGm++mao4rrwSamqatq9ZPUnPRER1fnq52zjMNjiVUEddU5OCzKpV6dNBw1qSA4dZAc1p3G7ujb+mBsaNSyUMKX2OG+ebv1WOjeJFTmbFqO/VVN+ltb5XEzTt5l2/TXOqimpqHCiscrmNwyxPS7QxmG0I3MZhG5XmVDU19zkIsw2dA4dtcJr7AF1rPHlr1pY5cNgGp7lDbrTKk7dmbZgDh21wmlvV5F5NZo1zryrb4PTtW7hxu5iqJvdqMmuYSxy2wXFVk1lpOXDYBsdVTWal5aoq2yC5qsmsdFziMDOzojhwWEVqzgN8ZlZarqqyitPcsaLMrLRc4rCK09wH+MystBw4rOJ4rCizyubAYRXHY0WZVbaSBg5JIyS9IqlO0sUF1veT9Iik6ZIel1SVpR8saWrOtFTSsdm6WyS9kbNuUCnPwVqfH+Azq2wlCxyS2gPXAUcCuwOjJe2et9nVwG0RsSdwOfCfABHxWEQMiohBwCHAEuChnP2+V78+IqaW6hxs/TWnV5Qf4DOrbKXsVTUUqIuImQCSJgCjgBk52+wO/Gs2/xgwqcBxTgAeiIglBdZZBWqJXlF+gM+scpWyqqoP8FbO8pwsLdc04Phs/jigm6QeeducDPw2L+3KrHrrl5I6tVSGrWW4V5TZhq3cjeMXAgdJeg44CJgLfFa/UlJv4IvAgzn7/ADYFdgb6A58v9CBJY2RVCupdv78+SXKvhXiXlFmG7ZSBo65wHY5y1VZ2moR8XZEHB8RewGXZGkLczb5GvD7iFiRs8+8SJYBN5OqxNYSEeMiojoiqnv16tUyZ2RN4l5RZhu2UgaOKcBOkgZI2oRU5TQ5dwNJPSXV5+EHwE15xxhNXjVVVgpBkoBjgRdKkHdrBveKMtuwlSxwRMRK4HxSNdNLwF0R8aKkyyWNzDYbDrwi6VVga2D1rUVSf1KJ5Ym8Q4+X9DzwPNATuKJU52Drx72izDZsiohy56Hkqquro7a2ttzZMDNrUyQ9ExHV+enlbhw3M7M2xoHDzMyK4sBhZmZFceAwM7OiOHBYQX4Dn5k1xG8AtLX4DXxm1hiXOGwtHmvKzBrjwGFr8VhTZtYYBw5bi8eaMrPGOHDYWjzWlJk1xoHD1uKxpsysMe5VZQX5DXxm1hCXOMzMrCgOHBsoP8BnZqXiqqoNkB/gM7NScoljA+QH+MyslBw4NkB+gM/MSsmBYwPkB/jMrJQcODZAfoDPzEqppIFD0ghJr0iqk3RxgfX9JD0iabqkxyVV5az7TNLUbJqckz5A0j+yY06UtEkpz6Et8gN8ZlZKiojSHFhqD7wKHAbMAaYAoyNiRs42vwP+EBG3SjoEOCMiTs3WfRwRXQsc9y7gnoiYIOnXwLSIuL6xvFRXV0dtbW2LnZuZ2cZA0jMRUZ2fXsoSx1CgLiJmRsRyYAIwKm+b3YFHs/nHCqxfgyQBhwD/myXdChzbYjk2M7N1KmXg6AO8lbM8J0vLNQ04Pps/DugmqUe23FlSraS/S6oPDj2AhRGxspFjAiBpTLZ/7fz585t7LmZmlil34/iFwEGSngMOAuYCn2Xr+mVFpFOAayTtUMyBI2JcRFRHRHWvXr1aNNNmZhuzUj45PhfYLme5KktbLSLeJitxSOoKfDUiFmbr5mafMyU9DuwF3A1sIalDVupY65hmZlZapSxxTAF2ynpBbQKcDEzO3UBST0n1efgBcFOWvqWkTvXbAPsBMyK15D8GnJDtcxpwbwnPwczM8pQscGQlgvOBB4GXgLsi4kVJl0samW02HHhF0qvA1kD9kwa7AbWSppECxc9yemN9H/hXSXWkNo8bS3UOZma2tpJ1x60k7o5rZla8cnTHNTOzDZADh5mZFcWBo0L5RUxmVqn8IqcK5BcxmVklc4mjAvlFTGZWyRw4KpBfxGRmlcyBowL5RUxmVskcOCqQX8RkZpXMgaMC+UVMZlbJ3KuqQtXUOFCYWWVyicPMzIriwGFmZkVx4DAzs6I4cJiZWVEcOMzMrCgOHGZmVhQHDjMzK4oDh5mZFaWkgUPSCEmvSKqTdHGB9f0kPSJpuqTHJVVl6YMk/U3Si9m6k3L2uUXSG5KmZtOgUp6DmZmtqWSBQ1J74DrgSGB3YLSk3fM2uxq4LSL2BC4H/jNLXwJ8PSL2AEYA10jaIme/70XEoGyaWqpzMDOztZWyxDEUqIuImRGxHJgAjMrbZnfg0Wz+sfr1EfFqRLyWzb8NvAf0KmFezcysiUoZOPoAb+Usz8nSck0Djs/mjwO6SeqRu4GkocAmwOs5yVdmVVi/lNSp0JdLGiOpVlLt/Pnzm3MeZmaWo9yN4xcCB0l6DjgImAt8Vr9SUm/gduCMiFiVJf8A2BXYG+gOfL/QgSNiXERUR0R1r14urJiZtZRSjo47F9guZ7kqS1stq4Y6HkBSV+CrEbEwW/4C8Efgkoj4e84+87LZZZJuJgUfMzNrJaUscUwBdpI0QNImwMnA5NwNJPWUVJ+HHwA3ZembAL8nNZz/b94+vbNPAccCL5TwHMzMLE/JAkdErATOBx4EXgLuiogXJV0uaWS22XDgFUmvAlsD9e+4+xpwIHB6gW634yU9DzwP9ASuKNU5mJnZ2hQR5c5DyVVXV0dtbW25s2Fm1qZIeiYiqvPTy904bmZmbYwDh5mZFcWBw8zMitKkwCFps/reT5J2ljRSUsfSZq1tGz8e+veHdu3S5/jx5c6RmVnLaGqJ40mgs6Q+wEPAqcAtpcpUWzd+PIwZA7NnQ0T6HDPGwcPMNgxNDRyKiCWkh/V+FREnAnuULltt2yWXwJIla6YtWZLSzczauiYHDkn7AjWkp7kB2pcmS23fm28Wl25m1pY0NXBcQHqy+/fZQ3zbk0aztQL69i0u3cysLWlS4IiIJyJiZERclTWSvx8R3y5x3tqsK6+ELl3WTOvSJaWbmbV1Te1VdaekL0jajDQ21AxJ3ytt1tqumhoYNw769QMpfY4bl9LNzNq6plZV7R4Ri0iDCj4ADCD1rLIG1NTArFmwalX6dNAwsw1FUwNHx+y5jWOByRGxAtjwB7kyM7O1NDVw/AaYBWwGPCmpH7CoVJkyM7PK1aQXOUXEWGBsTtJsSQeXJktmZlbJmto4vrmk/65/h7ekX5BKH2ZmtpFpalXVTcBi0guWvkaqprq5VJkyM7PK1dR3ju8QEV/NWf6JpKmlyJCZmVW2ppY4PpW0f/2CpP2AT0uTJTMzq2RNDRxnA9dJmiVpFnAt8M117SRphKRXJNVJurjA+n6SHpE0XdLjkqpy1p0m6bVsOi0nfYik57NjjpWkJp6DmZm1gKYOOTItIgYCewJ7RsRewCGN7SOpPXAdcCSwOzBa0u55m10N3BYRewKXA/+Z7dsd+DGwDzAU+LGkLbN9rge+AeyUTSOacg5mZtYyinoDYEQsyp4gB/jXdWw+FKiLiJkRsRyYAIzK22Z34NFs/rGc9UcAf46IDyLiQ+DPwAhJvYEvRMTfIyKA20gPJZqZWStpzqtj11VF1Ad4K2d5TpaWaxrpHR8AxwHdJPVoZN8+2XxjxzQzsxJqTuBoiSFHLgQOkvQccBAwF/isBY6LpDH1z53Mnz+/JQ5pZmasozuupMUUDhACNl3HsecC2+UsV2Vpq0XE22QlDkldga9GxEJJc4Hhefs+nu1flZe+xjFzjj0OGAdQXV3tcbXMzFpIoyWOiOgWEV8oMHWLiHU9AzIF2EnSAEmbACcDk3M3kNQze78HpBdF3ZTNPwgcLmnLrFH8cODBiJgHLJL0paw31deBe4s6YzMza5bmVFU1KiJWAueTgsBLwF3Z2wMvlzQy22w48IqkV4GtgSuzfT8AfkoKPlOAy7M0gHOB/wHqgNdJw7ybmVkrUeqctGGrrq6O2tracmfDzKxNkfRMRFTnp5esxGFmZhsmBw4zMyuKA4eZmRXFgcPMzIriwGFmZkVx4DAzs6I09UVOth6WLoUZM2CTTaBTp8JTO4duM2tjHDhK5JNPYNgwmD698e06dGg4qHTqBH36wGGHweGHww47gN8+Ymbl5sBRAhHwjW/ACy/AtdfCVlvBsmXrN02dCpMmpeMOGJACyOGHwyGHwBZblPc8zWzj5MBRAv/n/8Bvfwv/8R9w3nnNO1YEvP46PPRQmu68E37zm1TFNXTo54Fk6FDo2LFl8m9m1hgPOdLCnngCvvxlGDkS7r675auWVqyAf/zj80AyZQqsWgVf+EIqhdQHkh12aNnvNbONT0NDjjhwtKA5c2DIENhyS3j66XQzL7UPP4RHH01B5MEHYfbslL799qltZNCg1I4ipVJK/bSu5fq0qioYONBtK2YbIweOEgeOZctg+PDUrvH007DbbiX9uoIioK7u89LIo4/Cxx83/7g77ww1NXDKKbDjjs0/npm1DQ4cJQ4c55wDv/41/O//wle/WtKvarIVK+C991JAWbXq88/6KXe50Pxnn8G0aTB+fKqCi0htKTU1cNJJsPXW5T5DMyslB44SBo6bboKzzoLvfx9+9rOSfU1ZzZkDEyakIDJ1aqrKOvTQFESOOw66dSt3Ds2spTlwlChw1NbC/vvDAQfAAw+k9oQN3YwZKYDceSfMmgWbbpo6A9TUwBFHpAcezaztc+AoQeB4//3UGA7wzDPQs2eLf0VFi4CnnkoBZOJEWLAAuneHE09MQWS//fxkvFlb5hc5tbCVK+Hkk+Hdd1O3240taEDqabXffnDddTBvHvzhD6nEcfvtcOCB6YHFCy9MjfTLlpU7t2bWUhw41tOll8Ijj8D110P1WvF449OxIxx9dCp9vPsu3HEH7LEHjB2bnmvp3h2+8pX0JH1dXblza2bNUdLAIWmEpFck1Um6uMD6vpIek/ScpOmSjsrSayRNzZlWSRqUrXs8O2b9uq1KeQ6F3H03XHUVnH02nHFGa3975evaNVVV3X9/qr6aPBlOPz21jXzrW7DTTqlb73nnwX33tUyXYTNrPSVr45DUHngVOAyYA0wBRkfEjJxtxgHPRcT1knYH7o+I/nnH+SIwKSJ2yJYfBy6MiCY3WrRkG8dLL6UuqXvskbqodurUIofdaNTVwZ/+lB5WfPRRWLIklVb23x9GjEhVXXvu6QcOzSpBOdo4hgJ1ETEzIpYDE4BRedsEUP989ebA2wWOMzrbt+wWLUpdT7t0Sc9rOGgUb8cd4fzzU0njgw/g4YfhggtSR4Pvfz896d6nTyrJTZiQ0s2sspSyxHECMCIi/iVbPhXYJyLOz9mmN/AQsCWwGXBoRDyTd5zXgVER8UK2/DjQA/gMuBu4IgqchKQxwBiAvn37DpldPxbHelq1Kj3Yd999qW3joIOadTgrYO7c9MT7n/4Ef/5zGk4FUrD50pc+n/bc0wM6mrWGhkoc5X7qYDRwS0T8QtK+wO2S/ikiVgFI2gdYUh80MjURMVdSN1LgOBW4Lf/AETEOGAepqqq5Gb3qqjS8+S9/6aBRKvUljTPOSE+tT5mSqgP//vcUSO64I23XuXPqkJAbTPr0KW/ezTYmpQwcc4HtcparsrRcZwEjACLib5I6Az2B97L1JwO/zd0hIuZmn4sl3UmqElsrcLSkhx5KvahOPhm+851SfpPVa9/+86AA6ZmRN99MQaR+GjsWrr46re/TZ81AMmRIejAx36pVsHAhzJ/ftGmzzeDUU+HMM2HbbVvv/M0qWSmrqjqQGse/TAoYU4BTIuLFnG0eACZGxC2SdgMeAfpEREhqB7wFHBARM3OOuUVEvC+pIymoPBwRv24sL81pHJ81K92Ett023aw222y9DmMlsGxZGksrN5i88UZa16FDGtW3f//Us6s+ECxYkEozhXTrBr16rTnNnp0a8du3T92Nv/ENOPLItGy2oSvLk+NZ99prgPbATRFxpaTLgdqImJz1pLoB6EpqKL8oIh7K9h0O/CwivpRzvM2AJ4GO2TEfBv41Ihq4FSTrGzg+/TQ94DZzZhpaxCPDVr53303vK6kPJG+/nR7OzA8I+VPPnqkKrJC6OrjxRrj55nT8qqpUAjnrLOjbt3XPz6w1eciRIgNHRKprv/XW9ET00UeXKHPWZqxYkTpH3HBD6k4MqQvxN74BxxxTWQ32dXVwzz2pxPSVr6SHMCspf9Y2OHCsR+C44YZUvXHJJSXKmLVZs2enUshNN6XeYNtskx5y/Jd/Kc/bFyPSu2DuuSdN06en9E03TSXnHj1Sr8CTTkqdO1zVZk3hwNFKr461jcvKlan78Lhx8Mc/psb3L385lUKOPba0z/qsWpV6ntUHi7q69ODk/vvD8cenZ4623jqVjiZOTE/wf/JJSjvxxBREhg3zQJTWMAcOBw4rsblzUzvI//xPKpH07Jn+l7/zzulVvgMGpKk5rxReuRL+8pcUKH7/+/SdHTqkYHX88TBqVMMv2FqyJA0DM2FCCnJLl6b2mhNPTD0G997bT+zbmhw4HDislXz2WXoi/oYb0vMnixatub579xRAcoNJ/dSv39qllGXL0vHuuQfuvTf1DNt009S+cvzxqf1tyy2Ly+Pixam9ZuLE9B6ZFSvS93/taymINPU98x9/nDog5E5z534+v3hxCqBbbw1bbZU+C817FIbK5MDhwGFlEJGegH/jjTTNnPn5/BtvpO7ey5d/vr2UnkmpDyTLlqVSwuLFqaTyla+kYHHEES3XNXzhwvRw68SJKdB99lkqJZ10EgwfnoaGyQ0GuQFi8eK1j9elSzqHPn1SF+f330+90d59N1WVFbL55oUDyjbbpK7w9dNWW7l9pjU5cDhwWAVatSrdhHODSW6AWbky9dj66lfhkENK/3bF999PJZuJE+Hxx1P+6nXs+PkNvE+fNW/oucvdujVcWvnkE3jvvRRE6j/rp9zl995LAStf+/ZrB5NC+ene3dVuLcGBw4HDrCjvvJN6Z9XfqHv0aN2b8fLlKYDkl3Tyq8QKBZhOnVKeBwxID2yOHJlKUa1h0aJU/TdpUnozaFVV6mmXP22+eevkpzkcOBw4zDZIS5emN1AWamN54YU0ugDArrumzgMjR8I++7Rsldc776Rea5MmpUFQly9PD5buv39a9/rrKQjm6tFjzUCy/fafz/fuXbi3W0SqHvzwwxQwm/J5772w3XZrH6spHDgcOMw2SrNnp44A996bqt9WrkxtJccckwLJoYemdpli1dWlnm2TJsHf/pZu6ttvn7pBH3ss7LvvmsFp8eJUBfn662tPb7655lA4nTunY2277dqBoqEhcyBVJ3bvnjpLbLllmr/22jT0zvpw4HDgMNvoLVyYnru5997U6WDRotRD7bDDUhA55pgUVAqJgGef/TxYvJiNujd4cAoUxx4L//RP61edt2JFCnD5AeWdd1KVVn0QyA0IhdK6dGnZ6kQHDgcOM8uxfDk8+WQKIpMnp//1S6mkMHJkCiQ77JCem6kPFnPmpFLEgQemQDFqVOpCvaFy4HDgMLMGRKS2kPog8uyzKb1z59SGsummqQv0scemUkmPHuXNb2tx4HDgMLMmeuut1C4yY0ZqAzn88PVrB2nrKvUNgGZmFWe77eDcc8udi8rl4c3MzKwoDhxmZlYUBw4zMyuKA4eZmRWlpIFD0ghJr0iqk3RxgfV9JT0m6TlJ07N3lCOpv6RPJU3Npl/n7DNE0vPZMWibr+8AABMXSURBVMdKHsrMzKw1lSxwSGoPXAccCewOjJa0e95mlwJ3RcRewMnAr3LWvR4Rg7Lp7Jz064FvADtl04hSnYOZma2tlCWOoUBdRMyMiOXABGBU3jYB1L8PbXPg7cYOKKk38IWI+HukB1BuA45t2WybmVljShk4+gBv5SzPydJyXQb8s6Q5wP3At3LWDciqsJ6QdEDOMees45gASBojqVZS7fz585txGmZmlqvcjeOjgVsiogo4CrhdUjtgHtA3q8L6V+BOSUW9qTkixkVEdURU9+rVq8Uzbma2sSrlk+NzgdxR4KuytFxnkbVRRMTfJHUGekbEe8CyLP0ZSa8DO2f7V63jmGZmVkKlLHFMAXaSNEDSJqTG78l527wJfBlA0m5AZ2C+pF5Z4zqStic1gs+MiHnAIklfynpTfR24t4TnYGZmeUpW4oiIlZLOBx4E2gM3RcSLki4HaiNiMvBvwA2SvktqKD89IkLSgcDlklYAq4CzI6L+BZHnArcAmwIPZJOZmbUSj45rZmYFeXRcMyuLFStWMGfOHJYuXVrurFgDOnfuTFVVFR07dmzS9g4cZlZSc+bMoVu3bvTv3x8P9FB5IoIFCxYwZ84cBgwY0KR9yt0d18w2cEuXLqVHjx4OGhVKEj169CiqROjAYWYl56BR2Yr9+zhwmJlZURw4zKyijB8P/ftDu3bpc/z45h1vwYIFDBo0iEGDBrHNNtvQp0+f1cvLly9vdN/a2lq+/e1vr/M7hg0b1rxMtjFuHDezijF+PIwZA0uWpOXZs9MyQE3N+h2zR48eTJ06FYDLLruMrl27cuGFF65ev3LlSjp0KHwrrK6uprp6rd6oa3nqqafWL3NtlEscZlYxLrnk86BRb8mSlN6STj/9dM4++2z22WcfLrroIp5++mn23Xdf9tprL4YNG8Yrr7wCwOOPP84xxxwDpKBz5plnMnz4cLbffnvGjh27+nhdu3Zdvf3w4cM54YQT2HXXXampqaH+Wbn777+fXXfdlSFDhvDtb3979XFzzZo1iwMOOIDBgwczePDgNQLSVVddxRe/+EUGDhzIxRen1xvV1dVx6KGHMnDgQAYPHszrr7/esheqAS5xmFnFePPN4tKbY86cOTz11FO0b9+eRYsW8Ze//IUOHTrw8MMP8+///u/cfffda+3z8ssv89hjj7F48WJ22WUXzjnnnLWefXjuued48cUX2Xbbbdlvv/3461//SnV1Nd/85jd58sknGTBgAKNHjy6Yp6222oo///nPdO7cmddee43Ro0dTW1vLAw88wL333ss//vEPunTpwgcfpIE0ampquPjiiznuuONYunQpq1atavkLVYADh5lVjL59U/VUofSWduKJJ9K+fXsAPvroI0477TRee+01JLFixYqC+xx99NF06tSJTp06sdVWW/Huu+9SVVW1xjZDhw5dnTZo0CBmzZpF165d2X777Vc/JzF69GjGjRu31vFXrFjB+eefz9SpU2nfvj2vvvoqAA8//DBnnHEGXbp0AaB79+4sXryYuXPnctxxxwHpIb7W4qoqM6sYV14J2b1xtS5dUnpL22yzzVbP//CHP+Tggw/mhRde4L777mvwmYZOnTqtnm/fvj0rV65cr20a8stf/pKtt96aadOmUVtbu87G+3Jx4DCzilFTA+PGQb9+IKXPcePWv2G8qT766CP69EnvhLvlllta/Pi77LILM2fOZNasWQBMnDixwXz07t2bdu3acfvtt/PZZ58BcNhhh3HzzTezJGsA+uCDD+jWrRtVVVVMmjQJgGXLlq1eX2oOHGZWUWpqYNYsWLUqfZY6aABcdNFF/OAHP2CvvfYqqoTQVJtuuim/+tWvGDFiBEOGDKFbt25svvnma2137rnncuuttzJw4EBefvnl1aWiESNGMHLkSKqrqxk0aBBXX301ALfffjtjx45lzz33ZNiwYbzzzjstnvdCPDqumZXUSy+9xG677VbubJTdxx9/TNeuXYkIzjvvPHbaaSe++93vljtbqxX6OzU0Oq5LHGZmreCGG25g0KBB7LHHHnz00Ud885vfLHeW1pt7VZmZtYLvfve7FVXCaA6XOMzMrCgOHGZmVpSSBg5JIyS9IqlO0sUF1veV9Jik5yRNl3RUln6YpGckPZ99HpKzz+PZMadm01alPAczM1tTydo4JLUHrgMOA+YAUyRNjogZOZtdCtwVEddL2h24H+gPvA98JSLelvRPwINAn5z9aiLC3aTMzMqglCWOoUBdRMyMiOXABGBU3jYBfCGb3xx4GyAinouIt7P0F4FNJXXCzKxIBx98MA8++OAaaddccw3nnHNOg/sMHz6c+i78Rx11FAsXLlxrm8suu2z18xQNmTRpEjNmfP5/5R/96Ec8/PDDxWS/IpUycPQB3spZnsOapQaAy4B/ljSHVNr4VoHjfBV4NiKW5aTdnFVT/VB+tZiZNWL06NFMmDBhjbQJEyY0ONBgvvvvv58ttthivb47P3BcfvnlHHrooet1rEpS7u64o4FbIuIXkvYFbpf0TxGxCkDSHsBVwOE5+9RExFxJ3YC7gVOB2/IPLGkMMAagbylGSDOzol1wAWSvxmgxgwbBNdc0vP6EE07g0ksvZfny5WyyySbMmjWLt99+mwMOOIBzzjmHKVOm8Omnn3LCCSfwk5/8ZK39+/fvT21tLT179uTKK6/k1ltvZauttmK77bZjyJAhQHpGY9y4cSxfvpwdd9yR22+/nalTpzJ58mSeeOIJrrjiCu6++25++tOfcswxx3DCCSfwyCOPcOGFF7Jy5Ur23ntvrr/+ejp16kT//v057bTTuO+++1ixYgW/+93v2HXXXdfI06xZszj11FP55JNPALj22mtXv0zqqquu4o477qBdu3YceeSR/OxnP6Ouro6zzz6b+fPn0759e373u9+xww47rPc1L2WJYy6wXc5yVZaW6yzgLoCI+BvQGegJIKkK+D3w9YhYPch8RMzNPhcDd5KqxNYSEeMiojoiqnv16tUiJ2RmbU/37t0ZOnQoDzzwAJBKG1/72teQxJVXXkltbS3Tp0/niSeeYPr06Q0e55lnnmHChAlMnTqV+++/nylTpqxed/zxxzNlyhSmTZvGbrvtxo033siwYcMYOXIkP//5z5k6deoaN+qlS5dy+umnM3HiRJ5//nlWrlzJ9ddfv3p9z549efbZZznnnHMKVofVD7/+7LPPMnHixNVvKcwdfn3atGlcdNFFQBp+/bzzzmPatGk89dRT9O7du1nXtJQljinATpIGkALGycApedu8CXwZuEXSbqTAMV/SFsAfgYsj4q/1G0vqAGwREe9L6ggcA7T9CkOzjURjJYNSqq+uGjVqFBMmTODGG28E4K677mLcuHGsXLmSefPmMWPGDPbcc8+Cx/jLX/7Ccccdt3po85EjR65e98ILL3DppZeycOFCPv74Y4444ohG8/PKK68wYMAAdt55ZwBOO+00rrvuOi644AIgBSKAIUOGcM8996y1f7mHXy9ZiSMiVgLnk3pEvUTqPfWipMsl1V/xfwO+IWka8Fvg9EiDZ50P7Aj8KK/bbSfgQUnTgamkgHRDKfLf0u89NrPyGTVqFI888gjPPvssS5YsYciQIbzxxhtcffXVPPLII0yfPp2jjz66weHU1+X000/n2muv5fnnn+fHP/7xeh+nXv3Q7A0Ny17u4ddL+hxHRNwfETtHxA4RcWWW9qOImJzNz4iI/SJiYEQMioiHsvQrImKzLK1+ei8iPomIIRGxZ0TsERHfiYjPWjrf9e89nj0bIj5/77GDh1nb1LVrVw4++GDOPPPM1Y3iixYtYrPNNmPzzTfn3XffXV2V1ZADDzyQSZMm8emnn7J48WLuu+++1esWL15M7969WbFiBeNzbhTdunVj8eLFax1rl112YdasWdTV1QFplNuDDjqoyedT7uHX/eR4Aa313mMzaz2jR49m2rRpqwPHwIED2Wuvvdh111055ZRT2G+//Rrdf/DgwZx00kkMHDiQI488kr333nv1up/+9Kfss88+7Lfffms0ZJ988sn8/Oc/Z6+99lrjfeCdO3fm5ptv5sQTT+SLX/wi7dq14+yzz27yuZR7+HUPq15Au3appJFPSu8IMLOm87DqbYOHVW+mhnrvulevmZkDR0Gt+d5jM7O2xoGjgHK999hsQ7UxVIm3ZcX+fcr95HjFqqlxoDBrCZ07d2bBggX06NEDjxBUeSKCBQsWFPV8hwOHmZVUVVUVc+bMYf78+eXOijWgc+fOVFVVNXl7Bw4zK6mOHTsyYMCAcmfDWpDbOMzMrCgOHGZmVhQHDjMzK8pG8eS4pPnA7HLnowE9Sa/KrVTOX/M4f83j/DVPc/PXLyLWei/FRhE4Kpmk2kKP9FcK5695nL/mcf6ap1T5c1WVmZkVxYHDzMyK4sBRfuPKnYF1cP6ax/lrHueveUqSP7dxmJlZUVziMDOzojhwmJlZURw4WoGk7SQ9JmmGpBclfafANsMlfSRpajb9qJXzOEvS89l3r/W6RCVjJdVJmi5pcCvmbZec6zJV0iJJF+Rt06rXT9JNkt6T9EJOWndJf5b0Wva5ZQP7npZt85qk01oxfz+X9HL29/u9pC0a2LfR30IJ83eZpLk5f8OjGth3hKRXst/ixa2Yv4k5eZslaWoD+7bG9St4T2m132BEeCrxBPQGBmfz3YBXgd3zthkO/KGMeZwF9Gxk/VHAA4CALwH/KFM+2wPvkB5MKtv1Aw4EBgMv5KT9F3BxNn8xcFWB/boDM7PPLbP5LVspf4cDHbL5qwrlrym/hRLm7zLgwib8/V8Htgc2Aabl/1sqVf7y1v8C+FEZr1/Be0pr/QZd4mgFETEvIp7N5hcDLwF9ypuroo0Cbovk78AWknqXIR9fBl6PiLKOBBARTwIf5CWPAm7N5m8Fji2w6xHAnyPig4j4EPgzMKI18hcRD0XEymzx70DTx9FuYQ1cv6YYCtRFxMyIWA5MIF33FtVY/pReKvI14Lct/b1N1cg9pVV+gw4crUxSf2Av4B8FVu8raZqkByTt0aoZgwAekvSMpDEF1vcB3spZnkN5gt/JNPwPtpzXD2DriJiXzb8DbF1gm0q5jmeSSpCFrOu3UErnZ1VpNzVQzVIJ1+8A4N2IeK2B9a16/fLuKa3yG3TgaEWSugJ3AxdExKK81c+Sql8GAv8XmNTK2ds/IgYDRwLnSTqwlb9/nSRtAowEfldgdbmv3xoi1QlUZF93SZcAK4HxDWxSrt/C9cAOwCBgHqk6qBKNpvHSRqtdv8buKaX8DTpwtBJJHUl/4PERcU/++ohYFBEfZ/P3Ax0l9Wyt/EXE3OzzPeD3pCqBXHOB7XKWq7K01nQk8GxEvJu/otzXL/NuffVd9vlegW3Keh0lnQ4cA9RkN5a1NOG3UBIR8W5EfBYRq4AbGvjecl+/DsDxwMSGtmmt69fAPaVVfoMOHK0gqxO9EXgpIv67gW22ybZD0lDS32ZBK+VvM0nd6udJjagv5G02Gfh61rvqS8BHOUXi1tLg//TKef1yTAbqe6icBtxbYJsHgcMlbZlVxRyepZWcpBHARcDIiFjSwDZN+S2UKn+5bWbHNfC9U4CdJA3ISqAnk657azkUeDki5hRa2VrXr5F7Suv8BkvZ8u9pdS+G/UlFxunA1Gw6CjgbODvb5nzgRVIvkb8Dw1oxf9tn3zsty8MlWXpu/gRcR+rR8jxQ3crXcDNSINg8J61s148UwOYBK0h1xGcBPYBHgNeAh4Hu2bbVwP/k7HsmUJdNZ7Ri/upIddv1v8FfZ9tuC9zf2G+hlfJ3e/bbmk66AfbOz1+2fBSpF9HrrZm/LP2W+t9czrbluH4N3VNa5TfoIUfMzKworqoyM7OiOHCYmVlRHDjMzKwoDhxmZlYUBw4zMyuKA4fZepL0mdYctbfFRmqV1D93ZFazStKh3Bkwa8M+jYhB5c6EWWtzicOshWXvY/iv7J0MT0vaMUvvL+nRbBC/RyT1zdK3Vno/xrRsGpYdqr2kG7L3LTwkadNs+29n72GYLmlCmU7TNmIOHGbrb9O8qqqTctZ9FBFfBK4FrsnS/i9wa0TsSRpgcGyWPhZ4ItIAjYNJTxwD7ARcFxF7AAuBr2bpFwN7Zcc5u1QnZ9YQPzlutp4kfRwRXQukzwIOiYiZ2UB070RED0nvk4bRWJGlz4uInpLmA1URsSznGP1J70zYKVv+PtAxIq6Q9CfgY9IIwJMiG9zRrLW4xGFWGtHAfDGW5cx/xudtkkeTxg0bDEzJRmw1azUOHGalcVLO59+y+adIo7kC1AB/yeYfAc4BkNRe0uYNHVRSO2C7iHgM+D6wObBWqceslPw/FbP1t6mkqTnLf4qI+i65W0qaTio1jM7SvgXcLOl7wHzgjCz9O8A4SWeRShbnkEZmLaQ9cEcWXASMjYiFLXZGZk3gNg6zFpa1cVRHxPvlzotZKbiqyszMiuISh5mZFcUlDjMzK4oDh5mZFcWBw8zMiuLAYWZmRXHgMDOzovx/r0HwUn2mwu0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(epochs, acc, 'bo', label='Training acc')\n",
    "plt.plot(epochs, val_acc, 'b', label='Validation acc')\n",
    "plt.title('Training and validation accuracy')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Loss')\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> The dots are the training loss and accuracy, while the solid lines are the validation\n",
    "loss and accuracy. Note that your own results may vary slightly due to a different random\n",
    "initialization of your network.\n",
    "\n",
    "蓝色的点是训练损失和准确率，而蓝色的实线是验证损失和准确率。注意你自己运行的结果可能会由于网络随机初始化而产生一定差别。\n",
    "\n",
    "> As you can see, the training loss decreases with every epoch and the training accuracy\n",
    "increases with every epoch. That’s what you would expect when running gradient\n",
    "descent optimization—the quantity you are trying to minimize should get lower with\n",
    "every iteration. But that isn’t the case for the validation loss and accuracy: they seem to\n",
    "peak at the fourth epoch. This is an example of what we were warning against earlier: a\n",
    "model that performs better on the training data isn’t necessarily a model that will do\n",
    "better on data it has never seen before. In precise terms, what you are seeing is\n",
    "\"overfitting\": after the second epoch, we are over-optimizing on the training data, and we\n",
    "ended up learning representations that are specific to the training data and do not\n",
    "generalize to data outside of the training set.\n",
    "\n",
    "正如你在上面的图形可以看到，训练损失在每个epoch过后都会下降且训练准确率在每个epoch过后都会上升。这是当你运行梯度下降优化时可以预期的 - 你试图最小化的值在每次迭代后都会更低。但是对于验证损失和准确率来说这就不一样了：它们看起来在第四次epoch之后达到了极值。这个例子印证了我们之前警告过的内容：一个在训练数据上运行良好的模型并不代表它能在未知数据上很好工作。用更精确的术语来描述的话，你这里看到的是“过拟合”：第二次epoch之后，我们过度优化了训练数据，我们最终学习到了训练数据上特定的表现形式，但是无法泛化到训练集之外的数据上。\n",
    "\n",
    "> In this case, to prevent overfitting, we could simply stop training after three epochs.\n",
    "In general, there is a range of techniques you can leverage to mitigate overfitting, which\n",
    "we will cover in the next chapter.\n",
    "\n",
    "在这个例子中，我们可以简单的在三次epochs之后停止训练来防止过拟合。更通常的情况下，我们有许多的技术来减少过拟合的发生，它们将在下一章中进行介绍。\n",
    "\n",
    "> Let’s train a new network from scratch for four epochs, then evaluate it on our test\n",
    "data:\n",
    "\n",
    "让我们重新训练这个模型，只进行4个epochs，然后在我们的测试集上去评估模型的性能："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 25000 samples\n",
      "Epoch 1/4\n",
      "25000/25000 [==============================] - 1s 53us/sample - loss: 0.4426 - accuracy: 0.8214\n",
      "Epoch 2/4\n",
      "25000/25000 [==============================] - 2s 66us/sample - loss: 0.2555 - accuracy: 0.9101\n",
      "Epoch 3/4\n",
      "25000/25000 [==============================] - 1s 51us/sample - loss: 0.1979 - accuracy: 0.9304\n",
      "Epoch 4/4\n",
      "25000/25000 [==============================] - 1s 28us/sample - loss: 0.1666 - accuracy: 0.9412\n",
      "25000/25000 [==============================] - 2s 73us/sample - loss: 0.3030 - accuracy: 0.8804\n"
     ]
    }
   ],
   "source": [
    "model = models.Sequential()\n",
    "model.add(layers.Dense(16, activation='relu', input_shape=(10000,)))\n",
    "model.add(layers.Dense(16, activation='relu'))\n",
    "model.add(layers.Dense(1, activation='sigmoid'))\n",
    "model.compile(optimizer='rmsprop',\n",
    "loss='binary_crossentropy',\n",
    "metrics=['accuracy'])\n",
    "model.fit(x_train, y_train, epochs=4, batch_size=512)\n",
    "results = model.evaluate(x_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Our fairly naive approach achieves an accuracy of 88%. With state-of-the-art\n",
    "approaches, one should be able to get close to 95%.\n",
    "\n",
    "我们这个很原始的方案取得了88%的准确率。使用一系列的改进，这个准确率可以接近95%。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.4.5 使用训练好的网络生成新数据的预测\n",
    "\n",
    "> After having trained a network, you will want to use it in a practical setting. You can\n",
    "generate the likelihood of reviews being positive by using the predict method:\n",
    "\n",
    "有了训练好的网络之后，你会想将它使用在实际环境中。你可以调用predict方法来生成评论是积极的似然值："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.21216378],\n",
       "       [0.9984786 ],\n",
       "       [0.9738066 ],\n",
       "       ...,\n",
       "       [0.21477203],\n",
       "       [0.07282293],\n",
       "       [0.8112107 ]], dtype=float32)"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.predict(x_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> As you can see, the network is very confident for some samples (0.99 or more, or\n",
    "0.01 or less) but less confident for others (0.6, 0.4).\n",
    "\n",
    "你可以看到，网络在某些样本上很确定（0.99或更高，或者0.01或更低），但是在其他一些样本上就不太确定（0.6，0.4）。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.4.6 更多实验\n",
    "\n",
    "> - We were using 2 hidden layers. Try to use 1 or 3 hidden layers and see how it affects\n",
    "validation and test accuracy.\n",
    "- Try to use layers with more hidden units or less hidden units: 32 units, 64 units...\n",
    "- Try to use the mse loss function instead of binary_crossentropy .\n",
    "- Try to use the tanh activation (an activation that was popular in the early days of neural\n",
    "networks) instead of relu .\n",
    "\n",
    "- 我们使用了2个隐藏层。尝试使用1个或3个隐藏层，看看对验证和测试准确率有什么影响。\n",
    "- 尝试在层上使用更多的隐藏单元或更少的隐藏单元：32个，64个等等。\n",
    "- 尝试使用`mse`损失函数而不是`binary_crossentropy`。\n",
    "- 尝试使用`tanh`激活函数（这是一个神经网络早期很流行的激活函数）而不是`relu`。\n",
    "\n",
    "> These experiments will help convince you that the architecture choices we have made are\n",
    "all fairly reasonable, although they can still be improved!\n",
    "\n",
    "这些实验会令你确信我们前面结构的选择已经是很合理的，虽然它还可以有改进的地方！"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.4.7 小结\n",
    "\n",
    "> Here’s what you should take away from this example:\n",
    "\n",
    "> - There’s usually quite a bit of preprocessing you need to do on your raw data in order to\n",
    "be able to feed it—as tensors—into a neural network. In the case of sequences of words,\n",
    "they can be encoded as binary vectors—but there are other encoding options too.\n",
    "- Stacks of Dense layers with relu activations can solve a wide range of problems\n",
    "(including sentiment classification), and you will likely use them frequently.\n",
    "- In a binary classification problem (two output classes), your network should end with a\n",
    "Dense layer with 1 unit and a sigmoid activation, i.e. the output of your network should\n",
    "be a scalar between 0 and 1, encoding a probability.\n",
    "- With such a scalar sigmoid output, on a binary classification problem, the loss function\n",
    "you should use is binary_crossentropy .\n",
    "- The rmsprop optimizer is generally a good enough choice of optimizer, whatever your\n",
    "problem. That’s one less thing for you to worry about.\n",
    "- As they get better on their training data, neural networks eventually start overfitting and\n",
    "end up obtaining increasingly worse results on data never-seen-before. Make sure to\n",
    "always monitor performance on data that is outside of the training set.\n",
    "\n",
    "这个例子中你应该学习到下面的内容：\n",
    "\n",
    "- 你可能需要在原始数据上进行很多的预处理，才能使得这些数据能够以张量的形式输入到神经网络当中。在单词序列的场景中，它们被编码成二元矢量，当然也有其他的编码选择。\n",
    "- 使用relu激活的全连接层的堆叠可以解决很多领域的问题（包括情绪分类），你可能会经常使用它们。\n",
    "- 在一个二元分类问题中（两个输出类别），你的网络最后一层会是只有一个单元和sigmoid激活的全连接层，也就是网络的输出应该是一个0到1之间的标量，代表着概率。\n",
    "- 在一个二元分类问题中，需要标量sigmoid输出，应该使用`binary_crossentropy`作为损失函数。\n",
    "- `rmsprop`优化器通常来说都是一个好的选择，无论是哪种任务。这是你最后才需要关注的点。\n",
    "- 当模型在训练数据上获得越来越好的结果时，神经网络最终会开始过拟合，并且在未知数据上会获得越来越差的结果。确保你一直监视着模型在训练集之外的数据上的性能。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.5 多类别分类器例子：分类新闻专线\n",
    "\n",
    "> In the previous section we saw how to classify vector inputs into two mutually exclusive\n",
    "classes using a densely-connected neural network. But what happens when you have\n",
    "more than two classes?\n",
    "\n",
    "在上一节中我们看到如何使用全连接神经网络将矢量输入分到两个互斥的类别当中。但当有着多于两个类的时候会怎么样？\n",
    "\n",
    "> In this section, we will build a network to classify Reuters newswires into 46 different\n",
    "mutually-exclusive topics. Since we have many classes, this problem is an instance of\n",
    "\"multi-class classification\", and since each data point should be classified into only one\n",
    "category, the problem is more specifically an instance of \"single-label, multi-class\n",
    "classification\". If each data point could have belonged to multiple categories (in our case,\n",
    "topics) then we would be facing a \"multi-label, multi-class classification\" problem.\n",
    "\n",
    "在本节中，我们会构建一个网络来将路透社新闻专线分成46个不同的主题中。因为有多个类别，这个问题就是一个“多类别分类”的例子，而且因为每个数据点都应该被分到唯一的类别中，这个问题应该更明确的说是“单标签，多类别分类”。如果每个数据点能够从属于多个类别（在我们的例子中就是主题）那么我们面对的就是“多标签，多类别分类”问题"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.5.1 路透社数据集\n",
    "\n",
    "> We will be working with the Reuters dataset , a set of short newswires and their topics,\n",
    "published by Reuters in 1986. It’s a very simple, widely used toy dataset for text\n",
    "classification. There are 46 different topics; some topics are more represented than\n",
    "others, but each topic has at least 10 examples in the training set.\n",
    "\n",
    "我们将要使用路透社数据集来完成这个例子，这是一个由短通讯和它们的主题构成的数据集，由路透社在1986年发表。这是一个在文本分类当中广泛使用的简单数据集。它有46个不同主题；有些主题比其他的要丰富一些，但是训练集的每个主题中至少有10个样本。\n",
    "\n",
    "> Like IMDB and MNIST, the Reuters dataset comes packaged as part of Keras. Let’s\n",
    "take a look right away:\n",
    "\n",
    "和IMDB和MNIST一样，路透社数据集也被內建在Keras中。我们下面来看一下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.datasets import reuters\n",
    "\n",
    "(train_data, train_labels), (test_data, test_labels) = reuters.load_data(num_words=10000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Like with the IMDB dataset, the argument num_words=10000 restricts the data to the\n",
    "10,000 most frequently occurring words found in the data.\n",
    "\n",
    "> We have 8,982 training examples and 2,246 test examples:\n",
    "\n",
    "就像IMDB数据集，参数`num_words=10000`将数据限制在前10000个出现频率最高的单词内。\n",
    "\n",
    "我们有了8982个训练样本和2246个测试样本："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(8982,)"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2246,)"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> As with the IMDB reviews, each example is a list of integers (word indices):\n",
    "\n",
    "每个样本是一个整数列表（单词序号列表），与IMDB评论一样："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1,\n",
       " 245,\n",
       " 273,\n",
       " 207,\n",
       " 156,\n",
       " 53,\n",
       " 74,\n",
       " 160,\n",
       " 26,\n",
       " 14,\n",
       " 46,\n",
       " 296,\n",
       " 26,\n",
       " 39,\n",
       " 74,\n",
       " 2979,\n",
       " 3554,\n",
       " 14,\n",
       " 46,\n",
       " 4689,\n",
       " 4329,\n",
       " 86,\n",
       " 61,\n",
       " 3499,\n",
       " 4795,\n",
       " 14,\n",
       " 61,\n",
       " 451,\n",
       " 4329,\n",
       " 17,\n",
       " 12]"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data[10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Here’s how you can decode it back to words, in case you are curious:\n",
    "\n",
    "如果你感兴趣的话，下面是将整数列表解码回单词的方法："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'? ? ? said as a result of its december acquisition of space co it expects earnings per share in 1987 of 1 15 to 1 30 dlrs per share up from 70 cts in 1986 the company said pretax net should rise to nine to 10 mln dlrs from six mln dlrs in 1986 and rental operation revenues to 19 to 22 mln dlrs from 12 5 mln dlrs it said cash flow per share this year should be 2 50 to three dlrs reuter 3'"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word_index = reuters.get_word_index()\n",
    "reverse_word_index = dict([(value, key) for (key, value) in word_index.items()])\n",
    "' '.join([reverse_word_index.get(i - 3, '?') for i in train_data[0]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> The label associated with an example is an integer between 0 and 45: a topic index.\n",
    "\n",
    "一个样本对应的标签是一个介于0到45的整数：对应着主题的序号。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_labels[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.5.2 数据准备\n",
    "\n",
    "> We can vectorize the data with the exact same code as in our previous example:\n",
    "\n",
    "我们可以使用与上例中完全一样的代码对数据进行矢量化："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train = vectorize_sequences(train_data)\n",
    "x_test = vectorize_sequences(test_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> To vectorize the labels, there are two possibilities: we could just cast the label list as\n",
    "an integer tensor, or we could use a \"one-hot\" encoding. One-hot encoding is a widely\n",
    "used format for categorical data, also called \"categorical encoding\". For a more detailed\n",
    "explanation of one-hot encoding, you can refer to Chapter 6, Section 1. In our case,\n",
    "one-hot encoding of our labels consists in embedding each label as an all-zero vector\n",
    "with a 1 in the place of the label index, e.g.:\n",
    "\n",
    "要将标签矢量化，有两种做法：我们可以简单的将标签列表转换为整数张量，或者我们可以使用“one-hot”编码。One-hot编码在分类数据中是广泛使用的，因此也被称为“分类编码”。你可以参考第六章第1节的内容，学习有关one-hot编码更详细的说明。在我们的例子中，标签的one-hot编码是一个除了标签所在位置为1其他所有位置都为0的矢量，例如："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "def to_one_hot(labels, dimensions=46):\n",
    "    result = np.zeros((len(labels), dimensions))\n",
    "    for i, label in enumerate(labels):\n",
    "        result[i, label] = 1\n",
    "    return result\n",
    "\n",
    "one_hot_train_labels = to_one_hot(train_labels)\n",
    "one_hot_test_labels = to_one_hot(test_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Note that there is a built-in way to do this in Keras, which you have already seen in\n",
    "action in our MNIST example:\n",
    "\n",
    "注意在Keras中有內建的函数能够完成这项任务，你已经在MNIST例子中见过它了："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras import utils\n",
    "\n",
    "one_hot_train_labels = utils.to_categorical(train_labels)\n",
    "one_hot_test_labels = utils.to_categorical(test_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.5.3 构建网络\n",
    "\n",
    "> This topic classification problem looks very similar to our previous movie review\n",
    "classification problem: in both cases, we are trying to classify short snippets of text.\n",
    "There is however a new constraint here: the number of output classes has gone from 2 to\n",
    "46, i.e. the dimensionality of the output space is much larger.\n",
    "\n",
    "这个主题分类问题看起来非常类似前面的影评分类问题：两种情况中，我们都试图对简短的文本片段进行分类。但是这个问题中我们遇到了一个新的约束：输出类别的数量从2增加到了46，也就是说这个问题的输出空间在维度上远超上一个例子。\n",
    "\n",
    "> In a stack of Dense layers like what we were using, each layer can only access\n",
    "information present in the output of the previous layer. If one layer drops some\n",
    "information relevant to the classification problem, this information can never be\n",
    "recovered by later layers: each layer can potentially become an \"information bottleneck\".\n",
    "In our previous example, we were using 16-dimensional intermediate layers, but a 16-dimensional space may be too limited to learn to separate 46 different classes: such\n",
    "small layers may act as information bottlenecks, permanently dropping relevant\n",
    "information.\n",
    "\n",
    "在一个简单的全连接层堆叠的网络中，就像上一个例子那样，每个层次都仅能获取到上一个层次输出给它的信息。如果某个层次丢失了与分类问题相关信息的话，这些信息将永远无法被后续的层次复原：因此每一层都可能是潜在的“信息瓶颈”。在前面的例子中，我们使用了16维的中间层，但是一个16维的空间对于学习46个不同的分类问题来说可能存在着较大的限制：类似这种假设空间较小的层次可能就是信息的瓶颈，将会导致相关信息的永久丢失。\n",
    "\n",
    "> For this reason we will use larger layers. Let’s go with 64 units:\n",
    "\n",
    "因此我们将使用较大的层次，选择使用64单元："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = models.Sequential()\n",
    "model.add(layers.Dense(64, activation='relu', input_shape=(10000,)))\n",
    "model.add(layers.Dense(64, activation='relu'))\n",
    "model.add(layers.Dense(46, activation='softmax'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> There are two other things you should note about this architecture:\n",
    "\n",
    "> - We are ending the network with a Dense layer of size 46. This means that for each input\n",
    "sample, our network will output a 46-dimensional vector. Each entry in this vector (each\n",
    "dimension) will encode a different output class.\n",
    "- The last layer uses a softmax activation. You have already seen this pattern in the\n",
    "MNIST example. It means that the network will output a probability distribution over the\n",
    "46 different output classes, i.e. for every input sample, the network will produce a\n",
    "46-dimensional output vector where output[i] is the probability that the sample\n",
    "belongs to class i . The 46 scores will sum to 1.\n",
    "\n",
    "关于这个结构还有两个需要注意的点：\n",
    "\n",
    "- 网络中的最后一层是一个有着46单元的全连接层。这意味着对于每个输入样本，我们的网络会输出一个46维的向量。向量中的每个值（每个维度）实际上代表着不同的输出类别。\n",
    "- 最后一层使用了`softmax`激活函数。你在之前的MNIST例子中已经见过。它表示网络将会输出在46个不同输出类别上的概率值，也就是说，对于每个输入样本，网络会产生一个46维的输出向量，其中的`output[i]`表示这个样本从属于类别i的概率。全部46个概率值加起来为1。\n",
    "\n",
    "> The best loss function to use in this case is categorical_crossentropy . It\n",
    "measures the distance between two probability distributions: in our case, between the\n",
    "probability distribution output by our network, and the true distribution of the labels. By\n",
    "minimizing the distance between these two distributions, we train our network to output\n",
    "something as close as possible to the true labels.\n",
    "\n",
    "在这个场景中最好的损失函数是`categorical_crossentropy`。它衡量着两个概率分布之间的距离：在我们的例子中就是我们网络输出的概率分布与真实标签概率分布之间的距离。通过尽可能减少这两个分布之间的距离，我们将网络训练成能够输出尽可能符合真实标签的模型。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(optimizer='rmsprop',\n",
    "              loss='categorical_crossentropy',\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.5.4 验证我们的训练方法\n",
    "\n",
    "> Let’s set apart 1,000 samples in our training data to use as a validation set:\n",
    "\n",
    "下面我们将1000个样本从训练数据中抽取出来作为验证集："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_val = x_train[:1000]\n",
    "partial_x_train = x_train[1000:]\n",
    "\n",
    "y_val = one_hot_train_labels[:1000]\n",
    "partial_y_train = one_hot_train_labels[1000:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Now let’s train our network for 20 epochs:\n",
    "\n",
    "下面我们使用20次迭代来训练我们的网络："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 7982 samples, validate on 1000 samples\n",
      "Epoch 1/20\n",
      "7982/7982 [==============================] - 1s 117us/sample - loss: 2.7977 - accuracy: 0.4952 - val_loss: 1.8716 - val_accuracy: 0.5830\n",
      "Epoch 2/20\n",
      "7982/7982 [==============================] - 1s 108us/sample - loss: 1.5101 - accuracy: 0.6765 - val_loss: 1.3657 - val_accuracy: 0.7080\n",
      "Epoch 3/20\n",
      "7982/7982 [==============================] - 1s 106us/sample - loss: 1.1032 - accuracy: 0.7652 - val_loss: 1.1842 - val_accuracy: 0.7430\n",
      "Epoch 4/20\n",
      "7982/7982 [==============================] - 0s 56us/sample - loss: 0.8633 - accuracy: 0.8176 - val_loss: 1.0802 - val_accuracy: 0.7790\n",
      "Epoch 5/20\n",
      "7982/7982 [==============================] - 0s 45us/sample - loss: 0.6884 - accuracy: 0.8571 - val_loss: 1.0186 - val_accuracy: 0.7840\n",
      "Epoch 6/20\n",
      "7982/7982 [==============================] - 0s 42us/sample - loss: 0.5531 - accuracy: 0.8826 - val_loss: 0.9498 - val_accuracy: 0.8120\n",
      "Epoch 7/20\n",
      "7982/7982 [==============================] - 1s 110us/sample - loss: 0.4429 - accuracy: 0.9093 - val_loss: 0.9334 - val_accuracy: 0.8040\n",
      "Epoch 8/20\n",
      "7982/7982 [==============================] - 0s 47us/sample - loss: 0.3604 - accuracy: 0.9248 - val_loss: 0.9116 - val_accuracy: 0.8200\n",
      "Epoch 9/20\n",
      "7982/7982 [==============================] - 0s 42us/sample - loss: 0.2962 - accuracy: 0.9356 - val_loss: 0.9043 - val_accuracy: 0.8120\n",
      "Epoch 10/20\n",
      "7982/7982 [==============================] - 1s 82us/sample - loss: 0.2523 - accuracy: 0.9425 - val_loss: 0.9200 - val_accuracy: 0.8050\n",
      "Epoch 11/20\n",
      "7982/7982 [==============================] - 0s 41us/sample - loss: 0.2156 - accuracy: 0.9461 - val_loss: 0.9245 - val_accuracy: 0.8100\n",
      "Epoch 12/20\n",
      "7982/7982 [==============================] - 1s 92us/sample - loss: 0.1876 - accuracy: 0.9516 - val_loss: 0.9387 - val_accuracy: 0.8160\n",
      "Epoch 13/20\n",
      "7982/7982 [==============================] - 1s 108us/sample - loss: 0.1715 - accuracy: 0.9526 - val_loss: 0.9972 - val_accuracy: 0.8110\n",
      "Epoch 14/20\n",
      "7982/7982 [==============================] - 0s 41us/sample - loss: 0.1538 - accuracy: 0.9541 - val_loss: 0.9685 - val_accuracy: 0.8160\n",
      "Epoch 15/20\n",
      "7982/7982 [==============================] - 0s 42us/sample - loss: 0.1452 - accuracy: 0.9554 - val_loss: 0.9840 - val_accuracy: 0.8050\n",
      "Epoch 16/20\n",
      "7982/7982 [==============================] - 0s 42us/sample - loss: 0.1409 - accuracy: 0.9550 - val_loss: 1.0790 - val_accuracy: 0.7990\n",
      "Epoch 17/20\n",
      "7982/7982 [==============================] - 1s 108us/sample - loss: 0.1265 - accuracy: 0.9568 - val_loss: 1.0699 - val_accuracy: 0.8040\n",
      "Epoch 18/20\n",
      "7982/7982 [==============================] - 0s 41us/sample - loss: 0.1214 - accuracy: 0.9565 - val_loss: 1.0006 - val_accuracy: 0.8120\n",
      "Epoch 19/20\n",
      "7982/7982 [==============================] - 0s 43us/sample - loss: 0.1174 - accuracy: 0.9570 - val_loss: 1.0503 - val_accuracy: 0.8130\n",
      "Epoch 20/20\n",
      "7982/7982 [==============================] - 0s 39us/sample - loss: 0.1164 - accuracy: 0.9554 - val_loss: 1.0767 - val_accuracy: 0.8050\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(partial_x_train, partial_y_train, batch_size=512,\n",
    "                  epochs=20, validation_data=(x_val, y_val))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Let’s display its loss and accuracy curves:\n",
    "\n",
    "然后我们将损失和准确率曲线可视化："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f479b059b50>"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3deZRU5bnv8e8DNDSTzE4g3ZA4InMDCoo4iwMgjshSCYkIR0PEczUkepSbXNa5STw5hkRNcB46wVxFjh4hxgEERY2AiCB4nEARotDKZIPQ8Nw/3ipomq4e6N5V1b1/n7X2ql17qqeLzX7qHfa7zd0REZH4apDpAEREJLOUCEREYk6JQEQk5pQIRERiTolARCTmGmU6gOpq37695+fnZzoMEZE6ZfHixRvdvUN56+pcIsjPz2fRokWZDkNEpE4xszWp1qlqSEQk5pQIRERiTolARCTm6lwbgYik365du1i7di07duzIdChSidzcXDp16kROTk6V91EiEJFKrV27lpYtW5Kfn4+ZZTocScHdKSoqYu3atXTp0qXK+8WiaqiwEPLzoUGD8FpYmOmIROqWHTt20K5dOyWBLGdmtGvXrtolt3pfIigshHHjoLg4vF+zJrwHGD06c3GJ1DVKAnXDwfw71fsSwW237UsCScXFYbmIiMQgEXz2WfWWi0j2KSoqolevXvTq1YvDDz+cjh077n2/c+fOCvddtGgREydOrPQzBg4cWCuxzps3jwsvvLBWjpUu9T4RdO5cveUiUnO13S7Xrl07li5dytKlSxk/fjyTJk3a+75x48aUlJSk3LegoIBp06ZV+hkLFy6sWZB1WL1PBFOnQrNm+y9r1iwsF5Hal2yXW7MG3Pe1y9V2J40xY8Ywfvx4BgwYwK233so//vEPTj75ZHr37s3AgQP54IMPgP1/oU+ZMoWxY8cyZMgQunbtul+CaNGixd7thwwZwqWXXspxxx3H6NGjST7Jcfbs2Rx33HH07duXiRMnVvrL/+uvv2bEiBH06NGDk046iWXLlgHw6quv7i3R9O7dm61bt7J+/XoGDx5Mr169OPHEE1mwYEHtfmEVqPeNxckG4dtuC9VBnTuHJKCGYpFoVNQuV9v/79auXcvChQtp2LAhW7ZsYcGCBTRq1IiXXnqJn//85zz99NMH7LNq1Srmzp3L1q1bOfbYY5kwYcIBfe7feecdVqxYwZFHHsmgQYN4/fXXKSgo4Prrr2f+/Pl06dKFUaNGVRrfnXfeSe/evZk1axavvPIK11xzDUuXLuWuu+7innvuYdCgQWzbto3c3FymT5/Oueeey2233cbu3bspLvslRqjeJwIIJ58u/CLpkc52ucsuu4yGDRsCsHnzZq699lo+/PBDzIxdu3aVu88FF1xAkyZNaNKkCYceeihffvklnTp12m+b/v37713Wq1cvVq9eTYsWLejateve/vmjRo1i+vTpFcb32muv7U1GZ5xxBkVFRWzZsoVBgwZx8803M3r0aEaOHEmnTp3o168fY8eOZdeuXYwYMYJevXrV6LupjnpfNSQi6ZXOdrnmzZvvnf+3f/s3Tj/9dJYvX85zzz2Xsi99kyZN9s43bNiw3PaFqmxTE5MnT+aBBx5g+/btDBo0iFWrVjF48GDmz59Px44dGTNmDI899litfmZFlAhEpFZlql1u8+bNdOzYEYBHHnmk1o9/7LHH8sknn7B69WoAnnzyyUr3OfXUUylMNI7MmzeP9u3bc8ghh/Dxxx/TvXt3fvrTn9KvXz9WrVrFmjVrOOyww7juuuv40Y9+xJIlS2r9b0hFiUBEatXo0TB9OuTlgVl4nT49+urZW2+9lZ/97Gf07t271n/BAzRt2pR7772X8847j759+9KyZUtatWpV4T5Tpkxh8eLF9OjRg8mTJ/Poo48CcPfdd3PiiSfSo0cPcnJyGDp0KPPmzaNnz5707t2bJ598kp/85Ce1/jekYsnW8LqioKDA9WAakfRauXIlxx9/fKbDyLht27bRokUL3J0bbriBo48+mkmTJmU6rAOU9+9lZovdvaC87VUiEBGpovvvv59evXrRrVs3Nm/ezPXXX5/pkGpFLHoNiYjUhkmTJmVlCaCmVCIQEYk5JQIRkZhTIhARiTklAhGRmFMiEJGsd/rpp/PCCy/st+zuu+9mwoQJKfcZMmQIya7m559/Pps2bTpgmylTpnDXXXdV+NmzZs3i/fff3/v+jjvu4KWXXqpO+OXKpuGqlQhEJOuNGjWKGTNm7LdsxowZVRr4DcKooa1btz6ozy6bCH7xi19w1llnHdSxspUSgYhkvUsvvZTnn39+70NoVq9ezbp16zj11FOZMGECBQUFdOvWjTvvvLPc/fPz89m4cSMAU6dO5ZhjjuGUU07ZO1Q1hHsE+vXrR8+ePbnkkksoLi5m4cKFPPvss9xyyy306tWLjz/+mDFjxvDUU08B8PLLL9O7d2+6d+/O2LFj+e677/Z+3p133kmfPn3o3r07q1atqvDvy/Rw1bqPQESq5aabYOnS2j1mr15w992p17dt25b+/fszZ84chg8fzowZM7j88ssxM6ZOnUrbtm3ZvXs3Z555JsuWLaNHjx7lHmfx4sXMmDGDpUuXUlJSQp8+fejbty8AI0eO5LrrrgPg9ttv58EHH+THP/4xw4YN48ILL+TSSy/d71g7duxgzJgxvPzyyxxzzDFcc8013Hfffdx0000AtG/fniVLlnDvvfdy11138cADD6T8+zI9XHVkJQIzO8rM5prZ+2a2wswOGDjDzIaY2WYzW5qY7ogqHhGp20pXD5WuFvrrX/9Knz596N27NytWrNivGqesBQsWcPHFF9OsWTMOOeQQhg0btnfd8uXLOfXUU+nevTuFhYWsWLGiwng++OADunTpwjHHHAPAtddey/z58/euHzlyJAB9+/bdO1BdKq+99hpXX301UP5w1dOmTWPTpk00atSIfv368fDDDzNlyhTee+89WrZsWeGxqyLKEkEJ8K/uvsTMWgKLzexFdy/7r7TA3bOjxUREKlXRL/coDR8+nEmTJrFkyRKKi4vp27cvn376KXfddRdvv/02bdq0YcyYMSmHn67MmDFjmDVrFj179uSRRx5h3rx5NYo3OZR1TYaxnjx5MhdccAGzZ89m0KBBvPDCC3uHq37++ecZM2YMN998M9dcc02NYo2sRODu6919SWJ+K7AS6BjV54lI/daiRQtOP/10xo4du7c0sGXLFpo3b06rVq348ssvmTNnToXHGDx4MLNmzWL79u1s3bqV5557bu+6rVu3csQRR7Br1669Q0cDtGzZkq1btx5wrGOPPZbVq1fz0UcfAfD4449z2mmnHdTflunhqtPSRmBm+UBv4K1yVp9sZu8C64D/5e4HlMfMbBwwDqCznjovElujRo3i4osv3ltFlBy2+bjjjuOoo45i0KBBFe7fp08frrjiCnr27Mmhhx5Kv3799q775S9/yYABA+jQoQMDBgzYe/G/8sorue6665g2bdreRmKA3NxcHn74YS677DJKSkro168f48ePP6i/K/ks5R49etCsWbP9hqueO3cuDRo0oFu3bgwdOpQZM2bwm9/8hpycHFq0aFErD7CJfBhqM2sBvApMdfeZZdYdAuxx921mdj7wO3c/uqLjaRhqkfTTMNR1S1YNQ21mOcDTQGHZJADg7lvcfVtifjaQY2bto4xJRET2F2WvIQMeBFa6+29TbHN4YjvMrH8inqKoYhIRkQNF2UYwCLgaeM/Mkr2Ofw50BnD3PwKXAhPMrATYDlzpde2RaSIx4e4kfrdJFjuYS2hkicDdXwMqPGvc/Q/AH6KKQURqR25uLkVFRbRr107JIIu5O0VFReTm5lZrP91ZLCKV6tSpE2vXrmXDhg2ZDkUqkZubS6dOnaq1jxKBiFQqJyeHLl26ZDoMiYgGnRMRiTklAhGRmFMiEBGJOSUCEZGYUyIQEYk5JQIRkZhTIhARiTklAhGRmFMiEBGJOSUCEZGYUyIQEYk5JQIRkZhTIhARiTklAhGRmFMiEBGJOSUCEZGYUyIQEYk5JQIRkZhTIhARiTklAhGRmFMiEBGJOSUCEZGYUyIQEYk5JQIRkZhTIhARiTklAhGRmIssEZjZUWY218zeN7MVZvaTcrYxM5tmZh+Z2TIz6xNVPCIiUr5GER67BPhXd19iZi2BxWb2oru/X2qbocDRiWkAcF/iVURE0iSyEoG7r3f3JYn5rcBKoGOZzYYDj3nwJtDazI6IKiYRETlQWtoIzCwf6A28VWZVR+DzUu/XcmCywMzGmdkiM1u0YcOGqMIUEYmlyBOBmbUAngZucvctB3MMd5/u7gXuXtChQ4faDVBEJOYiTQRmlkNIAoXuPrOcTb4Ajir1vlNimYiIpEmUvYYMeBBY6e6/TbHZs8A1id5DJwGb3X19VDGJiMiBouw1NAi4GnjPzJYmlv0c6Azg7n8EZgPnAx8BxcAPIoxHRETKEVkicPfXAKtkGwduiCoGERGpnO4sFhGJOSUCEZGYi00i2LULZs8G90xHIiKSXWKTCB57DC64AN54I9ORiIhkl9gkgiuvhFatYNq0TEciIpJdYpMImjeHH/0InnoK1q7NdDQiItkjNokA4IYbYM8e+OMfMx2JiEj2iFUi6NIFhg2DP/0JduzIdDQiItkhVokAYOJE2LgRZszIdCQiItkhdong9NOhW7fQaKyupCIiMUwEZqFU8M478PrrmY5GRCTzYpcIAEaPhjZt1JVURARimgiSXUlnzoTPP698exGR+iyWiQDgX/4ltBGoK6mIxF1sE0F+PgwfHrqSbt+e6WhERDIntokAQqNxUZG6kopIvMU6EZx2GnTvrq6kIhJvsU4Eya6kS5fCa69lOhoRkcyIdSIAuOoqdSUVkXiLfSJo1gyuuw6eeQY++yzT0YiIpF/sEwHs60p6332ZjkREJP2UCIC8PBgxAqZPV1dSEYkfJYKEiRPh66/hz3/OdCQiIumlRJAweDD06KGupCISP0oECcmupMuWwfz5mY5GRCR9lAhKueoqaNtWXUlFJF6qlAjMrLmZNUjMH2Nmw8wsJ9rQ0q9pUxg3DmbNgjVrMh2NiEh6VLVEMB/INbOOwN+Bq4FHogoqkyZMCNVE9967b1lhYRikrkGD8FpYmKnoRERqX1UTgbl7MTASuNfdLwO6VbiD2UNm9pWZLU+xfoiZbTazpYnpjuqFHo3OneHii+H++6G4OFz0x40LJQT38DpunJKBiNQfVU4EZnYyMBp4PrGsYSX7PAKcV8k2C9y9V2L6RRVjidzEifDNN6Er6W23hYRQWnFxWC4iUh9UNRHcBPwMeMbdV5hZV2BuRTu4+3zg6xrGlxGnnAK9eoVG41RtBRqOQkTqiyolAnd/1d2HufuvEo3GG919Yi18/slm9q6ZzTGzlFVNZjbOzBaZ2aINGzbUwsdWLNmV9L334LDDyt+mc+fIwxARSYuq9hr6s5kdYmbNgeXA+2Z2Sw0/ewmQ5+49gd8Ds1Jt6O7T3b3A3Qs6dOhQw4+tmlGjoH17OOqoMDBdac2awdSpaQlDRCRyVa0aOsHdtwAjgDlAF0LPoYPm7lvcfVtifjaQY2bta3LM2pSbGxqFlywJF/28vFBSyMsLYxKNHp3pCEVEakdVE0FO4r6BEcCz7r4LqNFADGZ2uJlZYr5/IpaimhyztiW7kq5bB6tXw5494VVJQETqk6omgj8Bq4HmwHwzywO2VLSDmf0FeAM41szWmtkPzWy8mY1PbHIpsNzM3gWmAVe6Z9coP506wSWXhK6k336b6WhERKJhB3vtNbNG7l5Sy/FUqqCgwBctWpS2z3v99dCL6E9/ClVFIiJ1kZktdveC8tZVtbG4lZn9Ntlzx8z+g1A6qPcGDoQ+fTQqqYjUX1WtGnoI2Apcnpi2AA9HFVQ2SXYlXbEC5lZ454SISN1U1UTwPXe/090/SUz/G+gaZWDZ5IoroEMHjUoqIvVTVRPBdjM7JfnGzAYBsXmoY7Ir6bPPwvPPV769iEhdUtVEMB64x8xWm9lq4A/A9ZFFlYUmTQrDTgwbBv/5n2ovEJH6o6pDTLybuAO4B9DD3XsDZ0QaWZZp1w4WLAgjk958cygh7NyZ6ahERGquWk8oS9wNnLx/4OYI4slqzZvDX/8Kt98ODzwA55wDRVl1C5yISPXV5FGVVmtR1CENGsAvfxmeR/DmmzBgAKxcmemoREQOXk0SQaxrya+6CubNg23b4KST4IUXMh2RiMjBqTARmNlWM9tSzrQVODJNMWatk06Cf/wDunSB88+H3/9ejcgiUvdUmAjcvaW7H1LO1NLdG6UryGzWuTO89hpcdFG48WzCBNi1K9NRiYhUXU2qhiShRQuYORMmTw5jEp13HnxdJ5/NJiJxpERQSxo0gH//d3jssVBCOOkk+OCDTEclIlI5JYJadvXVYUyiTZtCMnjxxUxHJCJSMSWCCAwcCG+/HR5zOXQo3HtvpiMSEUlNiSAieXnhWQZDh8INN8CNN0JJ2p/eICJSOSWCCLVsCbNmwS23wD33hKTwP/+T6ahERPanRBCxhg3h17+Ghx4KjcjHHw9XXgnvvpvpyEREAiWCNPnBD8KD72+9FWbPDiOZXnQRvPFGpiMTkbhTIkijww4LXUzXrAnjFb3xRmhYPuMMeOkl3ZUsIpmhRJAGhYWQnx/uNcjPDyWC228PJYTf/jbcb3D22aG76X/9F+zZk+GARSRWlAgiVlgYnl2wZk34xb9mTXhfWBjuSJ40CT75JNyRvHEjjBgBPXvCn/+sXkYikh5KBBG77TYoLt5/WXFxWJ7UpElIDh98AE88ERLG6NFw3HFw//3w3XfpjVlE4kWJIGKffVb15Y0ahQSwbFnodtq2bUgQXbuGx2N++220sYpIPJnXsRbKgoICX7RoUabDqLL8/FAdVFZeXmgjqIg7vPwyTJ0ann3Qti1ccEG4H+Gcc8LjM0Xqu1274Omn4cEHw4+h5s3D1KxZ9eabNw8/qho2zPRfVLmSEli3Llw71qwJ14o1a0Jb4uWXH9wxzWyxuxeUt05DSUds6tTwq7509VCzZmF5ZczgrLPCtHBhuClt9mx4/PGwrn//kBTOOw8KCurGCS5SVUVFoWr0D3+AL76A730v/LD69lv46qvwWlwcXr/9FnbvrvyYHTrA8OHh2eNnnhmqZTPhu+9CrUDZC31yWrv2wL/n0EPh+9+PJh6VCNKgsDC0CXz2WXh+wdSpoQroYOzeDYsWwd/+BnPmhAfjuIfSwTnnhMRw7rnhpBGpi95/H6ZNCyP5bt8efgjddFM4txukqMx2h507908MZee/+SYMAvn887B1a7jz/8ILQ1IYOjR03qhtJSWwfHkYbubNN+HDD8OF/p//3H+7Bg2gY8dQU5CXFxJecj4vL1w3mjatWSwVlQiUCOq4jRvDyT1nTnhc5ldfheV9++4rLQwYENofRLLVnj3w97/D3XeH87hJkzCS78SJ0L177X7Wd9+FKteZM0N37Y0bw+ede25IChdddPDVrps3w1tvhQv/woXh4r9tW1h3xBFwwgnlX+w7doScnFr7E8uVkURgZg8BFwJfufuJ5aw34HfA+UAxMMbdl1R2XCWC1PbsgXfeCUnhb38LN6zt2QOtW4e6xaFDwy+gDh0yHalI8O23oarzd7+DVavCxfKGG0J1ajrO05KScNGeOROeeQY+/zxUsQ4ZEpLCiBHhIl0ed/j0030X/YUL4b33wvIGDaBHj3DD6KBB4TUvL1TpZkqmEsFgYBvwWIpEcD7wY0IiGAD8zt0HVHZcJYKq++abcMdyMjGsXx9O0FNOCSf5xReHk1Mk3T7/PLR5TZ8eztO+fcM9NZddBo0bZyYmd1i8OCSFmTP3PVhqwAAYORKGDQvPGSl94U9W8bRsCSefvO/C378/HHJIZv6OVDJWNWRm+cB/p0gEfwLmuftfEu8/AIa4+/qKjqlEcHDcQ2lh1qzwy2f58rC8T599SeGEEzL7i0VqX0lJeDbGiy+GC9ghh4SeM6Wnzp2jr5ZIevPNUP3z1FPhnBw5MtT/DxyYfefeypX7SgqLF++/rmvX/X/td+uW/Z01sjUR/Dfwf939tcT7l4GfuvsBV3kzGweMA+jcuXPfNeX1x5Rq+fDDkBRmzgz/OQGOPjokhJEjoV+/1A1zkr3cw7/tSy+Fi//cuaHe2izUte/YEXqo7Ny5b58GDcJDlMomiK5doUsXaN8+9UV69+5w/KKi8Jzu0lPZZZ99BitWQKtWcN114RkddaVEumZNKFV36BB++R9xRKYjqr46nwhKU4mg9q1bFxrNnnkmXDhKSuDII0P96MUXw2mnpe8Xo1Tfxo2h8fPFF8OUvFkxPz+0DZ11VugqmWwA3bMn/Jt/8smB06efHtijpUWLkBTy8kKf/tIX+E2bKh4ssXXrcP9L27bh8y+6CK69NpoeOlKxbE0EqhrKQt98E7rXPfNM+AVUXAxt2oRG5mHDwsWlffvwn7pFi+wrzsfBjh2hmid54X/nnXAxbtUqjGR79tlh+t73Du7f59tvQ6mhbIJYswZyc/dd2EtP7doduKx1a/VWyybZmgguAG5kX2PxNHfvX9kxlQjSp7g4XGieeQaeey78AiwtJydcAJJTMkGkWta+fbhAKHmktmdP6OP+zTf7pk2bwuuXX8Krr8KCBaF/faNGoZoieeEvKNCFV1LLyJ3FZvYXYAjQ3szWAncCOQDu/kdgNiEJfEToPvqDqGKRg9OsWbgLc/jwUF20ZEmoNigqCtPGjfvmi4pC97/kfKqRUxs3hsMPD1VPRx4Z6lrLm8+2hLF7N2zZEn6N79oVpp07qz+/eXP5F/nktHlzxcOQH398qF8/++xQZdeyZfq+A6m/dEOZ1Dr38Ku2bKLYsCEkknXrQlfWdevCtGnTgcdo3HhfYki+Hn54qI5q2jRUUTRtmnq+9LLGjfcllT17wsW2bINmqtfkfGV14dXRuHGobktOrVvv/768Zcm6dl345WBprCFJK7PQTTHZVbEy27fvSwxlX9etCyWNV14pP2FUNZ7c3HAB3rq14l/crVrtq+9u1y7UsyfnW7cOiSUnJ0yNG1c+X/p948bh+Lm52VXaEVEiqANqc6yibNS06b7uihXZsSMkjeRU+n1V5nfuDMmp9IW+dGOnGjclrnTaZ7nkE86So5cmn3AG9SsZVEVubpjatMl0JCL1i24ZynJVecKZiEhNKBFkueo84UxE5GAoEWS5zp2rt1xEpLqUCLLc1KmhP39pVX3CmYhIVSgRZLnRo8NQvcmxzPPywvu4NRSLSHTUa6gOGD1aF34RiY5KBCIiMadEICISc0oEIiIxp0QgIhJzSgQiIjGnRBADhYXhyWINGoTXwsJMRyQi2UTdR+s5DVonIpVRiaCe06B1IlIZJYJ6ToPWiUhllAjqOQ1aJyKVUSKo5zRonYhURomgntOgdSJSGfUaigENWiciFVGJQEQk5pQIRERiTolAqkR3J4vUX2ojkErp7mSR+k0lAqmU7k4Wqd8iTQRmdp6ZfWBmH5nZ5HLWjzGzDWa2NDH9KMp45ODo7mSR+i2yqiEzawjcA5wNrAXeNrNn3f39Mps+6e43RhWH1FznzqE6qLzlIlL3RVki6A985O6fuPtOYAYwPMLPk4jo7mSR+i3KRNAR+LzU+7WJZWVdYmbLzOwpMzsqwnjkIOnuZJH6LdONxc8B+e7eA3gReLS8jcxsnJktMrNFGzZsSGuAEoweDatXw5494VVJQKT+iDIRfAGU/oXfKbFsL3cvcvfvEm8fAPqWdyB3n+7uBe5e0KFDh0iClWjpPgSR7BVlIngbONrMuphZY+BK4NnSG5jZEaXeDgNWRhiPZEjyPoQ1a8B9330ISgYi2SGyRODuJcCNwAuEC/xf3X2Fmf3CzIYlNptoZivM7F1gIjAmqngkc3Qfgkh2M3fPdAzVUlBQ4IsWLcp0GFINDRqEkkBZZqHNQUSiZ2aL3b2gvHWZbiyWGNBT0kSymxKBRE73IYhkNyUCiVxt3IegXkci0dHoo5IWNXlKmkY/FYmWSgSS9dTrSCRaSgSS9TT6qUi0lAgk66nXkUi0lAgk69VGryM1NoukpkQgWa+mvY40xIVIxXRnsdR7+fnlP1gnLy+MpCoSB7qzWGKtNhqbVbUk9ZkSgdR7NW1sVtWS1HdKBFLv1bSxWfcxSH2nRCD1Xk0bm1W1JPWdhpiQWKjJEBedO5ff2FzdqiUNkSHZSiUCkUpkQ9WSShQSJSUCkUpkumqpNhqrlUikIkoEIlUwenS452DPnvBanSqdmvZaqmmJQolEKqNEIBKxmlYt1bREUR8SiRJRxNy9Tk19+/Z1kbrmiSfc8/LczcLrE09Ufd+8PPdwCd5/ysur2v5m5e9vlp7Pf+IJ92bN9t+3WbOqfwc13T95jIP9/usLYJGnuK5m/MJe3UmJQOKmphfCup5IMp2IkseoSSLJ9P7uSgQidV5NLgR1PZFkOhFlukRTG4nMXYlAJPbqciLJdCLKdPw13T+pokSgxmKRGKhJr6eadp+taWN5Tfevaa+tmjbWZ3r/qlAiEJFKZTKRZDoR1TSRZHr/KklVVMjWSVVDIlJdmaway/T+SaiNQETk4GW610/UvYb0hDIRkRjI2BPKzOw8M/vAzD4ys8nlrG9iZk8m1r9lZvlRxiMiIgeKLBGYWUPgHmAocAIwysxOKLPZD4Fv3P37wH8Cv4oqHhERKV+UJYL+wEfu/om77wRmAMPLbDMceDQx/xRwpplZhDGJiEgZUSaCjsDnpd6vTSwrdxt3LwE2A+3KHsjMxpnZIjNbtGHDhojCFRGJpzpxH4G7T3f3Ancv6NChQ6bDERGpV6J8VOUXwFGl3ndKLCtvm7Vm1ghoBRRVdNDFixdvNLNyHhyYFdoDGzMdRAWyPT7I/hgVX80ovpqpSXx5qVZEmQjeBo42sy6EC/6VwFVltnkWuBZ4A7gUeMUr6c/q7llbJDCzRam6Z2WDbI8Psj9GxVcziq9mooovskTg7iVmdiPwAtAQeMjdV5jZLwg3NjwLPAg8bmYfAV8TkoWIiKRRlCUC3H02MLvMsjtKzapLDjUAAAYsSURBVO8ALosyBhERqVidaCyuQ6ZnOoBKZHt8kP0xKr6aUXw1E0l8dW6ICRERqV0qEYiIxJwSgYhIzCkRVJOZHWVmc83sfTNbYWY/KWebIWa22cyWJqY7yjtWhDGuNrP3Ep99wFCtFkxLDPa3zMz6pDG2Y0t9L0vNbIuZ3VRmm7R/f2b2kJl9ZWbLSy1ra2YvmtmHidc2Kfa9NrHNh2Z2bRrj+42ZrUr8Gz5jZq1T7Fvh+RBhfFPM7ItS/47np9i3wsEpI4zvyVKxrTazpSn2jfT7S3VNSev5l2p8ak0pHuAARwB9EvMtgf8BTiizzRDgvzMY42qgfQXrzwfmAAacBLyVoTgbAv8E8jL9/QGDgT7A8lLLfg1MTsxPBn5Vzn5tgU8Sr20S823SFN85QKPE/K/Ki68q50OE8U0B/lcVzoGPga5AY+Ddsv+fooqvzPr/AO7IxPeX6pqSzvNPJYJqcvf17r4kMb8VWMmBYyhlu+HAYx68CbQ2syMyEMeZwMfunvE7xd19PuFeltJKD4r4KDCinF3PBV5096/d/RvgReC8dMTn7n/3MEYXwJuEu/czIsX3VxVVGZyyxiqKLzHQ5eXAX2r7c6uigmtK2s4/JYIaSDw/oTfwVjmrTzazd81sjpl1S2tg4MDfzWyxmY0rZ31VBgRMhytJ/Z8vk99f0mHuvj4x/0/gsHK2yZbvciyhlFeeys6HKN2YqLp6KEXVRjZ8f6cCX7r7hynWp+37K3NNSdv5p0RwkMysBfA0cJO7bymzegmhuqMn8HtgVprDO8Xd+xCeBXGDmQ1O8+dXyswaA8OA/1fO6kx/fwfwUA7Pyr7WZnYbUAIUptgkU+fDfcD3gF7AekL1SzYaRcWlgbR8fxVdU6I+/5QIDoKZ5RD+wQrdfWbZ9e6+xd23JeZnAzlm1j5d8bn7F4nXr4BnCMXv0qoyIGDUhgJL3P3Lsisy/f2V8mWyyizx+lU522T0uzSzMcCFwOjExeIAVTgfIuHuX7r7bnffA9yf4nMz/f01AkYCT6baJh3fX4prStrOPyWCakrUJz4IrHT336bY5vDEdphZf8L3XOGoqrUYX3Mza5mcJzQoLi+z2bPANYneQycBm0sVQdMl5a+wTH5/ZSQHRSTx+l/lbPMCcI6ZtUlUfZyTWBY5MzsPuBUY5u7FKbapyvkQVXyl250uTvG5ewenTJQSryR87+lyFrDK3deWtzId318F15T0nX9RtYTX1wk4hVBEWwYsTUznA+OB8YltbgRWEHpAvAkMTGN8XROf+24ihtsSy0vHZ4THiH4MvAcUpPk7bE64sLcqtSyj3x8hKa0HdhHqWX9IeEjSy8CHwEtA28S2BcADpfYdC3yUmH6Qxvg+ItQPJ8/DPya2PRKYXdH5kKb4Hk+cX8sIF7UjysaXeH8+oafMx+mML7H8keR5V2rbtH5/FVxT0nb+aYgJEZGYU9WQiEjMKRGIiMScEoGISMwpEYiIxJwSgYhIzCkRiCSY2W7bf2TUWhsJ08zyS498KZJNIn1msUgds93de2U6CJF0U4lApBKJ8eh/nRiT/h9m9v3E8nwzeyUxqNrLZtY5sfwwC88HeDcxDUwcqqGZ3Z8Yc/7vZtY0sf3ExFj0y8xsRob+TIkxJQKRfZqWqRq6otS6ze7eHfgDcHdi2e+BR929B2HAt2mJ5dOAVz0MmteHcEcqwNHAPe7eDdgEXJJYPhnonTjO+Kj+OJFUdGexSIKZbXP3FuUsXw2c4e6fJAYH+6e7tzOzjYRhE3Yllq939/ZmtgHo5O7flTpGPmHc+KMT738K5Lj7/zGzvwHbCKOszvLEgHsi6aISgUjVeIr56viu1Pxu9rXRXUAY+6kP8HZiREyRtFEiEKmaK0q9vpGYX0gYLRNgNLAgMf8yMAHAzBqaWatUBzWzBsBR7j4X+CnQCjigVCISJf3yENmnqe3/APO/uXuyC2kbM1tG+FU/KrHsx8DDZnYLsAH4QWL5T4DpZvZDwi//CYSRL8vTEHgikSwMmObum2rtLxKpArURiFQi0UZQ4O4bMx2LSBRUNSQiEnMqEYiIxJxKBCIiMadEICISc0oEIiIxp0QgIhJzSgQiIjH3/wF+M5X1/IkVawAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "loss = history.history['loss']\n",
    "val_loss = history.history['val_loss']\n",
    "\n",
    "epochs = range(1, len(loss) + 1)\n",
    "plt.plot(epochs, loss, 'bo', label='Training loss')\n",
    "plt.plot(epochs, val_loss, 'b', label='Validation loss')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Loss')\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<function matplotlib.pyplot.legend(*args, **kwargs)>"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3de5hU1Znv8e9Ly8VWg3LzAkKjwWtUxAomTpLxLlHERCcRh3OiYyLRxDiZnEliQibx+Bwec58ZjcnYmoszaaMmGR0aMajEUTSINAoqKAItIMQLICjaBmj7PX+sXVPVZXV3td27dlXt3+d56qmqfau3dlevd++11l7b3B0REUmvAUkHICIiyVIiEBFJOSUCEZGUUyIQEUk5JQIRkZTbI+kAemvEiBHe0NCQdBgiIlVl6dKlW9x9ZLF5VZcIGhoaaGlpSToMEZGqYmbru5qnqiERkZRTIhARSTklAhGRlFMiEBFJOSUCEZGUUyIQkZrX1AQNDTBgQHhuaqqu9eOmRCAiPUq6IOzL+k1NMHMmrF8P7uF55szSt5H0+tltxJpI3L2qHieccIKLSO/8+tfu48a5m4XnX/+6d+vW17uHYiw86utL30bS648b13nd7GPcuOpYv6/fPwto8S7K1cQL9t4+lAgkjZIsyJMuCPu6vlnx9c2qY/2+fv+s7hKBqoZEKlxfqxZmzYK2ts7T2trC9FJs2NC76ZW2/tixvZteaev39fuXQolAJGZ9rd9NuiBPuiDs6/qzZ0N9fedp9fVhejWs39fvX5KuThUq9aGqISm3JKtl3JOvWki6jr8/9mFf/oZJr682AiUCSVjS9ev9sY20F6S1oD++f3eJwML86pHJZFyjj0q5NDSEOvlC48bBunU9rz9gQCh6C5lBR0dpMWTbCPKrh+rrobERZswofRuzZoXqoLFjQ7VEqetKbTCzpe6eKTZPbQRS8/pSR590/TqEAruxMSQfs/DcmySQ3ca6dSH5rFunJCCdKRFITetrj5ukGyqzVJBLnJQIpKb1tcdNXwvy/jiaF4mb2gikpvVXHb3q16XadddGUHW3qhTpjbFjizf29raOXgW/1DJVDUnF60tjb3/V0YvUMiUCqWh9bexVHb1Iz9RGIBWtr/34RSTQdQRStcox4JZI2ikRSEUry4BbIimnRCAVTY29IvFTIpCKpsZekfgpEUjs+joev4ZXEImXLiiTWBWOnJnt/gkq0EUqhc4IJFZ9HetHROIXayIwsylmtsrM1pjZ1UXmjzOzBWb2lJn9t5mNiTMeKT91/xSpfLElAjOrA24EPg4cBVxkZkcVLPZD4N/d/VjgWuC6uOKRZKj7p0jli/OMYDKwxt1b3X0XcDtwXsEyRwF/jF4/WGS+VDl1/xSpfHEmgtHAi3nvN0bT8i0Hzo9efxLYx8yGxxiTlJm6f4pUvqR7Df0j8BMzuwR4GNgEvFO4kJnNBGYCjFWdQtXRMM4ilS3OM4JNwMF578dE0/6Hu//Z3c939+OBWdG07YUbcvdGd8+4e2bkyJExhiwikj5xJoIlwAQzG29mg4DpwJz8BcxshJllY/gG8IsY4xERkSJiSwTu3g5cCcwHngXudPcVZnatmU2LFjsZWGVmzwP7A2pCFBEps1ivI3D3ee5+mLsf6u6zo2nfdvc50evfufuEaJnPufvOOOOR96avQ0SISGVLurFYKpyGiBCpfRpiQrqlISJEap8SgXRLQ0SI1D4lAumWhogQqX1KBNItDREhUvuUCKRbGiJCpPap15D0SENEiNQ2nRGIiKScEoGISMopEYiIpJwSQQpoiAgR6Y4ai2uchogQkZ7ojKDGaYgIEemJEkGN0xARItITJYIapyEiRKQnSgQ1TkNEiEhPlAhqnIaIEJGeqNdQCmiICBHpjs4IRERSTolARCTllAhERFJOiUBEJOWUCEREUk6JQEQk5ZQIRERSTolARCTllAhERFJOiaAK6MYyIhInDTFR4XRjGXjxRfjZz+C+++CII+DEE8PjuONg8OCkoxOpfubuScfQK5lMxltaWpIOo2waGkLhX2jcOFi3rtzRlI87PPIIXH893HVXeP/hD0NrK7z0Ulhm0CCYODGXGCZPhve/PwyuV2va28P3HjUqmeS3axds3gwHHVSb+zcNzGypu2eKzdMZQYVL241l3n4bbrsNbrgBli+H/faDr3wFvvCFkBQBNm6ExYvh8cfD8y9+EZYHGDYsJIRsYpg8GUaMSOzr9FpbG6xaBc8+2/mxejXs3h0K4TFj4NBD4ZBDco/s++HD31tB7Q6vvRYSbWsrrF2be93aGs7KOjpg9Gg455zwOO002Guv/t8HUn46I6hwlXJG8NprMG8eHHZYPFUyGzbAT38KN98cPuuYY+BLXwrVX4X3UyjU3g4rV+YSw+LFsGJFKLggFJLZ5HD88SH+oUP7N/7e2rbt3YX9ypXhb539lxwwIBTuRx0FRx4Z/uavvNK5kH755c7b3Wefzokh/zF6dDirKFbQr10Lb7zReVv779850QwfDg8/HKroduwIv4GTT4apU0NiGD++LLuu7HbtCgclTz4Z9tHOnZ0fu3a9e1qx+e3toWoz+1v84Afhfe8r3/fo7oxAiaDCFbYRQCgYy3lPgUWLYPr03FnIoEGhQM0edZ94Yigoensk6g4PPRSO5u++O0z7xCfgqqvgYx/rWxXEm2/C0qW5xPD44+FMImv8+FCtNHFiSAwTJ4a7tvVntceOHfDCC7mCdu3aXKH/yiu55QYPhsMPD4V9/mPCBBgypPvPeOutcEBQWLBnHzt3dr3uoEFhPxQ7uxg/vuuj/V27YOFCuOee8Hj++TD9yCNzSeGkk2DgwF7trorgHvZl/kHFk0+G71xo8ODcY9Cgzu+LTTODp5/O7S+zsM+y/0Mnnggf+EB8+02JoMo1NYWbzW/YEAqr2bPLkwQ6OuBHP4JvfhMOPhhuuglefz1XsLa05BLU8OG5qphsghg+vPh229rCd7rhhvCPMWwYXHYZXHFFOOqNy0svhSO7Zctyj+efzx2B77tvLjlkH0ceGf6hi+nogD//ufjRdWtrqFPPt+++4YiwsMBvaIC6uv7/vh0duTOA1lbYtAkOPDBX8B90UDjr6KvVq3NJ4aGHQhXW0KFw1lkhMUyZAiNH9v1z4rB1a67Qf/zx8Ni6Ncyrr4dMJvebzmTCb3rw4FBYv9eDhm3bYMmSXKJZvBi2bAnz9twTJk3qfJCVvalUXyWWCMxsCvCvQB1wi7t/t2D+WOBWYN9omavdfV5320xjIkjC1q1w8cXhn/uCC+CWW0JBlq+9PVTB5NfXr1iRK1jf//7ORzvDhoWqn1tuCf8Mxx0Xqn/+9m/DP0AS3norJKNsYli+HJ56KpfgBg4MVTMTJ4Yj9JdfzhX0L7zQ+Uixri4k6mJ194ccEto7at2OHXD//eF3M29e2F9m4e9/9tmhkMsmv/5IQr2xc2f4G+efJa5ZE+aZwdFHdy6Ajz4a9ihDK6p7OKvLTwxPPJE7mxs1KhfTJz8Z4novEkkEZlYHPA+cAWwElgAXufvKvGUagSfd/WdmdhQwz90butuuEkH8/vQnuPBCePVV+PGPQ0NtqUckO3Z0rpJZvDgcNWfV1YUf81VXwUc+Upk9UN55JxQQ+WcOy5aFQm3o0OL174ceGs6aqrE6JC4dHaFaZe7ckBiWLMnNGzKkc3VYth1kwoSuz8BKsX171w3e69eHvy2Es6H83maZTGhfqRS7d4cDkvyDrOeeg5//HC699L1tM6lE8GHgGnc/K3r/DQB3vy5vmZuAVnf/XrT8j9z9pO62q0QQn44O+OEPQ1XQuHFw551wwgl93+6mTeGH/OKLcP75ocCsRm1t4cylEpNXNdi2LTSIFzaS53d6qKsLSbWw+uyII0JB3d4e2nq6avDetq3zZ44Y0TlZZ9u2xowp61fvF9u3h/3zXhNWUongb4Ap7v656P3/Bk509yvzljkQuA/YD9gLON3dlxbZ1kxgJsDYsWNPWF+sG430yZYt8JnPwL33wqc+Fapwku5ZI+mQ7TJbmCRWrw4Ff9aoUaFHWf60gQNDNVOxM7Tx48vbK6fSVfJ1BBcBv3L3H0VnBP9hZh9w9478hdy9EWiEcEaQQJw17ZFHQq+gzZvhxhtDo62OeqVc6uvDkfrxx3eevnt3555Wra0hGeRXzY0ZE09De9rEmQg2AfmVAGOiafk+C0wBcPdFZjYEGAG8GmNcEunogO9/H771rXBUtWhRaMwTqQQDB4YqoSOOCO1KEp842+2XABPMbLyZDQKmA3MKltkAnAZgZkcCQ4CCTncSh82bQ3/vb3wj9Ap64gklAZG0ii0RuHs7cCUwH3gWuNPdV5jZtWY2LVrs/wCXmdly4DfAJV5tFzZUoYULw2n4gw+Gwdxuv111qSJpFmsbQXRNwLyCad/Oe70S+Ks4Y5Ccjg743vfgn/4p1K8+9ljoHy8i6ZZ0Y7HEbPv2XDe7n/8c5s8PDcM33aSzABEJlAiqXHt76J9frF91a2vnftVDhoQEcNll6hUkIjlKBFVi585wyf7q1Z0L/PyrJSHXr/rQQ8OFM4V9q/feO7GvICIVSomgCrz5JkybFhp3IXe15OTJcNFF7x5qWP2qRaQ3lAgq3Ouvh8G6Fi8Og7V96lOq2xeR/qVEUMG2bg1D+S5fDnfcEfr7i4j0tx6vIzCzc82szAPG1pamptywuw0N4X1PXn0VTjkFnnkm3LRFSUBE4lJKAX8hsNrMvm9mR8QdUK3J3mEsewvC9evD++6SwaZN8Nd/HYZCnjs3XAEsIhKXHhOBu/8v4HhgLfArM1tkZjPNrIJG765cs2Z1vs0khPezZhVffv36cJvGjRtDn//TT48/RhFJt5KqfNz9DeB3wO3AgcAngSfM7EsxxlYTsvf5LWX6mjXw0Y+GoXYfeCC8FhGJWyltBNPM7C7gv4GBwGR3/zhwHGGsIOnG2LGlTV+5MpwJvP126CZ64onxxyYiAqWdEVwA/LO7H+PuP3D3VwHcvY0wjLR0Y/bsMN56vvr6MD1r2bLQJuAebv6t8X9EpJxKSQTXAI9n35jZnmbWAODuC2KJqobMmAGNjeHWj2bhubExTIdwP9JTTgm3QHz44XDvVhGRcirlOoLfAvn3EX4nmvbBWCKqQTNm5Ar+fAsXhh5BI0fCggWha6mISLmVckawh7vvyr6JXg+KL6R0eOABmDIFDjoonAkoCYhIUkpJBJvzbiSDmZ0HbIkvpNp3zz0wdWoYBO6hh8L4QCIiSSmlauhyoMnMfgIY8CLwmVijqmG//30YKO7YY8N1AsOHJx2RiKRdj4nA3dcCHzKzvaP3b8YeVY1qaoKLLw5dQ+fNg6FDk45IRKTEQefM7BzgaGCIRXc0cfdrY4yr5tx8M3z+86GbaHOz7gsgIpWjlAvK/o0w3tCXCFVDnwLGxRxXTfnBD8L4QlOmhDMBJQERqSSlNBaf5O6fAba5+/8FPgwcFm9YtcEdvvlN+NrX4MILwyiie+6ZdFQiIp2Vkgj+Ej23mdlBwG7CeEPSjY4O+OIX4brrcqONDlKnWxGpQKW0ETSb2b7AD4AnAAdujjWqKrd7N1xyCdx2Wzgb+O53dbN4Ealc3SaC6IY0C9x9O/B7M5sLDHH318sSXRV6+2349KfDfQSuuw6uvjrpiEREutdtInD3DjO7kXA/Atx9J7CzHIFVozfeCDeZf/hh+OlP4Yorko5IRKRnpbQRLDCzC8xUudGdLVvgtNPg0UdDe4CSgIhUi1LaCD4PfAVoN7O/ELqQuru/L9bIqsimTXDGGfDCC6FnkG4tKSLVpJQri3VLym6sWROSwNat8Ic/hAvGRESqSY+JwMw+Vmy6uz/c/+FUl6efhjPPDL2E/vhHyGSSjkhEpPdKqRr6at7rIcBkYClwaiwRVYnHHoOzzw53G1uwQDeUEZHqVUrV0Ln5783sYOBfYouoCjzwAHziE3DAAeG17iUgItWslF5DhTYCR/Z3INXirrtCY/Ahh8AjjygJiEj1K6WN4AbC1cQQEsdEwhXGqXPrrXDppTB5cri5zLBhSUckItJ3pbQRtOS9bgd+4+6PxhRPxbrnnjBsxOmnh7MCjSAqIrWilETwO+Av7v4OgJnVmVm9u7f1tKKZTQH+FagDbnH37xbM/2fglOhtPTDK3fftzRcol1/+MtxfeO5cGDw46WhERPpPSVcWA/mDJ+8JPNDTSmZWB9wIfBw4CrjIzDr1rXH3f3D3ie4+EbgB+M9SAy+nXbvCbSWnTlUSEJHaU0oiGJJ/e8rodX0J600G1rh7q7vvAm4Hzutm+YuA35Sw3bJ76CF4800499yelxURqTalJIK3zGxS9o2ZnQC8XcJ6owk3us/aGE17FzMbB4wH/tjF/Jlm1mJmLZs3by7ho/tXczMMGQKnpvrKCRGpVaW0EXwZ+K2Z/ZkwztABhFtX9qfpwO+y7RCF3L0RaATIZDJebJm4uId2gdNPDxePiYjUmlIuKFtiZkcAh0eTVrn77hK2vQk4OO/9mGhaMdOBL5awzbJbuTIMJqf7CohIrSrl5vVfBPZy92fc/RlgbzP7QgnbXgJMMLPxZjaIUNjPKbL9I4D9gEW9C7085s4NzxpRVERqVSltBJdFdygDwN23AZf1tJK7twNXAvOBZ4E73X2FmV1rZtPyFp0O3O7uZa3yKVVzM0yaBKOLtm6IiFS/UtoI6szMsgV11C20pNuwu/s8YF7BtG8XvL+mtFDLb8sWWLQIvvWtpCMREYlPKYngD8AdZnZT9P7zwL3xhVQ57r0XOjrUbVREalspVUNfJ3TrvDx6PE3nC8xqVnMzHHggPPtsGFxuwIDw3NSUdGQiIv2nx0Tg7h3AYmAd4SKxUwl1/jVt165wx7HDD4fLL4f160NX0vXrYeZMJQMRqR1dJgIzO8zMvmNmzxGGf9gA4O6nuPtPyhVgUhYuhB07YMUKaCsYVamtDWbNSiYuEZH+1l0bwXPAQmCqu68BMLN/KEtUFSB7NXFXFzJv2FDeeERE4tJd1dD5wEvAg2Z2s5mdRriyuOa5h0Rw6qkwblzxZcaOLW9MIiJx6TIRuPvd7j4dOAJ4kDDUxCgz+5mZnVmuAJPw3HPQ2hp6C82e/e6hJerrw3QRkVpQSmPxW+5+W3Tv4jHAk4SeRDWruTk8T50KM2ZAY2M4MzALz42NYbqISC2wCr2gt0uZTMZbWlp6XrAPPvrRMOz0k0/G+jEiImVjZkvdPVNs3nu5eX1N27oV/vQnXUQmIumhRFAgezXx1KlJRyIiUh5KBAWam+GAAyBT9ARKRKT2KBHk2b07XE18zjlhOAkRkTRQcZdn4UJ44w1VC4lIuigR5Jk7FwYPhjPOSDoSEZHyUSKI5F9NvNdeSUcjIlI+SgSRVatgzRpVC4lI+igRRLL3JlYiEJG0USKINDfDscdqMDkRSR8lAuC11+DRR3U1sYikkxIB4dqBd95RIhCRdFIiIFQLjRoFH/xg0pGIiJRf6hPB7t1hfCFdTSwiaZX6ou/RR+H111UtJCLplfpE0NwMgwbpamIRSS8lgmY45RTYe++kIxERSUaqE8Hzz8Pq1aoWEpF0S3UiyL83sYhIWqU+ERxzTLghvYhIWqU2EWzbBo88orMBEZHUJgJdTSwiEqQ2EcydCyNHwuTJSUciIpKsVCaC9vZwNfHZZ0NdXdLRiIgkK5WJ4NFHQxuBqoVERGJOBGY2xcxWmdkaM7u6i2U+bWYrzWyFmd0WZzxZc+fCwIFw5pnl+DQRkcq2R1wbNrM64EbgDGAjsMTM5rj7yrxlJgDfAP7K3beZ2ai44snX3Awnnwz77FOOTxMRqWxxnhFMBta4e6u77wJuB84rWOYy4EZ33wbg7q/GGA8QriRetUrVQiIiWXEmgtHAi3nvN0bT8h0GHGZmj5rZY2Y2pdiGzGymmbWYWcvmzZv7FJTuTSwi0lnSjcV7ABOAk4GLgJvNbN/Chdy90d0z7p4ZOXJknz6wuRmOPhrGj+/TZkREakaciWATcHDe+zHRtHwbgTnuvtvdXwCeJySGWGzfDgsXqlpIRCRfnIlgCTDBzMab2SBgOjCnYJm7CWcDmNkIQlVRa1wBzZ8friFQtZCISE5sicDd24ErgfnAs8Cd7r7CzK41s2nRYvOBrWa2EngQ+Kq7b40rpuZmGDECPvShuD5BRKT6mLsnHUOvZDIZb2lp6fV67e3hBvXnngu33hpDYCIiFczMlrp7pti8pBuLy2bRonA1saqFREQ6S00iWLAgXE181llJRyIiUllSkwi+8x1YsQLe976kIxERqSypSQRmMCG2jqkiItUrNYlARESKUyIQEUk5JQIRkZRTIhARSTklAhGRlFMiEBFJOSUCEZGUUyIQEUk5JQIRkZRTIhARSTklAhGRlFMiEBFJOSUCEZGUUyIQEUk5JQIRkZRTIhARSTklAhGRlFMiEBFJOSUCEZGUUyIQEUk5JQIRkZRTIhARSTklAhGRlFMiEBFJOSUCEZGUUyIQEUk5JQIRkZRTIhARSTklAhGRlFMiEBFJuVgTgZlNMbNVZrbGzK4uMv8SM9tsZsuix+fijEdERN5tj7g2bGZ1wI3AGcBGYImZzXH3lQWL3uHuV8YVh4iIdC/OM4LJwBp3b3X3XcDtwHkxfl6XmpqgoQEGDAjPTU1JRCEiUpniTASjgRfz3m+MphW6wMyeMrPfmdnBxTZkZjPNrMXMWjZv3tyrIJqaYOZMWL8e3MPzzJlKBiIiWUk3FjcDDe5+LHA/cGuxhdy90d0z7p4ZOXJkrz5g1ixoa+s8ra0tTBcRkXgTwSYg/wh/TDTtf7j7VnffGb29BTihv4PYsKF300VE0ibORLAEmGBm481sEDAdmJO/gJkdmPd2GvBsfwcxdmzvpouIpE1sicDd24ErgfmEAv5Od19hZtea2bRosavMbIWZLQeuAi7p7zhmz4b6+s7T6uvDdBERAXP3pGPolUwm4y0tLb1ap6kptAls2BDOBGbPhhkzYgpQRKQCmdlSd88UmxfbdQSVZMYMFfwiIl1JuteQiIgkTIlARCTllAhERFJOiUBEJOWUCEREUq7quo+a2WZgfdJxdGEEsCXpILqh+Pqm0uODyo9R8fVNX+Ib5+5Fx+ipukRQycyspat+upVA8fVNpccHlR+j4uubuOJT1ZCISMopEYiIpJwSQf9qTDqAHii+vqn0+KDyY1R8fRNLfGojEBFJOZ0RiIiknBKBiEjKKRH0kpkdbGYPmtnK6F4Kf19kmZPN7HUzWxY9vl3mGNeZ2dPRZ79rzG4LrjezNdH9oieVMbbD8/bLMjN7w8y+XLBM2fefmf3CzF41s2fypg0zs/vNbHX0vF8X614cLbPazC4uU2w/MLPnor/fXWa2bxfrdvtbiDnGa8xsU97f8ewu1p1iZqui3+PVZYzvjrzY1pnZsi7WjXUfdlWmlPX35+569OIBHAhMil7vAzwPHFWwzMnA3ARjXAeM6Gb+2cC9gAEfAhYnFGcd8DLhQpdE9x/wMWAS8EzetO8DV0evrwa+V2S9YUBr9Lxf9Hq/MsR2JrBH9Pp7xWIr5bcQc4zXAP9Ywm9gLXAIMAhYXvj/FFd8BfN/BHw7iX3YVZlSzt+fzgh6yd1fcvcnotc7CHdfG51sVL12HvDvHjwG7Ftw29ByOQ1Y6+6JXynu7g8DrxVMPg+4NXp9K/CJIqueBdzv7q+5+zbgfmBK3LG5+30e7gII8BjhnuCJ6WL/lWIysMbdW919F3A7Yb/3q+7iMzMDPg38pr8/txTdlCll+/0pEfSBmTUAxwOLi8z+sJktN7N7zezosgYGDtxnZkvNbGaR+aOBF/PebySZZDadrv/5ktx/Wfu7+0vR65eB/YssUwn78lLCGV4xPf0W4nZlVH31iy6qNiph/30UeMXdV3cxv2z7sKBMKdvvT4ngPTKzvYHfA1929zcKZj9BqO44DrgBuLvM4X3E3ScBHwe+aGYfK/Pn98jMBgHTgN8WmZ30/nsXD+fhFdfX2sxmAe1AUxeLJPlb+BlwKDAReIlQ/VKJLqL7s4Gy7MPuypS4f39KBO+BmQ0k/MGa3P0/C+e7+xvu/mb0eh4w0MxGlCs+d98UPb8K3EU4/c63CTg47/2YaFo5fRx4wt1fKZyR9P7L80q2yix6frXIMontSzO7BJgKzIgKincp4bcQG3d/xd3fcfcO4OYuPjvR36KZ7QGcD9zR1TLl2IddlCll+/0pEfRSVJ/4c+BZd/9xF8scEC2HmU0m7OetZYpvLzPbJ/ua0Kj4TMFic4DPRL2HPgS8nncKWi5dHoUluf8KzAGyvTAuBv6ryDLzgTPNbL+o6uPMaFqszGwK8DVgmru3dbFMKb+FOGPMb3f6ZBefvQSYYGbjo7PE6YT9Xi6nA8+5+8ZiM8uxD7spU8r3+4urJbxWH8BHCKdoTwHLosfZwOXA5dEyVwIrCD0gHgNOKmN8h0SfuzyKYVY0PT8+A24k9NZ4GsiUeR/uRSjYh+ZNS3T/EZLSS8BuQj3rZ4HhwAJgNfAAMCxaNgPckrfupcCa6PF3ZYptDaFuOPsb/Ldo2YOAed39Fsq4//4j+n09RSjUDiyMMXp/NqGnzNq4YiwWXzT9V9nfXd6yZd2H3ZQpZfv9aYgJEZGUU9WQiEjKKRGIiKScEoGISMopEYiIpJwSgYhIyikRiETM7B3rPDJqv42EaWYN+SNfilSSPZIOQKSCvO3uE5MOQqTcdEYg0oNoPPrvR2PSP25m74+mN5jZH6NB1RaY2dho+v4W7hGwPHqcFG2qzsxujsacv8/M9oyWvyoai/4pM7s9oa8pKaZEIJKzZ0HV0IV5815392OAnwD/Ek27AbjV3Y8lDPp2fTT9euAhD4PmTSJckQowAbjR3Y8GtgMXRNOvBo6PtnN5XF9OpCu6slgkYmZvuvveRaavA05199ZocLCX3X24mW0hDJuwO5r+kruPMLPNwBh335m3jQbCuPETovdfBwa6+/8zsz8AbxJGWb3bowH3RMpFZwQipfEuXvfGzrzX75BrozuHMPbTJGBJNCKmSNkoEYiU5sK850XR6z8RRssEmAEsjF4vAK4AMNAEnbgAAACbSURBVLM6Mxva1UbNbABwsLs/CHwdGAq866xEJE468hDJ2dM638D8D+6e7UK6n5k9RTiqvyia9iXgl2b2VWAz8HfR9L8HGs3ss4Qj/ysII18WUwf8OkoWBlzv7tv77RuJlEBtBCI9iNoIMu6+JelYROKgqiERkZTTGYGISMrpjEBEJOWUCEREUk6JQEQk5ZQIRERSTolARCTl/j+Hf25rioeyuAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "acc = history.history['accuracy']\n",
    "val_acc = history.history['val_accuracy']\n",
    "\n",
    "plt.plot(epochs, acc, 'bo', label='Training accuracy')\n",
    "plt.plot(epochs, val_acc, 'b', label='Validation accuracy')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> It seems that the network starts overfitting after 9 epochs. Let’s train a new network\n",
    "from scratch for 9 epochs, then let’s evaluate it on the test set:\n",
    "\n",
    "上面看出似乎网络在第9次迭代之后开始出现过拟合。因此下面我们可以重新训练一个网络，仅使用9次迭代，然后在测试集上来验证性能："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 7982 samples, validate on 1000 samples\n",
      "Epoch 1/9\n",
      "7982/7982 [==============================] - 1s 109us/sample - loss: 2.5417 - accuracy: 0.5326 - val_loss: 1.6613 - val_accuracy: 0.6550\n",
      "Epoch 2/9\n",
      "7982/7982 [==============================] - 0s 47us/sample - loss: 1.3697 - accuracy: 0.7176 - val_loss: 1.2834 - val_accuracy: 0.7170\n",
      "Epoch 3/9\n",
      "7982/7982 [==============================] - 1s 89us/sample - loss: 1.0266 - accuracy: 0.7793 - val_loss: 1.1689 - val_accuracy: 0.7460\n",
      "Epoch 4/9\n",
      "7982/7982 [==============================] - 0s 45us/sample - loss: 0.8160 - accuracy: 0.8279 - val_loss: 1.0345 - val_accuracy: 0.7780\n",
      "Epoch 5/9\n",
      "7982/7982 [==============================] - 0s 46us/sample - loss: 0.6544 - accuracy: 0.8623 - val_loss: 0.9832 - val_accuracy: 0.7920\n",
      "Epoch 6/9\n",
      "7982/7982 [==============================] - 0s 43us/sample - loss: 0.5297 - accuracy: 0.8919 - val_loss: 1.0068 - val_accuracy: 0.7740\n",
      "Epoch 7/9\n",
      "7982/7982 [==============================] - 0s 42us/sample - loss: 0.4304 - accuracy: 0.9128 - val_loss: 0.9093 - val_accuracy: 0.8180\n",
      "Epoch 8/9\n",
      "7982/7982 [==============================] - 1s 69us/sample - loss: 0.3557 - accuracy: 0.9230 - val_loss: 0.9094 - val_accuracy: 0.8090\n",
      "Epoch 9/9\n",
      "7982/7982 [==============================] - 1s 107us/sample - loss: 0.2917 - accuracy: 0.9372 - val_loss: 0.9160 - val_accuracy: 0.8010\n"
     ]
    }
   ],
   "source": [
    "model = models.Sequential()\n",
    "\n",
    "model.add(layers.Dense(64, activation='relu', input_shape=(10000,)))\n",
    "model.add(layers.Dense(64, activation='relu'))\n",
    "model.add(layers.Dense(46, activation='softmax'))\n",
    "\n",
    "model.compile(optimizer='rmsprop',\n",
    "              loss='categorical_crossentropy',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "history = model.fit(partial_x_train, partial_y_train, epochs=9,\n",
    "                    batch_size=512, validation_data=(x_val, y_val))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2246/2246 [==============================] - 0s 65us/sample - loss: 0.9941 - accuracy: 0.7805\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.9940675952449401, 0.7804987]"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.evaluate(x_test, one_hot_test_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Our approach reaches an accuracy of ~80%. With a balanced binary classification\n",
    "problem, the accuracy reached by a purely random classifier would be 50%, but in our\n",
    "case it is closer to 19%, so our results seem pretty good, at least when compared to a\n",
    "random baseline:\n",
    "\n",
    "可以看到我们的网络可以达到接近80%的准确率。在一个平均分布的二分分类问题中，如果使用一个纯随机的分类器准确率应该是50%，但在这个场景中，这个几率大约是19%，因此我们的结果看起来很不错，至少与随机的基线相比有大幅度提升："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.18432769367764915"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import copy\n",
    "test_labels_copy = copy.copy(test_labels)\n",
    "np.random.shuffle(test_labels_copy)\n",
    "float(np.sum(np.array(test_labels_copy) == np.array(test_labels))) / len(test_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.5.5 在新数据上生成预测值\n",
    "\n",
    "> We can verify that the predict method of our model instance returns a probability\n",
    "distribution over all 46 topics. Let’s generate topic predictions for all of the test data:\n",
    "\n",
    "我们可以验证一下模型的预测输出，这是一个46个主题的概率分布。我们可以使用`predict`产生所有测试数据上的预测值："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions = model.predict(x_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Each entry in predictions is a vector of length 46:\n",
    "\n",
    "`predictions`中的每个预测值都是一个长度为46的向量："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2246, 46)"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictions.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> The coefficients in this vector sum to 1:\n",
    "\n",
    "这个向量中所有的因子之和为1："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.99999994"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictions[0].sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.5.6 处理标签和损失的另一种方法\n",
    "\n",
    "> We mentioned earlier that another way to encode the labels would be to cast them as an\n",
    "integer tensor, like such:\n",
    "\n",
    "前面提到过我们还有另一种方法来对标签进行编码，将它转换成一个整数张量，就像下面这样；"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train = np.array(train_labels)\n",
    "y_test = np.array(test_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> The only thing it would change is the choice of the loss function. Our previous loss,\n",
    "categorical_crossentropy , expects the labels to follow a categorical encoding. With\n",
    "integer labels, we should use sparse_categorical_crossentropy :\n",
    "\n",
    "这里需要改变的是损失函数。我们之前使用`categorical_crossentropy`，期望这些标签遵循明确分类的编码。但在整数标签中，我们应该使用`sparse_categorical_crossentropy`损失函数："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(optimizer='rmsprop',\n",
    "              loss='sparse_categorical_crossentropy',\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> This new loss function is still mathematically\n",
    "categorical_crossentropy ; it just has a different interface.\n",
    "\n",
    "这个新的损失函数在数学上仍然等同于`categorical_crossentropy`；它只是另外一种接口形式而已。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.5.7 中间层具有足够大假设空间的重要性\n",
    "\n",
    "> We mentioned earlier that since our final outputs were 46-dimensional, we should avoid\n",
    "intermediate layers with much less than 46 hidden units. Now let’s try to see what\n",
    "happens when we introduce an information bottleneck by having intermediate layers\n",
    "significantly less than 46-dimensional, e.g. 4-dimensional.\n",
    "\n",
    "前面我们提到因为最终的输出是46维的，需要避免模型有着远少于46个隐藏单元的中间层。下面我们试着看一下如果在网络中定义一个少于46维（例如4维）的中间层，产生信息瓶颈的情况。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 7982 samples, validate on 1000 samples\n",
      "Epoch 1/20\n",
      "7982/7982 [==============================] - 1s 124us/sample - loss: 2.5492 - accuracy: 0.5190 - val_loss: 1.8208 - val_accuracy: 0.6230\n",
      "Epoch 2/20\n",
      "7982/7982 [==============================] - 1s 68us/sample - loss: 1.5487 - accuracy: 0.6473 - val_loss: 1.4841 - val_accuracy: 0.6480\n",
      "Epoch 3/20\n",
      "7982/7982 [==============================] - 1s 65us/sample - loss: 1.2746 - accuracy: 0.6825 - val_loss: 1.3780 - val_accuracy: 0.6730\n",
      "Epoch 4/20\n",
      "7982/7982 [==============================] - 1s 105us/sample - loss: 1.1194 - accuracy: 0.7228 - val_loss: 1.3334 - val_accuracy: 0.6900\n",
      "Epoch 5/20\n",
      "7982/7982 [==============================] - 1s 65us/sample - loss: 1.0080 - accuracy: 0.7434 - val_loss: 1.3202 - val_accuracy: 0.6910\n",
      "Epoch 6/20\n",
      "7982/7982 [==============================] - 1s 63us/sample - loss: 0.9225 - accuracy: 0.7607 - val_loss: 1.3206 - val_accuracy: 0.6950\n",
      "Epoch 7/20\n",
      "7982/7982 [==============================] - 1s 66us/sample - loss: 0.8495 - accuracy: 0.7770 - val_loss: 1.3391 - val_accuracy: 0.6990\n",
      "Epoch 8/20\n",
      "7982/7982 [==============================] - 1s 93us/sample - loss: 0.7854 - accuracy: 0.7900 - val_loss: 1.3378 - val_accuracy: 0.7080\n",
      "Epoch 9/20\n",
      "7982/7982 [==============================] - 1s 64us/sample - loss: 0.7301 - accuracy: 0.8059 - val_loss: 1.3916 - val_accuracy: 0.7020\n",
      "Epoch 10/20\n",
      "7982/7982 [==============================] - 1s 92us/sample - loss: 0.6823 - accuracy: 0.8190 - val_loss: 1.4300 - val_accuracy: 0.7010\n",
      "Epoch 11/20\n",
      "7982/7982 [==============================] - 1s 69us/sample - loss: 0.6379 - accuracy: 0.8281 - val_loss: 1.4262 - val_accuracy: 0.7040\n",
      "Epoch 12/20\n",
      "7982/7982 [==============================] - 1s 65us/sample - loss: 0.6002 - accuracy: 0.8411 - val_loss: 1.4936 - val_accuracy: 0.7010\n",
      "Epoch 13/20\n",
      "7982/7982 [==============================] - 1s 68us/sample - loss: 0.5668 - accuracy: 0.8524 - val_loss: 1.5643 - val_accuracy: 0.7050\n",
      "Epoch 14/20\n",
      "7982/7982 [==============================] - 1s 64us/sample - loss: 0.5360 - accuracy: 0.8572 - val_loss: 1.6263 - val_accuracy: 0.6980\n",
      "Epoch 15/20\n",
      "7982/7982 [==============================] - 1s 66us/sample - loss: 0.5110 - accuracy: 0.8631 - val_loss: 1.6555 - val_accuracy: 0.7040\n",
      "Epoch 16/20\n",
      "7982/7982 [==============================] - 1s 65us/sample - loss: 0.4858 - accuracy: 0.8691 - val_loss: 1.7311 - val_accuracy: 0.7070\n",
      "Epoch 17/20\n",
      "7982/7982 [==============================] - 1s 66us/sample - loss: 0.4650 - accuracy: 0.8711 - val_loss: 1.7850 - val_accuracy: 0.7050\n",
      "Epoch 18/20\n",
      "7982/7982 [==============================] - 1s 94us/sample - loss: 0.4484 - accuracy: 0.8731 - val_loss: 1.8535 - val_accuracy: 0.7000\n",
      "Epoch 19/20\n",
      "7982/7982 [==============================] - 2s 194us/sample - loss: 0.4301 - accuracy: 0.8789 - val_loss: 1.8933 - val_accuracy: 0.7030\n",
      "Epoch 20/20\n",
      "7982/7982 [==============================] - 1s 73us/sample - loss: 0.4180 - accuracy: 0.8811 - val_loss: 2.0100 - val_accuracy: 0.6910\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f47a8cf26d0>"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = models.Sequential()\n",
    "\n",
    "model.add(layers.Dense(64, activation='relu', input_shape=(10000,)))\n",
    "model.add(layers.Dense(4, activation='relu'))\n",
    "model.add(layers.Dense(46, activation='softmax'))\n",
    "\n",
    "model.compile(optimizer='rmsprop',\n",
    "              loss='categorical_crossentropy',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "model.fit(partial_x_train, partial_y_train, batch_size=128,\n",
    "          epochs=20, validation_data=(x_val, y_val))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Our network now seems to peak at ~71% validation accuracy, a 8% absolute drop.\n",
    "This drop is mostly due to the fact that we are now trying to compress a lot of\n",
    "information (enough information to recover the separation hyperplanes of 46 classes) into\n",
    "an intermediate space that is too low-dimensional. The network is able to cram most of\n",
    "the necessary information into these 8-dimensional representations, but not all of it.\n",
    "\n",
    "网络在这种情况下达到的验证准确率的峰值大约是71%，比前面那个网络下降了8%。造成这个结果的原因主要是我们试图将很多信息（能在超平面中将46个分类分开的足够信息）压缩到一个很低维度的中间空间当中。这个网络能够将大部分需要的信息塞进4维表示中，但无法包含全部需要的信息。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.5.8 更多实验\n",
    "\n",
    "> - Try using larger or smaller layers: 32 units, 128 units...\n",
    "- We were using two hidden layers. Now try to use a single hidden layer, or three hidden\n",
    "layers.\n",
    "\n",
    "- 尝试使用更大或更小的层：32隐藏单元，128隐藏单元...\n",
    "- 上面使用了2个隐藏层。尝试使用单个隐藏层，或者3个隐藏层。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.5.9 小结\n",
    "\n",
    "> Here’s what you should take away from this example:\n",
    "\n",
    "> - If you are trying to classify data points between N classes, your network should end with\n",
    "a Dense layer of size N.\n",
    "- In a single-label, multi-class classification problem, your network should end with a\n",
    "softmax activation, so that it will output a probability distribution over the N output\n",
    "classes.\n",
    "- Categorical crossentropy is almost always the loss function you should use for such\n",
    "problems. It minimizes the distance between the probability distributions output by the\n",
    "network, and the true distribution of the targets.\n",
    "- There are two ways to handle labels in multi-class classification:\n",
    "    - Encoding the labels via \"categorical encoding\" (also known as \"one-hot encoding\") and\n",
    "using categorical_crossentropy as your loss function.\n",
    "    - Encoding the labels as integers and using the sparse_categorical_crossentropy lo\n",
    "function.\n",
    "- If you need to classify data into a large number of categories, then you should avoid\n",
    "creating information bottlenecks in your network by having intermediate layers that are\n",
    "too small.\n",
    "\n",
    "下面列出这个例子中你应该学到的内容：\n",
    "\n",
    "- 如果你尝试将数据点分为N个类别，你的网络最后一层应该是具有N尺寸的全连接层。\n",
    "- 在一个单一标记、多类别分类问题中，你的网络最后应该使用`softmax`激活函数，使得它能够输出在N个输出类别上的概率分布。\n",
    "- 多类别交叉熵通常都是这类问题最合适的损失函数。它会尽可能减少网络输出的概率分布与真实目标概率分布之间的距离。\n",
    "- 在多类别分类中有两种方式处理标签：\n",
    "    - 将标签编码成`categorical编码`（也被称为`one-hot编码`），然后使用`categorical_crossentropy`作为损失函数。\n",
    "    - 将标签编码成整数张量，然后使用`sparse_categorical_crossentropy`作为损失函数。\n",
    "- 如果你需要将数据分成很多类别，你应该避免使用了过小的中间层从而造成信息瓶颈。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.6 回归的例子：预测房价\n",
    "\n",
    "> In our two previous examples, we were considering classification problems, where the\n",
    "goal was to predict a single discrete label of an input data point. Another common type of\n",
    "machine learning problem is \"regression\", which consists of predicting a continuous\n",
    "value instead instead of a discrete label. For instance, predicting the temperature\n",
    "tomorrow, given meteorological data, or predicting the time that a software project will\n",
    "take to complete, given its specifications.\n",
    "\n",
    "在前面两个例子当中，都是分类问题的应用，它们的目标是在输入数据点上预测一个离散标签。除此之外还有另一种常见的机器学习问题，叫做“回归”，它预测的是一个连续的值而不是一个离散的标签。例如给定气象数据来预测明天的温度，或者根据软件的规格说明来预测软件工程的完成时间。\n",
    "\n",
    "> Do not mix up \"regression\" with the algorithm \"logistic regression\": confusingly,\n",
    "\"logistic regression\" is not a regression algorithm, it is a classification algorithm.\n",
    "\n",
    "不要将这里的“回归”问题与“逻辑回归”算法混淆起来：这里很容易令人困惑，但实际上“逻辑回归”不是一个回归算法，而是一个分类算法。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.6.1 波斯顿房价数据集\n",
    "\n",
    "> We will be attempting to predict the median price of homes in a given Boston suburb in\n",
    "the mid-1970s, given a few data points about the suburb at the time, such as the crime\n",
    "rate, the local property tax rate, etc.\n",
    "\n",
    "我们尝试预测波斯顿近郊在1970年代中期的平均房价，依据是当时的一些统计数据，比如犯罪率、当地房产税率等。\n",
    "\n",
    "> The dataset we will be using has another interesting difference from our two previous\n",
    "examples: it has very few data points, only 506 in total, split between 404 training\n",
    "samples and 102 test samples, and each \"feature\" in the input data (e.g. the crime rate is a\n",
    "feature) has a different scale. For instance some values are proportions, which take a\n",
    "values between 0 and 1, others take values between 1 and 12, others between 0 and\n",
    "100...\n",
    "\n",
    "我们将要使用的数据和前面两个例子中的数据集有另外一个有趣的区别：它的数据点很少，一共只有506个，分为404个训练样本和102个测试样本，而且数据中的每个“特征”（例如犯罪率就是一个特征）有着不同的范围。如一些值是比例，它们的值处于0和1之间，另外的一些取值范围是1和12之间，还有一些范围是0和100之间等。\n",
    "\n",
    "> Let’s take a look at the data:\n",
    "\n",
    "我们先来看一下数据："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(404, 13)"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(train_data, train_targets), (test_data, test_targets) = datasets.boston_housing.load_data()\n",
    "train_data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(102, 13)"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> As you can see, we have 404 training samples and 102 test samples. The data\n",
    "comprises 13 features. The 13 features in the input data are as follow:\n",
    "\n",
    "> 1. Per capita crime rate.\n",
    "2. Proportion of residential land zoned for lots over 25,000 square feet.\n",
    "3. Proportion of non-retail business acres per town.\n",
    "4. Charles River dummy variable (= 1 if tract bounds river; 0 otherwise).\n",
    "5. Nitric oxides concentration (parts per 10 million).\n",
    "6. Average number of rooms per dwelling.\n",
    "7. Proportion of owner-occupied units built prior to 1940.\n",
    "8. Weighted distances to five Boston employment centres.\n",
    "9. Index of accessibility to radial highways.\n",
    "10. Full-value property-tax rate per $10,000.\n",
    "11. Pupil-teacher ratio by town.\n",
    "12. 1000 * (Bk - 0.63) ** 2 where Bk is the proportion of Black people by town.\n",
    "13. % lower status of the population.\n",
    "\n",
    "你可以看到，这里有404个训练样本和102个测试样本。数据中包含着13个特征。输入数据中的13特征分别是：\n",
    "\n",
    "1. 人均犯罪率。\n",
    "2. 25000平方英尺上住宅用地比例。\n",
    "3. 每个城镇中非零售工业的比例。\n",
    "4. 查尔斯河二元变量（河边为1，否则为0）。\n",
    "5. 一氧化氮浓度（每千万含量）。\n",
    "6. 每个住宅的平均房间数量。\n",
    "7. 在1940年之前自建住宅的比例。\n",
    "8. 到5个波斯顿就业中心区域的加权距离。\n",
    "9. 到达主要高速公路的排序编号。\n",
    "10. 每万美元的全额房产税率。\n",
    "11. 城镇的学生教师比。\n",
    "12. $1000 \\times (Bk - 0.63)^2$，其中的Bk是城镇中黑人人口比例。\n",
    "13. 低端人口百分比。\n",
    "\n",
    "> The targets are the median values of owner-occupied homes, in thousands of dollars:\n",
    "\n",
    "目标是自建住宅的平均价格，单位是千美元："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([15.2, 42.3, 50. , 21.1, 17.7, 18.5, 11.3, 15.6, 15.6, 14.4, 12.1,\n",
       "       17.9, 23.1, 19.9, 15.7,  8.8, 50. , 22.5, 24.1, 27.5, 10.9, 30.8,\n",
       "       32.9, 24. , 18.5, 13.3, 22.9, 34.7, 16.6, 17.5, 22.3, 16.1, 14.9,\n",
       "       23.1, 34.9, 25. , 13.9, 13.1, 20.4, 20. , 15.2, 24.7, 22.2, 16.7,\n",
       "       12.7, 15.6, 18.4, 21. , 30.1, 15.1, 18.7,  9.6, 31.5, 24.8, 19.1,\n",
       "       22. , 14.5, 11. , 32. , 29.4, 20.3, 24.4, 14.6, 19.5, 14.1, 14.3,\n",
       "       15.6, 10.5,  6.3, 19.3, 19.3, 13.4, 36.4, 17.8, 13.5, 16.5,  8.3,\n",
       "       14.3, 16. , 13.4, 28.6, 43.5, 20.2, 22. , 23. , 20.7, 12.5, 48.5,\n",
       "       14.6, 13.4, 23.7, 50. , 21.7, 39.8, 38.7, 22.2, 34.9, 22.5, 31.1,\n",
       "       28.7, 46. , 41.7, 21. , 26.6, 15. , 24.4, 13.3, 21.2, 11.7, 21.7,\n",
       "       19.4, 50. , 22.8, 19.7, 24.7, 36.2, 14.2, 18.9, 18.3, 20.6, 24.6,\n",
       "       18.2,  8.7, 44. , 10.4, 13.2, 21.2, 37. , 30.7, 22.9, 20. , 19.3,\n",
       "       31.7, 32. , 23.1, 18.8, 10.9, 50. , 19.6,  5. , 14.4, 19.8, 13.8,\n",
       "       19.6, 23.9, 24.5, 25. , 19.9, 17.2, 24.6, 13.5, 26.6, 21.4, 11.9,\n",
       "       22.6, 19.6,  8.5, 23.7, 23.1, 22.4, 20.5, 23.6, 18.4, 35.2, 23.1,\n",
       "       27.9, 20.6, 23.7, 28. , 13.6, 27.1, 23.6, 20.6, 18.2, 21.7, 17.1,\n",
       "        8.4, 25.3, 13.8, 22.2, 18.4, 20.7, 31.6, 30.5, 20.3,  8.8, 19.2,\n",
       "       19.4, 23.1, 23. , 14.8, 48.8, 22.6, 33.4, 21.1, 13.6, 32.2, 13.1,\n",
       "       23.4, 18.9, 23.9, 11.8, 23.3, 22.8, 19.6, 16.7, 13.4, 22.2, 20.4,\n",
       "       21.8, 26.4, 14.9, 24.1, 23.8, 12.3, 29.1, 21. , 19.5, 23.3, 23.8,\n",
       "       17.8, 11.5, 21.7, 19.9, 25. , 33.4, 28.5, 21.4, 24.3, 27.5, 33.1,\n",
       "       16.2, 23.3, 48.3, 22.9, 22.8, 13.1, 12.7, 22.6, 15. , 15.3, 10.5,\n",
       "       24. , 18.5, 21.7, 19.5, 33.2, 23.2,  5. , 19.1, 12.7, 22.3, 10.2,\n",
       "       13.9, 16.3, 17. , 20.1, 29.9, 17.2, 37.3, 45.4, 17.8, 23.2, 29. ,\n",
       "       22. , 18. , 17.4, 34.6, 20.1, 25. , 15.6, 24.8, 28.2, 21.2, 21.4,\n",
       "       23.8, 31. , 26.2, 17.4, 37.9, 17.5, 20. ,  8.3, 23.9,  8.4, 13.8,\n",
       "        7.2, 11.7, 17.1, 21.6, 50. , 16.1, 20.4, 20.6, 21.4, 20.6, 36.5,\n",
       "        8.5, 24.8, 10.8, 21.9, 17.3, 18.9, 36.2, 14.9, 18.2, 33.3, 21.8,\n",
       "       19.7, 31.6, 24.8, 19.4, 22.8,  7.5, 44.8, 16.8, 18.7, 50. , 50. ,\n",
       "       19.5, 20.1, 50. , 17.2, 20.8, 19.3, 41.3, 20.4, 20.5, 13.8, 16.5,\n",
       "       23.9, 20.6, 31.5, 23.3, 16.8, 14. , 33.8, 36.1, 12.8, 18.3, 18.7,\n",
       "       19.1, 29. , 30.1, 50. , 50. , 22. , 11.9, 37.6, 50. , 22.7, 20.8,\n",
       "       23.5, 27.9, 50. , 19.3, 23.9, 22.6, 15.2, 21.7, 19.2, 43.8, 20.3,\n",
       "       33.2, 19.9, 22.5, 32.7, 22. , 17.1, 19. , 15. , 16.1, 25.1, 23.7,\n",
       "       28.7, 37.2, 22.6, 16.4, 25. , 29.8, 22.1, 17.4, 18.1, 30.3, 17.5,\n",
       "       24.7, 12.6, 26.5, 28.7, 13.3, 10.4, 24.4, 23. , 20. , 17.8,  7. ,\n",
       "       11.8, 24.4, 13.8, 19.4, 25.2, 19.4, 19.4, 29.1])"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_targets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> The prices are typically between `$10,000` and `$50,000`. If that sounds cheap,\n",
    "remember this was the mid-1970s, and these prices are not inflation-adjusted.\n",
    "\n",
    "以上价格基本上都介于一万美元到五万美元之间。如果你觉得很便宜，请记住这是1970年代中期，这些价格没有基于通货膨胀进行调整。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.6.2 数据准备\n",
    "\n",
    "> It would be problematic to feed into a neural network values that all take wildly different\n",
    "ranges. The network might be able to automatically adapt to such heterogeneous data, but\n",
    "it would definitely make learning more difficult. A widespread best practice to deal with\n",
    "such data is to do feature-wise normalization: for each feature in the input data (a column\n",
    "in the input data matrix), we will subtract the mean of the feature and divide by the\n",
    "standard deviation, so that the feature will be centered around 0 and will have a unit\n",
    "standard deviation. This is easily done in Numpy:\n",
    "\n",
    "如果我们将上面这些极端不同范围的数据直接输入到神经网络的话，会有问题。网络也许可以自动适应这种极度异质的数据，但是这肯定会导致学习过程更为困难。处理这种数据有一种广泛使用的最佳实践，对逐个特征进行标准化：对于输入数据中的每个特征（即输入数据矩阵中的一列），我们将它减去该特征的均值然后除以标准差，这样处理过后，这个特征会集中在0点附近并且具有单位标准差。在Numpy中很容易完成："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "# train_data = (train_data - train_data.mean(axis=0)) / train_data.std(axis=)\n",
    "mean = train_data.mean(axis=0)\n",
    "train_data -= mean\n",
    "std = train_data.std(axis=0)\n",
    "train_data /= std\n",
    "\n",
    "# test_data = (test_data - mean) / std\n",
    "test_data -= mean\n",
    "test_data /= std"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Note that the quantities that we use for normalizing the test data have been computed\n",
    "using the training data. We should never use in our workflow any quantity computed on\n",
    "the test data, even for something as simple as data normalization.\n",
    "\n",
    "必须注意的是我们用来标准化测试数据时使用的仍然是在训练数据上计算得到的均值和标准差。永远不要在机器学习工作流中使用任何在测试数据上计算得到的量，即使只是面对诸如数据标准化这样的简单的任务时也是一样。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.6.3 构建网络\n",
    "\n",
    "> Because so few samples are available, we will be using a very small network with two\n",
    "hidden layers, each with 64 units. In general, the less training data you have, the worse\n",
    "overfitting will be, and using a small network is one way to mitigate overfitting.\n",
    "\n",
    "因为可用的样本非常少，我们会使用一个只有两个隐藏层的小型网络，每个有着64个单元。通常来说，训练数据越少，过拟合就会越严重，而使用小型网络是减轻过拟合的措施之一。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "def build_model():\n",
    "    '''\n",
    "    build_model 构建模型工具函数\n",
    "    因为我们需要多次实例化同一个模型，因此我们定义一个工具函数来简化代码\n",
    "    '''\n",
    "    model = models.Sequential()\n",
    "    model.add(layers.Dense(64, activation='relu',\n",
    "                input_shape=(train_data.shape[1],)))\n",
    "    model.add(layers.Dense(64, activation='relu'))\n",
    "    model.add(layers.Dense(1))\n",
    "    model.compile(optimizer='rmsprop', loss='mse', metrics=['mae'])\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Our network ends with a single unit, and no activation (i.e. it will be linear layer).\n",
    "This is a typical setup for scalar regression (i.e. regression where we are trying to predict\n",
    "a single continuous value). Applying an activation function would constrain the range\n",
    "that the output can take; for instance if we applied a sigmoid activation function to our\n",
    "last layer, the network could only learn to predict values between 0 and 1. Here, because\n",
    "the last layer is purely linear, the network is free to learn to predict values in any range.\n",
    "\n",
    "网络最后一层只有一个隐藏单元，并且没有激活函数（也就是说它是一个线性层）。这是标量回归（也就是回归的结果是单一的连续值）的标准设置。在这种情况下应用激活函数会限制输出值的范围；例如假设我们在最后一层应用了`sigmoid`激活函数的话，网络只能学会预测介于0和1之间的值。上面的网络中最后一层是纯线性的，因此可以学习到任何范围内的预测值。\n",
    "\n",
    "> Note that we are compiling the network with the mse loss function—Mean Squared\n",
    "Error, the square of the different between the predictions and the targets, a widely used\n",
    "loss function for regression problems.\n",
    "\n",
    "还需要注意我们使用`mse`损失函数（平均平方错误）来编译这个网络，它表示预测与目标之间的平方距离，这是回归问题中广泛使用的损失函数。\n",
    "\n",
    "> We are also monitoring a new metric during training: mae . This stands for Mean\n",
    "Absolute Error. It is simply the absolute value of the difference between the predictions\n",
    "and the targets. For instance, a MAE of 0.5 on this problem would mean that our\n",
    "predictions are off by $500 on average.\n",
    "\n",
    "还有一个地方，上面我们在训练过程中使用了一个新的监视指标：`mae`，也就是平均绝对错误。这是预测和目标之间距离的绝对值。例如，在这个例子中，一个0.5的MAE值表示我们的预测和目标之间有500美元的平均差距。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.6.4 使用K-折方法验证\n",
    "\n",
    "> To evaluate our network while we keep adjusting its parameters (such as the number of\n",
    "epochs used for training), we could simply split the data into a training set and a\n",
    "validation set, as we were doing in our previous examples. However, because we have so\n",
    "few data points, the validation set would end up being very small (e.g. about 100\n",
    "examples). A consequence is that our validation scores may change a lot depending on\n",
    "which data points we choose to use for validation and which we choose for training, i.e.\n",
    "the validation scores may have a high variance with regard to the validation split. This\n",
    "would prevent us from reliably evaluating our model.\n",
    "\n",
    "为了在持续调整网络参数的过程中验证网络性能（比如在训练中使用多次epochs迭代的方法），我们可以将数据进一步分为训练集和验证集，正如我们在前面的例子做的那样。然而，本例中我们的数据点很少，验证集会非常非常小（比如大约100个样本）。这会造成训练时的验证分值会十分依赖我们用来训练和验证的数据点而频繁变化，也就是验证分值对于不同的验证集划分而有着非常高的方差。这会造成难以很可靠的验证模型性能。\n",
    "\n",
    "> The best practice in such situations is to use K-fold cross-validation. It consists of\n",
    "splitting the available data into K partitions (typically K=4 or 5), then instantiating K\n",
    "identical models, and training each one on K-1 partitions while evaluating on the\n",
    "remaining partition. The validation score for the model used would then be the average of\n",
    "the K validation scores obtained.\n",
    "\n",
    "这种情况下的最佳实践是使用K-折交叉验证。它包括将可用的数据分为K个分区（通常K=4或5），然后同时实例化K个相同的模型，在每个模型上使用不同的K-1个分区数据进行训练和剩下的那个分区数据进行验证。模型使用的验证分数将会是所有K个分值的均值。\n",
    "\n",
    "![3-folds](imgs/f3.11.jpg)\n",
    "\n",
    "图3-11 3折交叉验证\n",
    "\n",
    "> In terms of code, this is straightforward:\n",
    "\n",
    "写成代码也很直接："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "正在进行的折： 0\n",
      "正在进行的折： 1\n",
      "正在进行的折： 2\n",
      "正在进行的折： 3\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "k = 4 # 4折交叉验证\n",
    "num_val_samples = len(train_data) // k # 验证分区数据的样本数\n",
    "num_epochs = 100 # 迭代次数\n",
    "all_scores = [] # 存储所有验证分值的集合\n",
    "for i in range(k):\n",
    "    print('正在进行的折：', i)\n",
    "    # 准备验证数据，使用第i个分区数据作为验证数据\n",
    "    val_data = train_data[i * num_val_samples: (i + 1) * num_val_samples]\n",
    "    val_targets = train_targets[i * num_val_samples: (i + 1) * num_val_samples]\n",
    "    # 准备训练数据，使用除了第i个分区数据外的其他数据作为训练数据\n",
    "    partial_train_data = np.concatenate(\n",
    "        [train_data[:i * num_val_samples], train_data[(i + 1) * num_val_samples:]],\n",
    "        axis=0)\n",
    "    partial_train_targets = np.concatenate(\n",
    "        [train_targets[:i * num_val_samples], train_targets[(i + 1) * num_val_samples:]],\n",
    "        axis=0)\n",
    "    # 使用刚才的工具函数构建每一折的网络\n",
    "    model = build_model()\n",
    "    # 训练模型（verbose=0 表示静默模式，无输出）\n",
    "    model.fit(partial_train_data, partial_train_targets,\n",
    "                epochs=num_epochs, batch_size=1, verbose=0)\n",
    "    # 在验证数据上验证模型性能\n",
    "    val_mse, val_mae = model.evaluate(val_data, val_targets, verbose=0)\n",
    "    all_scores.append(val_mae)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Running the above snippet with num_epochs = 100 yields the following results:\n",
    "\n",
    "在上面的代码执行100次迭代会得到下面的结果："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1.9331642, 2.6493096, 2.4610572, 2.3756325]"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.354791"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean(all_scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> As you can notice, the different runs do indeed show rather different validation\n",
    "scores, from 2.6 to 3.2. Their average (3.0) is a much more reliable metric than any single\n",
    "of these scores—that’s the entire point of K-fold cross-validation. In this case, we are off\n",
    "by `$3,000` on average, which is still significant considering that the prices range from\n",
    "`$10,000` to `$50,000`.\n",
    "\n",
    "你注意到，每个折在执行中产生了有差异的验证分值，从2.6到3.2。它们的均值（3.0）是一个更为可靠的指标作为模型性能的衡量 - 这就是K-折交叉验证方法的全部内涵。在这个例子中，我们与实际目标差距是3000美元，这对于价格取值区域从10000美元到50000美元来说，仍然是一个较大的差异。\n",
    "\n",
    "译者注：原文照译，如果按照译者自己跑出的结果，平均绝对错误的均值应为2.5左右，实际目标差距在2500美元左右。\n",
    "\n",
    "> Let’s try training the network for a bit longer: 500 epochs. To keep a record of how\n",
    "well the model did at each epoch, we will modify our training loop to save the per-epoch\n",
    "validation score log:\n",
    "\n",
    "让我们试试将网络训练更长的时间：500次迭代。为了记录每次迭代模型的表现，我们在训练循环中修改一下代码来保存每次迭代的验证分值日志："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "正在进行的折： 0\n",
      "正在进行的折： 1\n",
      "正在进行的折： 2\n",
      "正在进行的折： 3\n"
     ]
    }
   ],
   "source": [
    "num_epochs = 500\n",
    "all_mae_histories = []\n",
    "for i in range(k):\n",
    "    print('正在进行的折：', i)\n",
    "    \n",
    "    val_data = train_data[i * num_val_samples: (i + 1) * num_val_samples]\n",
    "    val_targets = train_targets[i * num_val_samples: (i + 1) * num_val_samples]\n",
    "    \n",
    "    partial_train_data = np.concatenate(\n",
    "        [train_data[:i * num_val_samples], train_data[(i + 1) * num_val_samples:]],\n",
    "        axis=0)\n",
    "    partial_train_targets = np.concatenate(\n",
    "        [train_targets[:i * num_val_samples], train_targets[(i + 1) * num_val_samples:]],\n",
    "        axis=0)\n",
    "    \n",
    "    model = build_model()\n",
    "    \n",
    "    history = model.fit(partial_train_data, partial_train_targets,\n",
    "                    validation_data=(val_data, val_targets),\n",
    "                    epochs=num_epochs, batch_size=1, verbose=0)\n",
    "    mae_history = history.history['val_mae']\n",
    "    all_mae_histories.append(mae_history)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> We can then compute the average of the per-epoch MAE scores for all folds:\n",
    "\n",
    "然后来计算所有折上每次epoch的MAE的均值："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "average_mae_history = [np.mean([x[i] for x in all_mae_histories]) for i in range(num_epochs)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "把所有epochs上的平均绝对错误绘制出来："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0, 0.5, 'Validation MAE')"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO2dd3gc1dWHf2erumTJsixX4V5wFxibbmOKqQFCCSGEUEISEhLSDIQaOgQSWhIg8NFDDc1gDDYGY+Peey+yJatYXdrVlvv9MXNnZ2arykqW57zPs8/OzszO3lmt7rmnkxACDMMwjHWxdfUAGIZhmK6FBQHDMIzFYUHAMAxjcVgQMAzDWBwWBAzDMBbH0dUDaC09e/YURUVFXT0MhmGYbsXKlSsrhRD5kY51O0FQVFSEFStWdPUwGIZhuhVEtDfaMTYNMQzDWBwWBAzDMBaHBQHDMIzFYUHAMAxjcVgQMAzDWBwWBAzDMBaHBQHDMIzFsYwg2FpWjyfmbkVlg7erh8IwDHNEYRlBsKO8AU/N34HDjS1dPRSGYZgjCssIAhspz0FuxMMwDGPAMoKASJEEwWAXD4RhGOYIwzKCgDUChmGYyFhIECiSgOUAwzCMEesIAvVOWSNgGIYxYhlBoPkIWBAwDMMYsI4gUJ+DLAcYhmEMWEYQSB8BwJKAYRhGj+UEAWsEDMMwRiwkCJTnIEsChmEYA5YRBMQaAcMwTEQsIwikRiA4aohhGMaAdQSBjTUChmGYSFhHEHCJCYZhmIhYRhBwQhnDMExkrCMI1GeWAwzDMEYsIwi0onOcUMYwDGPAcoKA+xEwDMMYsYwgIHYWMwzDRMQygoBLTDAMw0TGOoJAvVNOKGMYhjFiHUHAGgHDMExELCQIlGf2ETAMwxixjCCQmQQsCBiGYYwkXRAQkZ2IVhPRpxGOuYnobSLaQURLiagoWeOQGgHDMAxjpDM0glsAbI5y7DoA1UKIIQCeBPBIsgZh4xITDMMwEUmqICCifgDOBfBilFMuBPCKuv0egOlElJS1OyeUMQzDRCbZGsHfAfwJQLTpty+A/QAghPADqAWQZz6JiG4kohVEtKKioqJNA+GEMoZhmMgkTRAQ0XkAyoUQK9t7LSHE80KIYiFEcX5+fpuuIfsRsBxgGIYxkkyN4EQAFxDRHgD/BTCNiF43nXMAQH8AICIHgGwAVckYDIePMgzDRCZpgkAIcZsQop8QogjAFQDmCyF+bDrtYwDXqNuXquckZabmhDKGYZjIODr7A4noPgArhBAfA/gPgNeIaAeAw1AERnI+V31mjYBhGMZIpwgCIcQCAAvU7bt0+z0AftgZYyCtHwHDMAyjxzKZxdJHwEXnGIZhjFhIEMg8AhYEDMMweqwnCFgOMAzDGLCMICD1TtlZzDAMY8QygkBrXs9ygGEYxoCFBIHyzBoBwzCMEcsIAgL7CBiGYSJhHUHAGgHDMExELCMIbMmpbs0wDNPtsZAgUJ45j4BhGMaIhQQB+wgYhmEiYRlBwD4ChmGYyFhIEBCIuNYQwzCMGcsIAkAxD7FpiGEYxojFBAGbhhiGYcxYShAQWCNgGIYxYy1BQIDg1jQMwzAGLCUIbERcdI5hGMaExQQBJ5QxDMOYsZggYB8BwzCMGUsJAuKoIYZhmDAsJQhsNuKEMoZhGBPWEgRsGmIYhgnDUoKAwKYhhmEYM9YSBEScRcAwDGPCUoLAxkXnGIZhwrCYICAEg109CoZhmCMLiwkC9hEwDMOYsZQgII4aYhiGCcNSgsBmYx8BwzCMmaiCgIje0W0/Yjo2N5mDShZKHgELAoZhGD2xNIKhuu0ZpmP5SRhL0lHyCLp6FAzDMEcWsQRBrCmzW06nrBEwDMOE44hxLI2IJkARFqnqNqmP1M4YXEejNKZhGIZh9MQSBKUAnlC3y3Tb8nW3Q2lMw6KAYRhGT1RBIIQ4PdoxInImZzjJhRPKGIZhwkk4fJQUphPRfwCUJHFMSYP7ETAMw4QTVxAQ0QlE9BSAvQA+AvAtgBHJHlgy4DLUDMMw4cTKI3iQiLYDeADAOgATAFQIIV4RQlTHuzARpRDRMiJaS0QbiejeCOf8lIgqiGiN+ri+PTcTD04oYxiGCSeWs/h6ANsA/BPAJ0IILxG1Zhb1ApgmhGhQfQrfEdHnQoglpvPeFkLc3Lphtw0OH2UYhgknlmmoEMD9AM4HsJOIXoMSRhpLeGgIhQb1pVN9dOkszAllDMMw4UQVBEKIgBBijhDiGgCDAXwIYBGAA0T0ZiIXJyI7Ea0BUA7gSyHE0ginXUJE64joPSLqH+U6NxLRCiJaUVFRkchHRxsP5xEwDMOYSChqSAjhFUK8L4S4FMAQAHMSfF9ACDEeQD8AxxPRsaZTPgFQJIQYC+BLAK9Euc7zQohiIURxfn7bq1twYxqGYZhwopp5iOjWjvoQIUQNEX0N4GwAG3T7q3SnvQjg0Y76zEiwj4BhGCacWBrB4wB+DCAPQAaATN0jI96FiSifiHLU7VQoheu2mM4p1L28AMDm1gy+tXBCGcMwTDixHL8TAFwJ4FwAKwG8BWCeSNy2UgjgFSKyQxE47wghPiWi+wCsEEJ8DOA3RHQBAD+AwwB+2rbbSAxOKGMYhgknVomJtQDWAphFRFOhCIWniejP6iQeEyGEzD0w779Lt30bgNvaMvC2YCNCgMOGGIZhDCSSWZwPZUIfA6W0RHmyB5UsbDbWCBiGYczEchb/DMBlAFIAvAfgMiFEtxUCAEBgZzHDMIyZWD6CF6FE+OwFcBaAM4lIOyiEuCC5Q+t4bDZCgOUAwzCMgViCIGoZ6u6K00bwBzhsiGEYRk8sZ/E3nTmQzsBhJ/hZJWAYhjGQcD+CowGH3QY/JxIwDMMYsJYgsBH8HD7KMAxjwGKCwMamIYZhGBNxS0oT0TAAfwQwUH++EGJaEseVFJx2go+dxQzDMAYS6S3wLoB/AXgBQCC5w0kuDjtnFjMMw5hJRBD4hRD/TPpIOgGHzcYaAcMwjIlEfASfENEviaiQiHLlI+kjSwLsLGYYhgknEY3gGvX5j7p9AsCgjh9OcnHY2VnMMAxjJq4gEEIc0xkD6QycduI8AoZhGBOJRA05AfwCwCnqrgUA/i2E8CVxXEnBbiMEBRAMCthsFP8NDMMwFiARH8E/AUwC8Jz6mKTu63Y47crt+lgrYBiG0UjER3CcEGKc7vV8IlqbrAElE4eqBfgDAu5E7pxhGMYCJKIRBIhosHxBRIPQTfMJHKpGwA5jhmGYEImsi/8I4Gsi2gWAoGQYX5vUUSUJTSNg0xDDMIxGIlFD84hoKIDh6q6tQghvcoeVHBx2KQhYI2AYhpHEalU5TQgxn4guNh0aQkQQQnyQ5LF1OE6b6izm7GKGYRiNWBrBqQDmAzg/wjEBoNsJAk0jYB8BwzCMRqwOZXerm/cJIXbrjxFRt0wys9vYNMQwDGMmkaih9yPse6+jB9IZyDwCdhYzDMOEiOUjGAFgNIBsk58gC0BKsgeWDPR5BAzDMIxCLB/BcADnAciB0U9QD+CGZA4qWWiZxewsZhiG0YjlI/gIwEdENEUI8X0njilpSB8BN6dhGIYJkUhC2Woi+hUUM5FmEhJC/Cxpo0oSMmrIx6YhhmEYjUScxa8B6A3gLADfAOgHxTzU7WBnMcMwTDiJCIIhQog7ATQKIV4BcC6AyckdVnJgZzHDMEw4iQgC2XeghoiOBZANoFfyhpQ8HDapEbAgYBiGkSTiI3ieiHoAuBPAxwAyANyV1FEliVBmMZuGGIZhJIkUnXtR3fwG3bBPsR6ndBazRsAwDKMRK6Hs1lhvFEI80fHDSS6aaYg1AoZhGI1YGkGm+jwcwHFQzEKAkly2LJmDShZ2dhYzDMOEESuh7F4AIKJvAUwUQtSrr+8BMLtTRtfBuJ2KRtDCGgHDMIxGIlFDBQBadK9b1H3dDrfDDgDw+lkQMAzDSBKJGnoVwDIi+p/6+iIA/5e0ESURt0ORe15/t2y5zDAMkxTiagRCiAeg9CiuVh/XCiEeivc+IkohomVEtJaINhLRvRHOcRPR20S0g4iWElFR628hcTRB4GONgGEYRhIraihLCFFHRLkA9qgPeSxXCHE4zrW9AKYJIRqIyAngOyL6XAixRHfOdQCqhRBDiOgKAI8AuLyN9xIXIoLLYWPTEMMwjI5YpqE3oZShXgmlNaWE1NcxcwqEEAJAg/rSqT7M4ToXArhH3X4PwDNEROp7k4LbYWPTEMMwjI5YUUPnqc9tbktJRHYogmQIgGeFEEtNp/QFsF/9HD8R1QLIA1Bpus6NAG4EgAEDBrR1OAAUhzFrBAzDMCFimYYmxnqjEGJVvIsLIQIAxhNRDoD/EdGxQogNrR2kEOJ5AM8DQHFxcbu0BbfDBo+PNQKGYRhJLNPQ32IcEwCmJfohQogaIvoawNkA9ILgAID+AEqIyAGloF1VotdtC24n+wgYhmH0xDINnd6eCxNRPgCfKgRSAcyA4gzW8zGAawB8D+BSAPOT6R8AVNMQRw0xDMNoJJJHALX89CgYO5S9GudthQBeUf0ENgDvCCE+JaL7AKwQQnwM4D8AXiOiHQAOA7iiDffQKthZzDAMYySuICCiuwGcBkUQfAbgHADfQUk0i4oQYh2ACRH236Xb9gD4YatG3E7cHD7KMAxjIJESE5cCmA6gTAhxLYBxUGz53ZIUJ0cNMQxz5NHcEsAz87fD1wW10BIRBM1CiCAAPxFlASiH4uDtlrgdNng5aohhmCOM5xbswONzt+G9lSWd/tmJCIIVavjnC1ByAlZBce52S9xOO1pYI2CYNlM0azZufjNu9DjTSmRYe12zL86ZHU9UQUBEzxLRiUKIXwohaoQQ/4IS+XONaiLqlrCPgGHaz6frSrt6CEcFC7dXoGjWbByoaYbTrkzHXWEaiuUs3gbgcSIqBPAOgLeEEKs7Z1jJg6OGGCb5HKrzAAAKslLinJk8Nh2sQ5rLjqKe6ZizoQxOO2H6yCOjgn51YwvOf+Y7NLUoc9Gi7ZVwqUUxu8JiEVUjEEL8QwgxBcCpUJK8XiKiLUR0NxEN67QRdjAyjyAYFJj1/jpsLq3r6iExTLch0TSfyQ/Ow+QH5yVlDLsrG/HVpkMAgIc/34KiWZH7ZM18aiFOe3wBAOCm11fiuldWJGU8bWHhjkqUVDfjcKPS6mX9gVpNI2jRdVDcfqgeTS3+pI8nkTLUe4UQjwghJgC4Eko/gs1JH1mSkJnF+6ub8N/l+3Hja0fOj4NhjnR8R0Cb19MfX4DrX12BsloP/vXNTgBAIBh9XL97e03UY7sqGrBmf027xuMPBMNW8Yt2VOJLVVgBwJ7KRvxt7lZNkJoF6rqSGq2FrjQN7T/chBlPfotH52xt1/gSIa4gICIHEZ1PRG8A+BzAVgAXJ31kSSIv3YWWQBAl1c0AABtRF4+IYboPbTGrNnr9GP6Xz7VVfGvYVdGAJ7/cFlETOeGhkMYRa1z/W30g6rG/zd2GWyMICjkZL95RiW+3VUR9/5tL9+H8ZxZh2F8+x+IdoVqZV724FDe8ugKbDtahaNZszHjyGzw9fwf2VjUBAIKm+9lT1QSPeg+NXkUDeGfFfgDAgZrmqJ/fUcRyFs8gopcAlAC4AUqf4sFCiCuEEB8lfWRJQtoIP1uvOLtYEDBWpWjWbPzpvbWtek9rAy08vgB2VTTC6w/iya+2JfSe5paANvF/tr4U/5i3HRX13jifE9RtB6Kai/Rc/8oKzF5fioO1zQZBU1LdhKF3fI7Xl+zFj15cip+8tMzwvgM1zahpaoEQAn+bu1UzLz82N3zl/v4qJRRUalIN6iTf3BIab1FeGmqbfThUq/hV6jxK1NCmg8p1ZUOtZBLrE24DsBjASCHEBUKIN4UQjUkfUZI5pmc6emelYKP6JbMcYFpLg9ePFxfuQjCGOaK78M4KY8z6lrI6/OXD9VFNLdEEwZebDqGyIXyyrmzwwhdU3uOwx5/QSmubMfKuObjv00049bGvsb1caWlSpjqfo9Gsyw1avS++qccfCOKrzYqG4vEFUecJ2eFX7q0GAPzlw/BCyf5AECc+PB9X/2cZSms9qGoMtXNfva8GuysbDd/d7krjlFndpJwvJ3sAGNMvBwCwpaxeOdasjKVUFQwr91ajpLop7j21h1jO4mlCiBeFENVJHUEXkO62a7G6rBEwreX+Tzfh/tmbsWBbeVcPpc1Ec/re8tYavL5kH/ZURV7zRUrGbG4J4IZXV+BHLywJO1bZ0KIVeXTY4v+v7VAn/pcX7cHeqiZ8tOYggNCkGA19afllu6M3TwwEBdbsrwkztzz2xRbM2VAGANhcWh/2Pml6+k41/6w/UIsNB2q148MKMgAAy3cfxuDbP9P2z99i/I1I57A+V2BsX6VQw9ZDyufWqsdk5FVprQcnPfJ11HvqCJKvcxyBpLkcqNEEQRcPhul2yH9UvXofiw0HarG+pDb+iZ1ItJW9DGGsjGKKifQ+ubrddqgh7FhFvVezeccSBC8v2o0ZT3yjhVOaKWuFINh4MPp3/e6K/bjo2UV4/ttdhv2vL9mHm15fCUDRisyU1ynfh3QAj+idaRAEEwf0QI80pyYoolHV0ILP15fiuQU7tX2j+2YBCDm8S2ub4fUHDNoGgKRqoAlVHz3aSHXatX9m1giY1iJ/M2aHXzTOe/o7AMCeh89N2pgSxeMLwGmP3pwpK1WZEqKtwPWCQAgBIgrLhNUnRFU3tmjCxRnDNHTvJ5u08yMRzzTk8QURCAo0tfhxsDa6c7VeNQG9sXRfxONCCJTWhH/W/uomfLX5kPa+xhY/NhwMCQwiwoxRBWGmNjP/W30Am0wh6wVZKSjIcuOQKmwqG1qwPYJQ3VXZiCG9MmJev61YUiNIddk16cuCgGkt8ifTHT0EI+6cg9s+WGewqevJdDsBRI9UWVcSsr9LoVCrEwRef8Bw7dpmH+pVjcFhj/+/Jn0CZqRGEE2Azd1Yhvtnb8KYe+ZqkTmA4hPUE094H6z1aOPVc/dHGzVhlZ3qxP7DzQazjxACvz9zuPb6sUvHIsVp086XrD9Qi745qYZrZ6c6MTDXOM7FOxXNIjMltFaPZfJqL5YUBGkuu7bNcoBpLaT+aJLcQ6nDkbHu76woMUTZ6PGrC6SDEQTBrooG3PXRRu21VhtHN3GWVDfD02ISBAmYhiTbDoXb54GQIKj3RE6u+ve3u/Dyoj1h5/zq9CGG88zmFjM7yhtQ7/GjV6Zb25eb7sL28gZMHZyHzfedjWtPLNKOnT26NwAgL8NlyKIeWZgFacmR/gPJZcX9DK+zUpwYkJcGABioPi/aoTRqfP7qYjz4gzEYkJuGLzaWxRx7e7CkIEh1hgQBawRMa5HzWbLlwPwth/DJ2oMddj19hmq0lbVcDZfXe3HLf1djkc7mXdlgnESX7DqMWe+vQ02TSRDohExNcwsa1Ik5ke9ra1kUQaCahmqaYk/kZnqbSlxE831Iyus8aGjxo3d26H3P/mgiPrn5JDx95QSkuuzokeYCAEwZlIfnrpqI+y4cjV9PG2q4Tq8st7ZQGFqQCQBIcdpw5fH9cflxA/Ddn0MNIF0OG4aqJp+cNBdSnXZ8o+YujOqThR9NHoALx/fBN9sq8MriPa26/0Sxpo9ApxHY2FvMtJLW+gjays/+T8l6P39cnw65XqNupa4XBC3+oGbHl3HuB6qbsam0Dh+tOaj5NszCQzpX9aaPkuomw+T7+pJ9KFQnVU8CyWjlUSbq0tpmbC2rx09fXhbxeDR6Z7sNrz+IkVwGAPsON0EIWSNJcQZnpzoxqk+Wdo40+eRnumGzEX4ypUg7lul2oN7rR166WxN8w9RJvl+PNDx08Vjt3MWzpmGPGl564pCeAIC1+2tw/DG5WLb7MNJcdmSppqHfTB+KvVVNGJCb1qr7TxRLagR60xDLAaakuqlVWa/yJ9NaObCvqm2x4OVxHKWJIqN3AGPcfYNuv9zeVanY6u26f5CaKOWRN+tW8SXVzWHO2lLNvh/Ea0v24vEvwhOvYv0f9s5KgccXxFl//xaltR7cff6o6CeryAnUZbfjrxeOxo2nDIr7HiAU968XZvmZRmEizWdFJv8DAMz+zcl47qqJsNsI/XoovoBhqkZgriraJycVU1UBMKpQETRnjirAhP5KXkFmikMzQzrtNjx15QScPqJXQvfRWiwpCNg0xOi56NlFuP7VFQnb/OU/Z6CVkuCUx77G28sjR6tIAkGBfy7YaXDAylo4ZbUevLUs9vslby/fh6F3fAa/bvLRCwKvznyz73CTFpoozTjSvGPX/X/URjHLbDpYixSnDQPz0rDvcBOufXl5xPNW7avGvR9vxCuL94R913qBM6avsQHimH6h1ylOG346tSji9SVEwEc3n4RbZwxD/9xUXD2lCKN1K3pJJJ+FzJ/Qm4by0l2Gcy6Z2A+/O2MYfnna4LD3D8hLw8wxhQCA166bjEcvHYs+qnPYFyMr22YjrL3rTDz9owk4c7RS/UBGEXUG1hQErpBFjDWC5NPcEsBv3lodNxa8q5C270TLJ8i5sS1141ftjZ31+tn6UjwyZwv+/tU25KoTkEws+vlrK3DbB+s1DaG83qMlKAGKg1dO6Pd8vAm+gDCs9vUx+nqN4KJnF+GfavG2eq/RGdsSCGo5ENVNkTWCyoYWZKU4MbJ3FhbGqMsjhLKarvf6UaHLQvb6A1oJhjvPGxUWVXOW6pAFgHSXskqOFYrbK9ONY3qm4zfTh2pC2+2wh503um829jx8Lp68fBxcDhuGF2Rib6Witek1ArP5OMVpxy1nDEWKM/yaevrnpuGy4v5IdyvzTUF27JLc2WlOuB12TBzQA31zUnHDycfEPL8jsaQgMEYNsSRINnM2luLjtQfx8OdHdtHaaMlMZuS80Ja68fHs5NI04XLYNBOUnLQ1h6mqLRz/wDxM/OuXAJRol6kPz8dLi3YDAPxqWQf9Pek1ArO56ctNh+D1ByLe0/nPKHkQNVEEAaBE1kwdkmco1RCJkaoJ5PgH5qG0thmBoNBKQtx34Whcd9IxBiEBADNGFuDhi8eo9xVfC+tjEiSAUnU4DFUr+cGEfth2/zmw20gThL3jTNqtIT/TjScvH4d/Xz0pofOJCN/9+XTccW58E1hHYUlBoDcN2VkQJB2bZkrp4oHEodGbWN13eT9t0QiaTcJm+Z7DuPSfi7USBjJssyAzRdM8zGaaSDV9ZDbt8j1KrLmcMPWRQo26bXPCWKPXjx/+S+lAK0MY9bT4gzEjdvrkpGLq4DzDvosn9A07b9qIfG37zg834tE5W3DF80ppChkzX2oKXc1Oc2r2+FjlpiX9e4SPP1LhNvMiUC9Aepn8Au3lBxP6oVdm4sKlsxeo1hQEOo1AdMu0oO6FFmVzhBdpa0hQEFAUjaCi3hsxGUmPx/SeO/63Hiv2VmuagEzkCgqhTXpSI5CTQ0W9F1c8b2wbLid2GdooTfCNXr1GENo2R+dsL2/AOtUEJCNY9HPRtkP1qGn2wUbAH88aDjMFWSkYnJ+hOVbfvH4ynrh8PC4v7m84b8qgnvj5qYNwxshemLflEP6tK/Ugxy4n5B9M6Iuxqn9ARh5JTcdMH/X4xRP74k9nh4/PpWY1pzhtWH7HGfjhpH7422XjDOc8cskYXH3CQAAhjWDmmN6wApYMH9WbhvxH+jL1KEA6AvXhltsO1SMv3YW8jI5debWHRDUCSYvpt3PcA18hL92FlXfO0PaZnaIek0Yg/VWy4qQUBF5/UPNZSC1CTswl1c1YssuYZSqLtb27sgRXTR4YuqcWP3xq/w29dlBeH91fI803Z4/ujTvPG4WpD8/Hgq3lOFTnwanD8vGr04fgMTXyZ3B+OnZWNCI/0w0iwklDemLB1nItGua6k4/B4l1KNy4hFG3jpKEjUVbrwYJH5sOv+37y0pXfwj9/PAlbyupw8tCQ9iCTtaYO7qnte+EnxQgKgSmD8+B22NDoDSAn1RkxJFx+lxP690B+phuP/XBc2Dl5GW789aJjce8Fo2GzETbcexZSOqEE9JGANe7ShD4cLBGbI9M+5P+lXq0/88lvcdbfF3bK5+8or8cTUZqb6NEXEYuFdGxGsqebM1fNvy+zjyBdXZRU1HsRDAocUBsmtegEgccXQDAotDj+vabKoEII7D+sODkDQaHZ9AHgno834t5PNuL0xxdgZ3nofeVRIlL+9eNJyFSdm4XZqeiTk4rJx+Ti8bnbsPFgXZj9faxaQlly+8yReO26ydrrYQWZWPinaXjkEqXkglzZ985OwbljCw3vzc1QNIL8TLdBCACKg/bzW07G01dO0PbNGFWAs0b3RlaK4mTNTXdFzQvKSlFyHSYOzIl4XI+8RobbkVDp7KMBa9yliX46G2I0VZPpOChKAlYkW3cy+PGLy/DUvO2GkMxI3PPJJqwrqVGyS2NoBzIkc+3+Gjz0+Wbc8b/1Btu/XuCYNU6zjyBN1QjK6z04VO/RJv+mFr/BNHTL22u06Ka9pnyEBq8/6r1tO9SA15coIadvqx2vACUSyWw3L8hy4+xjeyNDEwTKpD3rnBHaOWZBcNpwZcI+pqfyP5Wf6caxpvBPALisuD+2/PUcw8T60MVjcN+Fo7XX5jBNMyMLs7QInNYypl823rtpCm6dEW42YiwqCHqkhTIh2TSUfKSPYN6WchTNmq2FQ3YW0saeSHhoZYMXP3lpGR6YHT3CyadO0N/vqsK/v9mFN5buw3srQ5Os3v7eYnIomzUCl4O09+ibmEhTEaBoBPpSE+ZmJ0pht8TMWlIbrvP4kaP7PwCgOTPlZCvt5BMG9NDMRTK0846ZI3HFcf1xwbg+ePemKbhofLhjOB5pLodmkwcQNxyzvRQX5RryFZgQlhQEeo98WyI/mLYhF8rrOqk2vxACe6saNdt6Ij4Ap92G8npvzJr2/gi/GX19nT26idr8+zL3MJAO3Ip6L3bqKm/qC7k1+4IYp2abFuWlhTl6NxyoCysFHYl0lx2LZ03TJkPpnJXISJkx/bLxgwl9DVFA2Wp5aqkt3HDKIDx8yVgQEY4rym1zlAuHbx8ZWFIQ6GEfQfIJmMxvbYm/bwvvrSzBqY8t0OLfE+RpwLcAAB8SSURBVMkT8PiC8PgC2FneACEEnvxyW1g2cCQtUr/y19fON5/b4PXhV2+s0vwRUjhVNnixpaweWSkOFGanGFb4npYAPC0BnD26N04dZrSdA0rNn3qvH1dNHoDXdfZ5PbfPHIElt0+H027Tyi/kprsw57cnazXu+6olETLcDjx5+XiDI//u80dj0sAeOH5QbsTrM90bywqC72+bhvH9c9g01AF4fAFc9OwirRSCmTCHaZTKlx3N97uqDK93VjRojcYlYSt2n1JPv7ElgEN1Xvxj3nb8+f31Bi3AF2HxoM/wLa/zQgiBD1cfCPM1eHxBzF5fij+8qzSNl8erm3zYUlaPEYVZcDtsJo0ggAavH+luR8RkKcng/AycNLQnzh1TqGUlA8D2B87BjacMRqbqMM1Si8T1SHNhRO8szVQ6aWCPqNceWZiF938xVXO6diQ3nz4EfzhzWIdfl0kcywqCwuxUDC/IZGdxB7DhQC3W7K/BvZ9sjHjcnAQUrSlKR+M11dy/5b9rcM4/jJFKZi2httmnmbB26Ew1Q+74HP9V6/xEMg3pzTXl9R58v7MKv317TdTvRCKTvKobW7C1rB4jemfC7bAbTD0eXwD1Hh8yU8IFgb6Gjpzgn75yApbcNl3bb+4MJifzHunKs/zzHFfUNav9P5w1HDebyjgznYtlBQGgdExijaD9yG8wmrXX/B23RyOobmzBXR9tQG2McgfxPufROVu0yB4ZxfPzU5XqlDW6lf16Uzjpst2HsXzP4YimLb0DvLzeq630F26P3MN2S1k9Pl130OAjaPAqDVFcDptmGspJc6KpRdEIMtwOraKlZKyuIJs0+dhspJWVjuQc7ZOjOIFzVR/B3y8fj0cvGRtT22CObiyZUCZx2OiIdhY/MHsTema48fNTw6scHknEa/tp1rraIwjeWr4Pr36/F00tATxuSgoqr/cgP8OtOSCjaR7PLdiJq04YiL45qVqSlazzflhXRmHZbqNp6dP1pVHr2cvaPbnpLhyq84Q1cYnEzW+u1iZsOdZUl8NgGspJdWKPGi6akeII61mbqTPVZKUazTZf3Xqq1oNYz/CCTHyx8ZD2mf1z09A/SXXume6BxTUCW0K1S7qKFxbuxkOfb+mUzyqaNRuPzmnbZ3l8xsxXM2YfQYM3fIJOtAT0yj3VABRH8JwNodZ9y/ccxvEPzMNn60P7DsdoS/jc1ztQXufRTEMyhl3fPP3rrcZKmrGc3HKyHpyfjr1VTSiL0UDdfE19gmO6yw6Xw6YlremFa7rbYZj4AaAoL1QT32y/H9IrI2J9G1lConc2awCMgsUFARkcf00t/k5zZHY2NU0tUe9N1gB6bsHONl27qcVYC8eMWdhGqsfzk5eWoWjW7LifteFgLc5V671vKQs5fj9YVQJAqa0PAF9vLcfWKP1vAeCNpftw/IPztFVxutsBl8OmlVo295k1YyPgmR9NwLs3TTHsv3pKEUprPXhuwU4ta9hMXrrLkDylt/OnuuyGRC99k5dMUzLVsjumGzSEvIzYCVmSyYPy8OmvT8I1UwbGP5mxBJYWBE6bzeD4G3XXF5j5lNGZuK+qCRc/tyghm/SRzPj7vsRl//4+4rFE6/BHQwqCaLk6Zh9BpFLFZlv6yr3VOPnR+WFCo6bJh749UpHushtCLGWdf1nB8vudVXDabRgUoYuUHlntM81lR6rTrlXYvG3mSMN5uaas19OG98J5Y/sYHKzP/GgCLhjXBy67Df6gQKHJ5i6TsV64phhzfnuKtr9YF62T7nJo5iIAePrKidq2bJH4wS+n4u+Xj0evzBQtrh8AeraibtOxfbMtUz6BiY+lfwl2GyEolBWxXBXvqjBmbT41fztW7avBF5vKIl0CgBIDXtVJ5RLaQ7RELrMtvarB2yrNSNrZyeQulppAIhqBmb/N3Yr9h5u1WvWAYoLy+oPITnUiM8WJumYfFu2oVGr0qBN6c0sA322vxOcbSpHpdmjFyiSTjzFGxqxQTU2pTgdSnDbNR5DqtOOtG07QzjM7aSOVdJDZt1IY9dSt0C8a3wef//Zk7HjgHEwc0MNgDpo0MDSmNLddcyD/6vTBmDGqAFdNHgAgZFKbOKAHLlJLPMvPAiI7hhkmESwtCJx2ta58UKnOGAmpMchzI1F8/1eYdP9XHT/ATkIvCIJBgUn3f4Wb31yd8PvlxKW3DK3aV43Bt3+GpbuqwnwEiZRDkCGPvkAQGw/W4nBjixZSmZXqRGaKQ6m0+eJSvLZkrxal0+wL4Mf/WYr9h5uR5rajV5ZxlSwna8mKvYog0DSCRuUzUpx2TBmch8cuVZqNm1sdRhIEUuhkqJNzXrpba+x+3UmDkJXijLgK19f/T3M5UKqag2Rc/5/OGoFrpgzUTGJ6MtpYe4dh9CRNEBBRfyL6mog2EdFGIrolwjmnEVEtEa1RH3clazyRkJNNc0sgqj1ZTmJ2W/eVmZGcnPpVur4QmvwevtqceDN3qRHo/b1fbykHoCR1mePuY5VDkE5jvSA496nvcNGzi7TJN1sVBJJlu0MlmfV5AekuR1g9fPPKXiaYpbnsSHHatQ5VsnnRDyb0xW3njMCd5xm7RUXq1iUnZTm27DSnZreP9POZ+7tT8OYNkw2lHtJddq23wEDVEZyd5sS9Fx5r6KOhfWYKCwKm/STzV+QH8HshxCoiygSwkoi+FEJsMp23UAhxXhLHEZXxav2WhdsrNSej2c4q7duRGl2bCQTFEamem00/a/bX4KJnF+HtG0/A5EF5BjOQnFRH9M5M+PpSI9B/jlyhZ7gdYRpAXQzTkD8o4LSTpoEdqFEmxX2HmwyCQB8qeUDX0Up/7VSXHVOH9MQb10/GqMIsrC2pMbQgHNM3W8sVSHXZDROtFAQOuy1i+G5tc/SIpHS1omiPNCcKMlOwq6LR0BRGMqwgE8MKlO/Z7bDB6w8q43Da0dQSiNhpy4wUmFMG5cU5k2GikzRBIIQoBVCqbtcT0WYAfQGYBUGXUVyUi54ZLszbfEhLijJP+H4tRj7+9Q7VefDSd7tR5/Hh0UvDG190FWZ7/6IdimP2Lx9uQHm911DjXda1N1emjIXUCPSf06BO/uluR9jqWV9Z09y1zOMLwGm3aRPcFl1JCDnJSx+BZM3+Grgdynv0Bd/khCzDJU8b3ku7P0AxvUhBkOZyGFqYpkTocZuT5kSG24GS6mYtvBMAvv7DaYaCdjLkMyfVhScvH4/nFuzAxAGx6+DnprtQWutBusuBt38+BRsO1BqcxrFYctv0Vv29GMZMp+iVRFQEYAKApREOTyGitQAOAviDECIsJ5+IbgRwIwAMGDCgw8ZltxHG9M3Gh2tCJX7Nq2eZDJVIZE1JdTNe/E5pHt5eQdBR+Q3rSmrCHOBytb1dLaGwtSxkFmtUTSuJtm3Uv8fQKF0VDnaiMB+B/jt+Z8V+nKOzfR+oacbNby7WSiFv1oWIPjpH6YplNg0BwIXj+2D5nmos3hlKAos0meubmI/rH8rKtdtIi+oBgJQIZpjld5wBglJu4scnhH6Hx5gik+TfLjvNid7ZKbjvwmPDrmUmJ00RBGluO3qkZ4QljsWiIxutM9Yk6YKAiDIAvA/gt0KIOtPhVQAGCiEaiGgmgA8BhBUdEUI8D+B5ACguLu7QDLChBZmGxCHz6tkfoxuVmYM1iSURxcMfCGLIHZ8ndO6a/TWoqPdixqiCiNe54JlFhn1efwDmZGr9xCxX3Q0J1rcHgOaWkKP2m20VqKz3auYgbyAYVn1Uz6wP1msrdgD4cPVBQ42fzaUhIbVFFVhZKY6wv9O0EQVYtc9Y9C5Sjpq+5v0AUzbtgxePwXurSiAEDNqBRGopux+aGbN8slw8mMs8xyI33Qm7jbTeugzTmST1V0dETihC4A0hxAfm40KIOiFEg7r9GQAnEfU0n5dMequRHk474adTi+D1B/HA7E34Xl1ZyhIU5gYjeqQKX1Hf/hDSa19e1qps4oueXYQbXl0Rtr/e48PrS/ZG2O9Hjcm+Xa0rq1ChtjCMpxF4fAH8+q3VKKluQoVaTsHTEsA1Ly3D799dq5lKvL5A3FLfhpLLpgk+EBQgAk4cErKBZ6U6w5rbTByYgyb1M2UUjrkjGgCkOEITfKEps9Zpt2HNnWfizesnhxVq0xOvhr60GmWnJm6uyUlzIc1l5/r8TJeQzKghAvAfAJuFEE9EOae3eh6I6Hh1PFWRzk0WU9TmG69dN1lTsV9YuBtXvrAEQEjNj6URyEzQWA3BE2VLWT3WlZhXtvGVILOt/eLnFuOeT8LdMXXNPkMZBQCo0tXFkfcQL8Tzm20V+GTtQdzz8UaUqHb3Jt0kLp2jXn/QkFBmjuMHgIoGY+VOM/kZblx5vGKKcan+g7vPH42fnXgMLivuB0DpriVNVJMGKGGXkeSPPgxYH8svyU5zao3X24rUgCKZpqIxvl+OoYAcw3QmyTQNnQjgagDriWiNuu92AAMAQAjxLwCXAvgFEfkBNAO4QiRadKaDGFmYpan6W0rNlqtQ7flYgkBqDdE0AiEEFm6vxElDekZtri3x+oNhNXJkJE0sqptatEYiXn9As/+bqWn24XCj0XlbVmusnCnH0eIPRnVYShNGbbMPVY0tyEwxRgc1+ZTtFn/QoBFE0qyueWmZtr2nsinseE6aU6upM1yNZhpWkIm7zldCOh+5RIn1b9Q0AuXcSBqBfsUda9XfHmaOKcSGA3Wtst3fcMog3HDKoKSMh2Hikcyooe8QvTKxPOcZAM8kawyJIieHSHHacnUXWxAoE060zN3P1pfhV2+uwl8vHI2rpxTFHIvHF0C1aS70BYJxJ61DdV5NEOwsb4x6XnmdRyujIFm2J3IcfqPXD5cjsp1bCjTZP3d0nyws2RW6jowM8vqNPgJz83Yz+w5HEASpLowszMINJx+Dq08oCjsu/35S4Mim611VUPAXpw7GVZMHtso0xDBdCWej6IjUPFsKgEjlqmubfKhq9GoTzi5d6OKPXliCk4b2xC9PG6I5kfdWhU9yZrz+YFjkUos/iHh+x0P1HoyCkv1q9gHoOVjjMZRajkWD14+z/v4tBudn4I5zR+LYviHThbTly3LLowqzDYJAxvz/6xtjIbt47SIj+Say0xRH6h3njorwjnDy1WziSBqBmTm/PTmsNEZ7ISIWAky3gkMUdJgFgRACh1Tn6YJtFWGr2ZlPLcS0v30T8VqLd1Zp4Y7SGhFvgeoLBBEIirBol1iOahlGWa5znsbK3C2r86Cy3hu1ZDQQqpFT7/GjvN6L73dV4bynvzOcY3bqDo1TrVOSSHcys/UsUgRPJGRSVfHAHuibk4rfnzk87ntG9M7SzE0MY1VYEOgwTzivfr9XW9muK6nFre+sMRzXZ7RKZ2YkpOlCILYkiJarEMssJROJ9CYVaZa587xR+OnUIm1/76wU7KlsRJ3Hj5G9jbVz9OSrNezNq3O9VmRe2evr4seiuSWgmW4eunhMxHMuM5WFCCToNnr52uOw7PbpyExxYtGsaV3WepFhuhssCHSYNYK1pmbs30VpOwgAIwszMbwg8spSLnDjzWfRKn7GEgTS/L61LOQclsLrsuJ+GFkYGtPAvDQtk9ZcfO3FnxRr273UaBqzL+GnLy/T6gbpBUGK04aCrMRKIDe1+PHRr07Eqz87PupKf2y/HDx26VitoXkgwXaiKU47emXFd9C+dt3x+OCXUxO6JsNYARYEOswT0+ayekOGZ32M2Hqn3QanI7atOVpA1Mq9h1E0aza2RSl85/EFo75XahFrS2q0c2qbfbCRUmIh1RVyA43tl60VNNMLCACYPrKXti0ndXOs/qIdVVo0kl5o5aa5wtokRiMogF5ZKThlWH5EnwwA9M5244fF/TE4X/nuE9UIEuXkofmYOKBH/BMZxiKwINAxsjATfzhzGN7/xRTYSKlMaW5IEg2n3QZHlAql0sb/0dqDBnOS5J3lSnetBabWiJKZTy3ECwt3aa+FEHj+250or/egxR9qfv7n99cBUARBVqoTNhsZhNslk/pp26NMZZX1YZWyvWFZXXhM/9ayemw6WIfHvtiq7euR7gprkxgNfTmGNF2U1iRdcxZZznl0H8U5fcG4Pgldm2GYtsGCQIfDbsPN04Zi0sBcnDw0H4Cy2tUjC6yZcTlsUWP95eq5psmHq18ML7ckbe+xwh0/WBVqmr7xYB0e/GwL/vjuOnj9QVxe3B9DemXgozUHUe/xoc7j06JW5GSb7rJjRO8sjFEjf4ry0vGPK8ZH/CypEZTVhudFbC6twz0fG8tBuRy2hAqkvXvTFLzz81BrRxmuW5Dlxrs/n4IiNSNY1vwZkJeGnQ/OxPksCBgmqXD4aBSG9srAN9sq0MOkEdQ0+ZDmCv/aXHaKGuuvj5SJtMr2RenkpWdLWT32VjViYF665sRtavGjJRBEQZYb9190LK54fgmW7T6M2uaQIJCTraxbf+Mpg/D0/O3Iz3TjwvF9Mbx3JtwOo4lm0sBc2MhoGvrtGUPx/c4qvLF0H3qkG1f/Ow6F/BOpTnvUyCCz81ZqK067DTYbYe7vTsX+6ibk6ITvkVjWm2GONlgQREHWqzE7amuafHh3RQnOHWvsFqUvnWzGY3CshtvFfX7pgI1d1mHuxkN4e8V+9FFXzHYbQQhlRS7HW1bnUUxDqqlGlr+QTVPOH9fHsMIeESF6aGRhJjLcoU5ZgFJa+c7zRuG8p78LiyaS+QWLZk1DZooDY++ZG/M+JPK7kJO9y2HT/AIMw3QeLAiikJuumEfMTVS2l9fjya+2hcXhRxMEi3dU4pXvQ8XfXHYbapt9qKj3ao5oWa2yOkLXK8nQXhl4c9k+7K5s1KpzykQot8OOnhluECk9BoQIt6sPTCC8883rJyMgBIgImSlOLYdC+SxEbJTy5e9O0XwK0qRz/DG5hq5h8bBzoTWG6VJYEERh4kClkcjFE/riy02hto1r1JDSRtPqXREExgmNCPj56ysN+8rqPBh3r7JiljWOZIkKczE4ALj2xCIML8jEvC3lhnEAofaHLrUpiz64RrZIHFWYhbvPH4WLxveNe8/6YmsZbofBsU0EZKU64HLYDFrS0Aghs6/+7HhsKq3Dxc8tDo01wlwva/HcFKEDGMMwnQcLgigUZqdiz8Pnhu1fr9YTajSZR1wOG26dMQzzt5RrIZ1Ouy1mbZ26Zj+y05yas7g6QumHa6YUoahnuiaA9Miqnu4Ijto81bdBRLj2xGOijiEa5uimSQNzQUTolelGSXXsvgspTjsmDuiB6SN6Yd6Wcqz4yxkRW31muB0Rv2OGYToXFgQJsPIvZ6C6yYcznvhGKyzXZOpB67LbMLQgE1/deipOfvRrbX+sWvyby+pw+wfrtRpF5paOd8wciSI13NLstAZCiWORInbyMhJL8IqGPm9h2R3TNfNPTpoTJdXNOGNkr7i1f168phhCIG7FVYZhuhYWBAmQl+FGbroLDhtpOQFmh6lMJivMTsHFE/qi3usPM+WYmbf5kKFQnVkjuPbEIm3bHMYKhASHjPo5rqgHlu+pVs5PMP8hGu/eNBV7qxphs5EmBABgwwGlVPePTxgY1qLRDBHFrGnEMMyRAecRJAgRGRqEm2vtSEexw27DE5ePx/j+4c3Kiwcas1kX7TD24JHKQ4rTBiJj6GQkjUAKDmkaevVnk7WOa3ntFASj+mThnDGFOGt0b8P+P5w5DD3SnDhFzbNgGKb7w4KgFehbG5o1AnOvWbPdflz/HC2W/7mrJsLlsGFTaR2cdsKm+87CIHV1nZPmREFWClx2myHbt0daeOau9EVI01Cqy47RasZwtPIN7eXmaUOx+q4z2dzDMEcRLAhagd4UUmsq9VxgKnbm1k3Eq+6cgfdumqJlGGenOrWJv39uGtJcDu3804f3QlaKM8zuLyfesf2yccv0oThbt1LXn/vopWPxl3NHagKBYRgmHiwIWkFhTmiyN7eTNE/ceo0gN90Fp92mrdLT3Q4th0DW05EF56aP7IUMtyNMwxilVgu9+fQh+N2MYQYzlb4wXl6GG9efPIiboDMMkzDsLG4Fvzx1COqa/fD6Avhg9YGY50pBMCA3lIT1yCVj8c7y/RjXL1ubqCcOUHwJsrzEKcPy8eHqg2GCpSArxRBqKUtHjCrMQs92RggxDGNtWCNoBdlpTjx08RgUmJqSD4vRnWuobrVekJWCX08fCiLC5WrzlTNGFgAA7jl/FK6aPABZKU7065GK/MzYk7usA3T2sb1jnscwDBMP1gjaQIMn5Ci+ZspA3DojvCWi1ss3iq3+pKE9DSv8n+qSvv589gh4/bFbOl48oR/2VjXhupNanyzGMAyjhwVBG9D7Co7tm43sCBE9lxX3Q0l1U5vKJ6S67JrpJxpnjCrAGaMKWn1thmEYM2waagPXnzRIM+1UNITX7AeAzBQn7j5/NNLdLGsZhjmy4VmqDbgcNtx74Whkpzlx1eSBXT0chmGYdsGCoI2kOO24febIrh4GwzBMu2HTEMMwjMVhQcAwDGNxWBAwDMNYHBYEDMMwFocFAcMwjMVhQcAwDGNxWBAwDMNYHBYEDMMwFof0Tcq7A0RUAWBvG9/eE0BlBw6nO8D3bA34nq1Be+55oBAiYo/ZbicI2gMRrRBCFHf1ODoTvmdrwPdsDZJ1z2waYhiGsTgsCBiGYSyO1QTB8109gC6A79ka8D1bg6Tcs6V8BAzDMEw4VtMIGIZhGBMsCBiGYSyOJQQBEZ1NRFuJaAcRzerq8XQURPQSEZUT0Qbdvlwi+pKItqvPPdT9RERPqd/BOiKa2HUjbztE1J+IviaiTUS0kYhuUfcftfdNRClEtIyI1qr3fK+6/xgiWqre29tE5FL3u9XXO9TjRV05/vZARHYiWk1En6qvj+p7JqI9RLSeiNYQ0Qp1X9J/20e9ICAiO4BnAZwDYBSAK4loVNeOqsP4PwBnm/bNAjBPCDEUwDz1NaDc/1D1cSOAf3bSGDsaP4DfCyFGATgBwK/Uv+fRfN9eANOEEOMAjAdwNhGdAOARAE8KIYYAqAZwnXr+dQCq1f1Pqud1V24BsFn32gr3fLoQYrwuXyD5v20hxFH9ADAFwBe617cBuK2rx9WB91cEYIPu9VYAhep2IYCt6va/AVwZ6bzu/ADwEYAZVrlvAGkAVgGYDCXD1KHu137nAL4AMEXddqjnUVePvQ332k+d+KYB+BQAWeCe9wDoadqX9N/2Ua8RAOgLYL/udYm672ilQAhRqm6XAShQt4+670FV/ycAWIqj/L5VE8kaAOUAvgSwE0CNEMKvnqK/L+2e1eO1API6d8Qdwt8B/AlAUH2dh6P/ngWAuUS0kohuVPcl/bfNzeuPYoQQgoiOyvhgIsoA8D6A3woh6ohIO3Y03rcQIgBgPBHlAPgfgBFdPKSkQkTnASgXQqwkotO6ejydyElCiANE1AvAl0S0RX8wWb9tK2gEBwD0173up+47WjlERIUAoD6Xq/uPmu+BiJxQhMAbQogP1N1H/X0DgBCiBsDXUMwiOUQkF3P6+9LuWT2eDaCqk4faXk4EcAER7QHwXyjmoX/g6L5nCCEOqM/lUAT+8eiE37YVBMFyAEPVaAMXgCsAfNzFY0omHwO4Rt2+BooNXe7/iRppcAKAWp262W0gZen/HwCbhRBP6A4dtfdNRPmqJgAiSoXiE9kMRSBcqp5mvmf5XVwKYL5QjcjdBSHEbUKIfkKIIij/s/OFEFfhKL5nIkonoky5DeBMABvQGb/trnaOdJIDZiaAbVDsqnd09Xg68L7eAlAKwAfFPngdFLvoPADbAXwFIFc9l6BET+0EsB5AcVePv433fBIUO+o6AGvUx8yj+b4BjAWwWr3nDQDuUvcPArAMwA4A7wJwq/tT1Nc71OODuvoe2nn/pwH49Gi/Z/Xe1qqPjXKu6ozfNpeYYBiGsThWMA0xDMMwMWBBwDAMY3FYEDAMw1gcFgQMwzAWhwUBwzCMxWFBwDAqRBRQqz7KR4dVqiWiItJViWWYIwkuMcEwIZqFEOO7ehAM09mwRsAwcVBrxD+q1olfRkRD1P1FRDRfrQU/j4gGqPsLiOh/av+AtUQ0Vb2UnYheUHsKzFWzhEFEvyGlv8I6IvpvF90mY2FYEDBMiFSTaehy3bFaIcQYAM9AqYoJAE8DeEUIMRbAGwCeUvc/BeAbofQPmAglSxRQ6sY/K4QYDaAGwCXq/lkAJqjXuSlZN8cw0eDMYoZRIaIGIURGhP17oDSG2aUWvCsTQuQRUSWU+u8+dX+pEKInEVUA6CeE8OquUQTgS6E0FwER/RmAUwhxPxHNAdAA4EMAHwohGpJ8qwxjgDUChkkMEWW7NXh12wGEfHTnQqkZMxHAcl11TYbpFFgQMExiXK57/l7dXgylMiYAXAVgobo9D8AvAK2hTHa0ixKRDUB/IcTXAP4MpXxymFbCMMmEVx4MEyJV7QImmSOEkCGkPYhoHZRV/ZXqvl8DeJmI/gigAsC16v5bADxPRNdBWfn/AkqV2EjYAbyuCgsC8JRQeg4wTKfBPgKGiYPqIygWQlR29VgYJhmwaYhhGMbisEbAMAxjcVgjYBiGsTgsCBiGYSwOCwKGYRiLw4KAYRjG4rAgYBiGsTj/DxwEA10tmmBRAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(np.arange(1, len(average_mae_history) + 1), average_mae_history)\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Validation MAE')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> It may be a bit hard to see the plot due to scaling issues and relatively high variance.\n",
    "Let’s:\n",
    "\n",
    "> - Omit the first 10 data points, which are on a different scale from the rest of the curve.\n",
    "- Replace each point with an exponential moving average of the previous points, to obtain\n",
    "a smooth curve\n",
    "\n",
    "这个图表很难看清楚，因为它的点很多而且相对来说方差也比较高。我们可以：\n",
    "\n",
    "- 忽略前面10个数据点，因为它们和曲线的其他部位显然处在不同的取值区间当中。\n",
    "- 将每个点替换成前一个数据点的指数移动平均数，来获得一条平滑的曲线。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0, 0.5, 'Validation MAE')"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3dd3gc1fXw8e9R773LkmW5d2zLGGMM2HRMDwQIoYbwQuAXSAjpkE4oCQQCoQQIgTghAdNCwBRjig0Y3JvcLVnValbv0n3/mNGqr9e2Vitpz+d59Hh25u7ojhA6e9u5YoxBKaWU9/LxdAWUUkp5lgYCpZTychoIlFLKy2kgUEopL6eBQCmlvJyfpytwpOLi4kxGRoanq6GUUsPKunXryowx8X1dG3aBICMjg7Vr13q6GkopNayISG5/17RrSCmlvJwGAqWU8nIaCJRSystpIFBKKS+ngUAppbycBgKllPJyGgiUUsrLaSBQSikPWr61iLc2F2KMoaWt3SN1GHYLypRSaqQor23i5n+sB+DjOaW8vC6fO04fzx2nTxjUemiLQCmlPOTtrcWO45fX5QPw7Kf7aWxpG9R6aCBQSikPWbOvnNSoYM6fmeI4V9PUyqe7ywa1HhoIlFLKQ7YVVjM9NZITMmMAmJQUjr+vsP7AIQA251fS1Or+1oGOESil1ABrbWunqbWd0EA/CiobSAgPxN+3++fumsYW9pfV8bXZqWSNtgJBfHgggf6+rM89RF5FPRc8thqAy7PS+P0l0/HxEbfUV1sESil1jDbmVbJ8axHNre1U1bdw4n0fsvCBlTy+cg8L7vuQP7y3k8r6Zkf5vIp6rvvbVwDMGBXFhMQwfn3hVB64dAaz0qLYnF/FhztKHOX/vTaPNzYVuK3+GgiUUuoYFFc1cuXTX3DzP9Zz3d++ZMWOg5TUNFFR18yD7+4E4KmP93HZk5/T3GpND71v+Q7W5VrdP1kZ0YgI18zPIDkymFnpUTS0tPH0J/tIjQpm12/PYWx8KM+vznHbM2ggUEqpY/D8Zzk0tbaxZHoyn+0t5/v/2USwv2+vcrtLannio73sL6tj44FKAIL8fQgJ6N5DPzs9GoCCygbOmppEgJ8P156Ywab8KjbYYwcDzW2BQETSRGSliGwXkW0icnsfZSJF5L8isskuc7276qOUUu7wQfZB5o+N5Z7zpzjO/f6S6Xzv9Al8d/E4rpibBsCExDAe/mAXi/7wEQWVDWTGhfL+907pdb9R0cFkxoUCcMFx1myiS2aPIjzQj1fXu6d7yJ2Dxa3AncaY9SISDqwTkfeNMdu7lLkV2G6MOV9E4oGdIrLUGNPc5x2VUmoIyS2vY09JLd84Pp3EiCBe/c6JjIkNJTo0wFGmrqmVxZMSOGl8HMvW5XOgop6XvsrjzjMnkhYT0uueIsI7dyzkQHk94xPDAQgL9OM/N89nfEKYW57DbYHAGFMEFNnHNSKSDaQCXQOBAcJFRIAwoAIrgCil1IB4Y2MBvj7CeTNSDl/4COwvq2PRHz4C4PTJiUBnt05XoYF+nDk1CYCr52cA8LMlU3qV6yrQz9cRBDpMTo44xhr3b1Cmj4pIBjALWNPj0mPAm0AhEA5cbozplWxDRG4CbgJIT093Z1WVUiPM7S9tBBiQQHCorplN+ZWcOjGBpV9YWwBHhfiTHtv7k/1w4vbBYhEJA5YBdxhjqntcPgvYCKQAxwGPiUivsGeMedoYk2WMyYqPj3d3lZVSI0B7u6GqvsXx2hhzzPf87ksbuO5vX7GvtJbVe8sZHRvCW/930jHf19PcGghExB8rCCw1xrzaR5HrgVeNZQ+wH5jkzjoppbzDz17fysxfv+d4fahLUDgaLW3tjtQPd/x7I9lF1Vw7P4NR0cO7NQDunTUkwLNAtjHmoX6KHQBOs8snAhOBfe6qk1LKe/zrywPdXueW13V7vTGvkoUPfMiPl23m9Q0F1DU5H55cZieFm5YaQcGhBiYlhXPVCSOjq9qdYwQLgKuBLSKy0T73UyAdwBjzJPAb4HkR2QII8CNjzOBmW1JKjTj1za34CIQE+HHLqWN58N2dfJVTwSx7MNcYw99W7yevooGXKvJ46as87jprIrcuGocxhoLKBscnfWMMz3+Ww71vZzMrPYpXbzkREcEYg/V5d/hz56yhVVh/3J2VKQTOdFcdlFLe6cMdJbQb+NPlx3Ha5AS+2FfO4yv3smhiAk2t7dy/fEevDJ/r7ZW+L32Vx09e3cLSG+fx8to83tlaTFNrO6dPTuSPl810/PEfKUEANOmcUmoEevDdnUxOjmDhhDhEhEvnjOLT3WWc8fAn3cpdOmcU01Mj2XDgEJ/uLsMYw2v2oq3v/2cjB6ubuPL4dGanR/G12aPclvTN0zQQKKWGpYLKBtrbTa9FWQ3NbeSW1/P9MyYQ6Gelepgzuvf8foAbF45hUlIEfr7C6xsLySmvZ789lnCwuomJieH8/pLp7n2QIUADgVJqWFpw34cA5Ny3pNv5HPsP+Rg7TQNAalQwAOGBftTYg8Kbf3kmEUH+AMwbEwvAK+vyKK1pcrxvSor7FnENJRoIlFLDSm55HXkVDY7XJTWNJIQHAVDd2MI5j3wKdA8EIsLmX56Jn49w+VNfcPUJox1BAGBsfCjx4YE88dFeAH538TT+snKvI9fPSKeBQCk1rJz/51VUN3ZO9Tz+dyvY9IsziQz2dwz4QvdAADj+8P+3jwVgIsIJmbH8d1Mhwf6+fD0rjavmjXbTEww9moZaKTWsdA0CHd7bZm0CvyW/CoCHL59JaOCRfc6dkx4FwLiEsF67iY103vW0Sqlh7e0tRd1ev3vHyaRGBfPW5iI7D1AVmfGhXDxr1BHf+4ypSQT6+fDjc7wvuYF2DSmlho3vLF3vOP7BmROYmBTOSePi+PfaPGb95n3CA/04d3ryUd07NSqYnb89Z6CqOqxoi0ApNWTtLa11JItrbGlznN94zxnctng80H1qaE1TKwsnxA1uJUcADQRKqSHpy/0VnPbHj7n37WyMMeQfqges1cJRIZ0bv5w3M5lvdsn5s3C8Zig+Uto1pJRym+VbiwHD2dOOvLvmpa+spHF//XQ/8eGBjI23dufqmfs/JMCP3140nWvmZ5AQHkhksH+veynnNBAopQbUVzkVxIYGkBkfxs3/WAf0XvTlijX7Kjh7ahK7Smr435Zipqdai7tG97G9I8CEHjt6KddpIFBKDZj2dsNlT34OwHPXZTnOl1Q3khAR5PJ9GprbKKhs4PK5aYxLCOOxlXvYlFfJN+alExsWOOD19nYaCJRSAya7uHMTwhueX+s4XrrmAOdOT2Zi0uE/tb+zpYiPdpYC1qKw8YlhrNhRwrXzR/P1rLSBr7TSQKCUGjhf7q8A4NSJ8Y4/5gG+PjyyYjePrNjNc9dlsXhSotN73NJlimhmfCiTkiJ45/aF7qu00llDSqmBk1teT3iQH09fnUV0iD8Bvj588sNFPP6N2QA88dFe2tv73ju4sLKBj3aWOF6LQGZc2KDU29tpi0ApNWCKqxpJiggiwM+HL392Ok2t7YQF+rFkRjLVjdP5yatbePGLXK49MaPXe2/953o2HKgE4PyZKdxz3hSCA3wH+Qm8k7YIlFJHraCygRN/v4ILH19NVX0LRdWNJEVag8L+vj6Edcn3c8XcNBaOj+OP7+2kqqH3RvIl1Z3pn785L534cB0UHiwaCJRSR+29bcUUVjWyKa+SP63YRXFVA8mRfc8OEhF+eNYkqhtbeW19fq/rLW3tAPzs3MlkZcS4td6qOw0ESqmj9unuMjJiQzhnWhJ/W53Dweomp9M7p4+KZMaoSP6ztnsgqGlsoaSmiR+ePZFvn5yJ7wjdEnKo0kCglHJJcVUjN7+4zpHqAWBzfiVzM2K4fsEYx7me+wD0dP6MFLYXVZNr7yQG8PnecgAmJ3vHjmBDjQYCpZRL7nx5I8u3FfPKunze336QgsoGymqbGZsQxvFjYsi5bwkf3nkKl8xKdXqfs6clAfDOVmsPgfZ2w9Of7CM5MoiF4zRhnCforCGllEtyyuykbx/sBqCj9yazSwsgM/7w0z3TYkKYMSqSV9blc92JGTz47k7W5h7ivkum4+dlG8IMFfpTV0q5pLyuqdvrjuUAmfHOu4L68s0TRrOnpJYfL9vMs6v2c9mcUVw+V1cNe4oGAqXUYTU0t9HY0s63F45hVHQwl83p3AEsrZ8kcM58PSuNOaOjeX1jIQAXHJeCiA4Qe4p2DSmlDqujNTAuIYxPf7gIEeH8mSlMSgon0O/oFn1NT41knb3Z/LgEXUHsSRoIlFKHdajOWgAWHRLg+OR+8oRj2wBmSpcZQklHkJlUDTwNBEqpbvaX1fHcqv2kRAVz8ymZiIijRRAbFnCYd7vu1EnxnDoxngmJ4dot5GFuCwQikga8ACQCBnjaGPNIjzJ3AVd1qctkIN4YU+Gueiml+lZU1UC7gTMe+phWeyQ4LiyAy7LSyD/UAFgtgoGSEB7E89cfP2D3U0fPnS2CVuBOY8x6EQkH1onI+8aY7R0FjDEPAg8CiMj5wPc0CCg1+IwxzP/9h47X3zt9Ait3lvCH93Zy6sQEfv76VgDiNP/PiOS2WUPGmCJjzHr7uAbIBpytNLkS+Je76qOU6t+Bis7Vwj4C/++UTH62ZDIHq5v4yatbAPjOqWOJCNL9gEeiQZk+KiIZwCxgTT/XQ4CzgWWDUR+lVHeb8qscxxfNSiXI35e5GTHMTIvig+yDgDX3X41Mbh8sFpEwrD/wdxhjqvspdj6wur9uIRG5CbgJID093S31VMqbbS+sxt9X+OiuRaR0yR56yvg4NuVVEuDn029WUTX8ubVFICL+WEFgqTHmVSdFr8BJt5Ax5mljTJYxJis+/timrCmleiusbCA5MpjUqOBuM3guOC6FyckR3HjSGJ3ZM4K5c9aQAM8C2caYh5yUiwROAb7prroopZwrrurcUKarcQnhul+wF3Bn19AC4Gpgi4hstM/9FEgHMMY8aZ+7GHjPGFPX+xZKqcFQVN3A7PRoT1dDeYjbAoExZhVw2LakMeZ54Hl31UMp1VtNYwvPfLqfW04dS4CvDwermvpsESjvoCuLlfIyz3y6j9/+LxuA5MggTp+SSHNbO8ma5sFraSBQyst0BAGAnQdrCLE3mB+fGO6pKikP00CglBc5VNeMCBh7L4GlXxzgv5uKGB0bwvzMWM9WTnmM7keglBfJP9SAMfD01XP4xflTaG5rp6y2iUevmIWPbhjvtbRFoJQXWLoml+2F1Zw+ORGwcgadOC6O51bv5+ZTxjIzLcrDNVSepIFAqRGurd3ws9espHEZsda2knGhgYQF+vHpDxd7smpqiNCuIaVGuE35lY7jNzdZW0PGhQ9cOmk1/GkgUGqEyy3vXKu5paCKYH9fQgK0M0B10kCg1AhXWNkIWHsEg7YGVG8aCJQa4QoqG4gJDSArw0ohMZC7jKmRod9AICL/6XJ8f49r77mzUkqpgVNY2UBKVBCLJyUQEuDLdSdmeLpKaohx1lE4vsvxGcCPurzWXNBKDUGb8iqJCvFndGwohZUNbMqrZFthNbPSolg4Pp7tvz7b01VUQ5CzQGCO8ppSykMufHw1ANfMH80Ln+cCEB3iz9XzdXcx1T9ngSBERGZhdR8F28difwUPRuWUUq6rqm9xHHcEgW+dNIY7Th9PuO41rJxwFgiKgI4NZYq7HHe8VkoNETlldazeW+Z4veyW+RgDWRkxHqyVGi76DQTGmEX9XbO3oFRKDQEFlQ2c+adPaG5tB2DVjxYxKjrEw7VSw4nL00fFcpqIPAvku7FOSql+tLcbPttbRmtbu+PcC5/n0Nzazqz0KK6Ym0ZqlPbcqiNz2OWFInIC8A3gIiAGuBX4gZvrpZSyZRdV8+r6fO46axLvbS/mtn9uAOAf35pHZLA/L36ey+JJCTx33VwP11QNV/0GAhG5F7gMOAD8C/gVsNYY8/dBqptSXq+93XD5U59T3djK/rI6PsgucVxbvbeMAxX1NLe288OzJ3qwlmq4c9YiuBHYBTwB/NcY0yQiOm1UKTd78fMcQgL8+NqcUeQfaqC6sRXAEQSC/X1Jjgwit7yOXQdrOXViApOSIjxYYzXcORsjSAZ+C5wP7BWRF7GmkWq2KqXc6O43tnHny5sA2G8njDshs3P2T2igH6NjQ9hZXMP+sjomJekWk+rY9BsIjDFtxpjlxphrgbHA68BqoEBE/jlYFVRqONt1sIaD1Y0ul++Y+QNwoLye/aW1AFw2Jw2AzPhQnr9+LhlxoewtraOt3TBRA4E6Ri59ujfGNAHLgGUiEg5c7NZaKTVCnPnwJwDk3LfEabmX1+YxITGc8KDO/yXf3lpEcVUjYYF+XDI7ldGxIcxOj8bHR9heWO0oN1fXCqhj5Gyw+PuDWRGlRpqWLlM8m1vbCfDruwFujOGuVzYD8Dd75k+Anw8vfJZDgJ8Pk5PDEZFui8POmJJofTQDkiKD3PQEyls4GyP4A/BNIBYIA8K7fIW5v2pKDW/FVZ1dQo+u2E1NY0uf5cpqmx3HByrqAbjllLEUVjWSU17PDQvG9HpPdGgAP18ymce+MWuAa628kbOuoVnAlcASYB3WFNIVxhidOaSUC/IPNTiOH1u5h31ltfzlqjm9ynX88QcorGrAz0c4Y0oij6zYDcDpUxL7vP+NCzMHuMbKWzkbLN5kjPmxMeY44FngQmC7iFwwaLVTahgrqLQCwc+XTAZg+dZi1uwrB+CVdfk8+O4OAPK6BIL1uYdICA9kcrI1HXRcQhj+vrp/lHKvw/6GiUg8VutgOlZqiRLn71BKARTYLYJvnjCaNT89jYzYUL73742U1jTxg5c38fjKvZTVNpFb3hkI1uYeIiEiCF8f4YPvn8Kym0/0VPWVF3G2Q9kNIrIceBkr9fTXjTFnGGO+GLTaKTUEldQ08vLaPMDaCObsP33CgS5/zDsUVNYTHx5IkL8viRFB/OicSRRWNXLfOzscZVZkH2RPaS2jooMZlxCGMZAYEQhYrYHIEM3vqNzPWYvgGSAFqAHOAp4RkTc7vg53YxFJE5GVIrJdRLaJyO39lDtVRDbaZT4+qqdQahD9vxfXcdcrm8mrqOfPH+5mR3ENv/rvtl7lCiobGBXdmQBu8aQEgv19Wba+M2fjjuIadhZXMykpnAtnpgCQEK6zgNTgcjZY3G8aahe1AncaY9bbaw/Wicj7xpjtHQVEJAr4C3C2MeaAiCQc4/dUyq1+/042Gw5UArC1oIpPd1t7AKzYUcLWgiqmpUYC1rqA1XvKOW9GsuO9/r4+zEqP4rO95SRHBhEW6EdueT37Sus4fXIiFxyXwh/f30VqtGYPVYPL2X4Ex/Tp3BhThLW5DcaYGhHJBlKB7V2KfQN41RhzwC6n4w9qyDLG8NTH+xyv395aTFNrO3efN4XfvLWdT3eXOQJBx7qAiODuXTsnjY/js73ltBtDSlQwH+6wfuUnJ0cwOjaUZbfMZ0KirhRWg2tQpiOISAbWgPOaHpcmANEi8pGIrBORa/p5/00islZE1paWlrq3skr1I8ceB/jmCelkxIbw302FAMwbE0NEkB9FVQ293pMU0b2b54YFY1g4Po5bThnr+OTv6yOcPD4egDmjY3RbSTXo3B4IRCQMaw3kHcaY6h6X/YA5WGsVzgLuFpEJPe9hjHnaGJNljMmKj493d5WV6uVgdSN3v74VgBtPyuSPX5/puDYuIYyUqGAKKzsXkMWGBjA1JYKbTu4+1z/I35cXvzWP6xaMIS7MGhSekx6tg8LKo9yaSdTe0nIZsNQY82ofRfKBcmNMHVAnIp8AM7HSXys1ZDy6Yjer9pQxOTmCjLhQMuJCeeKq2WzKryLI39cOBFaLwBhDZUMLVxyfRpC/b7/3PH1yAutyK3jw0pn9llFqMLiyQ9kE4C5gdNfyxpjFh3mfYC1EyzbGPNRPsTeAx+zU1gHAPOBh16qu1OCpbbL2BPjDZTMc586Znsw5063B4OTIIDYcOERVfQuVDc20tRuiggOc3nPGqCiW3niC+yqtlItcaRG8DDwJ/BVoO4J7LwCuBraIyEb73E+BdABjzJPGmGx7rcJmoB14xhiz9Qi+h1KDoq6pjcnJEUxNiezzelpMCIfqW5j56/cc56K0u0cNE64EglZjzBNHemNjzCqshWiHK/cg8OCR3l+pwVTd2EJkcP//u5wzLanbQjGAqBDnLQKlhgpXBov/KyLfEZFkEYnp+HJ7zZTyIGMMO4qraW+3cixWN7QQ4WQ2z+jYUP5v8Tju/9p0xyCwtgjUcOFKi+Ba+9+7upwzgKY+VCPWm5sKuf2ljZw+OZFnrs2iprG115qAnu4809pAPruohuc/y3EaOJQaSg4bCIwxvZOhKzUC7D5Yg4gwLqFze43GljZ+979slm8rBuCD7IN8sP0gBZUNLv9h//mSyZwzLUm3kFTDhivZR/1F5Lsi8or9dZs9LVSpYe2uVzbz/f9s7Hbuk12lvPhFLqU1TSyeZGU8ufGFtQBEOBkj6MrP14d5mbEDW1ml3MiVMYInsBZ9/cX+mmOfU2rYMsaw+2AN2wqrqWtqpbGljb9+so+bXlznKHPvxdO77SEcFujWZTdKeYwrv9lzjTFdV7x8KCKb3FUhpQZDUVUjdc3WbOiNeZW8tqGAV9ZZWUEXT0rggUtnEBcWyPq7z+AvK/fy8Ae7KKlp8mSVlXIbV1oEbSIytuOFiGRyZOsJlBpy9pbWOo5X7Snj7S1FXDAzhR2/OZvHvzHbMfPH39eHi2elAtYUUaVGIldaBHcBK0VkH9a6gNHA9W6tlVJutinPSiWdEhnEi5/nUt/cxtfmjOozJUR6bAg59y0Z7CoqNWhcmTW0QkTGAxPtUzuNMdpGVsPaRztLmZYawcxRUSxdcwCAaSkRHq6VUp7hbKvKxfa/l2BlBx1nfy2xzyk1LLW0tbMhr5IF4+LIyogGIDrEn1i7O0gpb+NsjOAU+9/z+/g6z831UmpANLe2M/We5Sxdk+s4V1nfQlu7ITUqmKzR1iL5sfFh/d1CqRHP2Q5lv7APf22M2d/1mojoIjM1LKzeU0ZdcxsPv7+Lq+aNBqCirhmAmNAARkUHMyYulJlpUZ6splIe5cpg8TJgdo9zr2CtJ1BqyKprauX+5VYiuMwun/gdgSAkABHhzdsWEOA3KJv1KTUk9RsIRGQSMBWI7DEmEAEE9f0upYaOT3aVsqO4BrBmCZ38wEoWTYzn+DHWqt/oUCs7qG4NqbydsxbBRKyxgCiscYEONcC33VkppY7Vy2vzWLY+nwA/H66Ym8YLn+dyoKKev3+e61hIFhuqaaKVAudjBG8Ab4jIfGPM54NYJ6WOSV5FPXe9shmAzLhQgu21ARfPSmX51mLHCmLdL0ApiytjBBtE5FasbiJHl5Ax5ga31UqpY1Bg7x08LTWC2xaNJz0mhA0HKvnZkskUVjawZn8F4UF+Oi6glM2V/xNeBJKAs4CPgVFY3UNKDUlFVVYgeOSKWZw9LYkpKRH85+b5xIUFOlJDp0QGe7KKSg0prgSCccaYu4E6Y8zfsRaXzXNvtZQ6eoWVjUDff+xTo6xz4xN13YBSHVwJBC32v5UiMg2IBBLcVyWljs3e0lqiQvwJDuidN2juGGsB2dez0ga7WkoNWa6METwtItHA3cCbQBhwj1trpdRRem7Vfl5dX8C01L7zBs1Oj2brr87SvQWU6sKVpHPP2Icfo/sUqyGsta2d+5bvYM7oaH51wdR+y2kQUKo7ZwvKvu/sjcaYhwa+OkodvaKqRppb27lsziimpUZ6ujpKDRvOPhp17Lw9EZiL1S0E1uKyL91ZKaWOxoGKesDaP0Ap5TpnC8p+BSAinwCzjTE19utfAv8blNop5cQv39zGyRPiWDwpEYDccisQjI4N9WS1lBp2XJk1lAg0d3ndbJ9TymPKapt4/rMcbnh+reNcbkUdAb4+JEVoKiyljoQro2YvAF+KyGv264uA591WI6VcsD73UK9zm/IqmZAUhq+PeKBGSg1frswa+p2IvAMstE9db4zZ4N5qKdVdaU0T4UF+VDe2EBcayNougWD1njJeXpvHutxDXDM/w3OVVGqYcjZrKMIYUy0iMUCO/dVxLcYYU+HsxiKShtWaSAQM8LQx5pEeZU4F3gA6Nr551Rjz6yN/DDWSGWOY+7sPSAgPpKSmiZtPGctHO0sc1696Zo3jeJ69YEwp5TpnLYJ/YqWhXof1h7yD2K8Pt6agFbjTGLNeRMKBdSLyvjFme49ynxpjdOtL1a+8Cit3UElNEwBPfrwXgJ8vmUyQvy8/f32ro+zxGgiUOmLOZg2dZ/97VNtSGmOKgCL7uEZEsoFUoGcgUMqpLQVVAExKCictJoS8inqSI4O4LCuNyGB/TpkQz8IHVgKaWlqpo+Gsa6jn9pTdGGPWu/pNRCQDmAWs6ePyfBHZBBQCPzDGbOvj/TcBNwGkp6e7+m3VCLG1sAo/H+GN2xYQ6Nc7f1BaTAg/XzKZRJ0tpNRRcdY19Ecn1wyw2JVvICJhWPse32GMqe5xeT0w2hhTKyLnAq8D43t9M2OeBp4GyMrKMj2vq5FtT0ktGXGhfQaBDjcu1OwnSh0tZ11Di4715iLijxUElhpjXu3je1R3OX5bRP4iInHGmLJj/d5q5NhXWsvYeE0brZS7uJR9y04/PYXuO5S9cJj3CPAskN1fXiIRSQIOGmOMiByPtcCt3MW6Ky/Q2tbOgYp6zpiS5OmqKDViHTYQiMgvgFOxAsHbwDnAKqypoc4sAK4GtojIRvvcT4F0AGPMk8ClwC0i0go0AFcYY9ze9bO1oIrIYH/SYjQnzVCXd6iBljZDZrymjVDKXVxpEVwKzAQ2GGOuF5FE4B+He5MxZhXWVFNnZR4DHnOlogPpvD+vAiDnviWD/a3VEdpXWgvAWA0ESrmNK7mGGowx7UCriEQAJcCw3d6ppKbR01XwSo0tbUf1vn2ldQBkxukYgVLu4kogWCsiUcBfsRaXrQc+d2ut3GhbQefEpbqmVg/WxHtU1bcw6e7lPGUvBDsS+8pqiQ7xJzpU1wco5S79BgIReVxEFhhjvmOMqbT79G+L7GsAABg6SURBVM8ArjXGXD94VRxY67rkqNldUuvBmniP4mqrFfb7d3bw768OHNF795bWkakzhpRyK2ctgl3AH0QkR0QeEJFZxpgcY8zmwaqcO7y3vZhR0cFA54pV5V6H6juzmD/58b4jeu++0jodH1DKzfoNBMaYR4wx84FTsKZ0PiciO0TkFyIyYdBqOID+8tEedh2s5dsLM4kNDWDjgUpPV8krVHYJBP6+rqeIrm5soay2SVsESrnZYccIjDG5xpj7jTGzgCux9iPIdnvNBtgX+8p58N2dnD8zhavmpTMzLYpl6/P515dH1lWhjtyh+hYAzp6aRFGl64P1nQPF2iJQyp0OGwhExE9EzheRpcA7wE7gErfXbIBFhfhz+uRE7rtkOn6+Pty6aCwAK7JLDvNOdaw6uoYmJ0dQ09RKdWMLty5dz41/X9vve+59O5uLHl8NoC0CpdzMWdK5M7BaAOdibVb/EnCTMaZukOo2oCYlRfDXa7Icr+eMjuGi41L4fJ8uZHanPSW1PLB8J4BjUVh2YTX/21IEWHsNWIvQO23Mq+TpT6yxhCXTk7VFoJSbOVtQ9hOsPQnuNMb03hdwBJiWGsnrGwspqWkkIVwzVw60huY2bvnHOsfrKSkRAPxoWed8g9KaJhJ6ZA2993/ZxIcH8v73Tta00koNAmeDxYuNMc+M1CAAMGNUFGClnFAD76lP9rKntHOK7tj4MG5dNJac8nrHuW2F1rqOwsoGfvDyJl78Ipevciv4xvHpGgSUGiSuLCgbsaamRCACW/J7ZsdWA2FPSS2jY0J4+PKZPHDpDAAWTUzoVmZ7kfWz/9/mIl5Zl8/dr2/FGJiVHjXo9VXKW7mUfXSkCg30IzMulC0FOo3UHSrrW4gODeDiWaMc56amRDqO48ICWZd7iOdW7WdjXvf/BjNHaSBQarB4dSAAaxC541OpGlgVdc0kR3bv/w8O8OWTuxYRHerPD1/ZzDtbi/lwR/eZW2kxwZpSQqlB5NVdQ2Btc5h/qJ7iqkYuf+pzDlZrUrqBUlnf3Gc/f3psCOFB/r02mp+QGMYPz57IsltOHKwqKqXQQMDo2BBa2gwPvruTNfsreP6zHE9XaVBtya9i2br8Ab9vbVMrxdWNRIf491vmmvkZfPe08TxyxXEAfD0rje+cOk5ncCk1yLy+a2i0vTnN/jJrdovrCRBGhrvf2MrGvEr8fIULj0sdkHu2txum/eJdAKddPL4+wvfPsLKVnD0tiQBfr/9copRHeP3/eR2rVrfa6am7Jkgbqcprmzj3kU/5KqeC6kYr/cOX+ysG7P6f7C51HEcG998i6CrQz7fXwjKl1ODw+kCQFBnEJbNTaW5rByD/UIOHa+R+z3+Ww/aiap5fnUNehTWnv6By4J57S37nuoyG5qPbkEYpNXi8PhAA/HzJFGLsLozsoupu2TJHos/2Wmk1souqaWmztoguGMAAWNtsbfhz/YIMvp41bDezU8praCAAYkIDWH77Qv5+w/FU1rfwxEdHvpPWcNIxM2pfmZU2alZ6FAWVDRhjBuT+tY2txIQG8IvzpxLpZLBYKTU0aCCwJUQEccqEeOZlxrBy58jNSGqMoaS6CV+fzv74xRMTqG9uo9JOF32s6ppaCQv0+nkISg0bGgh6OHl8PLsO1o7YTe4P1bfQ3NbORV1mCE1MCgcgt6K+v7cdkdqmNkI1ECg1bGgg6GF8ojWLaKQNGpfVNnHVM1849mxeNCkegOTIIMfMqX2lR7+H88odJfzuf9uBjhaB7zHWWCk1WPRjWw9JEdZ+xsVVI6tF8O62YlbvKWf1HmugODkyiA++fzIRwf5EBQfgI7C/zPWtJvIq6okODXB0AV3//FcA3HnmRGqbWokN0xQRSg0XGgh66MiNM5ICwQfbD/Kz17Z2Ozc2Pqxb+oe0mBDH4PHhNDS3sfCBlRyfEUNUiD83nZzpuLa3tJa6plZGx4YMTOWVUm6ngaCHqBB/Avx8KB4hOYda29q58YXOLSF/dPYkbjl1bK9ymXGh7C91LRB0DKZ/mWMtQlub27llxZJHVwH0yiOklBq6dIygBxEhOTJoxLQIun7Kv2R2ap9BAGBMXBj7y+pcmkLacxVyRV3vdRchAfoZQ6nhQgNBHxIjRk4g6Oj3X3bLfP542cx+y2XGh9LQ0sa/vsw77D0r65uJDw/sdf7EsbFcMTfNUUYpNTy4LRCISJqIrBSR7SKyTURud1J2roi0isil7qrPkUiODBrWXUOFlQ2c+fDHPP3JXnLsQDAuIdxpLp8x9gbxP31tC7VNrU7vf6i+hZTIINb89DT+dv1cx/n7LpnBBTNTgIFNWaGUci93tt9bsTa+Xy8i4cA6EXnfGLO9ayER8QXuB95zY12OSJIdCIwxwzIR2lubC9l1sJZ7395BalQwMaEBh03+NiU5wnG8vbDaaR9/ZUMLkSEBJEYEkdClZZAeG0JSZBBLpif32wWllBp63NYiMMYUGWPW28c1QDbQV57j/wOWAUNmOW9SRBDNre0cGqCVtoPtg+wSxieEkRoVTEFlAxPstRHORIcGsPrHiwH4+lOf85GT1dWV9c2OfQZEhEevnMULNxwPQICfD49fNZtpqZH9vl8pNbQMyhiBiGQAs4A1Pc6nAhcDTwxGPVzVMYW0qGp4dm/kltcxOz2ar8224u418zNcel9Kl20lr/vbV/1uWFNZ30JUlxbGBTNTOHlC/NFXWCnlUW4PBCIShvWJ/w5jTM/Ngf8E/MgY036Ye9wkImtFZG1paamzogMiMWL4riVobzeU1TYTFx7AbYvH88INx3POtCSX3isivPadE/n3TSeQFBHEn1bsoq29+yyitnZDdWNLn1tQKqWGJ7cGAhHxxwoCS40xr/ZRJAt4SURygEuBv4jIRT0LGWOeNsZkGWOy4uPd/8kzNdpaXZw3QLl3BtOh+mba2g3xYYEE+Plw8oT4IxrnmJUezbzMWO45fwp5FQ2syD7Y7Xp1QwvGWOstlFIjgztnDQnwLJBtjHmorzLGmDHGmAxjTAbwCvAdY8zr7qqTq+LDAokM9mdXydHn3vGU0tomAOKPcd/fM6ckkhwZxH/Wdu8e6tjBTQOBUiOHO2cNLQCuBraIyEb73E+BdABjzJNu/N7HRESYkBjG7oM1nq7KEdlbWsvKHVbXWV/z/I+En68PczNiHEnqOhywW0kpkcHHdH+l1NDhtkBgjFnFEewFb4y5zl11ORrjE8N5a1Mh7e0GH5/hMYX0tD9+7Dg+1kAAMD4hjDc3FVLf3OpYKbzHbiVNSAw/5vsrpYYGXVncjxPHxlLd2MqqPWV9Xv9g+0Gm3LOcmsahOcU0MeLYA8G4BGva6d6SzjQVuw/WEhcWQHSoDhYrNVJoIOjHGVMSiQjy4+0tRX1ef2TFbuqb29iUV9Xn9cHW2NK5Sfy9F08fkFw/HXsz7DxYQ01jC3kV9ewornYECKXUyKCZwfoR6OdLRlwohf1MIe3Y7H57URUnjY8bzKr1Kbfc6rt/9MpZjjQPx2pMXBihAb5syqtk9Z4yXttQAMDtp40fkPsrpYYGDQROJIQH9rlT2XOr9vPxLmtQ9t63d/DutoMsu+XEwa5eN/vLrL77TDtn0EDw9RFmpkXx4he53c6fNjlhwL6HUsrztGvIifjwIEprmnqd//Vb3dIlsS73ULeuGU/oSDedMYCBAGDBuM7WziWzUvnl+VOYrukjlBpRNBA4kRAeSHldMy1tnQuf+1tkdtDD2Ur3l9aREB7o2DpyoNxyyliOS4sCICsjhusWjBmWifiUUv3TQOBEgj3zpqy2s1Ww2p5F9I156bx52wLHeU+no9hfVudIJT2QfHzEkUl0xihtCSg1EmkgcCLBXp1bUt0ZCFbtKSMhPJDfXTSNGaOieP97JwN4dP8CYwx7S2vdEggAzpqaxJZfnqkZRZUaoTQQONGZhbTzj/xXORXMHxvr6B5JtMt4smtod0kth+pbmJ0e7bbvER6kKSWUGqk0EDjREQgK7d22GlvaOFjdxNj4znn04YF+hAb4UljpuUCwarfVXbVgCExjVUoNPxoInIgJDSDQz8exL0FHyyA1qjPPjoiQHhtKTnldn/dwVcem8e3txqUN5LvaWlhFUkRQt3oppZSrdB2BEyJCSlSwY1FZgb2mIKXHH9yx8aFszj/6FcbVjS0svH8lp01K4PN95QT6+fDit+aRFhPi0vtzy+vJiHOtrFJK9aQtgsNIjgxydA11/DsqunsgyIwPI/9QPU2tR7eWoKiykaqGFl7dUEBRVSM55fW9FnE5k1vunhlDSinvoIHgMFKigimy+/8LKhsQsTa372psfCjtxkrOtqekhvxDR7ahTVWDlbhuZloUT109h6kpEewodi0Fdk1jC2W1zYyO1UCglDo6GggOIyUyiOLqRj7eVUp5XRPRIQH4+3b/sc3NiAHg412lnP7QJ5x0/0oeXbHb5e9RbQeC31w4lbOmJjExMZxdTgLBGxsLWPjAh+SU1TlyDGVoIFBKHSUNBIfRMR5w7XNfsi63kug+duZKiQpmempktz/+D72/y+W0E9V2KusIe4rmhKRwiqsbeWtzYa89g6sbW7j9pY3kVTSwbH2+Y5BaxwiUUkdLA8FhJHcZGM4uqnZkHe3ptxdNo6HHH/7dB13b6rKjaygi2AoE505Lxs9HuO2fGxwZPzu8Yb8OD/Ljzx/u4V9fHgBgdIy2CJRSR0cDwWGk9BgPiA7pOxDMtPPxADxw6QwAdhRXu/Q9qhtaAeuPO0B6bIgjm+lnezs3xjHGsHTNAaalRvDe904mwNeH1XvKSYoIIjjA18UnUkqp7jQQHEbPqaL9tQgArjw+DYDFkxIIDfDlxS9yaWg+fPdQdWMLoQG+3cYeZqZFcdbURP67qZDvLF1HZX0ze0tr2VFcw+VZaSRHBjv2QRgdq91CSqmjp4HgMEID/fj7DceTbs/pd7ZF468vnMaHd55CXFggv7t4Opvzq/rd4ayr6oYWR7dQVydPiKelzfD2lmJW7ixhS4G1VuH4MbEAnD01CYDrTsw40sdSSikHDQQuOGVCvCMvf6Bf/z8yf18fMu30Excel0J6TAh3vrzJsYlNX/aU1PDyunwO1Tf3unZ+l53GiqoayS6qIcDXh8x4azzgsqxRbP7lmZwzPfmonksppUADgctOyLSmiLa2uZb+QUS49+LpALy8Nq/fcp/aeYL6Sg8REeTvSHW9r7SO7KJqxieGObqQRMQx00gppY6Wpphw0fkzUqhuaOHCWakuv+ek8XFcnpXG21uKaG1rx8+3d9wtr7VaAv/89gl93mPGqChOyIxhb2ktlfUtTEmOOLoHUEqpfmiLwEU+PsLV8zOO+BP4/LGx1DS1OraS7GpjXiXPrNpHYkQgiRFBfbzbMj4hnN0HazlY3Uh8eOAR110ppZzRQOBmU1KsT/DbC7tPJW1pa+eix1fT2NJObKjzP+5TUyKobWqlvrnNsWuaUkoNFA0EbpYZF0qAnw/bi6xAUFLTSHFVI+tzDznK1DW3Or3H1JTOncE6dk1TSqmBomMEbubn68PUlAjW7K8AYN69KzAGHr1ylqNMaU1Tf28HYEJS50Y4Cdo1pJQaYNoiGASLJiawOb+S7KJqOvacOdBlI5sLj0vp552WQL/OVcM6RqCUGmgaCAbBkhnJBPj6cM4jnzrOrdxZSoCvDxvuPoNfXzjtsPe47xJrKqqrm9UopZSr3BYIRCRNRFaKyHYR2SYit/dR5kIR2SwiG0VkrYic5K76eNLY+DA+vmsRx4+JcZxbl3uI+PBAokN7p7XuyxXHp7P/9+cSFqi9eUqpgeXOFkErcKcxZgpwAnCriEzpUWYFMNMYcxxwA/CMG+vjUUmRQfzzxnm8edsCFoyzUkREhx7ZVFQRcUfVlFJezm2BwBhTZIxZbx/XANlAao8ytaZzp/ZQ4Mh2bR9m/Hx9mDEqimvmZwBwqK7FsxVSSikGadaQiGQAs4A1fVy7GPg9kAAs6ef9NwE3AaSnp7urmoPmrKlJ3HPeFMYlhB2+sFJKuZl0fiB30zcQCQM+Bn5njHnVSbmTgXuMMac7u19WVpZZu3btANdSKaVGNhFZZ4zJ6uuaW2cNiYg/sAxY6iwIABhjPgEyRSTOnXVSSinVnTtnDQnwLJBtjHmonzLj7HKIyGwgECh3V52UUkr15s4xggXA1cAWEdlon/spkA5gjHkS+BpwjYi0AA3A5cbdfVVKKaW6cVsgMMasApzOdzTG3A/c7646KKWUOjxdWayUUl5OA4FSSnk5DQRKKeXlNBAopZSXc/uCsoEmIqVA7lG+PQ4oG8DqDBf63N7HW59dn7t/o40x8X1dGHaB4FiIyNr+VtaNZPrc3sdbn12f++ho15BSSnk5DQRKKeXlvC0QPO3pCniIPrf38dZn1+c+Cl41RqCUUqo3b2sRKKWU6kEDgVJKeTmvCAQicraI7BSRPSLyY0/XZ6CJyHMiUiIiW7ucixGR90Vkt/1vtH1eRORR+2ex2U7/PSyJSJqIrBSR7SKyTURut8+P6GcXkSAR+VJENtnP/Sv7/BgRWWM/379FJMA+H2i/3mNfz/Bk/Y+ViPiKyAYRect+PeKfW0RyRGSLiGwUkbX2uQH7PR/xgUBEfIHHgXOAKcCVIjLFs7UacM8DZ/c492NghTFmPLDCfg3Wz2G8/XUT8MQg1dEdWoE7jTFTgBOAW+3/tiP92ZuAxcaYmcBxwNkicgJWJt+HjTHjgEPAt+zy3wIO2ecfZvhn/L0daw/0Dt7y3IuMMcd1WS8wcL/nxpgR/QXMB97t8vonwE88XS83PGcGsLXL651Asn2cDOy0j58Cruyr3HD/At4AzvCmZwdCgPXAPKyVpX72ecfvPfAuMN8+9rPLiafrfpTPO8r+o7cYeAsr1b03PHcOENfj3ID9no/4FgGQCuR1eZ1vnxvpEo0xRfZxMZBoH4/In4fd7J8FrMELnt3uHtkIlADvA3uBSmNMq12k67M5ntu+XgXEDm6NB8yfgB8C7fbrWLzjuQ3wnoisE5Gb7HMD9nvuzh3K1BBhjDEiMmLnCYtIGNbe2HcYY6rt3U+Bkfvsxpg24DgRiQJeAyZ5uEpuJyLnASXGmHUicqqn6zPITjLGFIhIAvC+iOzoevFYf8+9oUVQAKR1eT3KPjfSHRSRZAD73xL7/Ij6eYiIP1YQWGqMedU+7RXPDmCMqQRWYnWJRIlIx4e7rs/meG77eiTDc2/wBcAFIpIDvITVPfQII/+5McYU2P+WYAX+4xnA33NvCARfAePtmQUBwBXAmx6u02B4E7jWPr4Wq/+84/w19syCE4CqLs3LYUWsj/7PAtnGmIe6XBrRzy4i8XZLABEJxhoXycYKCJfaxXo+d8fP41LgQ2N3Hg8nxpifGGNGGWMysP4//tAYcxUj/LlFJFREwjuOgTOBrQzk77mnB0EGaaDlXGAXVj/qzzxdHzc837+AIqAFqz/wW1h9oSuA3cAHQIxdVrBmUe0FtgBZnq7/MTz3SVh9p5uBjfbXuSP92YEZwAb7ubcC99jnM4EvgT3Ay0CgfT7Ifr3Hvp7p6WcYgJ/BqcBb3vDc9vNtsr+2dfwNG8jfc00xoZRSXs4buoaUUko5oYFAKaW8nAYCpZTychoIlFLKy2kgUEopL6eBQCmbiLTZ2R07vgYsU62IZEiX7LBKDSWaYkKpTg3GmOM8XQmlBpu2CJQ6DDsX/AN2PvgvRWScfT5DRD60c76vEJF0+3yiiLxm7xewSUROtG/lKyJ/tfcQeM9eFYyIfFesPRU2i8hLHnpM5cU0ECjVKbhH19DlXa5VGWOmA49hZcAE+DPwd2PMDGAp8Kh9/lHgY2PtFzAbazUoWPnhHzfGTAUqga/Z538MzLLvc7O7Hk6p/ujKYqVsIlJrjAnr43wO1kYw++wkd8XGmFgRKcPK895iny8yxsSJSCkwyhjT1OUeGcD7xtpEBBH5EeBvjPmtiCwHaoHXgdeNMbVuflSlutEWgVKuMf0cH4mmLsdtdI7RLcHKDTMb+KpLJk2lBoUGAqVcc3mXfz+3jz/DyoIJcBXwqX28ArgFHBvIRPZ3UxHxAdKMMSuBH2GlSu7VKlHKnfSTh1Kdgu1dvzosN8Z0TCGNFpHNWJ/qr7TP/R/wNxG5CygFrrfP3w48LSLfwvrkfwtWdti++AL/sIOFAI8aa48BpQaNjhEodRj2GEGWMabM03VRyh20a0gppbyctgiUUsrLaYtAKaW8nAYCpZTychoIlFLKy2kgUEopL6eBQCmlvNz/B2Yz9Fshhu26AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def smooth_curve(points, factor=0.9):\n",
    "    '''\n",
    "    smooth_curve 求移动平均曲线函数\n",
    "    参数 points 原始曲线数据点\n",
    "    参数 factor 前一数据点值对于目前数据点值影响的权重值\n",
    "    返回 平滑曲线数据点\n",
    "    '''\n",
    "    smoothed_points = [] # 记录平滑曲线数据点的列表\n",
    "    for point in points:\n",
    "        # 如果存在前一个点\n",
    "        if smoothed_points:\n",
    "            previous = smoothed_points[-1]\n",
    "            # 当前数据点的值设置为 previous * factor + point * (1 - factor)\n",
    "            smoothed_points.append(previous * factor + point * (1 - factor))\n",
    "        else:\n",
    "            smoothed_points.append(point)\n",
    "    return smoothed_points\n",
    "\n",
    "smooth_mae_history = smooth_curve(average_mae_history[10:]) # 去掉前10个数据点，平滑曲线\n",
    "\n",
    "plt.plot(range(1, len(smooth_mae_history) + 1), smooth_mae_history)\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Validation MAE')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> According to this plot, it seems that validation MAE stops improving significantly\n",
    "after after 80 epochs. Past that point, we start overfitting.\n",
    "\n",
    "从上图可以看出，验证的平均绝对错误从80个epochs开始就停止改善了。过了这个点之后，模型开始过拟合。\n",
    "\n",
    "> Once we are done tuning other parameters of our model (besides the number of\n",
    "epochs, we could also adjust the size of the hidden layers), we can train a final\n",
    "\"production\" model on all of the training data, with the best parameters, then look at its\n",
    "performance on the test data:\n",
    "\n",
    "当我们完成对我们其他参数的调整之后（除了迭代的次数外，我们还可以调整隐藏层的尺寸），我们就能使用它们在所有的训练数据上训练一个“生产”模型了，然后再看它在测试数据上的表现："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "102/102 [==============================] - 0s 628us/sample - loss: 18.4982 - mae: 2.7464\n"
     ]
    }
   ],
   "source": [
    "# 使用一个全新的模型\n",
    "model = build_model()\n",
    "# 在整个训练集上进行学习\n",
    "model.fit(train_data, train_targets,\n",
    "    epochs=80, batch_size=16, verbose=0)\n",
    "test_mse_score, test_mae_score = model.evaluate(test_data, test_targets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.7464004"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_mae_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> We are still off by about `$2,550`.\n",
    "\n",
    "我们的差距还有大约2550美元（译者注：本人计算机训练的模型差距为2750美元）。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.6.5 小结\n",
    "\n",
    "> Here’s what you should take away from this example:\n",
    "\n",
    "> - Regression is done using different loss functions from classification; Mean Squared Error\n",
    "(MSE) is a commonly used loss function for regression.\n",
    "- Similarly, evaluation metrics to be used for regression differ from those used for\n",
    "classification; naturally the concept of \"accuracy\" does not apply for regression. A\n",
    "common regression metric is Mean Absolute Error (MAE).\n",
    "- When features in the input data have values in different ranges, each feature should be\n",
    "scaled independently as a preprocessing step.\n",
    "- When there is little data available, using K-Fold validation is a great way to reliably\n",
    "evaluate a model.\n",
    "- When little training data is available, it is preferable to use a small network with very few\n",
    "hidden layers (typically only one or two), in order to avoid severe overfitting.\n",
    "\n",
    "下面列出了本节你需要了解的内容：\n",
    "\n",
    "- 回归使用的是和分类不同的损失函数；平均平方错误（MSE）在回归问题中是经常使用的损失函数。\n",
    "- 同样的，在回归中使用的评估指标也和分类不一样；“准确率”概念很自然的无法应用到回归问题之上。一个常用的回归指标是平均绝对错误（MAE）。\n",
    "- 如果数据量很小，K-折验证是一个能可靠验证模型的方法。\n",
    "- 如果训练数据很少，最好选择使用一个小型的网络，里面仅有少量的隐藏层（典型的只有1到2层），能够有效避免严重的过拟合。\n",
    "\n",
    "> This example concludes our series of three introductory practical examples. You are\n",
    "now able to handle common types of problems with vector data input:\n",
    "\n",
    "> - Binary (2-class) classification.\n",
    "- Multi-class, single-label classification.\n",
    "- Scalar regression.\n",
    "\n",
    "这个例子结束了我们本章三个介绍性的实际问题的系列讲解。你现在已经能够处理那些使用矢量数据作为输入的常见问题了：\n",
    "\n",
    "- 二分分类。\n",
    "- 多类别单标签分类\n",
    "- 标量回归\n",
    "\n",
    "> In the next chapter, you will acquire a more formal understanding of some of the\n",
    "concepts you have encountered in these first examples, such as data preprocessing, model\n",
    "evaluation, and overfitting.\n",
    "\n",
    "在下一章中，你会对本章这些例子遇到的一些概念进行更加正式深入的理解，例如数据预处理、模型验证和过拟合。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<< [第二章：开始之前：神经网络的数学知识](Chapter2_Mathematical_blocks_of_neural_networks.ipynb) || [目录](index.md) || [第四章：机器学习基础](Chapter4_Fundamentals_of_machine_learning.ipynb) >>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}