过拟合欠拟合及其解决方案.ipynb

{"cells":[{"cell_type":"markdown","metadata":{"graffitiCellId":"id_3a0i46j","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"44C527C86C2941789DE841DE4A782754","mdEditEnable":false},"source":"# 过拟合、欠拟合及其解决方案\n1. 过拟合、欠拟合的概念\n2. 权重衰减\n3. 丢弃法"},{"attachments":{"image.png":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAW8AAAEACAYAAAB8nvebAAAgAElEQVR4Ae2dCfg913jHvyoItTQlDbVEUBVFBG1RrSqtnfCUFq3YqhsSWi3airWbWkJbWlVLheqC2NUS8VhjS6it1VpiCUUQO9U+n/zfNzmZ/9x7597fnZkzc7/neebOzJmzzefMfOe9Z86cI9nVQuAHJB1TS2Fcjp0mcKwkrkc7EzCBJQSOkvQiSf8Xy5KgPmQCgxDIa5Hr8vaD5OhM1iZwvrVjOMI2CFxb0tGS7i3pYpK+K+mASPhnt5HBltI4TdKXtpSWk1lMACv3iMWHBz/yhsgxr8uzJD1D0rMlnTp4aZyhCVRCAHH+WGFpp5VT45q/z3b9E4BzjfXfLBPXbU3GRf814xxMoIUA1vezJH01blysHG4WOxMYmwDXYV6PXJ9cp1yvdiZgAg0C95D0cot3g4p3xyKAeHM9cl3amYAJdCBwxQ5hHMQE+ibg67Bvwk7fBEzABEzABEzABEzABEzABEzABEzABEzABEzABEzABEzABEzABEzABEzABEzABEzABEzABEzABEzABEzABEzABEzABExgEAJ8JNH16zbC5UcVzU+Y8V82FgXh5zgMLSM1lkOalowGqcA9ZJJ12ZYEddl2PK8VjjWPrzp3xlVpXjdteduvMgLfV1l5XJx9BLhJu95Q5c35pAZAbuRF4k36hD9S0jMb8WrafUSMrcG6uTBQUtMh2pxXORpiyagZvqZ9ys44IinGzbKtEm/iNeMuO/ccswReXa+3Zpm8PxKBHIZ0pOyd7QICWI6HSjqp5TjDcnLT4biZLxHDieZQnfjlTdy0wiLaOcfJB5EjPQT8nhmgsnWObtcsVttAXliSpSgRJx9UnCucah0tkfJRJwzJykI9lud447guWD8yrGyuE64B/NIhxi+ONPArH2QZJq8h8uChQfiMl2G8rpiAxbu+ykFouJmwiFmzpIWJ6LCfDqE+KMKeGJ7c9FiobX+hiXucpJuE8OdNzQ3MjYuwsZ0Pgsxn7DXlaSsTQlc6zo9moMNCsOCDSyasa3fUSVrBzfJyXoyxnizynxXXAHWajrpEnPP8MzzHSZsH9cnFQ6x8aOSDIa+NTNPryghYvOuqEMSHG+uBUSxuNG5AxBiHsJaChT/HuVFT4AnHzUtYXAoAaWHJHx/CxiwpTUc6WGCkm5ZZM8zQ+5SfcyzFKcuQD6zcp+xMMILwcA5ptWKdMvlFWqcIV3LJuDWsEWPKiqOMnAd+nBcODghxlp118uF8WXBcCyzNBzgPfxauD2bIaf6zO1PSlyMPrrNS9CNpr2ohYPGupSb2lYObCnHNm5ObNR03U96U6ZfrtMByn5udGy/FGH/iIoD4l+lmHNYZp7Tuy+NDb8MhBXhR3ggQTUkIHedYupwpiuO43I/d6lZZjzx42eZ8uCZYrxLSZp0i5MQrmcAzH8qL0iMM108ZrzpQLtC5U2+ZRR0E0lrO0nADYy1yI3JTlTccAov1hKWGsKfVlXG5+YifcRA3tvFbJIhMxZUWX6Yz5jotzewRg5DxACqnDEtLER6IDpxwJQ/8kmEcrnJFOVlSiKmztJTTryw4dck1kP/QymPUP/6kkS6vBdJadA0QJ8NlPK8rJGDLu8JKiSIhRtxkCFLewOWNyHHaOhEvRK284TjGTdhmPZVWXfPs81jTf8x9ziWbTDhf9vOlLdsIGA+vkg3lhV02kzSbTXjZNxVHvXJ+bY46zocXD6fygQUb6jMfZmV8/q20MYAlS9uDoozv7QoIWLwrqIQFRUCQ+YvLOi3KUqARZo7ljc3NSq8D2rKxnvHHjxuxeQOTbtOPYtR403IOKcKcX553YoNNmysfXHdovOhrC1+zH4JK3bMuXZ57+Y8NVoTlgZ/1SX03H26l0GeahG/zz+NeV0TA/bwrqowoCmKVLxO5kbjpuDm5UcubtFlybjx6IvCykzgs+CHwU3YIEZY33RgR5DwnHk4s2YbbPEfC8oBigQXp5H4zbE371DEiXdY1FnZ5HovKy7XDgztFmOuH+rcgLyI2Yf/aX+BMGO1GRecm46blhmuKLjcmftzYWFWIEeHpRUG3QgQKR/y0uvDLG7dcI3oZPqKdvUIkyKcWh2ClY5tz4zywLmlC4RywxnlgpaVNOM7/iRmxaELKc+YhR5jaHPx5GKX4sqaJI8tdlpemI+qK+qcLIXWaLh9o+QBIdsmI9m5EvulIj7zI184ETGANAtyAq8STGyv/PrPNDV+6pgCTXoYnXBm/jMd23uxN/1r2KR/nU4pL83zKMBxrW5rMajk/zq1Ztrby40fYZMG6dBwvXVn/+OeDvAzDNuFqvwaaZfa+CZiACZiACZiACZiACZiACZiACZiACZiACZiACZiACZiACZiACZiACZiACZiACZiACZiACZiACZiACZjA/AkcPv9T9BmagAmYwLwIXDZGWuNrRTsTMAETqJrAVMc26eMrsE9JOkPSraquMRfOBEzABCZKgE94GZuh+UnwNk6H8SJev42EnIYJmIAJ9ElgipZ3DsDTh/X9Wkk3kHRgn9CdtgmYgAnsIgFGRsPyfk8PJ3/pSPtmPaTtJE3ABExgZwkw4hrCnUtzBLZtgDlF0lO2kZDTGIQAzWcMm9pHM9ogJ+BMTGAXCDBOcQo36+aY19tg8ABJX5R0/m0k5jR6J8Dwp1wLzWFQe8/YGZiACXQnwEDxzCjDzcr6zO5RO4c8RNJ3Jd2icwwHHJMAFjeTDdjyHrMWnLcJrCBAM8l9Qrx/tWXg+hXROx9+haTndA7tgCZgAiZgAisJ/FiI99VXhtw8wF0lfV3ShTdPwjFNwARMoD8CU+wq2B+Nc1NmAuDvSbrduV7eMgETMIF6CHQRb9oSmeQ02xT5SIYJYJtLHs+zYz8nQk0/5s6bwoulb0Szyf2y4F6bgAmYQE0EDuhQGF4SMss8L4UQXj6OYc1+OvxY8EPcEXscAs8s3jn79THR5IE/XzPmbNYRfNQVs5D/bbSpnyXpLyW9XxLNMx8YtWTO3ARMwAQaBLqIN1HSgkaYcYh0OQN1aU0j1CdHuFzTD/cmhT+HU9Aj6Ogr/oXcWdJLJJ0Qgs2n8g8KQR+9gC5AKwH+4dHMdYcKr6nWAtvTBLZBoKt4k1cKOD0+0souy5BinGus63RY2OX+xxuWe4Ybc023w3dLumWIN2XB+n5+CPhXxiyc815IgOsR44F1+W9wYQQfMIE5EOgi3twQWDf0q8baPlXScS0nn6LNobI5BKE/sXKrO0/nVZJ+O5qJ6EtOuT8n6dclPS4DeV0VAa5H/uGxtjOBnSHQRbyxanIBDBZONp8gzGmRI/KINn7Z5p3h8T+iQfU0Scc2/MbeRbwfJul6kt4RPU7+StL9JT0+9scuo/M/LwGMhrLZ7rxHvWcCO06Am6Ns4wYH1viqv6nEacbbK8o++3nzMPuqpD8qCsl58gLzLoWfN03ABExgVAJdLO9mAVOwsb4Pjb+r2WRSWkBY4DSv8He2bO/O9HiBWZvjs/gXSrqjpEdH4Ti3J0h6lKQX2PqurcpcHhMwgWUEEGUGgXpmEQgRx6qmLbzp8Ee0sVpxxE/RD6+NV31a3hTq56M7I/mkS+v7V9LDaxMwARMYk0CXj3QoH+JFH21eDLGNiNOOnSLNfgo1Io1VjmCnRT7mOa6bNxMy8JKyFGrOgzZv/kl0ZbZuvg5vAiZgAlsngBDngkXdfNFIE0la1gh3dtvCb9Gy6XCufVvewKNnySei10nCTOubc7WrhwD14vG866kPl8QEFhIYQryvVXwJWhaEfxr0Ud/kXUGZjre3RwCjwuN5b4+nUzKB3ggMId4Unk/jn944i4vGGOJ0HbSrgwCWN//ustmujlK5FCZgAvsRGEq8aRpiWNimKPyOJL62PGi/ktnDBEzABExgIYGhxPtikr4m6cGNkjA92gdjEKvGIe+agAmYgAksIjCUeJP/8ZJOb+lh8lPR35sXs3YmYAImYAIdCAwp3lcKkb5TS7n4YOdtLf72MgETMAETaCEwpHiTPYNTvaWlHJeJZpWyP3hLMHuZgAmYgAlAYGjxzq5obU0kD4kPeg521YxGwP28R0PvjE1gPQJDizelYyad57UUk/7efLTEaIR24xDIhytrOxMwgYoJjCHefFX5HUmHtHA5XNK3PNtOC5lhvNzPexjOzsUE9kxgDPG+kKQzJP3JgtL/fgwle4UFx+1tAiZgAjtPYAzxBjqDUn1BEkLedAxWdYqkN7d0K2yG9b4JmIAJ7CSBscT7ByV9M6ZEawNPt0K+yPyDtoP2MwETMIFdJzCWeMOdoW95ebnIMdclgyTdbFEA+5uACZjArhIYU7x5OYk432IJ/BNiHHO3fy+B5EMmYAK7R2BM8Yb261d0DaRN/F2S3ivpIrtXPYOfsft5D47cGZrAZgTGFu/bhfV9nSXF/2FJn5X0r0vC+NB2CLif93Y4OhUT6J3A2OLNCb4vepYsO9nrS2JC4+aohMvi+Nj6BNzPe31mjmECoxCoQbyZ+Z62b2aZX+Z+S9L/+gXmMkQ+ZgImsCsEahBvWL9U0kclMb73Mvf3foG5DI+PmYAJ7AqBWsT7qtEssmpKtAv4BeauXJo+TxMwgWUEahFvyvjU+Ory4ssKLClfYL5G0gVXhPVhEzABE5glgZrE+4diTJNFY56UFcCM9F+S9HLPPl9i8bYJmMCuEKhJvGHO5/DfkHTZDhVwXUlnSXpRh7byDsk5SEwQfVLLRNGGYwImUBmB2sT7QEmflPScjpzoQsjExv/oQaw6ElsezP28l/PxUROohkBt4g2Yo6Pr4DU6UvqZsNYR/PN1jONg7QTcz7udi31NoDoCNYo3Q8Ly4Q6fznd1DF7FJA5/2zWCw5mACZjAlAnUKN7wzA93br0G3FvFDD1PWSOOg5qACZjAJAnUKt7ApCfJh9dsyz4q+os/bpK14UKbgAmYQEcCNYs3H+7wOfx9O55LBvulEHC+xlz1xWbG8doETMAEJkWgZvEG5NNivktm3lnH3TL6jL9W0qqPftZJ12FNwATqIHDRMOyOrKM4w5eidvFGtJnr8rkboLmmpE9I+pCkK28QfxejeDzvXaz1aZ3zL4QeMJgdy+9Nq/jbK23t4s2ZMtoglbTOy8skdMmYzPiLkm6Ynl4vJOB+3gvR+MCIBJh1i/dYnw4teKuk35R0iRHLNHrWUxBvIP1TTMiwSWUx/gnxvyPpbqMTr7sA7uddd/3sUun4132MpHeHYH9c0mMlHbZLEJad61TEO5tPeAm5qePT++9JevyaPVg2zc/xTMAE1iOAofWLkl4SxhbDXzxL0s/5A7z9QU5FvCn5XeIJfNP9T6Ozz+3ja0xGJOSFh50JmMD4BBjm4q/j/RY9zOhocHdJFx6/aPWWYEriDcWXxdgnexFe3k4zJyaTP/xkvVXjkpnArAkg2LRj/1cYZR+U9LAY8nnWJ76tk5uaeDNsLEPB/s0eARws6cToR/7nHhd8jzQd3QRWE2DYixtLerKk0wvBfoykne3utxrb4hBTE2/OhL9T9D650eLT6nyEF5hnRnfCIzrHckATMIEuBPhIjq59GFv82+W+fa+kh0tCe+z2QGCK4s3p0ib2sS21iR0iiTZweqMct+OTO7if9x5uJkc9mwAvHW8bLxrpootgv1PSQ/29xXavkKmKN5M18CZ6m4NQ/VqkeaqkH90u5smk5n7ek6mqqgp6kegl8nxJX4leXfTF/l1JV6iqpDMqzFTFmyq4T1wkt9hifVxe0pskfVPSg3ewS6H7eW/xYpp5Uj8i6QGSXhk9uOglcnL4Mc+sXc8EpizeoHlxtFl3mTatK0omdODjgK9Lwgqnzc7OBHadAN32GDOIF44fieYQhq54nqRflXSpXQc09PlPXbwvFl3+Tumhrfoq0VZHm90bJHWd2WfoOnR+JtAXgaZ1zUdu75L06Bhugh4kdiMRmLp4g41eIt+W9IQeGHJx0ruFPuFcuP8giaYVOxOYI4HSuv7PsK7pmsvwEveURBdbu0oIzEG8QfkbcaHxBWUf7gKS7hfD09Ie/heSDuojI6dpAgMT4B/m/aPtmqZC/mmeJulPJTE/rMfEH7hCumY3F/HmfF8Qb7qv2PXkNwjHW3W+AsMaoRsUb9PpGmVnAlMhwIdud5X0jPhHiVjTQ+RFkuhxdZmpnMiul3NO4o2w8lePl4xYyn06emVgmWCpMNoZL2zmMHO9+3n3edWMkzZ1yrDKfynpA2FZfyMm+P5DST/Vw/uicc50x3Kdk3hTdVePbksMcjOEw4p5arS58+B4kCRulqk69/Oeas2dW+7vj14hNO0xnCpd+Hgn9OZ40cjk3hc6N7i3pkpgbuJNPeTn83cesFLo18ob+DPCGucv6bUHzH9bWfHgoWfNlB9A22IxlXQQYgSZ6w+B5kthBPsdkhi3h+8gEHS7mRGYo3hTRYz7/TVJvIwZ2jFeCl+X0Zb4NklHD10A5zdrAkxIcitJj5J0UlxnXGvvk3S8pNvt+gwzs6794uTmKt4HSnp/LGyP4ZhD8+lxc/Fyk+EvDx2jIM5z0gSuJuleMRl3tlkj1rzbYXgI/mEy3Z/djhGYq3hTjVjdWN//MvLLRJog6JXCuMX0Feev7e94zIcdu9O6nS6GBsOmMojTSyV9Ph7+9G56VYzGx2QkexnPvltJHKp6AnMWb+DfJtr/eHkztqM3CpMg8zERPVSwnvhaja6HfMlmt1sE6GJ6vfhG4e/Ckqa9muuCl99M/3Xf+LJ3Dj2Zdqt2BzjbuYs3CJllmhuCdU3uxyX9WTGTCM08tGN6XPGaamk7ZeFDF15gM5ja02LYhW/FdfnJGKOHbns398df2wG+C6nsgnhTj4gkb+CxxGt03NjMKPKhuKEZ+IeeAkNP00YTDy/B3Ntk86uEybJvFt1GsZ7prsdXuRgQn5P0inhI82KRrqZ2JrARgV0Rb/520vbNRzX8Va3ZUSdMCkHvAW74/5H0QkkPlHTdnj9Xdj/v7lcGH4Lx0KVrKs1y/xZdRakzFtqpXxcfczEjep9f/nYvtUPOhsCuiDcVRn/Yt4QYTuVGov84nzLzV5sJWhEFJqF4tST+ZvNya5sfXGBxu5/3eW9vuubxFeK949/QSyR9OPpTUx8YBLy7eI6k35N065F7FfFAYWlz+DP+zxMXPEweEf73WJIG1wjGRTruJa7D5pLHc0040i/dkxaUowwzyPYBg+TiTDYlQBsjN9bbw0qiSYL5K2t2n47xkhkzGcdfcm4SLOSj4sMM/JlAgl4sb4w5AmlH3cRhLZL2LrqrRg8lhv5lJiVeImPcwDwdAzbRtMXokszFyAOVl4u1OZprEOBjQ4SpVxxruhgyaUL6xaGzBZ84KeBMM5gOwc7wrBmULfMgDsJchk8/4nM9cc3iGCefB146voHIe/DZjTQyzCBri/cgmPeUCRcKkyvw9dnL48s0RH0qjv7iDCDEgmM885+O0d8YAY7P8/k7zw32740FsckbJaLvzArrmem4yoWhfXOfyTy4f+ntwfC/vHOgrZqhUBFnFnoI8d6kdoc482BHQFnSMXEJ/oscVnAZPsNhrROXdQo4DwXC5rsRhJx/bOlKA6AUdR4auBT+3Mcv094XYuBfi/fAwDfMjouJGUGwUk+QdKeGNbBhsqNEowmFF2As6Y6MMV4Y54UPhejBcFgc5IUZH3qcLukTkrDs2cZSZ6F9fUqO8adpWmIkPJZLN7ZToHnIpaO/PefOwstZzhvBZuHY1B2WM0LcbKJAHBFvhLjpCI8rBZh9wnOMdVNcEWAc91Mzrzh0znHEPD9Qw0rH+k5rnEBt/wTKNHrfnmJfTP4WYqGx5qbeJUfPkxPjy8eHzPzEETmaA1j4eIkbCWsTcWNdfoX635I+JenL0b7OcKE8JFiX21jx27REuX/4+AShzYUxOS5e7Kd/inQpyojLZ2JhjBkeTCwp1Dyk8J+7QygRUwQXoUxhLs8b4Uy9yrDwS4sZvwyDhZ3WM9Y36WFx8xAgHH78s2k6/qmU8dJKx2rnIcE/hHSEy7DpN+jalveguPec2ctiggVGIOSmbrvI95xJJQkwBChNRSxtjvkHL1csh8QNinDS5osIIJQppKz7mIyCL2K/Gg8LHhgs5T7izAOEfxDUWYr1qDd+G9AR/RDGvJYRyBTkLBLimcKJoFK3iDD+OMSatmgMm/SLQ2fHI71c8Ee8SYMF/7TIKQf1QlnKbxdS7JsTp9DmnXEzP6+XENil3iaLMGB18zeueaEuCj9nf27mdft5Yx0j5FjDtCHni77rSLp+tMffSBIfJXETM1bHlcLi56HBg8Bu+wSaok0OWOMId1rBmSthEdsUX0S0LX4ZHsEvXcYt/ZrbpNvWrt4MN/i+Le/BkW8lQyZVwNF9CpdWS+zu1AqriJuQNTdzF4e1zGJXDwEEslmH+LFQv8326yw5dc7SFOY83lyTR94vWN40x7HG8ZAoDSLSxKInTHOkTXrxlGEjieFWFu/hWG87Jwv4PqLccLw8yr/V2+bs9IYhgBBiZWNh0z0PQeXfUduLxy4lQvAzzQxP2vizsP0eSY+M7VKIOUa/cF6aUwYeHPlwyHWmOdqamcbtpksAAWc0Nyzw8uKb7hmtX/J8abXIMls/RccYmkBa1rQ300yBaLKsK9yIbjq2ywc6DwOseKx0tmlqQ7gR4xRr4mY8Xn6vm3/m7fUCAm7z3h+M28D3Z2Kf6RDI9mqEM7sFIpwILe92cinFGMHneDr2EX9EmYVt/NKRHk0mCDjH2C8dIp55I+7ZhEMeixbSsluDgMW7HZYFvJ2LfedJAKFHYNd1xFvklh1bFMf+axCweC+GZQFfzMZHTMAERiZg8V5eARbw5Xx81ARMYCQCFu/V4FPAq3kzvrrIG4fgr+66/bw3zswRTcAENidg8e7Gjll4mH/yuT2Pp92tNP2F4qUUL7TKl1P95eaUTcAENiZg8e6Ojs95+cz8NZIu0j3apEJiedMbwC+bJlVtLuwuErB4r1frzMLzhfgggc/B7UzABGZAwB/pzKASV5zCOyX9RHwpxiBPDNpvZwImMHECFu+JV2DH4jPmM4MsfT5m5blBx3gOZgImYAJbI+Bmk81R0u5N+zczgzOhg50JmIAJDEbA4r031OeX9IzoicIUZHYmYAImMAgBi/d2MD88utgxsUPOULKdlIdNxf28h+Xt3ExgYwIW743R7RfxbpK+G00p5Yzj+wWs2MP9vCuuHBfNBEoCFu+Sxt63bxhTdDGpLRMBT825n/fUaszl3VkCFu/tVz39v98cLzLvu/3knaIJmIAJ7Js1ns+hr24YWyXAi8zHRzv4CZIutNXUnZgJmMDOE7Dl3e8lcLuY/fy9MZtJv7k5dRMwgY0I+COdjbDNOtJLou2b+U2ZZPWWsz5bn5wJmMBgBGx5D4P6wpL+OfqD/4kkP+iH4e5cTGC2BCzew1bt/SR9O0buO3jYrDvl5n7enTA5kAmMT8DiPXwdMLDVZyR9VtJth89+aY7u570Ujw+aQD0ELN7j1MUlJf1r9Ebh8/qLj1OM/XJ1P+/9kNjDBOokYPEet15+KcYH56Oem45bFOduAiYwJQIW7/Fri496XhkvM58s6cDxi+QSmIAJ1E7A4l1PDf2apLMk/UeMF15PyVwSEzCB6ghYvOuqkkPj03oGuPpjSReoq3gujQmYQC0ELN611MS55aAPOGODM9kxX2Ze89xD3jIBEzCBfQQs3vVeCYdLenf0C3+YJMZL6du5n3ffhJ2+CWyJgMV7SyB7SobP6o+T9J34vJ4+4n069/Puk67TNoEtErB4bxFmj0ldTdKbokfK0yXRT7wP537efVB1mibQAwGLdw9Qe0ySfuGnx8z195n4lGs9YnLSJjB/Ahbv6dUxs9bTE+Vbkt4u6VrTOwWX2ARMYK8ELN57JThe/CtJem3Mm3l8RZ/Yj0fEOZvADhGweE+/sm8j6SMx2NVdp386PgMTMIEuBCzeXSjVH4Zp1uhO+DVJJ0v60fqL7BKagAnshYDFey/06ot7OUnMmcm8pE+RdKk1i+h+3msCc3ATGIuAxXss8v3m+zOS3inp65KYuQdR7uLcz7sLJYcxgQoIWLwrqIQei3BrSadKOlPSQyXRU2WZcz/vZXR8zAQqImDxrqgyeirK+STRP/zDks6Q9ABJF+wpLydrAiYwEAGL90CgK8iGsVHuLekTkj4u6V4DjZdSwam7CCYwPwIW7/nV6aozwurG+mYOTazxO/tLzVXIfNwE6iNg8a6vToYqEe3fdC+kPfw9kmgftzMBE5gIAYv3RCqqx2LykpLP7ekj/i5Jv+zmlB5pO2kT2BIBi/eWQM4gGUYqpFvht6Nd/JgOvVNmcNo+BROQmAHFzgSmSuALkl4dU6/9s6TflsSs9o+VdPBUT8rlNoG5ErDlPdea3ey8yn7edDE8KubU/Kakv/Nn95tBdSwT6IOAxbsPqvNL8waSXhgjGL5UEl9w2pmACYxIwOI9IvwJZs0wtE+Nz+5PkXQnublwgtXoIs+BgMV7DrU4/DnwcvMRkv4nPvh5uKRLD18M52gCu0vA4r27db+NM2co2l+R9MaiSYXxxYeY6X4b5XcaJjBZAhbvyVZddQW/iqQnSKLXCr1UHinpstWV0gUygZkQsHjPpCK3dBr0NjlpjSFk27Ll83tm9GFSiP+V9ApJt7c13obKfiawOQGL9+bs5hhz2+N5Y40/Pma7/7Skx0i6/BzB+ZxMYGgCFu+hidedX9nPe5slxRq/S1j134uPge4o6YBtZuK0ziZAHR5XsLiiJB7KpWO/9Lu2pHuUAWK/DJOH29K/saTmkuFzTTl4yV26J0nCf3TnC3H0KnAB9kjgS42beo/JnROdT+6fHwvdDX9T0t/EQv/xF0h6gySE3W5vBKjDgyQ9KwQ4xRu+LKVws49DuPMYIpx+Lw5RZp9mMMI00ycueXxsX7Rz4qYok1+myQZRqwAAAAy8SURBVJALTNGX7ugYGI39ZzfSyDCDrC3eg2B2JhMn8N+SHhwz+9w8Jop4kaRvSPqXEPI3NW7yiZ/y4MU/thDuVZljSSOwxOGLWkQavy9LOrGIXIpzpk84HA+KfBCwX1rsZTzSxuWDJffx46EwmrN4j4beGU+QwHclvTwWmlUYkpYZf14V1hjjq2CRv22C51ZDkRFIXFrAsbvfCsGm+QILGtEl3hExTDBWcimwZeRMH3FuNoeU4ThOuoeGJ+Uh3bTG8SYPi3dJzdsmMBECNKtgfbMwzvhtY2harDmmbvunEHKGrLVbTgBmWMQ0eSwT1UyFMAgs7d7Z9o1ljaDTlFGKP8fxL9NnjtSyjT3TJV5a3YTJ7SMlHR9zq2bYPJb7g69teQ+O3BnOkAAz3mNxs1xU0h1CyN8aQ9XmsffO8Ny3cUpYubl0SQ+RLdutEX/8EHXSKR0inGnnMQSd8Cz4pUVOOogyYo8ln47wl4juo+nHmgdFxi39B9m2eA+C2Zn0SACLCusXwRz1b2yc41cl/UMs3PCMpULTyu+HMGSzC0KB9W63P4EU2f2P7POhzhFvmk8Im0IMU4SWNQ5hbRPX9Mt8mtY+Vnw6wo4q0lmQ5tri3STi/akR4GblJixv2lrOgRdoDEvL8oOSbhnt5P8oiTbz10X7OaMefqaWQldSjhTgtuIgqFjGrHPhwU2zC/G4HhB3jjUd1wmWNQ7Rp12bNQ4rvRRuRJ3eJYRhXbrTGmHLY4NsW7wHwexMeiTADcfLI9Y1uy9KOiEWxlFhyFrGVLl/dD9EDNIq54XnLnZBxKLOpo824c36RWCzzZk4WOAsiHc6xBshz3D4ExZhZ2GbeVAZEoHtUrTZp038sOjCyIMhrfNcZz5er0HAH+msActBJ0GA8VToR/6yGLqWsVYQej7Zp//zLjjEFpFEWEvBTQsc0WRBWBm6AHFFfLGieXCn9ZzhEfPyAZDpY3kTlrj44QiXLzBJn/ybIp35RxSvNiFg8d6EmuNMhQCjHt5K0l+FiNA9kT7kD5V0ramcxAblRLQRVcQY4U2XAluKJ2KLP0JLHLaJzxg3pfCXAkyaCDdhCVPmQV6ETcudNLMZjofBoiXLlmX1egUBi/cKQD48KwLXiJedOYRtdkO8n6RrSmLqtzk5BLnNIagse3WL0ifdZcf2mq/jS7J4+zLYVQJ0Q+Sl559KohsivVVoYsFifJCk63qWoF29NKZx3hbvadSTS9k/gQtLupmkR8fkEky6/BVJr5T0EEk39EBa/VeCc+hOwOLdndUuhOSv7l7H854LJ9rL+YSbKd7ohsjHQ1+LbfxoFyaMnQmMQsDiPQr2ajNFkBh3grXdeQlcIKxvXnYy/spZ0dTCJ/tPk3Sf6C/tKeDOy817PRGwePcEdqLJYnnTG8Avm1ZX4PdJ4gXoPSX9taRTJH0rRkekDf3Jku4u6fAZvghdTWdiIab4phrx/vd4cfmBifF2cU2gNgJY53RB/HFJ14s19xjD3b5b0jskvTPW/1Vb4Xe5PBbvXa59n7sJtBM4UNJ1CjFH2K8azS7vl/S+MKBY82Xome3J2LdPAhbvPuk6bROYDwGGvWU8Eaz0XOhnzuBbn20IOv+MWXhhatcTAY9t0hNYJ2sCMyOAENMuzlK6KzRE/RaSfiT6m380RBwhT2v9w5K+Uybg7c0I2PLejJtjmYAJLCdAUwuWOe3nfGqOtX6piMK45h+R9J+SmGKOtnT2P748SR8tCVi8SxreniIBepnUNJ73FBkOVebLFE0utKFfWdJVJF2+KMB/hJCzZgwSRB1x/1ARxpv++srXwAwIYNXRxzsHEprBKc32FBiznOXVjTNkbHOGX00xZ03TC0MB4M9xhsg9vbDS01rPNR8j7ZRzm/dOVfcsT3Yq43nPEv6WTooxWmgLZ2k6WgewzEthZ/snwu9iEYEeL5+S9MlYs93c/3wz8Snvu9lkyrXnspuACTBD0eWKhbHREfvSLwUeWjTFpKhjyTOwF8LftjCBRrXOlne1VeOCmYAJdCCAwLIsm9z5+2MqsxT0FHheph4cE14w6QUPgqZjKjsmiWgTd+YrpecMY66zLreX+TERxJ4fDBbvZlV53wRMYG4EaA/na+wuX2RfvBBzBD0XXoznNmseAIzqyEBffKWKlpbrcjuP0Vce90xJ94rtjVcW743ROaIJmMAMCTCkLkv13RYZqMbOBEzABExgYgQs3hOrMBd3PwL8nfV43vthscfcCVi8517D8z+/sp/3/M/WZ2gCQcDi7Uth6gTcz3vqNejyb0TALyw3wuZIFRGgG5dn0amoQlyUYQjY8h6Gs3MxARMwga0SsHhvFacTMwETMIFhCFi8h+HsXEzABEygKwGaAZ8Yg60tjGPxXojGB0zABExgFAJ0f72DJD6jZzmmbYLtKQ5MdTVJH5R0dIwgBt08j1wP4TdkXm3n4/z33VfMt8i18JyYCX2fbz/XRDJvrtvqZwi/LMcQebXl4fyhss8li1zjm9u57urHELi4wxui/WJJx0t6AwfLRPcFr//3EEln1F9Ml3BDAt+MeP8niQVXrsttjp1fEgMP5SBBGb4MV263HW/z2yTOttLZJO914qwTNs8p16virjrels4mcbaVziZ5bxJnnfJikFxd0m2IFO5ESYj3s9JjiuJN2ZnZmgFkSojlNmHYX+W36nhbOpvE2VY6Y+adosq51OT4i8lFfVSM/lZT2VwWE9iEwLHR5s34Kk+K65uhbO1MwARMwAQqJnDFVS8rKy67i2YCJmACJmACJmACJmACJmACJmACJmACJmACJmACJmACJmACJmACJmACJmACJmACJmACJmACJmACJmACJmACJmACJmACJmACJmACJmACJmACJmACJmACJmACJmACJmACJmACJmACJmACJmACJmACJmACJmACywlcuzjMkMCMwodjzf46royf8fBjiOF13bby75TvVMfz7nRyDmQCJjBLAswkwzyPLEfE8KlMUnAPSafGclqM707YHAc/YTxbUjk+Nuk8Ig/GNrMznTPxQXGMzZNzNpuGP+FZzp7ppnEsd8nrxrkTDxseFM28HlmEad08oNXXniZgAiZQPwGEEGuX5SaSDovZwQ6SxEQGXwqBT7FPgUYoc7sptFj1PARYFrlS+Mv4iDJlwbEu8wnvsx84HCMN8mLNwwRDOvcp90pn8V6JyAFMwAQqIYC4MbMMa0QTAb5EWK+I4EdDsLG62af5A8ucMAjroXEexG9z+J8UlnXbzEzZJMNMN+l4gODwI88U/Xw45DHKnY5JhfmHcPvIj21ccz+821cW73Yu9jUBE6iPACKHSD6zaDZ5YAg5VjcOq/bIEHbEFoscUU7LnDCLxBuLl1nbEeQzQ2D3pbrvl3gcbzoEG7FelC7HS/GmfPw7YMlt0iz3aZpZ6izeS/H4oAmYQGUEEEJELud2XFY8LGGaLhBVxDWtYfzSQi7jE54F8aYZo/lOsLnPgwJLnSYa8iAufjwEyIOFfwjMsVq69CeftMI5juWdx8rwrdsW71Ys9jQBE6iQAJY0CyKJQ/xoKkFA0yGiiHs6hLNs5mj605xCM0vT8cISMS0deX+58KAcWPRZHg7xUECwWShf0+HHMRxCjdiTBo4Z4su0wrt91XyStIeyrwmYgAmMTyCtbSxo2qQRU8QPhxCmNY0ApghyvNwmDIJOE0yKK9vNl4SkR1t52dxBPoh3tlHzoGgeT8u7mV7mhXBTBtJv9oLZdybninnut65tebdisacJmECFBNKipWiI5BOLMiK0xxX7tIWnQOeLSg7ni8sU9IyS1jAiSzqsjy+sYsKl8Kcln+kj4oRHyHmwZDnZxq8U8nyY8PDJfwWkw4OI81nZ1p0F9toETMAEpkYghZZyI3q0GyOSWLPsly6FFXFlIQzim/usEdR0HCcOC6JK8wnbNM1km3mGZU18HgSEwREGPxxin8LMPvk0y46AI9qEQ9A7O1venVE5oAmYQEUEEGnapLGwU7gRS8QRMaTvNCKKpV06rHCaK7DU0xEmP4pBfAlD+gg21jfiTByWTI92ctInPKJLGI7RNTEFmgcBjn3EPa1t0iYc+2zTfE18zodykTZivtS5zXspHh80AROokABizYL4IZBlswRiiGXNcQQzreBVp4EIE5f0EE7SRHRTRDlGWqSZebDGERarmocHjrzLMmVTCXER6Wx2If0UeuIRjnMiHGHKNPal7F8TMAETMAETMAETMAETMAETMAETMAETMAETMAETMAETMAETMAETMAETMAETMAETMAETMAETMAETMAETMAETMAETMAETMAETMAETMAETMAETMAETMAETMAETMAETmBaB/wer4vcn5EkwjQAAAABJRU5ErkJggg=="}},"cell_type":"markdown","metadata":{"graffitiCellId":"id_2rrhvg5","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"1C7884059BD2464DB639268E872E99C2","mdEditEnable":false},"source":"# 模型选择、过拟合和欠拟合\n\n## 训练误差和泛化误差\n在解释上述现象之前，我们需要区分训练误差（training error）和泛化误差（generalization error）。通俗来讲，前者指模型在训练数据集上表现出的误差，后者指模型在任意一个测试数据样本上表现出的误差的期望，并常常通过测试数据集上的误差来近似。计算训练误差和泛化误差可以使用之前介绍过的损失函数，例如线性回归用到的平方损失函数和softmax回归用到的交叉熵损失函数。\n\n机器学习模型应关注降低泛化误差。\n\n## 模型选择\n### 验证数据集\n从严格意义上讲，测试集只能在所有超参数和模型参数选定后使用一次。不可以使用测试数据选择模型，如调参。由于无法从训练误差估计泛化误差，因此也不应只依赖训练数据选择模型。鉴于此，我们可以预留一部分在训练数据集和测试数据集以外的数据来进行模型选择。这部分数据被称为验证数据集，简称验证集（validation set）。例如，我们可以从给定的训练集中随机选取一小部分作为验证集，而将剩余部分作为真正的训练集。\n\n### K折交叉验证  \n由于验证数据集不参与模型训练，当训练数据不够用时，预留大量的验证数据显得太奢侈。一种改善的方法是K折交叉验证（K-fold cross-validation）。在K折交叉验证中，我们把原始训练数据集分割成K个不重合的子数据集，然后我们做K次模型训练和验证。每一次，我们使用一个子数据集验证模型，并使用其他K-1个子数据集来训练模型。在这K次训练和验证中，每次用来验证模型的子数据集都不同。最后，我们对这K次训练误差和验证误差分别求平均。\n## 过拟合和欠拟合\n接下来，我们将探究模型训练中经常出现的两类典型问题：\n- 一类是模型无法得到较低的训练误差，我们将这一现象称作欠拟合（underfitting）；\n- 另一类是模型的训练误差远小于它在测试数据集上的误差，我们称该现象为过拟合（overfitting）。\n在实践中，我们要尽可能同时应对欠拟合和过拟合。虽然有很多因素可能导致这两种拟合问题，在这里我们重点讨论两个因素：模型复杂度和训练数据集大小。\n\n### 模型复杂度\n为了解释模型复杂度，我们以多项式函数拟合为例。给定一个由标量数据特征$x$和对应的标量标签$y$组成的训练数据集，多项式函数拟合的目标是找一个$K$阶多项式函数\n\n\n$$\n \\hat{y} = b + \\sum_{k=1}^K x^k w_k \n$$\n\n\n来近似 $y$。在上式中，$w_k$是模型的权重参数，$b$是偏差参数。与线性回归相同，多项式函数拟合也使用平方损失函数。特别地，一阶多项式函数拟合又叫线性函数拟合。\n\n给定训练数据集，模型复杂度和误差之间的关系：\n\n![Image Name](https://cdn.kesci.com/upload/image/q5jc27wxoj.png?imageView2/0/w/960/h/960)\n\n### 训练数据集大小\n影响欠拟合和过拟合的另一个重要因素是训练数据集的大小。一般来说，如果训练数据集中样本数过少，特别是比模型参数数量（按元素计）更少时，过拟合更容易发生。此外，泛化误差不会随训练数据集里样本数量增加而增大。因此，在计算资源允许的范围之内，我们通常希望训练数据集大一些，特别是在模型复杂度较高时，例如层数较多的深度学习模型。\n"},{"cell_type":"markdown","metadata":{"graffitiCellId":"id_rxqm2hz","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"662A794758DD42EFA08B79C1EF032C65","mdEditEnable":false},"source":"# 多项式函数拟合实验"},{"cell_type":"code","execution_count":1,"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","graffitiCellId":"id_yjb9esq","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"F39C60E42E09441D8602243557564E0A","collapsed":false,"scrolled":false},"outputs":[{"output_type":"stream","text":"1.3.0\n","name":"stdout"}],"source":"%matplotlib inline\nimport torch\nimport numpy as np\nimport sys\nsys.path.append(\"/home/kesci/input\")\nimport d2lzh1981 as d2l\nprint(torch.__version__)"},{"cell_type":"markdown","metadata":{"graffitiCellId":"id_1aakii8","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"193FB89ACE2E4D4888FB258A250939AD","mdEditEnable":false},"source":"## 初始化模型参数"},{"cell_type":"code","execution_count":2,"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","graffitiCellId":"id_3xv7jo8","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"9714C0B5EF3344AB89F6C05008E10951","collapsed":false,"scrolled":false},"outputs":[],"source":"n_train, n_test, true_w, true_b = 100, 100, [1.2, -3.4, 5.6], 5\nfeatures = torch.randn((n_train + n_test, 1))\npoly_features = torch.cat((features, torch.pow(features, 2), torch.pow(features, 3)), 1) \nlabels = (true_w[0] * poly_features[:, 0] + true_w[1] * poly_features[:, 1]\n          + true_w[2] * poly_features[:, 2] + true_b)\nlabels += torch.tensor(np.random.normal(0, 0.01, size=labels.size()), dtype=torch.float)"},{"cell_type":"code","execution_count":8,"metadata":{"graffitiCellId":"id_m2mu8wo","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"FDF06E080C72455296FF40E5A03C8A1D","collapsed":false,"scrolled":false},"outputs":[{"output_type":"execute_result","metadata":{},"data":{"text/plain":"(tensor([[-1.3396],\n         [ 0.3057]]), tensor([[-1.3396,  1.7946, -2.4040],\n         [ 0.3057,  0.0934,  0.0286]]), tensor([-16.1723,   5.2089]))"},"transient":{},"execution_count":8}],"source":"features[:2], poly_features[:2], labels[:2]"},{"cell_type":"markdown","metadata":{"graffitiCellId":"id_6fb98pt","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"04AB33FA366C42CDBF58FE28874F3F27","mdEditEnable":false},"source":"## 定义、训练和测试模型"},{"cell_type":"code","execution_count":9,"metadata":{"graffitiCellId":"id_tq07955","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"DD5CB5E380BE4BF498E4F3D5A9F97F43","collapsed":false,"scrolled":false},"outputs":[],"source":"def semilogy(x_vals, y_vals, x_label, y_label, x2_vals=None, y2_vals=None,\n             legend=None, figsize=(3.5, 2.5)):\n    # d2l.set_figsize(figsize)\n    d2l.plt.xlabel(x_label)\n    d2l.plt.ylabel(y_label)\n    d2l.plt.semilogy(x_vals, y_vals)\n    if x2_vals and y2_vals:\n        d2l.plt.semilogy(x2_vals, y2_vals, linestyle=':')\n        d2l.plt.legend(legend)"},{"cell_type":"code","execution_count":10,"metadata":{"graffitiCellId":"id_3lzy7bj","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"3DA4335947C44C018CD2C8B215A641ED","collapsed":false,"scrolled":false},"outputs":[],"source":"num_epochs, loss = 100, torch.nn.MSELoss()\n\ndef fit_and_plot(train_features, test_features, train_labels, test_labels):\n    # 初始化网络模型\n    net = torch.nn.Linear(train_features.shape[-1], 1)\n    # 通过Linear文档可知，pytorch已经将参数初始化了，所以我们这里就不手动初始化了\n    \n    # 设置批量大小\n    batch_size = min(10, train_labels.shape[0])    \n    dataset = torch.utils.data.TensorDataset(train_features, train_labels)      # 设置数据集\n    train_iter = torch.utils.data.DataLoader(dataset, batch_size, shuffle=True) # 设置获取数据方式\n    \n    optimizer = torch.optim.SGD(net.parameters(), lr=0.01)                      # 设置优化函数，使用的是随机梯度下降优化\n    train_ls, test_ls = [], []\n    for _ in range(num_epochs):\n        for X, y in train_iter:                                                 # 取一个批量的数据\n            l = loss(net(X), y.view(-1, 1))                                     # 输入到网络中计算输出，并和标签比较求得损失函数\n            optimizer.zero_grad()                                               # 梯度清零，防止梯度累加干扰优化\n            l.backward()                                                        # 求梯度\n            optimizer.step()                                                    # 迭代优化函数，进行参数优化\n        train_labels = train_labels.view(-1, 1)\n        test_labels = test_labels.view(-1, 1)\n        train_ls.append(loss(net(train_features), train_labels).item())         # 将训练损失保存到train_ls中\n        test_ls.append(loss(net(test_features), test_labels).item())            # 将测试损失保存到test_ls中\n    print('final epoch: train loss', train_ls[-1], 'test loss', test_ls[-1])    \n    semilogy(range(1, num_epochs + 1), train_ls, 'epochs', 'loss',\n             range(1, num_epochs + 1), test_ls, ['train', 'test'])\n    print('weight:', net.weight.data,\n          '\\nbias:', net.bias.data)"},{"cell_type":"markdown","metadata":{"graffitiCellId":"id_tvv0n31","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"FEBABF4CE4FC49F8869E557513F6CFD8","mdEditEnable":false},"source":"## 三阶多项式函数拟合（正常）"},{"cell_type":"code","execution_count":11,"metadata":{"graffitiCellId":"id_pc28vr5","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"CD685B472B744329A1CFC47C9F0B5E89","collapsed":false,"scrolled":false},"outputs":[{"output_type":"stream","text":"final epoch: train loss 0.00012718481593765318 test loss 0.00022606355196330696\nweight: tensor([[ 1.2121, -3.4013,  5.5954]]) \nbias: tensor([5.0013])\n","name":"stdout"},{"output_type":"display_data","metadata":{"needs_background":"light"},"data":{"text/plain":"<Figure size 432x288 with 1 Axes>","text/html":"<img src=\"https://cdn.kesci.com/rt_upload/CD685B472B744329A1CFC47C9F0B5E89/q5mwgfn2a0.png\">"},"transient":{}}],"source":"fit_and_plot(poly_features[:n_train, :], poly_features[n_train:, :], labels[:n_train], labels[n_train:])"},{"cell_type":"markdown","metadata":{"graffitiCellId":"id_pu3gps2","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"F5C7FE5EF88D44FE85E0B651B3B09959","mdEditEnable":false},"source":"## 线性函数拟合（欠拟合）"},{"cell_type":"code","execution_count":12,"metadata":{"graffitiCellId":"id_07e409i","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"33AD626DA0B94DB7A28D47697312B45D","collapsed":false,"scrolled":false},"outputs":[{"output_type":"stream","text":"final epoch: train loss 58.42559051513672 test loss 169.175048828125\nweight: tensor([[13.3642]]) \nbias: tensor([2.0402])\n","name":"stdout"},{"output_type":"display_data","metadata":{"needs_background":"light"},"data":{"text/plain":"<Figure size 432x288 with 1 Axes>","text/html":"<img src=\"https://cdn.kesci.com/rt_upload/33AD626DA0B94DB7A28D47697312B45D/q5mwgoekr9.png\">"},"transient":{}}],"source":"fit_and_plot(features[:n_train, :], features[n_train:, :], labels[:n_train], labels[n_train:])"},{"cell_type":"markdown","metadata":{"graffitiCellId":"id_hmoo3h2","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"BC253732133341DC8F028FB6834C68F0","mdEditEnable":false},"source":"## 训练样本不足（过拟合）"},{"cell_type":"code","execution_count":13,"metadata":{"graffitiCellId":"id_h1gobje","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"AB13F65A70A9484788F8004E427EC290","collapsed":false,"scrolled":false},"outputs":[{"output_type":"stream","text":"final epoch: train loss 1.9342325925827026 test loss 26.484249114990234\nweight: tensor([[ 3.4355, -2.5477,  3.8254]]) \nbias: tensor([2.3247])\n","name":"stdout"},{"output_type":"display_data","metadata":{"needs_background":"light"},"data":{"text/plain":"<Figure size 432x288 with 1 Axes>","text/html":"<img src=\"https://cdn.kesci.com/rt_upload/AB13F65A70A9484788F8004E427EC290/q5mwgvfscy.png\">"},"transient":{}}],"source":"fit_and_plot(poly_features[0:2, :], poly_features[n_train:, :], labels[0:2], labels[n_train:])"},{"cell_type":"markdown","metadata":{"graffitiCellId":"id_4tfoquk","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"A1A8C28C0A204B04AD8B5FFF9C2D2F5B","mdEditEnable":false},"source":"# 权重衰减\n## 方法  \n权重衰减等价于 $L_2$ 范数正则化（regularization）。正则化通过为模型损失函数添加惩罚项使学出的模型参数值较小，是应对过拟合的常用手段。\n\n##  L2 范数正则化（regularization）\n$L_2$范数正则化在模型原损失函数基础上添加$L_2$范数惩罚项，从而得到训练所需要最小化的函数。$L_2$范数惩罚项指的是模型权重参数每个元素的平方和与一个正的常数的乘积。以线性回归中的线性回归损失函数为例\n\n\n$$\n \\ell(w_1, w_2, b) = \\frac{1}{n} \\sum_{i=1}^n \\frac{1}{2}\\left(x_1^{(i)} w_1 + x_2^{(i)} w_2 + b - y^{(i)}\\right)^2 \n$$\n\n\n其中$w_1, w_2$是权重参数，$b$是偏差参数，样本$i$的输入为$x_1^{(i)}, x_2^{(i)}$，标签为$y^{(i)}$，样本数为$n$。将权重参数用向量$\\boldsymbol{w} = [w_1, w_2]$表示，带有$L_2$范数惩罚项的新损失函数为\n\n\n$$\n\\ell(w_1, w_2, b) + \\frac{\\lambda}{2n} |\\boldsymbol{w}|^2,\n$$\n\n\n其中超参数$\\lambda > 0$。当权重参数均为0时，惩罚项最小。当$\\lambda$较大时，惩罚项在损失函数中的比重较大，这通常会使学到的权重参数的元素较接近0。当$\\lambda$设为0时，惩罚项完全不起作用。上式中$L_2$范数平方$|\\boldsymbol{w}|^2$展开后得到$w_1^2 + w_2^2$。\n有了$L_2$范数惩罚项后，在小批量随机梯度下降中，我们将线性回归一节中权重$w_1$和$w_2$的迭代方式更改为\n\n\n$$\n \\begin{aligned} w_1 &\\leftarrow \\left(1- \\frac{\\eta\\lambda}{|\\mathcal{B}|} \\right)w_1 - \\frac{\\eta}{|\\mathcal{B}|} \\sum_{i \\in \\mathcal{B}}x_1^{(i)} \\left(x_1^{(i)} w_1 + x_2^{(i)} w_2 + b - y^{(i)}\\right),\\\\ w_2 &\\leftarrow \\left(1- \\frac{\\eta\\lambda}{|\\mathcal{B}|} \\right)w_2 - \\frac{\\eta}{|\\mathcal{B}|} \\sum_{i \\in \\mathcal{B}}x_2^{(i)} \\left(x_1^{(i)} w_1 + x_2^{(i)} w_2 + b - y^{(i)}\\right). \\end{aligned} \n$$\n\n\n可见，$L_2$范数正则化令权重$w_1$和$w_2$先自乘小于1的数，再减去不含惩罚项的梯度。因此，$L_2$范数正则化又叫权重衰减。权重衰减通过惩罚绝对值较大的模型参数为需要学习的模型增加了限制，这可能对过拟合有效。\n"},{"cell_type":"markdown","metadata":{"graffitiCellId":"id_6r7el1m","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"86B363E2493A4A2980C5E00729B925F5","mdEditEnable":false},"source":"## 高维线性回归实验从零开始的实现\n下面，我们以高维线性回归为例来引入一个过拟合问题，并使用权重衰减来应对过拟合。设数据样本特征的维度为$p$。对于训练数据集和测试数据集中特征为$x_1, x_2, \\ldots, x_p$的任一样本，我们使用如下的线性函数来生成该样本的标签：\n\n\n$$\n y = 0.05 + \\sum_{i = 1}^p 0.01x_i + \\epsilon \n$$\n\n\n其中噪声项$\\epsilon$服从均值为0、标准差为0.01的正态分布。为了较容易地观察过拟合，我们考虑高维线性回归问题，如设维度$p=200$；同时，我们特意把训练数据集的样本数设低，如20。"},{"cell_type":"code","execution_count":11,"metadata":{"graffitiCellId":"id_m6mthxc","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"51E803AFA42047D48605A9042CD266F8","collapsed":false,"scrolled":false},"outputs":[{"output_type":"stream","text":"1.3.0\n","name":"stdout"}],"source":"%matplotlib inline\nimport torch\nimport torch.nn as nn\nimport numpy as np\nimport sys\nsys.path.append(\"/home/kesci/input\")\nimport d2lzh1981 as d2l\n\nprint(torch.__version__)"},{"cell_type":"markdown","metadata":{"graffitiCellId":"id_d4sywej","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"5AC8DFCEDD734B8289B2CE7D06C82B6F","mdEditEnable":false},"source":"## 初始化模型参数\n与前面观察过拟合和欠拟合现象的时候相似，在这里不再解释。"},{"cell_type":"code","execution_count":12,"metadata":{"graffitiCellId":"id_3ymafxv","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"D66AE4FB939640948F151F6FF7C1740E","collapsed":false,"scrolled":false},"outputs":[],"source":"n_train, n_test, num_inputs = 20, 100, 200\ntrue_w, true_b = torch.ones(num_inputs, 1) * 0.01, 0.05\n\nfeatures = torch.randn((n_train + n_test, num_inputs))\nlabels = torch.matmul(features, true_w) + true_b\nlabels += torch.tensor(np.random.normal(0, 0.01, size=labels.size()), dtype=torch.float)\ntrain_features, test_features = features[:n_train, :], features[n_train:, :]\ntrain_labels, test_labels = labels[:n_train], labels[n_train:]"},{"cell_type":"code","execution_count":13,"metadata":{"graffitiCellId":"id_xd0w3ub","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"E1707F6923A742F69F84896E84B463FE","collapsed":false,"scrolled":false},"outputs":[],"source":"# 定义参数初始化函数，初始化模型参数并且附上梯度\ndef init_params():\n    w = torch.randn((num_inputs, 1), requires_grad=True)\n    b = torch.zeros(1, requires_grad=True)\n    return [w, b]"},{"cell_type":"markdown","metadata":{"graffitiCellId":"id_ejz9flm","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"01F0DAF1CD6F48BA92BAAE9F7554EA28","mdEditEnable":false},"source":"## 定义L2范数惩罚项"},{"cell_type":"code","execution_count":14,"metadata":{"graffitiCellId":"id_kkiguji","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"4709112E91504EF68681066730402A03","collapsed":false,"scrolled":false},"outputs":[],"source":"def l2_penalty(w):\n    return (w**2).sum() / 2"},{"cell_type":"markdown","metadata":{"graffitiCellId":"id_99qaypt","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"06C980C7C2534A728058234C02210A34","mdEditEnable":false},"source":"## 定义训练和测试"},{"cell_type":"code","execution_count":15,"metadata":{"graffitiCellId":"id_6crgl61","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"3A369C128AE64990854186DAD85DDD7F","collapsed":false,"scrolled":false},"outputs":[],"source":"batch_size, num_epochs, lr = 1, 100, 0.003\nnet, loss = d2l.linreg, d2l.squared_loss\n\ndataset = torch.utils.data.TensorDataset(train_features, train_labels)\ntrain_iter = torch.utils.data.DataLoader(dataset, batch_size, shuffle=True)\n\ndef fit_and_plot(lambd):\n    w, b = init_params()\n    train_ls, test_ls = [], []\n    for _ in range(num_epochs):\n        for X, y in train_iter:\n            # 添加了L2范数惩罚项\n            l = loss(net(X, w, b), y) + lambd * l2_penalty(w)\n            l = l.sum()\n            \n            if w.grad is not None:\n                w.grad.data.zero_()\n                b.grad.data.zero_()\n            l.backward()\n            d2l.sgd([w, b], lr, batch_size)\n        train_ls.append(loss(net(train_features, w, b), train_labels).mean().item())\n        test_ls.append(loss(net(test_features, w, b), test_labels).mean().item())\n    d2l.semilogy(range(1, num_epochs + 1), train_ls, 'epochs', 'loss',\n                 range(1, num_epochs + 1), test_ls, ['train', 'test'])\n    print('L2 norm of w:', w.norm().item())"},{"cell_type":"markdown","metadata":{"graffitiCellId":"id_55gl5e9","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"BA54CB5FD4E64380AC1357E8537265DC","mdEditEnable":false},"source":"## 观察过拟合"},{"cell_type":"code","execution_count":16,"metadata":{"graffitiCellId":"id_6pmjf6b","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"C27406AAA0FD41C6801D55ED4B25D5EA","collapsed":false,"scrolled":false},"outputs":[{"output_type":"stream","text":"L2 norm of w: 11.6444091796875\n","name":"stdout"},{"output_type":"display_data","metadata":{"needs_background":"light"},"data":{"text/plain":"<Figure size 252x180 with 1 Axes>","text/html":"<img src=\"https://cdn.kesci.com/rt_upload/C27406AAA0FD41C6801D55ED4B25D5EA/q5jf5cs7lp.svg\">"},"transient":{}}],"source":"fit_and_plot(lambd=0)"},{"cell_type":"markdown","metadata":{"graffitiCellId":"id_ejl383l","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"F2335EC469AB402C890CE258932583D1","mdEditEnable":false},"source":"## 使用权重衰减"},{"cell_type":"code","execution_count":17,"metadata":{"graffitiCellId":"id_x1tkbn7","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"0770D8C23B8144C59D13D24390E471F0","collapsed":false,"scrolled":false},"outputs":[{"output_type":"stream","text":"L2 norm of w: 0.04063604772090912\n","name":"stdout"},{"output_type":"display_data","metadata":{"needs_background":"light"},"data":{"text/plain":"<Figure size 252x180 with 1 Axes>","text/html":"<img src=\"https://cdn.kesci.com/rt_upload/0770D8C23B8144C59D13D24390E471F0/q5jf5d4mwp.svg\">"},"transient":{}}],"source":"fit_and_plot(lambd=3)"},{"cell_type":"markdown","metadata":{"graffitiCellId":"id_g2d3ly2","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"FD1464A14A6149AB81CF443A10F843A4","mdEditEnable":false},"source":"## 简洁实现"},{"cell_type":"code","execution_count":18,"metadata":{"graffitiCellId":"id_b6kxfkc","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"20CFA1339D054931892561E254150368","collapsed":false,"scrolled":false},"outputs":[],"source":"def fit_and_plot_pytorch(wd):\n    # 对权重参数衰减。权重名称一般是以weight结尾\n    net = nn.Linear(num_inputs, 1)\n    nn.init.normal_(net.weight, mean=0, std=1)\n    nn.init.normal_(net.bias, mean=0, std=1)\n    optimizer_w = torch.optim.SGD(params=[net.weight], lr=lr, weight_decay=wd) # 对权重参数衰减\n    optimizer_b = torch.optim.SGD(params=[net.bias], lr=lr)  # 不对偏差参数衰减\n    \n    train_ls, test_ls = [], []\n    for _ in range(num_epochs):\n        for X, y in train_iter:\n            l = loss(net(X), y).mean()\n            optimizer_w.zero_grad()\n            optimizer_b.zero_grad()\n            \n            l.backward()\n            \n            # 对两个optimizer实例分别调用step函数，从而分别更新权重和偏差\n            optimizer_w.step()\n            optimizer_b.step()\n        train_ls.append(loss(net(train_features), train_labels).mean().item())\n        test_ls.append(loss(net(test_features), test_labels).mean().item())\n    d2l.semilogy(range(1, num_epochs + 1), train_ls, 'epochs', 'loss',\n                 range(1, num_epochs + 1), test_ls, ['train', 'test'])\n    print('L2 norm of w:', net.weight.data.norm().item())"},{"cell_type":"code","execution_count":19,"metadata":{"graffitiCellId":"id_dxnud8x","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"525D01167F0E40509495588D6B0A0FB9","collapsed":false,"scrolled":false},"outputs":[{"output_type":"stream","text":"L2 norm of w: 13.361410140991211\n","name":"stdout"},{"output_type":"display_data","metadata":{"needs_background":"light"},"data":{"text/plain":"<Figure size 252x180 with 1 Axes>","text/html":"<img src=\"https://cdn.kesci.com/rt_upload/525D01167F0E40509495588D6B0A0FB9/q5jf5e5i21.svg\">"},"transient":{}}],"source":"fit_and_plot_pytorch(0)"},{"cell_type":"code","execution_count":20,"metadata":{"graffitiCellId":"id_qclwxdh","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"3FAACA854B9545A8ADADDEB6EE17A680","collapsed":false,"scrolled":false},"outputs":[{"output_type":"stream","text":"L2 norm of w: 0.051789578050374985\n","name":"stdout"},{"output_type":"display_data","metadata":{"needs_background":"light"},"data":{"text/plain":"<Figure size 252x180 with 1 Axes>","text/html":"<img src=\"https://cdn.kesci.com/rt_upload/3FAACA854B9545A8ADADDEB6EE17A680/q5jf5fa51u.svg\">"},"transient":{}}],"source":"fit_and_plot_pytorch(3)"},{"attachments":{"image.png":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAVAAAADICAYAAAC+qrlDAAAgAElEQVR4Ae2dB5hVxfn/55a9uzRdKwZRsUQUNZJYULFg1yiKLQkalVjiP5EISYyxJhgVY5pgSeJPjWCJJhpBMQkaC0ZRbBFj74BYUIkoCLt3b/k/n7sz63A459459d5d5n2eu+fuuXOmvGfmO++87zvvpIQlywHLAcuBaDnQKoTYSwgxVGY7SF7nyetcIcTDQogl0RabfG6p5Iu0JXYzDqw2gyHAewEgAAp4BI0QQsyS3wEHQAKwWB0IHpyYyTYdXSx07E6DW3r3KWy8xTZtuZZeffk/37Zi2YI3XmxpW/55lv8z2aZHi4WOO4QQU7srmFoAXR26tv82rpaDwZBNgOSJ2VzzAYV8+wCe+dImX1621rr9W9KZTLZULBY++XhR2/vzX6+ARjbX/F4h336fBAkFroZFdZtkP0+nMz9uam5uGX7QMU277DdKbLvzCNF3DTWvrNyOZZ8tES88OUvMuX+6mD3z9o6O9va2Uqn4WyHEhSunbPz/LIA2/jtKuoar7WCowWikzcuRMjfeYkj+gG+cmgMkNttarVJXffqtl+dWgOK+v16bX/DGSzkpnf6wB0mlQ9OZ7IxsU9P6R51yVm7kieM9QXNV7nTeAUxnTJ0k/nbdr/KFjo4PS8XCyO7EHwugXm929bu/2g+GKq8c4Bw/+Ku7tp3ww4kt2w1DCPVHzz8xS9x4+bltrz77eIsQYpIQAiDtzjQmncleudXQXVp++Kubsv0HKjVnsCYtWjhPXH7W8YVX5s5pKxULPxBCTAmWU7JPWQBNlt+NWpodDO5vpjWTbZqRyWZ3PuFHE3OHnTjePZWPu3dPnSRu/N25+WKh8GSx0IG01R0NKWOEEDfsc8SJYvwvo8W5SWePEQ9OQyUqvtMdQNQCqI/O30OT2sHg/mJbm3LNs7NNTYMvveWRTLWluvvj3ndZ2p9z3B7FQkfHqx359uHdDEQr/WXcpTeIfY/ka/T0wJ1TxORzwM/GB1ELoNG//+6Uox0M7m+rNZ3O/nvgZlsO/uWts3NexhD3R83uovs7e/Tw/MK3Xnu1VCrs2U1AdCiW871GHtsnasnTyTUk0Vl337KsVCzs0cg60Yyz4vb/1YYDDIY79j78+NyxP5gQW6OR3Ba9O0/Mf/3F/crl0j+FEB/EVlh0Gf+hpXfv/S645p6msLo9ryrlmlvEVl8bnnnorhvXKXR0YM2/yytto9xPZ7JPbTV0l9bz/3B3Ou46YcmfO/tf6cUfvndYuVxCB92QFDsjGrLVtlIC6+ng7Yc1xy1JwGrKwNhAmd2A9UjlY87/w4xIl+1u7WZyueCP9+ATWSnTLU0D3ZuQbcr1x2CUVJ0oizLpQkmV6bccK4H65VjPSD+hqbnl4Ium3N8Ux/LUjUXbDdsnPfO2a3oVCx1LhRBz3NI0wL3WVCp978gTzmg+ePT/S6Q6SLgs59944Zkdy+XSDUKItkQK9ldIazqd+esxp53Ta5f9R/l7MkRq+max0JF58alHdhOi/MdG5I2pBOruEWvOHG9nOfM8bMpoOMBg+NFRp/wktuWpWzUBCspMpdI4S4ftT25FRHFvfK6lpdfoGFUabpWkvAaXtMY0Nbf0ws8zaaLMXEuv3lJKT7r4muWZAii+Ck4QhJu4YLg5xTnT4vemCIcxp9MYvz9U46Oet9dwHLCDwYN/TCxHnvyTxKRyVQ0kLSYXylf3GunK9szhBx2dTWq1oredMimbOuj3G+W7CYBiYdheCHG4EOLnsuLobABJgJDfqwGms61u+h6eRzLx+riBtDNf+78BB+xg8GTSiFKp2G+fI+ieyRPlUr6HQJJ8hb4osbVY6BiOUadetN3OI1jK4+7VcCsXEwAlgsp8GRiB4AhIo/Syt4UQ44QQz8qdFWF7HkEZvD71enc9rVw7GLzf6KiBmw1eEZfV3bvYzl8od+3+A/JCiPohlXslK8IL21b9ErpdPmFp2Bfg7b8SYQuv8bypRQ3QRNLk5QKoLN91qRPJMWzjAGcv2tvrB3vfFwdCDQZKCruMqwyGTidp6jLdV+1jTNyUa97/q7sf1CtIEWxDjAJ4hx94TG7mbX/cvyPfHqQacT0ztKV332LfNVqNDc6A5qVjjxDlcll8/tkS0X/gpuLcq6cFrh99jjq0LV8G5jRMn6FBJgCKZAlwYiFkOgFAFTfYz0ujGAzOSDNlB8f0/wFcJ/E8ZW3i/EFKwC637S2fHLCDwYNhHfn2IX5BkB0zbM3ss0ar+PDdeWLcpVNEkH3yqkqUTz3U/41y3WTwdh1EnzOtz+Szx4hh+x4u1NbXc48fUeGT+t80Hz3dxlsMWfHaf5/UbzXEdxMABTQV6gNyfBQAopMA9Jzg6ZRG+d15T2cAS3eI2Ikq6Kq8VbmEXwfoua3G3+1g8H75m26lL6q80/EL2zFvvWqCmDR9bkUqJ1jIxNNHiVufDt5V/ZRfvXaR/jqoqSlHABQjQhp/+5W54rzfK8gQAh0mkigE35h0oH2PGGM84ciYok7jcyWfev4xAVClk0GCBARZyit9p1skAdKodF5tcwLqc/IZBaRuzwHWTqB2S2fveXMg0sGgirn2kvECVxzT5X0DDgbfAxMQGD32izYjeX6+9FPFEgFPoM+XLgkScIP6uAkSXfk36hck8fU3XJmdgKYyQiGdXnJT5zAeP2qouO7BbtnMLvabGJEAQyzwEK5GEMDpBp7yZ18Xpn24yAc9KL0QnSffAVZlwOrenPbFksZI7DUY1ABB6jrvhL3Fg9OmGINnY7QsfC2cvAEk1t+wU/sEuDKZnHreJNF/w0FdElf4UuuSw7x824rP/ZT84bsM2U6CLwRPVoagyXd1SuxO/qn0XldZh4bDABMJlKlUHUtwhDQgKb/OzmnWq9Vm9wFopiSYg5yvjFV8V1InEm/DMc+seQ2VSg2GPqa1chsM42QIM5ZqJ59zubhuor9u0ICDIVDfAgQU3XrlhMqSlP+drlB9+vn2vvkiY1VAHa8L3nip2bR4JPFNt9peTDz9iIokOuf+aYL+oq9O4BWTDOoPU/JTB9M8o0hnAqBqCY8PqFLwMGJUr3Bbrp8oXZJUHfEjVdIr9/hfEfkDkORHGiRbnud/1gJrSgMWUqkqXz1rrz454Kcj1hoMGAU4lgEdlx/yUwc/+YZNy4RgagRi+T75nDHiw/fmVySs3n3XFGqnjgILjEwEUjHd2UT5DUhz25Yvy2JZV+2qVUf0n0ie6D2/NfbnqzwHPzbdemhl5WLCG8qmDpogV6sKif1uAqBulWHZrYAPaRSgU1IpIIeRqdoSX6Ulb7TNzLgAJhFpCLGnetJkCZoALGVaHajb2zC/F/lgePvluZXBYFqFRh0MTbnmlxctnLe1aTsAWiQo2o/F2RkvlHBsSJ5+grVggKEeDebGVBlzLMOVHtOER05+8Aw8YdLlN6R3vBdM6In7uwxSDTf+/QCoih3KEhtSIOhcvwF+CgBl0lUu+jMqH4BX5e18oBoYO9Pa/705ENlgUEU8/+SsVZas6je3a6MOho58+33PPnovgoCxLygSmZvEClAglfsNODz73jvy1MONb3W8tySTbZo95/7poXcjIbUrdQ96dHTEJkQfow7FQkfDrUBNjEgmbbRpugcH1GCIrLYMBD/+k2owNKA6ZvrCt17phRQYhpCwka4emDZF4P8ImJoQ5f5v0bscPNclbpk8l0Qajh5+bOYdHbQtDNFPWN7zMQVPyqRsefxxmOJjedYCaCxsbdxMoxoMqoV+lqgNPhhmpdOZZXgUhCGk0ok3zer6mPKHcim/jmoqVoXOgD5s18Z9MZ9vb2vn9MykiTLb21YQ4i/ci4mp4hZAY2JsA2c7xQ4G97fD2eTTrv9NaEnLPXfvu0wsd17/67w8G907Yby/IF6uJf2xsQryQaXB4US/p27UMayE7qcJlHXrVReKcrn0swZcsVSaYgHUzxvtGWmX2MHg+SInIe3gZpMkoRfsyOc/0uwKSRZPWV8WQuzg2Eat/LGV5DeBOk766QmFpCpHWelM9t068qVmUy2A1mRRj0xQz8EAQ4niVW3XWb2YvqRcLp0x48bJgk0CSRDlcIxvqVg4vw5S1j7S8+U1IcRJQojr5f+AJxLoSkwoFQuHvvzs4+3yxMxY2UMZlEWZsRYUMnMLoCEZ2F0fr+NgwEiCmxuDE/0a4RAJVAOomvm1RMt0ymSnHQeXoQPk/6kX/b+RBXwZ4yTypxzKS1DHh5fBaUKIF4UQDwghOHPoWNnuq7QlvFvj55aKhbH4t/KJi1T+pWLh2wYePXFVwyhf5ZpklNgm6nEcwER8Q0JnfLOLTVmY8SFWEb1gKnv/AC6kHreBGzXj0e0B2JSnRxBh+zD3ONb4kYGbD97yl39+NL5jjY/dPb/wzVdfK5UK20XdQJf8NhJCjBVCnEpUQiHE7XLSeNqRlvdQy9xe6Te4adF3oiQkTwnO6F7jQ+mIKm0coiqi8mw2jcUBwGr+Ew/cNYodNX4cpU2awWDACCANEbdpz7wiN0wAVhArIQZLUgMGACUUIwLEBrIOLFt3kQeXtZXLpVuXL/vsyHtuumLtHfY8OL3WeiqZTB3iguR5xmFfKX72v4/eKBY6do/5sLQdhRBXCCE4lG0rDEJS4rxRCPGeSzNMDrXD12vF/Nde2Pm5x+5Pbzds77TpLiWX8iq3MBhd8r3DCk/N+seKcrmEhJxUX/CqktF9C6BGbOrRiSogOv+1F/aPaTBcIvVqTqmGJbwKlP2Y3M47WgjxkjztIE6mczb9TkKIg2XwGsK1UbYu/baVisVbyuXyng9Om7JBS6++mcFDwddwdPfUyeK3Zx6X78jnnyx05Pc1kPaCFNgkhPimEOI6qS4h0v15Us95vxACd6kghHT6UyEEk+H3y+XSnxcveu/we/9yTUuxUMgQjo/z7v0QHgh/u/ayyt75xYvee79ULOwnhJjpJ496prVL+Hpyv7HKHprOZO9pyuXWO/KUs3IjTxi3yh7mWtVlMGCAwYqN9VQaAJBWPpESBeKo7qnOgOR/loQLhBDXCiG+JoS4VQjBAWsAXdQ0WOochwkhLpbLWrYQV/N4xwFy3FZf263t+PGXtLjtPqpVSYxF11w0Nr/g9RdxlmeLMmCNSsM5sdTKqtrv6wohOI/5e1K3Sf6U9Ui1hwx/gz/Ew0B6V6oO9eiEVCr9o+aWXi3DDz6miZXMNjvt5dl/6CdPPHBXJYbA7H/e3oHng3RVSt7RVLUg4NUCaEDG9cDH0AXyGRTDYGA5RoAYwIJBomIcwEZd58ZSnsGP1AohNbHk1E8zkD/5vpA3y3ZAE2PRKUKI9WV90MnWAjJ4Q9332vjL2+QPOObU3LY777XKHni9VizVAYrZM29XwAnwAERMGuSF8Yo6Kd2w/rif7+hQyQdjENLmn4QQvxVCvOMnE4+0qFlQdAKcitz0k7zHMRxaKA+A4xiOwsZbbNPW3KsXOlfRvmLFsgVvvNgiA4Oo7Zl3yMm1Fv9V2fZqOdBwHGBQIyUqKYzBMD6TbXpUgle5pXffji2/MmzpdsNGlPnwnXvqd5lWj9KlNxKAAgTVh7Iw4ngRCsc/y/SzhRBhj7lg8M+RbTzeq1DD+wDKlGyuGf1hpT0DNt1y6TY77tEBX7jyv/pNpmMCUfpeVQz/K35gUIPnfgj125FSGiSf14UQZwghONkzSkLVwod3pupbq678zjv/pYxvQfv54GDLh99q5RFlG2xelgOxcQBJSA0M3SKtClSDQXX+oIMBqUuVg8RlInXhp/iGEAKgvkwI0VtVyvDKCoslLYD2DyHEAMPnTJPBLyYNxRuARn3nvhs/9byRuhRPak0q6jnex0+kFMuzBB85RBrEVJoor5T3oOQh75D3b0r0rVo8MM3LprMcaCgOIJXhg6kGMNc4icEESCApfSyE2NCwMIL5Ap6AKHpSU8dq2sdSHfAERJMgP+BCfZhEFP8BX77rS2W9zuhu/8ApIfLDd+7FTUiRy6XKg/ohPZoQ7SA9E4kly4EexQGkCqQfXSpkAMdJakChP1skjSjGkfHlMp7lPINyhhACv0Y30qVOANQLkNyeDXMPSYu6+ZG4UJnwDO1ichjoUgGkS6RM0vG+kD55f0kQ7wqd6v6yMKcaolodmEyos8lKo1o+9jfLgYbkAIMBPSMSGhZhJMS4SYELRg9caf7l57hcuUxlUCPB8jxgose0ZYmuS51JGkmVKsQPHwFCQOb7EkDvlC9A3y3E7xif0Hcm6XYIaAKeQaR3NVlSd1YdliwHehwHtpWD91fyqgxISTUUfz+2Mfpd9lK/daSPY0luScT1CeMQgxVjUVJSp84rpc/0CxhqUsEvFTUFk8r/pHM91vQkdinp7eD7V6R/rJ/JQM9DSZ8AKB/VRj2N/W450K05wBa+56VUQwdPalmoMw3XJgbY2fpNH993E0K8KvMAjM+Vu5p8ZBFJUqeXgameUBW+hxACd56ibMs1Qgh8OutBlIv7E/UJEitDSZ+oG5hU+Fg9aD3epC0zNg58VwiB9EYIs3oTBiLqwl55v3S0lDrZS49xBdciduAkTej5lATK1UTvh3EMqV8Z8rjyP7twCGQctTuSCU9QHbArjI/x0SaOjAFQJNd7hBAvS14ElWQdWdt/LQfqzwGWvywz2R/dCISeEt1fu9yLblKn9WQwDHR0BNtFUsKoRHQhJFqum5tkFFEapC2s4vCVK/97ET6uvxBCfChVGEh6SKCKWAksNARh9UwUV3hIXZA+o5B+mURMJpIo6m7zsBxIjAO3CCHelxF5Eiu0RkFsb3xc6v42qZEWqRPwwfClH4+tHkMCRRIlKAZbD43PNFcZBLwqx3gvSzVBPeA9oI+OE8nby5NA6UNPD1iXII/hrkRAFfSfUVBShsko6mrzsBww4oAa5EcZpU42EcdJILnhJ8p3JyGZEYJNSZ0EzPCiNYQQV0qdIo74OOTHTYq3OoBiNSdICYYtJGNicBJpyGR5fJZsq9skEXVbnO5KUeRvdZ9RcNHm0TAcQBJ7q8Ej3WwhpTOkUaRSRURN4mgHwuD5ARSs889I8MJlK7qYdKpmX1x1AAXs1W4h9Lv4reJ14Jf+mYA+NIy7kld7aD8Thl9jmld+9r7lQN05cJHcUeLmrF33ymkVIGYc+lD0ogxEXHmwsLPk9RcrrTNTdHsshZGI+PA9Dv9QBaBEdsKg9Zk0pmymtc3vV9ofpz40rLuSV3uIOwqAWvclLw7Z+92KA2z5w8eQZWF3ICzyDECc/JE6CT0XlqIOUKLqAxizW+gpWWdC8CF9RmVFV/pQnO2jpLDuStXqoiYTJgBLlgPdngO4pSifz0ZvDNs70V+y9AVECTsXJYUNUKLqAkAS/QidLfXEaMI1Dn0rQYzR/fpRX6h6ul2jcFdyy1fdwx3LJLK9Sm+vlgMNywE6MwO7EXw+azEJyQUfSD58Z0cLy/cg+sNqZQUNUEKeLMnxa2SJDkio3UJK6uIaB0WlD43aXcmtrUSkqubK5faMvWc50HAcYAmFbyLn4TQyKamTnThInyrICFZs9oGz7x29WtREnFGTACWUC4hjDEIyxqDFMcS6v2TcAMq7RD0Q1rcyanclt3fCxBd3cBq3cu09y4FIOUAkcQJvNLIuCv2mLnU6GYBLEjtaMKZwBG/UhA7TK0AJkqo6AhgpHnck3JLcXKjiBlDaPVy6ZhG1PwjF4a7kVg/AM0iMA7e87D3LgbpwYFe5dOeYh0YkLOpY1lmiswxWUqdbXYkdivT13xrp3J41vacHKGF/PXXCYR/dIw7wOMJXoyQAlPKJGxBEHxqHu5IXP1i+s4y3ZDnQLTlAiDes14R2a0RC6qR+LIXx8TShoCHwTPJWaQDJezUD1nNCCHxTTSgpAEViJkK8n/3ycbkrufGFvofHB7p3S5YD3ZID6OcwcGzaYLVn6atLnX5VC+gh0ZNGuTykTs7dQiyRz5G6V9MAJUkBKK8UadlUHxqnu5Jb98JdDHUH/LBkOdDtOABoAp6AaCMRLji4+rAsNpU63epPJCkGKL6WYQhggUdIwV67hfwEKEkSQGm30oeio/WiuN2V3MrFeZ73U4+YrG71sfcsB3xxgGU7y2M9UruvDCJOjIRHxCT0duxlJ5JSWPqNBL0gIfBQBaDfZJIx3S1kEqAkaQCFh8Q/XeFxamkS7kpu75HtmyzhG6X/udXR3rMccOUABiNmfwxIjUDsgFJSJ1GUoiL0gH+XWz7Z716LSB/2COBaAUrqAaBKH4ozv/PE0iTcldz4TvBk1AuWLAe6FQfQJ+KyhHRVb0L6+bGUjjhOOAqp09kmlqcECvlICOEVAs+5WyiKI4C9ApTUA0DhidKH/kVjUFLuSlqRXV/ZZMCkaclyoFtxAGf5RvD5ROrEXxIHfs4pipPQY7qFwNN3CxHYI+ojgN0ClNQLQOGv0oeiH07SXcnt3dpAym5csfcamgNs02TpXk/XEZaTutTJCZlJkB4C7yCX3UJ+Lf1+6qwHKCHWAO8AIK0HnSf1ugRgqecRGkif9Sy/Hry3ZXZjDrDdkcFLwJB6ERZX/TjhJOvBUh59nwo+Um23UFz1IoAI1nwAlLijTn1kXOXq+SKNYxgDQKttStCfieO7DaQcB1dtnrFxgBB1WD1ZOidNSJ2cG86gBUCTdF3BxQifUo7HwML/iASwi5NmgiyPpTMAys6qBUKIQxOsh3JXelIacHR9aILVqMRrhQc2kHKSXLdlBeYAwZGXy0PKAmcS8EGW6LrUCZgmQeoIYIAKn1IOaFOR5i+QIMZxyUmT0oFStmmAkijq6HRXUvrQU6PI3GceTKAAqA2k7JNxNnl9OMARuBzTkdTBaaqVGIcwEiUldeJLin7XeQSwW7tvk1LgnqqyCV0VgHJlMvEKUBJ1ddzcldgg4OUfGnX5en6KB3HqnfXy7HfLgcAcwK+S2Z5OmxQhdeKWxODEYBS31KnvFkLidB4B7NZuwJYQeJwwGUcIPLcyuafAQ38feoASDpTbzevhgPe93JV4L+yXd/MPDViU0WNMcuhhLVkONDQH+sqjiW9OsJZK6sRIE7e+Vd8tVOsIYDcWxB0Cz61MNwBV6QBOABRD13VCCOoXlmq5K7n5h4Yts9bzNpByLQ7Z3xuCA1fIJTSDJG7CCV6XOtG5xUF4E7jtFsJAEoT0EHhB8/BTbjUAJR+2NrJ/n+DQpgFKvMo3ja6k9KEne2UU8X0bSDlihtrsoucAPp9IMjhNx02oCTDU4NsXl9SJvkwdAYxKIordQoovSLIY2dj2GRfwq7JqAahKh/cAke1p6wNCiM3VD4ZX1BrvSHWGSZuIQ5CUPtQGUjZ8iTZZfTiQlM8nUieBP3APYgCiV4yaAGR2CLFTKI7dQqq+X5ch8K5RN2K6mgKoKh43p/elzvDnhoZA5a5EnFJTqTpJfagNpKzerr02JAcI1IAxJS5pkEYTak5JnVGdAqmYyWDmCGCkTCQwBhzSZ9xWWxUC70eqIjFc/QIoVagVoESvpu6uhPuaH0pKH4oBqZ674fzwxKZdzTiAryPLUdxW4iBAjEAkADQO6lFKnUhL+hHAWMnRdyJRJ0WXS9UHEmkcFARAVT28ApSo37m6uSvpv9f6rvShJ9VKGPB3+k/SXiEBq2ofWx05cKfUfZku3fzwCKmTrYjEEeW4jahI3y2EdAJAo5esByHBoQtlEjIJgee3jmEAlLKo3+lCCLZC8uG7chPzclfyW0dUBXHpQ20gZb9vw6ZPjAMHytmda5TEnmld6uSgtyhI3y0EMOPYjfGj3sTkUysEXtA6hgVQVa4eoIQdTVjQ0UWzZTYsAciPxuQfyvZNJNCo+lDYttrnLQcqHGDQY3XFqBMlMeDVccJRSJ3sCnLbLRSlKiCK9gPkHJGMk3kU/piqTlEBqMqPACXsqQeUnoowQAlHQxP2MOr98ujnkZwtWQ40FAc4voJgHWq/d9jKIXVeKS3TXMNG7qFe7EfH8GS6WyhsG8I+zw4ldiqhi40K4KMGUOWuhFqFYDFRBijZW+qDUQ1ERTaQclSctPlExoFtJSiNjShHJE0ldTLgwxBHAHNeOsvLILuFwpQdxbPslafu7J2PgqIEUFYdhCdU7kpDYghQwq6hKPWhNpByFL3I5hEZB9BXPS0/ypgQNHP0UvpxwkGlTqQ15xHAnAoZh2EraFv9PMdhcSyRL/TzkEfaqADUy12JPhBlgBLKiVIfagMpe3QMe7s+HPi+lD6RQsMQUifLQIw5QY8TxkWl1hHAYepYz2eRxABRwDQMRQWgtdyVogxQEqU+lIPk0INashyoOwfQK6L3/F2ImiAt6lJnEGd1fbeQ6RHAIapct0dZxrOcDxMCLwoA9eOuFFWAEqUPDRNDlX3+6GltIOW6dWFbsM4BBjTb/IIujdlBxJIqiNTJUlHfLYS1Gid4TrjsqcRkEzYEXlgArRVdyY33UQUowQjIdlrOlwpCNpByEK7ZZ2LhgBqIQWZzgIC960hTuD35kTqdRwCr3UJh9a+xMCmGTHFpYrLAxcmvzypO5OOkKoCr34jsptGVvJodNkCJ0ocScs/Uj3MvIYT6qLZH5Sni1U57fzXjADOz6mTqWg3U8KUkwjyR5v2SkjpxJyKKkinpRwDXe7eQaZ3jSsf58pwzj7N9Nen/cCHE5alMFv9c9KeuH/k7W0hJ70XKXYlg0QBZGAoSoESVp/ShbKxwEn2WNtyQSmdQ5bi2l/updPa/8CbAJOIss1v8v7pIF0m9DCSPvVLp7MnlUsFzy2IqnVlaLhUJZYbrx12aA/JEqYjfUkpCJvVm0GHg4UNeGJ8AgVq0n5SaWK6jLiAykjpbvtazPfl3tnlinea4kpHSX1K198RUJntxuVgY2DpoSH7zvY7JfWkoCwYhBsgr39+bS1Q3IeY9Ol0sfOZf+SXzXsqlMtmF5WKBdzS18mPnH0CafoBXxC7SrUj7OdBXJOlLZAQV8soAACAASURBVD/AZY1AKkSnNyH0odTnW0KIv8oVzLhUJntmuVjo23/Irm2b7f2NlnW26BSwVZvbly0Ri9+YK/LLllTa/tas2/PLF7+XE6n0AlEuoVvtZIhJDbpZGgug0bywEals00XlQsfuvdbeoG2jnQ5sGbT7KJHr2yr6bTCo8qGYj2Uno7O9+8z9HQvm/J0lN4QTMj6VjwshfurDeISRhwHJlUPEkGKqEQP2BKnTxLfwCSHEZPkcRgBLnRwg4AgxO4kej5vWCJFKTxXl0sZbHnii2GHMhK53asKwpR/ME89MmSBeu3eq0EDl3xKk8JLY1ceEaVIkaZgIrpXXW4UQRKLCUl6LLpJpf5zKZM9LZ5vWH/qts3LbHj1eNPettnhaOVvaPHvy2EofT2WbHi0XOn4g9fIrJ+zm/1kADfcC6VEsV8YwO+906sQWNSubZvvqzCl0tEKhbXlKiDJb7Ez0SEidPxRCcIQvktIpMsq5V7HkSXpAluNA0I9Sb/xMLblzAJcceMQ578duvMshHcPHXdXEhBiUdFCRUf53F0IQN4BlbxxEP/melEjJ/zwhxO/lErxaeagmBm571LjKZOEHOJ2ZIo3PvuKM4pL5Ly0vl4qH9TRp1AKo842b/z80nW3+e8ua66y7z/m35PwCp14MS6CHLztJzHt0Grc5FgGw89pXzAjGSo/UiXX8Jj0vx3d2C5HXMTI/lugMIBNJxJHVavnvyxxMt8OJP68ASVQceOr688WzN7PKrsRFjTpAjFs1mUBxiWMTBDucmEhfcknYmso235xKp/bf9sgzcruchhdcNDTrl2M6JfDOzQD08R5BScZj7BEMk40YlcpkZ6w/ZNiah1/5aNNaG4c7/DGbaxGb7/NNscnuo8Qb99+yXblUOlSUSyy79BMOmeyIyAPKsi96X7m9z8lX1AI4hbP8ZIdNUepHiQN5vzyTx/mM/X9VDkzKNPc6YIcxEzI7nIBjQ3S04df2Edne/cS7T9/HcR5rBTQa+qkQ5zD9TQjxiATRs+VKBDDtUt2kss13ZJuaDjr8qtlNWx7AGYLRkVJpLXzqXrxL5veU5bwFUP99BA36Q1secHzuoEvuTgN+UVHvtTcQW+x7bPqdp+9dq/2zTw7TQBSpE+Ak2hER3JE8cbTXCWsuei50qSju8QMlpiQ6Vb4DpJbMOACfL93zzGsz2x0Jq6OnDbbZTfTdYJCYP/sujEdJAQpGJY42odPSV2gnrluvoYfPNOW+ediVs5vWlUaiqFvdf8gugtXWhy8/AYhi8Oz2KyG7hPfXSwal0pn/br7vt5r3OffmnL9HzVOjK7vjpO2KhY6OmeVC+z1CiF9LfSU7VTgGQyes/SzTj5X+n7ihcGon7lCW/HOACfLZqJftXtV47Krx4oW/YccTX01YKsOIiDoH9zp04Tvu9dMbxOCDwNR4aeY5h3a88+TMtnKpuHEVVVW8lYgodwugPhiJj1vrJlsPOeyKf2fCKNZNisRif+epjKlKiDgMGldrz7FywC8PB2a2H74hQ9Td4CKZao/Zr7U4kMo2PbL+1sN2OfyKR9jlkwjddcYehQ9ffmJOudCBQSlpws3pDxt8ZY/0YZNxDIifkEL/euLWbSv+9wExSeNH7BibFNZxN8aqNVzW4zNNua0Omnh37OBJy1lGbXVo5eRiBjJHUUBY/VnCvyl1Wu1CCJyn8RtF6nQu6+Vj9mLIgVG4ou100kWJgSf1ojzKrbhLGVY0wmQDss29iwdecneEWVbPCuFj51MvRY2Aqim4W0P1YhL51QKoGZtbRSp94fbf+kkoNxazor5IteePrxF91t2QIMUEVcbRnb3tRA/6J9ZhIcQBElzZGWIpLAdS6clb7H9cPoxHRZAqUB5uUhVf0yAZBH+mNZXJ/ph+HfeKyllFVAWyb9Ofuy1ZADV7deOzzb1640ycNA05YizS0FEy0g2HgrHvGd++V5OuSw8vbxCO8l/5xpmx6bar8W/Hky9uovyEJbIR7DCqR7+GFzuefHE2lcnSt7stmQBoZXuiVDZ37uHqDFtlgibssNHJ+b/6TS9D7Rl3XlXaxK/ppuZjtjrklGzSszQN/eqxZ4vmfusQHASnPCRRor9bip4Do7ItfQp+LNBsgrj3/CAxX1atPOVSfsIh4UaxAcRPv2ZHFZ8oCMkbAE+4zVFUvSsPEwAF9LD+8oFz6OG4R89hjysO3wpYyZh76oOCWH3nqv+vK4/Jj324Xh+erRcNKnW0D8GPzQ/9397R2ec22e3QXCqdjes8bz/N6rFpU9mmozfb62hfuk+25PoB3FrMk+X762i1Mq3yeyqdOZK97VWSrPITO4vUXvhVfvR5g11dxBSok+7XZ23dk5t2GHYOoOwFKJ+VAS8IhMHLJpoCfoaKAEZAlQ/f9V0HADBgyFW/z7PVpjWWrvWiimQyYOgIU15VAiqssznBkaIhwPu1e6firsTk5bVDKZrCVtNcyoWO4X63aQKgvJv7Ljii4t/IPvkwbkCUn8pkdygXEUTjp3Kp2M8vGNJmrOiqzbuefnmoSWTgDvvnlsx7SRfA4m94hCWYSKCqOMBQRQ/CfYY92LjNgBR8V0Q6GOKUOPkfwFXA6VzO87vXh9iU9aLWdb5ccScyLp9OhpP0M1MvFDN+uLd4/GrcNIOTivjjkPSDZ2ifdOWAxmfX350333/u4cpkCYhsd9Q48Z+p4Y5Wony5pHUWFcf/nWGkfORcCYbz+afi/bmzBD6jg4YfLh6/ykST510I6oOmPmuyjO+WZCpVKVcDBXDOxnJfJ0B1TSHEEdJVQf0GEPLiAFGnJMV9gJfnnMQRtPWiEel0xpRPlTqyzKGTbXf0eEFAhr+dMlRsMvzwlUKe+WmMHx2Vn3xt2i4OINn7IsAk12dNMeLsKZUoRWx+6NufcKKdVOkDzz0sWIn4Vf+oPJK4+vE4QDCgPbRZkZvU/vwdkyp9X6Wpde34/NMdaqVp1N9NJVDATnUyJEcFpOqqi+DcY38vH4B3UwmkuhTpBFD1PFc3Paj6PXE+prJNWT+djArS0ehkPAf4ESfR2dGIFRmAfEsNAcpYHR9RAoJx25kgkRjV5Mb/ajkMeGJo+dL2ewkMTfzvk+rW36vVk36tTwYVQHVs+6TdAKgpKZ6Zpm+0dH4AlF7AAAZMWbLrn0+0hpFGRazmNtZ0pmb0mAqIndIn6ejE7NXV81Xfue+7k2t1Cvw129zbl5Id/dCyRfO7OhqSCaQAlN9Z2vu1ZK45cEsCQliKhwO6Dt+oBMBDn1iRSBUY8K4PuHh65XeMTKT1Sb4f8Jl/V3L6oynRDl3N4QRQJgv6u+KDSb7kkcpkOY+pW5IpgNI4BaBcCX6gPkiWupIPoGPJrZbdpEf3CZLQMdCR6vu5kWy5z3NInyz7CRLslETrAqAdn3+6zHcn2545o5Ocney+80eJ9qWf+Opk5PTpwtfQE5n3dlUBezXmACsFU3K+V/1/NVnOuuw7lSDaWxruL1eTrWkdQqarNJZ6mxI6XwWOjInFbz7XNYkAnkoq9+uZkGnKvWBah0ZLZwqgSJUElwX4sJYDZrwABYZokhXAcR/QVMQSHkAEcCHyYAmv0pM3b5FnVN7kqyz1/M53PU+ZVSKXJb46mbaUo3Ys33RJZeSkzqWe6og+W2De231mvLonT2eb3/fznlnK6u8V67sOHCztCUhCnmqZX4vHyz6YJ6hHrXQR/V7pS6aTBoBJe1RbeA7jGQTwE21/y4O+IwDZXD+0d2bEs4W25XrYRrMHGySVKYDCbCRCSCnvFIACcNwD5CD+ByQrM5xclrPEZzkOMAKcSJ1KE61b5vkdoORDPiz9UQdg7a8XzW379GN81YwIYNT1RPxPnE+ddH2Zft/re8KSiVc1evT9UqH9vo9efcbYf4hjPXRS//OukMYgJFEFOHpar+/v/ufBIvXw+j3q++mm5pdN9bOVCUFrM23DSArR5gHb7yXen/uQeO/Zh0R+qa7Rq17rd56+D/BUmFI9cQP+ampdBgwBRYCPxoIITD/cR+GNn4/6HQ0y4EcvUhITEihpeI705KEAFEAlHVcAFOJ59Sy+IdznXj2k0LmfzHsxxwxsMhh08KQhbn6BSCW69NLZZO+/dHL0ROVioR7t965Yz/pl1oI595xo+p69mg6wvDZzSkUPznt29gev5yj3g+cfIcpWYu+41NF+31uz7th8t7GTQm1fpS+r/oxxVE0mXm1V99Ebr1j8PjaGxNqsyo7qarJdBsDTJUrTsgFbJYU6n6n2mzMt//tN75ZH0HuU/UmUsRJnjB8hWMqb0sxzRhYWzLmHiEwri7KmGdh0JhyovOcDLppmDHrVMmXS41BBfVlfLT1S68OXsdmv4r3iNW6qZRHkN8b2s6NvfbvLyBkkE/UMkwDSqGmbiYX64vSr3+WUU5VHd7uaLOGRBIO80GrPVPvNjYd+07vlEfQeZd/13jMPdB19EDQjnqOT7TrW3M2D9Avm3MNK4T9yIlMeDs5rmGrZZzv7+NTZV/zAeBlfjWlIZKZAQj5P/+kCymWllmRfnytS6Xceu+KMSPo2KzTTNtOvX7nn2ny5WODomW5LJgDabRsXYcWnv37/zU0sOcKSn05GWS/gU5dKE+eTeJ9IDM856oAoW8+tro7qdOt/J3z+0cKs0mEm1RLKo1zNjhB10ai/cA1hByF9BfUbNonpolw6a/7jM5pMdaFRVQw3vkK+jXi25tJEVIVHmI/JEj7C4rpvVpxt3X/ILsMOm/xvU71x6MayHLp9zDb5QvtyjGicH4PUCYpzuNxBUlpBCc+pi4vk+e4cRjc7dOGrbwZTeq3zpW9+Y8pLLSY677BsQhL78zc2KnSsWMZZVsoGEDZb5/Oo4L7YJtX5K26GFVsDfXutjbfe5ejrn0vkjDQEEXnaAjoLZQtx1rlb/J8Iw7oFJ2pVslR6a9miBSdhVW8NeQpnraLU77MmntCx5J1XPxLl0j7SyMZs/S0hxBYyjXLyJ7jychmdnhMXTxBC9BdCfCg/Kkt7rc2BWaVCx1ELHpux9pDDvxf7Cm3694YVVyz58PVyqcjR03G58wCgmwkhNtSav2uXobZU+k/bpx8fv/SDt3OmRi8tH19fmTDuHrt7vtDRNkeUSvGc2OerRuESWwA15x+dcNP5j9+zzUa7fD3DCZpxEtvhXrjziowolzj7iLKRPPGl1Q1JT8jjPa6UR9YStZ7ja1lZALTnCSGOloaJhQnr1+JkT1x5oyIZVy4Vb2n79OOvxw0oONq/+8z9y8vFDs5u4R0iJSIZRnVa5WCZ78UyXyZZjGVIfjM1Jn4gyuXnF78x91gMX5yeGRcxYSz76J03yoU8x3LHNWHEVf1V8rUAugpLqt6YXhbiwPmP3rXB4K+flInySGO9VFxBpEWWjq77yAGibExQulBC3PXGS6BLmuiUOh+UOtMHZHAWYomeK4Q4WAjRRwKx3RrayXQmJI5+RrpHekcv9xNRLj+++I25Y5Yuml/Zvhjlu0YKe/Ty71XcnUS5fKAQ4l8S2H4p1TMs5XnHEKoZv0BDv2AyvUq+b87RQrWDxOdcNtP+S4UQYMHUyrntqVSXW5KsQ+gLbQY8P5n34opysYNzvBAKuj1ZHaj/V9iabmp+bI0vbTponwtu7WVqdTQtRnNnocM/KXVETssseiMkU472OEwIMVqezHmREOJGl7LQ2+4n0zFg2BaK8enPQog7JQC7PNajbwFSGFOQyHRiglKDe0QqnZmx1qBteo2c/HAkhwkCJDPG7dUJJKXiSIcPJO9E7QNGEiUyGZZ5U90ovtb4TZMvZ71fIoRAt1qUDVS+1Fw50I2+QPspCz9s+hll3bDJriM7Rpx7YyRnJaHznHn2ofm2zz5+s9TRzvHb4a2x+hur43crgfpnPudZ39K+dMmoN/5109prbTIkHZVOlLihT157DjViEPxASiMAKZ2bTq4GNlJpLyEEZ8ZfL4S4XQixrZSgvi0HAvuLMTpBJQmwPIekhSUfPSpbSZBOdpLpCNoSiUuLzK+RL4AF/R+9oAJR+K5L/PNEufzPtk8X7/HitCvXLhU60ujAg0ijAOdzt14m7p9wTGH54vdfKZeKHGHsBBL+x0AIod/+kxDiFwaqlx1lP/i1BEus7Uyu5Kf6AHnSf/jwOxKu0qHT6ZRjMs/c9dl7b57w6r1TSr3WXDdLm4MQbX7q2nMqqyl0nnLZrvpwkCwb7hkrgYZ7JUiCJ355/2/ndzzpopwKIuE3S1xIHpp4fGH5/z5oLxcLYzXLJMAJqCmi87GH8C6PQYUEcoGULF6WadGJAqBu1E8Gb0GCRUJl+cr5toD2vUII4y2sbpk3+D10cPDmM40/8NuNANjxqUz2zJbW9TJDR/+0hWDCJu8bT4p5s+8Sc2+9rK1tyUfFcrHAuVZMYs5VhSqX9wvAEaSHVcYMaRR0S4+0yvumLUyK6Dppkw6aKl91RcI8ROrGuYde3a3dtJl6nthnvYGFnU6+OEt4PpM2I3G+fu9U8fKM/8tXXJXKJVQjv1cV6ElXC6Dh3+aoVCZ7VblY2JCoO1seOEYMGKpWYd6ZMzvPn30Xnaxt0YuPIQkAikiEzhmaAaX8PJFCIYC7cyOyvOG4AKQs39B5viSB9I4aA2tdIQSWYMCUM8oZsAxGwBTpxAuEHUU3/L/0+fMlT/5PCIGeGAmeWA9KCvNqBKByXSqdObBcKvbtvc6G+c33PiaH4QX60vYjxPvPdWZBsI03H7o9v3zxu7lUOrOsXCoChN+vApx6mfQBwIu60S+QlFluc5wOxGQHcO4phHhGCIHqhnQmhG4UKZf0ADTRz3Sp25kH4DohlckeXS4W+qw1aJv8wB33r7SZdgOoKggLk8U7T93bxvbMVCbLDiOc5GkH/ZV+7JS4nWXZ/1djDoxJNzUDVuVsS5/8oN1HlXcYM6HyOeDi6eVdx07q+r//NrutkGCGpEDnRSflRYxOwIy0fH6lLTm9nlH3Maf+Qz7HTiZ0YyaTJkcnnykHJ2W+JwfCMJVxN72iU8RggzUaXZwiwMmEWMsCBBDfAYdZTX3WfDqVyWKUK3PlfwnG/K7Wv7xj06A4nYjcWQ6rBACe1QHg95h8n48LIb7emcToL3pvLO/0PYw4gFqtCUPPmDoRtrKrzbIelTZnW3rPkfkxsas2q+cpC0OnqS5XPWevqyEHmLHpRNNxUM4091IO8OWm3mu8ITsZxhulwDdhEYOWDsjyGkA7zeQhLQ1Ays4TnlVAqv1c9euWUgJ+RT7/ppRu0bl2J8IyzXL1LSHE1gEqDijwDhSA+s0CAIX/piCq54+hkLrz/MfScKj/Xus7et4XpXuUfnyG29LdKy/AttpE7/Uc9+EZdeejVlPV0tvfLAeqcgCpU5cyqiaWaXmmspySHZElnF9iACggRZJhKeiHUA0gAS+QdXheukdhuW5kQtrEFQhpHInOLynwBADCAih5mIAoK4WjNEMQy/mTtUmAd2FCACaTOACqO9KbPKvSIHVSb1NJXT2nrmryIA/Vfj/9X+Vjr5YDFQ6gC5oWghenSp0kOjyTJbmzKDo0S0A6M1e/kgVloifFMMBuJ/JhCYfxA11qoxBh2uARgToq7g0BKqaDJ+0MCqAUzfPq4wWi7H5CD628KFh2Y2VXhBoC/SUTApb2asRSnSU7fsFBT74ENFWdg7YdHqo86PuopFjWW7IcCMQB1Zn8Apde2JHS5QgjT9D9+UigSqeGZBqkPpTNvnz8FTFy4Qb1T2k5DiLt6W0M831jqa5YrPlW+s1Pgaeugw4KIpSt3rsCESQ7RRiK2IKrVCUYnapJmaiJ8JJAP8rmCCexOQDDHxNIUHdFVjyoLVS9w7YdNQB1ZueVlUCdb8z+b8QBfTbGRSlMRzpAGkRwOcIvNChhXEI3ykABSIPu5cObgGUnFn8kHz58557yOQxaRz/PMTEAerQp6LKV8gAQJKVHNLVFWBBBFQOfAUv6AhMQS3P04wAeng/oa01oZ7mkx2lePdMkJzOc539kkkmVNAroowBQ8gL02VnFqseS5UAgDiDlqQ7JVZdCgmTIIEJKYF98GDBmWa4DKfrCoEBKO5A+AQmkUaRSpFOkVKTVoBJzLf7QBowUgMfVQohQkdZlYbSDiQAne95XECld1RtQxxL9rhACf1DciXBZor63CSGGqIQ+rizpeVfUka2agD2nXLJ8D0P0JSTG38l28z1sX6U+9Cn4aC3yYd7OavwsEgyDhk7ErMwVSSQMEY0JVyOc59cPk5HUqeIPitGBujE4qy0lTYpDL4q+7t9SykJviv4UPWoQHa5bmbqLEgFUoiI2NgBIa0SUYbPUfyNtopu9SYuuFaYI9rPzvgDSKN3NUBGghwU8owBQ2shkQR+Iiqdh+Gaf7WYcUP53uAOh54pqJsZ383W5c8mPe4oX+zBmHC/zZLCzZz6IhOTM3+ljikUfy34YkA7rouSso/4/S+s/6jcCfkfFwpKaUzeRyPkg1UZBAOb/pIsWEylL+iCuWs66IMEzebCSiJKIOoURjOApliwHfHMAgGM2Z1kUJa0jhPiv1DMpnVjY/FluKyBlqYkPaxRASr2cPqYYUFiCc9+U0B8idQV1UapWDrpUpLow7cXAg0EHiQtn+GuEEBi4JgshCDEYNuYoYQvJFykRS/vaHpsFqrXT6zfULUyecXhWsBIDRJ2Bnr3qYu9bDqzEASRPBmfUhM7uUbnHe3iEmQOkuE/h0K2AVAV2jqIYJFDdx5QtiuyGQmJ1I5bCRBpiGQxAxUEYfIJOcrwHfHXxAgDgCS33Ja2SBDiGjxjYghKTDX0IS7sOxKhFcNuCN7hLBTUwokdGpxoH8f5QZbGct2Q54JsDyhofxijhVSidE/UAM3xYY4KzDJZ1CkhZhhIxKEogZfATgf9auSxFAkJ3ig5VSUJIcEjaSHW1Aw84W2D2P8ANwPnVp2J8URGUWP5ihOF0ADfCFzjI8Su8A9zXqB/bK71oN6kyYIvxl70SVbmPsYtoXXER0jMTQBhjZVx1s/l2Aw5glcW1Iw5CIkHfyiCjo0ZNDGJC7qFzU0AKsEVJlIFnAGoD9ppTDjFSAaan5FlQUZan54V+jh08umSn/+78ztIZQw7RnTgE8DIhxHrORI7/CWkHgOBJYUpryU0L8MBkcqQO7P+Hf/r+/1rlsSKgbhgo4yRcmqxbU5wc7sF5szyMe0cGR0UgxWFNjoNYHiogRReHdT1qIKXe6BJVRCkmBZbFcfmYIsGz9CZqUy0CoHBJAqBw1SKknB93Mjwx/lKrEPk7umGMjwD7VwyfIRlS/c/kZMpyn/bVItITNyBuUm5NcUzycdfd5l9nDqBIVyHK4qzKWVKaQEKKiwBSlnsfSaMGQKrr/MKUqxtG1BlPGI/YHw6YRu1jin4aaZdyvYi2YQgishMWcPSRuFL5peNkG2o5/KsQg2H2tKPuQO2B+qPWkp6IUkwMSRB6UPSh1q0pCW73oDLUPuMkmsQRDoANEq/psjRIvbAEKyAFXNABhgFSlpJYq72iKHG6H3vw2YvPkhOACOtjyg6mmz0aP1Dmj36ZyYKzpjAYBSWMc+zO+W2VDHAjYguksrRXSVrzJ/jFygeJ2Uu/O0DyMi79srOS1q3JyRH7vxEHcGdi0MdhSHKrADozBj4GJrb8xUkAKVZodkkpIFVGINNyMVZRX4JnmIAUUaEANKJEwdcgPqZqSek0bKCWIEAwQMYSmkkiqHXb2X4itqM7dctPOcg7Le3OPPz8zwRKEGYmVLcdW4RLRKoPuofeT11UWnTOvGvr1qQ4Yq9GHMCQFGZ/tVEhWiJcmxisROhxC0KhJY3kK/pABaQYVyZq1nSvAtDRKReloFZg4pUSlR+dIWBq6mOK5IkEqggPA7agsqTHKo3RL+p9/agKMArpBkXKYF88IIeEHQfh54rETnv1kIR/lzuk4ijTK0+W79TFujV5ccjed+UAvoZ8kiSc7AnMy8DB+T4JAkgBT0BUAambsUV3UcINJwpipw4bF/AYAEyVjynLWZ0AMoASHShnqmP9B8CQZDmOw8T4oufn5ztqB+VYj6SODhJQ5djjOAke4ErFpErgZo7Jxhj4jTgL9chb+UY7pX+P5Pa25UCn9KkfHJcUT5CsAAa2f3o5q8dRF8BBASnLe6RTBaQqihKBUZzgFkVdWLrqPqaAowpUjHsQVndWBEh+eC6gd0WNEFfwE71NONZTJlIomxX8Wtr1vPx+Z6mOqoDyUe/ggA+Q1oOsW1M9uN6Ny1TRmRSIJNkUjDsEIEEyi9vfz9kugBQDE/pRgBTDBgMYy3YSgOX0MUW3CaAioTKpIA0lqQOEPxjCqAMnbNayyjv5GcX/HEKIHhJpNA5XNJM6Kh20dWsy4ZZNU5G+GLRBj0sIy0KAG4kPEAsT0CNoPQh68Y4ELgwX6DyDRk8PUgeivWPd5h3wYQmPhZrlO078AG0ShKSrABwH+3oQ/qL4v+JShCQOoNaDlFtTnCqTerTLlhkTB+iwSRqSnM3A+otRCeDgeN+kSLkoET0IfSd6QPRvuAdFaeV2aw+Szn0SNPGLZLsjWytZyus+pvh4sq2UpX8c7l/kiX4W8MbSzorgr24VTuDerrIeqHTwAaVOXJNYEejNU25NeCdYshyoyQEV4q5mwhgT4NaE7o+lLMeFxE0EEwYscVHSvQFYOiogRbXALic3956g9QMIURcADhiTOBedaEv8v68jU/Swuo8p9QHsooq3iZ4R6RfJk2DIkPLXrccSHj0o6gNFSKBIoqxQkq4Pbk2sSKxbk3ob9urJAQwHLKHrTSzhkILQRbKkjIOUixJL5R9XKQAgJVAJ6RSQhllOY83G2gxQAgiHaGUTYo64n9XI6WMa9nhnAAmwclralWM9+uGkiV1ObEXVifeAtwZL+ySX9NatSX8L9ntVDihDUhSBkKsWZPgj6gSAxmQvuGGWxS+vAQAAHJlJREFUlWSAkIqiZOqihLeAAlKs0wC7HyBl8wABSGgPYf44R0onBiogdrp+s8Z3p4+p3+Od2ceOld3L0o5nAoYck80DNapq/DPvBh65BTaB3xj3mFiRUpMyrrFKoU7Wrcn4Na6+Ceko9TIkuXEdoGLAEIwEyTQsIb0ACkiBQVyUAFLll6mAtJpuDjUEkhN85aA8L90u4eAA0KBuO17HO3u1EUCnvGqWdvxRsYaHPRTOzztjFYTuuRrhJxrmHVbL2+03gNq6Nblxxt5bhQNE5UG/1kgUxdHJGEnYEQQYsywNK72gr1RAirsR0fIVkFLWN7WtnJxWWk16YWJg6c4kEZaog368M3pN3ceU/AFE7qP3rAXYqFKUY33Yupk8jxERY1ktQlJlYmLXEH67cZN1a4qbwz0kf8AT40ajUZijk5GkVOAKpJcoCSDlnCaAGSD9g7RgI3HeI4TAPakWIRWTfvNaCX3+7jzeGWMZnhaUhYHMxJqPXy7pk9gRhBoD53nctkyIJT176JkM2FNv0h6TfL3SWLcmL87Y+10cUNvYum400JcgRyfrLkq1QqcFbSrSLGHkWBIDNiwvMUyZDmj2fBNsOE5iKY+LFPUDpPz4mHIkNAavuGm09Ijwo1umTkRzoj1Mkl4qiyjqbt2aouBiD89DHfER9ojjuNjk5+hk9o27uShFVTdcrr4rTyBFCsJvkp0rHCwHUGFN5mjmarpbfB2RXg+PqlIu+WDBpi4APF4A7L7Sj3eu5WOK9E97TA1uLlUwusXxIHcbpVw1EZOjMgzGGf5OuTXVivK/ag3tndWGAwwWJNFGJUCHfft83DwG8NfkHHFcj6qd1RO0fbhAYS1nDz/SHBGbCPqhEzozjEbwEl2d17KU+JvkYyqt6mWYfN9BGmXQY7pFj4eXHJqHLyp19fIxxbGeyPtxEVI8/panhCiA94LOlsmMSPbVJq6gxSi3prhPcAhaP/tcA3CApVCjd5D1pcRBEGD96GSMCyxVOfc8aokJnSKADMgAzvCI8qoRrmEKSB9zGDwY8IBGXKd5YnzjyBEs7V4Hyul1dx7vrPuYAmwAU1yO7GwsQBJHOg5LnLnEkh61SBySonVrCvuGevjzGJKSOOIjLBvxT9SPTtZdlKIcOOxQ4jgSwJodUliJWRb7IYAUVxikPK78j4sWbkIYuaImAjoDSCaWdrey0R3rxzu/IAOu4AsbB9HnotSzsqSPayJFWkYlwnu0ZDmwCgeSPOJjlcJ93kCKU6HPAAz2TId1UVJVAKDPkzFLATosvhylEYZwuUESBUjRSWKgiZLQy6JLJH9AKaxqgGWw8jFVRjI2BLC1NAppUbWdkH3wOkpClcN59KhZOJ8+yiU9rmLw2EZrivKN9ZC8kj7iIwzbkDRRObCkZqBE0aE5lI1dUGxrJcwdQGSyBPbTDpbtDEA+LPGr+Yma5ksAkkfkUjuOLbCEHUQCJ8gyqgd4zgTAWUlhdittI/mgq2JM22ySjiU975ElfZTSPgHIcQtjErdkObASBwjeoB/tsNKPDfIPbk3oOomixJIt7NHJDC6c7QEHdGi/jkmHBvuw2LMExLikdiphvQ8KpARCRmepLO1xvSKW8BikkO6OkoYl9KxhjndGOuSYkjiJcIX0E+rutk00SNnWrSkI11aTZ5Dqkj7iww9rOV8eaQhrux5FSUl2RJs3JaRY9H2AJj6cPBulpOKsB9IsBhkkI4ilJe5O6NWQSAFSPzFRWWLjiuRlaZfFRHJRjvW6pI/0iRSKNIpUygTE+U0sc9XurGqFo9K4olqCiH6jn+AxQR2DnnHlrIp1a3JyxP5f4QBL2Hoc8VGL/cpFCfD0OuRMhWLDSl5NBwiQXa5Fo78w4Jnqters/P0XMqqQE1yoqwJS9LnscKoFpAAZvDC1tDvrEuT/+2V5bs86fUxrHe9MetqaxHZMVV/UG0jMTFSoa8KQcmv6Y5hM7LM9jwMqMlM9jvjw4qbuolRrGVbt6GQMQRiEMAxxqB3RnsLo8Lzq63Yf0CQkGyDqRQApe+vZGqqAlC2jTmL7IhIrRrRae9qdz4b5nxB8lMvJqtXI6WOKvyuSvj4pnCTVDlEZ/qrVR/8NfStLegLC1OpL+nNu33FrQv/u9o7c0tt7qwEHAE4GCUDaCKRH4DF1UWKAsyxXRyfjeoSjNbuTcEnCNUlf/ifRTpbtLN9NjFKArQJSniF4CYOUrY4Eneb9IEFHaV024QHloW+lDqbk9DFVxztj2PmLaSYRp+PdowJCgg+j71duTQSNsWQ50MWBeh/xQUXonEgtgAVXv5IK0g76QaQ+dF84weMMjyqgHoS+z+9RGQApy06kJYAUqZlrHJZ2U54gdVGHWhsJ3PLjnfAukUh5r6iK2A2FxFoPYlsrqxFOJQi6pLduTfV4cw1eZj3OitdZgqSpnxWu/2byndidN8vlFUss9HFRRzwyqYdKA3AAGHuqGz6vuPswEbCspz1YxGljPQjXHepCgOOghCoAEEZXzSRHu/4t9+lH6WNqUj/eDb6oTFJ8D0JsWrBuTUE410OfqecRH8pFid0kfqUclrksCxmcSDenSelGHZ1cL9AB8LC0ByECMeNahqV9e3lGk9pSSr5+d0YFqYPzGVy+8K8Mqj/mCBO8PSDUErh0oaZA7RKVj6nM3uhCO5BCUfH4ORlAZU6/Q5K1h9ApjqzmV2VIcgvYESdrONhMuSj5WWpjGFCxOQlQjIFCt3Sro5NZAgeVMoK2G7coBhfRm/wSLkJInE5LO7zhsDuAlEFPjM8kgZR4BGFcgvD9dIt2j24S7wIFZmF8TP3ymvT0P/iJftTv5GDdmoJwvAc/w5IzqSM+dBcl/DxNCTAkeDF1xTiB4cXLfYky6nF0MgYr/CP9WMsx1uDMT7uqWdppE36NHIWhgJRdQ0nQTVIq1icqk3IJNE27aq0unMc7+/UxNamLWxr6FMt5lvV+dkhZtyY3bq7G9zjiI4mz4vXgD6ZuJezawfLJQORANaQWL+DUX2HSRydTJwwmfk64BGgBTdrGcyaW9r4akLK05rm4gRR1AnUkGLIfwo0LlYofIliyfrxzLR9TP3m7pcWghK8oErCflYObWxOrH/yT2Zv/UK++a+CiBt/KueZe7+daerMrjS29tC/pFZ9b2+29CDhAUGWiMn3CC05nskvVS1+jdR22H/LC6RBhgwHjooTeC32YiYsSqgWkSDog9SNsmwnA6CwhfdCjk/0OBtqHgcTUgIWLE8v1oJZ2gJQTNdVefoA0TqMM7436QvQZovM/VKPPINldJp8JckFyJeIUEyf9wM3HNEi+bs9wuoBSKZm4vSm3JoxKI7K5ZrU6Kg/+6q4rRo/9eZnPKedeXj736mmV7/y/zxEnlpt79aaccjqTfUcCrlt9Vrnnt/OvkoG9ERkHAIdx2Vzzdwv59gH9WtdpX3eDgc277Ne5it9066Hiw3fnic8/w54hxOyZt+cXvPGSOoIBayq7ebBEmhAdjUGEHgy3FhWCzetZIqMDDGxdfEqeg4OUFoaoL8F3cabHKFKNGAxnFvLtlbPcB39117avDT+AGKGizxqtYv0NB4m3X0ZgF2LRu/PgTUf7iuVNQqTahCi/KsGlWv78RtBjtUOGoyrC+BbyLtGRwl+W2MQKAEzRAUdJSJ9/zmSyHxeLhXXXWm+Dtq/ufmBL/w07BSlnn3n0n3/Nv/Pmy0H7jFu9Od6ZOrACISYAvCYiFR+c5aMgVkXo15HsERhqSc/HSQ8QMWSH3Qv7H31ydth+o0TfNXgl1en5J2aJB6ZNEQ9OmyrSmezCUrHAO6y6rdoCaHWeJvXruFQq/YvmXr1673/0Kdl9jxwjNtu69skeixbOEy88OUvcPOn8wuJF7zJQAVJ8LTtR1r32SJp0CvRLnGBZLawbgAVwDpOh4NiBw+weFeFPiUWY3UkYEJBodAIJkLRHBBkMf7rsx+LNF/9jMhg4boNBiuTIZIEHQhTEqEUHy0CEAFImrGrvRyatean0mVQ61W/4gcekjvruT+PsMzUrI/sIYMoheKgvWCkBpLizfWCSQZU0GALJhxUQOnq8H9xoTCbbdFW/1nWaTznn8uyeh4Lr/olxdfPk8/MP330Lkw1j6jteuVgA9eJMMvdbM9mmGcVCx+6jx/5cjDxxvNFM6Va1B+6cIm783Tltny7+qKNUKuLv2CmSrZyY2RwLK0ABOHrtuUfsBTi/JoR4WEqcHNEbB9HLMYhwbAVGKKzeUGUwbDjoy02n/ezq3HbDgm3MMhgMADfSIfyCJ+yUippYxiONAqS0j/IA0yBAmnSf8csL9M68LMCUqFHoM1E14CalJim/ear0uCkxibMhgkj96EgVob6YMPKEceLU84iCGJ7m3D9d/O4n3y7k29pfLpUKjKlV3pcF0PB8DprD0KZc8y391lpni59d8/ecicRZq6Blny3hhXc8PevvGGuYNZk9FeFbimUZ6RP3HL3zkYa+wNKVJTXLWQJXAKJzVAYxXpH6qBcxNQFvpLa4BwPRnwhEgRSMroxgIrg7xUkAKeoSjB1Y7QFStoSihzahpPuMSZ2qpUGCQ7oHTNFH0y9ZwSCZogIiBKBf4rgYtrIiBADQLOlZpYwZd+kNgtVblPTWy3PFL047JP/5p5++3d72OcbTlUDUAmiU3DbPa2g6nfn3RlsM6X3pLf/OmOhnzLMW4tpLxosZN1Y2qgCi+NQx+7PljeU9Pos6ITEgBQKcxG5kSY9ukuC9SRJqAgYXyvz14xwMy5Z8Mi/fvgJjwb4SxNjO6FQfxNl2lrhMTifLMH4s6znXvhqQJtln9Ik3Kj5gBMJhHzClL+LHykmggCn6Zt67KaGGok8Tc+FGNmyce/U0oewFppmYpkMwGXvoNvlPF3/0ZLHQsYf+nAVQnRvJfG9tyjXPHjBoy8FXzviv3/3lxjW8e+okcd1E8LJiJaUcJDsdFLn3bXmcAzuEkAhwm6rnmUxIZ5d8b8IfxMGjEdKiJwbDSXttVGxbvoz2s1MKj4B6Ec73LEt1IL3KZXWQdJ9h95XaqRQHb/AxxYODAC8s9/ExRaoETCkXzwkTup5NG98a+zNx7A+wScZHSKI/HT0c4yRxTbt0orEN4Pia0r1zzmSb/rXG2utu97u/Pd2Ua64YkmNp0OChu4gnHpwhPvnofVxrAE+CaUAYm9gphGTKeocOy/IVnVxYZb8sItCFgXTjKedeLg79th9ffn9lwfOtvrpb+pF//KWjVCywDEQnXC8COP4upagBUk+Kbg9pDJ0s7lQiyT6DF8P81144qlwusxKJqz+gKmGiRnpkAsM7gdUAEyh+nwQ34d1Ui5bfms5kf7v9bvutccZEcDReWmu9DcTGW2yTeeQff8G6i12g4vFiJdB4+e7MHT3k5ZOmP2tkMXU+HOT/H4z8SnHhW688Xix0EDgXSYfI8cTlRBGPxInrSb2JwfD8TiMOGXje76t6jURWTwwEE08/gvzilrb81JmVACCCMY1tohcLIdhh85s69ZmVlqt+GhIwLT6mLPH54CKFz+ptUjLldFKdJuRaep875ZF3m6JWgemFOL+ffewehVefe2KOWsqb7B5x5mH/D8aB1lQqfSHW9igMRqZV+O75V2Sw8sszjHAXQuIcLDtpI4AnTRmfbcr1H/fLOFRv7pxCX4ZrVCbbhFW3UUjFEUAXzWYJpLNfH3WqmYtSVI3Q+kww14fgFcErhGNdcLHjA3iin8dpnw+TCyDbmk5nfnTUKT9JFDxp1ncvuDIrxxOrOqPtd8HZYZ/UOTA+19LSC1elJAn3nwGDtiwKkcJIgrMz29oYqI1CDTMYGoUh8v2gXvl9JpMt16PP7DjikI50OpvEeUlebEfi5LhldpHtKicUBg9S6TOlUrFf0nyhogg/8Eaqv4wBlJnI6crP/15n4yimVFBa/bM6X7O55lMPOObUxGdMeH7WpL9khCjjjBw0aG2cr25UowyGOBsZJO9srnnU1487Pb32emxFT5a+Pf7iplKpgBRYe0dH/FXDlQ5/XaJQ4fLWtNHm2xSTXLrrTRx+4NG4Y7ErqtV0Ce8MagF4ssRAnMYHy4sAUBOvVpaV1T5Juph4tSXM/aFsz4zaR820Qsyaa/cfgGEiWic50wpUTzeKGb0RBkP1aib+q+0z7iznKJKNjjzlzLoZwNkaKmmECYACbER+mSZBE/DkO46GiNQYotgF4EYMWGYwE10Kabw+bnl3p3ujWtf7UnuSuk8nc4YfeEwunc7u47zfAP8fLmf0ulRFHwx1qYB3obbPuPOmgiXb7mwCKZ0ZsBvt1qu+cHPCJQk3v6DEZL/xFkMQSIaaACggyVnTWCsxWwKoRIBBl4YUyiH3+HV51Yj7XgCrt4G8vD56uu74fcTXdj+AoxiMia2ZBDdQxAvnxQcldKFySRY0iziea6jBEEcDQ+Rp+0wV5vUfaB55jrQP3HlDZTzhB4z3BQFowtAaa6/HLqtBJgCqfAVZRrONCfkVUAVQ1QeQ9AJQ0gOwtaYMlZfbtVu7W+VaendFyPHz0m67unPWBDwB1DASbJ9+XSrscD3HTwMM0zbKYDCsbiLJbJ/xZHMgnezosRME42ni6aPEYSeOW2XXEsCKpGpKm25VqYYRgAKAOLVCe0k9GqIQH0AR4ARkvQYmJQG41fRvanSTl5sU6gXOslqNfSl05DmUzBehL1208G1x65UTKsuNS276QhpVGfl64V9Ed/J6TyrbJK+RDwZ44ocvNFYNhiQbXqusOPqM3xUM4fAkNVKfaR2wyZa+99Aznt566VmBIHGYiyfMdRPHiyceMPdBZhnfu++afU0kUAxISIWQumI4UsajLi7LNPqFZ6kVo5+X4PUi0LFCTslWSaPc77ZUKhb6ap3RuB28aHQ35149fZUoTZPOHuP7hRsXnFzCSAcDPOncwjq+IrGbNkMNBtP0SaSLss8AnOceP0I8cf/0yhVpy4TqZdirVbf/ffSeL3UY+bGCg5YvY/PXykS/ISyknEhX/rHKf8uXfbrMBEApWVnhlacz55bUOhwMqRLpVUmP5KGed1aLrVH8jprA66Pq4Hy2W/xPMGQ/RCcnsGuffmt2BVHmefSi4w4fWnnhfhTpfspOMm1UgwGpk0DChDJjNxOBcf0Qg8FP+iTSRtVnyOeUcyeJ0T+YUAEJFXw6iTbEUMastuXLfJ0Dxa4zJMxrH2R18vZKtgVWeNvtPKIimfoJmfj8k50rwloACggCXFjdgW4FhuxjrRZ0gudYivOsmu4oERRxSpOoAbhPWnSdRMBA+cd33KQwXPG92wJov9Z1n1OR5E071HnHjxDD9j280vGVLpRnUX5Pvmtu5YX70YnqBinTOiSQLrLBgB4VgICQQhkUpqQGg2n6JNJF2WfYdUW/eeKBu8Tbr8wVpquhBu0zvtiPIIL0iQoMiZqJBGkT4j4nGgRxLyyWSsQpmFULQAE/dJ2AGk7YSJQQgMYHck71gCeAi+XeqVRQulBdH0qeKh3qAP5XYAmw8r+eXhbbfS5Ll3ysJhGjSvNikS4BBF5u775rdun1AAo6BS8+IHX2noAP1/OxaoNB1Ut5KygwVferXdVgqJYm6d+i7DPUHSn0v3MeqkigAZbmjdRnKks5pEoToq2sSJSwwWSi+gZ58GFF9/nSTqA1yZM0b7/4HyJGzVMgaPIcS2slCQKq6C1ZegMO7EjCL5TG8RuSqpJWnXkz8nkhAC9peEbpR5FqeR5C4iUt+XMFfNVvMkm3uUzZ+MvbjL7qnhfUeTShKo5kwIv3E3kbYJnym7Pfx6E/VOHRPsx7fzuqWI7osljGqwFiWtVvDO3b0bbic6IAOYUB0yziSBdJn1EGNeXpMHrHVnHr02bzeYP2GZFOZ58fcfhx246PKHYC7cS4ZCqJMvZkIJpNa0mgesdQnqiAH28AgAPQkBrxC+U79/nfCzzJT6VRU4haawGkymjEFcmT37gCnmZvXa9x43yfvuD1F3OqM4etFksQNaOa5nXf7dflC/n2+0zTJ5RuXjqdfYEOGZbQZaVSKbHdsL3FC08+XJHSTfKk7LYVn7M1r5GkLKoeSZ9BskL/BwEU+xxhvphr0D6DP/Pf5s6+L7LTAzAe+bEnIMDQbxH+/ACoWlbT0fgAkoCakiRVfzUFuoooLiVQ9ayXhMl9lV6l7U7XCkIo3UvYiqPD8vPCWfoC4JqqJGwVIns+qsGAjm/9AZuI5594qPIxraA+GEyfSShdJH2GibbizXFl5/A1XbU0cp8Bc/734fstyrIe9n1gPFISeq284Mu/br+ug35LWj9L+Fp529+rc2DKuhsMPO5PD7/jy4JYPUuzX5HO/vL7i5eVSkUC1ZpOcGaZh09VWcbHcYRHraoxGE7acyDLd0KoKQGh1mNJ/m77jDe368IbxtJtV1+0tFwucZrAEj8SqHdT7C8mHJjw8QcLs1HNmiYFkgaQuPP6X+dLpeJvGxA8qSIri6m3TL5AncZp2rTQ6WZMnSTa21awFKymcgpdTogMbJ/xZl6FN6glkiJUcPhll8slzgyrCCIWQJPivgSKmyadnwfUkiL0Xx35/EcNDBKwoiEGQ1LvxEc5lcnF9hlXjsGbyZz75XeHlWtuNW4yZi/6f4fmpe6zC7UtgNZgXMQ/T1jy0Qf5y886gYCssRPSLs74pWKBQ4aSQ23/LWuIweC/2ok8YfuMN5vHZ3PN/zjnuD2LURlovYpizC5885V8qVTgZNEuqltMva4arF5flnBY13vzXvs+zfaz88Evm5iVLzz1YB7DvaxrxvSbT4LpZ2ZzzTs/NP3GzfY8ZHQ6gK+icVUvG/eNjleffRy1Bmf+NPLEQpu6+gxS0A57ciJwPNQN+4woFYv/KBVLI5+b88A6O+xxcCz9ZvI53xGP/vOvmXK5zNHbr+jctwCqcyOZ75x0OP+FJx8e1WeNtQSnZ0ZNDARmZZFKzSwViydEnX9c+dV7MMTVrgjyrfSZ1557wvaZVZnZVi6Xbl36yeKR/7rjunV22PPgNCdoRkFMWD8+ZljxuccfWFEul77p5upmATQKTvvPA7es+f95ZOaoD9+bv0poLf/ZffEEy/aK5JlKzSzk248TQkTmL/dFKbF9q+tgiK1V0WRs+4w3Hyv9ptCRHzDztmuGilSqsuMqzLHh+Af/4rRD8x8unPdhqVjgRFtXP2ELoN4vJe5fGBAPz3/1+aNn3/e3zEabbZ029UVzqxiz5R8mfE9F3p4sJc/uBJ6qWXUbDKoCDXy1fcb75dDX8Z1NvfTMozvdc9OV6bX7b5jGP9gPkOIX/NufHNc27frfZFd8vvTmcrl0WDUfdOsH6v1CkvplaCbbdFWx0DF8p71Hdhw37hdNfnYZAZwzbpws7rzuV3ms7dJgFH5rT1Ktr17OhHQme2auuaX5tJ9dlSW4ih/dKIPhpknntb3yn8daZGyG7r6jTXHL9hnFCfcrW7/x660cesm42n7XfSvjqne/Nbt28WF4+ui9+ZV98Eics2feUZCRnlR0OFepUy/SAqjOjfp+H5FKpW8sl0sbrd1/w/zuBx2Tw8jEHt1BW23fBRxsU4TQc953+7V5dhil0xmc5PHzxFjU6EYRv1xObDD4rVgDpLd9pvpLoO+wFXxUOp3Zt1Qq9vVKjntSqVS4Xi7VkfQtdVMOVGIJEDChSmzUcjqdWSq30dJB6Cg9nWgjG7mny7Z7xY0tS94hbcLL1YFsnzF/y8TX0D9eQd6NcrQSqBGb6ppIRarSK1FzaaEn7sHfVSAa1UT8Sfms7mT7TEI94P8D7LbGTloLO1gAAAAASUVORK5CYII="}},"cell_type":"markdown","metadata":{"graffitiCellId":"id_c8j5g7h","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"3547FD2B0E60465BB11AA6BADF4AC288","mdEditEnable":false},"source":"# 丢弃法\n\n多层感知机中神经网络图描述了一个单隐藏层的多层感知机。其中输入个数为4，隐藏单元个数为5，且隐藏单元$h_i$（$i=1, \\ldots, 5$）的计算表达式为\n\n\n$$\n h_i = \\phi\\left(x_1 w_{1i} + x_2 w_{2i} + x_3 w_{3i} + x_4 w_{4i} + b_i\\right) \n$$\n\n\n这里$\\phi$是激活函数，$x_1, \\ldots, x_4$是输入，隐藏单元$i$的权重参数为$w_{1i}, \\ldots, w_{4i}$，偏差参数为$b_i$。当对该隐藏层使用丢弃法时，该层的隐藏单元将有一定概率被丢弃掉。设丢弃概率为$p$，那么有$p$的概率$h_i$会被清零，有$1-p$的概率$h_i$会除以$1-p$做拉伸。丢弃概率是丢弃法的超参数。具体来说，设随机变量$\\xi_i$为0和1的概率分别为$p$和$1-p$。使用丢弃法时我们计算新的隐藏单元$h_i'$\n\n\n$$\n h_i' = \\frac{\\xi_i}{1-p} h_i \n$$\n\n\n由于$E(\\xi_i) = 1-p$，因此\n\n\n$$\n E(h_i') = \\frac{E(\\xi_i)}{1-p}h_i = h_i \n$$\n\n\n即丢弃法不改变其输入的期望值。让我们对之前多层感知机的神经网络中的隐藏层使用丢弃法，一种可能的结果如图所示，其中$h_2$和$h_5$被清零。这时输出值的计算不再依赖$h_2$和$h_5$，在反向传播时，与这两个隐藏单元相关的权重的梯度均为0。由于在训练中隐藏层神经元的丢弃是随机的，即$h_1, \\ldots, h_5$都有可能被清零，输出层的计算无法过度依赖$h_1, \\ldots, h_5$中的任一个，从而在训练模型时起到正则化的作用，并可以用来应对过拟合。在测试模型时，我们为了拿到更加确定性的结果，一般不使用丢弃法\n\n![Image Name](https://cdn.kesci.com/upload/image/q5jd69in3m.png?imageView2/0/w/960/h/960)\n"},{"cell_type":"markdown","metadata":{"graffitiCellId":"id_698weeu","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"53FBE46C03C040D0A123E957865CEC6A","mdEditEnable":false},"source":"## 丢弃法从零开始的实现"},{"cell_type":"code","execution_count":21,"metadata":{"graffitiCellId":"id_0cw2xsh","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"30FE9592111F415A8C913AC04AD9BF9A","collapsed":false,"scrolled":false},"outputs":[{"output_type":"stream","text":"1.3.0\n","name":"stdout"}],"source":"%matplotlib inline\nimport torch\nimport torch.nn as nn\nimport numpy as np\nimport sys\nsys.path.append(\"/home/kesci/input\")\nimport d2lzh1981 as d2l\n\nprint(torch.__version__)"},{"cell_type":"code","execution_count":22,"metadata":{"graffitiCellId":"id_7mt4bcl","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"C6790DFD4650485AB8315E04E12D4C74","collapsed":false,"scrolled":false},"outputs":[],"source":"def dropout(X, drop_prob):\n    X = X.float()\n    assert 0 <= drop_prob <= 1\n    keep_prob = 1 - drop_prob\n    # 这种情况下把全部元素都丢弃\n    if keep_prob == 0:\n        return torch.zeros_like(X)\n    mask = (torch.rand(X.shape) < keep_prob).float()\n    \n    return mask * X / keep_prob"},{"cell_type":"code","execution_count":23,"metadata":{"graffitiCellId":"id_ek4ildz","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"F961F801B5CB434683F8F82FD79C8F23","collapsed":false,"scrolled":false},"outputs":[{"output_type":"execute_result","metadata":{},"data":{"text/plain":"tensor([[ 0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.],\n        [ 8.,  9., 10., 11., 12., 13., 14., 15.]])"},"transient":{},"execution_count":23}],"source":"X = torch.arange(16).view(2, 8)\ndropout(X, 0)"},{"cell_type":"code","execution_count":24,"metadata":{"graffitiCellId":"id_i67qnbo","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"A2563ED1062D48728E6AA868A3A8C3F4","collapsed":false,"scrolled":false},"outputs":[{"output_type":"execute_result","metadata":{},"data":{"text/plain":"tensor([[ 0.,  0.,  0.,  6.,  8., 10.,  0., 14.],\n        [ 0.,  0., 20.,  0.,  0.,  0., 28.,  0.]])"},"transient":{},"execution_count":24}],"source":"dropout(X, 0.5)"},{"cell_type":"code","execution_count":25,"metadata":{"graffitiCellId":"id_gwsj0hd","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"56DF0D9E4EE84D04A6CCE5180FF9D164","collapsed":false,"scrolled":false},"outputs":[{"output_type":"execute_result","metadata":{},"data":{"text/plain":"tensor([[0., 0., 0., 0., 0., 0., 0., 0.],\n        [0., 0., 0., 0., 0., 0., 0., 0.]])"},"transient":{},"execution_count":25}],"source":"dropout(X, 1.0)"},{"cell_type":"code","execution_count":26,"metadata":{"graffitiCellId":"id_w81sasb","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"E3ECEB59A5F944898BE2C5ABEB513208","collapsed":false,"scrolled":false},"outputs":[],"source":"# 参数的初始化\nnum_inputs, num_outputs, num_hiddens1, num_hiddens2 = 784, 10, 256, 256\n\nW1 = torch.tensor(np.random.normal(0, 0.01, size=(num_inputs, num_hiddens1)), dtype=torch.float, requires_grad=True)\nb1 = torch.zeros(num_hiddens1, requires_grad=True)\nW2 = torch.tensor(np.random.normal(0, 0.01, size=(num_hiddens1, num_hiddens2)), dtype=torch.float, requires_grad=True)\nb2 = torch.zeros(num_hiddens2, requires_grad=True)\nW3 = torch.tensor(np.random.normal(0, 0.01, size=(num_hiddens2, num_outputs)), dtype=torch.float, requires_grad=True)\nb3 = torch.zeros(num_outputs, requires_grad=True)\n\nparams = [W1, b1, W2, b2, W3, b3]"},{"cell_type":"code","execution_count":27,"metadata":{"graffitiCellId":"id_d24i59y","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"C3C7D8DC21FA471AA4AF11DFAC41D7E5","collapsed":false,"scrolled":false},"outputs":[],"source":"drop_prob1, drop_prob2 = 0.2, 0.5\n\ndef net(X, is_training=True):\n    X = X.view(-1, num_inputs)\n    H1 = (torch.matmul(X, W1) + b1).relu()\n    if is_training:  # 只在训练模型时使用丢弃法\n        H1 = dropout(H1, drop_prob1)  # 在第一层全连接后添加丢弃层\n    H2 = (torch.matmul(H1, W2) + b2).relu()\n    if is_training:\n        H2 = dropout(H2, drop_prob2)  # 在第二层全连接后添加丢弃层\n    return torch.matmul(H2, W3) + b3"},{"cell_type":"code","execution_count":28,"metadata":{"graffitiCellId":"id_ticmy78","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"73C1F705FAA14F4390B8D24CCF9E5673","collapsed":false,"scrolled":false},"outputs":[],"source":"def evaluate_accuracy(data_iter, net):\n    acc_sum, n = 0.0, 0\n    for X, y in data_iter:\n        if isinstance(net, torch.nn.Module):\n            net.eval() # 评估模式, 这会关闭dropout\n            acc_sum += (net(X).argmax(dim=1) == y).float().sum().item()\n            net.train() # 改回训练模式\n        else: # 自定义的模型\n            if('is_training' in net.__code__.co_varnames): # 如果有is_training这个参数\n                # 将is_training设置成False\n                acc_sum += (net(X, is_training=False).argmax(dim=1) == y).float().sum().item() \n            else:\n                acc_sum += (net(X).argmax(dim=1) == y).float().sum().item() \n        n += y.shape[0]\n    return acc_sum / n"},{"cell_type":"code","execution_count":29,"metadata":{"graffitiCellId":"id_ryzllq8","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"B493E72017B54831A80790EE0AA3DB2C","collapsed":false,"scrolled":false},"outputs":[{"output_type":"stream","text":"epoch 1, loss 0.0046, train acc 0.549, test acc 0.704\nepoch 2, loss 0.0023, train acc 0.785, test acc 0.737\nepoch 3, loss 0.0019, train acc 0.825, test acc 0.834\nepoch 4, loss 0.0017, train acc 0.842, test acc 0.763\nepoch 5, loss 0.0016, train acc 0.848, test acc 0.813\n","name":"stdout"}],"source":"num_epochs, lr, batch_size = 5, 100.0, 256  # 这里的学习率设置的很大，原因与之前相同。\nloss = torch.nn.CrossEntropyLoss()\ntrain_iter, test_iter = d2l.load_data_fashion_mnist(batch_size, root='/home/kesci/input/FashionMNIST2065')\nd2l.train_ch3(\n    net,\n    train_iter,\n    test_iter,\n    loss,\n    num_epochs,\n    batch_size,\n    params,\n    lr)\n"},{"cell_type":"markdown","metadata":{"graffitiCellId":"id_ar5qzrs","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"99EEA36EC08E471E8EC87F0AF6FCEF34","mdEditEnable":false},"source":"## 简洁实现"},{"cell_type":"code","execution_count":30,"metadata":{"graffitiCellId":"id_t8mnig0","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"28E345EC41EB48D8876B6A15A3E18FBD","collapsed":false,"scrolled":false},"outputs":[],"source":"net = nn.Sequential(\n        d2l.FlattenLayer(),\n        nn.Linear(num_inputs, num_hiddens1),\n        nn.ReLU(),\n        nn.Dropout(drop_prob1),\n        nn.Linear(num_hiddens1, num_hiddens2), \n        nn.ReLU(),\n        nn.Dropout(drop_prob2),\n        nn.Linear(num_hiddens2, 10)\n        )\n\nfor param in net.parameters():\n    nn.init.normal_(param, mean=0, std=0.01)"},{"cell_type":"code","execution_count":31,"metadata":{"graffitiCellId":"id_iba0hj1","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"2EDA1430541148BB863DC9071ED92323","collapsed":false,"scrolled":false},"outputs":[{"output_type":"stream","text":"epoch 1, loss 0.0046, train acc 0.553, test acc 0.736\nepoch 2, loss 0.0023, train acc 0.785, test acc 0.803\nepoch 3, loss 0.0019, train acc 0.818, test acc 0.756\nepoch 4, loss 0.0018, train acc 0.835, test acc 0.829\nepoch 5, loss 0.0016, train acc 0.848, test acc 0.851\n","name":"stdout"}],"source":"optimizer = torch.optim.SGD(net.parameters(), lr=0.5)\nd2l.train_ch3(net, train_iter, test_iter, loss, num_epochs, batch_size, None, None, optimizer)"},{"cell_type":"markdown","metadata":{"graffitiCellId":"id_m2ujou7","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"88258E5A8E1B4FCAA576FB7FB1AAC42C","mdEditEnable":false},"source":"# 总结\n\n- 欠拟合现象：模型无法达到一个较低的误差\n    \n- 过拟合现象：训练误差较低但是泛化误差依然较高，二者相差较大\n"}],"metadata":{"kernelspec":{"name":"python3","display_name":"Python 3","language":"python"},"language_info":{"name":"python","version":"3.7.3","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}