Day031_HW.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 反爬：瀏覽器標頭與基本資訊\n",
    "\n",
    "\n",
    "* 了解「檢查 HTTP 標頭檔」的反爬蟲機制\n",
    "* 「檢查 HTTP 標頭檔」反爬蟲的因應策略\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 作業目標\n",
    "\n",
    "* 尋找一個網站是有作 Header 檢查的（Hint: 有加跟沒有加回傳結果不一樣）\n",
    "* 用上述網站說明該如何判斷 Header 中必須加上哪些資料才會正確\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 尋找一個網站是有作 Header 檢查的（Hint: 有加跟沒有加回傳結果不一樣）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "'<html>\\r\\n<head><title>403 Forbidden</title></head>\\r\\n<body>\\r\\n<center><h1>403 Forbidden</h1></center>\\r\\n<hr><center>nginx</center>\\r\\n</body>\\r\\n</html>\\r\\n'"
      ]
     },
     "metadata": {},
     "execution_count": 1
    }
   ],
   "source": [
    "import requests\n",
    "r = requests.get('https://makezine.com/')\n",
    "response = r.text\n",
    "\n",
    "response"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 用上述網站說明該如何判斷 Header 中必須加上哪些資料才會正確\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "'<!DOCTYPE html>\\n<html lang=\"en\" xmlns:fb=\"http://ogp.me/ns/fb#\" xmlns:fb=\"https://www.facebook.com/2008/fbml\">\\n<head>\\n  <meta charset=\"utf-8\">\\n  <meta http-equiv=\"X-UA-Compatible\" content=\"IE=edge\">\\n  <meta name=\"viewport\" content=\"width=device-width, initial-scale=1\">\\n\\n  <title>Make: DIY Projects and Ideas for Makers | </title>\\n  <meta name=\"twitter:widgets:csp\" content=\"on\">\\n  <meta name=\"p:domain_verify\" content=\"c4e1096cb904ca6df87a2bb867715669\" >\\n    <meta name=\"google-site-verification\" content=\"eAyS6vj89QDOsofiDC4sB-2YR3YNg_VGCYiQnWm2mqc\" />\\n  <meta name=\"norton-safeweb-site-verification\" content=\"4g4w71jm7qt9e7ghe2dxdhiq0mnnkwom6ue80rdet53q3figx8ooxrffbgkkl9kzo3qi85l2j-txt-fh8w-p2z5769ht01z8s6sxq3-8r7cojmimgp00homsjjjv96ww\" />\\n  <meta property=\"fb:admins\" content=\"1612614584\" />\\n\\n  <link rel=\"apple-touch-icon\" sizes=\"180x180\" href=\"https://1abxf1rh6g01lhm2riyrt55k-wpengine.netdna-ssl.com/apple-touch-icon.png\">\\n  <link rel=\"icon\" type=\"image/png\" href=\"https://1abxf1rh6g01lhm2riyrt55k-wpengine.netdna-ssl.com/favicon-32x32.png\" sizes=\"32x32\">\\n  <link rel=\"icon\" type=\"image/png\" href=\"https://1abxf1rh6g01lhm2riyrt55k-wpengine.netdna-ssl.com/favicon-16x16.png\" sizes=\"16x16\">\\n  <link rel=\"manifest\" href=\"/manifest.json\">\\n  <link rel=\"mask-icon\" href=\"https://1abxf1rh6g01lhm2riyrt55k-wpengine.netdna-ssl.com/safari-pinned-tab.svg\" color=\"#5bbad5\">\\n  <meta name=\"theme-color\" content=\"#ffffff\">\\n\\n  <!--[if lt IE 9]><script src=\"../../assets/js/ie8-responsive-file-warning.js\"></script><![endif]-->\\n  <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->\\n  <!--[if lt IE 9]>\\n    <script src=\"https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js\"></script>\\n    <script src=\"https://oss.maxcdn.com/respond/1.4.2/respond.min.js\"></script>\\n  <!--[endif]-->\\n\\n  <!-- Pingdom for site monitoring -->\\n  <script>\\n    var _prum = [[\\'id\\', \\'53fcea2fabe53d341d4ae0eb\\'],\\n      [\\'mark\\', \\'firstbyte\\', (new Date()).getTime()]];\\n    (function() {\\n      var s = '"
      ]
     },
     "metadata": {},
     "execution_count": 2
    }
   ],
   "source": [
    "import requests\n",
    "headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.104 Safari/537.36'}\n",
    "r = requests.get('https://makezine.com/', headers=headers)\n",
    "response = r.text\n",
    "\n",
    "response[:2000]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3-final"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}