Day008_HW.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 靜態網頁的資料爬蟲策略\n",
    "\n",
    "\n",
    "* 了解靜態網頁的資料爬蟲策略\n",
    "* 認識適用於靜態網頁爬蟲的相關套件工具：Request\n",
    "* 認識適用於靜態網頁爬蟲的相關套件工具：BeatifulSoup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 作業目標\n",
    "\n",
    "利用 Request + BeatifulSoup 爬取下列兩個網站內容並解析：\n",
    "\n",
    "1. Dcared 網址： https://www.dcard.tw/f\n",
    "2. 知乎： https://www.zhihu.com/explore\n",
    "\n",
    "並且回答下面問題：\n",
    "\n",
    "1. Request 取回之後該怎麼取出資料，資料型態是什麼？\n",
    "2. 為什麼要使用 BeatifulSoup 處理？處理後的型態是什麼？\n",
    "3. 觀察一下知乎回來的資料好像有點怪怪的，該怎麼解決？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Dcard 網址： https://www.dcard.tw/f"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "from bs4 import BeautifulSoup\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "<!DOCTYPE html><html lang=\"zh-TW\"><head prefix=\"og: http://ogp.me/ns#\" itemscope=\"\" itemType=\"https://schema.org/WebSite\"><meta charSet=\"utf-8\"/><meta http-equiv=\"X-UA-Compatible\" content=\"IE=edge\"/><meta name=\"apple-mobile-web-app-status-bar-style\" content=\"default\"/><meta name=\"application-name\" content=\"Dcard\"/><meta name=\"apple-itunes-app\" content=\"app-id=951353454\"/><meta name=\"theme-color\" content=\"#006aa6\"/><meta name=\"mobile-web-app-capable\" content=\"yes\"/><meta name=\"apple-mobile-web-app-capable\" content=\"yes\"/><meta name=\"supported-color-schemes\" content=\"light\"/><meta property=\"fb:app_id\" content=\"211628828926493\"/><meta property=\"fb:pages\" content=\"178875832200695,577748865730563,1333515469994506,619122564952487,804004803032067,178024805867764\"/><meta property=\"al:ios:app_store_id\" content=\"951353454\"/><meta property=\"al:ios:app_name\" content=\"Dcard\"/><meta property=\"al:android:package\" content=\"com.sparkslab.dcardreader\"/><meta property=\"al:android:app_name\" content=\"Dcard\"/><link rel=\"manifest\" href=\"/_next/static/manifest.json\"/><link rel=\"preconnect\" href=\"https://www.googletagmanager.com\" crossorigin=\"true\"/><link rel=\"preconnect\" href=\"https://fonts.googleapis.com\" crossorigin=\"true\"/><link rel=\"preconnect\" href=\"https://fonts.gstatic.com\" crossorigin=\"true\"/><link rel=\"dns-prefetch\" href=\"https://www.googletagmanager.com\"/><link rel=\"dns-prefetch\" href=\"https://fonts.googleapis.com\"/><link rel=\"dns-prefetch\" href=\"https://fonts.gstatic.com\"/><link rel=\"dns-prefetch\" href=\"https://www.google-analytics.com\"/><link rel=\"dns-prefetch\" href=\"https://sb.scorecardresearch.com\"/><link rel=\"dns-prefetch\" href=\"https://certify.alexametrics.com\"/><link rel=\"dns-prefetch\" href=\"https://cloudfront-labs.amazonaws.com\"/><link rel=\"dns-prefetch\" href=\"https://bilanx.dcard.tw\"/><link rel=\"dns-prefetch\" href=\"https://imgur.dcard.tw\"/><link rel=\"dns-prefetch\" href=\"https://megapx-assets.dcard.tw\"/><link rel=\"dns-prefetch\" href=\"https://browser.sentry-cdn.com\"/><link rel=\"dns-prefetch\" href=\"https://gcs.dcard.tw\"/><link rel=\"preload\" href=\"https://fonts.googleapis.com/css2?family=Roboto:wght@400;500;700&amp;display=swap\" as=\"style\"/><link rel=\"stylesheet\" href=\"https://fonts.googleapis.com/css2?family=Roboto:wght@400;500;700&amp;display=swap\"/><link rel=\"icon\" type=\"image/png\" sizes=\"16x16\" href=\"/_next/static/media/icon-16-8783abf2c964e446614d4d22e23ca2bb.png\"/><link rel=\"icon\" type=\"image/png\" sizes=\"24x24\" href=\"/_next/static/media/icon-24-32dfd860d03fae33428bcfcabbc07df5.png\"/><link rel=\"icon\" type=\"image/png\" sizes=\"32x32\" href=\"/_next/static/media/icon-32-7974ce7fd55403bdd4174737b963d694.png\"/><link rel=\"icon\" type=\"image/png\" sizes=\"192x192\" href=\"/_next/static/media/icon-192-8ddcd08c7f194cf623c2e405e1689557.png\"/><link rel=\"apple-touch-icon\" type=\"image/png\" sizes=\"57x57\" href=\"/_next/static/media/icon-57-f33c45592fe2c81b9f6fa9f7689e0d17.png\"/><link rel=\"apple-touch-icon\" type=\"image/png\" sizes=\"72x72\" href=\"/_next/static/media/icon-72-439\n"
     ]
    }
   ],
   "source": [
    "url = 'https://www.dcard.tw/f'\n",
    "\n",
    "import requests\n",
    "# 引入函式庫\n",
    "r = requests.get('https://www.dcard.tw/f')\n",
    "# 想要爬資料的目標網址\n",
    "response = r.text\n",
    "# 模擬發送請求的動作ere\n",
    "\n",
    "# print(response)\n",
    "# response[\"message\"]\n",
    "r.encoding = 'utf-8'\n",
    "print(r.text[0:3000])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Request 取回之後該怎麼取出資料，資料型態是什麼？ =>\n<class 'str'>\n"
     ]
    }
   ],
   "source": [
    "print('Request 取回之後該怎麼取出資料，資料型態是什麼？ =>')\n",
    "\n",
    "# import json\n",
    "# response = json.loads(response)\n",
    "\n",
    "print(type(response))\n",
    "# print(response['message'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "tags": [
     "outputPrepend"
    ]
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "058273\",\"235058387\",\"235061160\",\"235057753\",\"235054715\",\"235057474\",\"235054884\",\"235058692\",\"235054972\",\"235060777\",\"235055073\"],\"loading\":false,\"noMore\":false}},\"paging\":{\"head\":{},\"pageKey\":{},\"nextKey\":{}},\"listKeyOfPages\":{},\"idOfPrimaryKeys\":{\"235054715\":\"235054715\",\"235054759\":\"235054759\",\"235054844\":\"235054844\",\"235054884\":\"235054884\",\"235054972\":\"235054972\",\"235055073\":\"235055073\",\"235055756\":\"235055756\",\"235055932\":\"235055932\",\"235056171\":\"235056171\",\"235056464\":\"235056464\",\"235056566\":\"235056566\",\"235056681\":\"235056681\",\"235056960\":\"235056960\",\"235057056\":\"235057056\",\"235057150\":\"235057150\",\"235057474\":\"235057474\",\"235057753\":\"235057753\",\"235058040\":\"235058040\",\"235058273\":\"235058273\",\"235058380\":\"235058380\",\"235058387\":\"235058387\",\"235058551\":\"235058551\",\"235058692\":\"235058692\",\"235058749\":\"235058749\",\"235059040\":\"235059040\",\"235059270\":\"235059270\",\"235059411\":\"235059411\",\"235059934\":\"235059934\",\"235060777\":\"235060777\",\"235061160\":\"235061160\"},\"elevated\":null,\"errorDetails\":{},\"infiniteIndices\":{}},\"postCategory\":{\"loading\":false,\"fetched\":false,\"data\":{}},\"postDraft\":{\"data\":{},\"error\":null,\"paginations\":{},\"paging\":{\"head\":{},\"pageKey\":{},\"nextKey\":{}},\"listKeyOfPages\":{},\"idOfPrimaryKeys\":{}},\"ranking\":{\"loading\":false,\"fetched\":false,\"data\":{},\"paginations\":{},\"paging\":{\"head\":{},\"pageKey\":{},\"nextKey\":{}}},\"reaction\":{\"loading\":true,\"fetched\":false,\"data\":{\"aa0d425f-d530-4478-9a77-fe3aedc79eea\":{\"id\":\"aa0d425f-d530-4478-9a77-fe3aedc79eea\",\"name\":\"森77\",\"url\":\"https://megapx-assets.dcard.tw/images/9e806d2a-abda-4635-80a5-72b566d5c53c/orig.png\",\"thumbnail\":\"https://megapx-assets.dcard.tw/images/042b27f9-b507-473b-8f36-654aedcc37df/orig.png\",\"position\":1,\"startAt\":\"2018-01-01T00:00:00.000Z\",\"endAt\":null,\"rules\":{\"isGeneral\":true},\"localeKey\":null,\"createdAt\":\"2018-10-01T07:19:03.922Z\",\"updatedAt\":\"2020-09-17T03:38:21.226Z\",\"deletedAt\":null,\"active\":true,\"isGeneral\":true},\"514c2569-fd53-4d9d-a415-bf0f88e7329f\":{\"id\":\"514c2569-fd53-4d9d-a415-bf0f88e7329f\",\"name\":\"嗚嗚\",\"url\":\"https://megapx-assets.dcard.tw/images/ac0053ae-cc67-48c1-a553-6d3c24e9dcec/orig.png\",\"thumbnail\":\"https://megapx-assets.dcard.tw/images/039a093e-93ec-476c-976c-4a53b9b6d11c/orig.png\",\"position\":3,\"startAt\":\"2018-01-01T00:00:00.000Z\",\"endAt\":null,\"rules\":{\"isGeneral\":true},\"localeKey\":null,\"createdAt\":\"2018-10-01T07:19:03.922Z\",\"updatedAt\":\"2020-09-17T03:38:54.045Z\",\"deletedAt\":null,\"active\":true,\"isGeneral\":true},\"011ead16-9b83-4729-9fde-c588920c6c2d\":{\"id\":\"011ead16-9b83-4729-9fde-c588920c6c2d\",\"name\":\"跪\",\"url\":\"https://megapx-assets.dcard.tw/images/0daf1c9e-0a4d-44ac-9d2a-c623beda90b3/orig.png\",\"thumbnail\":\"https://megapx-assets.dcard.tw/images/8ba704c3-2661-4f21-b874-fb28ee5e22ef/orig.png\",\"position\":5,\"startAt\":\"2018-01-01T00:00:00.000Z\",\"endAt\":null,\"rules\":{\"isGeneral\":true},\"localeKey\":null,\"createdAt\":\"2018-10-01T07:19:03.922Z\",\"updatedAt\":\"2020-09-17T03:39:09.135Z\",\"deletedAt\":null,\"active\":true,\"isGeneral\":true},\"4b018f48-e184-445f-adf1-fc8e04ba09b9\":{\"id\":\"4b018f48-e184-445f-adf1-fc8e04ba09b9\",\"name\":\"驚訝\",\"url\":\"https://megapx-assets.dcard.tw/images/f86d9611-64c0-4821-8b62-7352d7691686/orig.png\",\"thumbnail\":\"https://megapx-assets.dcard.tw/images/2484e51a-2cdc-4f0e-a4ea-8d80b4589762/orig.png\",\"position\":7,\"startAt\":\"2018-01-01T00:00:00.000Z\",\"endAt\":null,\"rules\":{\"isGeneral\":true},\"localeKey\":null,\"createdAt\":\"2018-10-01T07:19:03.922Z\",\"updatedAt\":\"2020-09-17T03:39:23.180Z\",\"deletedAt\":null,\"active\":true,\"isGeneral\":true},\"e8e6bc5d-41b0-4129-b134-97507523d7ff\":{\"id\":\"e8e6bc5d-41b0-4129-b134-97507523d7ff\",\"name\":\"哈哈\",\"url\":\"https://megapx-assets.dcard.tw/images/be70b0c8-4c28-436d-8747-0197ede9e95f/orig.png\",\"thumbnail\":\"https://megapx-assets.dcard.tw/images/9a7cc9af-9f81-43ea-8d9b-968a6441ae51/orig.png\",\"position\":9,\"startAt\":\"2018-01-01T00:00:00.000Z\",\"endAt\":null,\"rules\":{\"isGeneral\":true},\"localeKey\":null,\"createdAt\":\"2018-10-01T07:19:03.922Z\",\"updatedAt\":\"2020-09-17T03:39:36.921Z\",\"deletedAt\":null,\"active\":true,\"isGeneral\":true},\"f4ab5118-bf78-40df-9531-b6f3817050cc\":{\"id\":\"f4ab5118-bf78-40df-9531-b6f3817050cc\",\"name\":\"已下架\",\"url\":\"https://megapx-assets.dcard.tw/images/15ec3de2-0cde-4eaa-96cb-7b420b7d0d3e/full.png\",\"thumbnail\":\"https://megapx-assets.dcard.tw/images/15ec3de2-0cde-4eaa-96cb-7b420b7d0d3e/full.png\",\"position\":10,\"startAt\":\"2020-08-18T02:57:39.000Z\",\"endAt\":\"2020-08-30T03:52:19.000Z\",\"rules\":{\"isGeneral\":true},\"localeKey\":null,\"createdAt\":\"2020-08-19T02:58:01.004Z\",\"updatedAt\":\"2020-08-31T03:52:26.025Z\",\"deletedAt\":null,\"active\":false,\"isGeneral\":true},\"286f599c-f86a-4932-82f0-f5a06f1eca03\":{\"id\":\"286f599c-f86a-4932-82f0-f5a06f1eca03\",\"name\":\"愛心\",\"url\":\"https://megapx-assets.dcard.tw/images/09a9f9ab-669e-4a73-9f0f-99f1d0686c46/orig.png\",\"thumbnail\":\"https://megapx-assets.dcard.tw/images/52057289-337a-4f2f-88c0-cb8a77ee422a/orig.png\",\"position\":11,\"startAt\":\"2018-01-01T00:00:00.000Z\",\"endAt\":null,\"rules\":{\"isGeneral\":true},\"localeKey\":null,\"createdAt\":\"2018-09-17T06:19:50.063Z\",\"updatedAt\":\"2020-09-17T03:39:52.431Z\",\"deletedAt\":null,\"active\":true,\"isGeneral\":true},\"ba4ebb00-877f-4dbc-8ba2-5ee60a4d0c5c\":{\"id\":\"ba4ebb00-877f-4dbc-8ba2-5ee60a4d0c5c\",\"name\":\"已下架\",\"url\":\"https://gcs.dcard.tw/reaction/ba4ebb00-877f-4dbc-8ba2-5ee60a4d0c5c1539664467287.png\",\"thumbnail\":\"https://gcs.dcard.tw/reaction/ba4ebb00-877f-4dbc-8ba2-5ee60a4d0c5c1539664467287.png\",\"position\":100,\"startAt\":\"2018-10-16T04:35:50.000Z\",\"endAt\":\"2018-10-16T04:57:32.000Z\",\"rules\":{\"isGeneral\":false},\"localeKey\":null,\"createdAt\":\"2018-10-16T04:34:27.803Z\",\"updatedAt\":\"2018-10-16T04:54:57.950Z\",\"deletedAt\":null,\"active\":false,\"isGeneral\":false},\"a9b7ae13-9e7b-4c6b-9d00-0e227c30ba18\":{\"id\":\"a9b7ae13-9e7b-4c6b-9d00-0e227c30ba18\",\"name\":\"已下架\",\"url\":\"https://megapx-assets.dcard.tw/images/a7061180-d673-4a13-954d-3e7ae76951cf/full.png\",\"thumbnail\":\"https://megapx-assets.dcard.tw/images/a7061180-d673-4a13-954d-3e7ae76951cf/full.png\",\"position\":1000,\"startAt\":\"2019-09-01T04:06:57.000Z\",\"endAt\":\"2020-08-31T04:06:57.000Z\",\"rules\":{\"isGeneral\":true},\"localeKey\":null,\"createdAt\":\"2020-09-01T04:07:23.435Z\",\"updatedAt\":\"2020-09-01T04:07:23.435Z\",\"deletedAt\":null,\"active\":false,\"isGeneral\":true}}},\"recommendedPost\":{\"data\":{},\"error\":null,\"paginations\":{},\"paging\":{\"head\":{},\"pageKey\":{},\"nextKey\":{}},\"listKeyOfPages\":{},\"idOfPrimaryKeys\":{}},\"region\":{\"loading\":false,\"fetched\":false,\"data\":{},\"paginations\":{},\"paging\":{\"head\":{},\"pageKey\":{},\"nextKey\":{}},\"defaultRegionCode\":\"\",\"guestCookieRegionCode\":\"\",\"queryRegionCode\":\"\"},\"remoteConfig\":{\"data\":{\"hot_comments_hidden_post_authors\":[\"dcardadangel\",\"dcardecangel\",\"dcardsvangel\",\"little_angel\"],\"web_adsenses\":[{\"id\":\"div-gpt-ad-1458631742813-0\",\"name\":\"Desktop 右一\",\"path\":\"/146730420/Desktop-Right-Top\",\"sizes\":[{\"width\":300,\"height\":250}]},{\"id\":\"div-gpt-ad-1466065536619-0\",\"name\":\"Desktop 右二(原臉書上方)\",\"path\":\"/146730420/Desktop-Beyond-FB\",\"sizes\":[{\"width\":300,\"height\":250}]},{\"id\":\"div-gpt-ad-1582095045953-0\",\"name\":\"Desktop 右三摩天\",\"path\":\"/146730420/Desktop-Right3\",\"sizes\":[{\"width\":300,\"height\":600}]},{\"id\":\"div-gpt-ad-1582095257338-0\",\"name\":\"Desktop 右四\",\"path\":\"/146730420/Desktop-Right4\",\"sizes\":[{\"width\":300,\"height\":250}]},{\"id\":\"div-gpt-ad-1458631598292-0\",\"name\":\"Desktop 右下摩天\",\"path\":\"/146730420/Desktop-RightBottom-Bottom\",\"sizes\":[{\"width\":300,\"height\":600}]},{\"id\":\"div-gpt-ad-1574770247619-0\",\"name\":\"Desktop 全部回應 (10,30)\",\"path\":\"/146730420/Desktop-Comments\",\"sizes\":[{\"width\":728,\"height\":90}]},{\"id\":\"div-gpt-ad-1467085023952-0\",\"name\":\"Desktop 單篇文章底部\",\"path\":\"/146730420/Desktop-Article-Bottom\",\"sizes\":[{\"width\":728,\"height\":90}]},{\"id\":\"div-gpt-ad-1487138000089-0\",\"name\":\"Mobile Web 文章底部 (new)\",\"path\":\"/146730420/Mobile-Web-Article-End\",\"sizes\":[{\"width\":300,\"height\":250}]},{\"id\":\"div-gpt-ad-1466051481599-0\",\"name\":\"Mobile Web 人氣回應結束\",\"path\":\"/146730420/Mweb-Popular-Comments\",\"sizes\":[{\"width\":300,\"height\":250}]},{\"id\":\"div-gpt-ad-1470381102159-0\",\"name\":\"Mobile Web 全部回應 (每 30 則)\",\"path\":\"/146730420/Mweb-Comments\",\"sizes\":[{\"width\":300,\"height\":250}]},{\"id\":\"div-gpt-ad-1458792275330-0\",\"name\":\"Mobile Web 畫面底部\",\"path\":\"/146730420/Mobile-Web-Screen-Bottom\",\"sizes\":[{\"width\":300,\"height\":250}]},{\"id\":\"div-gpt-ad-1471245470448-0\",\"name\":\"Mobile Web 畫面頂部\",\"path\":\"/146730420/Mobile-Web-Screen-Top\",\"sizes\":[{\"width\":300,\"height\":250}]}]},\"error\":null,\"loading\":false,\"fetched\":true},\"rule\":{\"data\":{},\"error\":null,\"paginations\":{},\"paging\":{\"head\":{},\"pageKey\":{},\"nextKey\":{}},\"listKeyOfPages\":{},\"idOfPrimaryKeys\":{}},\"searchHistory\":{\"data\":{}},\"searchKeyword\":{\"loading\":false,\"fetched\":false,\"data\":{},\"paginations\":{},\"paging\":{\"head\":{},\"pageKey\":{},\"nextKey\":{}}},\"school\":{\"loading\":false,\"fetched\":false,\"data\":{},\"paginations\":{},\"paging\":{\"head\":{},\"pageKey\":{},\"nextKey\":{}},\"abbrMap\":{},\"abbrPending\":{}},\"story\":{\"data\":{},\"error\":null,\"paginations\":{},\"paging\":{\"head\":{},\"pageKey\":{},\"nextKey\":{}},\"listKeyOfPages\":{},\"idOfPrimaryKeys\":{}},\"sudo\":{},\"surveyTeamLastSeenTime\":{\"data\":{},\"error\":null,\"loading\":false,\"fetched\":false},\"tab\":{\"data\":{\"country=TW\\u0026platform=web\":{\"forum\":[{\"uiType\":\"button\",\"alias\":\"hot\",\"name\":\"熱門\",\"listType\":\"legacy\",\"nameMessage\":{\"type\":\"string\",\"message\":\"熱門\"}},{\"uiType\":\"button\",\"alias\":\"latest\",\"name\":\"最新\",\"listType\":\"legacy\",\"nameMessage\":{\"type\":\"string\",\"message\":\"最新\"}},{\"uiType\":\"button\",\"alias\":\"rule\",\"name\":\"板規\",\"listType\":\"rule\",\"nameMessage\":{\"type\":\"string\",\"message\":\"板規\"}}],\"all\":[{\"uiType\":\"button\",\"alias\":\"hot\",\"name\":\"熱門\",\"listType\":\"legacy\",\"nameMessage\":{\"type\":\"string\",\"message\":\"熱門\"},\"isHome\":false},{\"uiType\":\"button\",\"alias\":\"latest\",\"name\":\"最新\",\"listType\":\"legacy\",\"nameMessage\":{\"type\":\"string\",\"message\":\"最新\"},\"isHome\":false},{\"uiType\":\"button\",\"alias\":\"subscribe\",\"name\":\"追蹤\",\"listType\":\"legacy\",\"nameMessage\":{\"type\":\"string\",\"message\":\"追蹤\"},\"isHome\":false}]}},\"error\":null,\"loading\":false,\"fetched\":true},\"tagging\":{\"loading\":false,\"fetched\":false,\"data\":{}},\"topic\":{\"data\":{},\"error\":null,\"paginations\":{},\"paging\":{\"head\":{},\"pageKey\":{},\"nextKey\":{}},\"listKeyOfPages\":{},\"idOfPrimaryKeys\":{},\"loading\":false},\"topicSuggestion\":{\"data\":{},\"error\":null,\"paginations\":{},\"paging\":{\"head\":{},\"pageKey\":{},\"nextKey\":{}},\"listKeyOfPages\":{},\"idOfPrimaryKeys\":{}},\"user\":{\"data\":{},\"error\":null,\"loading\":false,\"fetched\":false,\"fields\":{}},\"videoEngagement\":{\"previousEmitKey\":\"\",\"previousSourceKey\":\"\",\"eventId\":\"\",\"switchCount\":0},\"voteStatus\":{\"loading\":false,\"fetched\":false,\"data\":{},\"aliasToIdMap\":{}},\"haowuCampaign\":{\"loading\":false,\"fetched\":false,\"data\":{},\"paginations\":{},\"paging\":{\"head\":{},\"pageKey\":{},\"nextKey\":{}}},\"orderForm\":{\"shipFeeList\":[],\"coupons\":[],\"member\":{\"commonStores\":[],\"commonAddressList\":[],\"invoiceVehicle\":null,\"invoicePhoneId\":null,\"invoiceHumanId\":null,\"companyInfo\":null},\"correctedOrder\":{},\"orderCorrecting\":false,\"orderCorrectError\":false,\"orderHasBeenCorrected\":false,\"getShipFeeLoading\":false,\"orderValidating\":false,\"orderValidateError\":false,\"orderValidateErrorState\":{},\"ecPaymentInfo\":{\"orderItemId\":\"\",\"showExitConfirmation\":true}},\"order\":{\"loading\":false,\"fetched\":false,\"data\":{},\"paginations\":{},\"paging\":{\"head\":{},\"pageKey\":{},\"nextKey\":{}}},\"catalog\":{},\"catalogWidgets\":{\"loading\":false,\"fetched\":false,\"data\":{},\"paginations\":{},\"paging\":{\"head\":{},\"pageKey\":{},\"nextKey\":{}},\"widgetInfo\":{}},\"catalogImpression\":{\"sent\":{}},\"goodCategory\":{\"data\":{},\"error\":null,\"paginations\":{},\"paging\":{\"head\":{},\"pageKey\":{},\"nextKey\":{}},\"listKeyOfPages\":{},\"idOfPrimaryKeys\":{}},\"goodItem\":{\"data\":{},\"error\":null,\"paginations\":{},\"paging\":{\"head\":{},\"pageKey\":{},\"nextKey\":{}},\"listKeyOfPages\":{},\"idOfPrimaryKeys\":{}},\"groupBuyLastSeenTime\":{\"data\":{},\"error\":null,\"loading\":false,\"fetched\":false}},\"store\":{},\"isWebView\":false,\"locale\":\"zh-TW\",\"localeMessageRegistry\":{\"data\":{\"meta.app.description\":\"廣受年輕人喜愛的 Dcard 是台灣最大的匿名交流平台，舉凡時事話題、感情心情、吃喝玩樂、學習工作等，都有卡友陪你聊！\",\"search.main.action\":\"搜尋\",\"form_validation.is_required.message\":\"此欄位為必填。\",\"signup.login_or_signup.action\":\"註冊 / 登入\",\"settings.end_user_agreement.option\":\"服務條款\",\"settings.faq.option\":\"常見問題\",\"general.report_problem.description\":\"回報問題\",\"general.brand_identification.description\":\"品牌識別\",\"general.hiring.description\":\"徵才\",\"general.business.description\":\"商業合作\",\"general.free_to_download.description\":\"免費下載 App\",\"member.logout.action\":\"登出\",\"general.more.action\":\"更多\",\"forum_list.all_title.option\":\"所有看板\",\"forum_list.realtime_popular.title\":\"即時熱門看板\",\"ec.short.title\":\"好物研究室\",\"forum_list.moderator.title\":\"板主相關\",\"forum_list.favorites.title\":\"我的最愛\",\"forum_list.selected.title\":\"Dcard 精選看板\",\"forum.all.title\":\"全部\",\"comment.count.status_format\":\"回應 {commentCount}\",\"collection.put.action\":\"收藏\",\"post.anonymous.description\":\"匿名\",\"member.gender_female.option\":\"女\",\"member.gender_male.option\":\"男\"}}},\"page\":\"/forum\",\"query\":{},\"buildId\":\"4f8d1d95b292a11069d0289b504718e0aa48ef9a\",\"runtimeConfig\":{\"amplitude\":{\"androidApiKey\":\"cba46104d90853a7ce0afae9e2586bc5\",\"iosApiKey\":\"9e2fd13857b562968ac298aff9f6cbc5\",\"mobileApiKey\":\"c618800786141ad91fe6616b7d70ceeb\",\"apiKey\":\"bfb03c251442ee56c07f655053aba20f\"},\"bilanx\":{\"version\":\"v1\",\"host\":\"https://bilanx.dcard.tw\"},\"domain\":\"www.dcard.tw\",\"facebook\":{\"clientId\":\"211628828926493\",\"version\":\"v8.0\"},\"giftbox\":{\"host\":\"https://giftbox.dcard.tw\"},\"google\":{\"clientId\":\"578393968383-gb01vrjja5mcgsgpih2coq90nbernk39.apps.googleusercontent.com\"},\"googleAnalytics\":{\"trackId\":\"UA-27831574-1\"},\"imgur\":{\"clientId\":\"f0ea04148a54268\"},\"isEnvProd\":true,\"isNodeEnvProd\":true,\"megapx\":{\"host\":\"https://megapx.dcard.tw\"},\"pptr\":{\"host\":\"https://pptr.dcard.tw\"},\"recaptcha\":{\"site\":\"345541453\",\"id\":\"6LdNi5gUAAAAAFQ1jM75Mk82ptVmzA4npgfaPcd1\"},\"sentry\":{\"release\":\"dcard-web@1.235.0\",\"sampleRate\":0.1,\"environment\":\"production\",\"dsn\":\"https://6f4e12f41da04a64a46d3afec82e1277@sentry.io/1518416\",\"defaultIntegrations\":[{\"_options\":{},\"name\":\"InboundFilters\"},{\"name\":\"FunctionToString\"},{\"name\":\"Console\"},{\"name\":\"Http\",\"_breadcrumbs\":true,\"_tracing\":false},{\"_options\":{},\"name\":\"OnUncaughtException\"},{\"_options\":{\"mode\":\"warn\"},\"name\":\"OnUnhandledRejection\"},{\"name\":\"LinkedErrors\",\"_key\":\"cause\",\"_limit\":5}]},\"userAgent\":\"dcard-web\",\"version\":\"1.235.0\"},\"isFallback\":false,\"dynamicIds\":[\"/RKH\",\"00bG\",\"0R8F\",\"0v2e\",\"1USh\",\"2NhW\",\"3H2f\",\"3RLP\",\"3Rm+\",\"3uE+\",\"5BX3\",\"5X3t\",\"6Gik\",\"7giO\",\"BBVp\",\"BgsC\",\"BxyZ\",\"C3NB\",\"DAd4\",\"DZsy\",\"ENlB\",\"FB1i\",\"FP5w\",\"GxFv\",\"H0dQ\",\"ISC+\",\"IpA2\",\"J/Rj\",\"J2uS\",\"JUZD\",\"Jg5f\",\"KLoZ\",\"LhkU\",\"MNqy\",\"NIt9\",\"Oqde\",\"PLBH\",\"SH9S\",\"Sd/a\",\"UZAm\",\"VkfP\",\"XPLO\",\"XWic\",\"Ymfg\",\"avq1\",\"b0ZD\",\"bCPY\",\"cGKn\",\"czL1\",\"dDdM\",\"daRj\",\"eB27\",\"fgBh\",\"gLnJ\",\"iQib\",\"kcks\",\"lbiO\",\"m2GP\",\"oVC2\",\"ojop\",\"pBqJ\",\"pDRq\",\"rX8Z\",\"uOMN\",\"ug9z\",\"vKNz\",\"vUPX\",\"xBqQ\",\"28Xk\",\"4uJK\",\"6bhI\",\"A0Zq\",\"CbIe\",\"Lhmn\",\"MwrP\",\"ORyZ\",\"QRZr\",\"eQ7J\",\"sPdd\",\"tkOS\",\"uHtb\",\"zge7\",\"7VoR\",\"0Yt7\",\"4U32\",\"Cnka\",\"E/xX\",\"EHEA\",\"NC/j\",\"U4wA\",\"XCwa\",\"br6k\",\"la9/\",\"I9Rx\",\"VxxY\",\"nFl4\",\"nVo3\",\"/3cl\",\"1T7t\",\"2epp\",\"2wsP\",\"3vGx\",\"6YYi\",\"Aloh\",\"BDtS\",\"CwuV\",\"GVul\",\"GZgP\",\"JcJ6\",\"LGOb\",\"NxsW\",\"QYyL\",\"QZ90\",\"SU9P\",\"USq9\",\"Wi/K\",\"Xm/D\",\"Ybm3\",\"ZliH\",\"aDSF\",\"ehWz\",\"f2p0\",\"fLdg\",\"gEzS\",\"joOL\",\"muw5\",\"qBJQ\",\"rXFn\",\"sANR\",\"v9+8\",\"vL+r\",\"vczm\",\"wt6n\",\"zSRa\",\"466f\",\"6UbV\",\"AfYR\",\"gj/O\",\"lEDV\",\"1VLm\",\"30qI\",\"3ciz\",\"8hrp\",\"FGAU\",\"JqiK\",\"OEVo\",\"Q8hG\",\"Tkmt\",\"e7y7\",\"iJR/\",\"pN2L\",\"s5Av\",\"sna8\",\"71+A\",\"7Tip\",\"EHGq\",\"G5Zb\",\"M0s1\",\"PaJF\",\"RBiB\",\"U2d/\",\"dRmo\",\"eIKj\",\"h1LJ\",\"m2YG\",\"zCVg\",\"1bpS\",\"7n1t\",\"7u6U\",\"8NAe\",\"9IgF\",\"AKdL\",\"B9v6\",\"Br/J\",\"CX9e\",\"Cs9Q\",\"D4jl\",\"EWis\",\"KtRx\",\"LYY2\",\"Nhio\",\"SNCh\",\"UjHw\",\"YQ7a\",\"aO+1\",\"jZ6r\",\"kHPa\",\"keGA\",\"n+Ka\",\"nsNU\",\"oHI7\",\"tepD\",\"vljj\",\"wowS\",\"xok/\",\"/2Gn\",\"/wCD\",\"0Ocv\",\"Kkar\",\"LtXa\",\"SiwR\",\"a49g\",\"kn3Q\",\"p1C/\",\"r9Vz\",\"uZih\",\"x6rY\"],\"customServer\":true,\"gip\":true,\"appGip\":true}</script><script nomodule=\"\" src=\"/_next/static/chunks/polyfills-e25d44ddf3cf31672c54.js\"></script><script async=\"\" src=\"/_next/static/chunks/c8f7fe3b0e41be846d5687592cf2018ff6e22687.449fabe15332dd03d5e9.js\"></script><script async=\"\" src=\"/_next/static/chunks/78d62a35dcc6dedda879910d5b00bd9d449f477e.be546dad3639004706a3.js\"></script><script async=\"\" src=\"/_next/static/chunks/195.f6bd7090817943c0bd7c.js\"></script><script async=\"\" src=\"/_next/static/chunks/135.7840488cfebf9d30b614.js\"></script><script async=\"\" src=\"/_next/static/chunks/164.a20555c4770aa7687ef9.js\"></script><script async=\"\" src=\"/_next/static/chunks/d0e178a0b7a174cc355912fab2c2b5dbf6686e47.6cb147141079fd7e80f1.js\"></script><script async=\"\" src=\"/_next/static/chunks/139.e17f705458f7eb0b9e7c.js\"></script><script async=\"\" src=\"/_next/static/chunks/131.2be139dde2c51e254528.js\"></script><script async=\"\" src=\"/_next/static/chunks/main-eb3b89a39aaec4994aa7.js\"></script><script async=\"\" src=\"/_next/static/chunks/webpack-dad53eafae918c9a5d55.js\"></script><script async=\"\" src=\"/_next/static/chunks/framework.ae751f90465e16aa76b3.js\"></script><script async=\"\" src=\"/_next/static/chunks/commons.3ddadff88dc777002205.js\"></script><script async=\"\" src=\"/_next/static/chunks/dccd25a4c463633f108cbbc853ada3fe41251cea.f75e7f1b72743a71182e.js\"></script><script async=\"\" src=\"/_next/static/chunks/269bdeabb25f62b2c9a251215553624bff7f92ae.b3d34558bb61faa4814a.js\"></script><script async=\"\" src=\"/_next/static/chunks/f8721c61a0b54f165a8389496a953d0738c3e51b.82353b57448c2b6fb437.js\"></script><script async=\"\" src=\"/_next/static/chunks/b75497125ac60cca393cc1cd1b9cb818f6b1f287.1b280d6afb2b30c6161b.js\"></script><script async=\"\" src=\"/_next/static/chunks/6d6f8ab4d673300c7f9fd6044ff46ced10b06fce.9b240592e7d7bfbe9191.js\"></script><script async=\"\" src=\"/_next/static/chunks/a0a90e7cd7e608842a5a6b2dbd3f3a2250fade66.3797680e283a17a98e54.js\"></script><script async=\"\" src=\"/_next/static/chunks/ce8544ddb613456802626aea57a99d5e06c97783.72103bf25dd5031ee175.js\"></script><script async=\"\" src=\"/_next/static/chunks/89932b1910dac67f8f1d888df886c47cee6e32e1.d70f466923346e4ea4c0.js\"></script><script async=\"\" src=\"/_next/static/chunks/0675dfe7c6f6e766e59eb27d9666fbc1177470a8.fe6c163bc3d8ce3898a3.js\"></script><script async=\"\" src=\"/_next/static/chunks/912a71f9b725d3246164d5326be4edab67ea4c92.779ad53dafc6da6c0b46.js\"></script><script async=\"\" src=\"/_next/static/chunks/ddeb8ccf10a924ccb84e738c520b209d064c92ec.3f7245171fa9e58ebefd.js\"></script><script async=\"\" src=\"/_next/static/chunks/a08e225f50511737b62ce4e8a8af765e4bc753a1.668d3fea4f3c75a2772d.js\"></script><script async=\"\" src=\"/_next/static/chunks/b43e2fac06b263321f6566f878bec696cd0c59a2.e86cd1c1ce6fe64d50fa.js\"></script><script async=\"\" src=\"/_next/static/chunks/pages/_app-f27484d5e4fdd27956d2.js\"></script><script async=\"\" src=\"/_next/static/chunks/834c1bd6c615aa6f8af8b40d1105086149197c4a.2a637ec3ce6016de041b.js\"></script><script async=\"\" src=\"/_next/static/chunks/2cd985e84f0f9cdad0551af875a11646cf7f3372.255b243e86503a37802e.js\"></script><script async=\"\" src=\"/_next/static/chunks/76ee7bf2e8001b0abc3ad07ba6b80b1ce9dd2c4f.8d916592134c088e220c.js\"></script><script async=\"\" src=\"/_next/static/chunks/13b3731f4abb40403d48cd6fb8a8ede505e07c69.7fe5b44157231120156b.js\"></script><script async=\"\" src=\"/_next/static/chunks/02d5cd5e46a9bc7a1951a949fe4c8be1e0c7372d.466f6889fd7e6de62c55.js\"></script><script async=\"\" src=\"/_next/static/chunks/07ddc8feebba80388bd103699a5e8a2319231b85.d02a94561eb9cbe902a4.js\"></script><script async=\"\" src=\"/_next/static/chunks/812378942489f8f32ec5bf01fe4ee20d173e04ee.a8de361db68c725c4be3.js\"></script><script async=\"\" src=\"/_next/static/chunks/0d04c2db568e1348da93aa2450c44e065298f21f.82396e4a3d6714bc60a2.js\"></script><script async=\"\" src=\"/_next/static/chunks/f8821b6946d7dce977b57022b933cfab2256b3ea.e0b654eb99d63f9a5429.js\"></script><script async=\"\" src=\"/_next/static/chunks/0f2537373789941531e9c06242b5024dbfc81a35.7004608478d018b1e306.js\"></script><script async=\"\" src=\"/_next/static/chunks/pages/forum-0fbd285b3ead02b26f08.js\"></script><script async=\"\" src=\"/_next/static/4f8d1d95b292a11069d0289b504718e0aa48ef9a/_buildManifest.js\"></script><script async=\"\" src=\"/_next/static/4f8d1d95b292a11069d0289b504718e0aa48ef9a/_ssgManifest.js\"></script></body></html>\n<class 'bs4.BeautifulSoup'>\n"
     ]
    }
   ],
   "source": [
    "print('為什麼要使用 BeatifulSoup 處理？處理後的型態是什麼？ => ')\n",
    "\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "soup = BeautifulSoup(response, \"html5lib\")\n",
    "print(soup)\n",
    "print(type(soup))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. 知乎： https://www.zhihu.com/explore"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "<html>\n<head><title>403 Forbidden</title></head>\n<body bgcolor=\"white\">\n<center><h1>403 Forbidden</h1></center>\n<hr><center>openresty</center>\n</body>\n</html>\n\n"
     ]
    }
   ],
   "source": [
    "url = 'https://www.zhihu.com/explore'\n",
    "r = requests.get(url)\n",
    "r.encoding = 'utf-8'\n",
    "\n",
    "print(r.text[0:600])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. 觀察一下知乎回來的資料好像有點怪怪的，該怎麼解決？"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "<!doctype html>\n<html lang=\"zh\" data-hairline=\"true\" data-theme=\"light\"><head><meta charSet=\"utf-8\"/><title data-react-helmet=\"true\">发现 - 知乎</title><meta name=\"viewport\" content=\"width=device-width,initial-scale=1,maximum-scale=1\"/><meta name=\"renderer\" content=\"webkit\"/><meta name=\"force-rendering\" content=\"webkit\"/><meta http-equiv=\"X-UA-Compatible\" content=\"IE=edge,chrome=1\"/><meta name=\"google-site-verification\" content=\"FTeR0c8arOPKh8c5DYh_9uu98_zJbaWw53J-Sch9MTg\"/><meta name=\"description\" property=\"og:description\" content=\"有问题，上知乎。知乎，可信赖的问答社区，以让每个人高效获得可信赖的解答为使命。知乎凭借认真、专业和友善的社区氛围，结构化、易获得的\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "url = 'https://www.zhihu.com/explore'\n",
    "\n",
    "headers = {'user-agent': 'app'}\n",
    "r = requests.get(url, headers=headers)\n",
    "\n",
    "r.encoding = 'utf-8'\n",
    "print(r.text[0:600])\n",
    "r = requests.get(url)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3-final"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}