Day_014_Yahoo_Movie_HW.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# YAHOO電影爬蟲練習\n",
    "## 練習爬取電影放映資訊。必須逐步獲取電影的代號、放映地區、放映日期後，再送出查詢給伺服器。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "import re\n",
    "from bs4 import BeautifulSoup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 先搜尋全部的電影代號(ID)資訊"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Movie: 通靈駕訓班, ID: 11173\nMovie: 空中謎航, ID: 11152\nMovie: 致命天際線, ID: 11147\nMovie: 廢青四重奏, ID: 11130\nMovie: 妄想代理人：前篇, ID: 11102\nMovie: 水漾的女人, ID: 11065\nMovie: 午夜天鵝, ID: 11045\nMovie: 拆彈專家2, ID: 10986\nMovie: 杏林醫院, ID: 10781\nMovie: 靈魂急轉彎, ID: 11089\nMovie: 瑰麗卡萊爾：浮華紐約, ID: 11129\nMovie: 愛是您・愛是我, ID: 11123\nMovie: 來者弒客, ID: 11107\nMovie: 真愛鄰距離, ID: 11101\nMovie: 坑爹大作戰, ID: 11082\nMovie: 生為女人, ID: 10977\nMovie: 一家之煮, ID: 10955\nMovie: 腿, ID: 10934\nMovie: 鬼巢, ID: 11105\nMovie: 高校棋蹟, ID: 11099\nMovie: 說唱藝人：希望之聲, ID: 11086\nMovie: 85年的夏天, ID: 11076\nMovie: 順其自然的日子, ID: 11066\nMovie: 總是有個人在愛你, ID: 11063\nMovie: 神力女超人1984, ID: 10413\nMovie: 新解釋・三國志, ID: 11050\nMovie: 信用詐欺師JP：公主篇, ID: 11021\nMovie: 求婚好意外, ID: 10796\nMovie: 再見街貓BOB, ID: 11016\nMovie: 只有悲傷才是美麗的, ID: 11055\nMovie: 愛在午夜希臘時, ID: 11054\nMovie: 愛在黎明破曉時, ID: 11053\nMovie: 愛在日落巴黎時, ID: 11052\nMovie: 魔物獵人, ID: 10983\nMovie: 親愛的殺手, ID: 10861\nMovie: 十二夜2：回到第零天, ID: 11035\nMovie: 尋找小魔女Doremi, ID: 11027\nMovie: 緝毒風暴, ID: 11023\nMovie: 古魯家族：新石代, ID: 10958\nMovie: 同學麥娜絲, ID: 10935\nMovie: 名偵探柯南：紅之校外旅行 鮮紅篇&戀紅篇, ID: 10887\nMovie: 我的媽媽開GAYBAR, ID: 10973\nMovie: 愛麗絲與夢幻島, ID: 11018\nMovie: 綠洲：數位經典版, ID: 11013\nMovie: 孤味, ID: 10477\nMovie: 聖荷西謀殺案, ID: 10990\nMovie: 入魔, ID: 10989\nMovie: 玩美女人 經典數位修復, ID: 10910\nMovie: 鬼滅之刃劇場版 無限列車篇, ID: 10816\nMovie: 親愛的房客, ID: 10707\nMovie: 地下弒的秘密, ID: 10984\nMovie: 藥頭大媽, ID: 10951\nMovie: 看不見的目擊者, ID: 10946\nMovie: 無價之保, ID: 10959\nMovie: 倒數反擊, ID: 10906\nMovie: 阿公當家, ID: 10914\nMovie: 刻在你心底的名字, ID: 10902\nMovie: 急先鋒, ID: 10443\nMovie: 紫羅蘭永恆花園電影版, ID: 10431\nMovie: 殺戮荒村, ID: 10903\nMovie: 消失的情人節, ID: 10870\nMovie: 中央車站：數位修復版, ID: 10907\nMovie: 海霧, ID: 10872\nMovie: 花木蘭, ID: 8632\nMovie: TENET天能, ID: 10433\nMovie: 聖雅各的天空, ID: 10877\nMovie: 看不見的證人, ID: 10873\nMovie: 可不可以，你也剛好喜歡我, ID: 10473\nMovie: 東京教父：4K數位修復版, ID: 10860\nMovie: 劇場版 新幹線變形機器人—來自未來的神速ALFA-X, ID: 10823\nMovie: 巴亞拉魔幻冒險, ID: 10851\nMovie: 怪胎, ID: 10733\nMovie: 魔王的女兒, ID: 10730\nMovie: 藍色恐懼：數位修復版, ID: 10775\nMovie: 角落小夥伴電影版：魔法繪本裡的新朋友, ID: 10647\nMovie: 一首搖滾上月球, ID: 4887\nMovie: 錢不夠用2, ID: 3026\n"
     ]
    }
   ],
   "source": [
    "# 查看目前上映那些電影，並擷取出其ID資訊\n",
    "url = 'https://movies.yahoo.com.tw/'\n",
    "resp = requests.get(url)\n",
    "resp.encoding = 'utf-8'\n",
    "\n",
    "soup = BeautifulSoup(resp.text, 'lxml')\n",
    "html = soup.find(\"select\", attrs={'name':'movie_id'})\n",
    "movie_item = html.find_all(\"option\", attrs={'data-name':re.compile('.*')})\n",
    "\n",
    "for p in movie_item:\n",
    "    print(\"Movie: %s, ID: %s\" % (p[\"data-name\"], p[\"value\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 指定你有興趣的電影其ID，然後查詢其放映地區資訊。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 參考前一個步驟中擷取到的ID資訊，並指定ID\n",
    "movie_id = 10910"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "放映地區: 台北市, 代號(area_id): 28\n"
     ]
    }
   ],
   "source": [
    "url = 'https://movies.yahoo.com.tw/api/v1/areas_by_movie_theater'\n",
    "payload = {'movie_id':str(movie_id)}\n",
    "\n",
    "# 模擬一個header\n",
    "headers = {\n",
    "    'authority': 'movies.yahoo.com.tw',\n",
    "    'method': 'GET',\n",
    "    'path': '/api/v1/areas_by_movie_theater?movie_id=' + str(movie_id),\n",
    "    'scheme': 'https',\n",
    "    'accept': 'application/json, text/javascript, */*; q=0.01',\n",
    "    'accept-encoding': 'gzip, deflate, br',\n",
    "    'accept-language': 'zh-TW,zh;q=0.9,en-US;q=0.8,en;q=0.7,zh-CN;q=0.6',\n",
    "    'cookie': 'rxx=9s3x2fws06.1g16irnc&v=1; _ga=GA1.3.2056742944.1551651301; GUC=AQEBAQFczFpdm0IfmwSB&s=AQAAACoo4N5D&g=XMsVBw; BX=4hkdk1decm57t&b=3&s=mr; _ga=GA1.4.2056742944.1551651301; nexagesuid=82843256dd234e8e91aa73f2062f8218; browsed_movie=eyJpdiI6IlJXWWtiSWJaZlNGK2MxQnhscnVUYWc9PSIsInZhbHVlIjoiMXRhMmVHRXRIeUNjc1RBWDJzdGYwbnlIQURmWGsrcjJSMzhkbkcraDNJVUNIZEZsbzU3amlFcVZ1NzlmazJrTGpoMjVrbHk1YmpoRENXaHZTOUw1TmI2ZTZVWHdOejZQZm16RmVuMWlHTTJLaTZLVFZZVkFOMDlTd1wvSGltcytJIiwibWFjIjoiZWQ2ZjA4MmVjZmZlYjlmNjJmYmY2NGMyMDI0Njc0NWViYjVkOWE2NDg0N2RhODMxZjBjZDhiMmJhZTc2MDZhYiJ9; avi=eyJpdiI6Im1NeWFJRlVRWDR1endEcGRGUGJUbVE9PSIsInZhbHVlIjoickRpU3JuUytmcGl6cjF5OW0rNU9iZz09IiwibWFjIjoiY2VmY2NkNzZmM2NhNjY5YzlkOTcyNjE5OGEyMzU0NWYxOTdmMDRkMDY3OWNmMmZjOTMxYjc5MjI5N2Q5NGE5MiJ9; cmp=t=1559391030&j=0; _gid=GA1.4.779543841.1559391031; XSRF-TOKEN=eyJpdiI6IkhpS2hGcDRQaHlmWUJmaHdSS2Q2bHc9PSIsInZhbHVlIjoiOUVoNFk4OHI1UUZmUWRtYXhza0MyWjJSTlhlZ3RnT0VGeVJPN2JuczVRMGRFdWt2OUlsamVKeHRobFwvcHBGM0dhU3VyMXNGTHlsb2dVM2l0U1hpUGxBPT0iLCJtYWMiOiJkZWU4YzJhNjAxMTY3MzE4Y2ExNWIxYmE1ZjE1YWZlZTlhOTcyYjc4M2RlZGY4ZWNjZDYyMTA2NGYwZGViMzc2In0%3D; m_s=eyJpdiI6InpsZHZ2Tk1BZ0dxaHhETml1RjBnUXc9PSIsInZhbHVlIjoiSkNGeHUranRoXC85bDFiaDhySTJqNkJRcWdjWUxjeVRJSHVYZ1wvd2d4bWJZUTUrSHVDM0lUcW5KNHdETFZ4T1lieU81OUhzc1VoUXhZcWk0UDZSQXVFdz09IiwibWFjIjoiYmJkMDJkMDhlODIzMzcyMWY4M2NmYWNjNGVlOWRjMDIwZmVmNzAyMjE3Yzg3ZGY3ODBkZWEzZTI4MTI5ZWNmOSJ9; _gat=1; nexagesd=10',\n",
    "    'dnt': '1',\n",
    "    'mv-authorization': '21835b082e15b91a69b3851eec7b31b82ce82afb',\n",
    "    'referer': 'https://movies.yahoo.com.tw/',\n",
    "    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36',\n",
    "    'x-requested-with': 'XMLHttpRequest',\n",
    "}\n",
    "    \n",
    "resp = requests.get(url, params=payload, headers=headers)\n",
    "#print(resp.json())  # 若有需要，列印出json原始碼\n",
    "\n",
    "# 這裡回傳的格式是JSON格式的資料，要解析JSON擷取資料\n",
    "for p in resp.json():\n",
    "    print('放映地區: {0}, 代號(area_id): {1}'.format(p['title'], p['area_id']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 指定你想要觀看的放映地區，查詢有上映電影的場次日期"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 指定放映地區\n",
    "area_id = 28"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "一月 2\n一月 3\n一月 4\n一月 5\n一月 6\n"
     ]
    }
   ],
   "source": [
    "# 向網站發送請求\n",
    "url = 'https://movies.yahoo.com.tw/movietime_result.html'\n",
    "payload = {'movie_id':str(movie_id), 'area_id':str(area_id)}\n",
    "resp = requests.get(url, params=payload)\n",
    "resp.encoding = 'utf-8'\n",
    "\n",
    "soup = BeautifulSoup(resp.text, 'lxml')\n",
    "movie_date = soup.find_all(\"label\", attrs={'for':re.compile(\"date_[\\d]\")})\n",
    "\n",
    "# 列印播放日期\n",
    "for date in movie_date:\n",
    "    print(\"%s %s\" % (date.p.string, date.h3.string))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 最後指定觀看的日期，查詢並列印出放映的電影院、放映類型(數位、3D、IMAX 3D...)、放映時間等資訊。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 選定要觀看的日期\n",
    "date = \"2021-01-02\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "[<ul class=\"area_time _c jq_area_time\" data-theater_name=\"光點華山電影館\" data-theater_schedules=\"https://movies.yahoo.com.tw/theater_result.html/id=152\" data-theater_url=\"http://www.spot-hs.org.tw/\" id=\"theater_id_152\">\n<li class=\"adds\">\n<a href=\"https://movies.yahoo.com.tw/theater_result.html/id=152\">光點華山電影館</a>\n<span>02-23940622</span>\n</li>\n<li class=\"taps\">\n<span class=\"tapR\">數位</span> </li>\n<li class=\"time _c\">\n<div class=\"input_picker jq_input_picker\">\n<input class=\"gabtn\" data-ga=\"['電影頁_時刻表', '電影頁_時刻表_場次', '玩美女人 經典數位修復']\" data-movie_date=\"01.02\" data-movie_time=\"15:15\" data-movie_title=\"玩美女人 經典數位修復\" data-movie_type=\"數位\" data-news=\"news\" id=\"74756680\" name=\"schedule_list\" type=\"radio\" value=\"2021-01-02 15:15:00\"/>\n<label class=\"select\" for=\"74756680\">15:15</label>\n</div>\n</li>\n</ul>]\n"
     ]
    }
   ],
   "source": [
    "# 向網站發送請求，獲取上映的電影院及時間資訊\n",
    "url = \"https://movies.yahoo.com.tw/ajax/pc/get_schedule_by_movie\"\n",
    "payload = {'movie_id':str(movie_id),\n",
    "           'date':date,\n",
    "           'area_id':str(area_id),\n",
    "           'theater_id':'',\n",
    "           'datetime':'',\n",
    "           'movie_type_id':''}\n",
    "\n",
    "resp = requests.get(url, params=payload)\n",
    "#print(resp.json()['view'])  # 若有需要，列印出json原始碼\n",
    "\n",
    "soup = BeautifulSoup(resp.json()['view'], 'lxml')\n",
    "html = soup.find_all(\"ul\", attrs={'data-theater_name':re.compile(\".*\")})\n",
    "print(html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "電影院: 光點華山電影館\n類型: 數位\n語言: Unknown\n日期時間:\n\t 2021-01-02 15:15:00\n"
     ]
    }
   ],
   "source": [
    "\n",
    "\n",
    "  # 試著從上一步驟回傳的電影院資料中，擷取電影院名稱、影片放映類型以及時間表\n",
    "  \n",
    "  for i in html:\n",
    "    print(\"電影院: {}\".format(i['data-theater_name']))\n",
    "    movie_info = i.find_all(\"li\", attrs={\"class\":\"taps\"})\n",
    "    for j in movie_info:\n",
    "        info = j.find_all('span')\n",
    "        print(\"類型: {}\".format(info[0].string))\n",
    "        try:\n",
    "            print(\"語言: {}\".format(info[1].string))\n",
    "        except:\n",
    "            print(\"語言: {}\".format('Unknown'))\n",
    "    movie_time = i.find_all(\"input\")\n",
    "    print('日期時間:')\n",
    "    for time in movie_time:\n",
    "        print('\\t', time['value'])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3-final"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}